Quantification of signal in stained aggregates

Abstract

The present disclosure provides systems and methods for detecting and evaluating signals corresponding to one or more biomarkers in a biological sample that has been stained to detect the presence of protein and / or nucleic acid biomarkers.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 671,825, filed May 15, 2018, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Digital pathology involves scanning entire histopathology or cytopathology slides into digital images that can be interpreted on a computer screen. These images will then be processed by an imaging algorithm or interpreted by a pathologist. In order to examine tissue sections (which are actually transparent), colored histochemical stains that selectively bind to cellular components are used to prepare tissue sections. Clinicians or computer-aided diagnosis (CAD) algorithms use color-enhanced or stained cell structures to identify morphological markers of disease and treat accordingly. Observing the assay can enable a variety of processes, including disease diagnosis, evaluation of treatment response, and the development of new drugs to combat disease.

[0004] Immunohistochemistry (IHC) slide staining can be used to identify proteins in cells of tissue sections and is therefore widely used in studies of different types of cells, such as cancerous cells and immune cells in biological tissues. Therefore, IHC staining can be used in studies to understand the distribution and location of differentially expressed biomarkers of immune cells (such as T cells or B cells) in cancerous tissues for immune response studies. For example, tumors often contain infiltrates of immune cells that can prevent the development of tumors or facilitate the outgrowth of tumors.

[0005] In situ hybridization (ISH) can be used to find the existence of genetic abnormality or symptom, for example, when observed under a microscope, morphologically, it is shown as amplification of oncogenes specifically in malignant cells. Unique nucleic acid sequences occupy precise positions in chromosomes, cells and tissues, and in situ hybridization allows the existence, absence and / or amplification state of such sequences to be determined without seriously damaging the sequence. ISH uses DNA or RNA probe molecules with markers antisense to the target gene sequence or transcript to detect or locate the target nucleic acid target gene in a cell or tissue sample. ISH is carried out by exposing the cell or tissue sample fixed on a slide to a labeled nucleic acid probe, which can be hybridized with a given target gene specificity in a cell or tissue sample. By exposing the cell or tissue sample to a plurality of nucleic acid probes labeled with a plurality of different nucleic acid tags, several target genes can be analyzed simultaneously. By utilizing markers with different emission wavelengths, a single target cell or tissue sample can be analyzed simultaneously with multicolor analysis in a single step. Summary of the invention

[0006] In the past, clinical evaluation of proteins and nucleic acids in tissues relied on in situ immunoenzymatic assays. For example, detection of B-cell clonality aids in the diagnosis of B-cell lymphomas, and clonality assessment can be accomplished by evaluating KAPPA and LAMBDA light chain expression. Fig. 8A As shown, tonsil tissue stained for KAPPA mRNA can be detected with silver (Ag) (in black) and LAMBDA mRNA can be detected with Tyramide-sulforhodamine (in purple). The presence of the signal of interest appears as tiny spots (e.g., discrete dots), and these spots can accumulate to form larger areas of aggregated signal (hereinafter referred to as "signal aggregate plaques" or "blobs"), depending on the number of copies of the mRNA in the cell. For example, plasma cells have approximately 100,000 copies of mRNA per cell, so the signal in these cells may appear as plaques.

[0007] Quantitative ISH and IHC analysis is useful in clinical evaluation; however, due to limitations of existing technologies, it has not been widely performed. An automated technique for estimating the amount of signal in signal-aggregated plaques can facilitate enhanced clinical interpretation of stained biological samples and enable samples to be interpreted more quickly and accurately. In view of the above, the applicant has developed an image analysis system and method that can detect and quantify the amount of protein or nucleic acid signals present in a stained sample.

[0008] In one aspect of the present disclosure, a method for estimating the amount of signal corresponding to at least one biomarker in an image of a biological sample comprises: detecting isolated spots in a first image (e.g., an unmixed image channel image corresponding to a signal from a first biomarker); deriving an optical density value of a representative isolated spot (e.g., based on a calculated signal feature or characteristic from the detected isolated spot); and estimating the number of predicted spots in a signal aggregate in each of a plurality of sub-regions based on the derived optical density value of the representative isolated spot. In some embodiments, the method further comprises calculating the total number of spots in a sub-region by combining the number of detected isolated spots and the estimated number of predicted spots in a signal aggregate in each of the sub-regions.

[0009] In some embodiments, the number of predicted spots in the signal aggregates in each of the plurality of subregions is estimated by calculating the total optical density of the signal aggregates in one of the subregions divided by the derived optical density of the representative isolated spots. In some embodiments, the total optical density of the signal aggregates in one of the subregions is a measured value. In some embodiments, the representative isolated spot optical density is derived by the following steps: (i) generating a histogram based on the calculated descriptive signal features of all detected isolated spots; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measurement value based on the extracted size parameters; and (iv) multiplying the extracted intensity parameters by the calculated area measurement value. In some embodiments, the calculated descriptive signal features are derived by characterizing the signal features (e.g., mean, standard deviation, full width at half maximum, or size) from the detected isolated spots. In some embodiments, the calculated descriptive signal features include spot intensity, spot blur, spot circularity, and spot size. In some embodiments, the descriptive signal features are calculated in image blocks of predetermined size (eg, 7 x 7 pixels) using a one-dimensional Gaussian function fitting method.

[0010] In some embodiments, the size parameter is the mode of the radii from a full width at half maximum histogram. In some embodiments, the size parameter is the mean of the radii from a full width at half maximum histogram. In some embodiments, the intensity parameter is a uniform intensity metric. In some embodiments, the intensity parameter is a non-uniform intensity metric. In some embodiments, the uniform intensity metric is the mode of the intensity values ​​derived from the intensity histogram. In some embodiments, the uniform intensity metric is the mean of the intensity values ​​derived from the intensity histogram. In some embodiments, the uniform intensity metric is a weighted average intensity derived from the intensity histogram. In some embodiments, the non-uniform intensity metric is derived from a blurriness histogram, which is calculated based on the blurriness signal characteristics of each detected isolated spot.

[0011] In some embodiments, the estimation of the number of predicted spots is performed using a second image, the second image being substantially free of signals corresponding to the detected isolated spots. In some embodiments, the second image is a residual image, as described herein. In some embodiments, the second image is derived by the following steps: (i) generating a foreground segmentation mask based on the isolated spots detected in the first image; and (ii) filtering the first image with the generated foreground segmentation mask. In some embodiments, the second image is generated by the following steps: (i) generating an isolated spot image that includes only the detected isolated spots from the first image; and (ii) subtracting the isolated spot image from the first image. In some embodiments, the second region is segmented into a plurality of sub-regions.

[0012] In some embodiments, each of the plurality of sub-regions has pixels that are substantially uniform in at least one of biomarker staining intensity, biomarker staining presence, and local biomarker staining texture. In some embodiments, each of the plurality of sub-regions is a superpixel. In some embodiments, the superpixels are derived by: (i) grouping pixels using local k-means clustering; and (ii) merging small isolated regions into the nearest large superpixels using a connected component algorithm. In some embodiments, superpixels are generated using a 70x 70 pixel size and a regularization parameter of about 10. In some embodiments, segmentation is performed on the second image described above. In some embodiments, the sub-regions are derived by overlaying a sampling grid onto the second image, the sampling grid defining a non-overlapping region having a predetermined size and shape (e.g., a simple object shape, such as a square, a circle, etc.), such as an area of ​​about 70 pixels by 70 pixels to about 200 pixels by about 200 pixels. In some embodiments, the method further includes repeating at least some of the aforementioned steps to estimate a signal corresponding to a second biomarker.

[0013] In some embodiments, the method of estimating a signal corresponding to a second biomarker includes: detecting isolated spots in an unmixed image channel image corresponding to a signal from a second biomarker; deriving an optical density value for a representative isolated spot (e.g., based on a calculated signal feature from the detected isolated spot); and estimating the number of predicted spots in a signal aggregate in each of a plurality of sub-regions based on the calculated descriptive signal feature of the detected isolated spot. In some embodiments, the estimation of the number of predicted spots corresponding to the second biomarker is performed using an image derived from an unmixed image channel image having isolated spots corresponding to the second biomarker and signal aggregates, wherein the signal from the detected isolated spot is removed.

[0014] In some embodiments, the biological sample is stained to detect the presence of one or more protein biomarkers, such as a HER2 protein biomarker. In some embodiments, the biological sample is stained to detect the presence of one or more nucleic acid biomarkers. In some embodiments, the biological sample is stained to detect the presence of one or more mRNA sequences (e.g., one or more KAPPA mRNA sequences and / or one or more LAMBDA mRNA sequences). In some embodiments, each of the one or more KAPPA mRNA sequences is detected using the same reporter portion (e.g., silver). In some embodiments, each of the one or more LAMBDA mRNA sequences is detected using the same reporter portion (e.g., purple).

[0015] In some embodiments, the method further includes storing in a database an estimated number of predicted spots and detected isolated spots in each of a plurality of generated subregions of the entire whole slide image. In some embodiments, the method further includes using the stored data to generate an overlay that can be superimposed on an input image (e.g., a whole slide image or a portion thereof). In some embodiments, the method further includes generating an image overlay that independently visualizes the number of detected isolated spots and the estimated number of predicted spots in each subregion. In some embodiments, the method further includes generating an image overlay that visualizes the total amount of signal based on the estimated number of predicted spots and the number of detected isolated spots in each subregion. In some embodiments, the visualization includes information associated with more than one biomarker.

[0016] In another aspect of the present disclosure, a method for estimating the amount of a signal corresponding to at least one biomarker in an image of a biological sample comprises: detecting isolated spots in an input image (e.g., a single-channel image or an unmixed image channel image derived from a multi-channel image); calculating descriptive signal features based on the detected isolated spots; and estimating the number of predicted spots in signal aggregates in each of a plurality of sub-regions in a second image based on the calculated descriptive signal features of the detected isolated spots. In some embodiments, the method further comprises calculating the total number of spots in a sub-region by combining the number of detected isolated spots and the estimated number of predicted spots in signal aggregates in each of the sub-regions.

[0017] In some embodiments, the method further includes storing in a database an estimated number of predicted spots and / or detected isolated spots in each of the plurality of generated sub-regions and / or in the whole slide image. In some embodiments, the number of predicted spots in the signal aggregate in each of the plurality of sub-regions is estimated by the following steps: (i) deriving an optical density of a representative isolated spot, the optical density of the representative isolated spot being derived from the calculated descriptive signal feature; and (ii) calculating the quotient of the total optical density of the signal aggregate in one of the sub-regions of the second image divided by the derived optical density of the representative isolated spot. In some embodiments, the optical density of the representative isolated spot is derived by the following steps: (i) generating a histogram based on each calculated descriptive signal feature; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measurement value based on the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measurement value. In some embodiments, the calculated descriptive signal feature is derived by characterizing the signal features (e.g., mean, standard deviation, full width at half maximum, or size) from the detected isolated spots. In some embodiments, the intensity parameter is selected from a uniform intensity metric and a non-uniform intensity metric. In some embodiments, the calculated descriptive signal features are size, blurriness, intensity, and circularity.

[0018] In another aspect of the disclosure, a method for estimating the amount of signal corresponding to at least one biomarker in an image of a biological sample comprises: detecting a signal corresponding to a single gene copy in a first image (e.g., an unmixed image channel image corresponding to a signal from a first biomarker); deriving an optical density value of a representative signal corresponding to the single gene copy based on a signal feature calculated from all detected signals corresponding to the single gene copy in the first image; and estimating the amount of predicted signal (corresponding to the plurality of gene copies) in a signal patch within each of a plurality of sub-regions in a second image (e.g., an image in which detected isolated spots are substantially not signals within signal aggregates) based on the derived optical density value of the representative signal corresponding to the single gene copy. In some embodiments, the optical density of the representative signal corresponding to the single gene copy is derived by: (i) generating a histogram based on at least two calculated descriptive signal features; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measure value based on the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measure value. In some embodiments, the intensity parameter is selected from a uniform intensity measure and a non-uniform intensity measure. In some embodiments, the calculated descriptive signal features are size, blurriness, intensity, and circularity. In some embodiments, the method further comprises generating an overlay based on the estimated number of predicted spots and / or the detected isolated spots. In some embodiments, the method further comprises repeating at least some of the foregoing steps for another gene (by way of example only, performing the method for KAPPA mRNA for the first time and for LAMBDA mRNA for the second time).

[0019] In another aspect of the disclosure, is a method of estimating the amount of signal corresponding to at least one biomarker in an image of a biological sample, comprising: detecting isolated spots in a first image (e.g., an unmixed image channel image corresponding to a signal from a first biomarker); deriving an optical density value of a representative isolated spot based on a calculated signal feature from the detected isolated spot; generating a residual image derived from the first image substantially free of signal corresponding to the detected isolated spot; segmenting the residual image into a plurality of sub-regions; and estimating the number of predicted spots in a signal aggregate in each of the plurality of sub-regions based on the derived optical density of the representative isolated spot. In some embodiments, the first image is stained to detect the presence of at least one nucleic acid biomarker. In other embodiments, the first image is stained to detect the presence of at least one protein biomarker. In some embodiments, the calculated descriptive signal feature is derived by characterizing a signal feature (e.g., mean, standard deviation, full width at half maximum, or size) from the detected isolated spots. In some embodiments, the calculated descriptive signal features include spot intensity, spot blurriness, spot circularity, and spot size.

[0020] In another aspect of the present disclosure, a system for estimating the amount of one or more signals in a biological sample, the one or more signals corresponding to a staining target (e.g., protein, nucleic acid, etc.) in the biological sample, the system comprising: (i) one or more processors, and (ii) a memory connected to the one or more processors, the memory for storing computer executable instructions, when the instructions are executed by the one or more processors, causing the system to perform operations, the operations comprising: detecting isolated spots in a first image (e.g., an unmixed image channel image corresponding to a signal from a first biomarker); deriving an optical density value of a representative isolated spot based on a calculated signal feature from the detected isolated spot; and estimating the number of predicted spots in a signal aggregate in each of a plurality of sub-regions in a second image (e.g., an image in which the detected isolated spot does not substantially belong to a signal within the signal aggregate) based on the derived optical density value of the representative isolated spot. In some embodiments, the system further comprises instructions for calculating the total number of spots in a sub-region by combining the number of detected isolated spots and the estimated number of predicted spots in the signal aggregate in each of the sub-regions.

[0021] In some embodiments, the number of predicted spots in the signal aggregates in each of the plurality of subregions is estimated by calculating the quotient of the total optical density of the signal aggregates in one of the subregions of the second image divided by the derived optical density of the representative isolated spots. In some embodiments, the representative isolated spot optical density is derived by the following steps: (i) generating a histogram according to each calculated descriptive signal feature; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measurement value according to the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measurement value. In some embodiments, the calculated descriptive signal feature is derived from the signal feature of each detected isolated spot, including mean, standard deviation, half-height full width or size. In some embodiments, the size parameter is the mode of the radius from the half-height full width histogram. In some embodiments, the intensity parameter is a uniform intensity metric or a non-uniform intensity metric. In some embodiments, the uniform intensity metric is the mode of the intensity values ​​derived from the intensity histogram. In some embodiments, the uniform intensity metric is the mean of the intensity values ​​derived from the intensity histogram. In some embodiments, the uniform intensity metric is a weighted average intensity derived from an intensity histogram. In some embodiments, the non-uniform intensity metric is derived from a blurriness histogram calculated from blurriness signal characteristics of each detected isolated blob.

[0022] In some embodiments, the calculated descriptive signal features include spot intensity, spot blur, spot circularity, and spot size. In some embodiments, the descriptive signal features are calculated in image blocks of predetermined size (e.g., 7x7 pixels) using a one-dimensional Gaussian function fitting method.

[0023] In some embodiments, the second image is derived by: (i) generating a foreground segmentation mask based on the isolated spots detected in the first image; and (ii) filtering the first image with the generated foreground segmentation mask. In some embodiments, the second image is generated by: (i) generating an isolated speckle image that includes only the detected isolated spots from the first image; and (ii) subtracting the isolated speckle image from the first image.

[0024] In some embodiments, each of the plurality of sub-regions has pixels that are substantially uniform in at least one of biomarker staining intensity, biomarker staining presence, and local biomarker staining texture. In some embodiments, each of the plurality of sub-regions is a superpixel. In some embodiments, the superpixels are derived by: (i) grouping pixels using local k-means clustering; and (ii) merging small isolated regions into the nearest large superpixel using a connected component algorithm. In some embodiments, the sub-regions are derived by overlaying a sampling grid onto the second image, the sampling grid defining a non-overlapping region having a predetermined size and shape.

[0025] In another aspect of the present disclosure, a system for estimating the amount of one or more signals in a biological sample, wherein the one or more signals correspond to a stained target (e.g., protein, nucleic acid, etc.) in the biological sample, the system comprising: (i) one or more processors, and (ii) a memory connected to the one or more processors, wherein the memory is used to store computer executable instructions, which, when executed by the one or more processors, causes the system to perform operations, the operations comprising: detecting isolated spots in a first image (e.g., an unmixed image channel image corresponding to a signal from a first biomarker); calculating descriptive signal features based on all detected isolated spots; and estimating the number of predicted spots in a signal aggregate in each of a plurality of sub-regions in a second image (e.g., an image in which the detected isolated spots do not substantially belong to a signal within the signal aggregate) based on the calculated descriptive signal features. In some embodiments, the number of predicted spots in the signal aggregate in each of the plurality of subregions is estimated by: (i) deriving an optical density of a representative isolated spot, the optical density of the representative isolated spot being derived from the calculated descriptive signal feature; and (ii) calculating the quotient of the total optical density of the signal aggregate in one of the subregions of the second image divided by the derived optical density of the representative isolated spot. In some embodiments, the optical density of the representative isolated spot is derived by: (i) generating a histogram based on each calculated descriptive signal feature; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measurement value based on the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measurement value. In some embodiments, the intensity parameter is a uniform intensity measure or a non-uniform intensity measure. In some embodiments, the size parameter is the mode of the radius from a full width at half maximum histogram. In some embodiments, the first image is a first unmixed image channel image corresponding to a signal from a first biomarker. In some embodiments, the calculated descriptive signal features are derived from the signal features of each detected isolated spot, including mean, standard deviation, half-height full width or size. In some embodiments, the calculated descriptive signal features include spot intensity, spot blur, spot circularity, and spot size. In some embodiments, the descriptive signal features are calculated in an image block of a predetermined size using a one-dimensional Gaussian function fitting method. In some embodiments, the estimation of the number of predicted spots in the signal aggregate is performed using a second image that is substantially free of signals corresponding to the detected isolated spots.

[0026] In another aspect of the disclosure, a non-transitory computer-readable medium storing instructions for estimating the amount of different signals in a stained biological sample comprises: detecting isolated spots in a first image (e.g., an unmixed image channel image corresponding to a signal from a first biomarker); deriving an optical density value for a representative isolated spot based on a calculated signal feature from the detected isolated spot; and estimating the number of predicted spots in a signal aggregate in each of a plurality of sub-regions in a second image (e.g., an image in which the detected isolated spot does not substantially correspond to a signal within the signal aggregate) based on the derived optical density value for the representative isolated spot. In some embodiments, the optical density of the representative isolated spot is derived by: (i) generating a histogram based on each of the signal features from the detected isolated spot; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measurement value based on the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measurement value. In some embodiments, the isolated spot is detected using a shape detector having a disk-like shape. In some embodiments, the sub-region is a superpixel. In some embodiments, the superpixel is derived by the following steps: (i) grouping pixels using local k-means clustering; and (ii) merging small isolated regions into the nearest large superpixel using a connected component algorithm. In some embodiments, the number of predicted spots in the signal aggregate in each of the multiple subregions is estimated by calculating the quotient of the total optical density of the signal aggregate in one of the subregions divided by the derived optical density of the representative isolated spot. In some embodiments, the total optical density of the signal aggregate is a measured value. In some embodiments, the signal corresponds to at least two nucleic acid biomarkers. In some embodiments, the signal corresponds to at least one protein biomarker. In some embodiments, the non-transitory computer-readable medium further includes instructions for unmixing an input image. In some embodiments, the non-transitory computer-readable medium further includes instructions for segmenting an input image (e.g., an unmixed image channel image) into a plurality of subregions. In some embodiments, the non-transitory computer-readable medium further includes instructions for generating superpixels. In some embodiments, the non-transitory computer-readable medium further includes instructions for generating a second image that substantially does not include signals from the detected isolated spots. In some embodiments, the non-transitory computer readable medium further comprises instructions for generating a visualization and overlaying the generated visualization onto the input image. In some embodiments, the non-transitory computer readable medium further comprises instructions for combining the generated data from at least two image patches, each image patch corresponding to a portion of the whole slide image, and combining the data from the at least two patches. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to fully understand the features of the present disclosure, reference is made to the accompanying drawings. In the accompanying drawings, the same reference numerals are used throughout to identify the same elements.

[0028] According to some embodiments, Figure 1 A representative digital pathology system including an image acquisition device and a computer system is shown.

[0029] Figure 2 Various modules are presented that may be used in a digital pathology system or in a digital pathology workflow according to some embodiments.

[0030] Figure 3A A flowchart is presented that provides an overview of the steps for estimating the amount of one or more signals corresponding to one or more biomarkers, respectively, in a whole slide image or any portion thereof, and further illustrates optional steps for generating an overlay or storing the calculated data, according to some embodiments.

[0031] Figure 3B A flowchart is presented that illustrates steps for estimating the amount of one or more signals corresponding to one or more biomarkers, respectively, in a whole slide image or any portion thereof, and further illustrates optional steps for generating an overlay or storing the calculated data, according to some embodiments.

[0032] Figure 3C A flow chart illustrating a non-limiting method for estimating the amount of one or more signals corresponding to one or more protein biomarkers is presented according to some embodiments.

[0033] Figure 3D A flow chart illustrating a non-limiting method for estimating the amount of one or more signals corresponding to one or more nucleic acid biomarkers is presented according to some embodiments.

[0034] Figure 3E A flow chart illustrating a non-limiting method for estimating the amount of at least two signals corresponding to at least two biomarkers in a whole slide image or any portion thereof is presented, according to some embodiments.

[0035] Figure 4 A flow chart illustrating the generation of data from detected isolated blobs is presented according to some embodiments.

[0036] Figure 5 A flow chart is presented that illustrates the steps of calculating the optical density value of a representative isolated spot according to some embodiments.

[0037] Fig. 6AProvided is a portion of a whole slide image stained for KAPPA mRNA (silver or black / reporter, 601) in an in situ hybridization assay with hematoxylin counterstain.

[0038] Figure 6B An example of an image of image channels after demixing is provided, showing only the signal corresponding to KAPPA mRNA (silver / black / reporter, 602).

[0039] Figure 6C An example of demixed image channels is provided, showing the hematoxylin channel (eg, blue, 603).

[0040] Fig. 7A Provided is a portion of a whole slide image stained with hematoxylin counterstain for the LAMBDA mRNA probe (purple / reporter, 701) in an in situ hybridization assay.

[0041] Figure 7B An example of an image of image channels after demixing is provided, showing only the signal corresponding to LAMBDA mRNA (purple / reporter, 702).

[0042] Figure 7C An example of demixed image channels is provided, showing the hematoxylin channel (eg, blue, 703).

[0043] Fig. 8A A portion of a whole slide image stained for KAPPA mRNA (silver or black / reporter, 801) and LAMBDA mRNA probes (purple / reporter, 802) in an in situ hybridization assay is provided.

[0044] Figure 8B An example of an image of image channels after demixing is provided, showing only the signal corresponding to KAPPA mRNA (black / reporter, 803).

[0045] Figure 8C An example of an image of image channels after demixing is provided, showing only the signal corresponding to LAMBDA mRNA (purple / reporter, 804).

[0046] Fig.8D An example of demixed image channels is provided, showing the hematoxylin channel (eg, blue, 805).

[0047] Fig.9A An example of a portion of a whole slide image stained in an in situ hybridization assay is provided.

[0048] Fig. 9B Shows the Fig.9A The result of decomposing into individual channels (silver / black channels).

[0049] Fig. 9C Shown in Fig. 9B Point detection results on the unmixed image channel image (spot channel image).

[0050] Fig.9D The plaque channel image is shown, whereby Fig. 9C The signal of the detected isolated spots is substantially removed (eg, subtracted).

[0051] Fig.9E and 9F Shown in Fig. 9C The derived x,y positions of isolated blobs detected in the blob channel image.

[0052] Figure 9G and 9H Shown superimposed on Fig. 9C The derived seed center on each detected isolated blob in the blob channel image of .

[0053] Fig.9I and 9J Shown superimposed on Fig.9A Overlay of detected isolated spots on a portion of a whole slide image.

[0054] Fig. 10A A portion of a whole slide image used as an input image according to some embodiments of the present disclosure is shown, wherein a biological sample in the whole slide image is stained to detect the presence of a protein biomarker (eg, HER2).

[0055] Fig. 10B and Fig. 10C Shown from Fig. 10A Spot detection results of protein biomarkers.

[0056] Fig.11 Parameters that can be derived after extracting the characteristics of detected isolated spots using a one-dimensional Gaussian fitting function on data derived from a tile around a spot signal according to some embodiments of the present disclosure are shown. Panel (i) shows a window cropped out of a spot of interest on an input image; panel (ii) shows the optical density of the cropped window from panel (i) displayed in two dimensions; panel (iii) shows the optical density of the cropped window from panel (i) displayed in three dimensions; panel (iv) shows a three-dimensional graph of a Gaussian function, where the parameters of the input optical density are theoretically determined using a Gaussian fitting method to fit an ideal Gaussian function; and panel (v) shows a one-dimensional Gaussian function.

[0057] Fig.12Results of blob detection and calculated parameters generated from each detected isolated blob are shown, wherein the calculated parameters are grouped into several different histograms, according to some embodiments of the present disclosure.

[0058] Fig.13 Generating a residual image from an unmixed image channel image and a speckle channel image is shown according to some embodiments of the present disclosure, wherein the residual image is substantially free of signals from detected isolated speckles shown in the speckle channel image according to some embodiments of the present disclosure.

[0059] Fig.14A Shown is a portion of a whole slide image stained for at least one biomarker according to some embodiments of the present disclosure.

[0060] Fig. 14B Some embodiments according to the present disclosure are shown Fig.14A Segmentation result of a portion of a whole slide image.

[0061] Fig. 14C The result of overlaying the generated segmented fragments onto a portion of a whole slide image according to some embodiments of the present disclosure is shown.

[0062] Fig.15 An alternative flow chart is presented that illustrates the steps of estimating a signal corresponding to at least one biomarker in a whole slide image according to some embodiments of the present disclosure.

[0063] Fig.16A and 16B Shown are the results of overlaying detected isolated spots and estimated signal amounts in signal aggregates onto a whole slide image according to some embodiments of the present disclosure.

[0064] Fig.17A Shown are the results of cell nucleus segmentation according to some embodiments of the present disclosure.

[0065] Fig. 17B The nuclear boundaries of each cell are shown according to some embodiments of the present disclosure.

[0066] Fig.18 The results of expert observers and algorithms validated on 31 fields of view are shown (R 2 =0.99, CCC=0.99). The figure shows a good distribution of spot counts with a slope of 0.98 and an intercept of -2.3. The total spot counts identified by the observers (126,588) and the algorithm (124,040) are shown in the accompanying table. DETAILED DESCRIPTION

[0067] It should also be understood that in any method claimed herein that includes more than one step or action, the order of the steps or actions of the method is not necessarily limited to the order in which the steps or actions of the method are recited unless explicitly stated to the contrary.

[0068] As used herein, the singular terms "a", "an", and "the" include plural referents unless the context clearly indicates otherwise. Similarly, the word "or" is intended to include "and" unless the context clearly indicates otherwise. The term "comprising" is defined as inclusive, such that "comprising A or B" means comprising A, B, or A and B.

[0069] As used herein in the specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when the listed items are separated, "or" or "and / or" should be interpreted as inclusive, i.e., including multiple elements or at least one element in the listed elements, but also including more than one element, and (optionally) other unlisted items. Only when it is clearly indicated that they are contradictory, otherwise, such as "only one" or "just one" or "consisting of..." when used in the claims will refer to an element that just includes many elements or a list of elements. In general, when the term "or" as used herein is followed by exclusive terms such as "one of the two", "one of", "only one of" or "just one of", it should only be interpreted as indicating exclusive alternatives (that is, "one or the other but not two"). Its ordinary meaning when "consisting of substantially..." is used in the claims is as used in the field of patent law.

[0070] The terms "including," "comprising," "having," and the like are used interchangeably and have the same meaning. Similarly, the terms "including," "comprising," "having," and the like are used interchangeably and have the same meaning. Specifically, the definition of each term is consistent with the general definition of "comprising" in U.S. patent law and is therefore interpreted as an open term meaning "at least the following," and is also interpreted as not excluding additional features, limitations, aspects, etc. Thus, for example, "a device having components a, b, and c" means that the device includes at least components a, b, and c. Similarly, the phrase "a method involving steps a, b, and c" means that the method includes at least steps a, b, and c. In addition, although the steps and processes may be outlined in a particular order herein, those skilled in the art will recognize that the ordering of the steps and processes may vary.

[0071] As used in this specification and claims, with respect to a list of one or more elements, the phrase "at least one" should be understood to mean at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one element of each element specifically listed in the list of elements, and does not exclude any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than the elements specifically represented in the list of elements to which the phrase "at least one" refers, whether related or unrelated to those specifically represented. Thus, as a non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently "at least one of A and / or B") may refer to at least one A, optionally including more than one A, without the presence of B (and optionally including elements other than B) in one embodiment; to at least one B, optionally including more than one B, without the presence of A (and optionally including elements other than A) in another embodiment; to at least one A, optionally including more than one A and at least one B, optionally including more than one B (and optionally including other elements) in yet another embodiment; and so on.

[0072] As used herein, the terms "biological sample", "tissue sample", "specimen" and the like refer to any sample including biological molecules (e.g., proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans; beasts, such as cats, dogs, horses, cows, and pigs; and laboratory animals, such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (e.g., tissue sections and needle biopsies of tissues), cell samples (e.g., cytological smears, such as Pap smears or blood smears or cell samples obtained by microdissection), or cell components, fragments, or organelles (e.g., obtained by lysing cells and separating their components by centrifugation or other means). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, earwax, breast milk, vaginal fluid, saliva, swab (e.g., buccal swab), or any material containing biomolecules from the first biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor (e.g., a homogenized or liquefied sample) or a portion of a tumor obtained from a subject.

[0073] As used herein, the term "blob" refers to an area in a digital image that differs from surrounding areas in properties such as brightness or color. For example, a blob can be a group of adjacent pixels with a specific range of intensity values.

[0074] A "foreground segmentation mask" is, for example, an image mask created by a segmentation algorithm that allows one or more patches of pixels (serving as "foreground pixels") to be separated from other pixels (constituting the "background"). For example, a foreground segmentation mask may be generated by a nuclear segmentation algorithm, and application of the foreground segmentation mask on an image depicting a tissue section may allow identification of nuclear patches in the image.

[0075] As used herein, the term "image data" encompasses raw image data or pre-processed image data acquired from a biological tissue sample (such as by means of an optical sensor or a sensor array). In particular, the image data may include a pixel matrix.

[0076] As used herein, the terms "image", "image scan" or "scanned image" include raw image data acquired from a biological tissue sample, such as by an optical sensor or sensor array, or pre-processed image data. Specifically, the image data may include a pixel matrix.

[0077] As used herein, the term "multi-channel image" or "multiplexed image" includes digital images acquired from a biological tissue sample, in which different biological structures, such as cell nuclei and tissue structures, are simultaneously stained with specific fluorescent dyes, quantum dots, chromogens, etc., each of which emits fluorescence or is detectable in a different spectral band, thereby constituting one of the channels of the multi-channel image.

[0078] As used herein, the term "probe" or "oligonucleotide probe" refers to a nucleic acid molecule used to detect a complementary nucleic acid target gene.

[0079] As used herein, the term "slide" refers to any substrate of any suitable size (e.g., a substrate made in whole or in part of glass, quartz, plastic, silicon, etc.), and more specifically to a "microscope slide", such as a standard 3-inch by 1-inch microscope slide or a standard 75 mm by 25 mm microscope slide. Examples of biological samples that can be placed on a slide include, but are not limited to, cytological smears, thin tissue sections (e.g., from biopsies), and biological sample arrays, such as tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof. Thus, in one embodiment, tissue sections, DNA samples, RNA samples, and / or proteins are placed at specific locations on a slide. In some embodiments, the term slide can refer to SELDI and MALDI chips, as well as silicon wafers.

[0080] As used herein, the term "specific binding entity" refers to a member of a specific binding pair. A specific binding pair is a pair of molecules characterized in that they bind to each other to the substantial exclusion of binding to other molecules (e.g., the binding constant of the specific binding pair is at least 10"3M-1, 10"4M-1, or 10"5M-1 greater than the binding constant of either member of the binding pair to other molecules in the biological sample). Specific examples of specific binding moieties include specific binding proteins (e.g., antibodies, lectins, avidin such as streptavidin and protein A). Specific binding moieties may also include molecules (or portions thereof) that are specifically bound by such specific binding proteins.

[0081] The terms "staining", "stained", etc. as used herein generally refer to any treatment of a biological sample that detects and / or distinguishes the presence, location and / or quantity (such as concentration) of a specific molecule (such as lipid, protein or nucleic acid) or a specific structure (such as normal or malignant cells, cytosol, nucleus, Golgi apparatus or cytoskeleton) in a biological sample. For example, staining can provide a contrast between a specific molecule or a specific cell structure and the surrounding part of the biological sample, and the intensity of the staining can provide a measure of the amount of a specific molecule in the sample. Staining can be performed not only with a bright field microscope, but also with other observation tools, such as phase contrast microscopes, electron microscopes and fluorescent microscopes, to help observe molecules, cell structures and organisms. Some staining performed by the system can be used to visualize cell outlines. Other staining performed by the system may rely on certain cell components (such as molecules or structures) being stained, while other cell components will not or will hardly be stained. Examples of types of staining methods performed by the system include, but are not limited to, histochemical methods, immunohistochemical methods, and other methods based on intermolecular reactions (including non-covalent binding interactions), such as hybridization reactions between nucleic acid molecules. Specific staining methods include, but are not limited to, primary staining methods (eg, H&E staining, Papanicolaou staining, etc.), enzyme-linked immunohistochemistry methods, and in situ RNA and DNA hybridization methods, such as fluorescence in situ hybridization (FISH).

[0082] As used herein, the term "target" refers to any molecule whose presence, location and / or concentration has been or can be determined. Examples of target molecules include proteins, nucleic acid sequences, and haptens (e.g., haptens covalently bonded to proteins). Target molecules are typically detected using one or more conjugates of specific binding molecules and detectable markers.

[0083] Overview

[0084] The present disclosure provides systems and methods for detecting and estimating signals corresponding to one or more biomarkers in a biological sample that has been stained to detect the presence of protein and / or nucleic acid biomarkers. In some embodiments, the present disclosure enables accurate estimation (i.e., quantification) of the amount of signal corresponding to one or more biomarkers in patches of concentrated signal (e.g., in areas of dense biomarker staining).

[0085] In some embodiments, the systems and methods described herein facilitate the detection and counting of spot signals from individual isolated spots and signal aggregation plaques that are clustered together in in situ hybridization (ISH) (e.g., from high gene copy numbers, etc.) and immunohistochemistry (IHC) images. The systems and methods disclosed herein are capable of detecting and characterizing isolated spot signals (e.g., representing a single gene copy or a single protein target), and then applying the derived properties of the detected isolated spots to detect and estimate the number of spots in the form of aggregated signals (i.e., in signal aggregation plaques). As a result, the total number of spots for both isolated spots and aggregated plaques can be reported, and this number can be associated with the gene copy number (mRNA copy number) or the number of protein targets.

[0086] It is believed that the systems and methods described herein are capable of quantifying signals in situations where a significant amount of biomarker signal is concentrated in a small area (e.g., when a large number of gene copies are present in a cell). Although the methods used herein are applicable to detecting and estimating signals associated with protein and nucleic acid biomarkers, the systems and methods are particularly useful for elucidating the amount of signal when a large number of gene copies are present in a small area of ​​a biological sample.

[0087] At least some embodiments of the present disclosure relate to computer systems and methods for analyzing digital images captured from biological samples, the biological samples comprising tissue samples stained with one or more primary colors (e.g., hematoxylin and eosin (H&E)) and one or more detection probes (e.g., probes containing specific binding entities that facilitate labeling of targets within the sample). Although examples herein may refer to the application of specific tissues and / or specific stains or detection probes for detecting certain markers (and therefore detecting diseases), those skilled in the art will understand that different tissues and different stains / detection probes may be applied to detect different markers and different diseases. For example, although certain embodiments may involve quantifying the amount of signal corresponding to two different mRNA probes (or a series of mRNA probes with different sequences), the systems and methods described herein may be used to detect and estimate signals from a single nucleic acid probe, a single antibody probe, a combination of two or more nucleic acid probes, a combination of two or more antibody probes, and / or any combination of nucleic acid and antibody probes.

[0088] According to some embodiments, a digital pathology system 200 for imaging and analyzing a sample is provided in Figure 1 . The digital pathology system 200 may include an imaging device 12 (e.g., a device having a means for scanning a microscope slide bearing a sample) and a computer 14, whereby the imaging device 12 and the computer may be communicatively coupled together (e.g., directly or indirectly via a network 20). The computer system 14 may include a desktop computer, a laptop computer, a tablet computer, etc., digital electronic circuitry, firmware, hardware, memory, computer storage media, a computer program or instruction set (e.g., where the program is stored in a memory or storage medium), one or more processors (including a programmed processor), and any other hardware, software or firmware modules or combinations thereof. For example, Figure 1 The computing system 14 shown in the figure may be a computer having a display device 16 and a housing 18. The computer may store the digital image in binary form (locally such as in a memory, a server, or another network connection device). The digital image may also be divided into a matrix of pixels. A pixel may include a digital value having one or more bits defined by a bit depth. The skilled person will appreciate that other computer devices or systems may be utilized, and the computer system described herein may be communicatively coupled to additional components, such as a sample analyzer, a microscope, other imaging systems, automated slide preparation equipment, etc. Some of these additional components and various computers, networks, etc. that may be used will be further described herein.

[0089] Typically, the imaging device 12 (or other image sources including pre-scanned images stored in memory) may include, but are not limited to, one or more image capture devices. The image capture device may include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), an optical device (e.g., one or more lenses, a sensor focusing lens group, a microscope objective, etc.), an imaging sensor (e.g., a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, etc.), a film, etc. In a digital embodiment, the image capture device may include multiple lenses that cooperate to demonstrate instant focus. The image sensor (e.g., a CCD sensor) can capture a digital image of the sample. In some embodiments, the imaging device 12 is a bright field imaging system, a multispectral imaging (MS I) system, or a fluorescence microscope system. The digitized tissue data may be generated, for example, by an image scanning system, such as a VENTANA DP200 scanner or a VENTANA iSCAN HT scanner of VENTANA MEDICAL SYSTEMS, Inc. (Tucson, Arizona), or other suitable imaging equipment. Additional imaging devices and systems are also described herein. Those skilled in the art will appreciate that the digital color image acquired by the imaging device 12 may be conventionally composed of basic color pixels. Each color pixel can be encoded on three digital components, each comprising the same number of bits, each component corresponding to a primary color, typically red, green or blue, also denoted by the term "RGB" components.

[0090] Figure 2 An overview of various modules used in the currently disclosed digital pathology system is provided. In some embodiments, the digital pathology system employs a computer device 200 or a computer-implemented method having one or more processors 220 and at least one memory 201 storing non-transitory computer-readable instructions to be executed by the one or more processors so that the one or more processors 220 execute instructions (or stored data) in one or more modules (e.g., modules 202 to 210).

[0091] Reference Figure 2 and Figure 3A, the present disclosure provides a computer-implemented system and method for estimating an amount of a signal corresponding to at least one biomarker in an image of a biological sample. In some embodiments, the system may include: (a) an imaging module 202, which is suitable for generating image data of a stained biological sample (e.g., a sample stained for the presence of at least one or more protein biomarkers or one or more nucleic acid sequences (including one or more mRNA sequences)) (step 310); (b) a spot detection module 204, which is suitable for detecting isolated spots corresponding to biomarker signals in an input image (step 320); (c) an optical density derivation module 205, which is configured to derive optical density values ​​of typical isolated spots based on certain characteristics of all detected isolated spots (e.g., intensity, size, blurriness, circularity, etc.) (step 330); (d) a spot estimation module 208, which is suitable for estimating the number of spots in each of a plurality of subregions or a predicted number of spots within a signal spot aggregate based on the calculated spot characteristics (step 340); and (e) a database or other storage subsystem 240 for storing the estimated number of predicted spots and the number of detected spots in each of the plurality of subregions in the database 240 (step 350).

[0092] Those skilled in the art will also appreciate that additional modules may be incorporated into the workflow as desired. Figure 3B , the unmixing module 203 may be optionally run to provide an image channel image corresponding to a specific stain or biomarker (e.g., when a multi-channel image is used as an input image). Thus, after unmixing, the received input image is unmixed into an image having a signal corresponding to a single biomarker, and the unmixed image channel image may be used to detect isolated spots (step 320).

[0093] In some embodiments, the system includes a residual image generation module 206 that can be operated to calculate an image substantially free of detected isolated spots (step 370). Thus, the input image channel image can be processed so that the detected isolated spots from step 320 are removed, leaving only patches of signal aggregates.

[0094] In other embodiments, the segmentation module 207 can be run to segment the input image (e.g., the residual image from step 370) into a series of sub-regions (step 380). In some embodiments, the resulting segmented regions are substantially uniform in at least one of the presence of stain, stain intensity, or texture. In some embodiments, the segmented regions are each a superpixel, as described herein.

[0095] In other embodiments, the overlay generation module 209 may be run so that a visual representation of isolated spots detected in any signal aggregates and an estimated number of predicted spots may be superimposed on the input image (step 360).

[0096] Those skilled in the art will also appreciate that the system may be adapted to utilize received input images of biological samples stained in an in situ hybridization assay or an immunohistochemistry assay (or a combination of both assays). Figure 3C The steps for estimating the amount of a single protein biomarker are shown. Of course, the steps can be repeated for a second protein biomarker. Figure 3C The steps shown in , and the amount of the first estimated protein biomarker and the second estimated protein biomarker can be compared and / or used for further downstream analysis. Similarly, Figure 3D The steps for estimating the amount of a single nucleic acid biomarker are shown. Of course, Figure 3D The steps shown in can be repeated for a second nucleic acid biomarker, and the amounts of the first estimated nucleic acid biomarker and the second estimated nucleic acid biomarker can be compared and / or used for further downstream analysis.

[0097] also, Figure 2 Any of the modules identified in may be run more than once. For example, a first time modules 204 to 208 may be run to quantify a signal from a first biomarker (e.g., a signal corresponding to a first mRNA probe) and a second time modules 204 to 208 may be run to quantify a signal from a second biomarker (e.g., a signal corresponding to a second mRNA probe). For example, an unmixing module may be used to unmix a received image having at least two stains corresponding to different biomarkers into at least two corresponding image channel images (step 310). Modules 204 to 208 may be run a first time on a first image channel image to provide an estimate of predicted spots (corresponding to the first biomarker) in each of a plurality of subregions. Modules 204 to 208 may then be run a second time on a second image channel image to again provide an estimate of predicted spots (corresponding to the second biomarker) in each of a plurality of subregions. The results may then be combined so that an estimated number of predicted spots for each biomarker may be derived, and a comparison between the quantities of the first biomarker and the second biomarker may be made (e.g., to provide a prognosis or to develop a treatment regimen). These and other steps for comparing the estimated results of the quantities of the two biomarkers are described in Figure 3E Shown in.

[0098] Those skilled in the art will also understand that Figure 2Additional modules or databases not shown in the figure can be incorporated into the workflow. For example, an image preprocessing module can be run to apply certain filters to the acquired image, or to identify certain tissues and / or morphological structures within the tissue sample. In addition, a region of interest selection module can be used to select specific parts of the image for analysis.

[0099] Additionally, a module for removing chromatic aberration present in the input image may be run. Chromatic aberration refers to the effect caused by dispersion in an optical sensor, where the lens is unable to focus all colors to the same point of convergence. This effect may occur when images are acquired using a mounted microscope camera. Therefore, the microscope image should be corrected to reduce this effect, which may affect the spot count. In some embodiments, the correction is performed by simply shifting the pixels of the red (R), green (G), and blue (B) channels, and it can reduce the rainbow effect present in the image. Chromatic aberration removal can be used as an internal module to correct this effect on images acquired using a digital scanner.

[0100] Image acquisition module

[0101] In some embodiments, as an initial step, and with reference to Figure 2 , the digital pathology system 200 operates the imaging module 202 to capture an image or image data (e.g., from the scanning device 12) of a biological sample having one or more stains (step 310). In some embodiments, the received or acquired image is an RGB image or a multispectral image (e.g., a multi-channel bright field and / or dark field image). In some embodiments, the captured image is stored in the memory 201.

[0102] The image or image data (used interchangeably herein) can be acquired using the scanning device 12, for example, in real time. In some embodiments, the image is acquired from a microscope or other instrument capable of capturing image data of a microscope slide carrying a sample, as indicated herein. In some embodiments, the image is acquired using a two-dimensional scanner, such as a scanner capable of scanning image blocks, or a line scanner capable of scanning an image in a line-by-line manner, such as a VENTANA DP200 scanner. Alternatively, the image can be an image that has been previously acquired (e.g., scanned) and stored in the memory 201 (or, for that matter, retrieved from a server via the network 20).

[0103] In some embodiments, the image received as input is a whole slide image. In other embodiments, the image received as input is a portion of a whole slide image. In some embodiments, the whole slide image is decomposed into several parts, such as tiles, and each part or tile can be analyzed independently (e.g., using Figure 2 The modules given in and at least Figure 3A and Figure 3B After analyzing the multiple sections or blocks independently (e.g., providing an estimate of the amount of signal in each subregion of each block), the data from each section or block can be stored independently (step 350) and / or reported at the whole slide level (step 360) (see also Fig.15 ).

[0104] The biological sample can be stained by applying one or more stains, and the resulting image or image data includes signals corresponding to each of the one or more stains. In some embodiments, the input image is a single-channel image with only a single stain (e.g., stained with 3,3'-diaminobenzidine (DAB), silver (Ag+)). In some embodiments, two or more stains (thereby providing a multiplex image) can be performed on the biological sample in a multiplex assay (e.g., Ag+ for KAPPAmRNA sequences and sulforhodamine B (SRB) for LAMBDA mRNA sequences). In some embodiments, the biological sample is stained with at least two biomarkers. In other embodiments, the biological sample is stained to find the presence of at least two biomarkers and is also stained with a primary stain (e.g., hematoxylin).

[0105] In some embodiments, the biological sample is stained in an immunohistochemical assay to determine the presence of one or more protein biomarkers. For example, the biological sample can be stained for the presence of human epidermal growth factor receptor 2 protein (HER2 protein).

[0106] In other embodiments, the biological sample is stained in an in situ hybridization (ISH) assay to detect the presence of one or more nucleic acids (including mRNA). U.S. Patent No. 7,087,379 (the disclosure of which is incorporated herein by reference in its entirety) describes a method for staining a sample with an ISH probe so that individual spots representing a single copy can be observed and detected. In some embodiments, several target genes are analyzed simultaneously by exposing a cell or tissue sample to a plurality of nucleic acid probes that have been labeled with a plurality of different nucleic acid tags. For example, a plurality of nucleic acid probes can be labeled with a plurality of chromogenic and / or fluorescent compounds having different emission wavelengths, thereby allowing a single target cell or tissue sample to be analyzed simultaneously in a single step with multiple colors.

[0107] For example, the INFORM HER2 Dual ISH DNA Probe Mix Assay from Ventana Medical Systems, Inc. is designed to determine the HER2 gene status by counting the ratio of the HER2 gene to chromosome 17. The HER2 and chromosome 17 probes are detected using dual-color chromogenic in situ hybridization on formalin-fixed, paraffin-embedded tissue samples (such as human breast cancer tissue samples or human gastric cancer tissue samples). For the HER2 Dual ISH Assay, the signals are silver signals ("black signals") and red signals, corresponding to black and red dots in the input image, respectively. For the HER2 Dual ISH Assay, the cell-based scoring includes counting red and black dots within selected cells, where HER2 gene expression is expressed by black dots and chromosome 17 is expressed by red dots.

[0108] As another example, biological samples can be stained with multiple mRNA probes in ISH assays. Detecting B cell clonality contributes to the diagnosis of B cell lymphoma, and clonality assessment can be completed by assessing KAPPA and LAMBDA light chain expression. Therefore, a mixture of KAPPA oligomers and LAMBDA oligomers can be introduced into the sample to facilitate multiplex detection of KAPPA and LAMBDA mRNA. As described herein, KAPPA mRNA can be detected with silver (Ag) (shown as black) and LAMBDA mRNA can be detected with tyramide SRB (shown as purple). By analyzing the ratio of the amount of KAPPA mRNA detected and LAMBDA mRNA detected, it can be determined that the sample is non-lymphoma, lymphoma (KAPPA limited) or lymphoma (LAMBDA limited). U.S. Patent No. 8,236,502 describes specific KAPPA and LAMBDA mRNA probes that can be used and detection methods thereof, and its disclosure is incorporated herein by reference in its entirety.

[0109] The chromogenic stain may include hematoxylin, eosin, fast red, or 3,3'-diaminobenzidine (DAB). In some embodiments, the tissue sample is stained with a primary stain (e.g., hematoxylin). In some embodiments, the tissue sample is also stained with a secondary stain (e.g., eosin). In some embodiments, the tissue sample is stained in an IHC assay for a specific biomarker. Of course, one skilled in the art will appreciate that any biological sample may also be stained with one or more fluorophores.

[0110] Typically biological samples are processed on an automated staining / assay platform that applies a stain to the sample. There are a variety of commercial products on the market that are suitable for use as staining / assay platforms, one example of which is the Discovery TMThe camera platform may also include a bright field microscope, such as the VENTANA iScan HT or VENTANA DP 200 scanners from Ventana Medical Systems, Inc., or any microscope having one or more objectives and a digital imager. Other techniques for capturing images at different wavelengths may be used. Other camera platforms suitable for imaging stained biological samples are known in the art and are commercially available from companies such as Zeiss, Canon, Applied Spectral Imaging, and such platforms are readily adapted for use with the systems, methods, and apparatus of the subject disclosure.

[0111] In some embodiments, the input image is masked so that only the tissue area is present in the image. In some embodiments, a tissue area mask is generated to mask the non-tissue area by the tissue area. In some embodiments, a tissue area mask can be created by identifying the tissue area and automatically or semi-automatically (that is, with minimum user input) excluding background areas (e.g., corresponding to the area of ​​the full slide image of the glass without sample, such as only the area of ​​the white light from the imaging source). As will be appreciated by those skilled in the art, in addition to masking the non-tissue area by the tissue area, the tissue mask module can also mask other regions of interest as needed, such as, being identified as belonging to a certain tissue type or belonging to a part of the tissue of a suspected tumor area. In some embodiments, a tissue area mask image is generated by masking the tissue area by the non-tissue area in the input image using segmentation techniques. Suitable segmentation techniques are such techniques as known in the art (see Digital Image Processing, 3rd Edition, Rafael C. Gonzalez, Richard E. Woods, Chapter 10, page 689 and Handbook of Medical Imaging, Processing and Analysis, Isaac N. Bankman Academic Press, 2000, Chapter 2). In some embodiments, image segmentation techniques are used to distinguish between digitized tissue data and slides in the image, the tissue corresponding to the foreground and the slide corresponding to the background. In some embodiments, the component calculates an area of ​​interest (AOI) in the full slide image so as to detect all tissue regions in the AOI while limiting the number of background non-tissue areas analyzed. Various image segmentation techniques (e.g., image segmentation based on HSV color, laboratory image segmentation, mean shift color image segmentation, region growing, level setting methods, fast marching methods, etc.) can be used to determine, for example, the boundaries of tissue data and non-tissue or background data. Based at least in part on segmentation, the component can also generate a tissue foreground mask that can be used to identify these parts of the digital slide data corresponding to the tissue data. Alternatively, the component may generate a background mask for identifying those portions of the digitized slide data that do not correspond to tissue data.

[0112] This recognition can be achieved by image analysis operations such as edge detection. Tissue region masks can be used to remove non-tissue background noise in an image, such as non-tissue regions. In some embodiments, the generation of a tissue region mask includes one or more of the following operations (but not limited to the following operations): calculating the brightness of a low-resolution analysis input image, generating a brightness image, applying a standard deviation filter to the brightness image, generating a filtered brightness image, and applying a threshold to the filtered illumination image so that pixels with illumination above a given threshold are set to 1, and pixels below the threshold are set to 0, generating a tissue region mask. Additional information and examples related to the generation of tissue region masks are disclosed in US Publication No. 2017 / 0154420 entitled "An Image Processing Method and System for Analyzinga Multi-Channel Image Obtained from a Biological Tissue Sample Being Stainedby Multiple Stains", the disclosure of which is incorporated herein by reference in its entirety.

[0113] In some embodiments, the region of interest identification module can be used to select a portion of a biological sample for which image data should be acquired, for example, a region of interest having a large concentration of lymphocytes or a region suspected of having a large concentration of lymphocytes. A method for determining a region of interest is described in U.S. Publication No. 2017 / 0154420, the disclosure of which is incorporated herein by reference in its entirety. In general, U.S. Publication No. 2017 / 0154420 discloses: an image processing method for analyzing a multi-channel image obtained from a biological tissue sample stained with a plurality of stains, the method comprising: a. unmixing a multi-channel image to provide an unmixed image for each channel, b. spatially low-pass filtering at least one unmixed image, c. performing local maximum filtering on at least one unmixed image after spatial low-pass filtering, d. thresholding at least one of the unmixed images after spatial low-pass filtering to identify at least one group of adjacent pixels, and e. defining a region of interest by extracting an image portion of the multi-channel image from an image position given by the group of adjacent pixels, the region of interest having a predetermined size and shape.

[0114] Demixing module

[0115] In some embodiments, the image received as input may be a multiplexed image, i.e., the received image is an image of a biological sample stained with more than one stain (e.g., an image stained for the presence of KAPPA mRNA and LAMBDA mRNA probes; or HER2 and chromosome 17 probes). In these embodiments, and prior to further processing, the multiple images are first unmixed into their component channels, e.g., using unmixing module 203, where each unmixed channel corresponds to a specific stain or signal.

[0116] In some embodiments, in a sample including one or more stains, a separate image can be generated for each channel of the one or more stains. Those skilled in the art will appreciate that the features extracted from these channels can be used to describe different biological structures present in any image of the tissue (e.g., nuclei, membranes, cytoplasm, nucleic acids, etc.). For example, in the case of the KAPPA mRNA and LAMBA mRNA probes described herein, unmixing will result in a first unmixed image channel image with a silver (or black) signal, a second unmixed image channel image with a purple signal, and a third unmixed image channel with a hematoxylin signal. This unmixing result, also in the case of the KAPPA mRNA and LAMBA mRNA probes, FIG. 6A to FIG. 6C , 7A to 7C and FIG. 8A to FIG. 8D . In some embodiments, the unmixed image (often referred to as a "channel image" or "image-channel image") can be used as input to each module described herein. In some embodiments, the channel with the strongest color is unmixed first. For example, the silver / black channel can be unmixed before the purple channel.

[0117] The multispectral image provided by the imaging module 202 is a weighted mixture of the underlying spectral signals associated with the individual biomarkers and the noise component. At any particular pixel, the mixing weight is proportional to the biomarker expression of the underlying co-localized biomarkers at a particular location in the tissue and the background noise at that location. Therefore, the mixing weight varies from pixel to pixel. The spectral unmixing method disclosed herein decomposes the multi-channel pixel value vector at each pixel into a set of constituent biomarker end members or components and estimates the proportion of the individual constituent staining for each biomarker.

[0118] Unmixing is the process of decomposing the measured spectrum of a mixed pixel into a set of constituent spectra or end members and a set of corresponding fractions or abundances, which represent the proportion of each end member present in the pixel. Specifically, the unmixing process can extract stain-specific channels to determine the local concentration of a single stain using a reference spectrum known to a standard type of tissue and stain combination. Unmixing can use a reference spectrum retrieved from a control image or estimated from an observed image. Unmixing the component signals of each input pixel enables retrieval and analysis of stain-specific channels, such as the hematoxylin channel and the eosin channel in an H&E image, or the diaminobenzidine (DAB) channel and counterstaining (e.g., hematoxylin) channel in an IHC image. The terms "unmixing" and "color deconvolution" (or "deconvolution"), etc. (e.g., "deconvolution", "unmixing") can be used interchangeably in the art.

[0119] In some embodiments, linear unmixing is used to unmix the multiplexed images with the unmixing module 205. Linear unmixing is described, for example, in 'Zimmermann "Spectral Imaging and Linear Unmixing in Light Microscopy" Adv Biochem Engin / Biotechnol (2005) 95: 245-265' and CLLawson and RJHanson, "Solving least squares Problems", Prentice Hall, 1974, Chapter 23, page 161', the disclosures of which are incorporated herein by reference in their entirety. In linear dye unmixing, the measured spectrum (S (λ)) at any pixel is considered to be a linear mixture of the dye spectral components and is equal to the sum of the proportions or weights (A) of the color reference (R (λ)) of each individual dye represented at the pixel

[0120] S(λ)=A1·R1(λ)+A2·R2(λ)+A3·R3(λ).......A i ry(λ)

[0121] This can be expressed more generally in matrix form

[0122] S(λ)=∑A i ry(λ)|or|S=R·A

[0123] If M channel images are acquired and there are N individual dyes, then the columns of the M x N matrix R are the optimal color system as derived herein, the N x 1 vector A is the unknown ratio of the individual dyes, and the M x 1 vector S is the multi-channel spectral vector measured at the pixel. In these equations, the signal (S) in each pixel is measured during the acquisition of the multi-channel image and the reference spectrum, i.e., the optimal color system is derived as described herein. The contribution of the various dyes (Ai) can be determined by calculating their contribution to each point in the measured spectrum. In some embodiments, a least squares inverse fitting method is used to obtain a solution, which minimizes the squared difference between the measured spectrum and the calculated spectrum by solving the following system of equations,

[0124]

[0125] In this equation, j represents the number of detection channels and i equals the number of stains. Linear equation solutions often involve allowing constrained unmixing to force the weights (A) to sum to 1.

[0126] In other embodiments, unmixing is accomplished using the methods described in WO 2014 / 195193, entitled "Image Adaptive Physiologically Plausible Color Separation," filed on May 28, 2014, the disclosure of which is incorporated herein by reference in its entirety. In general, WO 2014 / 195193 describes a method for unmixing by separating component signals of an input image using an iteratively optimized reference vector. In some embodiments, image data from an assay is correlated with an expected or ideal result specific to the assay feature to determine a quality metric. In the case of low-quality images or poor correlation with the ideal result, one or more reference column vectors in the matrix R are adjusted, and unmixing is iteratively repeated using the adjusted reference vectors until the correlation shows a good quality image that matches the physiological and anatomical requirements. Anatomical, physiological, and assay information can be used to define rules that are applied to measure image data to determine quality metrics. This information includes how the tissue is stained, which structures within the tissue are intended to be stained or not stained, and the relationship between structures, stains, and markers specific to the assay being processed. The iterative process produces stain-specific vectors that generate images that accurately identify structures of interest and biologically relevant information, without any noise or unwanted spectra, and are therefore suitable for analysis. The reference vectors are adjusted into a search space. The search space defines the range of values ​​that the reference vectors can use to represent the stain. The search space can be determined by scanning a variety of representative training assays, including known or common problems, and identifying a set of high-quality reference vectors for the training assays.

[0127] In other embodiments, unmixing is accomplished using the method described in WO 2015 / 124772, filed on February 23, 215, entitled "Group Sparsity Model for Image Unmixing," the disclosure of which is incorporated herein by reference in its entirety. In general, WO 2015 / 124772 describes unmixing using a group sparsity framework, wherein the fractions of staining contributions from multiple co-existing markers are modeled within the "same group," and the fractions of staining contributions from multiple non-co-existing markers are modeled in different groups, the modeled group sparsity framework is provided with co-localization information of multiple co-existing markers, the modeled framework is solved using a group lasso to produce a least squares solution within each group, wherein the least squares solution corresponds to unmixing of co-existing markers, and a sparse solution is produced in the group corresponding to unmixing of non-co-existing markers. In addition, WO 2015124772 describes a method for unmixing by inputting image data acquired from a biological tissue sample, reading reference data from an electronic memory, reading co-existence data from an electronic memory, the reference data describing the stain color of each of a plurality of stains, the co-existence data describing groups of stains, each group including stains that can be juxtaposed in the biological tissue sample, and each group forming a group for a group lasso criterion, at least one group having a size of two or greater, and using the reference data as a reference matrix to calculate a solution to the group lasso criterion for obtaining an unmixed image. In some embodiments, the method for unmixing an image may include generating a group sparse model, wherein a portion of staining contributions from co-localized markers are assigned to a single group, a portion of staining contributions from non-co-localized markers are assigned to separate groups, and solving the group sparse model using an unmixing algorithm to produce a least squares solution within each group.

[0128] Blob and plaque detection module

[0129] After image acquisition and / or unmixing, images with a single biomarker channel are provided to a spot detection module 204 so that isolated spots within the image can be detected (as opposed to "plaques" or signal aggregates). In other embodiments, the unmixed image channel images are used as input to the spot and plaque detection module.

[0130] In some embodiments, morphological operations are performed to detect isolated spots, i.e., points, within an image. Based on the theory of mathematical morphology, binary morphological filtering is a technique for processing images with specific structural elements of a predetermined shape (e.g., a circle). The morphological filtering operation performs image filtering by examining the geometric and topological structures of objects with a predetermined shape. The basic idea is to detect an image with a simple predefined shape, where the algorithm draws a conclusion based on how this shape matches or does not match the shape in the image. In some embodiments, a disk-shaped structural element is used to perform morphological operations. In some embodiments, the radius of the disk-shaped element ranges from about 1 to about 2. In other embodiments, the radius of the disk-shaped element is 1. In some embodiments, Gaussian filtering / smoothing is performed before detection. In some embodiments, a scaling parameter of 1 can be used.

[0131] Other methods of detecting isolated spots in images are described in U.S. Patent Publication Nos. 2014 / 0377753 and 2017 / 0323148, the disclosures of which are incorporated herein by reference in their entirety.

[0132] In some embodiments, after each isolated spot in the input image is detected, the detected isolated spots are separated from the patches in the input image, thereby providing an "isolated spot image channel" and a "patch image channel". In some embodiments, the detected spots are "masked" from the patch image channel. In some embodiments, it is assumed that the input image contains objects with higher image values ​​and backgrounds with lower or near-zero values. In some embodiments, before "masking out", a smoothing operation is performed using a smoothing boundary with a width of 4 pixels and a morphological operation with a disk parameter of 2. From the "isolated spot image channel", the position of each detected isolated spot in the image can be determined, that is, the Cartesian coordinates are found for each detected isolated spot. For example, the x,y coordinates can be found by determining the seed center of each detected isolated spot. For example, the seed center can be calculated by determining the center of mass or mass center of each detected isolated spot.

[0133] In some embodiments, a multi-scale Gaussian difference (DoG) method can be used to detect small objects or blurred point sources. In general, Gaussian difference is a feature enhancement algorithm that involves subtracting a less blurred version from a blurred version of the original image. In the simple case of a grayscale image, a blurred image is obtained by convolving the original grayscale image with a Gaussian kernel with different standard deviations. It is believed that blurring an image using a Gaussian kernel suppresses only high-frequency spatial information. Subtracting one image from another preserves the spatial information between the frequency ranges preserved in the two blurred images. Therefore, the difference between the Gaussian filter is that it is a bandpass filter that discards all spatial frequencies except for a small amount of spatial frequencies present in the original grayscale image. In some embodiments, a DoG filter method is used to detect spots of different sizes. In some embodiments, multiple spot sizes are configured in ascending order (from small to large), but the processing is in order from large spots to small spots. In each iteration, a DoG filter is created based on the given internal and external filter sizes. The input or residual image is filtered with this DoG filter, and after local non-maximum suppression, an adaptive threshold decision is used to identify positive detections. The individual detections are collected in the result seed / annotation object.

[0134] For example, also in the case of KAPPA mRNA and LAMBDA mRNA detection, the black unmixed channel image (i.e., having signals corresponding to KAPPA mRNA copies) produced after unmixing can be used as an input image for detecting isolated KAPPA mRNA spots (i.e., single gene copies) (see Fig.9A Corresponds to a portion of the whole slide image, Fig. 9B In this particular example, the isolated speckle signal is extracted by detecting the disk shape of the speckle using a plurality of predetermined disk diameters. Fig. 9C and Fig.9D As shown in , the patch and speckle channel images are separated after speckle detection. Finally, the x,y coordinates of each detected isolated spot are calculated from the central seed of each spot signal, as FIG. 9E to FIG. 9H In some embodiments, an overlay can be generated and the detected isolated spots superimposed on the portion of the whole slide image used before unmixing ( Fig.9A and Fig.9I and Figure 9J for comparison).

[0135] As another example, in the case of detecting HER2 protein in an image stained in an immunohistochemical assay, Fig. 10A Shown is a portion of a whole slide image of a biological sample stained for the presence of HER2 protein. Fig. 10B and Fig. 10C Shows from Fig. 10A Detection results of isolated spots in the exported unmixed image channel image, where the detected isolated spots are superimposed on a portion of the whole slide image.

[0136] Optical Density Export Module

[0137] After detecting isolated spots in the image (320), the optical density derivation module 205 is used to determine the optical density of a representative isolated spot based on the calculated signal features from each detected isolated spot (step 330), that is, the optical density of a typical detected isolated spot is determined taking into account the specific features of all detected isolated spots in the input image.

[0138] refer to Figure 4 In some embodiments, the optical density derivation module 205 first calculates descriptive signal features for each detected isolated spot in the image (step 400). In some embodiments, the signal feature derivation module 205 implements a Gaussian fitting technique to analyze and parameterize certain characteristics of the detected isolated spots.

[0139] The phrase "Gaussian fitting" means fitting the data curve constituting the peak into a Gaussian distribution curve by determining the constant of the Gaussian function. Here, the data of each detected isolated spot from step 320 is used to generate a Gaussian function, from which the signal characteristics can be derived, and thus the characteristics of each detected isolated spot can be derived. In some embodiments, the fitting method is performed based on the assumption that the distribution of optical density and radius is a normal distribution.

[0140] In some embodiments, a one-dimensional Gaussian function fitting method is used to estimate the relevant spot parameters within a predefined tile size around the detected isolated spots. In some embodiments, the tile size is 5x 5 pixels. In other embodiments, the tile size is 7x 7 pixels. In other embodiments, the tile size is 11x 11 pixels. The technician will be able to determine the most appropriate tile size for any particular application, which will help provide the best histogram results.

[0141] In some embodiments, the characteristics derived from the Gaussian fitting technique include the size, intensity, blurriness, and circularity of the detected isolated points, and each of these characteristics is calculated using the parameters of the Gaussian function. In some embodiments, the estimated parameters from the fitting method include the mean, standard deviation (SD), and full width at half maximum by solving the linear system Ax=b, such as Fig.11 shown.

[0142] In some embodiments, the intensity characteristic of the detected isolated blob is calculated using the 98th percentile within a 5 pixel radius around the center of the detected isolated blob. In other embodiments, the entire intensity range in the intensity histogram can be used to calculate the intensity of a representative isolated blob by weighting the intensity by its histogram bin number and dividing by the sum of the intensity values ​​(e.g., intensityRef = sum(hIntensity.Values.*(hIntensity.BinEdges)) / sum(hIntensity.Values))

[0143] In some embodiments, the radius of a representative isolated blob is calculated by solving sizeRef = sum(hSize.Values.*(hSize.BinEdges)) / sum(hSize.Values).

[0144] In other embodiments, the blur characteristic is derived by finding the standard deviation (σ) of a Gaussian function fit method.

[0145] In other embodiments, the size characteristic is derived from the full width at half maximum (FWHM). FWHM is an expression for the range of a function given by the difference between the two extreme values ​​of the independent variable, with the dependent variable equal to half of its maximum value. In other words, it is the width of the spectral curve measured between those points on the y-axis that are half of the maximum amplitude (see, for example, Fig.11 ). Calculate FWHM using the following equation:

[0146]

[0147] In a further embodiment, the roundness characteristic is calculated based on a comparison between the actual optical density distribution within the image block and an ideal Gaussian model calculated from the estimated parameters. A consistency correlation coefficient (CCC) (a measure of the consistency between two variables, e.g., assessing reproducibility or inter-rater reliability) can be used to compare the relationship (or consistency), where CCC=1 indicates that the estimated parameters are completely consistent with the ideal Gaussian model; and where CCC=0 indicates that there is no consistency between the estimated parameters and the ideal Gaussian model.

[0148] Reference again Figure 4 Once the signal characteristic of each detected isolated spot is determined, the optical density of the representative isolated spot can be derived (step 401).

[0149] Figure 5A flow chart is provided showing the steps of calculating the optical density of a representative isolated spot. As a first step in the process, the calculated descriptive signal features (from step 400) are received as input (501). Next, a histogram can be generated for each calculated signal feature characteristic (step 502). For example, if a FWHM parameter representing the size of an isolated spot is determined, a histogram of the FMHM parameter can be generated. In some embodiments, the generated histogram shows the distribution of isolated spots in the input image having a particular value for each signal feature characteristic, such as Fig.12 In some embodiments, the y-axis of each histogram provides the normalized spot counts, while the x-axis gives the feature quantity. Fig.12 In , a histogram is shown, which shows the intensity of the black spot signal and the number of detected isolated spots with each of a series of interval intensity values. Fig.12 A histogram is shown, which shows the distribution of detected isolated blobs with a certain bin size.

[0150] Thus, the generated histogram provides an understanding of the density of detected isolated cells having a particular value or representative characteristics. Those skilled in the art will appreciate that the generated histogram thus provides insight into the characteristics of representative or typical detected isolated spots. For example, by viewing an intensity histogram (e.g., Fig.12 ), a person skilled in the art will be able to determine that the intensity value of the most frequently repeated detected isolated spots (i.e., the mode of the intensity values) has a certain value (e.g., a value of 0.9). It can then be assumed that a representative or typical detected isolated spot will have that particular determined intensity value (e.g., 0.9). In other embodiments, for example, a weighted average can be used.

[0151] Given the foregoing, the next step is to extract the size and intensity parameters from the generated histogram (step 502), so that the following equations can be solved:

[0152]

[0153] Here, the area can be referred to as a circle (πr 2 ) or rectangular (w x h) area, and Refers to the expected optical density of a single point.

[0154] In some embodiments, the expected optical density of a single point is an intensity value, which can be a uniform or non-uniform intensity value. In some embodiments, the expected optical density of a single point can be the mode of an intensity histogram, the average value of the total intensity of all detected isolated spots, or a weighted intensity, or a weighted average intensity of a histogram, etc. In some embodiments, isolated signal features are derived, and then derived parameters (such as size, intensity, etc.) are used as descriptive statistics to model a single point in the current data (image) to calculate an estimated count of the aggregated signal. The model of a single point can be a simple or more complex model, such as a circle, rectangle, sphere, or three-dimensional sphere, whose signal density changes from the center to the boundary (varying with fuzziness).

[0155] Next, an area measure is calculated from the extracted size parameters (from step 503) (step 504). For example, the radius can be derived from the mode of the FWHM values ​​in the corresponding histogram. This radius can then be multiplied by π to yield the area.

[0156] As another example, the following data can be calculated so that the optical density of a representative isolated spot can be derived:

[0157] (i) A threshold for background intensity (eg, TH = 0.45) was selected using the 10th percentile of the silver spot intensity histogram.

[0158] (ii) Use the mode of the silver intensity histogram (intensity = 0.9) as the expected intensity of the isolated point.

[0159] (iii) Using a size of FWHM = 1.06, the 10th percentile of the silver spot size histogram was used as the expected size of a representative isolated spot.

[0160] Finally, the derived intensity parameter is multiplied by the area to give the optical density of the representative isolated spot (step 505 ). The calculated optical density of the representative spot is then provided to the spot estimation module 208 .

[0161] Segmentation module and residual image generation module

[0162] In some embodiments, before estimating the number of predicted spots in the signal aggregate, the input image is segmented into multiple sub-regions using segmentation module 207. In some embodiments, since the calculation is based on a smaller local area rather than the entire image, it is believed that the generation of the sub-region minimizes the calculation error. In some embodiments, the input image is segmented into multiple sub-regions for reducing the complexity of estimating the signal in the aggregate signal plaque. In some embodiments, segmentation provides multiple sub-regions, wherein the presence of staining (e.g., the pixels present in the superpixel are a specific type of staining), staining intensity (e.g., the pixels have a specific relative intensity value or a range of values), and / or texture (e.g., the pixels have a specific color or spatial arrangement of intensity) is similar or uniform. Sub-regions with substantially uniformity are particularly important for fragments that are almost entirely composed of aggregate signals, and the uniformity on the sub-region creates a consistent approximation of spot counts within the fragment during signal estimation (as described herein). Segmentation is performed on a single-channel image, for example, the "purple" channel in the unmixed image showing the LAMBDA mRNA signal, or the "black" channel in the unmixed image showing the KAPPA mRNA signal.

[0163] In some embodiments, an image different from an image for detecting isolated spots or an image for calculating the optical density of a representative or typical isolated spot is segmented. In some embodiments, segmentation is performed on the residual image generated by the residual image generation module 206. In some embodiments, a residual image is generated by subtracting one image from another image. For example, an isolated spot image (as described herein) can be subtracted from an unmixed image channel image to provide a residual image having only signal aggregation patches. In other embodiments, a foreground segmentation mask can be generated based on the position of the detected isolated spots, and the foreground segmentation mask can be used to filter the unmixed image channel image to provide a residual image having only signal aggregation patches. Fig.13 An example of a residual image is provided in .

[0164] In some embodiments, a sub-region generates captured information in an area of ​​the input image that has a predetermined size or a size within a range set in an image processing algorithm (e.g., parameters of a SLIC superpixel generation algorithm as described herein).

[0165] In some embodiments, the input image is segmented into sub-regions having a predetermined shape, size, area, and / or spacing. For example, the sub-regions may be elliptical, circular, square, rectangular, etc. In some embodiments, the elliptical, circular, square, or rectangular sub-regions may have a size ranging from 50 pixels to about 100 pixels, or some other size such that the selected pixel groups have similar properties. In some embodiments, the sub-regions are non-overlapping and may be generated by a sampling grid. In some embodiments, the sub-regions are distributed on the image in a manner that captures representative samples of the relevant regions for analysis, such as regions where irregularly shaped cells are the main features.

[0166] In other embodiments, the input image is segmented by applying a series of algorithms to the image, including a global threshold filter, a local adaptive threshold filter, a morphological operation, and a watershed transform. The filters can be run sequentially or in any order deemed necessary by a person of ordinary skill in the art. Of course, any filter can be applied repeatedly until the desired result is achieved. In some embodiments, a first filter is applied to the input image to remove areas that are unlikely to be stained, such as those white image areas. In some embodiments, this is achieved by applying a global threshold filter. In some embodiments, global thresholding is based on the median and / or standard deviation calculated on the first principal component channel, such as similar to a grayscale channel. Then a filter is applied to the image to selectively remove artifacts, such as small patches, small interruptions, other small objects, and / or fill holes. In some embodiments, a morphological operator is applied to remove artifacts and / or fill holes. In some embodiments, a distance-based watershed is applied based on a binary image introduced as an input (e.g., a binary image generated by a previous filtering step).

[0167] In some embodiments, the input image is segmented into superpixels. It is believed that the superpixel algorithm segments the image into multiple fragments (pixel groups) representing entities with perceptual significance. Each superpixel is obtained by a low-level grouping process and has a perceptually consistent unit, i.e., all pixels in the biological object contained in the superpixel are as consistent as possible in terms of staining presence, staining intensity, and texture. FIG. 14A to FIG. 14C shows an example of superpixel segmentation (where Fig.14A A portion of a whole slide image is shown; Fig. 14B shows superpixel generation on an unmixed image channel image without detected isolated spots, i.e., a residual image; and Fig. 14C shows the superposition of Fig.14A As will be further described herein, a superpixel segmented image (e.g., Fig. 14B) can be used as input for estimating the amount of signal corresponding to the biomarker in each signal accumulation plaque.

[0168] A superpixel is a collection of pixels with similar characteristics, such as color, brightness, and texture. An image can be composed of a certain number of superpixels, which contain multiple combined features of pixels and can retain the side information of the original image. Compared with a single pixel, a superpixel contains rich feature information, which can greatly reduce the complexity of image post-processing and significantly improve the speed of image segmentation. Superpixels are also useful for estimating probabilities and making decisions with small neighborhood models.

[0169] Superpixel algorithms are methods of grouping pixels into meaningful atomic regions of similar size. Without wishing to be bound by any particular theory, it is believed that superpixels are powerful because they often fall on important boundaries within an image and tend to take on unusual or unique shapes when they contain salient object features. Consistent with the desire to acquire and store information at medium-resolution analysis, superpixels lie between the pixel and object levels: they carry more information than pixels by representing groups of pixels that have perceptual significance, rather than comprehensively representing image objects. Superpixels can be understood as a form of image segmentation that oversegments an image in a fraction of the computational time. The outlines of superpixels have been shown to adhere well to natural image boundaries, as most of the structure in the image is preserved. By computing image features for each superpixel rather than for each pixel, subsequent processing tasks are reduced in complexity and computational time. Superpixels are therefore considered a useful preprocessing step for object-level analysis such as image segmentation.

[0170] Without wishing to be bound by any particular theory, it is believed that superpixels over-segment images by forming compact and uniform groups of pixels with similar features in, for example, color or geometry. In the past, a variety of superpixel methods have been developed. They can be divided into (i) graph-based methods and (ii) gradient ascent-based methods. In graph-based methods, each pixel is considered to be a node in a graph. An edge weight proportional to their similarity is defined between all pairs of nodes. Then, a cost function defined on the graph is formulated and minimized in order to extract superpixel fragments. In gradient ascent-based methods, pixels are iteratively mapped to feature space to depict denser areas representing clusters. Each iteration refines each cluster to obtain a better segmentation until convergence.

[0171] Many superpixel algorithms have been developed, including normalized cutting, agglomerative clustering, fast shifting and Turbopixel algorithms. The normalized cutting algorithm uses contour and texture clues to recursively segment the graphics of all pixels in the image, and globally minimizes the cost function defined on the edge of the segmentation boundary. It produces very regular and beautiful superpixels (see Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, (PAMI), 22 (8): 888–905, August 2000, which is incorporated herein by reference in its entirety). Alastair Moore, Simon Prince, Jonathan Warrell, Umar Mohammed and Graham Jones describe a method in Superpixel Lattices. IEEE Computer Vision and Pattern Recognition (CVPR), 2008, which divides the image into smaller vertical or horizontal areas by finding the best path or seam, and generates superpixels that meet the grid. The optimal path is found using a graph cutting method (see Shai Avidan and Ariel Shamir. Seam carving for content-aware image resizing. ACM Transactions on Graphics (SIGGRAPH), 26 (3), 2007, the disclosure of which is incorporated herein by reference). Fast shift (see A. Vedaldi and S. Soatto. Quick shift and kernel methods for mode seeking. In European Conference on Computer Vision (ECCV), 2008, the disclosure of which is incorporated herein by reference) uses a pattern search segmentation scheme. It initializes the segmentation using a medoid shift process. Then, it moves each point in the feature space to its nearest neighbor, which increases the Parzen density estimate.The turbopixel method uses a level set-based geometric flow to gradually expand a set of seed locations (see A. Levinshtein, A. Stere, K. Kutulakos, D. Fleet, S. Dickinson, and K. Siddiqi. Turbopixels: Fast superpixels using geometric flows. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2009, the disclosure of which is incorporated herein by reference). The geometric flow relies on local image gradients with the goal of regularly distributing superpixels on the image plane. Unlike other methods, Turbopixel superpixels are constrained to have consistent size, compactness, and boundary adhesion. Radhakrishna Achanta describes other methods for generating superpixels in "SLIC Superpixels Compared to State-of-the-art", Journal of Latex Class Files, Vol. 6, No. 1, December 2011, the disclosure of which is incorporated herein by reference in its entirety.

[0172] A superpixel algorithm called Simple Linear Iterative Clustering (SLIC) is introduced, which is superior in terms of boundary adherence and efficiency compared to the current state-of-the-art superpixel methods. SLIC has two steps. First, it generates superpixels by grouping pixels with a local k-means clustering (KMC) method, where the distance is measured as the Euclidean distance combined with the data and spatial distance. Second, the connected component algorithm (CCA) is used to remove the generated small isolated regions by merging them into the nearest large superpixel.

[0173] K-means clustering aims to partition n observations into k clusters, where each observation belongs to the cluster with the nearest mean as a prototype of that cluster. Connected component labeling works by scanning the image pixel by pixel (from top to bottom and from left to right) to identify connected pixel regions, that is, regions of adjacent pixels that share the same set of intensity values ​​V. (For binary images V = {1}; however, in grayscale images, V will take on a range of values, for example: V = {51,52,53,...,77,78,79,80}.) Connected component labeling works on either binary or grayscale images, and different connectivity measures are possible. However, below we assume binary input images and 8-connectivity. The connected component labeling operator scans the image by moving along a row until it reaches a point p (where p represents the pixel to be labeled at any stage of the scanning process), for which V = {1}. When this is true, it checks the four neighbors of p that it has already encountered in the scan (i.e., (i) the neighbor to the left of p, (ii) the neighbor above p, and (iii and iv) the two upper diagonal entries). Based on this information, p is labeled as follows: if all four neighbors are 0, assign a new label to p, otherwise if only one neighbor has value V = {1}, assign its label to p, otherwise if more than one neighbor has value V = {1}, assign a label to p and note the equivalent entries.

[0174] After completing the scan, equivalent label pairs are classified into equivalent categories and a unique label is assigned to each category. As a final step, the image is scanned a second time, during which each label is replaced by the label assigned to its equivalent class. For display, the labels can be different grayscale or color.

[0175] SLIC is an adaptation of k-means for superpixel generation with two important differences: (i) by restricting the search space to an area proportional to the superpixel size, the number of distance calculations in the optimization is significantly reduced (this is considered to reduce the complexity linearly in the number of pixels and is independent of the number of superpixels k); and (ii) the weighted distance metric combines color and spatial proximity while providing control over the size and compactness of superpixels. (See Achanta et al., "SLIC Superpixels Compared to State-of-the-Art Superpixel Methods", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, No. 11, November 2012, the disclosure of which is incorporated herein by reference in its entirety).

[0176] SLIC considers image pixels in a 5D space, defined by their L*a*b values ​​in the CIELAB color space and their x and y coordinates. The pixels in the 5D space are clustered based on an adaptive k-means clustering that integrates color similarity and proximity in the image plane. Clustering is based on a distance metric D that measures color similarity (dc) in the L*a*b space and pixel proximity (ds) in the x,y space. The latter is normalized by a grid spacing (S) that defines the square root of the total number of image pixels divided by the number of superpixels (k). The compactness and regularity of the superpixels are controlled by a constant m. This parameter is used as a weighting criterion between spatial distance (dc) and spectral distance (ds). A larger m increases the weight of spatial proximity, which results in more compact superpixels whose boundaries adhere less to spectral contours in the image.

[0177]

[0178] The SLIC algorithm can be applied as follows. Let N p is the number of pixels in a given image (or its part or region of interest), and k is the number of superpixels to be generated. Next, the main steps of the SLIC algorithm are as follows:

[0179] (1) Initialize the cluster centers. Set the k initial cluster centers at intervals. We then move these cluster centers to the location with the lowest gradient in the 3 x 3 neighborhood. Without wishing to be bound by any particular theory, we think this is done to avoid centering superpixels on edges and to reduce the chance of seeding superpixels with noisy pixels.

[0180] (2) Assigning pixels: Each pixel is assigned to the nearest cluster center in the local search space through local KMC.

[0181] (3) Update cluster centers. Set each cluster center to the mean of all pixels in the corresponding cluster.

[0182] (4) Repeat steps (2)-(3) until the cluster remains unchanged or meets another given criterion.

[0183] (5) Post-processing. If the size of the isolated region is smaller than the minimum size S min , CCA is used to reassign isolated regions to nearby superpixels.

[0184] Local KMC is applied in step (2) of the SLIC method, where each pixel is associated with the nearest cluster center whose search region covers its position. In traditional KMC, the search region for each cluster center is the entire image, and then the distance from each cluster center to each pixel in the image is calculated. However, in local KMC, the search space for cluster centers is restricted to a local 2S x2S square region. Therefore, SLIC only calculates the distance from each cluster center to the pixels within its search region.

[0185] In local KMC, Euclidean distance is used for clustering. Let zi be the data of the center of the i-th cluster, whose spatial location is (x i ,y i ). Let z j is the intensity of a pixel within the center search region. Then, the integrated distance between this pixel and the center is:

[0186]

[0187] where d f =|z i -z j | and The intensity and spatial distance between the pixel and the center are respectively, and m is the integral distance D of df and ds. I A regularization parameter that weights the relative contribution of . A larger m means that ds is larger than d f More importantly. The equivalent integral distance D that directly describes the contribution of the two distances I It can be given by the following formula:

[0188]

[0189] Where N f is the average intensity of the entire image, and w∈[0,1] is the regularization parameter. In this case, w and (1-w) are the normalized intensity and spatial distance D, respectively. I ratio.

[0190] In some embodiments, the parameter k of the SLIC algorithm specifies the number of superpixels of approximately equal size. In some embodiments, the compactness parameter m can be set to control the compromise between the homogeneity and boundary adhesion of the superpixel. Without wishing to be bound by any particular theory, it is believed that by changing the compactness parameter, regular-shaped superpixels can be generated in untextured areas, and highly irregular superpixels can be generated in textured areas. Similarly, without wishing to be bound by any particular theory, it is believed that the parameter m also allows for the weighting of the relative importance between color similarity and spatial proximity. When m is larger, spatial proximity is more important, and the superpixels generated are tighter (i.e., they have a lower area to perimeter ratio). When m is smaller, the superpixels generated are more closely attached to the image boundary, but have less regular size and shape.

[0191] In some embodiments, both the pixel size and the compactness parameter are adjusted. In some embodiments, a pixel size ranging from about 40 pixels to about 70 pixels is used. In other embodiments, a pixel size ranging from about 60 pixels to about 100 pixels is used. In other embodiments, a pixel size ranging from about 70 pixels to about 100 pixels is used. In further embodiments, a pixel size ranging from about 80 pixels to about 100 pixels is used.

[0192] Blob Estimation Module

[0193] The speckle estimation module 208 is used to quantify the amount of staining corresponding to the biomarker in any signal aggregate or plaque (step 340). In some embodiments, the estimation module 208 is used to determine the amount of signal in each signal aggregate or plaque in each sub-region of a plurality of sub-regions in the input image (e.g., the sub-regions returned after executing the segmentation module).

[0194] In some embodiments, the estimation is based on the linear relationship between the sum of the optical densities of individual spots and the aggregate signal as follows:

[0195]

[0196] Where N is the number of spots within the aggregate signal region, ODA is the optical density of the aggregate signal, and ODS is the calculated optical density of a representative isolated spot (from step 401).

[0197] In some embodiments, the optical density of the spot aggregates is the measured intensity of the spot aggregates from an input image (eg, from a residual image).

[0198] Once the number of predicted spots in each sub-region is estimated (step 340), the data can be stored in a database or other storage module 240 (step 350). In some embodiments, the number of predicted spots for each signal aggregation patch is stored along with the coordinates (x, y) of the patch (e.g., the x, y coordinates of the seed center of the patch). In some embodiments, the detected isolated spots are also stored with their x, y coordinates, as described herein. In some embodiments, the total number of spots for each sub-region is stored, i.e., the total number of isolated spots actually detected and predicted spots.

[0199] Reference Fig.15 In embodiments where several portions of a whole slide image are evaluated independently, an estimated number of predicted spots or a total number of spots (as described above) may be stored for each portion of the whole slide image (step 350). Fig.15 In embodiments where several portions of a whole slide image are evaluated independently, the estimated number of predicted spots or the total number of spots in each portion of the whole slide image (as described above) can be combined so that data corresponding to the amount of signal can be stored (step 350) and / or reported (step 360) for the whole slide image.

[0200] In some embodiments, the total number of spots for each sub-region can be calculated and stored in a database. For example, the total number of detected isolated spots can be combined (i.e., added) with the estimated number of predicted spots for each sub-region to provide the total number of spots for each sub-region.

[0201] In some embodiments, the total number of spots per cell can be calculated. In some embodiments, cell nucleus segmentation can be performed to return a total cell count. After the total number of spots per sub-region is calculated (as described above), the result is used together with the total cell count: total spot count per cell = total spot count / total cell count. Fig.17A and Fig. 17B An example of a segmentation of a nucleus at the center of a seed is shown ( Fig.17A ) and the nuclear boundaries of each cell ( Fig. 17B ).

[0202] Covering Generation Module

[0203] After detecting isolated spots (step 320) and after estimating the number of predicted spots (step 340), data (e.g., quantitative data) can be reported, for example by overlaying the data onto the whole slide image or a portion thereof. In some embodiments, overlay generation module 209 is used to calculate an overlay that can then be overlaid onto an input image. Fig.16A and Fig. 16BAn example of a suitable overlay is provided, where individual detected spots (red dots, 1610) and an estimated number of predicted spots for each sub-region (integer value) are superimposed on a portion of a whole slide image.

[0204] In some embodiments, the total number of spots for each sub-region can be calculated and stored in a database (e.g., storage module 240), or visually depicted in an overlay. For example, the total number of detected isolated spots can be combined (i.e., added) with the estimated number of predicted spots for each sub-region to provide the total number of spots for each sub-region. In other words, total spot count = detected isolated spot count + estimated aggregate signal count.

[0205] Instead of displaying isolated spots and / or integers representing the estimated signal amount for each signal aggregation plaque, each sub-region can be color-coded to display the total density of detected and predicted spot signals. For example, when the total number of spots is less than 5, the region can be represented by blue; when the total number of spots is between 6 and 15, the region can be represented by yellow; and in the case where the total number of spots is greater than 15, the region can be represented by red.

[0206] In addition, when estimating the signal amount of more than one biomarker, such as estimating the signal corresponding to KAPPA and LAMBDA mRNA probes, an overlay can be generated in which sub-regions with a specific KAPPA to LAMBDA ratio are displayed. For example, the visualization can be (+) or (-), indicating a positive or negative for a specific ratio. Similarly, each sub-region can be color-coded according to the derived ratio.

[0207] Other components for practicing embodiments of the present disclosure

[0208] The computer system 200 of the present disclosure can be bound to a sample processing device that can perform one or more preparation processes on a tissue sample. The preparation process can include, but is not limited to, dewaxing the sample, conditioning the sample (e.g., cell conditioning), staining the sample, performing antigen retrieval, performing immunohistochemical staining (including labeling) or other reactions, and / or performing in situ hybridization (e.g., SISH, FISH, etc.) staining (including labeling) or other reactions, and other processes for preparing samples for microscopy, microanalysis, mass spectrometry, or other analytical methods.

[0209] The processing device can apply a fixative to the sample. The fixative can include cross-linking agents (such as aldehydes (e.g., formaldehyde, paraformaldehyde, and glutaraldehyde) and non-aldehyde cross-linking agents), oxidants (e.g., metal ions and complexes such as osmium tetroxide and chromic acid), protein denaturants (e.g., acetic acid, methanol, and ethanol), fixatives of unknown mechanism (e.g., mercuric chloride, acetone, and picric acid), combination reagents (e.g., Carnoy's fixative, methacarn, Bouin's fluid, B5 fixative, Rossman's fluid, Gendre's fluid), microwave and mixed fixatives (e.g., expelled volume fixation and vapor fixation).

[0210] If the sample is a paraffin-embedded sample, the sample can be deparaffinized using (a variety of) appropriate deparaffinization fluids. After the paraffin is removed, any number of substances can be applied to the sample in succession. The substances can be used for pretreatment (e.g., for reversing protein crosslinks, exposing nucleic acids, etc.), denaturation, hybridization, washing (e.g., stringent washing), detection (e.g., linking a visual or marker molecule to a probe), amplification (e.g., amplifying proteins, genes, etc.), counterstaining, coverslipping, etc.

[0211] The sample processing device can apply various substances to the sample. The substances include, but are not limited to, stains, probes, reagents, rinses and / or regulators. The substance can be a fluid (e.g., a gas, a liquid or a gas / liquid mixture), etc. The fluid can be a solvent (e.g., a polar solvent, a non-polar solvent, etc.), a solution (e.g., an aqueous solution or other types of solutions), etc. The reagent can include, but is not limited to, a stain, a wetting agent, an antibody (e.g., a monoclonal antibody, a polyclonal antibody, etc.), an antigen recovery fluid (e.g., an aqueous or non-aqueous antigen repair solution, an antigen recovery buffer, etc.), etc. The probe can be an isolated nucleic acid or an isolated synthetic oligonucleotide attached to a detectable marker or reporter molecule. The marker can include a radioactive isotope, an enzyme substrate, a cofactor, a ligand, a chemiluminescent or fluorescent agent, a hapten, and an enzyme.

[0212] The sample processing device can be an automated device, such as the BENCHMARK XT instrument and the SYMPHONY instrument sold by Ventana Medical Systems, Inc. Ventana Medical Systems, Inc. is the assignee of many U.S. patents that disclose systems and methods for performing automated analysis, including U.S. Patent Nos. 5,650,327, 5,654,200, 6,296,809, 6,352,861, 6,827,901, and 6,943,029 and U.S. Published Patent Application Nos. 20030211630 and 20040052685, the disclosures of which are incorporated herein by reference in their entirety. Alternatively, the sample can be processed manually.

[0213] After processing the sample, the user can transport the slide with the sample to the imaging device. In some embodiments, the imaging device is a bright field imager slide scanner. One bright field imager is the iScan Hyperthreading and DP200 (Griffin) bright field scanners sold by Ventana Medical Systems, Inc. In an automated embodiment, the imaging device is a digital pathology device as disclosed in International Patent Application No. PCT / US2010 / 002772 entitled IMAGING SYSTEM AND TECHNIQUES (Patent Publication No. WO / 2011 / 049608) or in U.S. Patent Publication No. 2014 / 0178169 entitled IMAGING SYSTEMS, CASSETTES, AND METHODS OF USING THE SAME filed on September 9, 2011.

[0214] The imaging system or device may be a multispectral imaging (MSI) system or a fluorescence microscopy system. The imaging system used herein is MSI. MSI is typically equipped for analysis of pathological samples with a computerized microscope-based imaging system by providing access to the spectral distribution of pixel-level images. Although there are various multispectral imaging systems, an operational aspect common to all of these systems is the ability to form multispectral images. A multispectral image is an image that captures image data of a specific wavelength or a specific spectral bandwidth across the electromagnetic spectrum. These wavelengths may be selected by optical filters or by using other instruments capable of selecting predetermined spectral components, which include electromagnetic radiation at wavelengths outside the visible range, such as, for example, infrared (IR).

[0215] The MSI system may include an optical imaging system, a portion of which includes a spectral selectivity system that is tunable to define a predetermined number of N discrete optical bands. The optical system may be suitable for imaging a tissue sample, irradiating an optical detector in transmission with a broadband light source. In one embodiment, the optical imaging system may include a magnification system (such as, for example, a microscope) having a single optical axis that is usually spatially aligned with a single optical output of the optical system. When the spectral selection system is adjusted or tuned (for example, with a computer processor), the system forms a series of images of the tissue, such as to ensure that images are acquired in different discrete spectral bands. The device may additionally include a display in which at least one visually perceptible tissue image from the sequence of acquired images appears. The spectral selection system may include an optical dispersive element (such as a diffraction grating), a collection of optical filters (such as thin film interference filters), or any other system suitable for selecting a specific passband from the spectrum transmitted by the light source through the sample toward the detector in response to a user input or a command of a preprogrammed processor.

[0216] In an alternative embodiment, the spectrally selective system defines a number of optical outputs corresponding to N discrete spectral bands. This type of system ingests the transmitted light output from the optical system and spatially redirects at least a portion of this light output along N spatially distinct optical paths in a manner that images a sample in an identified spectral band onto a detector system along the optical path corresponding to the identified spectral band.

[0217] The subject matter described in this specification and the embodiment of operation can be implemented in digital electronic circuit or in computer software, firmware or hardware (including the structure disclosed in this specification and its structural equivalent), or one or more combinations thereof.The embodiment of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions for executing or controlling the operation of data processing equipment on a computer storage medium.Any module described herein may include logic executed by one or more processors.As used herein, "logic" refers to any information with instruction signals and / or data forms that can be applied to affect the operation of a processor.Software is an example of logic.

[0218] A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. In addition, although a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be or may be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification may be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0219] The term "programmed processor" includes all kinds of devices, apparatuses, and machines for processing data, including, for example, a programmable microprocessor, a computer, a system on a chip or multiple systems on a chip, or a combination of the foregoing. The device may include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The device and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0220] A computer program (also referred to as a program, software, software application, script or code) can be written in any form of programming language, including compiled or interpreted languages, illustrative or procedural languages, and the computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object or other unit suitable for a computing environment. A computer program can but need not correspond to a file in a file system. The program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordination files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a computer or on multiple computers located at a site or distributed across multiple sites and interconnected by a communication network.

[0221] The processes and logic flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by a device, and the device can also be implemented as a special-purpose logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0222] For example, processors suitable for executing computer programs include general and special microprocessors, and any one or more processors of any type of digital computer. Generally speaking, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential element of a computer is a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from a large-capacity storage device or to transfer data or both to a large-capacity storage device. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, to name a few, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive). Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CDROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0223] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display), an LED (light emitting diode) display, or an OLED (organic light emitting diode) display) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. In some embodiments, a touch screen may be used to display information and receive input from the user. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user (e.g., by sending a web page to a web browser in response to a request received from a web browser on a user's client device).

[0224] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or a middleware component (e.g., an application server), or a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnections (e.g., the Internet), and peer-to-peer networks (e.g., self-organizing peer-to-peer networks). For example, Figure 1 The network 20 may include one or more local area networks.

[0225] The computing system may include any number of clients and servers. The client and server are usually far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of a computer program that runs on a corresponding computer and has a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., the purpose is to display data to a user interacting with the client device and to receive user input from a user interacting with the client device). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0226] Example

[0227] Quality control was performed based on a graphical user interface (GUI) where the detected isolated spots were overlaid on the original and the observer could make corrections, such as adding, deleting, moving spots. Verification was performed by an expert observer (scientist) on a single pass silver image using 31 FOVs. The consistency curve graph is shown below, R 2 = 0.99, and CCC = 0.99. Examples of spot counting results before and after correction are shown in Fig.18 middle.

[0228] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications mentioned in this specification and / or listed in the Application Data Sheet are incorporated herein by reference in their entirety. Aspects of the embodiments may be modified, if necessary, to employ concepts of the various patents, applications, and publications to provide further embodiments.

[0229] Although the present disclosure has been described with reference to a number of illustrative embodiments, it will be appreciated that many other modifications and embodiments can be devised by those skilled in the art within the spirit and scope of the principles of the present disclosure. More specifically, reasonable changes and modifications in the components and / or arrangements of the subject combination arrangement are possible within the scope of the foregoing disclosure, the accompanying drawings, and the appended claims without departing from the spirit of the present disclosure. In addition to changes and modifications in the components and / or arrangements, alternative uses will be apparent to those skilled in the art.

Claims

1. A method for estimating the amount of a signal corresponding to at least one biomarker in an image of a biological sample, comprising: (a) detecting isolated spots in the first image; (b) deriving an optical density value of a representative isolated spot based on signal characteristics from the detected isolated spots; (c) estimating the number of predicted spots in the signal aggregate in each of the plurality of sub-regions based on the derived optical density values ​​of the representative isolated spots; as well as (d) storing the predicted estimated number of blobs and the detected isolated blobs in each of the plurality of sub-regions in a database, wherein the number of the predicted spots in the signal aggregates in each of the plurality of sub-regions is estimated by calculating the quotient of the total optical density of the signal aggregates in one of the sub-regions divided by the derived optical density of the representative isolated spot, wherein the optical density of the representative isolated spot is derived by: (i) generating a histogram based on the calculated descriptive signal features from all detected isolated spots; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measure value based on the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measure value. 2 . The method of claim 1 , wherein the intensity parameter is a uniform intensity measure or a non-uniform intensity measure.

3. A method according to claim 2, wherein the non-uniform intensity measure is derived from a blurriness histogram, which is calculated based on the blurriness signal characteristics of each detected isolated blob.

4. The method of claim 2, wherein the uniform intensity metric is selected from a mode of intensity values ​​derived from an intensity histogram and a weighted average intensity derived from an intensity histogram. The method of claim 1 , wherein the size parameter is the mode of the radius from a full width at half maximum histogram.

6. The method of claim 1, wherein the first image is a first unmixed image channel image corresponding to a signal from a first biomarker.

7. The method of claim 1, wherein the calculated descriptive signal features include speckle intensity, speckle blurriness, speckle circularity, and speckle size.

8. The method according to claim 7, wherein the descriptive signal features are calculated in image blocks having a predetermined size using a one-dimensional Gaussian function fitting method.

9. The method of claim 1, wherein the estimating of the number of the predicted spots in the signal aggregate is performed using a second image that is free of signals corresponding to the detected isolated spots.

10. The method of claim 9, wherein the second image is derived by: (i) generating a foreground segmentation mask based on isolated blobs detected in the first image; and (ii) filtering the first image with the generated foreground segmentation mask.

11. The method of claim 9, wherein the second image is generated by: (i) generating an isolated blob image including only the detected isolated blob from the first image; and (ii) subtracting the isolated spot image from the first image.

12. The method of claim 1, wherein each of the plurality of sub-regions has pixels that are uniform in at least one of biomarker staining intensity, biomarker staining presence, and local biomarker staining texture.

13. The method of claim 1, wherein each of the plurality of sub-regions is a superpixel.

14. The method of claim 13, wherein the superpixels are derived by: (i) grouping pixels using local k-means clustering; and (ii) merging small isolated regions into the nearest large superpixel using a connected component algorithm.

15. The method of claim 9, wherein the sub-regions are derived by overlaying a sampling grid onto the second image, the sampling grid defining uncovered regions having a predetermined size and shape.

16. The method of claim 1, further comprising generating an image overlay that independently visualizes the number of detected isolated blobs and the number of estimated predicted blobs in each sub-region.

17. The method of claim 1, further comprising generating an image overlay that visualizes the total amount of signal based on the estimated number of predicted blobs and the number of detected isolated blobs in each sub-region.

18. A system (200) for estimating the amount of one or more signals in a biological sample, the one or more signals corresponding to a stained target in the biological sample, the system comprising: (i) one or more processors (220), and (ii) a memory (201) coupled to the one or more processors, the memory being configured to store computer executable instructions that, when executed by the one or more processors, cause the system (200) to perform operations, the operations comprising: (a) detecting isolated spots in the first image; (b) Calculate descriptive signal features based on all detected isolated spots; (c) estimating the number of predicted spots in the signal aggregates in each of a plurality of sub-regions in the second image based on the calculated descriptive signal features; and (d) storing in said memory at least an estimated number of said predicted spots, wherein the number of the predicted spots in the signal aggregate in each of the plurality of sub-regions is estimated by: (i) deriving the optical density of a representative isolated spot, the optical density of the representative isolated spot being derived from the calculated descriptive signal feature; and (ii) calculating the quotient of the total optical density of the signal aggregates in one of the sub-regions of the second image divided by the derived optical density of the representative isolated spot, The optical density of the representative isolated spots is derived by the following steps: (i) generating a histogram based on each calculated descriptive signal feature; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measurement value based on the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measurement value.

19. The system of claim 18, wherein the intensity parameter is a uniform intensity metric or a non-uniform intensity metric.

20. The system of claim 18, wherein the size parameter is the mode of the radius from a full width at half maximum histogram.

21. The system of claim 18, wherein the first image is a first unmixed image channel image corresponding to a signal from a first biomarker.

22. The system of claim 18, wherein the calculated descriptive signal features include speckle intensity, speckle blurriness, speckle circularity, and speckle size.

23. The system of claim 18, wherein the descriptive signal features are calculated in image blocks having a predetermined size using a one-dimensional Gaussian function fitting method.

24. The system of claim 18, wherein the estimating of the number of the predicted spots in the signal aggregate is performed using a second image that is devoid of signals corresponding to the detected isolated spots.

25. A non-transitory computer readable medium storing instructions for estimating the amount of different signals in a stained biological sample, comprising: (a) detecting isolated spots in the first image; (b) deriving an optical density value for a representative isolated spot based at least on the intensity and size signal characteristics from the detected isolated spots; as well as (c) estimating the number of predicted spots in the signal aggregates in each of the plurality of sub-regions in the second image based on the derived optical density values ​​of the representative isolated spots, wherein the number of the predicted spots in the signal aggregates in each of the plurality of sub-regions is estimated by calculating the quotient of the total optical density of the signal aggregates in one of the sub-regions divided by the derived optical density of the representative isolated spot, wherein the optical density of the representative isolated spot is derived by the following steps: (i) generating a histogram based on each of the signal features from the detected isolated spots; (ii) extracting size and intensity parameters from the histogram; (iii) calculating an area measure value based on the extracted size parameter; and (iv) multiplying the extracted intensity parameter by the calculated area measure value.

26. The non-transitory computer readable medium of claim 25, wherein the isolated spots are detected using a shape detector having a disk-like shape.

27. The non-transitory computer readable medium of claim 25, wherein the sub-region is a superpixel.

28. The non-transitory computer-readable medium of claim 27, wherein the superpixels are derived by: (i) grouping pixels using local k-means clustering; and (ii) merging small isolated regions into the nearest large superpixel using a connected component algorithm.

29. The non-transitory computer readable medium of claim 25, wherein the total optical density of the signal aggregate is a measured value.

30. The non-transitory computer readable medium of claim 25, wherein the signal corresponds to at least two nucleic acid biomarkers.

31. The non-transitory computer readable medium of claim 25, wherein the signal corresponds to at least one protein biomarker.

Images