Non-tumor segmentation for supporting tumor detection and analysis

Abstract

The present disclosure relates to machine learning techniques for segmenting non-tumor regions in sample images to support tumor detection and analysis. In particular, aspects of the present disclosure relate to accessing one or more images comprising non-target regions (e.g., non-tumor regions) and target regions (e.g., tumor regions); predicting, by a two-dimensional segmentation model, a segmentation map of the non-target regions based on discriminative features encoded from the one or more images; generating a segmentation mask of the one or more images based on the segmentation map; applying the segmentation mask to the one or more images to generate a non-target region-masked image that excludes the non-target regions from the one or more images; and classifying biological material or structures within the target regions by an image analysis model based on a feature set extracted from the non-target region-masked image.

Description

[0001] Cross Reference to Related Patent Applications

[0002] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 963,145, filed January 19, 2020, which is incorporated by reference herein in its entirety for all purposes. TECHNICAL FIELD

[0003] The present disclosure relates to digital pathology, and in particular to machine learning techniques for segmenting non-tumor regions in sample images to support or improve tumor detection and analysis. BACKGROUND

[0004] Digital pathology involves scanning a pathology slide (e.g., a histopathology or cytopathology glass slide) containing tissue and / or cells into a digital image. The tissue and / or cells within the digital image can then be examined and / or interpreted by a pathologist through digital pathology image analysis for a variety of reasons including disease diagnosis, assessment of response to treatment, and development of drug formulations to combat the disease. To examine the tissue and / or cells within the digital image (which is nearly transparent), the pathology slide can be prepared using color stains (e.g., immunostains) that selectively bind to tissue and / or cell components. Immunohistochemistry (IHC) is a common application of immunostaining and is a process that selectively identifies antigens (proteins) in cells of a tissue section by exploiting the principle of specific binding of antibodies and other compounds (or chemicals) to antigens in biological tissue. In some assays, the target antigen stained in the sample can be referred to as a biomarker. Thereafter, digital pathology image analysis can be performed on the digital image of the stained tissue and / or cells to identify and quantify the staining for the antigen (e.g., a biomarker indicative of tumor cells) in the biological tissue.

[0005] Machine learning techniques have shown great promise in digital pathology image analysis, such as in tumor region identification, metastasis detection, and patient prognosis. Many computing systems equipped with machine learning techniques, including convolutional neural networks (CNNs), have been proposed for image classification and digital pathology image analysis, such as tumor region and metastasis detection. For example, a CNN can have a series of convolutional layers as hidden layers, and such network structure is capable of extracting representative features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques for image segmentation have also been implemented. Image segmentation is the process of partitioning a digital image into multiple segments (sets of pixels, also known as image objects). The goal of segmentation is to simplify and / or change the representation of an image into something more meaningful and easier to analyze. For example, image segmentation is often used to locate objects in an image, such as tumors and boundaries (lines, curves, etc.). To perform image segmentation on large data (e.g., whole slide pathology images), the image is first divided into many small pieces. A computing system equipped with machine learning techniques is trained to classify these pieces, and all pieces in the same class are combined into one segmented region. Thereafter, based on the representative features associated with the segmented region, machine learning techniques can be further implemented to make predictions or classifications on the segmented region (e.g., negative tumor cells or tumor cells without staining expression). SUMMARY

[0006] In various embodiments, a computer-implemented method is provided, the method comprising: accessing a plurality of images of a sample, wherein one or more images of the plurality of images contain a non-target region and a target region; predicting, by a two-dimensional segmentation model, a segmentation map of the non-target region based on discriminative features encoded from the one or more images, wherein the discriminative features are associated with a first biological material or structure; generating a segmentation mask of the one or more images based on the segmentation map; applying the segmentation mask to the one or more images to generate a non-target region masked image that excludes the non-target region from the one or more images; classifying, by an image analysis model, a second biological material or structure within the target region into a type of cell or nucleus based on a feature set extracted from the non-target region masked image; and providing, for the target region, the type of cell or nucleus classified for the second biological material or structure

[0007] In some embodiments, the sample is stained for one or more biomarkers, the first biological material or structure is a lymphoid aggregate or an immune cell cluster, and the second biological material or structure is a tumor cell or a tumor cell cluster.

[0008] In some embodiments, the computer-implemented method further includes: dividing the one or more images into image patches having a predetermined size; encoding, by the two-dimensional segmentation model, the image patches into discriminative features; projecting, by the two-dimensional segmentation model, the discriminative features onto a pixel space; and determining, by the two-dimensional segmentation model, a classification of the first biological material or structure for each pixel space based on a predetermined threshold.

[0009] In some embodiments, the computer-implemented method further includes: extracting, by the image analysis model, a feature set from the non-target region masked image; and computing one or more metrics of the second biological material or structure based on the feature set, wherein providing the type of cell or nucleus for the second biological material or structure classification further includes providing the one or more metrics of the second biological material or structure.

[0010] In some embodiments, the two-dimensional segmentation model is a modified U-Net model including a contracting path and an expansive path, each of the contracting path and the expansive path having at most 256 channels, and one or more layers of the contracting path implementing spatially randomized inactivation.

[0011] In various embodiments, a computer-implemented method is provided that includes: accessing a plurality of images of a sample, wherein one or more images of the plurality of images include a non-target region and a target region; predicting, by a two-dimensional segmentation model, a segmentation map of the non-target region based on discriminative features encoded from the one or more images, wherein the discriminative features are associated with a first biological material or structure; generating a first segmentation mask of the one or more images based on the segmentation map of the non-target region; classifying, by an image analysis model, a second biological material or structure within the target region into a type of cell or nucleus based on a second set of features extracted from the one or more images; generating a second segmentation mask of the one or more images based on the classification of the second biological material or structure within the target region; applying the first segmentation mask and the second segmentation mask to the one or more images to generate a target region and non-target region masked image that excludes the non-target region from the one or more images; and providing the target region and non-target region masked image and the type of cell or nucleus for the second biological material or structure classification.

[0012] In some embodiments, the sample is stained for one or more biomarkers, the first biological material or structure is a lymphoid aggregate or an immune cell cluster, and the second biological material or structure is a tumor cell or a tumor cell cluster.

[0013] In some embodiments, the computer-implemented method further comprises: dividing the one or more images into image patches of a predetermined size; encoding, by the two-dimensional segmentation model, the image patches into discriminative features; projecting, by the two-dimensional segmentation model, the discriminative features onto a pixel space; and determining, by the two-dimensional segmentation model, a classification of the first biological material or structure for each pixel space based on a predetermined threshold.

[0014] In some embodiments, the computer-implemented method further comprises: extracting, by the image analysis model, a feature set from the one or more images; and computing one or more metrics of the second biological material or structure based on the feature set, wherein providing the image masked by the target region and the non-target region mask and the type of cell or nucleus classified for the second biological material or structure further comprises providing the one or more metrics of the second biological material or structure.

[0015] In some embodiments, the two-dimensional segmentation model is a modified U-Net model comprising a contracting path and an expansive path, each of the contracting path and the expansive path having at most 256 channels, and one or more layers of the contracting path implementing spatial random dropout.

[0016] In various embodiments, a computer-implemented method is provided, the method comprising: accessing a plurality of tile images of a sample, wherein one or more tile images of the plurality of tile images comprise annotations of segmented non-target regions and target regions; training a two-dimensional segmentation model on the one or more tile images, wherein the training comprises: performing an iterative operation to find a set of parameters of the two-dimensional segmentation model that minimizes a loss function of the two-dimensional segmentation model, wherein each iteration involves finding a set of parameters of the two-dimensional segmentation model such that a value of the loss function using the set of parameters is less than a value of the loss function using another set of parameters in a previous iteration, and wherein the loss function is structured to measure a difference between an output predicted using the two-dimensional segmentation model and the annotations contained in the one or more tile images; and providing the trained two-dimensional segmentation model.

[0017] In some embodiments, the computer-implemented method further comprises dividing the one or more tile images into patches of 256 pixels x 256 pixels in size, wherein the training of the two-dimensional segmentation model is performed on the small patches.

[0018] In some embodiments, the two-dimensional segmentation model is a modified U-Net model comprising a contracting path and an expansive path, each of the contracting path and the expansive path having at most 256 channels, and one or more layers of the contracting path implementing spatial random dropout.

[0019] In some embodiments, the training further comprises adjusting a learning rate by lowering the learning rate of the modified U-Net according to a predefined schedule.

[0020] In some embodiments, the predefined schedule is a step decay schedule that decreases the learning rate by a predetermined factor every predetermined number of epochs for optimizing the loss function, and wherein the loss function is a binary cross-entropy loss function.

[0021] In some embodiments, a system is provided that includes one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0022] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and that includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0023] Some embodiments of the present disclosure include a system that includes one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine- readable storage medium that includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes.

[0024] The terms and expressions employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the application claimed. Thus, it should be understood that although the present application has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed can be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of the application as defined by the appended claims BRIEF DESCRIPTION OF DRAWINGS

[0025] The present disclosure is described in connection with the appended drawings:

[0026] FIG. 1 Non-target regions (e.g., lymphoid aggregation regions or immune cell groups) and target regions (e.g., negative tumor cells) within the same image are shown in accordance with various embodiments;

[0027] FIG. 2 A block diagram illustrating a computing environment for non-tumor segmentation and image analysis using deep convolutional neural networks is shown, in accordance with various embodiments;

[0028] FIG. 3 An exemplary schematic diagram 300 representative of a model architecture for non-target region segmentation is shown, in accordance with various embodiments;

[0029] FIG. 4 Ground truth generation and prediction model training is shown, in accordance with various embodiments;

[0030] FIG. 5A Loss during modified U-Net training with respect to batches is shown, in accordance with various embodiments;

[0031] FIG. 5B Accuracy achieving best performance during modified U-Net training is shown, in accordance with various embodiments;

[0032] FIG. 6 An exemplary U-Net is shown, in accordance with various embodiments;

[0033] FIG. 7 A process for training a prediction model is shown, in accordance with various embodiments;

[0034] FIG. 8A A process for generating a segmentation mask of non-target regions in image data is shown, in accordance with various embodiments;

[0035] FIG. 8B An example of an original image is shown, in accordance with various embodiments;

[0036] FIG. 8C An example of a predicted tile mask is shown, in accordance with various embodiments;

[0037] FIG. 9A A process for applying a segmentation mask to support or improve image analysis is shown, in accordance with various embodiments;

[0038] FIG. 9B An example of an original image with overlapping analysis results is shown, in accordance with various embodiments; and

[0039] FIG. 9C An example of an original image with overlapping analysis results implemented with the segmentation framework described herein is shown, in accordance with various embodiments.

[0040] In the drawings, like parts and / or features can have the same reference label. Also, various parts of the drawings can be referred to, by the first reference label and a second reference label that is differentiated by a dash and a distinction between the first and second reference labels. If only the first reference label is used in the description, the description is applicable to any one of the similar parts having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION

[0041] I. SUMMARY

[0042] The present disclosure describes techniques for automated non-tumor segmentation. More specifically, some embodiments of the present disclosure provide machine learning techniques for segmenting non-tumor regions in sample images to support or improve tumor detection and analysis.

[0043] Immunohistochemistry (IHC) slide staining can be used to identify proteins in cells of a tissue section, and is therefore widely used to study different types of cells, such as cancer cells and immune cells in biological tissues. IHC-stained cells of a tissue section can be evaluated under a high-power microscope and / or analyzed automatically using digital pathology algorithms to analyze digital images of a biological sample. Generally, in a whole slide analysis, the evaluation of a stained biological sample requires segmentation of regions of the stained biological sample, including identifying target regions (e.g., positive and negative tumor cells) and excluding non-target regions (e.g., normal tissue or blank slide regions). In some cases, the non-target regions to be excluded include biological materials or structures that are difficult to distinguish from other biological materials or structures of the target regions, and are therefore excluded from the evaluation of the biological sample. For example, lymphoid aggregates are clusters of immune cells, and their morphology and staining characteristics can be very similar to negative-stained tumor cells. FIG. 1 A region 100 of a stained pathology slide is shown that includes a cluster of immune cells (e.g., a lymphoid aggregate) 105 located next to a cluster of negative-stained tumor cells 110. The cluster of immune cells 105 has very similar morphology and staining characteristics to the cluster of negative-stained tumor cells 110, and it is therefore difficult to distinguish the lymphoid aggregate from the negative-stained tumor cells. As a result, in this case, a pathologist would typically provide a manual tumor annotation while excluding the non-target region. However, manual tumor annotation is prone to errors, pathologist bias, and is laborious due to the large size of the whole slide image under high magnification and the large amount of data to be processed.

[0044] Automated segmentation of tumors and substructures from digital images has the potential for precise and reproducible delineation of tumors that can aid in more efficient and better diagnosis, surgical planning, and treatment evaluation of tumors. Most automated tumor segmentation methods use hand-designed features that focus on characteristics of tumor cells. These methods implement a classical machine learning pipeline according to which features of the image are first extracted and then provided to a classifier whose training procedure does not affect the nature of these features. Another approach for designing task-adaptive feature representations is to learn a hierarchy of increasingly complex features directly from in-domain data. However, in both cases, accurate automated segmentation of tumors or tumor cells from whole slide images is a challenging task for a number of reasons. First, the boundaries between tumors and normal tissue, especially lymphoid aggregation regions, are often ambiguous due to specific and non-specific staining, heterogeneity in low resolution images, sparse signals (e.g., less than about 1% of the image), and the sheer number of biological materials or structures that need to be distinguished within a whole slide image. Second, tumors and tumor cells vary greatly in size, shape, and location from patient to patient. This hinders the use of strong priors on shape and location that are typically used for robust image analysis in many other applications such as face recognition or navigation. As a result, conventional image analysis algorithms often provide undesirable detection results (e.g., over-detection or misclassification) in these difficult regions (e.g., lymphoid aggregation regions) as negative tumor cells.

[0045] To address these limitations and problems, the techniques for automated tumor segmentation of embodiments of the present invention include the use of machine learning architectures that are fast and allow the model to handle the regional and whole slide image size and highly imbalanced nature. One illustrative embodiment of the present disclosure relates to a computer-implemented method for automatically detecting and excluding or masking non-target regions (e.g., lymphoid aggregation regions) on a whole slide image. In some cases, the method is performed as part of a pre-processing step prior to performing an image analysis algorithm to segment and classify target regions (e.g., tumor cells) within the image. In other cases, the method is performed as part of a post-processing step after performing an image analysis algorithm to segment and classify target regions (e.g., tumor cells) within the image. However, as will be appreciated by those of ordinary skill in the art, the concepts discussed herein are not limited to pre-processing or post-processing procedures, but can also be integrated into the overall image analysis process according to various embodiments.

[0046] The computer-implemented method can include the use of a convolutional neural network (CNN) architecture or model that utilizes a two-dimensional segmentation model (modified U-Net) to automatically detect and exclude biological structures or non-tumor cells (such as immune cell clusters or lymphoid aggregation areas) or mask these non-target areas across the whole slide image prior to performing standard image analysis algorithms to learn and identify target areas. However, the present disclosure is not limited to only segmenting out immune cell clusters or lymphoid aggregation areas, the techniques described herein can also be applied to distinguish other non-target or non-tumor areas, such as scanner artifacts, anesthetics, etc. The neural network architecture or model can be trained using pre-labeled images of non-target areas (e.g., lymphoid aggregation areas, necrosis, artifacts, and other cells) and target areas (e.g., tumor cells or staining expression targets). Thus, the trained convolutional neural network architecture or model can be used to segment non-target areas, which can then be masked from the whole slide analysis prior to, during, or after inputting the image to the image analysis algorithm. The image analysis algorithm performs a classification task and outputs a tumor readout of the target areas. Advantageously, this proposed architecture and technique can improve the accuracy of tumor cell classification via the image analysis algorithm, as well as reduce the likelihood of over-detection or misclassifying these difficult non-target areas (e.g., lymphoid aggregation areas) as target areas (e.g., negative tumor cells).

[0047] II. DEFINITIONS

[0048] As used herein, when an action is "based on" something, it means the action is based at least in part on at least a portion of that something.

[0049] As used herein, the terms "substantially," "about," and "approximately" are defined as largely but not necessarily wholly that which is specified (and include wholly that which is specified), as understood by persons of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "about," or "approximately" can be replaced with "within [some percent] of" for the specified percent, including 0.1%, 1%, 5%, and 10%.

[0050] As used herein, the term "sample," "biological sample," or "tissue sample" refers to any sample obtained from any organism, including viruses, that includes biological molecules such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof. Examples of other organisms include mammals (such as humans; veterinary animals, such as cats, dogs, horses, cows, and pigs; and laboratory animals, such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (e.g., tissue sections and punch biopsies of tissue), cell samples (e.g., cytological smears, such as a cervical smear or a blood smear or a cell sample obtained by microdissection), or cell fractions, debris, or organelles (e.g., obtained by lysing cells and separating their components by centrifugation or otherwise). Other examples of biological samples include blood, serum, urine, semen, fecal matter, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or punch biopsy), nipple aspirate, cerumen, milk, vaginal secretion, saliva, swabs (e.g., buccal swabs), or any material containing biological molecules derived from a first biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample (e.g., a sample that has been homogenized or liquefied) prepared from a tumor or a portion thereof obtained from a subject.

[0051] As used herein, the term "biological material or structure" refers to a natural material or structure that includes whole or partial living structures (e.g., nuclei, cell membranes, cytoplasm, chromosomes, DNA, cells, clusters of cells, etc.).

[0052] As used herein, the term "non-target region" refers to a region of an image that includes image data that is not intended to be evaluated in the image analysis process. The non-target region can include a non-tissue region of the image that corresponds to a substrate (such as glass without a sample), e.g., where only white light from the imaging source is present. The non-target region can additionally or alternatively include a tissue region of the image that corresponds to biological material or structures that are not intended to be analyzed in the image analysis process or that are difficult to distinguish from biological material or structures within the target region (e.g., lymphoid aggregates, immune cell clusters, necrosis, stromal cells, normal cells, scanning artifacts).

[0053] As used herein, the term "target region" refers to a region of an image that includes image data that is intended to be evaluated in the image analysis process. The target region includes any region of the image that is intended to be analyzed in the image analysis process (e.g., tumor cells or stained expression), such as a tissue region of the image.

[0054] As used herein, the term“tile” or“tile image” refers to a single image corresponding to a portion of an entire image or an entire slide. In some embodiments, a“tile” or“tile image” refers to a region of an entire slide scan or a target region having (x, y) pixel dimensions (e.g., 1000 pixels x 1000 pixels). For example, consider dividing an entire image into M columns of tiles and N rows of tiles, where each tile in the M x N tiling contains a portion of the entire image, i.e., the tile at position Ml, Nl contains a first portion of the image, while the tile at position M3, N4 contains a second portion of the image, the first and second portions being different. In some embodiments, the tiles can each have the same dimensions (pixels dimension x pixels dimension).

[0055] As used herein, the term“block” or“image block” refers to a container of pixels corresponding to a portion of a tile image, an entire image, or an entire slide. In some embodiments, a“block” or“image block” refers to a region of a tile image or a target region having (x, y) pixel dimensions (e.g., 256 pixels x 256 pixels). For example, a tile image of 1000 pixels x 1000 pixels is divided into blocks of 100 pixels x 100 pixels, the tile image will contain 100 blocks (each block containing 1000 pixels). In other embodiments, a block overlaps with each“block” or“image block” having (x, y) pixel dimensions and shares one or more pixels with another“block” or“image block”.

[0056] III. TECHNIQUES FOR AUTOMATED NON-TUMOR SEGMENTATION

[0057] Image segmentation is a procedure that separates similar parts of an image that exhibit similarity in different characteristics such as shape, size, color, etc. Segmentation of tumors and tumor cells visualizes the size, mass, and location of tumors or tumor cells within a certain region of the body (e.g., the brain or the lung) and can also provide a basis for analyzing stain uptake in digital images. For a long time, the gold standard for tumor and tumor cell segmentation has been manual segmentation, which is time-consuming and labor-intensive and thus not suitable for clinical applications and large-scale research and development efforts. There have been numerous studies dedicated to trying to make the process of tumor segmentation fully or partially automated. For example, image segmentation techniques such as thresholding, region growing, fuzzy clustering, use of watershed algorithms, etc. have been used to separate abnormal tissue (e.g., tumor mass) from normal tissue such as white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) of the brain. Nonetheless, the process of segmentation remains challenging not only because of the diversity of target regions containing tumors or tumor cells but also because of the difficulty in distinguishing target regions from non-target regions (e.g., normal tissue or blank slide regions).

[0058] Described herein is an end-to-end method that combines a model that segments and masks non-target regions (e.g., lymphoid aggregates) from images of specimen slides using a two-dimensional CNN and an image analysis algorithm that analyzes target regions (e.g., tumor cells) from images of specimen slides. The developed model is computationally inexpensive and aims to accommodate the size of the whole slide images, the extreme imbalance between tumor or tumor cells and healthy tissue, and the heterogeneous nature of the input images. The model has comparable performance to traditional algorithms that rely on human intervention for non-tumor and tumor segmentation.

[0059] III. A. EXAMPLE COMPUTING ENVIRONMENT

[0060] FIG. 2A block diagram of a computing environment 200 for non-tumor segmentation and image analysis using deep convolutional neural networks is shown in accordance with various embodiments. The computing environment 200 can include an analysis system 205 to train and execute predictive models, such as two-dimensional CNN models. More specifically, the analysis system 205 can include training subsystems 210a-n (‘a’ and ‘n’ representing any natural number) that build and train their respective predictive models 215a-n (which can be referred to individually as predictive model 215 or collectively as predictive models 215 herein) for use by other components of the computing environment 200. The predictive models 215 can be machine learning (“ML”) models, such as deep convolutional neural networks (CNNs), such as inception neural networks, residual neural networks (“Resnet”), or recurrent neural networks, such as long short-term memory (“LSTM”) models or gated recurrent unit (“GRUs”) models. The predictive models 215 can also be any other suitable ML model trained to segment non-target regions (e.g., lymphoid aggregation regions), segment target regions, or provide image analysis of target regions, such as two-dimensional CNNs (“2DCNNs”), Mask R-CNNs, feature pyramid networks (FPNs), dynamic time warping (“DTW”) techniques, hidden Markov models (“HMMs”), and the like, or a combination of one or more of such techniques— e.g., CNN-HMMs or MCNNs (multi-scale convolutional neural networks). The computing environment 200 can employ the same type of predictive models or different types of predictive models trained to segment non-target regions, segment target regions, or provide image analysis of target regions. For example, the computing environment 200 can include a first predictive model (e.g., a U-Net) for segmenting non-target regions (e.g., lymphoid aggregation regions). The computing environment 200 can also include a second predictive model (e.g., a 2DCNN) for segmenting target regions (e.g., tumor cell regions). The computing environment 200 can also include a third model (e.g., a CNN) for image analysis of target regions. The computing environment 200 can also include a fourth model (e.g., an HMM) for treatment of a disease diagnosis, or prognosis of a subject (e.g., a patient). In other examples in accordance with the present disclosure, other types of predictive models can be implemented.

[0061] In various embodiments, each prediction model 215a-n corresponding to training subsystem 210a-n is trained based on one or more sets of input image elements 220a-n, respectively. In some embodiments, each of input image elements 220a-n includes image data from one or more scanned slides. Each of input image elements 220a-n can correspond to image data from a single sample and / or image data on the day corresponding to the underlying image data that was collected. The image data can include an image, as well as any information related to the imaging platform that generated the image. For example, a tissue section can have been stained by applying a staining assay containing one or more different biomarkers associated with a chromogenic stain or fluorophore for brightfield imaging or fluorescence imaging. The staining assay can use a chromogenic stain for brightfield imaging, an organic fluorophore, quantum dots, or an organic fluorophore together with quantum dots for fluorescence imaging, or any other combination of stain, biomarker, and viewing or imaging device. Further, a typical tissue section is processed in an automated staining / platform that applies the staining assay to the tissue section, resulting in a stained sample. There are a variety of commercial products on the market suitable for use as a staining / platform assay, one example being the assignee Ventana Medical Systems, Inc.’s product VENTANA SYMPHONY. The stained tissue section can be provided to an imaging system, such as a microscope or a whole slide scanner with a microscope and / or imaging components, one example being the assignee Ventana Medical Systems, Inc.’s product VENTANA iScan Coreo. A multiplexed tissue slide can be scanned on an equivalent multiplexed slide scanner system. Additional information provided by the imaging system can include any information related to the staining platform, including concentrations of chemicals used for staining, reaction times of chemicals applied to the tissue in staining, and / or pre-analytic conditions of the tissue, such as tissue age, fixation method, duration, how the tissue was embedded, sectioned, etc.

[0062] The input image elements 220a-n can include one or more training input image elements 220a-d, validation input image elements 220e-g, and unlabeled input image elements 220h-n. It should be appreciated that the input image elements 220a-n corresponding to the training, validation, and unlabeled sets need not all be accessed at the same time. For example, an initial set of training and validation input image elements 220a-n can be accessed first and used to train the predictive model 215, and subsequently unlabeled input image elements can be accessed or received (e.g., at a single or multiple subsequent times) and used by the trained predictive model 215 to provide a desired output (e.g., segmentation of non-target regions). In some cases, the predictive model 215a-n is trained using supervised training, and each of the training input image elements 220a-d and optional validation input image elements 220e-g are associated with one or more labels 225 that identify a“correct” interpretation of the identification of non-target regions, target regions, and various biological materials and structures in the training input image elements 220a-d and validation input image elements 220e-g. The labels can alternatively or additionally be used to classify the corresponding training input image elements 220a-d and validation input image elements 220e-g or pixels therein with respect to the presence and / or interpretation of staining related to normal or abnormal biological structures (e.g., tumor cells). In certain cases, the labels can alternatively or additionally be used to classify the corresponding training input image elements 220a-d and validation input image elements 220e-g at a time point corresponding to a time point at which the underlying images were taken or a subsequent time point (e.g., a predefined duration of time after the time at which the images were taken).

[0063] In some embodiments, the classifier subsystem 210a-n includes a feature extractor 230, a parameter data store 235, a classifier 240, and a trainer 245 that collectively function to train the predictive model 215 based on training data (e.g., training input image elements 220a-d) and to optimize the parameters of the predictive model 215 during supervised or unsupervised training. In some cases, the training process includes an iterative operation to find a set of parameters for the predictive model 215 that minimizes a loss function for the predictive model 215. Each iteration can involve finding a set of parameters for the predictive model 215 such that the value of the loss function using that set of parameters is less than the value of the loss function using another set of parameters in a previous iteration. The loss function can be constructed to measure the difference between the output predicted using the predictive model 215 and the output predicted using the labels 225 contained in the training data. Once that set of parameters is identified, the predictive model 215 has been trained and can be used as designed for segmentation and / or prediction.

[0064] In some embodiments, the classifier subsystem 210a-n accesses training data from the training input image elements 220a-d at an input layer. The feature extractor 230 can pre-process the training data to extract relevant features (e.g., edges) detected at particular portions of the training input image elements 220a-d. The classifier 240 can receive the extracted features and convert the features into one or more output metrics that segment non-target or target regions, provide image analysis, provide disease diagnosis and prognosis for treatment for a subject (e.g., a patient), or a combination thereof, in accordance with weights associated with a set of hidden layers of the one or more predictive models 215. The trainer 245 can train the feature extractor 230 and / or the classifier 240 by facilitating learning of one or more parameters using the training data corresponding to the training input image elements 220a-d. For example, the trainer 245 can use a backpropagation technique to facilitate learning of weights associated with a set of hidden layers of the predictive model 215 used by the classifier 240. Backpropagation can use, for example, a stochastic gradient descent (SGD) algorithm to accumulate updates to parameters of the hidden layers. Learned parameters can include, for example, weights, biases, and / or other hidden layer-related parameters, which can be stored in the parameter data store 235.

[0065] The individually trained predictive models or the set of trained predictive models can be deployed to process unlabeled input image elements 220h-n to segment non-target or target regions, provide image analysis, provide disease diagnosis and prognosis for treatment for a subject (e.g., a patient), or a combination thereof. More specifically, the trained version of the feature extractor 230 can generate a feature representation of the unlabeled input image elements, which can then be processed by the trained version of the classifier 240. In some embodiments, image features can be extracted from the unlabeled input image elements 220h-n based on one or more convolutional blocks, convolutional layers, residual blocks, or pyramidal layers that optimize the bloating of the predictive models 215 in the classifier subsystem 210a-n. These features can be organized in a feature representation (e.g., a feature vector of an image). The predictive models 215 can be trained to learn feature types based on subsequent adjustments of parameters in the classification and hidden layers, including fully connected layers of the predictive models 215.

[0066] In some embodiments, the image features extracted by the convolutional blocks, convolutional layers, residual blocks, or pyramid layers include feature maps that are matrices of values representing one or more portions of the specimen slide on which one or more image processing operations have been performed (e.g., edge detection, sharpening image resolution). These feature maps can be flattened for processing by fully connected layers of the prediction model 215 that output a non-target region mask, a target region mask, or one or more metrics corresponding to a current or future prediction related to the specimen slide. For example, input image elements can be fed to an input layer of the prediction model 215. The input layer can include nodes corresponding to particular pixels. A first hidden layer can include a set of hidden nodes, each of which is connected to a plurality of input layer nodes. Nodes in subsequent hidden layers can be similarly configured to receive information corresponding to multiple pixels. Thus, the hidden layers can be configured to learn to detect features that span multiple pixels. Each of the one or more hidden layers can include a convolutional block, a convolutional layer, a residual block, or a pyramid layer. The prediction model 215 can further include one or more fully connected layers (e.g., softmax layers).

[0067] At least a portion of the training input image elements 220a-d, validation input image elements 220e-g, and / or unlabeled input image elements 220h-n can include or can have been derived from data obtained directly or indirectly from a source, which can but need not be an element of the analysis system 205. In some embodiments, the computing environment 200 includes an imaging device 250 that images a sample to obtain image data, such as a multi-channel image (e.g., a multi-channel fluorescence or brightfield image) having a plurality (e.g., between ten and ten) channels. The imaging device 250 can include, without limitation, a camera (e.g., an analog camera, a digital camera, etc.), optics (e.g., one or more lenses, a sensor focusing lens group, a microscope objective, etc.), an imaging sensor (e.g., a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.), photographic film, etc. In digital embodiments, the image capture device can include multiple lenses that can work together to prove instant focus capabilities. An image sensor, e.g., a CCD sensor, can capture a digital image of a sample. In some embodiments, the imaging device 250 is a brightfield imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. The imaging device 250 can utilize non-visible electromagnetic radiation (e.g., UV light) or other imaging techniques to capture images. For example, the imaging device 250 can include a microscope and a camera arranged to capture images magnified by the microscope. The image data received by the image analysis system 205 can be the same as and / or can be derived from the raw image data captured by the imaging device 250.

[0068] In some cases, the labels 225 associated with the training input image elements 220a-d and / or the validation input image elements 220e-g can have been received or derivable from data received from one or more provider systems 255, each of which can be associated with, for example, a physician, a nurse, a hospital, a pharmacist, etc. associated with a particular subject. The received data can include, for example, one or more medical records corresponding to the particular subject. The medical records can indicate, for example, a diagnosis or characterization by a professional that indicates whether the subject has a tumor and / or a stage of progression of the subject’s tumor (e.g., along a standard scale and / or by identifying a metric such as total metabolic tumor volume (TMTV)) corresponding to a time period or a subsequently defined time period corresponding to the time at which one or more input image elements associated with the subject were collected. The received data can further include pixels of a tumor or tumor cells located within the one or more input image elements associated with the subject. Thus, the medical records can include or be used to identify one or more labels for each of the training / validation input image elements 220a-g. The medical records can further indicate each of one or more treatments (e.g., drug treatments) that the subject has received and a time period during which the subject received the treatment. In some cases, images or scans input to one or more classifier subsystems are received from the provider systems 255. For example, the provider systems 255 can receive images from the imaging devices 250 and can then transmit the images or scans (e.g., along with a subject identifier and one or more labels) to the analysis system 205.

[0069] In some embodiments, data received or collected at one or more of the imaging devices 250 can be aggregated with data received or collected at one or more of the vendor systems 255. For example, the analysis system 205 can identify corresponding or identical identifiers for subjects and / or time periods in order to correlate image data received from the imaging devices 250 with label data received from the vendor systems 255. The analysis system 205 can further use metadata or automated image analysis to process the data to determine which classifier subsystem to feed particular data components to. For example, image data received from the imaging devices 250 can correspond to an entire slide or multiple regions of a slide or tissue. Metadata, automated correlation, and / or image processing can indicate for each image which region of a slide or tissue the image corresponds to. For example, automated correlation and / or image processing can include detecting whether an image has image attributes corresponding to a slide substrate or biological structures and / or shapes associated with particular cells, such as white blood cells. Label-related data received from the vendor systems 255 can be slide-specific, region-specific, or subject-specific. When label-related data is specific to a slide or specific to a region, metadata or automated analysis (e.g., using natural language processing or text analysis) can be used to identify which region particular label-related data corresponds to. When label-related data is specific to a subject, the same label data (for a given subject) can be fed to each classifier subsystem 210a-n during training.

[0070] In some embodiments, the computing environment 200 can further include a user device 260 that can be associated with a user that requests and / or coordinates performance of one or more iterations of the analysis system 205 (e.g., where each iteration corresponds to a run of a model and / or a production of output of a model). The user can correspond to a physician, investigator (e.g., associated with a clinical trial), subject, medical professional, etc. Thus, it should be appreciated that in some cases, the vendor systems 255 can include and / or function as the user device 260. Each iteration can be associated with a particular subject (e.g., person) that can (but need not) not be the user. The request for an iteration can include and / or be accompanied by information about the particular subject (e.g., a name or other identifier of the subject, such as a de-identified patient identifier). The request for an iteration can include an identifier of one or more other systems from which data is collected, e.g., input image data corresponding to the subject. In some cases, a communication from the user device 260 includes an identifier of each subject in a set of subjects that corresponds to a request to perform an iteration for each subject represented in the set.

[0071] Upon receiving a request, the analysis system 205 may send a request (e.g., a request including the subject's identifier) ​​for unlabeled input image elements to one or more corresponding imaging systems 250 and / or supplier systems 255. The trained prediction model 215 can then process the unlabeled input image elements to segment non-target or target regions, provide image analysis, provide disease diagnosis and prognosis for treatment of the subject (e.g., a patient), or a combination thereof. The results for each identified subject may include, or may be based on, segmentation and / or one or more output metrics from the trained prediction model 215 deployed from the classifier subsystems 110a-n. For example, segmentation and / or one or more output metrics may include, or may be based on, the output generated by fully connected layers of one or more CNNs. In some cases, such outputs may be further processed using, for example, a softmax function. Furthermore, aggregation techniques (e.g., random forest aggregation) may then be used to aggregate the outputs and / or further processed outputs to generate one or more subject-specific metrics. One or more results (e.g., including plane-specific outputs and / or one or more subject-specific outputs and / or their processed versions) may be transmitted to and / or utilized by the user device 260. In some cases, some or all of the communication between the analysis system 205 and the user device 260 occurs via a website. It should be understood that the CNN system 205 may gate access to results, data, and / or processing resources based on authorized analysis.

[0072] Although not explicitly shown, it should be understood that computing environment 200 may also include a developer device associated with the developer. Communication from the developer device may instruct each predictive model 215 in analysis system 205 what type of input image elements to use, the number of neural networks to use, the configuration of each neural network (including the number of hidden layers and the number of hyperparameters), and how to format data requests and / or which training data to use (e.g., and how to access the training data).

[0073] III. B. OVERVIEW OF MODEL ARCHITECTURE

[0074] FIG. 3 Representative model architectures for non-target region segmentation according to various embodiments are shown (e.g., regarding...). FIG. 2An exemplary schematic diagram 300 of a portion of the analysis system 205. The model architecture can include a pre-processing stage 305 that includes an image acquisition module 310 to generate or obtain input images that include single-channel image data (e.g., where each single-channel image data has a single stain image) and / or multiplexed image data (e.g., images with multiple stains), an optional image annotation module 315 to electronically annotate a portion of the input images (e.g., portions for further analysis), such as portions representing tumor regions or immune cell regions, and an optional unmixing module 320 to generate image channel images corresponding to one or more staining channels. The model architecture can further include a processing stage 325 that includes an image analysis module 330 to detect and / or classify biological material or structures including cells or nuclei (such as tumor cells, stromal cells, lymphocytes, etc.) based on features within the input images (e.g., within hematoxylin and eosin stain images, biomarker images, or unmixing image channel images).

[0075] The model architecture can further include a post-processing stage 335 that includes an optional scoring module 340 to derive expression predictions and / or scores for each biomarker in each of the identified regions or biological structures and an optional metric generation module 345 to derive described metrics and optionally provide a disease diagnosis and prognosis for treatment for a subject (e.g., a patient). The metrics describe variability between the derived expression predictions and / or scores in different regions or biological structures. The model architecture can further include a segmentation and mask module 350 to segment regions or biological structures (such as lymphocyte aggregates or tumor cell clusters) in the input images and generate masks based on the segmented regions or biological structures and an optional registration module 355 to map the identified regions or biological structures (e.g., tumor cells or immune cells) from a first image or first set of images within the input images to at least one additional image or multiple additional images. The segmentation and mask module 350 and the optional registration module 355 can be implemented in the pre-processing stage 305, the processing stage 325, the post-processing stage 335, or any combination thereof.

[0076] The image acquisition module 310 generates or obtains images or image data (e.g., images can be single-channel images or multiplexed images) of a biological sample with one or more stains. In certain embodiments, the generated or obtained images are RGB images or multispectral images. In some embodiments, the generated or obtained images are stored in a memory device. The images or image data (used interchangeably herein) can be generated or obtained using an imaging device (e.g., with respect to the image acquisition module 310) and / or a microscope (e.g., with respect to the image acquisition module 310). FIG. 2The images described are generated or obtained (e.g., in real-time) by the imaging device 250). In some embodiments, the images are generated or obtained from a microscope or other instrument capable of capturing image data of a microscope slide bearing a sample, as described herein. In some embodiments, the images are generated or obtained using a 2D scanner, such as a scanner capable of scanning image tiles. Alternatively, the images can be images that have been previously generated (e.g., scanned) and stored in a memory device (or retrieved from a server via a communication network, as the case can be).

[0077] The image acquisition module 310 is configured to select a portion of a biological sample for which one or more images or image data should be acquired. For example, the image acquisition module 310 can receive an identified region of interest or field of view (FOV). In some embodiments, the region of interest is identified by a user of the system of the present disclosure or a user of another system communicatively coupled to the system of the present disclosure. Alternatively, and in other embodiments, the image acquisition module 305 retrieves a location or identification of the region of interest from a storage / memory device. In some embodiments, the image acquisition module 310 automatically generates a field of view or region of interest (ROI) via the methods described in PCT / EP2015 / 062015, the contents of which are incorporated by reference in their entirety for all purposes. In some embodiments, the ROI is automatically determined by the image acquisition module 305 based on some predetermined criteria or features in or of the image (e.g., for a biological sample stained with more than two stains, identifying a region of the image that contains only two stains). In some cases, the image acquisition module 310 outputs the ROI.

[0078] In some embodiments, the image acquisition module 310 generates or obtains at least two images as input. In some embodiments, the images generated or obtained as input are derived from consecutive tissue sections, e.g., consecutive sections of the same tissue sample. Typically, each of the at least two images received as input includes signals corresponding to a stain, including chromogens, fluorophores, quantum dots, etc. In some embodiments, one of the images has been stained with at least one primary stain (hematoxylin or eosin (H&E)), while the other of the images has been stained in at least one of an IHC assay or an in situ hybridization (ISH) assay to identify a particular biomarker. In some embodiments, one of the images has been stained with both hematoxylin and eosin, while the other of the images has been stained in at least one of an IHC assay or an ISH assay to identify a particular biomarker. In some embodiments, the input images are multiplexed images, e.g., stained for multiple different markers in a multiplexed assay according to methods known to those of ordinary skill in the art.

[0079] The generated or obtained images can be optionally annotated to the image analysis by a user (e.g., a medical professional such as a pathologist) using the image annotation module 315. In some embodiments, the user identifies portions (e.g., sub-regions) of the image that are suitable for further analysis. The target regions or non-target regions (e.g., tumor regions or immune regions) that are annotated to generate a slide score can be the entire tissue region or a set of specific regions on the digital slide. For example, in some embodiments, the identified portions represent overexpressed tumor regions of a particular biomarker (e.g., a particular IHC marker). In other embodiments, the user, medical professional, or pathologist can annotate lymphoid aggregation regions within the digital slide. In some embodiments, the representative fields of annotation can be selected by the pathologist to reflect biomarker expression that the pathologist uses for the overall slide interpretation. The annotations can be drawn using annotation tools provided in the viewer application (e.g., VENTANA VIRTUOSO software), and can be drawn at any magnification or resolution. Alternatively, or in addition, automated image analysis operations can be used to automatically detect target regions and non-target regions or other regions using automated image analysis operations such as segmentation, thresholding, edge detection, etc., as well as fields of view (FOV - a portion of an image having a predetermined size and / or shape) that are automatically generated based on the detected regions.

[0080] In some embodiments, the generated or obtained images can be multiplexed images, i.e., the received images belong to a biological sample that is stained with more than one stain. In these embodiments, the multiplexed images are first de-mixed into their constituent channels (e.g., with de-mixing module 320) before further processing, where each de-mixed channel corresponds to a particular stain or signal. In certain embodiments, the de-mixed images (often referred to as “channel images” or “image channel images”) and can be used as input to each of the modules described herein. For example, a model architecture can be implemented for assessing heterogeneity between markers (an indication of the amount of biological marker protein expression heterogeneity in the sample) determined from a first H&E image stained with clusters for multiple differentiation markers (CD3, CD8, etc.), a second multiplexed image, and multiple single-channel images each stained for a particular biomarker (e.g., ER, PR, Ki67, etc.). In this example, the multiplexed image is first de-mixed into its constituent channel images, and those channel images can be used along with the H&E image and the multiple single-channel images to determine the heterogeneity between markers.

[0081] After image acquisition and / or unmixing, the input images or unmixed image channel images are processed with image analysis algorithms provided by the image analysis module 330 to identify and classify cells and / or nuclei. The procedures and algorithms described herein can be adapted to identify and classify various types of cells or nuclei based on features within the input images, including identifying and classifying tumor cells, non-tumor cells, stromal cells, lymphocytes, non-target stains, etc. Those of ordinary skill in the art will appreciate that nuclei, cytoplasm, and cell membranes have different features and that staining different tissue samples can reveal different biological features. In fact, those of ordinary skill in the art will recognize that certain cell surface receptors can have staining patterns that are localized to the cell membrane or localized to the cytoplasm. Thus, a "cell membrane" staining pattern is analytically different from a "cytoplasm" staining pattern. Likewise, a "cytoplasm" staining pattern is analytically different from a "nucleus" staining pattern. Each of these different staining patterns can be used as a feature to identify cells and / or nuclei. For example, stromal cells can be strongly stained by FAP, but tumor epithelial cells can be strongly stained by EpCAM, while panCK can be stained by cytokeratin. Thus, during image analysis, different cell types can be distinguished by utilizing different stains to provide a classification solution.

[0082] Methods of identifying, classifying, and / or scoring nuclei, cell membranes, and cytoplasm in images of biological samples having one or more stains are described in U.S. Patent No. 7,760,927 ("the '927 patent"), the contents of which are incorporated herein in their entirety for all purposes. For example, the '927 patent describes an automated method for simultaneously identifying a plurality of pixels in an input image of biological tissue stained with a biomarker, the method including considering a first color plane of a plurality of pixels in a foreground of the input image for simultaneously identifying cytoplasm and cell membrane pixels, wherein the input image is processed to remove a background portion and a counterstain component of the input image; determining a threshold level between cytoplasm and cell membrane pixels in the foreground of the digital image; and using the determined threshold level to simultaneously determine whether a selected pixel is a cytoplasm pixel, a cell membrane pixel, or a transition pixel in the digital image using the selected pixel and its eight neighboring pixels. In some embodiments, tumor nuclei are automatically identified by first identifying candidate nuclei, and then automatically distinguishing tumor nuclei from non-tumor nuclei. Various methods of identifying candidate nuclei in tissue images are known in the art. For example, automated candidate nucleus detection is performed by applying a radial symmetry based method, such as that of Parvin, which detects on a hematoxylin image channel or a biomarker image channel after deconvolution (see Parvin, Bahram, et al. "Iterative voting for inference of structural saliency and characterization of subcellular events." Image Processing, IEEE Transactions on 16.3 (2007): 615-623, which is incorporated herein in its entirety for all purposes).

[0083] More specifically, in some embodiments, the obtained image as input is processed, e.g., to detect nucleus centers (seeds) and / or segment nuclei. For example, instructions can be provided and executed to detect nucleus centers based on radial symmetry voting using the techniques of Parvin (described above). In some embodiments, nuclei are detected using radial symmetry to detect the center of a nucleus, and then classify the nucleus based on the intensity of the stain around the cell center. In some embodiments, a radial symmetry based nucleus detection operation is used as described in commonly assigned and co-pending patent application WO2014140085A1, which is incorporated by reference in its entirety. For example, the image size can be computed within the image and one or more votes at each pixel are accumulated by adding the sum of the sizes within a selected region. Mean shift clustering can be used to find the local center of this region, which represents the actual nucleus location. Radial symmetry voting based nucleus detection is performed on color image intensity data, and explicitly uses the prior domain knowledge that nuclei are elliptical-like spots with different sizes and eccentricities. To accomplish the above, in addition to the color intensity in the input image, image gradient information is used for radial symmetry voting and combined with an adaptive segmentation process to accurately detect and localize nuclei. For example, as used herein, a "gradient" refers to the intensity gradient of a particular pixel computed in consideration of the gradient of intensity values of a set of pixels around that particular pixel. Each gradient can have a particular "direction" with respect to a coordinate system whose x and y axes are defined by the two orthogonal edges of the digital image. For example, the detection of nucleus seeds includes defining a seed as a point that is assumed to be located within a nucleus and serves as a starting point for localizing the nucleus. The first step is to detect seed points associated with each nucleus using a very stable method based on radial symmetry to detect elliptical-like spot structures similar to nuclei. The radial symmetry method operates on the gradient image using a kernel-based voting procedure. Each pixel that passes the voting kernel accumulates a vote, thereby creating a voting response matrix. The kernel is based on the gradient direction computed at that particular pixel, the expected range of minimum and maximum nucleus sizes, and the voting kernel angle (typically in the range [p / 4, p / 8]). In the resulting voting space, the local maxima locations with voting values above a predetermined threshold are saved as seed points. Unassociated seeds are discarded in the subsequent segmentation or classification process. Other methods are discussed in U.S. Patent Publication No. 2017 / 0140246, the disclosure of which is incorporated by reference in its entirety.

[0084] After identifying candidate nuclei, the candidate nuclei can be further analyzed to distinguish tumor nuclei from other candidate nuclei. The other candidate nuclei can be further classified (e.g., by identifying lymphocyte nuclei and stromal nuclei). In some embodiments, a learned supervised classifier is applied to identify tumor nuclei, as further described herein. For example, a learned supervised classifier is trained on features of nuclei to identify tumor nuclei, and is subsequently applied to classify candidate nuclei in test images as tumor nuclei or non-tumor nuclei. Optionally, the learned supervised classifier can be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the learned supervised classifier used to identify tumor nuclei is a random forest classifier. For example, a random forest classifier can be trained by (i) creating a training set of tumor and non-tumor nuclei, (ii) extracting features of each nucleus, and (iii) training the random forest classifier to distinguish between tumor nuclei and non-tumor nuclei based on the extracted features. The trained random forest classifier can then be applied to classify nuclei in test images as tumor nuclei and non-tumor nuclei. Optionally, the random forest classifier can be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei.

[0085] Nuclei can be identified using other techniques known to those of ordinary skill in the art. For example, image sizes can be computed from a particular image channel of either the FI&E or IHC images, and a plurality of votes based on the sum of sizes within a region around each pixel can be assigned to each pixel around a particular size. Alternatively, a mean shift clustering operation can also be performed to locate local centers within a voting image that represent the actual locations of nuclei. In other embodiments, nuclei segmentation can be used to segment entire nuclei based on currently known centers of nuclei, through morphological operations and local thresholding. In other embodiments, model-based segmentation can be utilized to detect nuclei (i.e., learn a shape model of nuclei from a training data set, and use it as prior knowledge to segment nuclei in test images).

[0086] In some embodiments, the nuclei are then segmented using a threshold value that is calculated separately for each nucleus. For example, since it is believed that pixel intensity in the region of a nucleus can vary, Otsu's method can be used to segment in the region around the identified nucleus. As will be appreciated by one of ordinary skill in the art, Otsu's method is used to determine the best threshold by minimizing the intra-class variance, and the method is known to those of skill in the art. More specifically, Otsu's method is used to perform automatic clustering-based image thresholding, or reducing a grayscale image to a binary image. The algorithm assumes that the image contains two classes of pixels following a bimodal histogram (foreground pixels and background pixels). Then, the best threshold that separates the two classes of pixels is computed such that the combined spread (intra-class variance) is minimal or equal (since the sum of squared distances is constant), maximizing their inter-class variance.

[0087] In some embodiments, the systems and methods further comprise automatically analyzing spectral and / or shape features of the nuclei identified in the image to identify nuclei of non-tumor cells. For example, blobs can be identified in the first digital image of the first step. As used herein, a "blob" can be, for example, a region of a digital image in which some attribute (such as intensity or grayscale value) remains constant or varies within a specified range of values. In a sense, all pixels in a blob can be considered similar to one another. Blobs can be identified, for example, using differential methods based on derivatives of position functions on the digital image and methods based on local extrema. A nuclear blob is a blob in which the pixels and / or the contour shape suggest that it can have been produced by a nucleus stained with the first stain. For example, the radial symmetry of a blob can be evaluated to determine whether the blob should be identified as a nuclear blob or any other structure, such as a staining artifact. For example, where a blob is long and thin and does not have radial symmetry, the blob can not be identified as a nuclear blob, but rather as a staining artifact. According to embodiments, blobs identified as "nuclear blobs" can represent a set of pixels identified as candidate nuclei and can be further analyzed to determine whether the nuclear blob represents a nucleus. In some embodiments, any kind of nuclear blob is used directly as an "identified nucleus." In some embodiments, filtering operations are performed on the identified nuclei or nuclear blobs to identify nuclei that do not belong to a tumor cell that is positive for the biomarker and remove the identified non-tumor cell nuclei from the list of identified nuclei or not add the nuclei to the list of identified nuclei from the start. For example, additional spectral and / or shape features of the identified nuclear blobs can be analyzed to determine whether the nuclei or nuclear blobs are nuclei of tumor cells. For example, nuclei of lymphocytes are larger than nuclei of other tissue cells, such as lung cells. Where the tumor cells are derived from lung tissue, nuclei of lymphocytes can be identified by identifying nuclear blobs that have a minimum size or diameter that is significantly larger than the average size or diameter of nuclei of normal lung cells. The identified nuclear blobs associated with lymphocyte nuclei can be removed (i.e., "filtered") from the set of identified nuclei. By filtering nuclei of non-tumor cells, the accuracy of the method can be improved. Depending on the biomarker, non-tumor cells can also express the biomarker to some extent, and thus can produce an intensity signal in the first digital image that is not derived from a tumor cell. By identifying and filtering nuclei that do not belong to a tumor cell from the total number of identified nuclei, the accuracy of identifying biomarker-positive tumor cells can be improved. These and other methods are described in U.S. Patent Publication 2017 / 0103521, the contents of which are incorporated herein in their entirety for all purposes. In some embodiments, once a seed is detected, a locally adaptive thresholding method can be used and blobs can be created around the center of the detection.In some embodiments, other methods can also be introduced, for example, a marker-based watershed algorithm can also be used to identify the nuclei spots around the detected nuclei centers. These and other methods are described in PCT / EP2016 / 051906, published as WO2016 / 120442, the contents of which are hereby incorporated by reference in their entirety for all purposes.

[0088] In some embodiments, the scoring module 340 is used to compute various marker expression scores for each stain or biomarker within each cell cluster within each image (single pass images or unmixed image channel images from multiplexed images). In some embodiments, the scoring module 340 utilizes data gathered by the image analysis module 330 during detection and classification of cells. For example, as described herein, the cell analysis module 330 can include a series of image analysis algorithms, and can be used to determine whether one or more of a nucleus, cell wall, tumor cell, or other structure is present within an identified cell cluster. In some embodiments, the scoring module 340 can use the stain intensity values and specific nuclear counts derived for each field to determine various marker expression scores, such as a percent positive or H-Score score. Scoring methods are described in further detail in commonly-assigned and co-pending applications WO / 2014 / 102130 Al “Image analysis for breast cancer prognosis” filed December 19, 2013 and WO / 2014 / 140085 Al “Tissue object-based machine learning system for automated scoring of digital whole slides,” filed March 12, 2014, the contents of each of which are hereby incorporated by reference in their entirety. For example, automated image analysis algorithms in the image analysis module 330 can be used to interpret each of the IFIC slides in the series to detect tumor nuclei that are positive and negative for a particular biomarker, such as Ki67, ER, PR, HER2, etc. Based on the detected positive and negative tumor nuclei, various slide-level scores can be computed using the scoring module 340, such as a marker percent positive, H-Score score, etc.

[0089] In some embodiments, the expression score is an H-score score, which is used to assess the percentage of tumor cells with membrane staining graded as "weak," "moderate," or "strong." The individual grades are added together, resulting in a total score that can be up to 300, and the cut-off point that distinguishes "positive" from "negative" is 100. For example, the intensity of membrane staining (0, 1+, 2+, or 3+) is determined for each cell in a fixed field of view (or for each cell in a tumor or cell cluster, as the case can be). The H-score can simply be based on one predominant staining intensity, or more complexly, can include a sum of individual H-score scores for each intensity level seen. In some embodiments, the expression score is an Allred score. The Allred score is a scoring system that shows both the percentage of cells that test positive for a hormone receptor, and the degree to which the receptor is present after staining (called "intensity"). In other embodiments, the expression score is a percentage positive. In the context of scoring breast cancer samples that are stained for PR and Ki-67 biomarkers, for PR and Ki-67 slides, the percentage positive is calculated in a single slide (e.g., the total number of cell nuclei that are positive (e.g., malignant cells) in each field of view within a digital image of the slide after staining is added together and divided by the total number of cell nuclei that are positive and negative for staining from each field of view of the digital image), as follows: % positive = number of cells positive for staining / (number of cells positive for staining + number of cells negative for staining). In other embodiments, the expression score is an IHC composite score, which is a prognostic score based on several IHC markers, where the number of markers is greater than one. IHC4 is one such score based on four measured IHC markers in a breast cancer sample (i.e., ER, HER2, Ki-67, and PR) (see, e.g., Cuzick et al., J. Clin. Oncol. 29:4273-8, 2011, and Barton et al., Br. J. Cancer 1-6, Apr. 24, 2012, both of which are incorporated herein by reference).

[0090] After image analysis and optionally determining expression scores for each marker in each identified cluster or mapped cluster, metrics can be derived from the various identified clusters and biological structures using a metric generation module 345. In still other cases, morphological metrics can be computed by applying various image analysis algorithms on top of the pixels contained within or around the nuclear spots or seeds. In some embodiments, morphological metrics include area, minor and major axis length, perimeter, radius, volume, etc. At the cellular level, such metrics are used to classify the cell nuclei as belonging to healthy cells or diseased cells. At the tissue level, these statistical features are leveraged across the tissue to classify the tissue as diseased tissue or non-diseased tissue. In some cases, appearance metrics for a particular cell nucleus can be computed by comparing pixel intensity values of pixels contained within or around the nuclear spot or seed used to identify the cell nucleus, where the compared pixel intensities are derived from different image channels (e.g., background channel, channel for staining of a biomarker, etc.). In some embodiments, the metrics derived from appearance features are computed from the pixel intensities and from the percentiles of the gradient magnitude computed from different image channels (e.g., 10th, 50th, and 95th percentiles). For example, a number P of an X-percentile value (X = 10, 50, 95) of pixel values in each of a plurality of image channels IC (e.g., three channels: HTX, DAB, Brightness) is first determined within a nuclear spot representing a target cell nucleus. It can be advantageous to compute appearance feature metrics because the derived metrics can describe properties of the cell nucleus region as well as the membrane region surrounding the cell nucleus.

[0091] In some cases, background metrics can be computed that are indicative of the appearance and / or staining present in the cytoplasm and membrane features of a cell that includes the nucleus from which the background features are extracted from the image. For example, by identifying a nuclear spot or seed representing a cell nucleus, background features and corresponding metrics can be computed for the cell nucleus and corresponding cell depicted in the digital image; a region of pixels directly adjacent to the group of identified cells (e.g., a band of 20 pixels, about 9 microns thick, around the nuclear spot boundary) is analyzed to capture the appearance and staining present in the cytoplasm and membrane of the cell and the region directly adjacent to the cell along with this nucleus. In some cases, color metrics can be derived from the colors including color ratios, R / (R+G+B), or color principal components. In other embodiments, color metrics derived from the colors include local statistics (mean / median / variance / standard deviation) and / or color intensity correlations of each of the colors in a local image window. In some cases, intensity metrics can be derived from the group of adjacent cells having certain attribute values of the intensity set between the black and white shades of the gray cells represented in the image. In this way, the intensity of these colored cells is determined to be affected from the cluster of dark cells surrounding them, as the correlation of color features can define an example of size grading.

[0092] In some cases, other features such as texture features or spatial features can be considered and used as a basis for calculating metrics, such as the expression score. For example, the expression score can be utilized as a predictive measure or to guide treatment. For example, in the context of breast cancer and ER and PR biomarkers, a sample that tests positive can guide the decision to provide hormone therapy during treatment. Those skilled in the art will also appreciate that not all clusters in a biological sample can have the same score for any particular marker. By being able to determine a heterogeneity score or metric that describes variability between clusters, additional guidance can be provided to make informed treatment decisions. In some embodiments, heterogeneity is determined to measure how different clusters compare to each other. Heterogeneity can be measured by a variability metric as described in WO2019110567A1, the contents of which are incorporated herein in their entirety for all purposes, which describes, for example, how protein expression levels differ between various identified and mapped clusters relative to each other. In some embodiments, heterogeneity is measured between all identified clusters. In other embodiments, heterogeneity is measured only between a subset of identified clusters, such as clusters that meet certain predetermined criteria.

[0093] In some embodiments, the images received as input can be segmented and masked by segmentation and masking module 350. For example, a trained convolutional neural network architecture or model can be used to segment non-target regions and / or target regions, which can then be masked for analysis prior to, during, or after inputting the images to the image analysis algorithm. In some embodiments, the input images are masked such that only tissue regions are present in the images. In some embodiments, a tissue region mask is generated to mask non-tissue regions from the tissue regions. In some embodiments, the tissue region mask can be created by identifying tissue regions and excluding background regions, such as regions of the entire slide image that correspond to glass with no sample, such as regions where only white light from the imaging source is present.

[0094] In some embodiments, a tissue region mask image is generated using segmentation techniques by masking tissue regions from non-tissue regions in the input image. In some embodiments, the image segmentation techniques are utilized to distinguish between digitized tissue data and the slide in the image, with tissue corresponding to foreground and the slide corresponding to background. In some embodiments, segmentation and masking module 350 computes a region of interest (AOI) in the entire slide image to detect all tissue regions in the AOI while limiting the amount of background non-tissue regions analyzed. A variety of different image segmentation techniques (e.g., HSV color based image segmentation, lab image segmentation, mean shift color image segmentation, region growing, level set methods, fast marching methods, etc.) can be used to determine the boundaries of, for example, tissue data and non-tissue or background data. Based at least in part on the segmentation techniques, segmentation and masking module 350 can also generate a tissue foreground mask that can be used to identify portions of the digitized slide data that correspond to tissue data. Alternatively, the component can generate a background mask that is used to identify portions of the digitized slide data that do not correspond to tissue data.

[0095] Such identification can be achieved through image analysis operations (e.g., edge detection, etc.). The tissue region mask can be used to remove non-tissue background noise in the image (e.g., non-tissue regions). In some embodiments, the generation of the tissue region mask includes one or more of the following operations (but is not limited to the following operations): computing the intensity of the low resolution input image, generating an intensity image, applying a standard deviation filter to the intensity image, generating a filtered intensity image, and applying a threshold to the filtered intensity image, setting pixels with intensity above a given threshold to 1 and below the threshold to 0, generating the tissue region mask. Additional information and examples related to the generation of the tissue region mask are disclosed in PCT / EP / 2015 / 062015 entitled "An Image Processing Method and System for Analyzing a Multi-Channel Image Obtained from a Biological Tissue Sample Being Stained by Multiple Stains", the contents of which are incorporated herein in their entirety for all purposes.

[0096] In addition to masking non-tissue regions from tissue regions, the segmentation and masking module 350 can also mask other target regions as needed, such as tissue identified as belonging to a non-target region or part of certain tissue types (e.g., part of a lymphocyte aggregation region) or tissue identified as belonging to a target region or certain tissue types (e.g., a suspected tumor region). In various embodiments, non-target region segmentation (such as lymphocyte aggregation region segmentation) is performed by a CNN model (e.g., with respect to...). FIG. 2 The classifier subsystem 210a shown is associated with a CNN model. In some embodiments, the CNN model is a two-dimensional segmentation model. For example, the CNN model may be a U-Net with residual blocks, dilation, and depthwise convolution. Preprocessed or processed image data (e.g., a two-dimensional region or an entire slide image) may be used as input to the U-Net. The U-Net includes a contraction path and an expansion path, wherein pooling operations of successive layers in the expansion path are replaced with upsampling operators. Thus, these successive layers increase the resolution of the previous layer's output and make the final output of the U-Net have the same resolution as the input image. Based on at least a partial segmentation technique, the U-Net can generate a foreground mask of non-target regions that can be used to identify portions of the digitized slide data that correspond to non-target region data. Alternatively, the component can generate a background mask for identifying portions of the digitized slide data that do not correspond to non-target region data. The output of the U-Net may be a foreground non-target region mask representing the location of non-target regions present in the underlying image, or a background non-target region mask representing portions of the digitized slide data that do not correspond to non-target region data (e.g., target regions).

[0097] In some embodiments, the biological material or structures identified in one or more images, such as tumor cells or cell clusters, are mapped to one or more additional images using the registration module 355 and a registration process. Registration is the process of transforming different data sets (here, images or cell clusters within images) into one coordinate system. More specifically, registration is the process of aligning two or more images, which generally involves designating one image as a reference (also referred to as a reference image or a fixed image) and performing a geometric transformation on the other images to align them to the reference. A geometric transformation maps locations in one image to new locations in another image. The step of determining the correct geometric transformation parameters is the key to the image registration process. In some embodiments, image registration is performed using the methods described in WO / 2015 / 049233, entitled "Line-Based Image Registration and Cross-Image Annotation Devices, Systems and Methods," filed September 30, 2014, the contents of which are incorporated herein in their entirety for all purposes. WO / 2015 / 049233 describes a registration process that includes a coarse registration process used alone or in combination with a fine registration process. In some embodiments, the coarse registration process can include selecting digital images for comparison, generating a foreground image mask from each of the selected digital images, and matching tissue structures between the foreground images thus generated. In further embodiments, generating a foreground image mask involves generating a soft-weighted foreground image from an entire slide image of a stained tissue section and applying OTSU thresholding to the soft-weighted foreground image to produce a binary soft-weighted image mask. In other further embodiments, generating a foreground image mask involves generating a binary soft-weighted image mask from an entire slide image of a stained tissue section, separately generating a gradient magnitude image mask from the same entire slide image, applying OTSU thresholding to the gradient image mask to produce a binary gradient magnitude image mask, and merging the binary soft-weighted image and the binary gradient magnitude image masks using a binary OR operation to generate the foreground image mask. For example, "gradient" as used herein refers to the intensity gradient of a particular pixel calculated in consideration of the intensity values of a set of pixels surrounding that particular pixel. Each gradient can have a particular "direction" with respect to a coordinate system, the x and y axes of which are defined by two orthogonal edges of the digital image. A "gradient direction feature" can refer to a data value indicative of the direction of the gradient within that coordinate system.

[0098] In some embodiments, matching the tissue structures includes computing line-based features from the boundaries of each thus-generated foreground image mask, computing global transformation parameters between a first set of line features on the first foreground image mask and a second set of line features on the second foreground image mask, and globally aligning the first and second images based on the transformation parameters. In yet another embodiment, the coarse registration process includes mapping the selected digital images to a common grid based on the global transformation parameters, which can encompass the selected digital images. In some embodiments, the fine registration process can involve identifying a first sub-region of a first digital image in the aligned digital image set; identifying a second sub-region on a second digital image in the aligned digital image set, where the second sub-region is larger than the first sub-region and the first sub-region is substantially within the second sub-region on the common grid; and computing an optimized position of the first sub-region in the second sub-region.

[0099] III. C. EXAMPLE U-NET FOR TWO-DIMENSIONAL SEGMENTATION

[0100] Two-dimensional segmentation separately uses a tile-based approach and a modified U-Net to extract features from input images (e.g., one or more images of a sample slide) to generate a two-dimensional non-target region map with high resolution. As FIG. 4As shown, the tile images 400 (e.g., IHC images) can be used to train a modified U-Net with human annotations 405 to identify non-target regions (e.g., lymphoid aggregates) and target regions (e.g., tumors and other cells). For example, the modified U-Net can be trained using ground truth segmentation masks 410 and output a foreground / background probability for each pixel, where a loss function is constructed to measure the difference between the foreground / background probability for each pixel predicted using the U-Net and the ground truth segmentation masks 410. The ground truth segmentation masks 410 can be generated from the tile images 400 with human annotations 405. The tile images 400 can be segmented into relatively small patches 415 of a predetermined size (e.g., a size of 256 x 256 for both original black and white or color images) to generate corresponding ground truth segmentation masks 410. A pair of small patches 415 of a black and white or color image or corresponding ground truth segmentation masks 410 are input to the modified U-Net. In some cases, the data can be split into an 80% training set and a 20% test or validation set. In some cases, a learning rate schedule is used to adjust the learning rate of the modified U-Net during training by lowering the learning rate according to a predefined schedule. In some cases, the learning rate schedule is a step decay schedule that lowers the learning rate by a predetermined factor every few epochs for optimizing the loss function. For example, the step decay schedule can be used to lower the learning rate by a quarter every 15 epochs. In some cases, a binary cross-entropy loss function is used to train the modified U-Net using hyperparameters set to: 50 epochs, a patch size of 2, and a learning rate of 1 x 10-5. Training can be performed using two different magnifications (e.g., 20x and 10x) to find the best modified U-Net. FIG. 5A The loss is shown with respect to these patches during training of the modified U-Net. FIG. 5B The accuracy is shown during training of the modified U-Net to achieve the best performance.

[0101] As FIG. 6As shown, the U-Net 600 can include a contracting path 605 (encoder) and an expansive path 610 (decoder), which gives it a u-shaped architecture. The contracting path 605 is a CNN network that includes repeated applications of convolutions (e.g., 3x3 convolutions (unpadded convolutions)), each followed by a rectified linear unit (ReLU) and a max-pooling operation (e.g., 2x2 max-pooling with a stride of 2) for down-sampling. At each down-sampling step or pooling operation, the number of feature channels can be doubled. During the contraction, the spatial information of the image data decreases while the feature information increases. The expansive path 610 is a CNN network that combines the features and spatial information from the contracting path 605 (up-sampling of the feature maps from the contracting path 605) with repeated applications of convolutions (e.g., two 3x3 convolutions) each followed by a rectified linear unit (ReLU), concatenation with the corresponding cropped feature maps from the contracting path 605, and a final convolution (e.g., one 1x1 convolution) to generate a two-dimensional non-target region mask. For localization, high-resolution features from the contracting path 605 are combined with the up-sampled output from the expansive path 610.

[0102] In various embodiments, the U-Net 600 implements a reduced number of channels overall compared to a traditional U-Net architecture. Specifically, the number of channels of the intermediate activation output layers is reduced by a predetermined factor, e.g., two or four (see, e.g., bold numbers within the contracting path 605 and the expansive path 610), e.g., in the second layer, the number of channels is reduced by a factor of four from 64 to 16, etc., and the maximum number of channels is also reduced from 1024 used in a traditional U-Net architecture to 256. This reduction in channels is implemented to reduce computational expense and model complexity. Experimental evidence shows that this compressed U-Net architecture provides better results compared to a traditional U-Net architecture with a maximum number of 1024 channels and no reduction in the number of channels. In addition, the U-Net 600 includes spatial random dropout 615 to combat overfitting. The spatial random dropout 615 can be implemented in the last few layers (i.e., the last, second last, third last, or fourth last) of the contracting path 605. The spatial random dropout 615 deactivates entire two-dimensional feature maps, rather than individual elements as performed by traditional dropout. For example, if neighboring pixels in a feature map are strongly correlated (as is often the case in early convolutional layers), then traditional dropout does not regularize the activations, otherwise leading to an effective learning rate decrease. In contrast, the spatial random dropout 615 will regularize the activations and help promote independence between feature maps, otherwise leading to an effective learning rate increase.

[0103] III. D. TECHNIQUES FOR SEGMENTATION AND ANALYSIS

[0104] FIG. 7 A process 700 for training a prediction model is shown in accordance with various embodiments.

[0105] The process 700 begins at block 705, where a plurality of tile images of a sample are accessed. One or more tile images of the plurality of tile images contain annotations to segment non-target regions and target regions of the one or more tile images. At block 710, the one or more tile images are divided into image patches of a predetermined size. For example, tile images often have random sizes, while two-dimensional segmentation models, such as U-Net, learn better on normalized image sizes, and thus the tile images can be divided into image patches of a particular size to optimize training. In some embodiments, the one or more tile images are divided into image patches of a predetermined size of 64 pixels x 64 pixels, 128 pixels x 128 pixels, 256 pixels x 256 pixels, or 512 pixels x 512 pixels. At block 715, a prediction model, such as a two-dimensional segmentation model, is trained on the image patches. In some cases, the two-dimensional segmentation model is a modified U-Net model that includes a contracting path and an expansive path, each of which has a maximum predetermined number of channels, and one or more layers of the contracting path implement spatial random dropout. In some embodiments, the maximum predetermined number of channels is 64, 128, 256, or 512. Training can include performing an iterative operation to find a set of parameters of the prediction model that minimizes a loss function of the prediction model. Each iteration can involve finding a set of parameters of the prediction model such that a value of the loss function using the set of parameters is less than a value of the loss function using another set of parameters in a previous iteration. The loss function is structured to measure a difference between an output predicted using the prediction model and the annotations contained in the one or more tile images or image patches. In some cases, training further includes adjusting a learning rate by decreasing a learning rate of the modified U-Net according to a predefined schedule. The predefined schedule can be a step decay schedule that decreases the learning rate by a predetermined factor every predetermined number of epochs to optimize the loss function. In certain cases, the loss function is a binary cross-entropy loss function. At block 720, the trained prediction model is provided. For example, as discussed with respect to FIG. 1, the trained prediction model can be deployed for execution in an image analysis environment. FIG. 2 and FIG. 3

[0106] FIG. 8A A process 800 for generating a segmentation mask for non-target regions in image data is shown in accordance with various embodiments.

[0107] The process 800 begins at block 805, where a plurality of images of a sample are accessed. One or more images of the plurality of images include non-target regions and target regions. FIG. 8B ​An example of raw images of one or more images is shown. In some cases, a sample is stained for one or more biomarkers. At block 810, one or more images (e.g., tile images) are divided into image patches of a predetermined size. In some embodiments, one or more tile images are divided into image patches of a predetermined size of 64 pixels x 64 pixels, 128 pixels x 128 pixels, 256 pixels x 256 pixels, or 512 pixels x 512 pixels. At block 815, the image patches are encoded into discriminative features. The discriminative features (e.g., lower resolution features) can be associated with biological material or structures (e.g., lymphoid aggregates or immune cell groups) that are at least a portion of the non-target region. The encoding can be performed by a prediction model (e.g., a two-dimensional segmentation model). In some cases, the two-dimensional segmentation model is a modified U-Net model including a contracting path and an expansive path, each of the contracting path and the expansive path having at most 256 channels, and one or more layers of the contracting path implementing spatial stochastic dropout. The contracting path can perform the encoding of the image patches into discriminative features at multiple different levels.

[0108] At block 820, the discriminative features are projected to pixel space (e.g., higher resolution) and a classification of the biological material or structures is determined for each pixel space. The projection and determination of the classification can be performed by a prediction model (e.g., a two-dimensional segmentation model). In some cases, the two-dimensional segmentation model is a modified U-Net model. The expansive path can perform the projection of the discriminative features and the determination of the classification at multiple different levels. In some cases, the multiple different levels perform upsampling (i.e., expanding the feature dimension to the original size of the input image patch) and concatenation followed by regular convolutional operations to project the discriminative features and determine the classification. In some cases, the discriminative features are semantically projected to assign a class to each individual pixel in the image patch (dense prediction) and determine the classification of the biological material or structures as individual instances. The classification of each pixel in the image patch can be determined based on a predetermined threshold, and in some cases, the predetermined threshold is a hyperparameter optimized in the training of the prediction model.

[0109] At block 825, a segmentation map of the non-target region is predicted and output based on the classification of the biological material or structures. The segmentation map is output as a high resolution image patch in which all pixels are classified (e.g., presence or absence of the biological material or structures). In some embodiments, the segmentation map is output at a size of 256 pixels x 256 pixels in which each pixel has a classification in a range of 0 to 1, where 0 indicates no biological material or structures and 1 indicates presence of the biological material or structures. The generation can be performed by a prediction model (e.g., a two-dimensional segmentation model). In some cases, the two-dimensional segmentation model is a modified U-Net model.

[0110] At box 830, the segmentation maps of the non-target regions are magnified and combined to generate images masked by the non-target regions. These masked images provide segmentation masks for the non-target regions of one or more images containing biological material or structures. For example, 256×256 blocks of images including segmentation maps of the non-target regions can be stitched together to create non-target regions (see [link to documentation]). FIG. 8C The image shown is a tile mask image that provides a segmentation mask compared to the predicted tile mask image (which is the actual tile mask image). In some cases, the image masked by the non-target region excludes the non-target region from one or more images, so that the non-target region can be excluded from downstream processing (e.g., with respect to the image analysis discussed in Figure 9). At box 835, the image masked by the non-target region is provided. For example, the image masked by the non-target region can be provided to a memory storage device, to a display of a computing device, or to a user in one or more types of media (such as a user interface, etc.).

[0111] FIG. 9A A process 900 for applying a segmentation mask to support or improve image analysis is illustrated according to various embodiments.

[0112] Procedure 900 begins at box 905, where multiple images of the sample are accessed. One or more of these images include both non-target and target regions, see, for example... FIG. 8B In some cases, the sample is stained against one or more biomarkers. In some embodiments, the accessed image is an RGB image or a multispectral image. In some embodiments, the accessed image is stored in a memory device. An imaging device (e.g., regarding...) can be used. FIG. 2 The imaging apparatus 250 shown generates or acquires images (e.g., in real time). In some embodiments, as described herein, the images are acquired from a microscope or other instrument capable of capturing image data of a microscope slide carrying a sample. In some embodiments, the images are accessed from, for example, a 2D scanner capable of scanning image blocks. Alternatively, the images may be images that have been previously generated (e.g., scanned) and stored in a memory device (or, in this respect, retrieved from a server via a communication network). In some cases, the images are non-target region masked images that provide a segmentation mask for non-target regions containing biological material or structures, such as lymphoid aggregates or immune cell groups. Non-target region masked images for access can be generated and / or acquired, as per [reference to...]. FIG. 8A As shown.

[0113] In some embodiments, the input image accessed is a multiplexed image, i.e., the received image is of a biological sample stained with more than one stain. In these embodiments at block 910, and prior to further processing, the multiplexed image is unmixed into its constituent channels, where each unmixed channel corresponds to a particular stain or signal. After image acquisition and / or unmixing, the image or unmixed image channel images are processed with image analysis algorithms in blocks 915-945 to identify and classify cells and / or nuclei. The processes and analysis algorithms described herein can be adapted to identify and classify various types of cells or nuclei based on features within the input image, including identifying and classifying tumor cells, non-tumor cells, stromal cells, lymphocytes, non-target stains, etc.

[0114] At block 915, candidate nuclei are identified. In some embodiments, the image is input into image analysis to detect nuclei centers (seeds) and / or segment nuclei. For example, a plurality of pixels in an image stained with a biomarker can be identified, including considering one or more color planes of the plurality of pixels in the foreground of the input image, for simultaneous identification of cytoplasm and cell membrane pixels. In some cases, the image is pre-processed to remove portions of the image that are not required for analysis or cannot be determined to be target regions, such as slide background, non-target regions as shown with respect to FIG. 8, and / or complex staining components of the image. Thereafter, a threshold level between cytoplasm and cell membrane pixels in the foreground of the digital image is determined, and selected pixels from the foreground and their predetermined number of neighboring pixels are processed based on the determined threshold level to determine whether the selected pixels are cytoplasm pixels, cell membrane pixels, or transition pixels in the digital image. In some embodiments, tumor nuclei are automatically identified on the hematoxylin image channel or biomarker image channel after unmixing by applying a radial symmetry based method (Parvin’s radial symmetry based method).

[0115] At block 920, features are extracted from the candidate nuclei. At least some of the features can be related to another biological material or structure within the target region, such as tumor cells or tumor cell clusters. The extraction can be performed by an image analysis prediction model, such as Mask R-CNN capable of semantic or instance segmentation. For example, after identifying the candidate nuclei, the candidate nuclei can be further analyzed by the image analysis prediction model to distinguish tumor nuclei from other candidate nuclei, such as lymphocyte nuclei. In some cases, the other candidate nuclei can be a further process to identify specific classes of nuclei and / or cells, such as identifying lymphocyte nuclei and stromal nuclei. At block 925, based on the features extracted from the candidate nuclei, the other biological material or structure within the target region is classified as a type of cell or nucleus. The classification can be performed by the image analysis prediction model. In some embodiments, a learned supervised classifier is applied to identify tumor nuclei from the candidate nuclei. For example, the learned supervised classifier can be trained on the features of the nuclei to identify tumor nuclei, and subsequently applied to classify candidate nuclei in a test image as tumor nuclei or non-tumor nuclei. Optionally, the learned supervised classifier can be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the learned supervised classifier for identifying tumor nuclei is a random forest classifier. For example, the random forest classifier can be trained by (i) creating a training set of tumor and non-tumor nuclei, (ii) extracting features of each nucleus, and (iii) training the random forest classifier to distinguish between tumor nuclei and non-tumor nuclei based on the extracted features. The trained random forest classifier can then be applied to classify nuclei in a test image as tumor nuclei and non-tumor nuclei. Optionally, the random forest classifier can be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei.

[0116] At block 930, a segmentation map of the target region is predicted and output based on the classification of the other biological material or structure. The segmentation map can be output by the image analysis prediction model. At block 935, the segmentation map of the target region is upscaled and combined to generate target region masked images that provide a segmentation mask for the target region of one or more images containing the other biological material or structure. In other cases, the segmentation map of the target region and a segmentation map of non-target regions (from about FIG. 8AThe images (obtained in the process described) are scaled up and combined to generate target region and non-target region masked images that provide a segmentation mask for target regions containing another biological material or structure (e.g., tumor cells) and a segmentation mask for non-target regions containing biological material or structures (e.g., lymphoid aggregates or immune cell groups). In this way, an observer (human user or computing device) can distinguish between target regions and non-target regions.

[0117] At optional block 935, metrics are derived from the various identified nuclei, cells, cell clusters, and / or biological material or structures, as discussed in detail herein. In some cases, one or more metrics can be computed by applying various image analysis algorithms on pixels contained in or surrounding the classified nuclei, cells, cell clusters, and / or biological material or structures. In some embodiments, the metrics include disease status, area, short and long axis length, perimeter, radius, solidity, etc. As FIG. 9B As shown, the original image with the overlay analysis results can tend to provide false positives or over-detection of tumor cells that are actually lymphoid aggregates or immune cell groups. However, as FIG. 9C As shown, the original image with the overlay analysis results and the implementation of the segmentation framework described herein (i.e., the non-target region masked image) are able to detect lymphoid aggregates or immune cell groups and remove false positives or over-detection of tumor cells. At block 940, the target region masked image or the target region and non-target region masked image is provided. For example, the target region masked image or the target region and non-target region masked image can be provided to a memory storage device, to a display of a computing device, to a user (e.g., a user interface, etc.) in one or more types of media, in some cases, providing the target region masked image or the target region and non-target region masked image includes providing the associated metrics, or the metrics can be provided separately.

[0118] IV. ADDITIONAL CONSIDERATIONS

[0119] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer- readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine- readable storage medium including instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes.

[0120] The terminology and expressions employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the application claimed. Thus, it should be understood that although the present application has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed can be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this application as defined by the appended claims.

[0121] The following description is provided in relation to preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the following description of preferred exemplary embodiments will provide those skilled in the art with an enabling description of how the various embodiments are implemented. It should be understood that various changes can be made to the function and arrangement of elements without departing from the spirit and scope of the disclosure as set forth in the appended claims.

[0122] In the following description, specific details are set forth to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form, rather than in detail in order to avoid obscuring the embodiments. In other instances, such circuits, processes, algorithms, structures, and techniques can be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. A computer-implemented method comprising: accessing a plurality of images of a specimen, wherein one or more images of the plurality of images contain a non-target region and a target region; predicting, by a two-dimensional segmentation model, a segmentation map of the non-target region based on discriminative features encoded from the one or more images, wherein the discriminative features are associated with a first biological material or structure, wherein the first biological material or structure is a lymphoid aggregate, and wherein the two-dimensional segmentation model is a modified U-Net model comprising a contracting path and an expansive path, each of the contracting path and the expansive path having at most 256 channels, and one or more layers of the contracting path implementing spatial drop out; generating a segmentation mask of the one or more images based on the segmentation map; applying the segmentation mask to the one or more images to generate a non-target region masked image that excludes the non-target region from the one or more images; classifying, by an image analysis model, a second biological material or structure within the target region into a type of cell or nucleus based on a feature set extracted from the non-target region masked image, wherein the second biological material or structure is a tumor cell or tumor cell cluster; and providing, for the target region, the type of cell or nucleus classified for the second biological material or structure.

2. The computer-implemented method of claim 1, wherein the specimen is stained for one or more biomarkers.

3. The computer-implemented method of claim 1 or 2, further comprising: partitioning the one or more images into image patches having a predetermined size; encoding, by the two-dimensional segmentation model, the image patches into the discriminative features; projecting, by the two-dimensional segmentation model, the discriminative features onto a pixel space; and determining, by the two-dimensional segmentation model, a classification of the first biological material or structure for each pixel space based on a predetermined threshold.

4. The computer-implemented method of claim 1, 2, or 3, further comprising: extracting, by the image analysis model, the feature set from the non-target region masked image; and computing one or more metrics of the second biological material or structure based on the feature set, wherein the providing the type of cell or nucleus classified for the second biological material or structure further comprises providing the one or more metrics of the second biological material or structure.

5. A non-transitory computer-readable storage medium comprising computer program instructions that, when executed by a computer processor, cause the computer processor to perform the acts or operations of the computer-implemented method of any of claims 1-4.

6. A computer system comprising a processor and a non-transitory memory, the memory comprising computer program instructions that, when executed by the processor, cause the processor to perform the acts or operations of the computer-implemented method of any of claims 1-4.

7. A computer-implemented method comprising: ​ ​ accessing a plurality of images of a sample, wherein one or more images of the plurality of images contain a non-target region and a target region; predicting, by a two-dimensional segmentation model, a segmentation map of the non-target region based on discriminative features encoded from the one or more images, wherein the discriminative features are associated with a first biological material or structure, wherein the first biological material or structure is a lymphoid aggregate, and wherein the two-dimensional segmentation model is a modified U-Net model that includes a contracting path and an expansive path, each of the contracting path and the expansive path having at most 256 channels, and one or more layers of the contracting path implementing spatial random dropout; generating a first segmentation mask of the one or more images based on the segmentation map of the non-target region; classifying, by an image analysis model, a second biological material or structure within the target region as a type of cell or nucleus based on a second set of features extracted from the one or more images, wherein the second biological material or structure is a tumor cell or tumor cell cluster; generating a second segmentation mask of the one or more images based on the classification of the second biological material or structure within the target region; applying the first segmentation mask and the second segmentation mask to the one or more images to generate a target region and non-target region masked image that excludes the non-target region from the one or more images; and providing the target region and non-target region masked image and the type of cell or nucleus classified for the second biological material or structure.

8. The computer-implemented method of claim 7, wherein the sample is stained for one or more biomarkers.

9. The computer-implemented method of claim 7 or 8, further comprising: partitioning the one or more images into image patches having a predetermined size; encoding, by the two-dimensional segmentation model, the image patches into the discriminative features; projecting, by the two-dimensional segmentation model, the discriminative features onto a pixel space; and determining, by the two-dimensional segmentation model, a classification of the first biological material or structure for each pixel space based on a predetermined threshold.

10. The computer-implemented method of claim 7, 8, or 9, further comprising: extracting, by the image analysis model, the second set of features from the one or more images; and computing one or more metrics of the second biological material or structure based on the second set of features, wherein the providing the target region and non-target region masked image and the type of cell or nucleus classified for the second biological material or structure further comprises providing the one or more metrics of the second biological material or structure.

11. A non-transitory computer-readable storage medium comprising computer program instructions that, when executed by a computer processor, cause the computer processor to perform the acts or operations of the computer-implemented method of any one of claims 7-10. ​ ​ ​ 12. A computer system comprising a processor and a non-transitory memory, the memory comprising computer program instructions which, when executed by the processor, cause the processor to perform the actions or operations of the computer-implemented method of any of claims 7-10.

13. A computer-implemented method comprising: accessing a plurality of tile images of a sample, wherein one or more tile images of the plurality of tile images comprise annotations of segmented non-target regions and target regions; training a two-dimensional segmentation model on the one or more tile images, wherein the two-dimensional segmentation model is a modified U-Net model comprising a contracting path and an expansive path, each of the contracting path and the expansive path having at most 256 channels, and one or more layers of the contracting path implementing spatial random dropout, wherein the training comprises performing an iterative operation to find a set of parameters of the two-dimensional segmentation model that minimizes a loss function of the two-dimensional segmentation model, wherein each iteration involves finding the set of parameters of the two-dimensional segmentation model such that a value of the loss function using the set of parameters is less than a value of the loss function using another set of parameters in a previous iteration, and wherein the loss function is structured to measure a difference between an output predicted using the two-dimensional segmentation model and the annotations contained in the one or more tile images; and providing the trained two-dimensional segmentation model, wherein the trained two-dimensional segmentation model is configured to predict a segmentation map of non-target regions based on discriminative features encoded from one or more images, wherein the discriminative features are associated with lymphoid aggregates.

14. The computer-implemented method of claim 13, further comprising dividing the one or more tile images into patches, wherein the two-dimensional segmentation model is trained on the patches.

15. The computer-implemented method of claim 13 or 14, wherein the training further comprises adjusting a learning rate of the modified U-Net by decreasing the learning rate according to a predefined schedule.

16. The method of claim 15, wherein the predefined schedule is a step decay schedule that decreases the learning rate by a predetermined factor every predetermined number of epochs to optimize the loss function, and wherein the loss function is a binary cross-entropy loss function.

17. A non-transitory computer-readable storage medium comprising computer program instructions which, when executed by a computer processor, cause the computer processor to perform the actions or operations of the computer-implemented method of any of claims 13-16.

18. A computer system comprising a processor and a non-transitory memory, the memory comprising computer program instructions which, when executed by the processor, cause the processor to perform the actions or operations of the computer-implemented method of any of claims 13-16.

Images