Machine learning models for cell localization and classification trained using repel coding

Repel coding enhances digital pathology image analysis by training machine learning models to accurately detect and classify cells, addressing issues of cell morphology and crowding, thereby improving diagnostic efficiency.

JP2024161519AActive Publication Date: 2024-11-19VENTANA MEDICAL SYSTEMS INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024137511
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-08-13
Filing Date
2024-08-19
Publication Date
2024-11-19
Estimated Expiration
2041-08-12

AI Technical Summary

Technical Problem

Existing digital pathology image analysis techniques face challenges in accurately detecting and classifying cells due to variations in cell morphology, imaging protocols, and cell crowding, which complicates the segmentation and classification process, especially when using machine learning models.

Method used

The use of Repel coding to train machine learning models by encoding cell centers more prominently and reducing the emphasis on pixels between cells, enhancing the accuracy of cell detection and classification by emphasizing reversibility criteria, thereby improving the performance of convolutional neural networks in digital pathology image analysis.

Benefits of technology

Repel coding effectively trains machine learning models to generate precise segmentation masks for biomarkers, leading to improved accuracy in cell detection and classification, supporting faster and more accurate disease diagnosis and treatment decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161519000001_ABST
    Figure 2024161519000001_ABST
Patent Text Reader

Abstract

To effectively train machine learning models to detect, characterize, or classify some or all of digital pathology images.SOLUTION: An aspect of the present disclosure comprises: accessing an image of a biological sample, wherein the image displays cells containing a staining pattern of a biomarker; inputting the image into a machine learning model; encoding the image into a feature representation including extracted discriminative features using the machine learning model; combining cellular features and spatial information with the staining patterns of the biomarkers through concatenation with the extracted discriminative features from an upconvolution sequence and the feature representation by the machine learning model; and generating, by the machine learning model, two or more segmentation masks for the biomarkers in the image based on the combined cellular features and spatial information and the staining patterns of the biomarkers.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 065,268, filed August 13, 2020, which is incorporated by reference herein in its entirety for all purposes.

[0002] Field The present disclosure relates to digital pathology, and in particular to techniques for using repel coding to effectively train machine learning models to automatically detect, characterize and / or classify part or all of digital pathology images. [Background technology]

[0003] background Digital pathology involves scanning slides (e.g., histopathology or cytopathology glass slides) into interpretable digital images on a computer screen. The tissues and / or cells in the digital images can then be examined by digital pathology image analysis and / or interpreted by a pathologist for a variety of reasons, including diagnosing disease, evaluating response to treatment, and developing drugs to combat disease. To examine the tissues and / or cells (which are substantially transparent) in the digital images, pathology slides can be prepared using a variety of staining assays (e.g., immunohistochemistry) that selectively bind to tissue and / or cellular components. Immunofluorescence (IF) is a technique for analyzing assays that bind to fluorescent dyes for antigens. Multiple assays that respond to various wavelengths can be utilized on the same slide. These multiplexed IF slides allow for understanding the complexity and heterogeneity of the immune landscape of the tumor microenvironment and the potential impact on tumor response to immunotherapy. In some assays, the target antigen for the stain in the tissue may be referred to as a biomarker. Digital pathology image analysis can then be performed on the digital images of the stained tissues and / or cells to identify and quantitate staining for antigens (e.g., biomarkers indicative of various cells, such as tumor cells) in the biological tissue.

[0004] Machine learning techniques have shown great promise in digital pathology image analysis, such as cell detection, counting, localization, classification, and patient prognosis. Many computing systems with machine learning techniques, including convolutional neural networks (CNNs), have been proposed for image classification and digital pathology image analysis, such as cell detection and classification. For example, a CNN can have a series of convolutional layers as hidden layers, and this network structure allows for the extraction of representational features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques have also been implemented for image segmentation. Image segmentation is the process of dividing a digital image into multiple segments (sets of pixels, also known as image objects). The purpose of segmentation is to simplify and / or change the representation of an image to one that is more meaningful and easier to analyze. For example, image segmentation is typically used to find objects such as cells and boundaries (lines, curves, etc.) in an image. To perform image segmentation for large data (e.g., whole-slide pathology images), the image is first divided into many small patches. A computing system with machine learning techniques is trained to classify these patches, and all patches in the same class are combined into one segmented region. Then, the machine learning techniques are further implemented. The segmented regions can then be predicted or further classified (e.g., positive cells for a given biomarker, negative cells for a given biomarker, or cells with no staining expression) based on the phenotypic features associated with the segmented regions. Summary of the Invention

[0005] overview A method, system, and computer-readable storage medium are disclosed for using repel coding to effectively train machine learning models to automatically detect, characterize, and / or classify some or all of a digital pathology image.

[0006] The method, system, and computer readable storage medium may be embodied in a variety of ways.

[0007] In various embodiments, a computer-implemented method includes accessing an image of a biological sample, the image displaying cells including a staining pattern of a biomarker; inputting the image into a machine learning model, the machine learning model including a convolutional neural network including an encoder and a decoder, one or more layers of the encoder including a residual block with skip connections, parameters of the machine learning model learned from training images and a label mask for each biomarker in the training images, the label mask being generated using repel coding in combination with the labels of the respective biomarkers; and filtering the image by the machine learning model based on the extracted discriminative features. combining, by a machine learning model, the cellular features and spatial information and the staining pattern of the biomarkers via concatenation with the sequence of upconvolution and the extracted discriminatory features from the feature representation; and generating, by the machine learning model, two or more segmentation masks for the biomarkers in the image based on the combined cellular features and spatial information and the staining pattern of the biomarkers, wherein the two or more segmentation masks include a positive segmentation mask for cells expressing the biomarkers and a negative segmentation mask for cells not expressing the biomarkers.

[0008] In some embodiments, the method further includes overlaying two or more segmentation masks on the image to generate an instance segmentation image, and outputting the instance segmentation image.

[0009] In some embodiments, the method further includes determining, by a user, a diagnosis of a subject associated with the biological sample, where the diagnosis is determined based on (i) cells expressing the biomarker in the instance segmented image, and / or (ii) cells not expressing the biomarker in the instance segmented image.

[0010] In some embodiments, the method further includes administering a treatment to the subject by a user based on (i) cells in the instance segmented image that express the biomarker, (ii) cells in the instance segmented image that do not express the biomarker, and / or (iii) a diagnosis of the subject.

[0011] In some embodiments, the image displays cells comprising a staining pattern of the biomarker and another biomarker, and the machine learning model detects staining patterns for the other biomarker in the image. Two or more segmentation masks are generated, where the two or more segmentation masks for the other biomarker include a positive segmentation mask for cells expressing the other biomarker and a negative segmentation mask for cells not expressing the other biomarker.

[0012] In some embodiments, the method further includes overlaying two or more segmentation masks, each of the biomarker and another biomarker, on the image to generate an instance segmented image, and outputting the instance segmented image.

[0013] In some embodiments, generating the label masks includes (i) coding cells in each training image using repel coding, where the coding includes a cell center and a periphery represented by a response attenuation away from the cell center, and (ii) generating two or more label masks for each of the biomarkers in the images based on the coding and the labels of each of the biomarkers.

[0014] In some embodiments, combining the cellular features and spatial information with the staining patterns of the biomarkers comprises projecting the extracted discriminatory features into a pixel space and classifying each pixel space, where the classification comprises cell detection and classification of the cells based on the staining patterns of the biomarkers.

[0015] In various embodiments, a computer-implemented method includes accessing images of a biological sample, the images displaying cells comprising staining patterns of biomarkers, the cells being annotated with labels providing information comprising: (i) cell centers; and (ii) expression of the biomarkers; and generating two or more repel coding masks for each of the images, the generating comprising: (i) coding cells in the images using a repel coding algorithm, the output of the coding being an initial cell localization mask comprising a cell center and a periphery represented by a response attenuation away from the cell center; (ii) segmenting the initial cell localization mask using the labels to classify each instance of a cell based on the expression of the biomarkers; and (iii) dividing the initial cell localization mask into two or more repel coding masks based on the segmentation and classification of the instance of the cell. and generating a repel coding mask for each of the images; labeling each of the images with the two or more repel coding masks to generate a set of training images; and training a machine learning algorithm on the set of training images to generate a machine learning model, wherein the training includes performing iterative operations to learn a set of parameters for segmenting and classifying cells that maximizes or minimizes an objective function, each iteration including finding a set of parameters for the machine learning algorithm such that a value of the objective function using the set of parameters is greater or smaller than a value of the objective function using another set of parameters in a previous iteration, wherein the objective function is configured to measure a difference between a segmentation mask predicted using the machine learning algorithm and the two or more repel coding masks for the images.

[0016] In some embodiments, the two or more recoding masks include a positive mask for cells expressing the biomarker and a negative mask for cells not expressing the biomarker.

[0017] In some embodiments, the trained machine learning model includes a convolutional neural network including an encoder and a decoder, where one or more layers of the encoder include residual blocks with skip connections.

[0018] In some embodiments, the image is an image patch of a predetermined size.

[0019] In some embodiments, the method further includes dividing the two or more repel coding masks into mask patches of a predetermined size, and labeling each of the image patches with the mask patch to generate a set of training images.

[0020] In some embodiments, a method is provided that includes determining, by a user, a diagnosis of a subject based on results generated by a machine learning model trained using some or all of one or more techniques disclosed herein, and potentially selecting, recommending and / or administering a particular treatment to the subject based on the diagnosis.

[0021] In some embodiments, a method is provided that includes determining, by a user, a treatment to select, recommend and / or administer to a subject based on results generated by a machine learning model trained using some or all of one or more techniques disclosed herein.

[0022] In some embodiments, a method is provided that includes determining, by a user, whether a subject is eligible to participate in a clinical trial or assign the subject to a particular cohort in a clinical trial based on results generated by a machine learning model trained using some or all of one or more of the techniques disclosed herein.

[0023] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. [Brief description of the drawings]

[0024] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of various embodiments will become more apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings, in which: FIG. [Figure 1] 1 illustrates an example of a histological staining process according to various embodiments of the present disclosure. [Diagram 2] FIG. 1 shows a block diagram illustrating a computing environment for processing digital pathology images using machine learning models, in accordance with various embodiments of the present disclosure. [Diagram 3] 1 shows a flowchart illustrating a process for implementing a repel coding scheme for generating a repel coding mask for use in training a machine learning algorithm, according to various embodiments of the present disclosure. [Figure 4] 1 illustrates an annotated image with raw dot labels according to various embodiments of the present disclosure. [Diagram 5] 1 illustrates an image being coded by a repel coding algorithm to provide an initial cell localization mask, according to various embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates a modified U-Net model used for biomarker segmentation according to various embodiments of the present disclosure. [Figure 7]FIG. 1 shows a flowchart illustrating a process for training a machine learning algorithm to localize and classify cells based on biomarker expression patterns, according to various embodiments of the present disclosure. [Figure 8-1] 1 shows a flowchart illustrating a process of using a machine learning model to localize and classify cells based on biomarker expression patterns, according to various embodiments of the present disclosure. [Figure 8-2] 1 shows a flowchart illustrating a process of using a machine learning model to localize and classify cells based on biomarker expression patterns, according to various embodiments of the present disclosure. [Figure 9] 1 shows a flowchart illustrating a process for applying a segmentation mask to support or improve image analysis, according to various embodiments of the present disclosure. [Figure 10] 1 shows different patterns of PD1 according to various embodiments of the present disclosure. [Figure 11] 1 shows visual results from both the training and validation sets in Example 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Detailed Description Although specific embodiments are described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The devices, methods, and systems described herein can be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications of the form of the exemplary methods and systems described herein can be made without departing from the scope of protection.

[0026] I. Overview The present disclosure describes techniques for detecting, characterizing, and / or classifying some or all of a digital pathology image. More specifically, some embodiments of the present disclosure provide machine learning techniques that use repel coding to effectively train machine learning models to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cellular analysis.

[0027] The ability to characterize biomarkers in cell and tissue samples and measure the heterogeneity of the presence and levels of such biomarkers within and between tissues can provide valuable information in biomedicine to understand and characterize various disease states and / or to appropriately select available targeted therapeutics for a patient's disease state. Furthermore, the ability to identify and characterize regions within a tissue that have differential distribution of key biomarkers can provide important information to inform the development of targeted and combination therapies. The development and selection of appropriate combination therapies may also be an important factor in the prevention of recurrence.

[0028] Automated detection and classification of cells with different biomarker expression levels from digital images has the potential for faster and more accurate extraction of information for disease diagnosis and treatment. Since most cells and tissues are virtually transparent in digital images, modern laboratory techniques have been developed to aid in the visualization of cells and tissues. Immunofluorescence (IF) is a staining technique that aids in the visualization of components in tissues or cells. IF uses a combination of antibodies and fluorophores to mark specific proteins and organelles in a cell or tissue sample. Thus, IF techniques are widely used in the study of various types of cells, tracking and localizing proteins in cell or tissue samples, and identifying biological structures in cell or tissue samples. It is possible to evaluate IF stained cells in tissue sections under a microscope at high magnification and / or to automatically analyze digital images of biological samples using digital pathology algorithms. Often in whole slide analysis, the evaluation of stained biological specimens involves detecting cells or cellular structures in the stained biological sample, localizing the centers of cells or cellular structures, and identifying biomarkers and biomarker patterns. It is necessary to recognize the above.

[0029] After cells are made visible for analysis, cell detection and classification techniques are desired. Cell detection and classification techniques have evolved from using hand-crafted features to machine learning-based detection techniques. Most machine learning-based cell detection and classification techniques implement a classical feature-based machine learning pipeline, where a cell classifier or detector is trained in pixel space and the location and features of target cells are labeled. However, accurate cell detection and classification using classical feature-based machine learning is challenging for several reasons. First, the morphology of different cells differs dramatically, and parametric models must be redesigned and trained to fit different cell targets. Second, even for the same type of cells, different preparation and imaging protocols result in different cell appearances. More importantly, certain types of cells (e.g., immune cells) tend to cluster in groups, and the crowding or density of cells makes it difficult to define the boundaries between cells and associated features. The problem of cell crowding or density increases the difficulty of not only cell detection but also classification.

[0030] To address these limitations and issues, various embodiments disclosed herein relate to machine learning techniques for using repel coding to efficiently train machine learning models to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cell analysis. Repel coding of raw data labels for cell detection is based on proximity coding, which is typically used for cell counting. Proximity coding produces local maxima at cell centers. However, in cell detection, it is a challenge to distinguish adjacent cells from each other, especially cells that tend to cluster in groups. Proximity coding focuses on entropy, and the valleys in response between adjacent cells are usually not strong enough for accurate cell detection. To overcome this challenge, repel coding is configured to increase the valleys in response between adjacent cells and more accurately align the local maxima at cell centers compared to typical proximity coding. In essence, repel coding codes the centers of cells in a way that increases reversibility (obtaining a better balance between entropy and reversibility), which allows cells to be more legibly distinguished and labeled.

[0031] Further, the repel coding is configured to intelligently apply cell center coding based on the image label for each of one or more characteristics associated with the cells to generate a repel coding mask. The one or more characteristics can be a biomarker staining pattern (e.g., bright field staining or immunofluorescence (IF)), a type of cell (e.g., tumor cells, immune cells, tissue cells, etc.), the presence or absence of various organelles such as nuclei or mitochondria, etc. For example, given a tissue stained for a single biomarker, the repel coding can be configured to (i) code the centers of cells identified by the annotation as positive for the biomarker and output a repel coding mask for the positive cells, and (ii) code the centers of cells identified by the annotation as negative for the biomarker and output a repel coding mask for the negative cells. The repel coding mask is then used as a label for the original image to effectively train a machine learning model to automatically detect, characterize, and / or classify cells in the biological sample image to support or improve cell analysis. When training a machine learning model on images and / or masks coded by proximity coding, the training tends to merge and place nearby cell centers together. On the other hand, when training a machine learning model on images and / or masks coded by repel coding, the training tends to emphasize more the pixels in the center of the cells (repulsive code labels have stronger pixels in the center of the cells) and less emphasize the pixels between the cell centers (repulsive code labels have weaker pixels on the periphery of the cells). As a result, repel coding emphasizes the reversibility criterion and more effectively trains the machine learning model to be able to detect cells.

[0032] Furthermore, the repel coding mask, which is the actual ground truth value of the corresponding image, indicates whether the image contains cells, and if so, where and what characteristics each cell may have, such as biomarker patterns. The machine learning model encodes the image into a feature representation that includes the extracted discriminatory features of the cells, and then combines the cell features and spatial information with the staining patterns of the biomarkers through a sequence of upconvolution and concatenation with the extracted discriminatory features from the feature representation. The machine learning model generates two or more probabilities or segmentation masks for each biomarker in the image based on the combination of the cell features and spatial information, and the staining patterns of the biomarkers. The cell centers of the two or more probabilities or segmentation masks are extracted by local maxima detection based on repel coding. The cost function is used to measure the difference or distance between the output probabilities or segmentation masks and the ground truth values ​​of the corresponding image (i.e., the repel coding mask). The objective of training the machine learning model is to find model parameters, weights, or structures that minimize or maximize the cost function.

[0033] Once trained, the machine learning model can be used in a computer-implemented method to automatically generate segmentation masks for biomarkers in an image. In some cases, the computer-implemented method is performed as part of pre-processing before running an image analysis algorithm to segment and classify target regions (e.g., tumor cells) in an image. In other examples, the computer-implemented method is performed as part of post-processing after running an image analysis algorithm to segment and classify target regions (e.g., tumor cells) in an image. However, as will be appreciated by those skilled in the art, the concepts described herein are not limited to pre-processing or post-processing procedures, but may be integrated into the overall image analysis process according to various embodiments.

[0034] The computer-implemented method may include the use of machine learning models including a convolutional neural network ("CNN") architecture or model that utilizes a two-dimensional segmentation model (modified U-Net) to automatically detect biological structures such as cells or cell nuclei, and biomarkers such as PD1, before running standard image analysis algorithms to learn and recognize target regions. However, the present disclosure is not limited to segmenting only cells, cell nuclei, or biomarkers, and the techniques described herein may also be applied to distinguish other cellular organelles, such as ribosomes, mitochondria, and the like. The convolutional neural network architecture may be trained using pre-labeled images of different biomarker regions or biomarker positive and negative regions. As a result, the trained convolutional neural network architecture or model may be used to automatically encode the image into a feature representation, which may then be masked from the whole slide analysis before, during, or after inputting the image into the image analysis algorithm. The feature representation and extracted features may be further combined with spatial information of the biomarkers, and a biomarker mask may be generated accordingly. The image analysis algorithm may further perform a classification task and output a classification label for the detected cell or cell structure. Advantageously, the proposed architecture and techniques can improve the accuracy of cell detection and biomarker classification by image analysis algorithms.

[0035] II. Definition As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a part of the something.

[0036] As used herein, "substantially" and "approximately The terms "approximately," "about," and "approximately" are defined as being largely, but not necessarily completely, specified (and including that which is completely specified), as would be understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" can be replaced with "within [a percentage]" of that which is specified, where the percentage includes 0.1, 1, 5, and 10%.

[0037] As used herein, the terms "sample", "biological sample", "tissue" or "tissue sample" refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as veterinary animals such as humans, cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, primates, etc.), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or needle biopsies of tissues), cell samples (such as cytological smears such as Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components, such as by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucous membranes, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as cheek swabs), or any material containing biomolecules derived from an initial biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or portion thereof obtained from a subject (such as a homogenized or liquefied sample).

[0038] As used herein, the terms "biological material," "biological structure," or "cellular structure" refer to naturally occurring materials or structures that contain all or part of a biological structure (e.g., a cell nucleus, cell membrane, cytoplasm, chromosomes, DNA, cells, cell masses, etc.).

[0039] As used herein, "digital pathology image" refers to a digital image of a stained sample.

[0040] As used herein, the term "cell detection" refers to the detection of a cell or a cell structure (eg, a cell nucleus, a cell membrane, a cytoplasm, a chromosome, DNA, a cell, a cell mass, etc.).

[0041] As used herein, the term "target region" refers to a region of an image that contains image data that is intended to be evaluated in an image analysis process. Target regions include any region, such as a tissue region of an image, that is intended to be analyzed in an image analysis process (e.g., tumor cells or stain expression).

[0042] As used herein, the term "tile" or "tile image" refers to a single image that corresponds to a portion of an entire image or an entire slide. In some embodiments, a "tile" or "tile image" refers to an area of ​​an entire slide scan or a region of interest having (x,y) pixel dimensions (e.g., 1000 pixels by 1000 pixels). For example, consider an entire image divided into M columns of tiles and N rows of tiles, where each tile in the MxN mosaic contains a portion of the entire image, i.e., tiles at locations MI, NI contain a first portion of the image, tiles at locations M3, N4 contain a second portion of the image, and the first and second portions are different. In some embodiments, the tiles can each have the same dimensions (pixel size by pixel size).

[0043] As used herein, a "patch," "image patch," or "mask patch" The term refers to a container of pixels that corresponds to a portion of an entire image, entire slide, or entire mask. In some embodiments, a "patch", "image patch", or "mask patch" refers to a region of an image or mask, or a region of interest having (x,y) pixel dimensions (e.g., 256 pixels by 256 pixels). For example, a 1000 pixel by 1000 pixel image divided into 100 pixel by 100 pixel patches will contain 10 patches (each patch containing 1000 pixels). In other embodiments, the patches have (x,y) pixel dimensions and overlap with each "patch", "image patch", or "mask patch" sharing one or more pixels with another "patch", "image patch", or "mask patch".

[0044] III. Generation of digital pathology images Histological staining is widely used to highlight features of interest and enhance the contrast of sectioned tissues or cells of biological samples. For example, staining can be used to mark specific types of cells and / or flag specific types of nucleic acids and / or proteins to aid in microscopy. The stained sample can then be evaluated to determine or estimate the amount of the feature of interest in the sample (which can include, for example, counts, density or expression levels) and / or one or more characteristics of the feature of interest (e.g., the location of the features of interest relative to each other or relative to other features, shape characteristics, etc.). The process of histological staining can include several stages, such as fixing, processing, embedding, sectioning, staining and imaging.

[0045] In some embodiments, immunohistochemical staining of tissue sections (e.g., bright field staining or IF) is a type of histological staining used to identify the presence of a particular protein in a biological sample. For example, the expression level of a particular protein (e.g., an antigen) is determined by (a) performing an immunohistochemical analysis of a sample with a particular antibody type, and (b) determining the presence and / or expression level of the protein in the sample. In some embodiments, the immunohistochemical staining intensity is determined relative to a reference determined from a reference sample (e.g., a control cell line stained sample, a tissue sample from a non-cancerous subject, a reference sample known to have a predetermined level of protein expression).

[0046] FIG. 1 shows an example of a histological staining process 100. Stage 110 of the histological staining process 100 includes sample fixation, which can be used to preserve the sample and slow the degradation of the sample. In histology, fixation generally refers to an irreversible process that uses chemicals to retain chemical composition, preserve the native sample structure, and maintain cellular structures from degradation. Fixation can also harden cells or tissues for sectioning. Fixatives can enhance sample and cell preservation using crosslinking proteins. Fixatives can bind and crosslink some proteins and denature others by dehydration, which can harden the tissue and inactivate enzymes that might otherwise degrade the sample. Fixatives can also kill bacteria.

[0047] Fixatives can be administered, for example, by perfusion and immersion of the prepared sample. Various fixatives can be used, including methanol, buine fixatives and / or formaldehyde fixatives, such as neutral buffered formalin (NBF) or paraffin-formalin (paraformaldehyde-PFA). If the sample is a liquid sample (e.g., a blood sample), the sample may be smeared onto a slide and dried before fixation.

[0048] Although the fixation process can help preserve the structure of the sample and cells for the purposes of histological examination, fixation can result in the concealment of tissue antigens, thereby reducing antigen detection. Thus, fixation is generally considered a limiting factor in immunohistochemistry, since formalin can crosslink antigens and mask epitopes. In some cases, fixed samples are treated with citraconic anhydride (a reversible protein crosslinker) to improve detection. Additional processes are performed to reverse the effects of cross-linking, including applying heat and / or pressure.

[0049] Stage 120 of the histological staining process 100 includes sample processing and embedding. Sample processing can include infiltrating a fixed sample (e.g., a fixed tissue sample) with a suitable histological wax, such as paraffin wax. The histological wax may be insoluble in water or alcohol, but may be soluble in a paraffin solvent, such as xylene. Thus, the water in the tissue may need to be replaced by xylene. To do so, the sample can first be dehydrated by gradually replacing the water in the sample with alcohol, which can be achieved by passing the tissue through increasing concentrations of ethyl alcohol (e.g., from 0 to about 100%). After the water has been replaced by the alcohol, the alcohol may be replaced by xylene, which is miscible with the alcohol. Embedding can include embedding the sample in warm paraffin wax. Because paraffin wax can be soluble in xylene, the molten wax can fill the spaces filled by xylene and previously filled by water. The wax-filled sample can be cooled to form a hardened block that can be clamped to a microtome for cross-sectioning. In some cases, deviations from the above exemplary procedure result in penetration of the paraffin wax resulting in inhibition of penetration of antibodies, chemicals, or other fixatives.

[0050] Stage 130 of the histological staining process 100 involves sectioning the sample. Sectioning is the process of cutting thin slices of a sample from an embedded block with the purpose of mounting the sample (e.g., an embedded and fixed tissue sample) on a microscope slide for examination. Sectioning can be performed using a microtome. In some cases, the tissue can be quickly frozen in dry ice or isopentane and then cut by a cold knife in a refrigerated cabinet (e.g., a cryostat). Other types of coolants, such as liquid nitrogen, can be used to freeze the tissue. Sections for use with brightfield and fluorescent microscopes are generally on the order of 4-10 μm thick. In some cases, the sections can be embedded in epoxy or acrylic resin, which can allow thinner sections (e.g., <2 μm) to be cut. The sections can then be mounted on one or more glass slides. A cover slip can be placed on top to protect the sample sections.

[0051] Stage 140 of the histological staining process 100 involves staining (of tissue samples or sections of fixed liquid samples). The purpose of staining is to identify different sample components by color reaction. Most cells are colorless and transparent. Therefore, it may be necessary to stain histological sections to visualize the cells. The staining process generally involves adding a dye or staining agent to the sample to confirm or quantify the presence of a particular compound, structure, molecule, or feature (e.g., subcellular feature). For example, staining can help to identify or highlight a particular biomarker from a tissue section. In other examples, staining can be used to identify or highlight biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or organelles within individual cells.

[0052] Many staining solutions are water-based. Therefore, to stain tissue sections, the wax may need to be dissolved and replaced with water (rehydrated) before the staining solution is applied to the section. For example, the sections can be passed sequentially through xylene, decreasing concentrations of ethyl alcohol (from about 100% to 0%), and water. Once stained, the sections may be dehydrated again and placed in xylene. The sections can then be mounted on microscope slides in mounting medium dissolved in xylene. To protect the sample section, a coverslip can be placed on top. The vapor of xylene around the edges of the coverslip will dry out. This allows the mounting medium to dry and the coverslip to be firmly bonded to the slide.

[0053] Various types of staining protocols can be used to perform staining. For example, an exemplary immunohistochemical staining protocol includes using a hydrophobic barrier line around the sample (e.g., tissue section) to prevent leakage of reagents from the slide during incubation, treating the tissue section with reagents to block endogenous sources of nonspecific staining (e.g., enzymes, free aldehyde groups, immunoglobulins, other unrelated molecules that may mimic specific staining), incubating the sample with a permeabilization buffer to facilitate penetration of antibodies and other staining reagents into the tissue, incubating the tissue section with a primary antibody at a specific temperature (e.g., room temperature, 6-8 °C) for a period of time (e.g., 1-24 hours), rinsing the sample using a wash buffer, incubating the sample (tissue section) with a secondary antibody at another specific temperature (e.g., room temperature) for another period of time, rinsing the sample again using a water buffer, incubating the rinsed sample with a chromogen (e.g., DAB), and washing off the chromogen to stop the reaction. In some cases, a counterstain is then used to distinguish the entire "landscape" of the sample and serve as the primary color reference used to detect tissue targets. Counterstains can include, for example, hematoxylin (blue to purple stain), methylene blue (blue stain), toluidine blue (nuclear deep blue, polysaccharide pink to red stain), nuclear fast red (also called Kernechtrot dye, red stain), methyl green (green stain), non-nucleogenic stains such as eosin (pink stain), etc. Those skilled in the art will recognize that other immunohistochemical staining techniques can be implemented to perform the staining.

[0054] In another example, an H&E staining protocol can be performed for staining tissue sections. The H&E staining protocol involves applying hematoxylin stain mixed with a metal salt or mordant to the sample. The sample can then be rinsed with a weak acid solution to remove excess stain (differentiation), followed by blueing in weak alkaline water. After application of hematoxylin, the sample can be counterstained with eosin. It will be understood that other H&E staining techniques can be implemented.

[0055] In some embodiments, different types of stains can be used to perform staining depending on which feature of interest is. For example, DAB can be used on various tissue sections for IHC staining, where DAB results in a brown color that displays the feature of interest in the stained image. In another example, alkaline phosphatase (AP) can be used on skin tissue sections for IHC staining, since the DAB color can be masked by melanin pigment. With regard to the primary staining technique, applicable stains can include, for example, basophilic and acidophilic stains, hematin and hematoxylin, silver nitrate, trichrome stains, etc. Acidic dyes can react with cationic or basic components in tissues or cells, such as proteins and other components in the cytoplasm. Basic dyes can react with anionic or acidic components in tissues or cells, such as nucleic acids. As mentioned above, one example of a staining system is H&E. Eosin can be a negatively charged pink acidic dye, and hematoxylin can be a purple or blue basic dye that contains hematein and aluminum ions. Other examples of stains include Periodic Acid-Schiff (PAS) stains, Masson's Trichrome, Alcian Blue, Van Gieson, Reticulin stains, etc. In some embodiments, different types of stains can be used in combination.

[0056] Stage 150 of the histological staining process 100 involves medical imaging. A microscope (e.g., an electron microscope or an optical microscope) can be used to magnify the stained sample. For example, an optical microscope has a resolution of less than 1 μm, such as on the order of a few hundred nanometers. A microscope can be used to image the biological specimen. To observe finer details in the nanometer or sub-nanometer range, an electron microscope can be used. An imaging device (combined with the microscope or separate from the microscope) images the magnified biological specimen to obtain image data, such as a multichannel image (e.g., multichannel fluorescence) having several (e.g., between 10 and 16) channels. The imaging device includes, but is not limited to, a camera (e.g., analog camera, digital camera, etc.), an optical system (e.g., one or more lenses, a sensor focus lens group, a microscope objective lens, etc.), an imaging sensor (e.g., a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, etc.), photographic film, etc. In digital embodiments, the imaging device can include multiple lenses that cooperate to demonstrate on-the-fly focusing. The image sensor, e.g., a CCD sensor, can capture a digital image of the specimen. In some embodiments, the imaging device is a bright-field imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. The imaging device can capture an image utilizing invisible electromagnetic radiation (e.g., UV light) or other imaging techniques. For example, the imaging device may include a microscope and a camera configured to capture an image magnified by the microscope. The image data received by the analysis system may be identical to and / or derived from the raw image data captured by the imaging device.

[0057] At step 160, images of the stained sections are stored. The images can be stored locally, remotely, and / or on a cloud server. Each image may be stored in association with a subject identifier and a date (e.g., the date the sample was collected and / or the date the image was taken). The images may also be transmitted to another system (e.g., a system associated with a pathologist, or an automated or semi-automated image analysis system).

[0058] It will be understood that modifications to process 100 are contemplated. For example, if the sample is a liquid sample, step 120 (processing and embedding) and / or step 130 (sectioning) may be omitted from the process.

[0059] IV. Exemplary Process Flow for Digital Pathology Image Conversion 2 shows a block diagram illustrating a computing environment 200 for processing digital pathology images using a machine learning model. As described further herein, processing the digital pathology image may include training a machine learning algorithm using the digital pathology image or converting some or all of the digital pathology image into one or more results using a trained (or partially trained) version of a machine learning algorithm (i.e., a machine learning model).

[0060] As shown in FIG. 2, the computing environment 200 includes several stages: an image storage stage 205 , a pre-processing stage 210 , a labeling stage 215 , a training stage 220 , and a result generation stage 225 .

[0061] The image storage stage 205 includes one or more image data stores 230 that are accessed (e.g., by the pre-processing stage 210) to provide a set of digital images 235 of pre-selected regions from a biological specimen slide (e.g., a tissue slide) or from the entire biological specimen slide. Each digital image 235 stored in each image data store 230 and accessed in the image storage stage 210 may include a digital pathology image generated according to some or all of the process 100 shown in FIG. 1. In some embodiments, each digital image 235 includes image data from one or more scanned slides. Each of the digital images 235 may include an underlying image corresponding to a single specimen and / or image. The image data can correspond to image data from one day on which it was collected.

[0062] The image data may include the image, as well as any information regarding the color or color wavelength channels, as well as details regarding the imaging platform on which the image was generated. For example, the tissue section may need to be stained by application of a staining assay that includes one or more different biomarkers associated with a chromogenic stain for brightfield imaging or a fluorophore for fluorescent imaging. The staining assay may use a chromogenic stain for brightfield imaging, an organic fluorophore, a quantum dot, or an organic fluorophore together with a quantum dot for fluorescent imaging, or any other combination of stain, biomarker, and observation or imaging device. Examples of biomarkers include biomarkers such as estrogen receptor (ER), human epidermal growth factor receptor 2 (HER2), human Ki-67 protein, progesterone receptor (PR), programmed cell death protein 1 ("PD1"), where the tissue section is detectably labeled by a respective binding agent (e.g., an antibody) of ER, HER2, Ki-67, PR, PD1, etc. In some embodiments, digital image and data analysis operations such as classification, scoring, Cox modeling and risk stratification depend on the type of biomarkers used and the field of view (FOV) selection and annotation. Furthermore, a typical tissue section is processed in an automated staining / assay platform that applies a staining assay to the tissue section to obtain a stained sample. There are various commercial products on the market that are suitable for use as a staining / assay platform, one example being the VENTANA SYMPHONY product from the assignee Ventana Medical Systems, Inc. The stained tissue section can be fed into an imaging system, for example, on a microscope or a whole slide scanner with a microscope and / or imaging components, one example being the VENTANA iScan Coreo product from the assignee Ventana Medical Systems, Inc. Multiple tissue slides can be scanned in a comparable multiplexed slide scanner system.Additional information provided by the imaging system can include any information regarding the staining platform, including the concentration of chemicals used for staining, the reaction time of the chemicals applied to the tissue in the staining, and / or the pre-analysis conditions of the tissue, such as the age of the tissue, fixation method, duration, how the sections were embedded, cut, etc.

[0063] In the pre-processing stage 210, each, a plurality, or all of the set of digital images 235 are pre-processed using one or more techniques to generate corresponding pre-processed images 240. Pre-processing may include cropping the image. In some cases, pre-processing may further include standardization or rescaling (e.g., normalization) to bring all features to the same scale (e.g., the same size scale or the same color scale or saturation scale). In a particular example, the image is resized with a minimum size (width or height) of a given number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a given number of pixels (e.g., 3000 pixels), optionally maintaining the original aspect ratio. Pre-processing may further include removing noise. For example, the image may be smoothed to remove undesired noise, such as by applying a Gaussian function or a Gaussian blur.

[0064] The preprocessed images 240 can include one or more of a training image, a validation input image, and an unlabeled image. It should be understood that the preprocessed images 240 corresponding to the training group, the validation group, and the unlabeled group do not have to be accessed at the same time. For example, an initial set of training and validation preprocessed images 240 may be initially accessed and used to train the machine learning algorithm 255, and the unlabeled input image elements can be subsequently accessed or received (e.g., at one or more subsequent times) and used by the trained machine learning model 260 to provide a desired output (e.g., cell classification).

[0065] In some cases, the machine learning algorithm 255 is trained using supervised training, and some or all of the preprocessed image 240 is partially or fully labeled in the labeling stage 215, either manually, semi-automatically, or automatically, with labels 245 that identify the "correct" (i.e., "ground truth") interpretation of various biological materials and structures in the preprocessed image 240. For example, the labels 245 may identify (for example) features of interest, classifications of cells, binary indications as to whether a given cell is a particular type of cell, binary indications as to whether the preprocessed image 240 (or a particular region having the preprocessed image 240) contains a particular type of indication (e.g., necrosis or artifacts), categorical features of slide-level or region-specific indications (e.g., identifying a particular type of cell), counts (e.g., identifying the amount of a particular type of cell in a region, the amount of artifacts displayed, or the amount of necrotic area), the presence or absence of one or more biomarkers, etc. In some cases, the labels 245 include location. For example, the labels 245 may identify the point location of the nucleus of a particular type of cell or the point location of a particular type of cell (e.g., a raw dot label). As another example, the labels 245 may include the edge or border of a displayed tumor, blood vessel, necrotic area, etc. As another example, the labels 245 may include one or more biomarkers identified based on a biomarker pattern observed using one or more stains. For example, a tissue slide stained for a biomarker, e.g., programmed cell death protein 1 ("PD1"), may be observed and / or processed to label cells as either positive or negative cells, taking into account the expression level and pattern of PD1 in the tissue. Depending on the feature of interest, a given labeled preprocessed image 240 may be associated with a single label 245 or multiple labels 245. In the latter case, each label 245 may be associated with (for example) an indication as to which location or portion within the preprocessed image 245 the label corresponds to.

[0066] The labels 245 assigned in the labeling stage 215 can be identified based on input from a human user (e.g., a pathologist or image scientist) and / or an algorithm (e.g., an annotation tool) configured to define the labels 245. In some cases, the labeling stage 215 can include transmitting and / or presenting some or all of the one or more preprocessed images 240 to a computing device operated by the user. In some cases, the labeling stage 215 can include utilizing an interface (e.g., using an API) presented by the labeling controller 250 at the computing device operated by the user, the interface including an input component for accepting input identifying the labels 245 for features of interest. For example, a user interface may be provided by the labeling controller 250 that allows selection of an image or a region of an image (e.g., FOV) for labeling. A user operating a terminal can select an image or FOV using the user interface. Several image or FOV selection mechanisms can be provided, such as specifying a known or irregular shape or defining an anatomical region of interest (e.g., tumor region). In one example, the image or FOV is an entire tumor region selected on an IHC slide stained with a combination of hematoxylin and eosin (H&E) stains. The selection of the image or FOV can be performed by a user or by an automated image analysis algorithm, such as tumor region segmentation on an H&E tissue slide. For example, the user can select that the image or FOV can be automatically designated as the entire slide or entire tumor, or the entire slide or entire tumor region using a segmentation algorithm. The user operating the terminal can then select one or more labels 245 to be applied to the selected image or FOV, such as point locations on the cells, positive markers for biomarkers expressed by the cells, negative biomarkers for biomarkers not expressed by the cells, a boundary around the cells, etc.

[0067] In some cases, the interface can identify which and / or how many particular labels 245 are requested, which can be communicated to the user via (for example) text instructions and / or visualizations. For example, a particular color, size, and / or symbol can represent that a label 245 is requested for a particular representation (e.g., a particular cell or region or staining pattern) in the image relative to other representations. If labels 245 corresponding to multiple representations are requested, the interface can identify each of the representations simultaneously, or can identify each representation sequentially (such that providing a label for one identified representation triggers the identification of the next representation for labeling). In some cases, each image is presented until the user identifies a particular number of labels 245 (e.g., of a particular type). For example, a given whole slide image or a given patch of a whole slide image can be presented until the user identifies the presence or absence of three different biomarkers, at which point the interface can present a different whole slide image or a different patch image (e.g., until a threshold number of images or patches are labeled). Thus, in some cases, the interface is configured to request and / or accept labels 245 of an incomplete subset of features of interest, allowing the user to determine which of the potentially many representations will be labeled.

[0068] In some cases, the labeling stage 215 includes a labeling controller 250 that implements an annotation algorithm to semi-automatically or automatically label various features of an image or a region of interest within an image. For example, intensities across an image (e.g., a pre-processed image) can be normalized or regularized, intensities can be thresholded or filtered, and / or an algorithm (e.g., configured to detect objects, lines, and / or shapes such as Equation (1)) can be applied. Each boundary, point location, or location coding can then be identified as a feature of interest. In some cases, a metric is associated with each feature of interest (e.g., a metric indicating a confidence in localizing the feature of interest), and the interest level can be scaled based on the metric. According to aspects of the present disclosure, the annotation algorithm can be configured to use a coding scheme to localize the feature of interest (e.g., localize where exactly an object is in the image). Localizing the feature of interest can include predicting the point location and / or boundary of the feature of interest. For example, localization may include identifying or predicting a point location corresponding to each displayed cell and / or identifying or predicting a closed shape corresponding to each displayed cell. In a particular example, the localization technique includes coding the raw dot labels into a mask (e.g., labels 245 of locations of objects such as cells) using a repel coding scheme. The repel coding scheme is defined as follows:

number

number

number

number

number

number

[0069] The label masks can be generated using a repel coding scheme in combination with annotations for each of the biomarkers. For example, the localization technique can include coding the raw dot labels into an initial cell localization mask (e.g., labels for cell locations 245) using repel coding defined in formula (1), segmenting the initial cell localization mask using label annotations of one or more biomarkers based on the biomarker staining pattern, and outputting two or more repel coding masks for each biomarker (e.g., one mask for positive cells and one mask for negative cells). The segmentation is an instance segmentation, and along with pixel-level classification for localization, the algorithm further classifies each instance of a class separately (e.g., positive cells for PD1, negative cells for PD1, positive cells for HER2, negative cells for HER2, etc.). For example, given tissue stained for a single biomarker, the repel coding scheme can be configured to (i) code the centers of cells identified by annotation as positive for the biomarker and output a repel coding mask of positive cells, and (ii) code the centers of cells identified by annotation as negative for the biomarker and output a repel coding mask of negative cells. As another example, given tissue stained for two biomarkers, the repel coding can be configured to (i) code the centers of cells identified by annotation as positive for the first biomarker and output a repel coding mask of positive cells, (ii) code the centers of cells identified by annotation as negative for the first biomarker and output a repel coding mask of negative cells, (iii) code the centers of cells identified by annotation as positive for the second biomarker and output a repel coding mask of positive cells, and (iv) code the centers of cells identified by annotation as negative for the second biomarker and output a repel coding mask of negative cells.As another example, given tissue stained for a single biomarker, repel coding can be configured to: (i) code centers of cells identified by annotation as positive for the biomarker and output a repel coding mask of positive cells (e.g., tumor cells), (ii) code centers of cells identified by annotation as negative for the biomarker and output a repel coding mask of negative cells (e.g., immune cells), and (iii) code centers of cells identified by annotation as tissue cells and output a repel coding mask of tissue cells.

[0070] In some cases, the labeling controller 250 may also include a user input or annotation. The labeling controller 250 annotates the image or FOV on the first slide according to a algorithm and maps the annotations to the rest of the slide. Depending on the defined FOV, several methods for annotation and registration are possible. For example, the entire tumor area annotated on the H&E slide from among multiple consecutive slides may be selected automatically or by the user on an interface such as VIRTUOSO / VERSO™. Since the other tissue slides correspond to consecutive sections from the same tissue block, the labeling controller 250 performs a marker-to-marker registration operation to map and transfer the entire tumor annotation from the H&E slide to each of the remaining IHC slides in the series. Exemplary methods for marker-to-marker registration are described in more detail in commonly assigned, co-pending International Publication No. WO 2014140070, filed March 12, 2014, entitled "Whole slide image registration and cross-image annotation devices, systems and methods," which is incorporated herein by reference in its entirety for all purposes. In some embodiments, any other method for image registration and generation of whole tumor annotations can be used. For example, a qualified reader, such as a pathologist, can annotate all tumor regions on any other IHC slides and execute the labeling controller 250 to map all tumor annotations on other digitized slides. For example, a pathologist (or an automated detection algorithm) can annotate all tumor regions on an H&E slide to trigger analysis of all adjacent serially sectioned IHC slides to determine a whole-slide tumor score for the annotated regions on all slides.

[0071] In the training phase 220, the labels 245 and the corresponding preprocessed images 240 can be used by the training controller 265 to train the machine learning algorithm 255. In some cases, the machine learning algorithm 255 includes a CNN, a modified CNN with the encoding layer replaced by a residual neural network ("Resnet"), or a modified CNN with the encoding and decoding layers replaced by Resnet. In other examples, the machine learning algorithm 255 can be any suitable machine learning algorithm configured to localize, classify, and / or analyze the preprocessed images 240, such as a two-dimensional CNN ("2DCNN"), a Mask R-CNN, a feature pyramid network (FPN), a dynamic time warping ("DTW") technique, a hidden Markov model ("HMM"), or the like, or a combination of one or more of such techniques, such as a CNN-HMM or an MCNN (multi-scale convolutional neural network). The computing environment 200 can use the same type of machine learning algorithm or different types of machine learning algorithms trained to detect and classify different cells. For example, the computing environment 200 may include a first machine learning algorithm (e.g., U-Net) for detecting and classifying PD1. The computing environment 200 may also include a second machine learning algorithm (e.g., 2DCNN) for detecting and classifying cluster of differentiation 68 ("CD68"). The computing environment 200 may also include a third machine learning algorithm (e.g., U-Net) for combined detection and classification of PD1 and CD68. The computing environment 200 may also include a fourth machine learning algorithm (e.g., HMM) for diagnosis of disease for treatment or prognosis of a subject, such as a patient. In other examples according to the present disclosure, still other types of machine learning algorithms may be implemented.

[0072] In some embodiments, the training phase includes a parameter data store and a trainer controller that are collectively used to train the machine learning algorithm 255 based on the training data (e.g., the labels 245 and the corresponding preprocessed images 240) and optimize parameters of the machine learning algorithm 255 during supervised or unsupervised training. In some cases, the training process includes iterative operations to learn a set of parameters (e.g., one or more coefficients and / or weights) of the machine learning algorithm 255. Each parameter is a parameter The cost function or objective function may be an adjustable variable such that the value of the meter is adjusted during training. For example, the cost function or objective function may be configured to optimize accurate classification of the displayed representation, optimize characterization of a given type of feature (e.g., characterize shape, size, uniformity, etc.), optimize detection of a given type of feature, and / or optimize accurate localization of a given type of feature. Each iteration may include learning a set of parameters of the machine learning algorithm 255 that minimizes or maximizes a cost function of the machine learning algorithm 255, such that the value of the cost function using the set of parameters is smaller or larger than the value of the cost function using another set of parameters in the previous iteration. The cost function may be constructed to measure the difference between the predicted output using the machine learning algorithm 255 and the label 245 included in the training data. Once a set of parameters is identified, the machine learning algorithm 255 may be trained and utilized for the designed localization and / or classification.

[0073] The training iterations continue until a stopping condition is met. The training completion condition can be configured to be met (for example) when a predetermined number of iterations are completed, when a statistical value generated based on testing or validation exceeds a predetermined threshold (e.g., a classification accuracy threshold), when a statistical value generated based on a confidence metric (e.g., an average or median confidence metric, or a percentage of confidence metrics above a particular value) exceeds a predetermined confidence threshold, and / or when a user device that was involved in the training review closes the training application executed by the training controller 265. In some cases, a new training iteration can be initiated in response to receiving a corresponding request or trigger condition from a user device (e.g., drift is determined within the trained machine learning model 260).

[0074] The trained machine learning model 260 can then be used (in the result generation stage 225) to process the new pre-processed image 240 to generate predictions or inferences, such as predicting cell center and / or location probabilities, classifying cell types, generating cell masks (e.g., segmentation masks for each pixel of the image), predicting a diagnosis or prognosis of a disease in a subject, such as a patient, or a combination thereof. In some cases, the masks display the locations of indicated cells associated with one or more biomarkers. For example, given tissue stained for a single biomarker, the trained machine learning model 260 can be configured to (i) infer cell centers and / or locations, (ii) classify cells based on features of the staining pattern associated with the biomarker, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cells. As another example, given tissue stained for two biomarkers, the trained machine learning model 260 can be configured to (i) infer cell centers and / or locations, (ii) classify cells based on staining pattern features associated with the two biomarkers, and (iii) output a cell detection mask for cells positive for the first biomarker, a cell detection mask for cells negative for the first biomarker, a cell detection mask for cells positive for the second biomarker, and a cell detection mask for cells negative for the second biomarker. As another example, given tissue stained for a single biomarker, the trained machine learning model 260 can be configured to (i) infer cell centers and / or locations, (ii) classify cells based on cell features and staining patterns associated with the biomarker, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cell codes, as well as a mask cells classified as tissue cells.

[0075] In some cases, analysis controller 280 generates analysis results 285 that are made available to the entity that requested processing of the underlying image. Analysis results 285 may include a mask output from trained machine learning model 260 overlaid on new pre-processed image 240. Additionally or alternatively, analysis results 285 may include a whole slide tumor score, etc. The information may include information calculated or determined from the output of a trained machine learning model of. In an exemplary embodiment, the automated analysis of the tissue slide uses an FDA-cleared 510(k) approved algorithm of assignee VENTANA. Alternatively or additionally, any other automated algorithm may be used to analyze selected regions of the image (e.g., masked images) and generate a score. In some embodiments, the analysis controller 280 may further respond to instructions received from a computing device from a pathologist, physician, investigator (e.g., associated with a clinical trial), patient, medical professional, etc. In some cases, the communication from the computing device includes an identifier for each subject of the set in response to a request to perform an iteration of the analysis for each subject represented in the set of particular subjects. The computing device may further perform an analysis based on the machine learning model and / or the output of the analysis controller 280 and / or provide a recommended diagnosis / treatment for the subject.

[0076] It will be appreciated that the computing environment 200 is exemplary and that computing environments 200 having different stages and / or using different components are contemplated. For example, in some cases, the network may omit the pre-processing stage 210 such that the images used to train the algorithm and / or the images processed by the model are raw images (e.g., from an image data store). As another example, it will be appreciated that each of the pre-processing stage 210 and the training stage 220 may include a controller for performing one or more operations described herein. Similarly, the labeling stage 215 is shown in relation to the labeling controller 250 and the result generation stage 225 is shown in relation to the analysis controller 280, although the controllers associated with each stage may additionally or alternatively facilitate other operations described herein other than generating labels and / or generating analysis results. 2 lacks displayed representations of devices associated with a programmer (e.g., who selected the architecture of the machine learning algorithm 255 that defined how the various interfaces would work, etc.), devices associated with a user providing an initial label or label review (e.g., in the labeling stage 215), and devices associated with a user requesting model processing of a given image (which may be the same user as the user who provided the initial label or label review or a different user). Despite the lack of representations of these devices, the computing environment 200 may include use of one, more, or all of the devices, and may in fact include use of multiple devices associated with a corresponding number of users providing an initial label or label review, and / or a corresponding number of users requesting model processing of various images.

[0077] V. Techniques for Automated Object Localization and Classification Automated instance segmentation (e.g., cell localization and classification) of digital images provides meaningful information for image analysis in the biomedical industry and plays an important role in disease diagnosis and treatment. For example, automated localization and classification of programmed cell death protein 1 ("PD1") positive and negative cells based on the expression level and pattern of PD1 in a biological sample can be used to determine the complex immunological status and disease diagnosis of a sample subject. The accuracy of the generated segmentation mask of an object such as a cell depends on the corresponding machine learning model, and the performance of the machine learning model depends on the training and architecture of the machine learning algorithm. In various embodiments, the performance of the machine learning model is optimized by training the machine learning algorithm using images labeled by masks generated by a repel coding scheme that takes into account the various biomarker staining patterns observed in the image. Furthermore, in some embodiments, the performance of the machine learning model is optimized by using residual blocks with skip connections to not only localize the object but also to capture complex features for multi-class segmentation. It is optimized by using a modified U-Net architecture.

[0078] An Exemplary Repel Coding Scheme for VA Object Localization and Classification FIG. 3 illustrates a flowchart showing a process 300 for implementing a repel coding scheme to generate a repel coding mask for use in training a machine learning algorithm, according to various embodiments. The process 300 shown in FIG. 3 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., memory device). The process 300 presented in FIG. 3 and described below is intended to be exemplary and non-limiting. FIG. 3 illustrates various processing steps performed in a particular sequence or order, but is not limited thereto. In certain alternative embodiments, the steps may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, such as those illustrated in FIG. 1 and FIG. 2, the process illustrated in FIG. 3 can be performed by a pre-processing or labeling subsystem (e.g., computing environment 200) to generate a repel coding mask for training by one or more machine learning algorithms (e.g., machine learning algorithm 255).

[0079] The process 300 begins at block 310, where an image of a biological sample is acquired or accessed by a computing device (e.g., the labeling controller 250 of the computing environment 200 described with respect to FIG. 2). In some cases, the image displays cells having a staining pattern associated with a biomarker. In other examples, the image displays cells having multiple staining patterns associated with multiple biomarkers. The image is annotated with raw dot labels. The raw dot labels provide information including (i) cell center, and (ii) expression of one or more biomarkers. As shown in FIG. 4, the image 400 includes raw dot labels 405 located at the center of each cell, where the color, symbol, pattern, etc. of the raw dot labels 405 indicates the expression of one or more biomarkers. An "x" label 405 indicates that the cell is negative for (does not express) the biomarker. A "+" label 405 indicates that the cell is positive for (expresses) the biomarker. As described with respect to FIG. 2, the raw dot labels 405 may be annotated by a user or automatically by an annotation tool in a pre-processing stage.

[0080] In block 320, the image is coded into an initial cell localization mask using a repel coding algorithm. The repel coding algorithm is defined in equation (1). The repel coding algorithm calculates the distance between a pixel and its closest cell center label, and the distance between the pixel and its second closest cell center label. For each pixel, a pseudo distance is calculated based on the distance between the pixel and its closest cell center label, and the distance between the pixel and its second closest cell center label. If the pseudo distance is equal to or greater than a threshold, the corresponding pixel is assigned a value of 0. Otherwise, a non-zero value is assigned to the pixel according to the pseudo distance. FIG. 5 shows an image 500 (e.g., image 400 described with respect to FIG. 4) being coded by the repel coding algorithm, which outputs an initial cell localization mask 505 including a cell center 510 (represented by high intensity pixels) and a periphery 515 represented by response attenuation (low intensity pixels) away from the cell center 510.

[0081] In block 330, the initial cell localization mask is segmented based on the biomarker staining pattern using the raw dot labels. The segmentation is performed using a segmentation algorithm. The segmentation is performed using an instance segment. The segmentation algorithm is a pixel-level classification for localization, and along with the pixel-level classification for localization, the segmentation algorithm further classifies each instance of a class separately (e.g., positive cells for PD1, negative cells for PD1, positive cells for HER2, negative cells for HER2, etc.). The segmentation algorithm can implement any known instance segmentation technique known in the art. For example, in some cases, the segmentation algorithm is ruled based on the biomarker information provided by the raw dot labels to segment the cell localization mask. In other examples, the segmentation algorithm is cluster-based, such as k-means, to segment the cell localization mask based on the biomarker information provided by the raw dot labels. FIG. 5 shows that the segmented initial cell localization mask 505 is segmented to generate a segmented cell localization mask 520, where each pixel belongs to a particular class (either background 525 or cell 530) based on the repel coding algorithm, and a particular biomarker class (either negative for biomarker 535 or positive for biomarker 540) is assigned to each pixel of the image based on the segmentation algorithm. It should be appreciated that the order of blocks 320 and 330 can be reversed where a segmentation algorithm is first applied to the image to generate a segmentation mask and then a repel coding algorithm is applied to the segmentation mask to generate a segmented cell localization mask.

[0082] In block 340, the segmented cell localization mask is split into two or more repulsion coding masks. Essentially, each class represented in the segmented cell localization mask is split into its own separate repulsion coding mask. For example, if the segmented cell localization mask has two classes: (1) cells positive for the biomarker, and (2) cells negative for the biomarker, the segmented cell localization mask is split into two repulsion masks: (1) one mask for cells positive for the biomarker, and (2) another mask for cells negative for the biomarker. The splitting may be performed using an image processing algorithm that hides or removes various classes that can be selected as parameters. For example, continuing with the above example, a mask for cells positive for the biomarker can be generated by hiding or removing cells negative for the biomarker from the segmented cell localization mask. On the other hand, a mask for cells negative for the biomarker can be generated by hiding or removing cells positive for the biomarker from the segmented cell localization mask. FIG. 5 shows that the segmented cell localization mask 520 is split into two repel coding masks: (1) one mask 545 for cells positive for the biomarker, and (2) another mask 550 for cells negative for the biomarker.

[0083] Example U-Net for VB instance segmentation The modified U-Net model can be used in a biomarker segmentation method to individually extract complex features from an input image (e.g., one or more images of a biological sample) and generate a high-resolution two-dimensional segmentation mask. As shown in FIG. 6, the modified U-Net model 600 can include an encoder 605 and a decoder 610, which gives a U-shaped architecture. The encoder 605 is a CNN network that includes repeated application of convolutions (e.g., 3×3 convolutions (unpadded convolutions)) with one or more layers each comprising a residual block with skip connections followed by a rectified linear unit (ReLU), and a max pooling operation for downsampling (e.g., 2×2 max pooling with stride 2). One or more layers of the encoder 605 include residual blocks with skip connections. With the help of the residual blocks, the original input of the convolution is also added to the output of the convolution. Each downsampling step or In the pooling operation, the number of feature channels can be doubled. During encoding, the spatial information of the image data is reduced and the feature information is increased. The decoder 610 is a CNN network that combines the features and spatial information from the encoder 605 (upsampling of the feature maps from the encoder 605). The upsampling of the feature maps is followed by an upconvolution (upsampling operator) that halves the number of channels, concatenation with the correspondingly cropped feature maps from the encoder 605, repeated application of convolutions (e.g., two 3x3 convolutions), each followed by a rectified linear unit (ReLU), and a final convolution (e.g., a 1x1 convolution) to generate a two-dimensional segmentation mask. It should be understood that one or more layers of the decoder 610 can also include residual blocks with skip connections. For localization, the high-resolution features from the encoder 605 are combined with the upsampled output from the decoder 610.

[0084] Techniques for training machine learning algorithms for VC instance segmentation FIG. 7 shows a flow chart illustrating a process 700 for training a machine learning algorithm (e.g., modified U-Net) to localize and classify cells based on biomarker expression patterns, according to various embodiments. The process 700 shown in FIG. 7 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., memory device). The process 700 presented in FIG. 7 and described below is intended to be exemplary and non-limiting. FIG. 7 shows various processing steps performed in a particular sequence or order, but is not limited thereto. In certain alternative embodiments, the steps may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, such as those shown in Figures 1 and 2, the process shown in Figure 7 can be performed as part of a training phase (e.g., algorithm training 220) to generate a prediction or segmentation mask that includes pixel-level classifications that localize cells (e.g., background and cells) and pixel-by-pixel biomarker expression classifications for each instance of a class (e.g., cell) that classifies cells based on biomarker expression.

[0085] Process 700 begins at block 705, where a training image of a biological sample (e.g., preprocessed image 240 of computing environment 200 described with respect to FIG. 2) is obtained or accessed by a computing device. In some cases, the training image displays cells having a staining pattern associated with a biomarker. In other examples, the training image displays cells having multiple staining patterns associated with multiple biomarkers. The training image is annotated with a repel coding mask (e.g., a repel coding mask generated according to process 300 described with respect to FIG. 3). The repel coding mask provides information about the training image, including (i) cell location, and (ii) classification of the cells with respect to one or more biomarkers.

[0086] At block 710, one or more images and mask labels are divided into image patches of a predetermined size. For example, images usually have random sizes, and machine learning algorithms such as modified CNNs learn better with normalized image sizes, and thus, training images can be divided into image patches having specific sizes to optimize training. In some embodiments, training images are divided into image patches having a predetermined size of 64 pixels by 64 pixels, 128 pixels by 128 pixels, 256 pixels by 256 pixels, or 512 pixels by 512 pixels. The division of training images may be performed before the repel coding mask is generated, so that the subsequently generated repel coding mask may be divided into image patches having a predetermined size of 64 pixels by 64 pixels, 128 pixels by 128 pixels, 256 pixels by 256 pixels, or 512 pixels by 512 pixels. has the same size as the size of the corresponding training image. Alternatively, the segmentation may be performed after the repel coding mask is generated. In such a case, the repel coding mask is pre-processed and segmented into mask patches of the same predefined size as the training images before, after, or simultaneously with the segmentation of the training images.

[0087] At block 715, a machine learning algorithm is trained on the image patch. In some cases, the machine learning algorithm is a modified U-Net that includes an encoder and a decoder, where one or more layers of the encoder include a residual block with skip connections (or Resnet). The training can include performing an iterative operation to find a set of parameters of the machine learning algorithm that minimizes or maximizes a cost function of the machine learning algorithm. The output from the iterations is a prediction or segmentation mask that includes a pixel-level classification (e.g., background and cell) that localizes the cell, and a biomarker expression classification per pixel of each instance of a class (e.g., cell) that classifies the cell based on biomarker expression. Each iteration includes finding a set of parameters of the machine learning algorithm such that the value of the cost function using the set of parameters is smaller or larger than the value of the cost function using another set of parameters in the previous iteration. The cost or objective function is constructed to measure the difference between (i) the predicted pixel-level classification and predicted biomarker expression classification per pixel, and (ii) the ground truth repel coding mask. In certain cases, the cost function is a binary cross-entropy loss function.

[0088] In some cases, training further includes adjusting a learning rate of the machine learning algorithm by maximizing or minimizing the learning rate according to a predetermined schedule, which may be a step-decay schedule that decreases the learning rate by a predetermined factor every predetermined number of epochs to optimize the cost function.

[0089] The training iterations continue until a stopping condition is met. The training completion condition can be configured to be met (for example) when a predetermined number of iterations are completed, when a statistical value generated based on testing or validation exceeds a predetermined threshold (e.g., a classification accuracy threshold), when a statistical value generated based on a confidence metric (e.g., an average or median confidence metric, or a percentage of confidence metrics above a particular value) exceeds a predetermined confidence threshold, and / or when a user device that was involved in the training review closes the training application. As a result of the training, the machine learning algorithm has learned a nonlinear relationship in the image that is used by the machine learning algorithm to predict, for each instance of a class (e.g., cell) that classifies cells based on biomarker expression, a pixel-level classification (e.g., background and cell) that localizes the cell and a biomarker expression classification per pixel. The output of the training includes a trained machine learning model having a learned set of parameters associated with the nonlinear relationship that derived the minimum or maximum of the cost function from all the iterations.

[0090] At block 720, a trained machine learning model is provided. For example, the trained machine learning model can be deployed for execution in an image analysis environment, as described with respect to FIG.

[0091] Techniques for Instance Segmentation Using VD Machine Learning Models FIG. 8 shows a flowchart illustrating a process 800 of using a machine learning model (e.g., a modified U-Net model) to localize and classify cells based on biomarker expression patterns, according to various embodiments. The process 800 shown in FIG. 8 may be implemented in software (e.g., code, instructions, etc.) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be implemented in a program (e.g., instruction, program). The software may be stored in a non-transitory storage medium (e.g., a memory device). The process 800 presented in FIG. 8 and described below is intended to be exemplary and non-limiting. FIG. 8 illustrates various processing steps occurring in a particular sequence or order, but is not limited thereto. In certain alternative embodiments, the steps may be performed in a number of different orders, or some steps may be performed in parallel. In certain embodiments, such as those illustrated in FIGS. 1 and 2, the process illustrated in FIG. 8 may be performed as part of a result generation stage (e.g., result generation 280) to generate a prediction or segmentation mask that includes pixel level classifications (e.g., background and cell) that localize cells, and pixel-by-pixel biomarker expression classifications for each instance of a class (e.g., cell) that classifies cells based on biomarker expression.

[0092] Process 800 begins at block 805, where one or more images for a biological sample are accessed or obtained. The images display cells containing staining patterns of one or more biomarkers. In certain cases, the one or more images display cells containing a staining pattern of a biomarker and another biomarker. As described with respect to FIG. 1, the images can be pre-processed by immunochemical staining techniques (e.g., IF) so that specific proteins and organelles in the biological sample are visible for processing and analysis in an analytical system. In some embodiments, the images are stained using multiple stains or binders, such as antibodies, so that information about different biomarkers can be reported under multi-channel analysis or similar techniques.

[0093] At block 810, one or more images can be divided into image patches of a predetermined size. For example, images typically have random sizes, and machine learning algorithms such as modified CNNs learn better with normalized image sizes, and therefore images can be divided into image patches having specific sizes to optimize analysis. In some embodiments, images are divided into image patches having a predetermined size of 64 pixels by 64 pixels, 128 pixels by 128 pixels, 256 pixels by 256 pixels, or 512 pixels by 512 pixels.

[0094] At block 815, the one or more images or image patches are input to a machine learning model for further analysis. In some cases, the machine learning model is a modified CNN (e.g., U-Net) model including an encoder and a decoder, where one or more layers of the encoder include a residual block with skip connections (or Resnet). The machine learning model further includes parameters learned from the training images and corresponding label masks for each biomarker in the training images (as described in detail with respect to the computing environment 200 and process 700). The label masks were generated using a repel coding scheme in combination with the labels for each of the biomarkers (as described in detail with respect to the computing environment 200 and process 300). As described with respect to FIG. 3, generating the label masks includes (i) coding cells in each training image using repel coding, where the coding includes cell centers and peripheries represented by response decay away from the cell centers, and (ii) generating two or more label masks for each of the biomarkers in the image based on the coding and the respective labels of the biomarkers.

[0095] In block 820, one or more images or image patches are encoded into a feature representation that includes the discriminatory features extracted by the provided machine learning model. The discriminatory features (e.g., lower resolution features) can be associated with biological materials or biological structures, such as cells. The machine learning model may be used to encode multiple different levels in multiple different sub-networks. The encoding of images or image patches into discriminatory features can be performed in a time-domain manner, with each sub-network being associated with at least one expression (e.g., positive or negative) of a biomarker.

[0096] In block 825, the extracted discriminatory features are combined with the cell features and spatial information and the staining patterns of the biomarkers in the image or image patch through a sequence of upconvolution and concatenation with discriminatory features. The combination can be performed by projecting the extracted discriminatory features into a pixel space (e.g., a higher resolution) and classifying each pixel space. The decoder with the machine learning model can perform the combination and projection of the extracted discriminatory features at multiple different levels in multiple different sub-networks. In certain cases, the multiple different levels perform upsampling (i.e., expanding the feature dimension to the original size of the input image patch) and concatenation, followed by regular convolution operations to project the extracted discriminatory features. The classification includes cell detection and classification of cells based on the staining patterns of the biomarkers. Specifically, the classification includes pixel-level classification (e.g., background and cell) to localize the cells and biomarker expression classification per pixel for each instance of a class (e.g., cell) to classify the cells based on biomarker expression.

[0097] At block 830, two or more segmentation masks for each of the one or more biomarkers are generated and output based on the combined cellular features and spatial information and the staining patterns of the biomarkers. The two or more segmentation masks include a positive segmentation mask for cells expressing the biomarker and a negative segmentation mask for cells not expressing the biomarker. In some cases, the segmentation masks are output as high-resolution image patches, with each pixel on each mask assigned a value representing the probability that the cell location and biomarker type is expressed (or reaches a certain expression criterion). In some embodiments, the segmentation masks are output with a size of 256 pixels by 256 pixels, with each pixel having a value ranging from 0 to 1, with 0 representing that the biomarker corresponding to the segmentation mask is not expressed (or does not reach an expression criterion) at the pixel location, and 1 representing that the biomarker corresponding to the segmentation mask is expressed (or reaches an expression criterion) at the pixel location.

[0098] In optional block 835, two or more segmentation masks can be overlaid on one or more images or image patches to generate an instance segmentation image. In some cases, the overlay algorithm can include selecting a maximum pixel value for each pixel on the segmentation mask and assigning the maximum value to a corresponding pixel on the instance segmentation image. In some cases, the overlay algorithm can include assigning weights to pixels on each segmentation mask, combining the weighted pixel values ​​of each pixel, and assigning the combined value to a corresponding pixel on the instance segmentation image. The overlay algorithm can further include a checking step to verify that each pixel value is less than or equal to 1. It should be understood that the overlay algorithm is not limited to the above-mentioned algorithms and can be any algorithm known to those skilled in the art.

[0099] At block 840, the two or more segmentation masks and / or instance segmentation images are output. For example, the two or more segmentation masks and / or instance segmentation images may be presented locally or transmitted to another device. The two or more segmentation masks and / or instance segmentation images may be output along with a subject's identifier. In some cases, the two or more segmentation masks and / or instance segmentation images may be output along with a subject's identifier. The data is output to a end user or to a storage device.

[0100] In optional block 845, a diagnosis of a subject associated with the biological sample is determined using the two or more segmentation masks and / or the instance segmented images. In some cases, the diagnosis is determined based on cells expressing the biomarker in the instance segmented image. In some cases, the diagnosis is determined based on cells not expressing the biomarker in the example segmented image. In some cases, the diagnosis is determined based on a combination of cells expressing the biomarker in the instance segmented image and cells not expressing the biomarker in the instance segmented image.

[0101] In optional block 850, a treatment is administered to a subject associated with the biological sample. In some cases, the treatment is administered based on cells expressing the biomarker in the example segmented image. In some cases, the treatment is administered based on cells not expressing the biomarker in the example segmented image. In some cases, the treatment is administered based on the subject's diagnosis determined in block 845. In some cases, the treatment is administered based on any combination of cells expressing the biomarker in the instance segmented image, cells not expressing the biomarker in the instance segmented image, and the subject's diagnosis determined in block 845.

[0102] Techniques for Using Instance Segmentation in VE Image Analysis FIG. 9 illustrates a flow chart showing a process 900 for applying a segmentation mask to support or improve image analysis, according to various embodiments. The process 900 shown in FIG. 9 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., memory device). The process 900 presented in FIG. 9 and described below is intended to be exemplary and non-limiting. FIG. 9 illustrates various processing steps performed in a particular sequence or order, but is not limited thereto. In certain alternative embodiments, the steps may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, such as those shown in FIG. 1 and FIG. 2, the process shown in FIG. 9 can be performed as part of a result generation stage (e.g., result generation 225) to support or improve image analysis.

[0103] Process 900 begins at block 905, where multiple images of a sample are accessed. In some cases, the specimen is stained for one or more biomarkers. In some embodiments, the accessed images are RGB images or multispectral images. In some embodiments, the accessed images are stored in a memory device. The images can be generated or acquired, for example, in real time, using an imaging device (e.g., an imaging device described with respect to FIG. 1). In some embodiments, the images are accessed from a microscope or other instrument capable of capturing image data of a microscope slide holding a specimen, as described herein. In some embodiments, the images are accessed from a 2D scanner, such as one capable of scanning image tiles. Alternatively, the images may be images previously generated (e.g., scanned) and stored in a memory device (or retrieved from a server over a communication network, in which case). In some cases, the images are instance segmented images that include one or more segmentation masks for biological materials or biological structures, such as cells. The instance segmented images can be generated and / or retrieved for access as described with respect to process 800 and FIG. 8.

[0104] In some embodiments, the accessed image is a multiplex image, i.e., the received image is an image of a biological sample stained with two or more stains. In these embodiments, at block 910, prior to further processing, each multiplex image is unmixed into its constituent channels, with each unmixed channel corresponding to a particular stain or signal. Following image acquisition and / or unmixing, at blocks 915-945, the image or unmixed image channel images are processed by image analysis algorithms to identify and classify cells and / or nuclei. The processes and analysis algorithms described herein can be adapted to identify and classify various types of cells or cell nuclei based on features within the input image, including identifying and classifying tumor cells, non-tumor cells, stromal cells, lymphocytes, non-target stains, etc.

[0105] At block 915, candidate nuclei are identified. In some embodiments, the image is input to image analysis to detect nuclear centers (seeds) and / or segment the nuclei. For example, a plurality of pixels in the image stained with a biomarker can be identified, which includes considering one or more color planes of a plurality of pixels in the foreground of the input image for simultaneous identification of cytoplasmic and cell membrane pixels of a cell. In some cases, the image is pre-processed to remove portions of the image that are not analyzed or determined to be background or negative for a given biomarker, such as slide background, cells not expressing the given biomarker as described with respect to Figures 2-8, and / or counterstain components of the image. A threshold level between cytoplasmic and cell membrane pixels in the foreground of the digital image is then determined, and the selected pixel and a predetermined number of its neighboring pixels from the foreground are processed to determine whether the selected pixel is a cytoplasmic pixel, a cell membrane pixel, or a transition pixel in the digital image based on the determined threshold level. In some embodiments, tumor nuclei are automatically identified by applying Parvin's radial symmetry-based method, such as radial symmetry-based method, hematoxylin image channel or biomarker image channel after unmixing.

[0106] At block 920, features are extracted from the candidate nuclei. At least some of the features can be associated with biological materials or biological structures within the target region of the image, such as tumor cells or clusters of tumor cells. The extraction may be performed by an image analysis predictive model, such as Mask R-CNN, capable of semantic or instance segmentation. For example, after the candidate nuclei are identified, the candidate nuclei can be further analyzed by an image analysis predictive model to distinguish tumor nuclei from other candidate nuclei (e.g., lymphocyte nuclei). In certain examples, the other candidate nuclei can be further processed to identify specific classes of nuclei and / or cells, for example, to identify lymphocyte nuclei and stromal nuclei.

[0107] At block 925, the biological material or structures in the target region are classified into types of cells or cell nuclei based on the features extracted from the candidate nuclei. The classification may be performed by an image analysis predictive model. In some embodiments, a learned supervised classifier is applied to distinguish tumor nuclei from the candidate nuclei. For example, a learned supervised classifier can be trained on nuclei features to identify tumor nuclei and then applied to classify nuclei candidates in the test image as either tumor nuclei or non-tumor nuclei. Optionally, the learned supervised classifier may be further trained to distinguish different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the learned supervised classifier used to identify tumor nuclei is a random forest classifier. For example, the random forest classifier can be trained by (i) creating a training set of tumor nuclei and non-tumor nuclei, (ii) extracting features of each nuclei, and (iii) training the random forest classifier to distinguish tumor nuclei from non-tumor nuclei based on the extracted features. The trained random forest classifier is then applied to classify nuclei in the test image as tumor nuclei. and non-tumor nuclei. Optionally, the random forest classifier may be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei.

[0108] At block 930, a segmentation mask of the target region is predicted and output based on the classification of the biological material or structure. The segmentation mask may be output by an image analysis predictive model. The segmentation mask can be overlaid on the image to generate a target region mask image.

[0109] In optional block 935, metrics are derived from the various identified nuclei, cells, clusters of cells, and / or biological material or structures, as described in detail herein. In some cases, one or more metrics can be calculated by applying various image analysis algorithms to pixels contained in or surrounding the classified nuclei, cells, clusters of cells, and / or biological material or structures. In some embodiments, the metrics include disease state, area, minor and major axis lengths, perimeter, radius, hardness, etc.

[0110] At block 940, a non-target region mask image is provided. For example, the target region mask image or the target region may be provided to a memory storage device, provided to a display of a computing device, provided to a user in one or more types of media, such as a user interface, etc. In some cases, providing the target region mask image includes providing associated metrics, or the metrics may be provided separately.

[0111] VI. Working Examples Example 1. Experiments using PD1 The machine learning model was trained using images from the ImageNet dataset. The ImageNet dataset included multiple IF stained tissue slides with indications of stomach, pancreas, lung, breast, colon and bladder. The dataset included variable size images covering tumor, peritumoral and normal tissue regions in the slides. The resolution of the images was 0.325 μm / pixel. The images were pre-processed using repel coding as described herein with respect to equation (1). Patches were extracted from these images and masks to train the model. In this example, the training image patches had a size of 250 pixels by 250 pixels. The biomarker used for classification was PD1. This biomarker is expressed primarily on lymphocytes and appears in a number of patterns including membranous (partial or complete), punctate, nuclear, spherical and combinations of these patterns. PD1 shows a pattern of expression with a wide range of intensities. Figure 10 shows the different staining patterns of PD1.

[0112] In this experiment, 100 images were selected from 15 different slides. The images were divided into patches of size 250 pixels x 250 pixels before feeding into the machine learning model as described with respect to Figure 6. Figure 11 shows the visual results from both the training and validation sets. The model did not detect any false positives for erythrocyte expression. Table 1 shows the recall and precision of PD1+ and PD1- cells for both the training and validation sets. [Table 1]

[0113] VII. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0114] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be left to those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0115] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0116] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

Claims

1. 1. A computer-implemented method, comprising: accessing an image of a biological sample, the image displaying cells containing a staining pattern of a biomarker; inputting the image into a machine learning model, the machine learning model comprises a convolutional neural network including an encoder and a decoder; one or more layers of the encoder include residual blocks having skip connections; parameters of the machine learning model are learned from training images and a label mask for each biomarker in the training images; The label mask was generated using recoding in combination with the labels of each of the biomarkers. inputting the image into a machine learning model; encoding the image into a feature representation including extracted discriminative features by the machine learning model; combining the cellular features and spatial information with the staining patterns of the biomarkers by the machine learning model through a sequence of upconvolution and concatenation with the extracted discriminative features from the feature representation; generating, by the machine learning model, two or more segmentation masks for the biomarkers in the image based on the combined features and spatial information of the cells and the staining patterns of the biomarkers, the two or more segmentation masks including a positive segmentation mask for cells expressing the biomarkers and a negative segmentation mask for cells not expressing the biomarkers; A method comprising:

2. superimposing the two or more segmentation masks on the image to generate an instance segmented image; outputting the instance segmentation image; The method of claim 1 further comprising:

3. 3. The method of claim 2, further comprising determining, by a user, a diagnosis of a subject associated with the biological sample, the diagnosis being determined based on (i) the cells in the instance segmented image that express the biomarker and / or (ii) the cells in the instance segmented image that do not express the biomarker.

4. 4. The method of claim 3, further comprising administering, by the user, a treatment to the subject based on (i) the cells in the instance segmented image that express the biomarker, (ii) the cells in the instance segmented image that do not express the biomarker, and / or (iii) the diagnosis of the subject.

5. the image displays cells containing a staining pattern of the biomarker and another biomarker; the machine learning model generates two or more segmentation masks for the different biomarkers in the image; The two or more segmentation masks for the additional biomarkers may be a positive segmentation mask for cells expressing the additional biomarkers, and a positive segmentation mask for cells expressing the additional biomarkers. and a negative segmentation mask for cells that do not express the specified biomarker.

6. superimposing the two or more segmentation masks for each of the biomarkers and the further biomarkers on the image to generate an instance segmented image; outputting the instance segmentation image; The method of claim 5 further comprising:

7. 2. The method of claim 1, wherein generating the label mask comprises: (i) coding cells in each training image using the repel coding, the coding including a cell center and a periphery represented by a response attenuation away from the cell center; and (ii) generating two or more label masks for each of the biomarkers in the images based on the coding and the label for each of the biomarkers.

8. 2. The method of claim 1, wherein combining the characteristic and spatial information of the cell with the staining pattern of the biomarker comprises projecting the extracted discriminatory features into a pixel space and classifying each pixel space, wherein the classification comprises cell detection and classification of the cell based on the staining pattern of the biomarker.

9. 1. A system comprising: one or more data processors; A non-transitory computer readable storage medium comprising instructions, which when executed on said one or more data processors, cause said one or more data processors to perform any of the steps of the method of any one of claims 1 to 8. A system comprising:

10. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform any of the steps of the method of any one of claims 1 to 8.

11. 1. A computer-implemented method, comprising: accessing an image of a biological sample, the image displaying cells containing staining patterns of biomarkers, the cells being annotated with labels providing information including: (i) cell centroids; and (ii) expression of the biomarkers; generating two or more repel coding masks for each of the images, the generating comprising: (i) coding the cells in the images using a repel coding algorithm, wherein an output of the coding is an initial cell localization mask including the cell center and a periphery represented by a response attenuation away from the cell center; (ii) segmenting the initial cell localization mask using the labels to classify each instance of the cell based on the expression of the biomarkers; and (iii) dividing the initial cell localization mask into the two or more repel coding masks based on the segmentation and classification of the instance of the cell; labeling each of the images with the two or more repel coding masks to generate a set of training images; training a machine learning algorithm on the set of training images to generate a machine learning model, the training comprising performing iterative operations to learn a set of parameters for segmenting and classifying cells that maximizes or minimizes an objective function, each iteration comprising finding a set of parameters for the machine learning algorithm such that a value of the objective function using the set of parameters is greater or smaller than a value of the objective function using another set of parameters in a previous iteration, the objective function being configured to measure a difference between a segmentation mask predicted using the machine learning algorithm and the two or more repel coding masks for the images; Providing the machine learning model; A method comprising:

12. The method of claim 11 , wherein the two or more recoding masks comprise a positive mask for cells expressing the biomarker and a negative mask for cells not expressing the biomarker.

13. 12. The method of claim 11, wherein the trained machine learning model comprises a convolutional neural network including an encoder and a decoder, and one or more layers of the encoder include residual blocks with skip connections.

14. The method of claim 11 , wherein the image is an image patch of a predetermined size.

15. 15. The method of claim 14, further comprising: dividing the two or more repel coding masks into mask patches of the predetermined size; and labeling each of the image patches with the mask patch to generate a set of training images.

16. 1. A system comprising: one or more data processors; A non-transitory computer readable storage medium comprising instructions, which when executed on said one or more data processors, cause said one or more data processors to perform any of the steps of the method of any one of claims 11 to 15. A system comprising:

17. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform any of the steps of the method of any one of claims 11 to 15.

Citation Information

Patent Citations

  • Processing of histology images with a convolutional neural network to identify tumors

    US20190206056A1

  • Deep Learning Method For Tumor Cell Scoring On Cancer Biopsies

    US20190392580A1