Machine learning models for cell localization and classification trained using repelcoding

Abstract

To effectively train machine learning models to detect, characterize, or classify some or all of digital pathology images.SOLUTION: An aspect of the present disclosure comprises: accessing an image of a biological sample, wherein the image displays cells containing a staining pattern of a biomarker; inputting the image into a machine learning model; encoding the image into a feature representation including extracted discriminative features using the machine learning model; combining cellular features and spatial information with the staining patterns of the biomarkers through concatenation with the extracted discriminative features from an upconvolution sequence and the feature representation by the machine learning model; and generating, by the machine learning model, two or more segmentation masks for the biomarkers in the image based on the combined cellular features and spatial information and the staining patterns of the biomarkers.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 065,268, filed August 13, 2020, which is incorporated herein by reference in its entirety for all purposes.

[0002] Field The present disclosure relates to digital pathology, and in particular to techniques for using repel coding to effectively train machine learning models to automatically detect, characterize and / or classify part or all of digital pathology images. [Background technology]

[0003] background Digital pathology involves scanning slides (e.g., histopathology or cytopathology glass slides) into interpretable digital images on a computer screen. The tissues and / or cells within the digital images can then be examined by digital pathology image analysis and / or interpreted by a pathologist for a variety of reasons, including diagnosing disease, assessing response to treatment, and developing drugs to combat disease. To examine the tissues and / or cells within the digital images (which are substantially transparent), pathology slides can be prepared using various staining assays (e.g., immunohistochemistry) that selectively bind to tissue and / or cellular components. Immunofluorescence (IF) is a technique for analyzing assays that bind to fluorescent dyes specific to antigens. Multiple assays responding to various wavelengths can be utilized on the same slide. These multiplexed IF slides enable understanding of the complexity and heterogeneity of the immune landscape of the tumor microenvironment and its potential impact on tumor response to immunotherapy. In some assays, the target antigen for the stain in the tissue is sometimes referred to as a biomarker. Digital pathology image analysis can then be performed on the digital images of the stained tissues and / or cells to identify and quantify staining for antigens (e.g., biomarkers indicative of various cells, such as tumor cells) in the biological tissue.

[0004] Machine learning techniques have shown great promise in digital pathology image analysis, such as cell detection, counting, localization, classification, and patient prognosis. Many computing systems equipped with machine learning techniques, including convolutional neural networks (CNNs), have been proposed for image classification and digital pathology image analysis, such as cell detection and classification. For example, a CNN can have a series of convolutional layers as hidden layers, and this network structure enables the extraction of representational features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques have also been implemented for image segmentation. Image segmentation is the process of dividing a digital image into multiple segments (sets of pixels, also known as image objects). The purpose of segmentation is to simplify and / or modify the representation of the image to make it more meaningful and easier to analyze. For example, image segmentation is typically used to find objects such as cells and boundaries (lines, curves, etc.) within an image. To perform image segmentation for large data sets (e.g., whole-slide pathology images), the image is first divided into many small patches. A computing system equipped with machine learning techniques is trained to classify these patches, and all patches in the same class are combined into one segmented region. Then, machine learning techniques are further implemented. The segmented regions can then be predicted or further classified (e.g., positive cells for a given biomarker, negative cells for a given biomarker, or cells with no staining expression) based on the phenotypic features associated with the segmented regions. Summary of the Invention

[0005] overview A method, system, and computer-readable storage medium are disclosed for using repel coding to effectively train machine learning models to automatically detect, characterize, and / or classify some or all of digital pathology images.

[0006] The method, system, and computer-readable storage medium may be embodied in various ways.

[0007] In various embodiments, a computer-implemented method includes accessing an image of a biological sample, the image displaying cells containing a staining pattern of biomarkers; inputting the image to a machine learning model, the machine learning model including a convolutional neural network including an encoder and a decoder, one or more layers of the encoder including residual blocks with skip connections, parameters of the machine learning model learned from training images and label masks for each biomarker in the training images, the label masks being generated using repel coding in combination with the labels of the respective biomarkers; and filtering the image by the machine learning model to extract discriminative features. encoding the cellular features and spatial information and the staining pattern of the biomarkers using a machine learning model via concatenation with the upconvolution sequence and extracted discriminative features from the feature representation; and generating, using the machine learning model, two or more segmentation masks for the biomarkers in the image based on the combined cellular features and spatial information and the staining pattern of the biomarkers, wherein the two or more segmentation masks include a positive segmentation mask for cells expressing the biomarkers and a negative segmentation mask for cells not expressing the biomarkers.

[0008] In some embodiments, the method further includes overlaying two or more segmentation masks on the image to generate an instance segmentation image, and outputting the instance segmentation image.

[0009] In some embodiments, the method further includes determining, by the user, a diagnosis of a subject associated with the biological sample, wherein the diagnosis is determined based on (i) cells in the instance segmented image that express the biomarker, and / or (ii) cells in the instance segmented image that do not express the biomarker.

[0010] In some embodiments, the method further includes administering a treatment to the subject by a user based on (i) cells that express the biomarker in the instance segmented image, (ii) cells that do not express the biomarker in the instance segmented image, and / or (iii) a diagnosis of the subject.

[0011] In some embodiments, the image displays cells comprising a staining pattern of the biomarker and another biomarker, and the machine learning model detects the staining pattern of the other biomarker in the image. Two or more segmentation masks are generated, the two or more segmentation masks for the other biomarkers including a positive segmentation mask for cells expressing the other biomarker and a negative segmentation mask for cells not expressing the other biomarker.

[0012] In some embodiments, the method further includes overlaying two or more segmentation masks, one for the biomarker and one for the other biomarker, on the image to generate an instance segmented image, and outputting the instance segmented image.

[0013] In some embodiments, generating the label masks includes (i) coding cells in each training image using repel coding, where the coding includes a cell center and a periphery represented by response attenuation away from the cell center; and (ii) generating two or more label masks for each of the biomarkers in the images based on the coding and the labels of each of the biomarkers.

[0014] In some embodiments, combining the cellular features and spatial information with the staining patterns of the biomarkers comprises projecting the extracted discriminatory features into a pixel space and classifying each pixel space, wherein the classifying comprises cell detection and classification of the cells based on the staining patterns of the biomarkers.

[0015] In various embodiments, a computer-implemented method includes accessing images of a biological sample, the images displaying cells comprising staining patterns of biomarkers, the cells being annotated with labels providing information including (i) cell centers and (ii) expression of the biomarkers; and generating two or more repel coding masks for each of the images, the generating comprising: (i) coding cells in the images using a repel coding algorithm, wherein an output of the coding is an initial cell localization mask comprising a cell center and a periphery represented by a response attenuation away from the cell center; (ii) segmenting the initial cell localization mask using the labels to classify each instance of a cell based on the expression of the biomarkers; and (iii) dividing the initial cell localization mask into two or more repel coding masks based on the segmentation and classification of the instance of the cell. generating a segmentation mask; labeling each of the images with two or more repelcoding masks to generate a set of training images; training a machine learning algorithm on the set of training images to generate a machine learning model, wherein the training includes performing iterative operations to learn a set of parameters for segmenting and classifying cells that maximizes or minimizes an objective function, each iteration including finding a set of parameters for the machine learning algorithm such that a value of the objective function using the set of parameters is greater or smaller than a value of the objective function using another set of parameters in a previous iteration, wherein the objective function is configured to measure a difference between the segmentation mask predicted using the machine learning algorithm and the two or more repelcoding masks for the image; and providing the machine learning model.

[0016] In some embodiments, the two or more recoding masks comprise a positive mask for cells expressing the biomarker and a negative mask for cells not expressing the biomarker.

[0017] In some embodiments, the trained machine learning model includes a convolutional neural network including an encoder and a decoder, where one or more layers of the encoder include residual blocks with skip connections.

[0018] In some embodiments, the image is an image patch of a predetermined size.

[0019] In some embodiments, the method further includes dividing the two or more repel coding masks into mask patches of a predetermined size, and labeling each of the image patches with the mask patch to generate a set of training images.

[0020] In some embodiments, a method is provided that includes determining a diagnosis of a subject by a user based on results generated by a machine learning model trained using some or all of one or more techniques disclosed herein, and potentially selecting, recommending, and / or administering a particular treatment to the subject based on the diagnosis.

[0021] In some embodiments, a method is provided that includes determining, by a user, a treatment to select, recommend, and / or administer to a subject based on results generated by a machine learning model trained using some or all of one or more techniques disclosed herein.

[0022] In some embodiments, a method is provided that includes determining, by a user, whether a subject is eligible to participate in a clinical trial or whether the subject is eligible to be assigned to a particular cohort in a clinical trial based on results generated by a machine learning model trained using some or all of one or more techniques disclosed herein.

[0023] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawings]

[0024] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of various embodiments will become more apparent by way of example only and with reference to the accompanying drawings. [Figure 1] 1 illustrates an example of a histological staining process according to various embodiments of the present disclosure. [Figure 2] FIG. 1 shows a block diagram illustrating a computing environment for processing digital pathology images using machine learning models, according to various embodiments of the present disclosure. [Figure 3] 1 shows a flowchart illustrating a process for implementing a repel coding scheme for generating repel coding masks for use in training machine learning algorithms, according to various embodiments of the present disclosure. [Figure 4] 1 illustrates an annotated image with raw dot labels according to various embodiments of the present disclosure. [Figure 5] 10 shows an image being coded by a repel coding algorithm to provide an initial cell localization mask, according to various embodiments of the present disclosure. [Figure 6] 1 illustrates a modified U-Net model used for biomarker segmentation according to various embodiments of the present disclosure. [Figure 7]FIG. 1 shows a flowchart illustrating a process for training a machine learning algorithm to localize and classify cells based on biomarker expression patterns, according to various embodiments of the present disclosure. [Figure 8-1] 1 shows a flowchart illustrating a process of using a machine learning model to localize and classify cells based on biomarker expression patterns, according to various embodiments of the present disclosure. [Figure 8-2] 1 shows a flowchart illustrating a process of using a machine learning model to localize and classify cells based on biomarker expression patterns, according to various embodiments of the present disclosure. [Figure 9] 1 shows a flowchart illustrating a process for applying a segmentation mask to support or improve image analysis, according to various embodiments of the present disclosure. [Figure 10] 1 illustrates different patterns of PD1 according to various embodiments of the present disclosure. [Figure 11] 1 shows the visual results from both the training and validation sets in Example 1. DETAILED DESCRIPTION OF THE INVENTION

[0025] Detailed Description While specific embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The apparatus, methods, and systems described herein can be embodied in various other forms. Furthermore, various omissions, substitutions, and changes in the form of the exemplary methods and systems described herein can be made without departing from the scope of protection.

[0026] I. Overview The present disclosure describes techniques for detecting, characterizing, and / or classifying some or all of a digital pathology image. More specifically, some embodiments of the present disclosure provide machine learning techniques that use repel coding to effectively train machine learning models to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cellular analysis.

[0027] The ability to characterize biomarkers in cell and tissue samples and measure the heterogeneity of the presence and levels of such biomarkers within and between tissues can provide valuable information in biomedicine for understanding and characterizing various disease states and / or for appropriately selecting available targeted therapeutics for a patient's disease state. Furthermore, the ability to identify and characterize regions within tissues with differential distribution of key biomarkers can provide important information to inform the development of targeted and combination therapies. The development and selection of appropriate combination therapies may also be an important factor in preventing recurrence.

[0028] Automated detection and classification of cells with different biomarker expression levels from digital images has the potential for faster and more accurate extraction of information for disease diagnosis and treatment. Because most cells and tissues are virtually transparent in digital images, modern laboratory techniques have been developed to aid in cell and tissue visualization. Immunofluorescence (IF) is a staining technique that aids in the visualization of components within tissues or cells. IF uses a combination of antibodies and fluorophores to mark specific proteins and organelles in cell or tissue samples. Therefore, IF techniques are widely used in the study of various cell types, the tracking and localization of proteins in cell or tissue samples, and the identification of biological structures in cell or tissue samples. It is possible to evaluate IF-stained cells in tissue sections under a microscope at high magnification and / or automatically analyze digital images of biological samples using digital pathology algorithms. Often, in whole-slide analysis, evaluation of stained biological specimens involves detecting cells or cellular structures in the stained biological specimen, localizing the centers of the cells or cellular structures, and identifying biomarkers and biomarker patterns. It is necessary to recognize this.

[0029] Once cells are visualized for analysis, cell detection and classification techniques are desirable. Cell detection and classification techniques have evolved from using handcrafted features to machine learning-based detection techniques. Most machine learning-based cell detection and classification techniques implement a classical feature-based machine learning pipeline, in which a cell classifier or detector is trained in pixel space and the location and features of target cells are labeled. However, accurate cell detection and classification using classical feature-based machine learning is challenging for several reasons. First, the morphology of different cells varies dramatically, and parametric models must be redesigned and trained to fit different cell targets. Second, even for the same type of cell, different preparation and imaging protocols result in different cell appearances. More importantly, certain types of cells (e.g., immune cells) tend to cluster within groups, and cell crowding or density makes it difficult to define the boundaries between cells and associated features. The issue of cell crowding or density increases the difficulty not only of cell detection but also of classification.

[0030] To address these limitations and issues, various embodiments disclosed herein relate to machine learning techniques for using repel coding to efficiently train machine learning models to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cell analysis. Repel coding of raw data labels for cell detection is based on proximity coding, which is typically used for cell counting. Proximity coding generates a local maximum at the cell center. However, in cell detection, a challenge is distinguishing adjacent cells from each other, especially cells that tend to cluster in groups. Proximity coding focuses on entropy, and the response valleys between adjacent cells are usually not strong enough for accurate cell detection. To overcome this challenge, repel coding is configured to increase the response valleys between adjacent cells and more accurately align the local maximum at the cell center compared to typical proximity coding. Essentially, repel coding codes the centers of cells in a way that increases reversibility (achieving a better balance between entropy and reversibility), which allows cells to be more easily distinguished and labeled.

[0031] Furthermore, repel coding is configured to intelligently apply cell center coding based on image labels for each of one or more characteristics associated with a cell to generate a repel coding mask. The one or more characteristics can be biomarker staining patterns (e.g., brightfield staining or immunofluorescence (IF)), cell types (e.g., tumor cells, immune cells, tissue cells, etc.), the presence or absence of various organelles such as nuclei or mitochondria, etc. For example, given tissue stained for a single biomarker, repel coding can be configured to (i) code the centers of cells identified by annotation as positive for the biomarker and output a repel coding mask for the positive cells, and (ii) code the centers of cells identified by annotation as negative for the biomarker and output a repel coding mask for the negative cells. The repel coding masks are then used as labels for the original images to effectively train machine learning models to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cellular analysis. When training a machine learning model on images and / or masks coded by proximity coding, the training tends to merge and position nearby cell centers together. On the other hand, when training a machine learning model on images and / or masks coded by repel coding, the training tends to emphasize pixels at the center of cells more (repulsive code labels have stronger pixels at the center of cells) and de-emphasize pixels between cell centers (repulsive code labels have weaker pixels at the periphery of cells). As a result, repel coding emphasizes the reversibility criterion and more effectively trains machine learning models to be able to detect cells.

[0032] Furthermore, the repel coding mask, which is the actual ground truth value of the corresponding image, indicates whether the image contains cells, and if so, where and what characteristics each cell may have, such as biomarker patterns. The machine learning model encodes the image into a feature representation containing the extracted discriminative features of the cells, and then combines the cell features and spatial information with the biomarker staining patterns through a sequence of upconvolution and concatenation with the extracted discriminative features from the feature representation. The machine learning model generates two or more probabilities or segmentation masks for each biomarker in the image based on the combination of the cell features and spatial information and the biomarker staining patterns. The cell centers of the two or more probabilities or segmentation masks are extracted by local maxima detection based on repel coding. A cost function is used to measure the difference or distance between the output probability or segmentation mask and the ground truth value of the corresponding image (i.e., the repel coding mask). The goal of training a machine learning model is to find model parameters, weights, or structure that minimize or maximize the cost function.

[0033] Once trained, the machine learning model can be used in a computer-implemented method for automatically generating segmentation masks for biomarkers in an image. In some cases, the computer-implemented method is performed as part of pre-processing before running an image analysis algorithm to segment and classify target regions (e.g., tumor cells) in an image. In other examples, the computer-implemented method is performed as part of post-processing after running an image analysis algorithm to segment and classify target regions (e.g., tumor cells) in an image. However, as will be understood by those skilled in the art, the concepts described herein are not limited to pre-processing or post-processing procedures, but may be integrated into the overall image analysis process according to various embodiments.

[0034] The computer-implemented method can include using a machine learning model, including a convolutional neural network (CNN) architecture or model that utilizes a two-dimensional segmentation model (modified U-Net) to automatically detect biological structures, such as cells or cell nuclei, and biomarkers, such as PD1, before running a standard image analysis algorithm to learn and recognize target regions. However, the present disclosure is not limited to segmenting only cells, cell nuclei, or biomarkers; the techniques described herein can also be applied to distinguish other cellular organelles, such as ribosomes and mitochondria. The convolutional neural network architecture can be trained using pre-labeled images of different biomarker regions or biomarker-positive and -negative regions. The resulting trained convolutional neural network architecture or model can be used to automatically encode images into feature representations, which can then be masked from full-slide analysis before, during, or after inputting the images into the image analysis algorithm. The feature representations and extracted features are further combined with spatial information of the biomarkers to generate biomarker masks accordingly. The image analysis algorithm can further perform classification tasks and output classification labels for the detected cells or cellular structures. Advantageously, the proposed architecture and techniques can improve the accuracy of cell detection and biomarker classification by image analysis algorithms.

[0035] II. Definition As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a part of the something.

[0036] As used herein, "substantially" and "approximately" mean The terms "approximately," "about," and "approximately" are defined as being largely, but not necessarily fully, specified (and including that which is fully specified), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" can be substituted with "within [percentage]" of what is specified, where percentages include 0.1, 1, 5, and 10%.

[0037] As used herein, the terms "sample," "biological sample," "tissue," or "tissue sample" refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans, veterinary animals such as cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or needle biopsies of tissue), cell samples (such as cytological smears, such as Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components, such as by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucous membranes, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as cheek swabs), or any material containing biomolecules derived from an initial biological sample. In certain embodiments, the term "biological sample," as used herein, refers to a sample prepared from a tumor or portion thereof obtained from a subject (such as a homogenized or liquefied sample).

[0038] As used herein, the terms "biological material," "biological structure," or "cellular structure" refer to naturally occurring materials or structures that comprise all or part of a biological structure (e.g., a cell nucleus, cell membrane, cytoplasm, chromosomes, DNA, cell, cell mass, etc.).

[0039] As used herein, "digital pathology image" refers to a digital image of a stained sample.

[0040] As used herein, the term "cell detection" refers to the detection of a cell or cell structure (eg, a cell nucleus, cell membrane, cytoplasm, chromosome, DNA, cell, cell mass, etc.).

[0041] As used herein, the term "target region" refers to a region of an image containing image data intended to be evaluated in an image analysis process. Target regions include any region, such as a tissue region of an image, intended to be analyzed in the image analysis process (e.g., tumor cells or stain expression).

[0042] As used herein, the term "tile" or "tile image" refers to a single image corresponding to a portion of an entire image or an entire slide. In some embodiments, a "tile" or "tile image" refers to a region of an entire slide scan or a region of interest having (x,y) pixel dimensions (e.g., 1000 pixels by 1000 pixels). For example, consider an entire image divided into M columns of tiles and N rows of tiles. Each tile in the MxN mosaic contains a portion of the entire image; i.e., tiles at locations MI and N contain a first portion of the image, tiles at locations M3 and N4 contain a second portion of the image, and the first and second portions are different. In some embodiments, the tiles can each have the same dimensions (pixel size by pixel size).

[0043] As used herein, the terms "patch," "image patch," or "mask patch" The term "patch" refers to a container of pixels corresponding to a portion of an entire image, entire slide, or entire mask. In some embodiments, a "patch," "image patch," or "mask patch" refers to a region of an image or mask, or a region of interest having (x,y) pixel dimensions (e.g., 256 pixels by 256 pixels). For example, a 1000 pixel by 1000 pixel image divided into 100 pixel by 100 pixel patches will contain 10 patches (each patch containing 1000 pixels). In other embodiments, the patches have (x,y) pixel dimensions and overlap with each "patch," "image patch," or "mask patch" sharing one or more pixels with another "patch," "image patch," or "mask patch."

[0044] III. Generation of digital pathology images Histological staining is widely used to highlight features of interest and enhance the contrast of sectioned tissues or cells in biological samples. For example, stains can be used to mark specific types of cells and / or flag specific types of nucleic acids and / or proteins to aid in microscopy. The stained sample can then be evaluated to determine or estimate the amount of the feature of interest in the sample (e.g., count, density, or expression level) and / or one or more characteristics of the feature of interest (e.g., the location of the feature of interest relative to each other or relative to other features, shape characteristics, etc.). The histological staining process can include several steps, such as fixation, processing, embedding, sectioning, staining, and imaging.

[0045] In some embodiments, immunohistochemical staining of tissue sections (e.g., brightfield staining or IF) is a type of histological staining used to identify the presence of specific proteins in biological samples. For example, the expression level of a specific protein (e.g., an antigen) is determined by (a) performing immunohistochemical analysis of the sample with a specific antibody type, and (b) determining the presence and / or expression level of the protein in the sample. In some embodiments, the immunohistochemical staining intensity is determined relative to a reference sample (e.g., a control cell line staining sample, a tissue sample from a non-cancerous subject, or a reference sample known to have a predetermined level of protein expression).

[0046] Figure 1 shows an example of a histological staining process 100. Stage 110 of the histological staining process 100 involves sample fixation, which can be used to preserve the sample and slow its degradation. In histology, fixation generally refers to an irreversible process that uses chemicals to preserve the chemical composition, preserve the native sample structure, and protect cellular structures from degradation. Fixation can also harden cells or tissues for sectioning. Fixatives can enhance sample and cell preservation by cross-linking proteins. Fixatives can bind and cross-link some proteins and denature others through dehydration, which hardens the tissue and can inactivate enzymes that might otherwise degrade the sample. Fixatives can also kill bacteria.

[0047] Fixatives can be administered, for example, by perfusion or immersion of the prepared sample. Various fixatives can be used, including methanol, buprenorphine fixatives, and / or formaldehyde fixatives, such as neutral buffered formalin (NBF) or paraffin-formalin (paraformaldehyde-PFA). If the sample is a liquid sample (e.g., a blood sample), the sample may be smeared onto a slide and dried before fixation.

[0048] Although the fixation process can help preserve sample and cellular structure for histological examination purposes, fixation can result in the concealment of tissue antigens, thereby reducing antigen detection. Therefore, fixation is generally considered a limiting factor in immunohistochemistry, as formalin can crosslink antigens and mask epitopes. In some cases, fixed samples are treated with citraconic anhydride (a reversible protein crosslinker) to improve detection. Additional processes are performed to reverse the effects of cross-linking, including annealing and heating.

[0049] Stage 120 of the histological staining process 100 involves sample processing and embedding. Sample processing can include infiltrating a fixed sample (e.g., a fixed tissue sample) with an appropriate histological wax, such as paraffin wax. Histological wax may be insoluble in water or alcohol, but soluble in paraffin solvents such as xylene. Therefore, water within the tissue may need to be replaced with xylene. To do so, the sample can first be dehydrated by gradually replacing the water in the sample with alcohol, which can be achieved by passing the tissue through increasing concentrations of ethyl alcohol (e.g., from 0 to approximately 100%). After the water has been replaced by alcohol, the alcohol may be replaced with xylene, which is miscible with alcohol. Embedding can include embedding the sample in warm paraffin wax. Because paraffin wax can be soluble in xylene, molten wax can fill the spaces previously filled by xylene and water. The wax-filled sample can be cooled to form a hardened block that can be clamped to a microtome for cross-sectioning. In some cases, deviations from the above exemplary procedure result in penetration of the paraffin wax, which results in inhibition of penetration of antibodies, chemicals, or other fixatives.

[0050] Stage 130 of the histological staining process 100 involves sectioning the sample. Sectioning is the process of cutting thin slices of a sample from an embedded block in order to mount the sample (e.g., an embedded and fixed tissue sample) on a microscope slide for examination. Sectioning can be performed using a microtome. In some cases, the tissue can be rapidly frozen in dry ice or isopentane and then cut with a cold knife in a refrigerated cabinet (e.g., a cryostat). Other types of coolants, such as liquid nitrogen, can be used to freeze the tissue. Sections for use with brightfield and fluorescence microscopes are generally on the order of 4–10 μm thick. In some cases, the sections can be embedded in epoxy or acrylic resin, which can allow thinner sections (e.g., <2 μm) to be cut. The sections can then be mounted on one or more glass slides. A coverslip can be placed on top to protect the sample sections.

[0051] Stage 140 of the histological staining process 100 involves staining (of a tissue sample or a section of a fixed liquid sample). The purpose of staining is to identify different sample components through a color reaction. Most cells are colorless and transparent. Therefore, it may be necessary to stain histological sections to visualize the cells. The staining process generally involves adding a dye or staining agent to a sample to confirm or quantify the presence of a specific compound, structure, molecule, or feature (e.g., a subcellular feature). For example, staining can help identify or highlight specific biomarkers in a tissue section. In other examples, staining can be used to identify or highlight biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or organelles within individual cells.

[0052] Many staining solutions are aqueous. Therefore, to stain tissue sections, the wax may need to be dissolved and replaced with water (rehydrated) before the staining solution is applied to the section. For example, sections can be passed sequentially through xylene, decreasing concentrations of ethyl alcohol (from about 100% to 0%), and water. Once stained, the sections may be dehydrated again and placed in xylene. The sections can then be mounted on microscope slides in mounting medium dissolved in xylene. A coverslip can be placed on top to protect the sample section. The vaporized xylene around the edges of the coverslip can be removed. This allows the mounting medium to dry and the coverslip to be firmly bonded to the slide.

[0053] Various types of staining protocols can be used to perform staining. For example, an exemplary immunohistochemical staining protocol includes using a hydrophobic barrier around the sample (e.g., tissue section) to prevent leakage of reagents from the slide during incubation, treating the tissue section with reagents to block endogenous sources of nonspecific staining (e.g., enzymes, free aldehyde groups, immunoglobulins, and other unrelated molecules that may mimic specific staining), incubating the sample with a permeabilization buffer to promote penetration of antibodies and other staining reagents into the tissue, incubating the tissue section with a primary antibody at a specific temperature (e.g., room temperature, 6-8 °C) for a certain period of time (e.g., 1-24 hours), rinsing the sample using a wash buffer, incubating the sample (tissue section) with a secondary antibody at another specific temperature (e.g., room temperature) for another period of time, rinsing the sample again using a water buffer, incubating the rinsed sample with a chromogen (e.g., DAB), and washing off the chromogen to stop the reaction. In some cases, a counterstain is then used to distinguish the entire "landscape" of the sample and serve as the primary color reference used to detect tissue targets. Counterstains can include, for example, hematoxylin (blue to purple stain), methylene blue (blue stain), toluidine blue (nuclear deep blue, polysaccharide pink to red stain), nuclear fast red (also known as Kernechtrot dye, red stain), methyl green (green stain), non-nucleogenic stains such as eosin (pink stain), etc. Those skilled in the art will recognize that other immunohistochemical staining techniques can be implemented to perform the staining.

[0054] In another example, an H&E staining protocol can be performed to stain tissue sections. The H&E staining protocol involves applying hematoxylin stain mixed with a metal salt or mordant to the sample. The sample can then be rinsed with a weak acid solution to remove excess stain (differentiation), followed by bluing in weak alkaline water. After application of hematoxylin, the sample can be counterstained with eosin. It will be understood that other H&E staining techniques can be implemented.

[0055] In some embodiments, various types of stains can be used to stain the target feature. For example, DAB can be used on various tissue sections for IHC staining, producing a brown color that displays the feature of interest in the stained image. In another example, alkaline phosphatase (AP) can be used on skin tissue sections for IHC staining because the DAB color can be masked by melanin pigments. Regarding primary staining techniques, applicable stains can include, for example, basophilic and eosinophilic stains, hematin and hematoxylin, silver nitrate, trichrome stains, etc. Acidic dyes can react with cationic or basic components in tissues or cells, such as proteins and other components in the cytoplasm. Basic dyes can react with anionic or acidic components in tissues or cells, such as nucleic acids. As mentioned above, one example of a staining system is H&E. Eosin can be a negatively charged pink acidic dye, and hematoxylin can be a purple or blue basic dye containing hematein and aluminum ions. Other examples of stains include periodic acid-Schiff (PAS) stain, Masson's trichrome, Alcian blue, van Gieson, reticulin stain, etc. In some embodiments, different types of stains can be used in combination.

[0056] Stage 150 of the histological staining process 100 involves medical imaging. A microscope (e.g., an electron microscope or an optical microscope) can be used to magnify the stained sample. For example, an optical microscope has a resolution of less than 1 μm, such as on the order of several hundred nanometers. Electron microscopes can be used to observe finer details in the nanometer or subnanometer range. An imaging device (combined with or separate from the microscope) images the magnified biological sample to obtain image data, such as a multichannel image (e.g., multichannel fluorescence) having several (e.g., between 10 and 16) channels. Imaging devices include, but are not limited to, cameras (e.g., analog cameras, digital cameras, etc.), optical systems (e.g., one or more lenses, a sensor-focusing lens group, a microscope objective, etc.), imaging sensors (e.g., charge-coupled devices (CCDs), complementary metal-oxide semiconductor (CMOS) image sensors, etc.), photographic film, etc. In digital embodiments, the imaging device can include multiple lenses that cooperate to demonstrate on-the-fly focusing. An image sensor, e.g., a CCD sensor, can capture a digital image of the specimen. In some embodiments, the imaging device is a bright-field imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. The imaging device can capture images using invisible electromagnetic radiation (e.g., UV light) or other imaging techniques. For example, the imaging device may include a microscope and a camera configured to capture an image magnified by the microscope. The image data received by the analysis system may be identical to and / or derived from the raw image data captured by the imaging device.

[0057] In step 160, images of the stained sections are stored. The images can be stored locally, remotely, and / or on a cloud server. Each image may be stored in association with a subject identifier and a date (e.g., the date the sample was collected and / or the date the image was taken). The images may also be transmitted to another system (e.g., a system associated with a pathologist or an automated or semi-automated image analysis system).

[0058] It will be understood that modifications to process 100 are contemplated. For example, if the sample is a liquid sample, step 120 (processing and embedding) and / or step 130 (sectioning) may be omitted from the process.

[0059] IV. Exemplary Process Flow for Digital Pathology Image Conversion 2 shows a block diagram illustrating a computing environment 200 for processing digital pathology images using a machine learning model. As described further herein, processing the digital pathology image may include training a machine learning algorithm using the digital pathology image or converting some or all of the digital pathology image into one or more results using a trained (or partially trained) version of a machine learning algorithm (i.e., a machine learning model).

[0060] As shown in FIG. 2, the computing environment 200 includes several stages: an image storage stage 205 , a preprocessing stage 210 , a labeling stage 215 , a training stage 220 , and a result generation stage 225 .

[0061] The image storage stage 205 includes one or more image data stores 230 that are accessed (e.g., by the pre-processing stage 210) to provide a set of digital images 235 of pre-selected regions from a biological specimen slide (e.g., a histology slide) or from the entire biological specimen slide. Each digital image 235 stored in each image data store 230 and accessed in the image storage stage 210 can include a digital pathology image generated according to some or all of the process 100 shown in FIG. 1. In some embodiments, each digital image 235 includes image data from one or more scanned slides. Each of the digital images 235 includes underlying image data corresponding to a single specimen and / or image. The image data can correspond to image data from the same day that the image data was collected.

[0062] The image data can include the image, as well as any information about the color or wavelength channels, as well as details about the imaging platform on which the image was generated. For example, a tissue section may need to be stained by applying a staining assay containing one or more different biomarkers associated with a chromogenic stain for brightfield imaging or a fluorophore for fluorescent imaging. The staining assay can use a chromogenic stain for brightfield imaging, an organic fluorophore, quantum dots, or an organic fluorophore together with quantum dots for fluorescent imaging, or any other combination of stain, biomarker, and observation or imaging device. Examples of biomarkers include estrogen receptor (ER), human epidermal growth factor receptor 2 (HER2), human Ki-67 protein, progesterone receptor (PR), programmed cell death protein 1 ("PD1"), etc., where the tissue section is detectably labeled with a respective binding agent (e.g., antibody) for ER, HER2, Ki-67, PR, PD1, etc. In some embodiments, digital image and data analysis operations such as classification, scoring, Cox modeling, and risk stratification depend on the type of biomarker used and the field of view (FOV) selection and annotation. Furthermore, typical tissue sections are processed in an automated staining / assay platform that applies a staining assay to the tissue section, resulting in a stained sample. Various commercially available products suitable for use as staining / assay platforms exist on the market, one example being the VENTANA SYMPHONY product from the assignee, Ventana Medical Systems, Inc. The stained tissue sections can be fed into an imaging system, for example, a microscope or a whole slide scanner with a microscope and / or imaging components, one example being the VENTANA iScan Coreo product from the assignee, Ventana Medical Systems, Inc. Multiple tissue slides can be scanned using an equivalent multiplexed slide scanner system.Additional information provided by the imaging system can include any information regarding the staining platform, including the concentration of chemicals used for staining, the reaction time of the chemicals applied to the tissue in the stain, and / or the pre-analysis conditions of the tissue, such as the age of the tissue, fixation method, duration, how the sections were embedded, cut, etc.

[0063] In the preprocessing stage 210, each, multiple, or all of the set of digital images 235 are preprocessed using one or more techniques to generate corresponding preprocessed images 240. Preprocessing can include cropping the image. In some cases, preprocessing can further include standardizing or rescaling (e.g., normalizing) all features to the same scale (e.g., the same size scale or the same color or saturation scale). In a particular example, the image is resized to a minimum size (width or height) of a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels), optionally maintaining the original aspect ratio. Preprocessing may further include removing noise. For example, the image may be smoothed to remove unwanted noise, such as by applying a Gaussian function or a Gaussian blur.

[0064] The preprocessed images 240 can include one or more training images, validation input images, and unlabeled images. It should be understood that the preprocessed images 240 corresponding to the training group, validation group, and unlabeled group do not need to be accessed simultaneously. For example, an initial set of training and validation preprocessed images 240 may be initially accessed and used to train the machine learning algorithm 255, and unlabeled input image elements can be subsequently accessed or received (e.g., at one or more subsequent times) and used by the trained machine learning model 260 to provide a desired output (e.g., cell classification).

[0065] In some cases, the machine learning algorithm 255 is trained using supervised training, and some or all of the preprocessed image 240 is partially or fully labeled in the labeling stage 215, manually, semi-automatically, or automatically, with labels 245 that identify the "correct" (i.e., "ground truth") interpretation of various biological materials and structures within the preprocessed image 240. For example, the labels 245 may identify (for example) features of interest, cellular classifications, binary indications as to whether a given cell is a particular type of cell, binary indications as to whether the preprocessed image 240 (or a particular region of the preprocessed image 240) contains a particular type of indication (e.g., necrosis or artifact), categorical features of slide-level or region-specific indications (e.g., identifying a particular type of cell), counts (e.g., identifying the amount of a particular type of cell within a region, the amount of indicated artifact, or the amount of necrotic area), the presence or absence of one or more biomarkers, etc. In some cases, the labels 245 include location. For example, the label 245 may identify the point location of the nucleus of a particular type of cell or the point location of a particular type of cell (e.g., a raw dot label). As another example, the label 245 may include the edge or boundary of a displayed tumor, blood vessel, necrotic region, etc. As another example, the label 245 may include one or more biomarkers identified based on biomarker patterns observed using one or more stains. For example, a tissue slide stained for a biomarker, such as programmed cell death protein 1 ("PD1"), can be observed and / or processed to label cells as either positive or negative cells, taking into account the expression level and pattern of PD1 in the tissue. Depending on the feature of interest, a given labeled preprocessed image 240 can be associated with a single label 245 or multiple labels 245. In the latter case, each label 245 may be associated with (for example) an indication as to which location or portion within the preprocessed image 245 the label corresponds to.

[0066] The labels 245 assigned in the labeling stage 215 can be identified based on input from a human user (e.g., a pathologist or image scientist) and / or an algorithm (e.g., an annotation tool) configured to define the labels 245. In some cases, the labeling stage 215 can include transmitting and / or presenting some or all of the one or more preprocessed images 240 to a computing device operated by a user. In some cases, the labeling stage 215 can include utilizing an interface (e.g., using an API) presented by the labeling controller 250 on the computing device operated by the user, the interface including an input component for accepting input identifying labels 245 for features of interest. For example, a user interface may be provided by the labeling controller 250 that enables selection of an image or a region of an image (e.g., FOV) for labeling. A user operating a terminal can select an image or FOV using the user interface. Several image or FOV selection mechanisms can be provided, such as specifying a known or irregular shape or defining an anatomical region of interest (e.g., tumor region). In one example, the image or FOV is an entire tumor region selected on an IHC slide stained with a combination of hematoxylin and eosin (H&E) staining. The selection of the image or FOV can be performed by a user or by an automated image analysis algorithm, such as tumor region segmentation on an H&E tissue slide. For example, the user can select that the image or FOV be the entire slide or the entire tumor, or that the entire slide or the entire tumor region be automatically designated as the image or FOV using a segmentation algorithm. The user operating the terminal can then select one or more labels 245 to be applied to the selected image or FOV, such as point locations on cells, positive markers for biomarkers expressed by cells, negative biomarkers for biomarkers not expressed by cells, or a boundary around the cells.

[0067] In some cases, the interface can identify which and / or how many specific labels 245 are required, which can be communicated to the user via (e.g.,) text instructions and / or visualizations. For example, a specific color, size, and / or symbol can indicate that a label 245 is required for a particular representation within an image (e.g., a particular cell or region or staining pattern) relative to other representations. If labels 245 corresponding to multiple representations are required, the interface can identify each of the representations simultaneously or sequentially (such that providing a label for one identified representation triggers the identification of the next representation for labeling). In some cases, each image is presented until the user identifies a specific number of labels 245 (e.g., of a particular type). For example, a given whole slide image or a given patch of a whole slide image can be presented until the user identifies the presence or absence of three different biomarkers, at which point the interface can present a different whole slide image or a different patch image (e.g., until a threshold number of images or patches have been labeled). Thus, in some cases, the interface is configured to request and / or accept labels 245 of an incomplete subset of features of interest, allowing the user to decide which of the potentially many representations will be labeled.

[0068] In some cases, the labeling stage 215 includes a labeling controller 250 that implements an annotation algorithm to semi-automatically or automatically label various features of an image or a region of interest within the image. For example, intensities across the image (e.g., a preprocessed image) can be normalized or regularized, intensities can be thresholded or filtered, and / or an algorithm (e.g., configured to detect objects, lines, and / or shapes, such as Equation (1)) can be applied. Each boundary, point location, or location coding can then be identified as a feature of interest. In some cases, a metric is associated with each feature of interest (e.g., a metric indicating the confidence in localizing the feature of interest), and the interest level can be scaled based on the metric. According to aspects of the present disclosure, the annotation algorithm can be configured to use the coding scheme to localize the feature of interest (e.g., to localize where exactly an object is located within the image). Localizing the feature of interest can include predicting the point location and / or boundary of the feature of interest. For example, localization may include identifying or predicting a point location corresponding to each displayed cell and / or identifying or predicting a closed shape corresponding to each displayed cell. In a particular example, the localization technique includes coding the raw dot labels into a mask (e.g., labels 245 of the locations of objects such as cells) using a repel coding scheme. The repel coding scheme is defined as follows:

number

number

number

number

number

number

[0069] The label masks can be generated using a repel coding scheme in combination with annotations for each biomarker. For example, the localization technique can include coding raw dot labels into an initial cell localization mask (e.g., label 245 for cell location) using repel coding defined in Equation (1), segmenting the initial cell localization mask using label annotations of one or more biomarkers based on the biomarker staining pattern, and outputting two or more repel coding masks for each biomarker (e.g., one mask for positive cells and one mask for negative cells). The segmentation is instance segmentation, and along with pixel-level classification for localization, the algorithm further classifies each instance of a class separately (e.g., positive cells for PD1, negative cells for PD1, positive cells for HER2, negative cells for HER2, etc.). For example, given tissue stained for a single biomarker, the repel coding scheme can be configured to (i) code the centers of cells identified by the annotation as positive for the biomarker and output a repel coding mask of positive cells, and (ii) code the centers of cells identified by the annotation as negative for the biomarker and output a repel coding mask of negative cells. As another example, given tissue stained for two biomarkers, the repel coding can be configured to (i) code the centers of cells identified by the annotation as positive for the first biomarker and output a repel coding mask of positive cells, (ii) code the centers of cells identified by the annotation as negative for the first biomarker and output a repel coding mask of negative cells, (iii) code the centers of cells identified by the annotation as positive for the second biomarker and output a repel coding mask of positive cells, and (iv) code the centers of cells identified by the annotation as negative for the second biomarker and output a repel coding mask of negative cells.As another example, given tissue stained for a single biomarker, repel coding can be configured to (i) code the centers of cells identified by annotation as positive for the biomarker and output a repel coding mask of positive cells (e.g., tumor cells), (ii) code the centers of cells identified by annotation as negative for the biomarker and output a repel coding mask of negative cells (e.g., immune cells), and (iii) code the centers of cells identified by annotation as tissue cells and output a repel coding mask of tissue cells.

[0070] In some cases, the labeling controller 250 may use input or annotations from a user. The labeling controller 250 annotates the image or FOV on the first slide according to a algorithm and maps the annotations to the remainder of the slide. Depending on the defined FOV, several methods for annotation and registration are possible. For example, the entire tumor region annotated on the H&E slide from among multiple consecutive slides may be selected automatically or by the user on an interface such as VIRTUOSO / VERSO™. Because the other tissue slides correspond to serial sections from the same tissue block, the labeling controller 250 performs a marker-to-marker registration operation to map and transfer the entire tumor annotation from the H&E slide to each of the remaining IHC slides in the series. Exemplary methods for marker-to-marker registration are described in further detail in commonly assigned, co-pending International Publication No. WO 2014140070, filed March 12, 2014, entitled "Whole slide image registration and cross-image annotation devices, systems and methods," which is incorporated herein by reference in its entirety for all purposes. In some embodiments, any other method for image registration and generation of whole tumor annotations may be used. For example, a qualified reader, such as a pathologist, can annotate all tumor regions on any other IHC slide and execute labeling controller 250 to map all tumor annotations on other digitized slides. For example, a pathologist (or an automated detection algorithm) can annotate all tumor regions on an H&E slide and trigger analysis of all adjacent serially sectioned IHC slides to determine a whole-slide tumor score for the annotated regions on all slides.

[0071] During the training phase 220, the labels 245 and the corresponding preprocessed images 240 can be used by the training controller 265 to train a machine learning algorithm 255. In some cases, the machine learning algorithm 255 includes a CNN, a modified CNN with the encoding layer replaced by a residual neural network (“Resnet”), or a modified CNN with the encoding and decoding layers replaced by a Resnet. In other examples, the machine learning algorithm 255 can be any suitable machine learning algorithm configured to localize, classify, and / or analyze the preprocessed images 240, such as a two-dimensional CNN (“2DCNN”), a Mask R-CNN, a feature pyramid network (FPN), a dynamic time warping (“DTW”) technique, a hidden Markov model (“HMM”), or the like, or a combination of one or more of such techniques, e.g., a CNN-HMM or an MCNN (multiscale convolutional neural network). The computing environment 200 can use the same type of machine learning algorithm or different types of machine learning algorithms trained to detect and classify different cells. For example, the computing environment 200 may include a first machine learning algorithm (e.g., U-Net) for detecting and classifying PD1. The computing environment 200 may also include a second machine learning algorithm (e.g., 2DCNN) for detecting and classifying cluster of differentiation 68 ("CD68"). The computing environment 200 may also include a third machine learning algorithm (e.g., U-Net) for combined detection and classification of PD1 and CD68. The computing environment 200 may also include a fourth machine learning algorithm (e.g., HMM) for diagnosing disease for treatment or prognosis of a subject, such as a patient. In other examples consistent with the present disclosure, still other types of machine learning algorithms may be implemented.

[0072] In some embodiments, the training phase includes a parameter data store and a trainer controller that are collectively used to train the machine learning algorithm 255 based on training data (e.g., the labels 245 and corresponding preprocessed images 240) and optimize parameters of the machine learning algorithm 255 during supervised or unsupervised training. In some cases, the training process includes iterative operations to learn a set of parameters (e.g., one or more coefficients and / or weights) of the machine learning algorithm 255. Each parameter is a parameter The cost function or objective function can be an adjustable variable such that the value of the metric is adjusted during training. For example, the cost function or objective function can be configured to optimize accurate classification of displayed representations, optimize characterization of a given type of feature (e.g., characterizing shape, size, uniformity, etc.), optimize detection of a given type of feature, and / or optimize accurate localization of a given type of feature. Each iteration can include learning a set of parameters for the machine learning algorithm 255 that minimizes or maximizes a cost function of the machine learning algorithm 255, such that the value of the cost function using a set of parameters is less than or greater than the value of the cost function using a different set of parameters in a previous iteration. The cost function can be constructed to measure the difference between the predicted output using the machine learning algorithm 255 and the labels 245 included in the training data. Once the set of parameters is identified, the machine learning algorithm 255 can be trained and utilized for the designed localization and / or classification.

[0073] The training iterations continue until a stopping condition is met. The training completion condition can be configured to be met (for example) when a predetermined number of iterations are completed, when a statistical value generated based on testing or validation exceeds a predetermined threshold (e.g., a classification accuracy threshold), when a statistical value generated based on a confidence metric (e.g., an average or median confidence metric, or a percentage of confidence metrics above a particular value) exceeds a predetermined confidence threshold, and / or when a user device involved in the training review closes the training application executed by the training controller 265. In some cases, a new training iteration can be initiated in response to receiving a corresponding request or trigger condition from a user device (e.g., drift is determined within the trained machine learning model 260).

[0074] The trained machine learning model 260 can then be used (in the result generation stage 225) to process the new preprocessed image 240 to generate predictions or inferences, such as predicting cell center and / or location probabilities, classifying cell types, generating cell masks (e.g., segmentation masks for each pixel of the image), predicting a disease diagnosis or prognosis for a subject, such as a patient, or a combination thereof. In some cases, the masks display the locations of indicated cells associated with one or more biomarkers. For example, given tissue stained for a single biomarker, the trained machine learning model 260 can be configured to (i) infer cell centers and / or locations, (ii) classify the cells based on features of the staining pattern associated with the biomarker, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cells. As another example, given tissue stained for two biomarkers, the trained machine learning model 260 can be configured to (i) infer cell centers and / or locations, (ii) classify the cells based on features of the staining patterns associated with the two biomarkers, and (iii) output a cell detection mask for cells positive for the first biomarker, a cell detection mask for cells negative for the first biomarker, a cell detection mask for cells positive for the second biomarker, and a cell detection mask for cells negative for the second biomarker. As another example, given tissue stained for a single biomarker, the trained machine learning model 260 can be configured to (i) infer cell centers and / or locations, (ii) classify the cells based on features of the cells and the staining patterns associated with the biomarker, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cell codes, as well as mask cells classified as tissue cells.

[0075] In some cases, analysis controller 280 generates analysis results 285 that are made available to the entity that requested the processing of the underlying image. Analysis results 285 may include a mask output from trained machine learning model 260 overlaid on new preprocessed image 240. Additionally or alternatively, analysis results 285 may include a whole-slide tumor score, etc. The analysis results may include information calculated or determined from the output of a trained machine learning model from the assignee, VENTANA. In an exemplary embodiment, the automated analysis of the tissue slides uses an FDA-cleared, 510(k)-approved algorithm from the assignee, VENTANA. Alternatively, or additionally, any other automated algorithm may be used to analyze selected regions of an image (e.g., a masked image) and generate a score. In some embodiments, analysis controller 280 may further respond to instructions received from a pathologist, physician, investigator (e.g., associated with a clinical trial), patient, medical professional, etc., received from the computing device. In some cases, the communication from the computing device includes an identifier for each subject in the set in response to a request to perform an iteration of the analysis for each subject represented in the particular subject set. The computing device may further perform an analysis based on the machine learning model and / or the output of analysis controller 280 and / or provide a recommended diagnosis / treatment for the subject.

[0076] It will be understood that computing environment 200 is exemplary and that computing environments 200 having different stages and / or using different components are contemplated. For example, in some cases, a network may omit pre-processing stage 210 such that the images used to train an algorithm and / or processed by a model are raw images (e.g., from an image data store). As another example, it will be understood that each of pre-processing stage 210 and training stage 220 may include a controller for performing one or more operations described herein. Similarly, while labeling stage 215 is shown in relation to labeling controller 250 and result generation stage 225 is shown in relation to analysis controller 280, the controllers associated with each stage may additionally or alternatively facilitate other operations described herein other than generating labels and / or generating analysis results. 2 lacks displayed representations of devices associated with a programmer (e.g., who selected the architecture of the machine learning algorithm 255, which defined how various interfaces would function, etc.), devices associated with a user providing an initial label or label review (e.g., in the labeling stage 215), and devices associated with a user requesting model processing of a given image (which may be the same user as or a different user from the user who provided the initial label or label review). Despite the lack of representations of these devices, the computing environment 200 may include use of one, more, or all of the devices, and may in fact include use of multiple devices associated with a corresponding number of users providing initial labels or label reviews and / or a corresponding number of users requesting model processing of various images.

[0077] V. Techniques for Automated Object Localization and Classification Automated instance segmentation (e.g., cell localization and classification) of digital images provides meaningful information for image analysis in the biomedical industry and plays an important role in disease diagnosis and treatment. For example, automated localization and classification of programmed cell death protein 1 ("PD1")-positive and -negative cells based on the expression level and pattern of PD1 in a biological sample can be used to determine the complex immunological status and disease diagnosis of a sample subject. The accuracy of the generated segmentation mask of an object such as a cell depends on the corresponding machine learning model, and the performance of the machine learning model depends on the training and architecture of the machine learning algorithm. In various embodiments, the performance of the machine learning model is optimized by training the machine learning algorithm using images labeled with masks generated by a repel coding scheme that takes into account the various biomarker staining patterns observed in the image. Furthermore, in some embodiments, the performance of the machine learning model is improved not only by localizing objects but also by using residual blocks with skip connections to capture complex features for multi-class segmentation. It is optimized by using a modified U-Net architecture.

[0078] An exemplary repel coding scheme for VA object localization and classification FIG. 3 illustrates a flowchart depicting a process 300 for implementing a repel coding scheme to generate repel coding masks for use in training machine learning algorithms, according to various embodiments. The process 300 depicted in FIG. 3 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The process 300 presented in FIG. 3 and described below is intended to be exemplary and non-limiting. While FIG. 3 depicts various processing steps performed in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, steps may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, such as those depicted in FIGS. 1 and 2, the process depicted in FIG. 3 can be performed by a preprocessing or labeling subsystem (e.g., computing environment 200) to generate repel coding masks for training by one or more machine learning algorithms (e.g., machine learning algorithm 255).

[0079] Process 300 begins at block 310, where an image of a biological sample is acquired or accessed by a computing device (e.g., labeling controller 250 of computing environment 200 described with reference to FIG. 2). In some cases, the image displays cells having a staining pattern associated with a biomarker. In other examples, the image displays cells having multiple staining patterns associated with multiple biomarkers. The image is annotated with raw dot labels. The raw dot labels provide information including (i) cell center and (ii) expression of one or more biomarkers. As shown in FIG. 4, image 400 includes raw dot labels 405 positioned at the center of each cell, and the color, symbol, pattern, etc. of the raw dot labels 405 indicate the expression of one or more biomarkers. An "x" label 405 indicates that the cell is negative for (does not express) the biomarker. A "+" label 405 indicates that the cell is positive for (expresses) the biomarker. As described with reference to FIG. 2, the raw dot labels 405 may be annotated by a user or automatically by an annotation tool during a preprocessing stage.

[0080] In block 320, the image is coded into an initial cell localization mask using a repel coding algorithm. The repel coding algorithm is defined in Equation (1). The repel coding algorithm calculates the distance between a pixel and its nearest cell center label and the distance between the pixel and its second nearest cell center label. For each pixel, a pseudo-distance is calculated based on the distance between the pixel and its nearest cell center label and the distance between the pixel and its second nearest cell center label. If the pseudo-distance is greater than or equal to a threshold, the corresponding pixel is assigned a value of 0. Otherwise, a non-zero value is assigned to the pixel according to the pseudo-distance. FIG. 5 shows an image 500 (e.g., image 400 described with reference to FIG. 4) being coded by the repel coding algorithm, which outputs an initial cell localization mask 505 including a cell center 510 (represented by high-intensity pixels) and a periphery 515 represented by response decay (low-intensity pixels) away from the cell center 510.

[0081] In block 330, the initial cell localization mask is segmented based on the biomarker staining pattern using raw dot labels. The segmentation is performed using a segmentation algorithm. The segmentation is performed using instance segments. In addition to the pixel-level classification for localization, the segmentation algorithm further classifies each instance of a class separately (e.g., positive cells for PD1, negative cells for PD1, positive cells for HER2, negative cells for HER2, etc.). The segmentation algorithm can implement any known instance segmentation technique known in the art. For example, in some cases, the segmentation algorithm is ruled based on biomarker information provided by the raw dot labels to segment the cell localization mask. In other examples, the segmentation algorithm is cluster-based, such as k-means, to segment the cell localization mask based on biomarker information provided by the raw dot labels. Figure 5 shows that the segmented initial cell localization mask 505 is segmented to generate a segmented cell localization mask 520, in which each pixel belongs to a specific class (either background 525 or cell 530) based on a repel coding algorithm, and a specific biomarker class (either negative for biomarker 535 or positive for biomarker 540) is assigned to each pixel of the image based on the segmentation algorithm. It should be understood that the order of blocks 320 and 330 can be reversed if the segmentation algorithm is first applied to the image to generate a segmentation mask, and then the repel coding algorithm is applied to the segmentation mask to generate a segmented cell localization mask.

[0082] In block 340, the segmented cell localization mask is divided into two or more repulsion coding masks. Essentially, each class represented in the segmented cell localization mask is divided into its own separate repulsion coding mask. For example, if the segmented cell localization mask has two classes: (1) cells positive for the biomarker and (2) cells negative for the biomarker, the segmented cell localization mask is divided into two repulsion masks: (1) one mask for cells positive for the biomarker and (2) another mask for cells negative for the biomarker. The division may be performed using an image processing algorithm that hides or removes various classes, selectable as parameters. For example, continuing with the above example, a mask for cells positive for the biomarker can be generated by hiding or removing cells negative for the biomarker from the segmented cell localization mask. Meanwhile, a mask for cells negative for the biomarker can be generated by hiding or removing cells positive for the biomarker from the segmented cell localization mask. FIG. 5 shows that the segmented cell localization mask 520 is split into two repel coding masks: (1) one mask 545 for cells positive for the biomarker, and (2) another mask 550 for cells negative for the biomarker.

[0083] An example U-Net for VB instance segmentation The modified U-Net model can be used in a biomarker segmentation method to individually extract complex features from an input image (e.g., one or more images of a biological sample) and generate a high-resolution two-dimensional segmentation mask. As shown in Figure 6, the modified U-Net model 600 can include an encoder 605 and a decoder 610, which provides a U-shaped architecture. The encoder 605 is a CNN network that includes repeated application of convolutions (e.g., 3x3 convolutions (unpadded convolutions)), each with one or more layers comprising a residual block with a skip connection followed by a rectified linear unit (ReLU), and a max-pooling operation for downsampling (e.g., 2x2 max-pooling with stride 2). One or more layers of the encoder 605 include a residual block with a skip connection. With the help of the residual block, the original input of the convolution is also added to the output of the convolution. At each downsampling step or In the pooling operation, the number of feature channels can be doubled. During encoding, the spatial information of the image data is reduced and the feature information is increased. The decoder 610 is a CNN network that combines the features and spatial information from the encoder 605 (upsampling the feature maps from the encoder 605). The upsampling of the feature maps is followed by an upconvolution (upsampling operator) that halves the number of channels, concatenation with the corresponding cropped feature maps from the encoder 605, repeated application of convolutions (e.g., two 3x3 convolutions), each followed by a rectified linear unit (ReLU), and a final convolution (e.g., a 1x1 convolution) to generate a two-dimensional segmentation mask. It should be understood that one or more layers of the decoder 610 can also include residual blocks with skip connections. For localization, the high-resolution features from the encoder 605 are combined with the upsampled output from the decoder 610.

[0084] Techniques for training machine learning algorithms for VC instance segmentation FIG. 7 shows a flowchart illustrating a process 700 for training a machine learning algorithm (e.g., a modified U-Net) to localize and classify cells based on biomarker expression patterns, according to various embodiments. The process 700 shown in FIG. 7 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The process 700 presented in FIG. 7 and described below is intended to be exemplary and non-limiting. While FIG. 7 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, steps may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, such as those shown in Figures 1 and 2, the process shown in Figure 7 can be performed as part of a training phase (e.g., algorithm training 220) to generate a prediction or segmentation mask that includes pixel-level classifications that localize cells (e.g., background and cell) and pixel-by-pixel biomarker expression classifications for each instance of a class (e.g., cell) that classifies cells based on biomarker expression.

[0085] Process 700 begins at block 705, where a training image of a biological sample (e.g., preprocessed image 240 of computing environment 200 described with respect to FIG. 2) is acquired or accessed by a computing device. In some cases, the training image displays cells having a staining pattern associated with a biomarker. In other examples, the training image displays cells having multiple staining patterns associated with multiple biomarkers. The training image is annotated with a repel coding mask (e.g., a repel coding mask generated according to process 300 described with respect to FIG. 3). The repel coding mask provides information about the training image, including (i) cell location and (ii) classification of the cells with respect to one or more biomarkers.

[0086] At block 710, one or more images and mask labels are divided into image patches of a predetermined size. For example, images typically have random sizes, and machine learning algorithms such as modified CNNs learn better with normalized image sizes; therefore, training images can be divided into image patches having specific sizes to optimize training. In some embodiments, training images are divided into image patches having a predetermined size of 64 pixels by 64 pixels, 128 pixels by 128 pixels, 256 pixels by 256 pixels, or 512 pixels by 512 pixels. The division of training images may be performed before the repel coding mask is generated, so that the subsequently generated repel coding mask can be optimized. has the same size as the size of the corresponding training image. Alternatively, the segmentation may be performed after the repel coding mask is generated. In such a case, the repel coding mask is preprocessed and segmented into mask patches of the same predetermined size as the training images before, after, or simultaneously with the segmentation of the training images.

[0087] In block 715, a machine learning algorithm is trained on the image patch. In some cases, the machine learning algorithm is a modified U-Net including an encoder and a decoder, where one or more layers of the encoder include a residual block with skip connections (or a ResNet). Training can include performing iterative operations to find a set of parameters for the machine learning algorithm that minimizes or maximizes a cost function of the machine learning algorithm. The output from the iterations is a prediction or segmentation mask that includes a pixel-level classification (e.g., background and cell) that localizes cells and a per-pixel biomarker expression classification for each instance of a class (e.g., cell) that classifies cells based on biomarker expression. Each iteration involves finding a set of parameters for the machine learning algorithm such that the value of the cost function using the set of parameters is smaller or larger than the value of the cost function using another set of parameters in the previous iteration. The cost or objective function is constructed to measure the difference between (i) the predicted pixel-level classification and predicted biomarker expression classification for each pixel and (ii) the ground truth repel coding mask. In certain cases, the cost function is a binary cross-entropy loss function.

[0088] In some cases, training further includes adjusting the learning rate of the machine learning algorithm by maximizing or minimizing the learning rate according to a predetermined schedule, which may be a step-decay schedule that decreases the learning rate by a predetermined factor every predetermined number of epochs to optimize the cost function.

[0089] The training iterations continue until a stopping condition is met. The training completion condition can be configured to be met (for example) when a predetermined number of iterations are completed, when a statistical value generated based on testing or validation exceeds a predetermined threshold (e.g., a classification accuracy threshold), when a statistical value generated based on a confidence metric (e.g., an average or median reliability metric, or the percentage of reliability metrics above a particular value) exceeds a predetermined confidence threshold, and / or when a user device involved in the training review closes the training application. As a result of the training, the machine learning algorithm has learned, for each instance of a class (e.g., cell) that classifies cells based on biomarker expression, a pixel-level classification (e.g., background and cell) that localizes the cell and a nonlinear relationship within the image that the machine learning algorithm uses to predict the biomarker expression classification for each pixel. The training output includes a trained machine learning model having a learned set of parameters associated with the nonlinear relationship that derived the minimum or maximum cost function value from all iterations.

[0090] The trained machine learning model is provided at block 720. For example, the trained machine learning model can be deployed for execution in an image analysis environment, as described with respect to FIG.

[0091] Techniques for instance segmentation using VD machine learning models FIG. 8 shows a flowchart illustrating a process 800 of using a machine learning model (e.g., a modified U-Net model) to localize and classify cells based on biomarker expression patterns, according to various embodiments. The process 800 shown in FIG. 8 may be implemented by software (e.g., code, instructions) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be implemented in a non-transitory storage medium (e.g., a memory device). The process 800 presented in FIG. 8 and described below is intended to be exemplary and non-limiting. While FIG. 8 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, such as those depicted in FIGS. 1 and 2, the process depicted in FIG. 8 may be performed as part of a result generation stage (e.g., result generation 280) to generate a prediction or segmentation mask that includes pixel-level classifications (e.g., background and cell) that localize cells and pixel-by-pixel biomarker expression classifications for each instance of a class (e.g., cell) that classifies cells based on biomarker expression.

[0092] Process 800 begins at block 805, where one or more images of a biological sample are accessed or acquired. The images display cells containing staining patterns of one or more biomarkers. In certain cases, the one or more images display cells containing a biomarker and a staining pattern of another biomarker. As described with respect to FIG. 1, the images can be preprocessed by immunochemical staining techniques (e.g., IF) so that specific proteins and organelles in the biological sample are visible for processing and analysis in an analytical system. In some embodiments, the images are stained using multiple stains or binders, such as antibodies, so that information about different biomarkers can be reported under multichannel analysis or similar techniques.

[0093] At block 810, one or more images may be divided into image patches of a predetermined size. For example, images typically have random sizes, and machine learning algorithms such as modified CNNs learn better with normalized image sizes; therefore, images may be divided into image patches having specific sizes to optimize analysis. In some embodiments, images are divided into image patches having a predetermined size of 64 pixels by 64 pixels, 128 pixels by 128 pixels, 256 pixels by 256 pixels, or 512 pixels by 512 pixels.

[0094] In block 815, one or more images or image patches are input to a machine learning model for further analysis. In some cases, the machine learning model is a modified CNN (e.g., U-Net) model including an encoder and a decoder, where one or more layers of the encoder include a residual block with skip connections (or ResNet). The machine learning model further includes parameters learned from training images and corresponding label masks for each biomarker in the training images (as described in detail with respect to computing environment 200 and process 700). The label masks were generated using a repel coding scheme in combination with labels for each of the biomarkers (as described in detail with respect to computing environment 200 and process 300). As described with respect to FIG. 3, generating the label masks includes (i) coding cells in each training image using repel coding, where the coding includes cell centers and peripheries represented by response decay away from the cell centers, and (ii) generating two or more label masks for each of the biomarkers in the images based on the coding and the respective labels of the biomarkers.

[0095] In block 820, one or more images or image patches are encoded into a feature representation that includes the discriminative features extracted by the provided machine learning model. The discriminative features (e.g., lower resolution features) can be associated with biological materials or biological structures, such as cells. The machine learning model can be used to generate a feature representation for multiple different levels within multiple different sub-networks. The encoding of images or image patches into discriminatory features can be performed in a network, where each sub-network is associated with the expression (e.g., positive or negative) of at least one biomarker.

[0096] In block 825, the extracted discriminative features are combined with the cell features and spatial information and the biomarker staining patterns within the image or image patch through a sequence of upconvolution and concatenation with the discriminative features. Combination can be performed by projecting the extracted discriminative features into pixel space (e.g., higher resolution) and classifying each pixel space. The machine learning model decoder can perform the combination and projection of the extracted discriminative features at multiple different levels within multiple different subnetworks. In certain cases, the multiple different levels perform upsampling (i.e., expanding the feature dimensions to the original size of the input image patch) and concatenation, followed by periodic convolution operations to project the extracted discriminative features. Classification includes cell detection and classification of cells based on biomarker staining patterns. Specifically, classification includes pixel-level classification (e.g., background and cell) to localize cells and pixel-by-pixel biomarker expression classification for each instance of a class (e.g., cell) to classify cells based on biomarker expression.

[0097] In block 830, two or more segmentation masks for each of the one or more biomarkers are generated and output based on the combined cellular features and spatial information and the staining patterns of the biomarkers. The two or more segmentation masks include a positive segmentation mask for cells expressing the biomarker and a negative segmentation mask for cells not expressing the biomarker. In some cases, the segmentation masks are output as high-resolution image patches, with each pixel on each mask assigned a value representing the probability that the cell location and biomarker type are expressed (or reach a certain expression criterion). In some embodiments, the segmentation masks are output in a size of 256 pixels by 256 pixels, with each pixel having a value ranging from 0 to 1, where 0 represents that the biomarker corresponding to the segmentation mask is not expressed (or does not reach the expression criterion) at the pixel location, and 1 represents that the biomarker corresponding to the segmentation mask is expressed (or reaches the expression criterion) at the pixel location.

[0098] At optional block 835, two or more segmentation masks can be overlaid on one or more images or image patches to generate an instance segmented image. In some cases, the overlay algorithm can include selecting the maximum pixel value for each pixel on the segmentation mask and assigning the maximum value to the corresponding pixel on the instance segmented image. In some cases, the overlay algorithm can include assigning weights to pixels on each segmentation mask, combining the weighted pixel values ​​for each pixel, and assigning the combined value to the corresponding pixel on the instance segmented image. The overlay algorithm can further include a checking step that confirms that each pixel value is less than or equal to 1. It should be understood that the overlay algorithm is not limited to the algorithms described above and can be any algorithm known to those skilled in the art.

[0099] At block 840, the two or more segmentation masks and / or instance segmentation images are output. For example, the two or more segmentation masks and / or instance segmentation images may be presented locally or transmitted to another device. The two or more segmentation masks and / or instance segmentation images may be output along with a subject identifier. In some cases, the two or more segmentation masks and / or instance segmentation images may be output along with a subject identifier. The data is output to a user or a storage device.

[0100] In optional block 845, a diagnosis of a subject associated with the biological sample is determined using two or more segmentation masks and / or instance segmented images. In some cases, the diagnosis is determined based on cells expressing a biomarker in the instance segmented image. In some cases, the diagnosis is determined based on cells not expressing a biomarker in the instance segmented image. In some cases, the diagnosis is determined based on a combination of cells expressing a biomarker in the instance segmented image and cells not expressing a biomarker in the instance segmented image.

[0101] In optional block 850, treatment is administered to a subject associated with the biological sample. In some cases, the treatment is administered based on cells expressing the biomarker in the example segmented image. In some cases, the treatment is administered based on cells not expressing the biomarker in the example segmented image. In some cases, the treatment is administered based on the subject's diagnosis determined in block 845. In some cases, the treatment is administered based on any combination of cells expressing the biomarker in the instance segmented image, cells not expressing the biomarker in the instance segmented image, and the subject's diagnosis determined in block 845.

[0102] Techniques for using instance segmentation in VE image analysis FIG. 9 shows a flowchart illustrating a process 900 for applying a segmentation mask to support or improve image analysis according to various embodiments. The process 900 shown in FIG. 9 can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The process 900 presented in FIG. 9 and described below is intended to be exemplary and non-limiting. While FIG. 9 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, steps may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, such as those shown in FIGS. 1 and 2, the process shown in FIG. 9 can be performed as part of a result generation stage (e.g., result generation 225) to support or improve image analysis.

[0103] Process 900 begins at block 905, where multiple images of a sample are accessed. In some cases, the specimen is stained for one or more biomarkers. In some embodiments, the accessed images are RGB images or multispectral images. In some embodiments, the accessed images are stored in a memory device. The images can be generated or acquired, e.g., in real time, using an imaging device (e.g., an imaging device described with reference to FIG. 1). In some embodiments, the images are accessed from a microscope or other instrument capable of capturing image data of a microscope slide holding a specimen, as described herein. In some embodiments, the images are accessed from a 2D scanner, such as one capable of scanning image tiles. Alternatively, the images may be images previously generated (e.g., scanned) and stored in a memory device (or, in that case, retrieved from a server via a communications network). In some cases, the images are instance segmented images that include one or more segmentation masks for biological materials or structures, such as cells. The instance segmented images can be generated and / or acquired for access as described with reference to process 800 and FIG. 8.

[0104] In some embodiments, the accessed image is a multiplexed image, i.e., the received image is an image of a biological sample stained with two or more stains. In these embodiments, in block 910, prior to further processing, each multiplexed image is unmixed into its constituent channels, with each unmixed channel corresponding to a particular stain or signal. Following image acquisition and / or unmixing, in blocks 915-945, the image or unmixed image channel images are processed by image analysis algorithms to identify and classify cells and / or nuclei. The processes and analysis algorithms described herein can be adapted to identify and classify various types of cells or cell nuclei based on features within the input image, including identifying and classifying tumor cells, non-tumor cells, stromal cells, lymphocytes, non-target stains, etc.

[0105] In block 915, candidate nuclei are identified. In some embodiments, the image is input to image analysis to detect nuclear centers (seeds) and / or segment nuclei. For example, multiple pixels in the image stained with a biomarker can be identified, which involves considering one or more color planes of multiple pixels in the foreground of the input image for simultaneous identification of cytoplasmic and membrane pixels of cells. In some cases, the image is preprocessed to remove portions of the image that are not analyzed or determined to be background or negative for a given biomarker, such as the slide background, cells not expressing the given biomarker as described with respect to FIGS. 2-8, and / or counterstain components of the image. A threshold level between cytoplasmic and membrane pixels in the foreground of the digital image is then determined, and the selected pixel and a predetermined number of its neighboring pixels from the foreground are processed to determine whether the selected pixel is a cytoplasmic pixel, membrane pixel, or transition pixel in the digital image based on the determined threshold level. In some embodiments, tumor nuclei are automatically identified by applying Parvin's radial symmetry-based method, such as a radial symmetry-based method, hematoxylin image channel, or biomarker image channel after unmixing.

[0106] At block 920, features are extracted from the candidate nuclei. At least some of the features can be associated with biological materials or structures within the target region of the image, such as tumor cells or clusters of tumor cells. The extraction may be performed by an image analysis predictive model, such as Mask R-CNN, that is capable of semantic or instance segmentation. For example, after the candidate nuclei are identified, the candidate nuclei can be further analyzed by the image analysis predictive model to distinguish tumor nuclei from other candidate nuclei (e.g., lymphocyte nuclei). In certain examples, the other candidate nuclei can be further processed to identify specific classes of nuclei and / or cells, for example, to distinguish lymphocyte nuclei from stromal nuclei.

[0107] In block 925, biological materials or structures within the target region are classified into cell or cell nuclei types based on features extracted from the candidate nuclei. The classification may be performed by an image analysis predictive model. In some embodiments, a trained supervised classifier is applied to distinguish tumor nuclei from candidate nuclei. For example, a trained supervised classifier can be trained on nuclei features to identify tumor nuclei and then applied to classify nuclei candidates in a test image as either tumor nuclei or non-tumor nuclei. Optionally, the trained supervised classifier may be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the trained supervised classifier used to identify tumor nuclei is a random forest classifier. For example, the random forest classifier can be trained by (i) creating a training set of tumor nuclei and non-tumor nuclei, (ii) extracting features for each nuclei, and (iii) training the random forest classifier to distinguish between tumor nuclei and non-tumor nuclei based on the extracted features. The trained random forest classifier is then applied to distinguish nuclei in a test image from tumor nuclei. and non-tumor nuclei. Optionally, the random forest classifier may be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei.

[0108] At block 930, a segmentation mask of the target region is predicted and output based on the classification of the biological material or structure. The segmentation mask may be output by an image analysis predictive model. The segmentation mask can be overlaid on the image to generate a target region mask image.

[0109] In optional block 935, metrics are derived from the various identified nuclei, cells, clusters of cells, and / or biological material or structures, as described in detail herein. In some cases, one or more metrics can be calculated by applying various image analysis algorithms to pixels contained in or surrounding the classified nuclei, cells, clusters of cells, and / or biological material or structures. In some embodiments, the metrics include disease state, area, minor and major axis lengths, perimeter, radius, stiffness, etc.

[0110] At block 940, a non-target region mask image is provided. For example, the target region mask image or the target region may be provided to a memory storage device, provided to a display of a computing device, or provided to a user in one or more types of media, such as a user interface. In some cases, providing the target region mask image includes providing associated metrics, or the metrics may be provided separately.

[0111] VI. Working Examples Example 1. Experiments using PD1 We trained the machine learning model using images from the ImageNet dataset. The ImageNet dataset included multiple IF-stained tissue slides, including stomach, pancreas, lung, breast, colon, and bladder indications. The dataset contained variable-size images covering tumor, peritumoral, and normal tissue regions within the slide. The image resolution was 0.325 μm / pixel. The images were preprocessed using repel coding, as described herein with respect to Equation (1). Patches were extracted from these images and masks to train the model. In this example, the training image patches had a size of 250 pixels by 250 pixels. The biomarker used for classification was PD1. This biomarker is primarily expressed on lymphocytes and appears in multiple patterns, including membranous (partially or completely), punctate, nuclear, spherical, and combinations of these patterns. PD1 exhibits a wide range of expression intensities. Figure 10 shows the different staining patterns of PD1.

[0112] In this experiment, 100 images were selected from 15 different slides. The images were divided into patches of 250 pixels x 250 pixels in size before being fed into the machine learning model, as described with respect to Figure 6. Figure 11 shows the visual results from both the training and validation sets. The model did not detect any false positives for erythrocyte expression. Table 1 shows the recall and precision of PD1+ and PD1- cells for both the training and validation sets. [Table 1]

[0113] VII. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0114] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described, or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be left to those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0115] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0116] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. 1. A computer-implemented method comprising: accessing an image of the biological sample, the image displaying cells containing a staining pattern of the biomarker; inputting the image into a trained machine learning model; the trained machine learning model comprises a convolutional neural network including an encoder and a decoder; one or more layers of the encoder include residual blocks with skip connections; parameters of the trained machine learning model were learned from training images depicting cells having staining patterns associated with biomarkers, the training images being annotated with dot labels providing information including cellular centers of the cells depicted in the training images, and corresponding label masks for each of the biomarkers; the corresponding label masks are generated using a repel-coding scheme in combination with the labels of each of the biomarkers, and using the repel-coding scheme includes applying the repel-coding scheme to the training images to calculate distances between pixels of the training images and the cell centers; encoding the image into a feature representation including the extracted discriminative features by the trained machine learning model; combining, by the trained machine learning model, the cellular features and spatial information described by the image of the biological sample with the staining pattern of the biomarkers through a sequence of upconvolution and concatenation with the extracted discriminative features from the feature representation; generating, by the trained machine learning model, two or more segmentation masks for the biomarkers in the image based on the combined feature and spatial information of the cells and the staining pattern of the biomarkers, wherein the two or more segmentation masks include a positive segmentation mask for cells of the image that express the biomarkers and a negative segmentation mask for cells of the image that do not express the biomarkers; A method comprising:

2. superimposing the two or more segmentation masks on the image to generate an instance segmented image; outputting the instance segmented image; The computer-implemented method of claim 1 , further comprising:

3. the image of the biological sample displays cells containing a staining pattern of a biomarker and another biomarker; the trained machine learning model generates two or more segmentation masks for the different biomarkers in the image; The two or more segmentation masks for the other biomarkers may include a positive segmentation mask for cells expressing the other biomarkers and a positive segmentation mask for cells expressing the other biomarkers. and a negative segmentation mask for cells that do not express the specified biomarker.

4. superimposing the two or more segmentation masks for each of the biomarker and the further biomarker on the image to generate an instance segmented image; outputting the instance segmented image; The computer-implemented method of claim 3 further comprising:

5. 2. The computer-implemented method of claim 1, wherein the label mask is generated based on: (i) coding a cell in each training image of the training images using the repel coding scheme, the coding including a cell center and a periphery represented by a response attenuation away from the cell center; and (ii) generating two or more label masks for the biomarkers in the image based at least in part on the coding.

6. 2. The computer-implemented method of claim 1, wherein combining the characteristic and spatial information of the cell with the staining pattern of the biomarker comprises projecting the extracted discriminatory features into a pixel space and classifying each pixel space, wherein the classification comprises cell detection and classification of the cell based on the staining pattern of the biomarker.

7. The trained machine learning model comprises, at least in part: accessing an image of a biological sample, the image displaying cells containing staining patterns of biomarkers, the cells annotated with labels providing information including (i) cell center and (ii) expression of the biomarkers; generating two or more repel-coding masks for each of the images, said generating comprising: (i) coding the cells in the images using a repel-coding algorithm, wherein an output of the coding is an initial cell localization mask including the cell center and a periphery represented by a response attenuation away from the cell center; (ii) segmenting the initial cell localization mask using the labels to classify each instance of the cell based on the expression of the biomarkers; and (iii) dividing the initial cell localization mask into the two or more repel-coding masks based on the segmentation and classification of the cell instances; labeling each of the images with the two or more repel coding masks to generate a set of training images; training a machine learning algorithm on the set of training images to generate the trained machine learning model, the training comprising performing iterative operations to learn a set of parameters for segmenting and classifying cells that maximizes or minimizes an objective function, each iteration comprising finding a set of parameters for the machine learning algorithm such that a value of the objective function using the set of parameters is greater or less than a value of the objective function using another set of parameters in a previous iteration, the objective function being configured to measure a difference between a segmentation mask predicted using the machine learning algorithm and the two or more repel coding masks for the images; The computer-implemented method of claim 1 , wherein the method is trained based on

8. 8. The computer-implemented method of claim 7, wherein the two or more recoding masks comprise a positive mask for cells expressing the biomarker and a negative mask for cells not expressing the biomarker.

9. 8. The computer-implemented method of claim 7, wherein the trained machine learning model comprises a convolutional neural network including an encoder and a decoder, and one or more layers of the encoder include residual blocks with skip connections.

10. The computer-implemented method of claim 7 , wherein the image is an image patch of a predetermined size.

11. 11. The computer-implemented method of claim 10, further comprising: dividing the two or more repel coding masks into mask patches of the predetermined size; and labeling each of the image patches with the mask patch to generate a set of training images.

12. 1. A system comprising: one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on said one or more data processors, cause said one or more data processors to perform any of the steps of the method of any one of claims 1 to 11; A system comprising:

13. A computer program comprising instructions configured to cause one or more data processors to carry out the computer-implemented method of any one of claims 1 to 11.

Images