Machine learning model for cell localization and classification using repulsive coding learning

Abstract

The present disclosure relates to computer-implemented techniques for cell localization and classification. Specifically, aspects of the present disclosure relate to: accessing an image of a biological sample, wherein the image depicts a cell comprising a staining pattern of a biomarker; inputting the image into a machine learning model; encoding, by the machine learning model, the image into a feature representation comprising extracted discriminative features; combining, by the machine learning model, through an up-convolution sequence and concatenation, feature and spatial information of the cell and the staining pattern of the biomarker with the extracted discriminative features from the feature representation; and generating, by the machine learning model, two or more segmentation masks for the biomarker in the image based on the combined feature and spatial information of the cell and the staining pattern of the biomarker.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 065,268, filed August 13, 2020, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This disclosure relates to digital pathology, and more specifically to techniques for using exclusion coding to efficiently train machine learning models to automatically detect, characterize, and / or classify parts or all of digital pathology images. Background Technology

[0004] Digital pathology involves scanning slides (e.g., histopathology or cytopathology slides) into digital images that can be interpreted on a computer screen. For a variety of reasons, including disease diagnosis, assessment of response to treatment, and development of drug formulations to combat disease, the tissues and / or cells within the digital images can subsequently be examined and / or interpreted by a pathologist through digital pathology image analysis. To examine the tissues and / or cells within the digital images (which are virtually transparent), pathology slides can be prepared using various staining assays (e.g., immunohistochemistry) that selectively bind to tissue and / or cellular components. Immunofluorescence (IF) is a technique used to analyze assays that bind fluorescent dyes to antigens. Multiple assays responding to various wavelengths can be utilized on the same slide. These multiplexed IF slides enable understanding of the complexity and heterogeneity of the immune environment of the tumor microenvironment, and its potential impact on the tumor's response to immunotherapy. In some assays, the target antigen to be stained in the tissue can be referred to as a biomarker. Subsequently, digital pathological image analysis can be performed on digital images of stained tissues and / or cells to identify and quantify staining in biological tissues against antigens (e.g., biomarkers indicating various cells such as tumor cells).

[0005] Machine learning techniques have shown great promise in digital pathology image analysis, such as in cell detection, counting, localization, classification, and patient prognosis. Many computational systems equipped with machine learning techniques, including convolutional neural networks (CNNs), have been proposed for image classification and digital pathology image analysis (such as cell detection and classification). For example, a CNN can have a series of convolutional layers as hidden layers, and this network structure is capable of extracting representative features for object / image classification and digital pathology image analysis. In addition to object / image classification, machine learning techniques for image segmentation have also been implemented. Image segmentation is the process of dividing a digital image into multiple segments (sets of pixels, also called image objects). The goal of segmentation is to simplify and / or change the representation of the image to something more meaningful and easier to analyze. For example, image segmentation is often used to locate objects in an image, such as cells and boundaries (lines, curves, etc.). To perform image segmentation on large datasets (e.g., entire slide pathology images), the image is first divided into many small blocks. A computational system equipped with machine learning techniques is trained to classify these blocks and combine all blocks of the same category into a single segmentation region. Subsequently, based on representative features associated with the segmented regions, machine learning techniques can be further implemented to predict or further classify the segmented regions (e.g., cells that are positive for a given biomarker, cells that are negative for a given biomarker, or cells that do not express staining). Summary of the Invention

[0006] This invention discloses a method, system, and computer-readable storage medium for using exclusion coding to efficiently train machine learning models to automatically detect, characterize, and / or classify parts or all of digital pathology images.

[0007] These methods, systems, and computer-readable storage media can be embodied in a variety of ways.

[0008] In various embodiments, a computer-implemented method includes: accessing an image of a biological sample, wherein the image depicts cells including staining patterns of biomarkers; inputting the image into a machine learning model, wherein: the machine learning model includes a convolutional neural network including an encoder and a decoder, one or more layers of the encoder including residual blocks with skip connections, parameters of the machine learning model being learned from training images and label masks for each biomarker in the training images, and the label masks being generated using exclusion coding combined with labels for each biomarker in the biomarkers; encoding the image into a feature representation including extracted discriminative features by the machine learning model; combining cell features and spatial information, as well as staining patterns of biomarkers, with extracted discriminative features from the feature representation by the machine learning model through an overconvolutional sequence and concatenation; and generating two or more segmentation masks for biomarkers in the image based on the combined cell features and spatial information, as well as staining patterns of biomarkers, wherein the two or more segmentation masks include a positive segmentation mask for cells expressing the biomarker and a negative segmentation mask for cells not expressing the biomarker.

[0009] In some embodiments, the method further includes: overlaying two or more segmentation masks onto an image to generate an instance segmentation image; and outputting the instance segmentation image.

[0010] In some embodiments, the method further includes: a user determining a diagnosis for a subject associated with a biological sample, wherein the diagnosis is determined based on: (i) cells expressing a biomarker within an instance segmentation image, and / or (ii) cells not expressing a biomarker within an instance segmentation image.

[0011] In some embodiments, the method further includes: the user administering treatment to the subject based on: (i) segmenting cells expressing biomarkers within an image, (ii) segmenting cells not expressing biomarkers within an image, and / or (iii) diagnosing the subject.

[0012] In some embodiments, the image depicts cells with staining patterns including a biomarker and another biomarker, and the machine learning model generates two or more segmentation masks for the other biomarker in the image, wherein the two or more segmentation masks for the other biomarker include a positive segmentation mask for cells expressing the other biomarker and a negative segmentation mask for cells not expressing the other biomarker.

[0013] In some embodiments, the method further includes: overlaying two or more segmentation masks for each of the biomarker and another biomarker onto an image to generate an instance segmentation image; and outputting the instance segmentation image.

[0014] In some embodiments, generating a label mask includes: (i) encoding cells in each training image using an exclusion coding scheme, the coding including the cell center and a perimeter represented by response decay away from the cell center; and (ii) generating two or more label masks for each biomarker in the image based on the coding and a label for each biomarker in the image.

[0015] In some embodiments, combining cell features and spatial information with staining patterns of biomarkers includes: projecting extracted discriminative features onto a pixel space, and classifying each pixel space, wherein the classification includes cell detection and classification based on staining patterns of biomarkers.

[0016] In various embodiments, a computer-implemented method includes: accessing an image of a biological sample, wherein the image depicts cells including a staining pattern of a biomarker, and wherein the cells are annotated with labels providing information including: (i) cell center, and (ii) biomarker expression; generating two or more exclusion-coded masks for each image, wherein generation includes: (i) encoding the cells in the image using an exclusion-coding algorithm, wherein the output of the encoding is an initial cell localization mask including the cell center and a perimeter represented by response decay away from the cell center, (ii) segmenting the initial cell localization mask using the labels to classify each instance of the cell based on the biomarker expression, and (iii) segmenting the cells based on the cell instance and The classification process involves splitting an initial cell localization mask into two or more exclusion-coded masks; labeling each image in the image with the two or more exclusion-coded masks to generate a training image set; training a machine learning algorithm on the training image set to generate a machine learning model, wherein training includes performing iterative operations to learn a set of parameters for cell segmentation and classification, which maximizes or minimizes an objective function, wherein each iteration involves finding a set of parameters for the machine learning algorithm such that the value of the objective function using the parameter set is greater than or less than the value of the objective function using another set of parameters in a previous iteration, and wherein the objective function is constructed to measure the difference between the segmentation mask predicted using the machine learning algorithm and two or more exclusion-coded masks of the image; and providing the machine learning model.

[0017] In some embodiments, two or more exclusion coding masks include a positive mask for cells expressing the biomarker and a negative mask for cells not expressing the biomarker.

[0018] In some embodiments, the trained machine learning model includes a convolutional neural network that includes an encoder and a decoder, and one or more layers of the encoder include residual blocks with skip connections.

[0019] In some embodiments, an image is an image block of a predetermined size.

[0020] In some embodiments, the method further includes: dividing two or more exclusion coding masks into mask blocks of a predetermined size and using the mask blocks to label each image block in the image block to generate a training image set.

[0021] In some embodiments, a method is provided that includes: determining a diagnosis for a subject by a user based on results generated by a machine learning model trained using some or all of the techniques disclosed herein, and potentially selecting, recommending, and / or administering specific treatments to the subject based on the diagnosis.

[0022] In some embodiments, a method is provided that includes: determining, by a user, a treatment to be selected, recommended, and / or administered to a subject based on results generated by a machine learning model trained using some or all of the techniques disclosed herein.

[0023] In some embodiments, a method is provided that includes: determining, by a user, whether a subject is eligible to participate in a clinical study or assigning the subject to a specific group in a clinical study based on results generated by a machine learning model trained using some or all of the techniques disclosed herein.

[0024] The terms and expressions used are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features shown and described or portions thereof. However, it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed by way of examples and optional features, modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. Attached Figure Description

[0025] Aspects and features of various embodiments will become more apparent from the examples described with reference to the accompanying drawings, in which:

[0026] Figure 1 Examples of histological staining procedures according to various embodiments of the present disclosure are shown.

[0027] Figure 2A block diagram illustrating a computational environment for processing digital pathology images using a machine learning model, according to various embodiments of the present disclosure, is shown.

[0028] Figure 3 A flowchart illustrating a process for implementing an exclusion coding scheme to generate an exclusion coding mask to be used in training a machine learning algorithm, according to various embodiments of the present disclosure, is shown.

[0029] Figure 4 Annotated images with original point labels are shown according to various embodiments of this disclosure.

[0030] Figure 5 Images encoded by an exclusion coding algorithm to provide an initial cell localization mask according to various embodiments of the present disclosure are shown.

[0031] Figure 6 An improved U-Net model for biomarker segmentation according to various embodiments of the present disclosure is shown.

[0032] Figure 7 A flowchart illustrating a process for training a machine learning algorithm to locate and classify cells based on biomarker expression patterns, according to various embodiments of the present disclosure, is shown.

[0033] Figure 8 A flowchart illustrating a process for locating and classifying cells based on biomarker expression patterns using a machine learning model, according to various embodiments of the present disclosure, is shown.

[0034] Figure 9 A flowchart illustrating a process for applying a segmentation mask to support or improve image analysis, according to various embodiments of the present disclosure, is shown.

[0035] Figure 10 PD1 is shown in different modes according to various embodiments of this disclosure.

[0036] Figure 11 The visual results from both the training and validation sets are shown in Example 1. Detailed Implementation

[0037] While specific embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The apparatuses, methods, and systems described herein can be embodied in many other forms. Furthermore, various omissions, substitutions, and changes can be made to the form of the exemplary methods and systems described herein without departing from the scope of protection.

[0038] I. Overview

[0039] This disclosure describes techniques for detecting, characterizing, and / or classifying parts or all of digital pathology images. More specifically, some embodiments of this disclosure provide machine learning techniques for efficiently training machine learning models using exclusion coding to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cell analysis.

[0040] The ability to characterize biomarkers in cellular and tissue samples, and to measure the heterogeneity of the presence and levels of such biomarkers within and between tissues, can provide valuable information in biomedical science for understanding and characterizing various disease states and / or for appropriate selection of available targeted therapies for a patient's disease state. Furthermore, the ability to identify and characterize regions within tissues with varying distributions of key biomarkers can provide important information for the development of targeted and combination therapies. Developing and selecting appropriate combination therapies may also be an important factor in preventing relapse.

[0041] Automated detection and classification of cells with different biomarker expression levels from digital images could potentially extract information for disease diagnosis and treatment more quickly and accurately. Since most cells and tissues are almost transparent in their digital images, modern laboratory techniques have been developed to aid in the visualization of cells and tissues. Immunofluorescence (IF) is a staining technique that helps visualize components within tissues or cells. IF uses a combination of antibodies and fluorophores to label specific proteins and organelles in cell or tissue samples. Therefore, IF technology is widely used in the study of different cell types, tracking and locating proteins in cell or tissue samples, and identifying biological structures within cell or tissue samples. IF-stained cells in tissue sections can be evaluated under high-magnification microscopy and / or digital images of biological samples can be automatically analyzed using digital pathology algorithms. Typically, in whole-slide analysis, the evaluation of stained biological samples requires detecting cells or cellular structures within the stained biological sample, locating the center of cells or cellular structures, and performing biomarker identification and biomarker pattern recognition.

[0042] Once cells are visible for analysis, cell detection and classification techniques are needed. These techniques have evolved from using handcrafted features to machine learning-based detection methods. Most machine learning-based cell detection and classification techniques implement the classic feature-based machine learning pipeline, where the cell classifier or detector is trained in a pixel space that labels the location and features of target cells. However, accurate cell detection and classification using classic feature-based machine learning is challenging for several reasons. First, different cells exhibit significant morphological variations, requiring parametric models to be redesigned and trained to adapt to diverse cell targets. Second, even for the same cell type, different preparation and imaging protocols can produce different cell appearances. More importantly, certain cell types (e.g., immune cells) tend to cluster, and cell crowding or density makes it difficult to define the boundaries between cells and their associated features. This cell crowding or density problem not only increases the difficulty of cell detection but also the difficulty of classification.

[0043] To address these limitations and problems, the various embodiments disclosed herein relate to machine learning techniques for efficiently training machine learning models using exclusion coding to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cell analysis. Exclusion coding of raw data labels for cell detection is based on neighbor coding, which is typically used for cell counting. Neighbor coding produces local maxima at the cell center. However, distinguishing adjacent cells (especially those that tend to cluster) from each other is a challenge for cell detection. Neighbor coding focuses on entropy, while the response valleys between adjacent cells are often insufficient for accurate cell detection. To overcome this challenge, exclusion coding is configured to increase the response valleys between adjacent cells and more accurately align local maxima at the cell center compared to typical neighbor coding. Essentially, exclusion coding encodes the cell center in a manner that increases reversibility (achieving a better balance between entropy and reversibility), making cells more distinguishable and labelable.

[0044] Furthermore, exclusion coding is configured to intelligently apply cell center coding based on image labels to generate exclusion-coded masks for each of one or more characteristics associated with a cell. These one or more characteristics can be biomarker staining patterns (e.g., bright-field staining or immunofluorescence (IF)), cell types (e.g., tumor cells, immune cells, tissue cells, etc.), the presence or absence of various organelles (such as the nucleus or mitochondria), and so on. For example, given tissue stained for a single biomarker, exclusion coding can be configured to: (i) encode the centers of cells identified by annotation as positive for the biomarker and output an exclusion-coded mask for positive cells, and (ii) encode the centers of cells identified by annotation as negative for the biomarker and output an exclusion-coded mask for negative cells. The exclusion-coded masks are then used as labels for the original images to effectively train machine learning models to automatically detect, characterize, and / or classify cells in biological sample images to support or improve cell analysis. When training a machine learning model on images and / or masks encoded with proximity coding, training tends to group and place nearby cell centers together. When training machine learning models on images and / or masks encoded with exclusion coding, the training tends to emphasize pixels at the cell center (pixels with higher intensity of exclusion coding labels at the cell center) and less pixels between cell centers (pixels with lower intensity of exclusion coding labels at the cell periphery). Therefore, exclusion coding emphasizes the reversibility criterion and trains machine learning models more effectively to detect cells.

[0045] Furthermore, the exclusion-coded mask, serving as the actual ground truth for the corresponding image, indicates whether the image contains cells, and if so, where and what characteristics each cell might possess, such as biomarker patterns. The machine learning model encodes the image into a feature representation that includes extracted discriminative features of the cells, and then combines the cell features and spatial information, along with the staining patterns of the biomarkers, with the extracted discriminative features from the feature representation through convolutional sequences and concatenation. Based on the combined cell features and spatial information, and the staining patterns of the biomarkers, the machine learning model generates two or more probability or segmentation masks for each biomarker in the image. Cell centers for the two or more probability or segmentation masks are extracted using local maxima detection based on exclusion coding. A cost function is used to measure the difference or distance between the output probability or segmentation mask and the ground truth of the corresponding image (i.e., the exclusion-coded mask). The goal of training the machine learning model is to find the model parameters, weights, or structure that minimize or maximize the cost function.

[0046] Once trained, the machine learning model can be used in a computer-implemented method to automatically generate segmentation masks for biomarkers in an image. In some cases, the computer-implemented method is performed as part of preprocessing before executing image analysis algorithms to segment and classify target regions (e.g., tumor cells) within an image. In other cases, the computer-implemented method is performed as part of postprocessing after executing image analysis algorithms to segment and classify target regions (e.g., tumor cells) within an image. However, as will be understood by those skilled in the art, the concepts discussed herein are not limited to preprocessing or postprocessing procedures, but can also be integrated into the overall image analysis processing according to various embodiments.

[0047] Computer implementation methods may include using a machine learning model comprising a convolutional neural network (“CNN”) architecture or model, prior to executing standard image analysis algorithms to learn and identify target regions, which utilizes a two-dimensional segmentation model (a modified U-Net) to automatically detect biological structures (such as cells or nuclei) and biomarkers (such as PD1). However, this disclosure is not limited to segmenting only cells, nuclei, or biomarkers; the techniques described herein can also be applied to distinguish other organelles, such as ribosomes, mitochondria, etc. The CNN architecture can be trained using pre-labeled images of different biomarker regions or biomarker positive and negative regions. Therefore, the trained CNN architecture or model can be used to automatically encode images into feature representations, which can then be masked from whole-slide analysis before, during, or after the image is input into the image analysis algorithm. The feature representations and extracted features are further combined with spatial information of the biomarkers, and a biomarker mask is generated accordingly. The image analysis algorithm can further perform a classification task and output classification labels for the detected cells or cellular structures. Advantageously, the proposed architecture and technology can improve the accuracy of cell detection and biomarker classification via image analysis algorithms.

[0048] II. Definition

[0049] As used in this article, when an action is “based on” something, it means that the action is based at least in part on at least a part of something.

[0050] As used herein, the terms “substantially,” “about,” and “approximately” are defined as substantially but not necessarily exactly as specified (and include exactly as specified), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms “substantially,” “about,” or “approximately” may be replaced with “within [a certain percentage]” for the specified meaning, where percentages include 0.1%, 1%, 5%, and 10%.

[0051] As used herein, the terms “sample,” “biological sample,” “tissue,” or “tissue sample” refer to any sample obtained from any organism, including viruses, that includes biomolecules such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof. Examples of other organisms include mammals (such as humans; mammals such as cats, dogs, horses, cattle, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (e.g., tissue sections and needle biopsies of tissues), cell samples (e.g., cytological smears such as cervical smears or blood smears or cell samples obtained through microdissection), or cell fractions, fragments, or organelles (e.g., obtained by lysing cells and separating their components by centrifugation or other methods). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspiration, cerumen, breast milk, vaginal secretions, saliva, swabs (e.g., oral swabs), or any material containing biomolecules derived from the first biological sample. In some embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject (e.g., a homogenized or liquefied sample).

[0052] As used herein, the terms “biomaterials,” “biological structures,” or “cellular structures” refer to natural materials or structures that comprise whole or part of living structures (e.g., cell nuclei, cell membranes, cytoplasm, chromosomes, DNA, cells, cell clusters, etc.).

[0053] As used in this article, “digital pathological images” refers to digital images of stained samples.

[0054] As used in this article, the term "cell detection" refers to the detection of cells or cell structures (such as cell nuclei, cell membranes, cytoplasm, chromosomes, DNA, cells, cell clusters, etc.).

[0055] As used herein, the term "target region" refers to a region of an image that includes image data intended for evaluation in image analysis processing. Target regions include any region of tissue in an image intended for analysis during image analysis (e.g., tumor cells or staining expression).

[0056] As used herein, the term "tile" or "tile image" refers to a single image corresponding to a portion of the full image or a full slide. In some embodiments, "tile" or "tile image" refers to a region scanned across the entire slide or a region of interest having (x, y) pixel dimensions (e.g., 1000 pixels x 1000 pixels). For example, consider dividing the full image into M columns of tiles and N rows of tiles, where each tile in the M x N stitching contains a portion of the full image; that is, the tile at positions M1, N1 contains a first portion of the image, while the tile at positions M3, N4 contains a second portion of the image, and the first and second portions are distinct. In some embodiments, the tiles may each have the same dimensions (pixel size x pixel size).

[0057] As used herein, the terms “block,” “image block,” or “mask block” refer to a container of pixels corresponding to a portion of a full image, a full slide, or a full mask. In some embodiments, a “block,” “image block,” or “mask block” refers to a region of an image or mask or a region of interest having (x, y) pixel dimensions (e.g., 256 pixels x 256 pixels). For example, a 1000-pixel x 1000-pixel image is divided into 100-pixel x 100-pixel blocks, and the image will comprise 10 blocks (each containing 1000 pixels). In other embodiments, blocks overlap with each “block,” “image block,” or “mask block” having (x, y) pixel dimensions and share one or more pixels with another “block,” “image block,” or “mask block.”

[0058] III. Generation of Digital Pathology Images

[0059] Histological staining is widely used to highlight features of interest and enhance the contrast of sections of tissue or cells in biological samples. For example, staining can be used to label specific types of cells and / or specific types of nucleic acids and / or proteins to aid in microscopic examination. The stained sample can then be evaluated to determine or estimate the quantity of features of interest in the sample (e.g., which may include counts, density, or expression levels) and / or one or more properties of the features of interest (e.g., position, shape characteristics, etc. of the features of interest relative to each other or relative to other features). The process of histological staining can include several stages, such as fixation, processing, embedding, sectioning, staining, and imaging.

[0060] In some embodiments, immunohistochemical staining of tissue sections (e.g., bright-field staining or IF) is a histological staining used to identify the presence of a specific protein in a biological sample. For example, the expression level of a specific protein (e.g., an antigen) is determined by the following steps: (a) performing immunohistochemical analysis on a sample having a specific antibody type; and (b) determining the presence and / or expression level of the protein in the sample. In some embodiments, the immunohistochemical staining intensity is determined relative to a reference determined from a reference sample (e.g., a stained sample of a control cell line, a tissue sample from a non-cancer subject, or a reference sample known to have a predetermined protein expression level).

[0061] Figure 1 An example of a histological staining procedure 100 is shown. Stage 110 of the histological staining procedure 100 includes sample fixation, which can be used to preserve the sample and slow down sample degradation. In histology, fixation generally refers to the irreversible process of using chemicals to retain chemical components, preserve the natural sample structure, and maintain cellular structures from degradation. Fixation can also harden cells or tissues for sectioning. Fixatives can use cross-linked proteins to enhance the preservation of samples and cells. Fixatives can bind to and cross-link certain proteins and denature other proteins through dehydration, which can harden tissues and inactivate enzymes that might otherwise degrade the sample. Fixatives can also kill bacteria.

[0062] Fixatives can be applied, for example, by perfusion and impregnation of the prepared sample. Various fixatives can be used, including methanol, Bouin fixatives, and / or formaldehyde fixatives such as neutral buffered formalin (NBF) or paraffin-formaldehyde (PFA). In the case of liquid samples (e.g., blood samples), the sample can be smeared onto a glass slide and dried before fixation.

[0063] While fixation can be used to preserve the structure of samples and cells for histological research purposes, it can also lead to the concealment of tissue antigens, thus reducing antigen detection. Therefore, fixation is generally considered a limiting factor in immunohistochemistry because formalin can cross-link antigens and mask epitopes. In some cases, additional processes are performed to reverse the effects of cross-linking, including treating the fixed sample with citral anhydride (a reversible protein cross-linking agent) and heating.

[0064] Stage 120 of the histological staining process 100 includes sample preparation and embedding. Sample preparation may include wetting a fixed sample (e.g., a fixed tissue sample) with a suitable histological wax (such as paraffin). Histological wax may be insoluble in water or alcohol, but soluble in paraffin solvents such as xylene. Therefore, water in the tissue may need to be replaced with xylene. For this purpose, the sample may first be dehydrated by gradually replacing the water in the sample with alcohol, which can be achieved by passing the tissue through ethanol of increasing concentrations (e.g., from 0 to approximately 100%). After replacing the water with alcohol, the alcohol may be replaced with xylene, which is miscible with alcohol. Embedding may include embedding the sample in warm paraffin. Because paraffin is soluble in xylene, the molten wax can fill the space that was previously filled with water but is now filled with xylene. The wax-filled sample may be cooled to form a hardened block, which can be clamped into a microtome for sectioning. In some cases, deviating from the exemplary process described above may result in paraffin wetting, thereby inhibiting the penetration of antibodies, chemicals, or other fixatives.

[0065] Stage 130 of the histological staining procedure 100 includes sample sectioning. Sectioning is the process of cutting a thin slice of a sample (e.g., an embedded and fixed tissue sample) from an embedded block for mounting on a microscope slide for examination purposes. A microtome can be used to perform sectioning. In some cases, tissue can be rapidly frozen in dry ice or isopentane and then cut with a cold knife in a freezer (e.g., a cryostat). Other types of coolants can be used to freeze tissue, such as liquid nitrogen. Sections used with bright-field and fluorescence microscopy are typically about 4 to 10 μm thick. In some cases, sections can be embedded in epoxy or acrylic resin, allowing for the cutting of thinner sections (e.g., <2 μm). The section can then be mounted on one or more slides. A coverslip can be placed on top to protect the sample section.

[0066] Stage 140 of the histological staining process 100 includes staining (of sections of tissue samples or fixed liquid samples). The purpose of staining is to identify different sample components through color reactions. Most cells are colorless and transparent. Therefore, it may be necessary to stain tissue sections to make cells visible. The staining process typically involves adding dyes or staining agents to the sample to qualitatively or quantitatively determine the presence of specific compounds, structures, molecules, or features (e.g., subcellular features). For example, staining can help identify or highlight specific biomarkers from tissue sections. In other examples, staining agents can be used to identify or highlight biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or organelles within individual cells.

[0067] Many staining solutions are aqueous. Therefore, to stain tissue sections, it may be necessary to first dissolve the wax and replace it with water (rehydration) before applying the staining solution to the sections. For example, sections can be passed sequentially through xylene, alcohol in decreasing concentrations (from approximately 100% to 0%), and water. Once stained, the sections can be dehydrated again and placed in xylene. The sections can then be mounted on a microscope slide in a mounting medium dissolved in xylene. A coverslip can be placed on top to protect the sample sections. Evaporation of xylene around the edges of the coverslip may dry the mounting medium and firmly adhere the coverslip to the slide.

[0068] Various types of staining protocols can be used to perform staining. For example, exemplary immunohistochemical staining protocols include: using a hydrophobic barrier line around the sample (e.g., tissue section) to prevent reagent leakage from the slide during incubation; treating the tissue section with reagents to block endogenous sources of nonspecific staining (e.g., enzymes, free aldehydes, immunoglobulins, other irrelevant molecules that can mimic specific staining); incubating the sample with permeation buffer to facilitate the penetration of antibodies and other staining reagents into the tissue; incubating the tissue section with a primary antibody at a specific temperature (e.g., room temperature, 6 to 8 °C) for a period of time (e.g., 1 to 24 hours); rinsing the sample with a washing buffer; incubating the sample (tissue section) with a secondary antibody at another specific temperature (e.g., room temperature) for another period of time; rinsing the sample again with a water buffer; incubating the rinsed sample with a chromogenic agent (e.g., DAB); and washing away the chromogen to terminate the reaction. In some cases, counterstaining is subsequently used to identify the entire “landscape” of the sample and serves as a reference for the primary color used to detect tissue targets. Examples of counterstaining agents may include hematoxylin (stains from blue to purple), methylene blue (stains to blue), toluidine blue (stains cell nuclei to deep blue and polysaccharides from pink to red), nuclear solid red (also known as Kernechtrot dye, stains to red), and methyl green (stains to green); non-nuclear staining agents, such as eosin (stains to pink), etc. Those skilled in the art will recognize that other immunohistochemical staining techniques can be performed to perform the staining.

[0069] In another example, a hematoxylin and eosin (H&E) staining protocol can be performed on tissue sections. The H&E staining protocol involves applying a hematoxylin staining agent or mordant mixed with a metal salt to the sample. The sample can then be rinsed in a weakly acidic solution to remove excess staining (differentiation), followed by blue staining in weakly alkaline water. After hematoxylin application, the sample can be counterstained with eosin. It should be understood that other H&E staining techniques can be performed.

[0070] In some embodiments, various types of staining agents can be used to perform staining, depending on the feature of interest targeted. For example, DAB can be used for IHC staining of various tissue sections, where DAB produces a brown color that delineates the feature of interest in the stained image. In another example, alkaline phosphatase (AP) can be used for IHC staining of skin tissue sections because the DAB color can be masked by melanin. Regarding primary staining techniques, suitable staining agents can include, for example, basophilic and eosinophilic staining agents, heme and hematoxylin, silver nitrate, trichrome staining agents, etc. Acidic dyes can react with cationic or basic components in tissues or cells, such as proteins and other components in the cytoplasm. Basic dyes can react with anionic or acidic components in tissues or cells, such as nucleic acids. As mentioned above, an example of a staining system is H&E. Eosin can be a negatively charged pink acidic dye, and hematoxylin can be a purple or blue basic dye comprising hematoxylin oxide and aluminum ions. Other examples of dyes may include iodic acid-Schiff reaction (PAS) dyes, Masson's tricolor dyes, Alsin blue dyes, Vangelson dyes, reticular fiber dyes, etc. In some embodiments, different types of dyes may be used in combination.

[0071] Stage 150 of the histological staining process 100 includes medical imaging. A microscope (e.g., an electron microscope or an optical microscope) can be used to magnify the stained sample. For example, an optical microscope can have a resolution of less than 1 μm, such as about a few hundred nanometers. To observe finer details in the nanometer or sub-nanometer range, an electron microscope can be used. An imaging device (in conjunction with or separate from the microscope) images the magnified biological sample to obtain image data, such as multichannel images (e.g., multichannel fluorescence) having several (e.g., such as between ten and sixteen) channels. The imaging device can include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), optics (e.g., one or more lenses, sensor focusing lens groups, microscope objectives, etc.), imaging sensors (e.g., charge-coupled devices (CCDs), complementary metal-oxide-semiconductor (CMOS) image sensors, etc.), photographic film, etc. In digital embodiments, the imaging device may include multiple lenses that cooperate to demonstrate instantaneous focusing capability. An image sensor, such as a CCD sensor, can capture digital images of the biological sample. In some embodiments, the imaging device is a bright-field imaging system, a multispectral imaging (MSI) system, or a fluorescence microscopy system. Imaging devices can capture images using invisible electromagnetic radiation (such as UV light) or other imaging techniques. For example, an imaging device may include a microscope and a camera arranged to capture images magnified by the microscope. Image data received by an analysis system may be the same as and / or derived from the original image data captured by the imaging device.

[0072] At stage 160, images of the stained slides are stored. Images can be stored locally, remotely, and / or in the cloud. Each image can be stored in association with the subject's identifier and date (e.g., the date the sample was collected and / or the date the image was captured). Images can be further transferred to another system (e.g., a system associated with a pathologist or an automated or semi-automated image analysis system).

[0073] It should be understood that modifications to process 100 are conceivable. For example, if the sample is a liquid sample, stage 120 (processing and embedding) and / or stage 130 (slicing) can be omitted from the process. IV. Exemplary Process of Digital Pathology Image Conversion

[0074] Figure 2 A block diagram illustrating a computing environment 200 for processing digital pathology images using a machine learning model is shown. As further described herein, processing digital pathology images may include using the digital pathology images to train a machine learning algorithm, or using a trained (or partially trained) version of a machine learning algorithm (i.e., a machine learning model) to transform part or all of the digital pathology images into one or more results.

[0075] like Figure 2 As shown, the computing environment 200 includes several stages: image storage stage 205, preprocessing stage 210, labeling stage 215, training stage 220 and result generation stage 225.

[0076] Image storage stage 205 includes one or more image data memories 230, which are accessed (e.g., via preprocessing stage 210) to provide a digital image set 235 of a preselected region or the entire biological sample slide (e.g., a tissue slide). Each digital image 235 stored in each image data memory 230 and accessed in image storage stage 210 may include, according to... Figure 1 The process 100 depicted herein generates some or all of the digital pathology images. In some embodiments, each digital image 235 includes image data from one or more scanned slides. Each digital image 235 may correspond to image data from a single sample and / or image data from the day the underlying image data corresponding to that image was collected.

[0077] Image data may include images, as well as any information associated with color channels or color wavelength channels, and details about the imaging platform on which the images are generated. For example, tissue sections may need to be stained by applying a staining assay that includes one or more different biomarkers associated with a chromogenic staining agent or fluorophore for bright-field or fluorescence imaging. The staining assay may use a chromogenic staining agent for bright-field imaging, an organic fluorophore, quantum dots, or an organic fluorophore together with quantum dots for fluorescence imaging, or any other combination of staining agents, biomarkers, and observation or imaging devices. Exemplary biomarkers include biomarkers for estrogen receptor (ER), human epidermal growth factor receptor 2 (HER2), human Ki-67 protein, progesterone receptor (PR), programmed cell death protein 1 (“PD1”), etc., wherein the tissue sections are detectably labeled with a binder (e.g., an antibody) of each of ER, HER2, Ki-67, PR, PD1, etc. In some embodiments, digital image and data analysis operations, such as classification, scoring, Cox modeling, and risk stratification, depend on the type of biomarker used and the field of view (FOV) selection and annotation. Furthermore, typical tissue sections are processed in an automated staining / assay platform that applies staining assays to the tissue sections to produce stained samples. Several commercially available products are suitable for use as staining / assay platforms; one example is the VENTANA SYMPHONY product from the licensor Ventana Medical Systems, Inc. The stained tissue sections can be fed onto imaging systems, such as microscopes or full-slide scanners with microscope and / or imaging components; an example is the VENTANA iScan Coreo product from the licensor Ventana Medical Systems, Inc. Multiplexed tissue slides can be scanned on equivalent multiplexed slide scanner systems. Additional information provided by the imaging system may include any information related to the staining platform, including the concentration of the chemicals used for staining, the reaction time of the chemicals applied to the tissue during staining, and / or the pre-analysis conditions of the tissue, such as tissue age, fixation method, duration, how the sections are embedded, cut, etc.

[0078] In preprocessing stage 210, each of one, more, or all images in the digital image set 235 is preprocessed using one or more techniques to generate a corresponding preprocessed image 240. Preprocessing may include cropping the image. In some cases, preprocessing may further include normalization or correction (e.g., normalization) to place all features on the same scale (e.g., the same size scale, or the same color scale or color saturation scale). In some cases, the image is resized using a minimum size (width or height) of a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels), optionally preserving the original aspect ratio. Preprocessing may further include noise removal. For example, the image may be smoothed to remove unwanted noise, such as by applying a Gaussian function or Gaussian blur.

[0079] Preprocessed images 240 may include one or more training images, validation input images, and unlabeled images. It should be understood that it is not necessary to access the preprocessed images 240 corresponding to the training group, validation group, and unlabeled group simultaneously. For example, an initial set 240 of training and validation preprocessed images may be accessed first and used to train a machine learning algorithm 255, and unlabeled input image elements may subsequently be accessed or received (e.g., at one or more subsequent times) and used by the trained machine learning model 260 to provide the desired output (e.g., cell classification).

[0080] In some cases, the machine learning algorithm 255 is trained using supervised training, and some or all of the preprocessed images 240 are partially or entirely manually, semi-automatically, or automatically labeled during the labeling phase 215, where labels 245 identify the “correct” interpretation (i.e., “truth”) of various biological materials and structures within the preprocessed images 240. For example, labels 245 may identify features of interest, such as cell classification, binary indications of whether a given cell is a specific type of cell, binary indications of whether the preprocessed image 240 (or a specific region of the preprocessed image 240) contains a specific type of depiction (e.g., necrosis or artifacts), slide-level or region-specific depiction categorical characterization (e.g., identifying a specific type of cell), quantity (e.g., the number of specific types of cells within an identified region, the number of depicted artifacts, or the number of necrotic regions), the presence or absence of one or more biomarkers, and so on. In some cases, labels 245 include location. For example, labels 245 may identify point locations of cell nuclei of a specific type of cell or point locations of a specific type of cell (e.g., original point labels). As another example, label 245 may include a border or boundary, such as a delineated border of a tumor, blood vessels, necrotic areas, etc. As another example, label 245 may include one or more biomarkers identified based on patterns of biomarkers observed using one or more staining agents. For example, tissue slides stained against a biomarker (e.g., programmed cell death protein 1 (“PD1”)) may be observed and / or processed to label cells as positive or negative cells based on the expression level and pattern of PD1 in the tissue. Depending on the feature of interest, a given labeled pre-processed image 240 may be associated with a single label 245 or multiple labels 245. In the latter case, each label 245 may be associated with, for example, an indication of which location or portion within the pre-processed image 245 the label corresponds to.

[0081] The labels 245 assigned in labeling stage 215 can be identified based on input from a human user (e.g., a pathologist or imaging scientist) and / or an algorithm (e.g., an annotation tool) configured to define labels 245. In some cases, labeling stage 215 may include transferring and / or presenting part or all of one or more preprocessed images 240 to a user-operated computing device. In some cases, labeling stage 215 includes utilizing an interface (e.g., using an API) to be presented by labeling controller 250 at a user-operated computing device, wherein the interface includes input components for accepting input for identifying labels 245 of interest. For example, a user interface may be provided by labeling controller 250 that enables the selection of an image or region of an image (e.g., field of view) for labeling. A user operating the terminal can use the user interface to select an image or FOV. Several image or FOV selection mechanisms may be provided, such as specifying a known or irregular shape, or defining an anatomical region of interest (e.g., a tumor region). In one example, the image or FOV is a selected whole tumor region on an IHC slide stained with a combination of hematoxylin and eosin (H&E) staining. Image or FOV selection can be performed by the user or by automated image analysis algorithms, such as tumor region segmentation on an H&E tissue slide. For example, the user can select the image or FOV as the whole slide or the whole tumor, or a segmentation algorithm can be used to automatically assign the whole slide or the whole tumor region as the image or FOV. Subsequently, the user of the terminal can select one or more labels 245 to be applied to the selected image or FOV, such as point locations on cells, positive biomarkers for cell-expressed biomarkers, negative biomarkers for cell-unexpressed biomarkers, cell boundaries, etc.

[0082] In some cases, the interface can identify which specific label 245 is being requested and / or the degree of request, which can be communicated to the user via, for example, text instructions and / or visualizations. For instance, a specific color, size, and / or symbol can indicate that a label 245 is being requested for a specific depiction within an image relative to other depictions (e.g., a specific cell or region or staining pattern). If labels 245 corresponding to multiple depictions are to be requested, the interface can identify each depiction simultaneously, or it can identify each depiction sequentially (such that providing a label for a depiction for one identification triggers the identification of the next depiction for labeling). In some cases, each image is presented until the user has identified a specific number of labels 245 (e.g., of a specific type). For example, a given full slide image or a given block of a full slide image can be presented until the user has identified the presence or absence of three different biomarkers, at which point the interface can present different full slide images or images of different blocks (e.g., until a threshold number of images or blocks are labeled). Therefore, in some cases, the interface is configured to request and / or accept labels 245 for an incomplete subset of features of interest, and the user can determine which of the many potential depictions will be labeled.

[0083] In some cases, the labeling stage 215 includes a labeling controller 250 that implements an annotation algorithm to semi-automatically or automatically label various features of an image or a region of interest within an image. For example, the intensity can be normalized or regularized over the entire image (e.g., a preprocessed image), the intensity can be thresholded or filtered, and / or an algorithm (e.g., configured to detect objects, lines, and / or shapes, such as equation (1)) can be applied. Each boundary, point location, or location encoding can then be identified as a feature of interest. In some cases, a metric is associated with each feature of interest (e.g., a metric indicating the confidence in locating the feature of interest), and the degree of interest can be scaled based on the metric. According to aspects of this disclosure, the annotation algorithm can be configured to locate features of interest using an encoding scheme (e.g., locating the exact position of an object in an image). Locating features of interest can include predicting the point locations and / or boundaries of the features of interest. For example, localization can include identifying or predicting point locations corresponding to each depicted cell and / or identifying or predicting closed shapes corresponding to each depicted cell. In some cases, localization techniques involve encoding raw point labels into masks using an exclusion coding scheme (e.g., label 245 for the location of an object, such as a cell). The exclusion coding scheme is defined as follows:

[0084]

[0085] in, It is the distance from pixel (i,j) to its nearest cell center, and It is the distance from pixel (i,j) to its second nearest cell center. Intermediate variable D′ ij Can be taken Divide by

[0086] A label mask can be generated using an exclusion coding scheme in conjunction with annotations for each biomarker in the biomarker. For example, the localization technique may include: encoding the original point labels into an initial cell localization mask (e.g., label 245 for cell location) using exclusion coding as defined in Equation (1); segmenting the initial cell localization mask using label annotations for one or more biomarkers based on the biomarker staining pattern; and outputting two or more exclusion-coded masks for each biomarker (e.g., one mask for positive cells and one mask for negative cells). The segmentation is instance segmentation, where, along with pixel-level classification for localization, the algorithm further classifies each instance of the class separately (e.g., PD1 positive cells, PD1 negative cells, HER2 positive cells, HER2 negative cells, etc.). For example, given tissue stained for a single biomarker, the exclusion coding scheme may be configured to: (i) encode the centers of cells identified by the annotation as positive for the biomarker and output an exclusion-coded mask for the positive cells, and (ii) encode the centers of cells identified by the annotation as negative for the biomarker and output an exclusion-coded mask for the negative cells. In another example, given a tissue stained for two biomarkers, exclusion coding can be configured to: (i) encode the centers of cells identified by annotation as positive for the first biomarker and output an exclusion coding mask for positive cells; (ii) encode the centers of cells identified by annotation as negative for the first biomarker and output an exclusion coding mask for negative cells; (iii) encode the centers of cells identified by annotation as positive for the second biomarker and output an exclusion coding mask for positive cells; and (iv) encode the centers of cells identified by annotation as negative for the second biomarker and output an exclusion coding mask for negative cells. In another example, given a tissue stained for a single biomarker, exclusion coding can be configured to: (i) encode the centers of cells identified by annotation as positive for the biomarker and output an exclusion coding mask for positive cells (e.g., tumor cells); (ii) encode the centers of cells identified by annotation as negative for the biomarker and output an exclusion coding mask for negative cells (e.g., immune cells); and (iii) encode the centers of cells identified by annotation as tissue cells and output an exclusion coding mask for tissue cells.

[0087] In some cases, the labeling controller 250 annotates the image or field of view (FOV) on the first slide based on user input or an annotation algorithm, and maps the annotations to the remaining slides. Depending on the defined FOV, various annotation and registration methods can be used. For example, these can be done automatically or by the user on an interface (such as VIRTUOSO / VERSO). TM (or a similar interface) selects the whole tumor region to be annotated on an H&E slide from multiple consecutive slides. Since the other tissue slides correspond to consecutive slices from the same tissue block, the label controller 250 performs an inter-label registration operation to map and transfer the whole tumor annotation from the H&E slide to each of the remaining IHC slides in the series. An exemplary method for inter-label registration is described in more detail in co-assigned and co-pending application WO2014140070A2, filed March 12, 2014, “Whole slide image registration and cross-image annotation devices, systems and methods,” which is incorporated herein by reference in its entirety for all purposes. In some embodiments, any other method for image registration and generating whole tumor annotations may be used. For example, a qualified reader (such as a pathologist) may annotate the whole tumor region on any other IHC slide, and the label controller 250 performs the mapping of the whole tumor annotation onto other digitized slides. For example, a pathologist (or an automated detection algorithm) can annotate the entire tumor region on an H&E slide, triggering an analysis of all adjacent consecutive IHC slides to determine a whole-slide tumor score for the annotated region on all slides.

[0088] During training phase 220, training controller 265 may train machine learning algorithm 255 using labels 245 and corresponding preprocessed images 240. In some cases, machine learning algorithm 255 includes CNN, a modified CNN with encoding layers replaced by residual neural networks (“ResNet”), or a modified CNN with encoding and decoding layers replaced by ResNet. In other cases, machine learning algorithm 255 may be any suitable machine learning algorithm configured to localize, classify, and / or analyze preprocessed images 240, such as two-dimensional CNN (“2DCNN”), Mask R-CNN, Feature Pyramid Network (FPN), Dynamic Time Warping (“DTW”) technique, Hidden Markov Model (“HMM”), or a combination of one or more such techniques—e.g., CNN-HMM or MCNN (Multi-Scale Convolutional Neural Network). Computing environment 200 may employ the same type of machine learning algorithm or different types of machine learning algorithms trained to detect and classify different cells. For example, computing environment 200 may include a first machine learning algorithm (e.g., U-Net) for detecting and classifying PD1. The computing environment 200 may also include a second machine learning algorithm (e.g., 2DCNN) for detecting and classifying differentiation cluster 68 (“CD68”). The computing environment 200 may also include a third machine learning algorithm (e.g., U-Net) for combined detection and classification of PD1 and CD68. The computing environment 200 may also include a fourth machine learning algorithm (e.g., HMM) for diagnosing a disease or for predicting the prognosis of a subject (such as a patient). Other types of machine learning algorithms may also be implemented in other examples according to this disclosure.

[0089] In some embodiments, the training phase includes a parameter data store and a trainer controller, which together train the machine learning algorithm 255 based on training data (e.g., labels 245 and corresponding preprocessed images 240) and optimize the parameters of the machine learning algorithm 255 during supervised or unsupervised training. In some cases, the training process includes iterative operations to learn a set of parameters (e.g., one or more coefficients and / or weights) for the machine learning algorithm 255. Each parameter can be an adjustable variable, such that the value of the parameter can be adjusted during training. For example, a cost function or objective function can be configured to optimize the accurate classification of the depicted representation, optimize the representation of a given type of feature (e.g., representing shape, size, uniformity, etc.), optimize the detection of a given type of feature, and / or optimize the accurate localization of a given type of feature. Each iteration may involve learning a set of parameters for the machine learning algorithm 255 that minimizes or maximizes the cost function of the machine learning algorithm 255, such that the value of the cost function using the parameter set is less than or greater than the value of the cost function using another set of parameters in a previous iteration. The cost function can be constructed to measure the difference between the output predicted using the machine learning algorithm 255 and the output predicted using the labels 245 contained in the training data. Once the parameter set is identified, the machine learning algorithm 255 has been trained and can be used as designed for localization and / or classification.

[0090] Training iterations continue until a stopping condition is met. The training completion condition can be configured to be met when: (e.g.) a predefined number of iterations have been completed; when statistics generated based on testing or validation exceed a predetermined threshold (e.g., a classification accuracy threshold); when statistics generated based on confidence metrics (e.g., average or median confidence metrics or a percentage of confidence metrics above a specific value) exceed a predefined confidence threshold; and / or when the user device participating in training review closes the training application executed by training controller 265. In some cases, a new training iteration can be initiated in response to a corresponding request or trigger condition received from the user device (e.g., drift is determined within the trained machine learning model 260).

[0091] The trained machine learning model 260 can then be used (in the results generation phase 225) to process the new preprocessed image 240 to generate predictions or inferences, such as predicting cell center and / or location probabilities, classifying cell types, generating cell masks (e.g., pixel-by-pixel segmentation masks of the image), predicting diagnoses of diseases or prognoses for subjects (e.g., patients), or combinations thereof. In some cases, the mask identifies the location of depicted cells associated with one or more biomarkers. For example, given tissue stained for a single biomarker, the trained machine learning model 260 can be configured to: (i) infer cell centers and / or locations, (ii) classify cells based on features of staining patterns associated with the biomarker, and (iii) output cell detection masks for positive cells and cell detection masks for negative cells. In another example, given tissue stained for two biomarkers, a trained machine learning model 260 can be configured to: (i) infer the center and / or location of cells, (ii) classify cells based on features of staining patterns associated with the two biomarkers, and (iii) output cell detection masks for cells positive for the first biomarker, cell detection masks for cells negative for the first biomarker, cell detection masks for cells positive for the second biomarker, and cell detection masks for cells negative for the second biomarker. In another example, given tissue stained for a single biomarker, a trained machine learning model 260 can be configured to: (i) infer the center and / or location of cells, (ii) classify cells based on features of cells and staining patterns associated with the biomarker, and (iii) output cell detection masks for positive cells and cell detection masks for negative cells and classify the masked cells as tissue cells.

[0092] In some cases, the analysis controller 280 generates an analysis result 285 that can be used to request processing of entities in the underlying image. The analysis result 285 may include a mask, output from a trained machine learning model 260, superimposed on a new preprocessed image 240. Additionally or alternatively, the analysis result 285 may include information calculated or determined based on the output of the trained machine learning model, such as a whole-slide tumor score. In an exemplary embodiment, automated analysis of tissue slides uses an FDA-approved 510(k) approved algorithm by assignee VENTANA. Alternatively or additionally, any other automated algorithm may be used to analyze selected image regions (e.g., masked images) and generate scores. In some embodiments, the analysis controller 280 may further respond to instructions received from a computing device from pathologists, physicians, researchers (e.g., associated with clinical trials), subjects, medical experts, etc. In some cases, communications from the computing device include an identifier for each subject in a set of specific subjects, corresponding to a request to perform iterations of analysis for each subject represented in that set. The computing device can further perform analysis based on the machine learning model and / or the output of the analysis controller 280, and / or provide recommended diagnosis / treatment for the subject.

[0093] It should be understood that computing environment 200 is exemplary, and computing environments 200 with different stages and / or using different components are conceivable. For example, in some cases, the network may omit the pre-processing stage 210, such that the images used to train the algorithm and / or processed by the model are raw images (e.g., from an image data store). As another example, it should be understood that each of the pre-processing stage 210 and the training stage 220 may include a controller to perform one or more actions described herein. Similarly, while the labeling stage 215 is depicted as associated with the labeling controller 250 and the result generation stage 225 is depicted as associated with the analysis controller 280, the controllers associated with each stage may further or alternatively facilitate other actions described herein besides label generation and / or analysis result generation. As yet another example, Figure 2The depiction of the computing environment 200 lacks representation of the following devices: devices associated with the programmer (e.g., selecting the architecture of the machine learning algorithm 255, defining how various interfaces will operate, etc.), devices associated with the user providing initial labels or label review (e.g., in the labeling phase 215), and devices associated with the user requesting model processing on a given image (this user may be the same user as the user providing the initial labels or label review or a different user). Despite the lack of depiction of these devices, the computing environment 200 may involve the use of one, more, or all of these devices, and in practice may involve the use of multiple devices associated with corresponding multiple users providing initial labels or label review and / or multiple devices associated with corresponding multiple users requesting model processing on various images.

[0094] V. Automated target localization and classification technology

[0095] Automated instance segmentation of digital images (e.g., cell localization and classification) provides meaningful information for image analysis in the biomedical industry and plays a crucial role in disease diagnosis and treatment. For example, automated localization and classification of programmed cell death protein 1 (“PD1”) positive and negative cells based on PD1 expression levels and patterns in biological samples can be used to determine the complex immunological status of sample subjects and for disease diagnosis. The accuracy of the segmentation mask generated for objects such as cells depends on the corresponding machine learning model, and the performance of the machine learning model depends on the training and architecture of the machine learning algorithm. In various embodiments, the performance of the machine learning model is optimized by training the machine learning algorithm with images labeled with masks generated by a exclusion coding scheme that takes into account various biomarker staining patterns observed in the images. Furthermore, in some embodiments, the performance of the machine learning model is optimized by using a U-Net architecture that has been modified with residual blocks having skip connections to not only localize objects but also understand the complex features used for multi-class segmentation.

[0096] Exemplary exclusion coding scheme for VA for object location and classification

[0097] Figure 3 A flowchart illustrating a process 300 for implementing an exclusion coding scheme to generate an exclusion coding mask to be used in training a machine learning algorithm is shown according to various embodiments. Figure 3 The process 300 described herein can be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processor, core), hardware, or a combination thereof of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 3The processes presented and described below 300 are illustrative and not limiting. Although Figure 3 Various processing steps are described as occurring in a specific sequence or order, but this is not limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1 and Figure 2 In the embodiments described herein, Figure 3 The processing described herein can be performed by a preprocessing or labeling subsystem (e.g., computing environment 200) to generate an exclusion coding mask for training by one or more machine learning algorithms (e.g., machine learning algorithm 255).

[0098] Process 300 begins at box 310, where it is controlled by a computing device (e.g., regarding...). Figure 2 The labeling controller 250 of the computing environment 200 acquires or accesses images of biological samples. In some cases, the images depict cells with staining patterns associated with biomarkers. In other cases, the images depict cells with multiple staining patterns associated with multiple biomarkers. The images are annotated with raw point labels. The information provided by the raw point labels includes: (i) cell centers, and (ii) the expression of one or more biomarkers. Figure 4 As shown, image 400 includes a primitive dot label 405 located at the center of each cell, and the color, symbol, pattern, etc., of the primitive dot label 405 indicates the expression of one or more biomarkers. An "x" label 405 indicates that the cell is negative for the biomarker (does not express the biomarker). A "+" label 405 indicates that the cell is positive for the biomarker (expresses the biomarker). (See also: [link to image description]) Figure 2 The original point label 405 discussed can be annotated by the user or automatically by an annotation tool during the preprocessing stage.

[0099] At box 320, the image is encoded into an initial cell localization mask using a rejection coding algorithm. The rejection coding algorithm is defined in equation (1). The rejection coding algorithm calculates the distance between a pixel and its nearest cell center label and the distance between a pixel and its second nearest cell center label. For each pixel, a pseudo-distance is calculated based on the distance between the pixel and its nearest cell center label and the distance between the pixel and its second nearest cell center label. If the pseudo-distance is not less than a threshold, the corresponding pixel is assigned a value of 0; otherwise, a non-zero value is assigned to the pixel based on the pseudo-distance. Figure 5 Image 500 encoded by a rejection coding algorithm is shown (e.g., regarding...). Figure 4The algorithm outputs an initial cell localization mask 505, which includes a cell center 510 (represented by higher intensity pixels) and a perimeter 515 (represented by response decay away from the cell center 510).

[0100] At box 330, the initial cell localization mask is segmented using original point labels based on biomarker staining patterns. A segmentation algorithm is used to perform segmentation. Segmentation is instance segmentation, where, along with pixel-level classification for localization, the segmentation algorithm further classifies each instance of a class separately (e.g., PD1-positive cells, PD1-negative cells, HER2-positive cells, HER2-negative cells, etc.). The segmentation algorithm can implement any known instance segmentation technique known in the art. For example, in some cases, the segmentation algorithm is rule-based and segments the cell localization mask based on biomarker information provided by the original point labels. In other cases, the segmentation algorithm is clustering-based (such as k-means) and segments the cell localization mask based on biomarker information provided by the original point labels. Figure 5 As shown, the initial cell localization mask 505 is segmented to generate a segmented cell localization mask 520, where each pixel belongs to a specific class (background 525 or cell 530) based on an exclusion coding algorithm, and a specific biomarker class (biomarker negative 535 or biomarker positive 540) has been assigned to each pixel of the image based on the segmentation algorithm. It should be understood that the order of boxes 320 and 330 can be reversed, where the segmentation algorithm is first applied to the image to generate the segmentation mask, and then the exclusion coding algorithm is applied to the segmentation mask to generate the segmented cell localization mask.

[0101] At box 340, the segmented cell localization mask is split into two or more exclusion-coded masks. Essentially, each class represented within the segmented cell localization mask is split into its own separate exclusion-coded mask. For example, if the segmented cell localization mask has two classes: (1) biomarker-positive cells and (2) biomarker-negative cells; then the segmented cell localization mask is split into two exclusion masks: (1) one mask for biomarker-positive cells, and (2) another mask for biomarker-negative cells. Segmentation can be performed using various image processing algorithms where hiding or removing classes is optional as a parameter. For example, continuing the example above, a mask for biomarker-positive cells can be generated by hiding or removing biomarker-negative cells from the segmented cell localization mask; and a mask for biomarker-negative cells can be generated by hiding or removing biomarker-positive cells from the segmented cell localization mask. Figure 5As shown, the segmented cell localization mask 520 is split into two exclusion coding masks: (1) a mask 545 for cells that are positive for the biomarker, and (2) another mask 550 for cells that are negative for the biomarker.

[0102] An exemplary U-Net for instance segmentation in VB

[0103] The improved U-Net model can be used in biomarker segmentation methods to extract complex features from input images (e.g., one or more images of a biological sample) and generate high-resolution two-dimensional segmentation masks. Figure 6 As shown, the improved U-Net model 600 may include an encoder 605 and a decoder 610, giving it a U-shaped architecture. The encoder 605 is a CNN network comprising repeated applications of convolutions (e.g., 3x3 convolutions (unpadded convolutions)), where one or more layers include residual blocks with skip connections, each convolution followed by a rectified linear unit (ReLU) and a max-pooling operation for downsampling (e.g., a 2x2 max-pooling with a stride of 2). One or more layers of the encoder 605 include residual blocks with skip connections. With the help of the residual blocks, the original input to the convolution is also added to the output of the convolution. The number of feature channels can be doubled in each downsampling step or pooling operation. During encoding, the spatial information of the image data decreases while the feature information increases. The decoder 610 is a CNN network that combines the features and spatial information (upsampling from the feature maps of the encoder 605) from the encoder 605. The upsampling of the feature map is followed by a sequence of upconvolutions that halves the number of channels (upsampling operators), a concatenation with the corresponding cropped feature map from encoder 605, repeated application of convolutions (e.g., two 3x3 convolutions) followed by a Corrected Linear Unit (ReLU) for each convolution, and a final convolution (e.g., a 1x1 convolution) to generate a two-dimensional segmentation mask. It should be understood that one or more layers of decoder 610 may also include residual blocks with skip connections. For localization, high-resolution features from encoder 605 are combined with the upsampled output from decoder 610.

[0104] VC techniques for training machine learning algorithms based on instance segmentation

[0105] Figure 7 A flowchart illustrating a process 700 for training a machine learning algorithm (e.g., a modified U-Net) to locate and classify cells based on biomarker expression patterns, according to various embodiments, is shown. Figure 7 The process 700 described herein can be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processor, core), hardware, or a combination thereof of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 7 The processes 700 presented and described below are illustrative and not limiting. Although Figure 7 Various processing steps are described as occurring in a specific sequence or order, but this is not limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1 and Figure 2 In the embodiments described herein, Figure 7 The processing described herein can be performed as part of a training phase (e.g., algorithm training 220) to generate a prediction or segmentation mask that includes pixel-level classification of localized cells (e.g., background and cells) and per-pixel biomarker expression classification for each instance of a class (e.g., cell) that classifies cells based on biomarker expression.

[0106] Process 700 begins at box 705, where a computing device acquires or accesses training images of the biological sample (e.g., regarding...). Figure 2 The preprocessed image 240 of the computing environment 200. In some cases, the training image depicts cells with staining patterns associated with biomarkers. In other cases, the training image depicts cells with multiple staining patterns associated with multiple biomarkers. The training images are annotated with exclusion-coded masks (e.g., based on information about...). Figure 3 The process 300 generates an exclusion coding mask. The exclusion coding mask provides information about the training images, including: (i) cell location, and (ii) cell classification for one or more biomarkers.

[0107] At box 710, one or more images and mask labels are divided into image patches of a predetermined size. For example, images often have random sizes, while machine learning algorithms (such as modified CNNs) learn better on normalized image sizes, and therefore training images can be divided into image patches of a specific size to optimize training. In some embodiments, training images are split into image patches of predetermined sizes of 64 pixels x 64 pixels, 128 pixels x 128 pixels, 256 pixels x 256 pixels, or 512 pixels x 512 pixels. The partitioning of training images can be performed before generating the exclusion-coded mask, so that the subsequently generated exclusion-coded mask has the same size as the corresponding training image. Alternatively, the partitioning can be performed after generating the exclusion-coded mask. In this case, the exclusion-coded mask is preprocessed before, after, or simultaneously with the partitioning of the training images, and the exclusion-coded mask is divided into mask patches of the same predetermined size as the training images.

[0108] At box 715, a machine learning algorithm is trained on an image patch. In some cases, the machine learning algorithm is a modified U-Net comprising an encoder and a decoder, and one or more layers of the encoder comprise residual blocks (or ResNet) with skip connections. Training may include performing iterative operations to find a set of parameters for the machine learning algorithm that minimizes or maximizes its cost function. The output of the iteration is a prediction or segmentation mask comprising pixel-level classification of localized cells (e.g., background and cells) and per-pixel biomarker expression classification for each instance of a class (e.g., cell) class that classifies cells based on biomarker expression. Each iteration involves finding a set of parameters for the machine learning algorithm such that the value of the cost function using that set is less than the value of the cost function using a different set of parameters in a previous iteration. The cost or objective function is constructed to measure the difference between (i) the predicted pixel-level classification and the predicted per-pixel biomarker expression classification, and (ii) the ground-value exclusion encoded mask. In some cases, the cost function is a binary cross-entropy loss function.

[0109] In some cases, training further includes adjusting the learning rate by maximizing or minimizing the learning rate of the machine learning algorithm according to a predefined schedule. The predefined schedule could be a step decay schedule, which reduces the learning rate by a predetermined factor every predetermined number of periods to optimize the cost function.

[0110] Training iterations continue until a stopping condition is met. The training completion condition can be configured to be met when: (e.g.) a predefined number of iterations have been completed; when statistics generated based on testing or validation exceed a predetermined threshold (e.g., a classification accuracy threshold); when statistics generated based on confidence metrics (e.g., average or median confidence metrics or a percentage of confidence metrics above a specific value) exceed a predefined confidence threshold; and / or when the user device participating in training review closes the training application. As a result of training, the machine learning algorithm has learned nonlinear relationships within the image, which it uses to predict pixel-level classifications of localized cells (e.g., background and cells) and per-pixel biomarker expression classifications for each instance (e.g., cells) of a class that classifies cells based on biomarker expression. The training output includes a trained machine learning model with a learned set of parameters associated with nonlinear relationships that derive the minimum or maximum value of the cost function from all iterations.

[0111] At box 720, the trained machine learning model is provided. For example, as per [reference to...] Figure 2 The trained machine learning model can be deployed for execution in an image analysis environment.

[0112] VD uses machine learning models for instance segmentation.

[0113] Figure 8 A flowchart illustrating a process 800 for locating and classifying cells based on biomarker expression patterns using a machine learning model (e.g., a modified U-Net model) according to various embodiments is shown. Figure 8 The process 800 described herein can be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processor, core), hardware, or a combination thereof of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 8 The processes presented and described below are illustrative and not limiting. Although Figure 8 Various processing steps are described as occurring in a specific sequence or order, but this is not limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1 and Figure 2 In the embodiments described herein, Figure 8 The processing described herein can be performed as part of a result generation phase (e.g., result generation 280) to generate a prediction or segmentation mask that includes pixel-level classification of localized cells (e.g., background and cells) and per-pixel biomarker expression classification for each instance (e.g., cell) of a class that classifies cells based on biomarker expression.

[0114] Process 800 begins at box 805, where one or more images of a biological sample are accessed or obtained. The images depict staining patterns of cells including one or more biomarkers. In some cases, one or more images depict staining patterns of cells including one biomarker and another biomarker. (See also: [link to documentation]) Figure 1 The images can be preprocessed using immunochemical staining techniques (e.g., IF) to make specific proteins and organelles in the biological sample visible to the analytical system for processing and analysis. In some embodiments, the images are stained using multiple staining agents or binding agents (e.g., antibodies) to allow information about different biomarkers to be reported according to multichannel analysis or similar techniques.

[0115] At box 810, one or more images can be divided into image patches of a predetermined size. For example, images often have random sizes, while machine learning algorithms (such as modified CNNs) learn better at normalizing image sizes, and therefore images can be divided into image patches of a specific size to optimize analysis. In some embodiments, the image is divided into image patches of a predetermined size of 64 pixels x 64 pixels, 128 pixels x 128 pixels, 256 pixels x 256 pixels, or 512 pixels x 512 pixels.

[0116] At box 815, one or more images or image patches are input into a machine learning model for further analysis. In some cases, the machine learning model is a modified CNN (e.g., U-Net) model including an encoder and a decoder, and one or more layers of the encoder include residual blocks (or ResNet) with skip connections. The machine learning model further includes parameters learned from training images and corresponding label masks for each biomarker in the training images (as described in detail with respect to computational environment 200 and process 700). The label mask is generated using an exclusion coding scheme combined with labels for each biomarker in the biomarkers (as described in detail with respect to computational environment 200 and process 300). Figure 3 The process of generating a label mask includes: (i) encoding cells in each training image using the exclusion coding, the coding including a cell center and a perimeter represented by response decay away from the cell center; and (ii) generating two or more label masks for each of the biomarkers in the image based on the coding and the label for each of the biomarkers.

[0117] At box 820, a provided machine learning model encodes one or more images or image patches into feature representations that include extracted discriminative features. These discriminative features (e.g., lower-resolution features) may be associated with biological materials or structures such as cells. The machine learning model may encode images or image patches into discriminative features at multiple different levels within multiple different subnetworks, and each subnetwork is associated with at least one expression of a biomarker (e.g., positive or negative).

[0118] At box 825, the extracted discriminative features are combined with the features and spatial information of cells in the image or image patch, as well as the staining patterns of biomarkers, through an upconvolution sequence and concatenation. This combination can be performed by projecting the extracted discriminative features into a pixel space (e.g., at a higher resolution) and classifying each pixel space. The decoder of the machine learning model can perform the combination and projection of the extracted discriminative features at multiple different levels within multiple different subnetworks. In some cases, multiple different levels perform upsampling (i.e., expanding the feature dimension to the original size of the input image patch) and concatenation, followed by regular convolution operations to project the extracted discriminative features. Classification includes cell detection and classification based on staining patterns of biomarkers. Specifically, classification includes pixel-level classification of cells (e.g., background and cells) and per-pixel biomarker expression classification for each instance (e.g., cell) of a class that classifies cells based on biomarker expression.

[0119] At box 830, based on the combined features and spatial information of cells and the staining patterns of biomarkers, two or more segmentation masks are generated and output for each of one or more biomarkers. The two or more segmentation masks include a positive segmentation mask for cells expressing the biomarker and a negative segmentation mask for cells not expressing the biomarker. In some cases, the segmentation masks are output as high-resolution image patches, where each pixel on each mask is assigned a value representing the probability of the cell's location and the type of biomarker expressed (or meeting a specific expression criterion). In some embodiments, the segmentation masks are output in a size of 256 pixels x 256 pixels, where each pixel has a value ranging from 0 to 1, where 0 indicates that the biomarker corresponding to the segmentation mask at the pixel location is not expressed (or has not met the expression criterion), and 1 indicates that the biomarker corresponding to the segmentation mask at the pixel location is expressed (or meets the expression criterion).

[0120] At option 835, two or more segmentation masks can be overlaid on one or more images or image patches to generate an instance segmentation image. In some cases, the overlay algorithm may include selecting the maximum pixel value for each pixel on the segmentation mask and assigning the maximum value to the corresponding pixel on the instance segmentation image. In some cases, the overlay algorithm may include assigning weights to pixels on each segmentation mask, combining the weighted pixel values ​​of each pixel, and assigning the combined value to the corresponding pixel on the instance segmentation image. The overlay algorithm may further include a checking step to ensure that each pixel value is not greater than 1. It should be understood that the overlay algorithm is not limited to the algorithms described above and may be any algorithm known to those skilled in the art.

[0121] At box 840, two or more segmentation masks and / or instance segmentation images are output. For example, the two or more segmentation masks and / or instance segmentation images may be rendered locally or transferred to another device. The two or more segmentation masks and / or instance segmentation images may be output along with the subject's identifier. In some cases, the two or more segmentation masks and / or instance segmentation images are output to the end user or a storage device.

[0122] At optional box 845, two or more segmentation masks and / or instance segmentation images are used to determine a diagnosis for a subject associated with a biological sample. In some cases, the diagnosis is determined based on cells expressing a biomarker within the instance segmentation image. In some cases, the diagnosis is determined based on cells not expressing a biomarker within the instance segmentation image. In some cases, the diagnosis is determined based on a combination of cells expressing a biomarker and cells not expressing a biomarker within the instance segmentation image.

[0123] At optional box 850, treatment is administered to the subject associated with the biological sample. In some cases, treatment is administered based on cells expressing biomarkers within the instance segmentation image. In some cases, treatment is administered based on cells not expressing biomarkers within the instance segmentation image. In some cases, treatment is administered based on the diagnosis of the subject determined at box 845. In some cases, treatment is administered based on any combination of cells expressing biomarkers within the instance segmentation image, cells not expressing biomarkers within the instance segmentation image, and the diagnosis of the subject determined at box 845.

[0124] VE uses instance segmentation techniques in image analysis.

[0125] Figure 9 A flowchart illustrating a process 900 for applying a segmentation mask to support or improve image analysis, according to various embodiments, is shown. Figure 9 The process 900 described herein can be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processor, core), hardware, or a combination thereof of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 9 The processes presented and described below are illustrative and not restrictive. Although Figure 9 Various processing steps are described as occurring in a specific sequence or order, but this is not limiting. In some alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 1 and Figure 2 In the embodiments described herein, Figure 9The processing described herein can be performed as part of the result generation phase (e.g., result generation 225) to support or improve image analysis.

[0126] Process 900 begins at box 905, where multiple images of the sample are accessed. In some cases, the sample is stained for one or more biomarkers. In some embodiments, the accessed images are RGB images or multispectral images. In some embodiments, the accessed images are stored in a memory device. Imaging devices (e.g., regarding...) can be used. Figure 1 The imaging apparatus described herein generates or acquires images (e.g., in real time). In some embodiments, as described herein, the images are acquired from a microscope or other instrument capable of capturing image data of a microscope slide carrying a sample. In some embodiments, the images are accessed from, for example, a 2D scanner capable of scanning image patches. Alternatively, the images may be images that have been previously generated (e.g., scanned) and stored in a memory device (or, in this respect, retrieved from a server via a communication network). In some cases, the images are instance segmentation images, which include one or more segmentation masks for biological materials or structures (such as cells). (See regarding process 800 and...) Figure 8 The instance segmentation image can be generated and / or obtained for access.

[0127] In some embodiments, the input image accessed is a multiplexed image, i.e., the received image is an image of a biological sample stained with more than one staining agent. In these embodiments at block 910, and prior to further processing, the multiplexed image is demixed into its constituent channels, each demixed channel corresponding to a specific staining agent or signal. After image acquisition and / or demixing, the image or demixed image channel images are processed using the image analysis algorithms in blocks 915-945 to identify and classify cells and / or cell nuclei. The processes and analysis algorithms described herein are adaptable to the identification and classification of various types of cells or cell nuclei based on features within the input image, including the identification and classification of tumor cells, non-tumor cells, stromal cells, lymphocytes, non-target staining, etc.

[0128] At box 915, candidate cell nuclei are identified. In some embodiments, the image is input into an image analysis to detect the nucleus center (seed) and / or segment the cell nucleus. For example, multiple pixels in an image stained with a biomarker can be identified, including one or more color planes considering multiple pixels in the foreground of the input image, for simultaneously identifying cytoplasmic and cell membrane pixels. In some cases, the image is preprocessed to remove portions of the image that are not required for analysis or cannot be determined as background or are negative for a given biomarker, such as slide background, as per [reference to...]. Figures 2 to 8The cell and / or image counterstaining components do not express a given biomarker. Subsequently, a threshold level is determined between cytoplasmic and cell membrane pixels in the foreground of the digital image, and selected pixels from the foreground and a predetermined number of their neighboring pixels are processed based on the determined threshold level to determine whether the selected pixels are cytoplasmic pixels, cell membrane pixels, or transition pixels in the digital image. In some embodiments, tumor cell nuclei are automatically identified by applying a radial symmetry-based method (Parvin's radial symmetry-based method), for example, on the unmixed hematoxylin image channel or biomarker image channel.

[0129] At box 920, features are extracted from candidate cell nuclei. At least some of these features may be associated with biological material or structures within the target region of the image, such as tumor cells or tumor cell clusters. Extraction can be performed using an image analysis prediction model, such as Mask R-CNN capable of semantic or instance segmentation. For example, after identifying candidate cell nuclei, the image analysis prediction model can further analyze the candidate cell nuclei to distinguish tumor cell nuclei from other candidate cell nuclei (e.g., lymphocyte nuclei). In some cases, other candidate cell nuclei may be further processed to identify specific categories of cell nuclei and / or cells, such as identifying lymphocyte nuclei and stromal cell nuclei.

[0130] At box 925, based on features extracted from candidate cell nuclei, the biological material or structure within the target region is classified as a cell or nucleus type. Classification can be performed by an image analysis prediction model. In some embodiments, a learned supervised classifier is applied to identify tumor cell nuclei from candidate cell nuclei. For example, the learned supervised classifier can be trained on cell nucleus features to identify tumor cell nuclei and subsequently applied to classify candidate cell nuclei in a test image as tumor cell nuclei or non-tumor cell nuclei. Optionally, the learned supervised classifier can be further trained to distinguish between different categories of non-tumor cell nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the learned supervised classifier used to identify tumor cell nuclei is a random forest classifier. For example, a random forest classifier can be trained by (i) creating a training set of tumor and non-tumor cell nuclei, (ii) extracting features for each cell nucleus, and (iii) training a random forest classifier to distinguish between tumor and non-tumor cell nuclei based on the extracted features. The trained random forest classifier can then be applied to classify cell nuclei in a test image as tumor cell nuclei or non-tumor cell nuclei. Optionally, the random forest classifier can be further trained to distinguish between different categories of non-tumor cell nuclei, such as lymphocyte nuclei and stromal nuclei.

[0131] At box 930, a segmentation mask for the target region is predicted and output based on the classification of biological materials or structures. The segmentation mask can be output by an image analysis prediction model. The segmentation mask can be overlaid on the image to generate an image masked by the target region.

[0132] At optional box 935, as discussed in detail herein, metrics can be derived from various identified nuclei, cells, cell clusters, and / or biological materials or structures. In some cases, one or more metrics can be computed by applying various image analysis algorithms to pixels contained in or surrounding the classified nuclei, cells, cell clusters, and / or biological materials or structures. In some embodiments, metrics include disease state, area, minor and major axis lengths, perimeter, radius, stiffness, etc.

[0133] At box 940, an image masked by a target region is provided. For example, the image masked by the target region or the target region may be provided to a memory storage device, to a display of a computing device, or to a user (such as a user interface) in one or more types of media. In some cases, providing the image masked by the target region includes providing associated metrics, or metrics may be provided separately.

[0134] VI. Examples

[0135] Example 1. Experiment using PD1

[0136] Images from the ImageNet dataset were used to train the machine learning model. The ImageNet dataset includes multiple IF-stained tissue slides, including the stomach, pancreas, lung, breast, colon, and bladder as indications. The dataset includes variable-sized images covering tumor, peritumor, and normal tissue areas within the slides. The images have a resolution of 0.325 μm / pixel. The images were preprocessed using exclusion coding as described herein with respect to equation (1). Patches were extracted from these images and masks to train the model. In this example, the training image patches were 250 pixels x 250 pixels in size. The biomarker used for classification was PD1. This biomarker is primarily expressed on lymphocytes and exhibits a variety of patterns, including membranous (partial or intact), punctate, nuclear, globular, and combinations of these patterns. PD1 exhibits a pattern of expression across a wide range of intensities. Figure 10 Different staining patterns of PD1 are shown.

[0137] In this experiment, 100 images were selected from 15 different slides. Before being fed into the machine learning model, the images were divided into 250 pixel x 250 pixel blocks, as shown in the diagram. Figure 6 As stated above. Figure 11Visual results from both the training and validation sets are shown. The model did not detect any false positives regarding erythrocyte expression. Table 1 shows the recall and precision for PD1+ and PD1- cells on both the training and validation sets.

[0138] Table 1: Recall and Precision on Training and Validation Sets

[0139] PD1+ (training set) 94.37 98.74 PD1 (Training Set) 98.12 98.5 PD1+ (Validation Set) 87.45 90.12 PD1 (Validation Set) 86.5 88.37

[0140] VII. Other Precautions

[0141] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0142] The terms and expressions used are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features shown and described or portions thereof; however, it should be recognized that various modifications are possible within the scope of the claimed invention. Therefore, it should be understood that although the claimed invention has been specifically disclosed by way of examples and optional features, those skilled in the art can employ modifications and variations of the concepts disclosed herein, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0143] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of preferred exemplary embodiments will provide those skilled in the art with a feasible description for implementing various embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0144] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as parts in block diagram form to avoid obscuring the embodiments with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

Claims

1. A computer-implemented method, comprising: Access images of biological samples, wherein the images depict staining patterns of cells including biomarkers; The image is input into the machine learning model, where: The machine learning model includes a convolutional neural network, which comprises an encoder and a decoder. One or more layers of the encoder include residual blocks with skip connections. The parameters of the machine learning model are learned from training images and label masks for each biomarker in the training images, and The label mask is generated using exclusion coding combined with labels for each of the biomarkers; The machine learning model encodes the image into a feature representation that includes extracted discriminative features; The machine learning model combines the cell's features and spatial information, along with the staining pattern of the biomarker, with the extracted discriminative features from the feature representation through convolutional sequences and cascades; and The machine learning model generates two or more segmentation masks for the biomarker in the image based on the combined features and spatial information of the cells and the staining pattern of the biomarker, wherein the two or more segmentation masks include a positive segmentation mask for cells expressing the biomarker and a negative segmentation mask for cells not expressing the biomarker.

2. The method according to claim 1, further comprising: The two or more segmentation masks are superimposed on the image to generate an instance segmentation image; as well as Output the segmented image of the instance.

3. The method according to claim 2, further comprising: The diagnosis of a subject associated with the biological sample is determined by the user, wherein the diagnosis is based on: (i) the cells expressing the biomarker in the instance segmentation image, and / or (ii) the cells not expressing the biomarker in the instance segmentation image.

4. The method of claim 3, further comprising: The treatment is administered to the subject by the user based on: (i) cells expressing the biomarker in the instance segmentation image, (ii) cells not expressing the biomarker in the instance segmentation image, and / or (iii) the diagnosis of the subject.

5. The method according to claim 4, wherein: The image depicts staining patterns in cells that include biomarkers and another type of biomarker. The machine learning model generates two or more segmentation masks for the other biomarker in the image, and The two or more segmentation masks for the other biomarker include a positive segmentation mask for cells expressing the other biomarker and a negative segmentation mask for cells not expressing the other biomarker.

6. The method of claim 5, further comprising: The two or more segmentation masks for each of the biomarker and the other biomarker are superimposed on the image to generate an instance segmentation image; as well as Output the segmented image of the instance.

7. The method of claim 1, wherein generating the tag mask comprises: (i) Encode the cells in each training image using the exclusion coding, the coding including the cell center and the perimeter represented by response decay away from the cell center, and (ii) Generate two or more label masks for each of the biomarkers in the image based on the coding and the label for each of the biomarkers.

8. The method of claim 1, wherein combining the characteristics and spatial information of the cell with the staining pattern of the biomarker comprises: The extracted discriminative features are projected onto a pixel space, and each pixel space is classified, wherein the classification includes cell detection and classification of the cells based on the staining patterns of the biomarkers.

9. A system comprising: One or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform any one of the steps of the method according to any one of claims 1 to 8.

10. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform any one of the steps of the method according to any one of claims 1 to 8.

11. A computer-implemented method, comprising: Access to images of biological samples, wherein the images depict staining patterns of cells including biomarkers, and wherein the cells are annotated with informational labels including: (i) cell centers, and (ii) expression of the biomarkers; Generating two or more exclusion-coded masks for each of the images, wherein the generation includes: (i) encoding the cells in the images using an exclusion-coded algorithm, wherein the output of the encoding is an initial cell localization mask including the cell center and a perimeter represented by response decay away from the cell center; (ii) segmenting the initial cell localization mask using the label to classify each instance of the cell based on the expression of the biomarker; and (iii) splitting the initial cell localization mask into the two or more exclusion-coded masks based on the segmentation and classification of the instances of the cell. Each image in the images is labeled with one or more exclusion coding masks to generate a training image set; A machine learning algorithm is trained on the training image set to generate a machine learning model, wherein the training includes performing iterative operations to learn a set of parameters for segmenting and classifying cells, the parameter set maximizing or minimizing an objective function, wherein each iteration involves finding the parameter set of the machine learning algorithm such that the value of the objective function using the parameter set is greater than or less than the value of the objective function using a different parameter set in a previous iteration, and wherein the objective function is constructed to measure the difference between a segmentation mask predicted using the machine learning algorithm and two or more exclusion-coded masks of the image; and Provide the machine learning model.

12. The method of claim 11, wherein the two or more exclusion coding masks comprise a positive mask for cells expressing the biomarker and a negative mask for cells not expressing the biomarker.

13. The method of claim 11, wherein the trained machine learning model comprises a convolutional neural network, the convolutional neural network comprising an encoder and a decoder, and one or more layers of the encoder comprising residual blocks having skip connections.

14. The method of claim 11, wherein the image is an image block of a predetermined size.

15. The method of claim 14, further comprising: The two or more exclusion-coded masks are divided into mask blocks of the predetermined size, and each image block in the image block is labeled with the mask blocks to generate a training image set.

16. A system comprising: One or more data processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform any one of the steps of the method according to any one of claims 11 to 15.

17. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform any one of the steps of the method according to any one of claims 11 to 15.

Images