Automated identification of necrotic areas in digital images of multiplex immunofluorescence stained tissue
By using the U-Net model and preprocessing techniques, necrotic regions in multiplex immunofluorescence images are automatically identified, solving the problem of difficult identification in existing technologies, improving the accuracy and efficiency of image analysis, and supporting more efficient disease diagnosis and treatment assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- VENTANA MEDICAL SYSTEMS INC
- Filing Date
- 2021-08-11
- Publication Date
- 2026-05-29
Smart Images
Figure CN116113983B_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims the benefit and patent priority of U.S. Provisional Application No. 63 / 065,350, filed August 13, 2020, which is incorporated herein by reference in its entirety for all purposes. Technical Field
[0003] This disclosure relates to digital pathology, and more particularly to techniques for depicting necrotic tissue regions in digital images of multiplex immunofluorescence stained tissues using machine learning models. Background Technology
[0004] Digital pathology involves scanning slides of samples (e.g., tissue samples, blood samples, urine samples, etc.) into digital images. Samples can be stained so that selected proteins (antigens) in the cells are visually differentiated relative to the rest of the sample. Target proteins in the sample are called biomarkers. For tissue samples, digital images can be generated depicting the sample using multiple staining agents targeting different biomarkers. Digital images depicting samples stained with multiple staining agents are called multiplex immunofluorescence images. Multiplex immunofluorescence images can visualize the spatial relationships between tumor cells and non-tumor cells in tissue samples.
[0005] Image analysis can be performed to identify and quantify biomarkers in tissue samples. Image analysis can be performed by a computational system or a pathologist to facilitate the characterization of biomarkers (e.g., in terms of presence, size, shape, and / or location) to inform, for example, the diagnosis of a disease, the determination of a treatment plan, or the assessment of the response to therapy.
[0006] Digital pathology algorithms can quantify the expression and co-expression of biomarkers and characterize their spatial relationships within the tumor microenvironment (TME) on a cell-by-cell basis. These algorithms typically rely on manual identification of tumor regions in whole-slide images (WSI) by pathologists. Necrotic regions are also manually annotated to exclude them from phenotypic detection and reporting. Necrotic regions are a series of morphological changes in living tissue following cell death. The variations in size, shape, and number of necrotic regions across WSIs make their manual annotation an error-prone and time-consuming task. Summary of the Invention
[0007] In some embodiments, a method is provided. Access a multiplex immunofluorescence image of a sample slice. The multiplex immunofluorescence image may include a first channel targeting nuclear markers and a second channel targeting epithelial tumor markers. The sample slice may include one or more necrotic tissue regions. A machine learning model (e.g., a U-Net model) is used to process the multiplex immunofluorescence image. The output of the processing may correspond to a prediction regarding whether the multiplex immunofluorescence image includes one or more necrotic tissue regions. This prediction may identify one or more specific portions of the multiplex immunofluorescence image predicted to depict necrotic tissue. A mask is generated based on the output of the machine learning model. The mask is output and used for subsequent image processing of the multiplex immunofluorescence image.
[0008] In some embodiments, nuclear markers include 4',6-diamidinyl-2-phenylindole (DAPI), and epithelial tumor markers include broad-spectrum cytokeratin (PanCK).
[0009] In some embodiments, the method includes: preprocessing the multiplex immunofluorescence images using gamma correction and normalization before feeding the multiplex immunofluorescence images to a machine learning model.
[0010] In some cases, subsequent image processing includes: generating a modified version of the multiplex immunofluorescence image by applying a mask to the multiplex immunofluorescence image; and processing the modified version of the multiplex immunofluorescence image. The output of the processed image corresponds to a depiction of the detection of a set of tumor cells in a sample slice, and characterizes the presence, number, and / or size of the set of tumor cells.
[0011] In some embodiments, the computer-implemented method may include generating multiple tile images from a multiplex immunofluorescence image. The multiple tile images may be provided to a machine learning model as a multiplex immunofluorescence image.
[0012] In some embodiments, subsequent image processing may further include: accessing a second image of the sample slice and applying a mask to the second image. The presence, number, and / or size of a group of tumor cells in the sample slice can be determined by processing the second image with the mask. The presence, number, and / or size of a group of tumor cells in the sample slice can be output.
[0013] Some embodiments of this disclosure include a system comprising one or more data processors. The system may further include a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods disclosed herein.
[0014] In some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium is provided. The computer program product may include instructions configured to cause one or more data processors to perform part or all of the methods disclosed herein.
[0015] The terms and expressions used are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features or portions thereof shown and described, but it should be recognized that various modifications can be made within the scope of the claimed invention. Therefore, it should be understood that while the invention has been specifically disclosed by way of examples and optional features, modifications and variations of the concepts disclosed herein can be made by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined in the appended claims. Attached Figure Description
[0016] Aspects and features of various embodiments will become more apparent from the description of examples with reference to the accompanying drawings, in which:
[0017] Figure 1 Exemplary depictions of necrotic tissue regions in multiplex immunofluorescence images according to some aspects of this disclosure are shown;
[0018] Figure 2 An exemplary computational system for training and using machine learning models to identify depictions of necrotic tissue, according to some aspects of this disclosure, is shown;
[0019] Figure 3 A block diagram of a process for predicting necrotic tissue regions according to some aspects of this disclosure is shown;
[0020] Figure 4 An exemplary U-Net is shown according to some aspects of this disclosure;
[0021] Figure 5 A diagram showing gamma correction according to some aspects of this disclosure is shown;
[0022] Figure 6 Exemplary results of preprocessing multiplex immunofluorescence images according to some aspects of this disclosure are shown;
[0023] Figure 7 Examples of image patches generated according to some aspects of this disclosure are shown;
[0024] Figure 8 Exemplary input images and corresponding masks according to some aspects of this disclosure are shown;
[0025] Figure 9Exemplary images are shown during the process of identifying necrotic tissue areas according to some aspects of this disclosure; and
[0026] Figure 10 An exemplary process for identifying necrotic tissue regions using a machine learning model, according to some aspects of this disclosure, is shown.
[0027] In the accompanying drawings, similar parts and / or features may have the same reference numerals. Furthermore, various parts of the same type can be distinguished by adding a dash after the reference numeral and a second reference numeral to differentiate similar parts. If only the first reference numeral is used in the description, the description applies to any similar parts having the same first reference numeral, regardless of the second reference numeral. Detailed Implementation
[0028] I. Overview
[0029] While certain embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The devices, methods, and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and variations may be made to the form of the exemplary methods and systems described herein without departing from the scope of protection.
[0030] This disclosure describes techniques for automatically identifying depictions of necrotic tissue in digital pathology images. More specifically, some embodiments of this disclosure provide machine learning techniques for identifying depictions of necrotic tissue in digital pathology images to support or improve disease detection and analysis.
[0031] Multiplex immunofluorescence staining of slides allows for the simultaneous detection of multiple proteins in tissue sections. Multiplex immunofluorescence images can be used to study different types of cells in biological tissues, blood, or urine. Biological tissue biopsies can be collected, fixed (e.g., using formaldehyde solution), and embedded relative to the tissue, after which the sample can be cut into smaller sections. Each section can be applied to a glass slide for staining and analysis. During staining, a primary antibody and one or more types of secondary antibodies can be applied to the sample to amplify the immunofluorescence signal. Staining agents can include biomarkers targeting different biological materials, such as biomarkers for cell nuclei and biomarkers for epithelial tumor cells. After applying each of the primary and secondary antibodies, the slide can be scanned to generate an image of the slide. The slide can be washed between the application of the primary and secondary antibodies. Images can be combined after the staining process to generate multiplex immunofluorescence images. Alternatively, two antibodies can be applied and a single image can be taken.
[0032] During imaging and analysis, regions in multiplex immunofluorescence images can be segmented into target regions (e.g., tumor cells) and non-target regions (e.g., normal tissue or blank slide areas). Each target region may include a target area that can be characterized and / or quantified. In some cases, multiplex immunofluorescence images may include biological material or other structures that do not correspond to fluorescent target regions, making it difficult to distinguish between target and non-target regions. For example, necrotic tissue may lead to incorrect detection of cell nuclei in the DAPI channel due to fluorescence of the necrotic tissue, misclassification of individual markers, and incorrect segmentation of tumor areas.
[0033] Figure 1 An exemplary depiction of a necrotic tissue region is shown in a multiplex immunofluorescence image. Image 100 shows tissue stained with hematoxylin and eosin (H&E) staining agents, and multiplex immunofluorescence image 110 shows the fluorescence multiplicity of the tissue. The tissue includes tumor cells 104 and necrotic regions 102. While a pathologist can attempt to avoid annotating the necrotic tissue region 102, a pathologist can annotate the target region (e.g., tumor cells 104) in the multiplex immunofluorescence image 110. Alternatively, a pathologist can annotate the necrotic region 102, causing further analysis of the multiplex immunofluorescence image to exclude the necrotic region 102 from the analysis. These annotations can be error-prone and time-consuming when dealing with a large number of slides.
[0034] Automatic identification and segmentation of necrotic tissue from multiplex immunofluorescence images can provide more accurate and faster differentiation between necrotic tissue regions and target regions, leading to more efficient and better assessment of disease (e.g., cancer and / or infectious diseases) diagnosis and treatment. The staining agents used in multiplex immunofluorescence images may stain both the target biomarkers and the necrotic tissue, making detection and segmentation difficult or inaccurate. Therefore, conventional image analysis algorithms often provide over-detection, under-detection, or inaccurate detection of necrotic regions in multiplex immunofluorescence images.
[0035] In some embodiments, a machine learning model is provided to predict the location of necrotic tissue in multiplex immunofluorescence images. In some cases, the machine learning model has a convolutional neural network (CNN) architecture that utilizes a U-Net model to automatically identify depictions of necrotic tissue in multiplex immunofluorescence images. The machine learning model can be trained using training images comprising a set of positive data elements (images depicting necrotic tissue) and a set of negative data elements (images not depicting necrotic tissue). Each positive data element may include one or more indications about where each region of necrotic tissue is depicted within the image. Training the machine learning model may include, for example, learning necrotic tissue characteristics or signatures (e.g., intensity and texture features). Training may also, or alternatively, include learning how to modify the image based on the characteristics of the necrotic tissue (e.g., removing or otherwise modifying the depicted necrotic tissue). The trained machine learning model can be used to predict specific portions of necrotic tissue depicted in multiplex immunofluorescence images. In some examples, the machine learning model can be trained against multichannel data so that the trained machine learning model can process multiplex immunofluorescence images of samples stained with multiple staining agents. Staining agents may include: nuclear markers, such as 4',6-diamidinyl-2-phenylindole (DAPI); and epithelial tumor markers, such as broad-spectrum cytokeratin (PanCK). Machine learning models can be trained against additional or alternative staining agents. In some cases, as part of preprocessing, a machine learning model is executed, followed by image analysis algorithms to detect and / or characterize depictions of other cell types (e.g., tumor cells, immune cells, or leukocytes). However, as will be understood by those skilled in the art, the concepts discussed herein are not limited to processing procedures but can also be integrated into overall image analysis processing according to various embodiments.
[0036] Outputs depicting specific portions of necrotic tissue indicative of multiplex immunofluorescence images can be generated. In some examples, machine learning models can modulate multiplex immunofluorescence images based on predictions and learned tuning parameters. For instance, a mask can be generated based on the predicted portion and applied to the multiplex immunofluorescence image in subsequent image processing. Subsequent image processing may include: processing one or more multiplex immunofluorescence images and / or modified versions thereof (e.g., processed modified versions of multiplex immunofluorescence images, processing one or more other multiplex immunofluorescence images and / or processing one or more modified versions of other multiplex immunofluorescence images); and outputting results. The results can characterize the presence, location, number, and / or size of a group of tumor cells in the multiplex immunofluorescence image. The results can be used to determine diagnoses, treatment plans, or evaluate ongoing treatments targeting tumor cells.
[0037] II. Definitions and Abbreviations
[0038] As used in this article, when an action is “based on” something, it means that the action is based at least in part on at least a part of something.
[0039] As used herein, the terms “substantially,” “about,” and “approximately” are defined as substantially but not necessarily exactly as specified (and include exactly as specified), as understood by one of ordinary skill in the art. In any disclosed embodiment, the terms “substantially,” “about,” or “approximately” may be replaced with “within [a certain percentage]” for the specified meaning, where percentages include 0.1%, 1%, 5%, and 10%.
[0040] As used herein, the terms “sample,” “biological sample,” or “tissue sample” refer to any sample obtained from any organism, including viruses, that includes biomolecules such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof. Examples of other organisms include mammals (such as humans; mammals such as cats, dogs, horses, cattle, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (e.g., tissue sections and needle biopsies of tissues), cell samples (e.g., cytological smears such as cervical smears or blood smears or cell samples obtained through microdissection), or cell fractions, fragments, or organelles (e.g., obtained by lysing cells and separating their components by centrifugation or other methods). Other examples of biological samples include blood, serum, urine, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), cerumen, breast milk, saliva, swabs (e.g., oral swabs), or any material containing biomolecules derived from the first biological sample. In some embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject (e.g., a homogenized or liquefied sample).
[0041] As used herein, the term "biomaterial or structure" refers to natural materials or structures, including whole or part of living structures (e.g., cell nucleus, cell membrane, cytoplasm, chromosomes, DNA, cells, cell clusters, etc.).
[0042] As used herein, the term "non-target region" refers to a region of an image containing image data that is not intended for evaluation in image analysis processing. A non-target region may include non-tissue regions of an image corresponding to a substrate (such as glass without a sample), where only white light from the imaging source is present. A non-target region may additionally or alternatively include tissue regions of an image that correspond to biological material or structures that are not intended for analysis during image analysis or that are difficult to distinguish from biological material or structures (e.g., stromal cells, normal cells, scanning artifacts) within the target region.
[0043] As used herein, the term "target region" refers to a region of an image that includes image data intended for evaluation in image analysis processing. Target regions include any region of an image, such as a tissue region (e.g., tumor cells), intended for analysis during image analysis.
[0044] As used herein, the term "tile" or "tile image" refers to a single image corresponding to the entire image or a portion of the entire slide. In some embodiments, a "tile" or "tile image" refers to an area scanned across the entire slide or a target area having a (x, y) pixel size (e.g., 1000 pixels x 1000 pixels). For example, consider dividing the entire image into M columns and N rows of tiles, where each tile in an M x N stitch comprises a portion of the entire image; that is, the tile at positions M1, N1 comprises a first portion of the image, while the tile at position M3, N4 comprises a second portion of the image, and the first and second portions are distinct. In some embodiments, the tiles may each have the same scale (pixel size x pixel size).
[0045] As used herein, the term "multiplex immunofluorescence image" refers to a full slide image or a patch of full slide images of a biological sample (e.g., tissue section, blood smear, or urine smear) stained for multiple biomarkers. Each of the biomarkers emits fluorescence at a different wavelength when excited, allowing for unique identification of each biomarker.
[0046] As used herein, the term "sample slide" refers to any biological sample (such as a tissue slide, tissue biopsy, blood, and urine) derived from an organism. Examples of organisms include humans, veterinary animals, and laboratory animals. In some embodiments, the term "sample slide" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject.
[0047] III. Computing Environment
[0048] Figure 2An exemplary computational system 200 is illustrated for training and using a necrotic tissue detection machine learning model to identify necrotic depictions and for processing digital pathology images. The computational system 200 may include an analysis system 205 for training and executing the necrotic tissue detection machine learning model. Examples of necrotic tissue detection machine learning models include convolutional neural networks, deep neural networks, U-Net, V-Net, residual neural networks, and / or recurrent neural networks. The necrotic tissue detection machine learning model may be trained and / or used to predict, for example, whether a multiplex immunofluorescence image includes a depiction of one or more necrotic tissue regions. The analysis system 205 may be further trained and / or used with one or more other machine learning models to perform another type of detection (e.g., depicting tumor cells). One or more other machine learning models may include, for example, convolutional neural networks, U-Net, V-Net, and / or deep neural networks. In some cases, the results from the necrotic tissue detection machine learning model are used to process the image (e.g., to remove or mask each necrotic tissue region), and one or more other machine learning models may receive and use the processed image to perform other types of detection. In some cases, the results from the necrotic tissue detection machine-learning model are used to identify image patches and / or images that should be excluded when performing other types of detection.
[0049] In some embodiments, the generated multiplex immunofluorescence images are stored in a memory device. Imaging apparatus 235 can be used to generate multiplex immunofluorescence images. In some embodiments, as described herein, the images are generated or obtained from a microscope or other instrument capable of capturing image data of a microscope slide carrying a sample. In some embodiments, the multiplex immunofluorescence images are generated or obtained using a 2D scanner, such as a scanner capable of scanning image patches. Alternatively, the multiplex immunofluorescence images may have been previously generated (e.g., scanned) and stored in a memory device (or, in this regard, retrieved from a server via a communication network).
[0050] In some cases, the training controller 210 determines or learns preprocessing parameters and / or techniques for multiplex immunofluorescence images. For example, preprocessing may include dividing the multiplex immunofluorescence image into patches and modifying the patches. Each of the patches may be divided into one or more image blocks. Image blocks may be relatively small in size (e.g., 512x512). Image blocks may be selected based on features identified in the image blocks (e.g., edges). The selected image blocks may correspond to the multiplex immunofluorescence image and be processed using a trained necrotic tissue detection machine learning model. The analysis system 205 may select one or more image blocks of a full slide image for processing using the necrotic tissue detection machine learning model. Preprocessing modifications may include gamma correction of the multiplex immunofluorescence image or patches. Gamma correction can reduce the bit range information of the multiplex immunofluorescence image from 16 bits to 8 bits, which can produce pixel intensities from 0 to 255. Other exemplary preprocessing may include normalizing the gamma-corrected multiplex immunofluorescence image by dividing each pixel by 255 to make the overall intensity range between 0 and 1.
[0051] Training controller 210 can execute code to train a necrotic tissue detection machine learning model and / or one or more other machine learning models using one or more training datasets 215. Each training dataset 215 may include a set of training multiplex immunofluorescence images. Each of the multiplex immunofluorescence images may include a digital pathology image depicting one or more biological objects (e.g., a group of cells or one or more types of cells). The training data may depict cells using, for example, nuclear markers (e.g., DAPI, propidium iodide, etc.). Hoechst or The sample is stained with staining agents and epithelial tumor markers (e.g., PanCK, epithelial cell adhesion molecule (EpCAM), or tissue polypeptide antigen (TPA)). Each image in a first subgroup of a set of training multiplex immunofluorescence images may include a depiction of one or more necrotic tissue regions, and each image in a second subgroup of a set of training multiplex immunofluorescence images may omit the depiction of necrotic tissue regions (e.g., no necrotic tissue regions may be depicted). Each of the multiplex immunofluorescence images may depict a portion of a sample (such as a tissue sample (e.g., colorectal, bladder, breast, pancreas, lung, or stomach tissue)). In some cases, each of one or more multiplex immunofluorescence images depicts multiple tumor cells. Training data 215 may have been collected (e.g., from image source 220, such as a public data repository) and / or from data received (e.g., from one or more laboratories).
[0052] The computing system 200 may include a label mapper 225 that maps multiplex immunofluorescence images from image source 220 depicting necrotic tissue regions to a "necrotic tissue" label and multiplex immunofluorescence images not depicting necrotic tissue to a "non-necrotic" label. The labels may be image-level labels indicating whether necrosis is depicted in the multiplex immunofluorescence image. Alternatively, the labels may be data indicating the location of necrosis depicted in the multiplex immunofluorescence image. The mapping data may be stored in a mapping data repository (not shown). The mapping data can identify each multiplex immunofluorescence image mapped to either the necrotic tissue label or the non-necrotic label.
[0053] In some cases, labels associated with training multiplex immunofluorescence images may have been received or may be derived from data received from one or more provider systems 230, each of which may be associated with (e.g.) a pathologist, imaging scientist, physician, nurse, hospital, pharmacist, laboratory, etc., associated with a particular subject. The received data may include (e.g.) one or more medical records corresponding to a particular subject. The medical records may indicate (e.g.) a professional diagnosis or characterization of the time period corresponding to the collection of one or more input image elements associated with the subject, or a subsequently defined time period indicating whether the subject has a tumor and / or the stage of progression of the subject's tumor (e.g., along standard scales and / or by identifying metrics such as total metabolic tumor volume (TMTV)). The received data may further identify the location of tumors or tumor cells within one or more multiplex immunofluorescence images associated with the subject (e.g., by identifying point pixels for each tumor or tumor cell or the boundaries for each tumor or tumor cell). In some cases, images or scans that are input to one or more classifier subsystems are received from provider system 230. For example, provider system 230 can receive images from imaging device 235 and can then transmit the images or scans (e.g., together with subject identifiers and one or more tags) to analysis system 205.
[0054] Training controller 210 can use the mapping of training data 215 to train a necrotic tissue detection machine learning model. More specifically, training controller 210 has access to the architecture of the model (e.g., a U-Net model), defines (fixed) hyperparameters for the model (the hyperparameters being parameters that affect the learning process, such as the model's learning rate, size / complexity, etc.), and trains the model to learn a set of parameters. More specifically, the set of parameters can be learned by identifying parameter values associated with low or minimal loss, cost, or error generated by comparing the predicted output (obtained using given parameter values) with the actual output. The mask prediction performance of the necrotic tissue detection machine learning model can be trained and evaluated by training controller 210 using the sum of the binary cross-entropy loss and the Dice coefficients.
[0055] In some cases, the necrotic tissue detection machine learning model can be configured to retrain or fine-tune the model to improve the accuracy of the output estimation (e.g., it includes a metric or identifier corresponding to the estimate or likelihood of the portion depicting necrotic tissue in the multiplex immunofluorescence image). Furthermore, the necrotic tissue detection machine learning model can be trained to modulate the multiplex immunofluorescence image by removing or masking portions predicted to depict necrotic tissue. Training data 215 may include the input multiplex immunofluorescence image and the modulated image based on the output. For example, the necrotic tissue detection machine learning model can learn to filter or remove necrotic tissue regions in the multiplex immunofluorescence image. Pixels identified as part of a necrotic tissue region may be set to an intensity of 0 or an average or median intensity value determined using the intensities of other pixels.
[0056] The machine learning (ML) execution processor 240 can use this architecture and learned parameters to process non-training data and generate results. For example, the ML execution processor 240 can access multiplex immunofluorescence images of sample slices that are not represented in the training data 215. Multiplex immunofluorescence images can depict samples stained with the same staining agents as the training data or different staining agents. For example, multiplex immunofluorescence images can depict nuclear markers (e.g., DAPI, propidium iodide, TO-). Hoechst or Multiplex immunofluorescence images may include staining agents and epithelial tumor markers (e.g., PanCK, EpCAM, or TPA). As an example, multiplex immunofluorescence images may include 5-Plex assays targeting differentiation cluster 3 (CD3), differentiation cluster 8 (CD8), differentiation cluster 68 (CD68), programmed death-ligand 1 (PD-L1), and PanCK (represented by cyanine 5 (Cy5), rhodamine 6G, rhodamine 610, carboxyfluorescein (FAM), and N,N'-dicyclohexylcarbodiimide (DCC) fluorophores and DAPI as counterstains). Multiplex immunofluorescence images may (but do not necessarily) be preprocessed according to learned or recognized preprocessing techniques. (Preprocessed or raw) multiplex immunofluorescence images may be fed into a necrotic tissue detection machine learning model with an architecture (e.g., U-Net) that is used and configured with learned parameters during training. The necrotic tissue detection machine learning model can output the following prediction: one or more specific portions of a multiplex immunofluorescence image contain a depiction of necrotic tissue cells. The necrotic tissue depiction can include a depiction of an area where at least 30%, at least 50%, at least 75%, or at least 90% of the cells appear to be undergoing necrosis. For each pixel, the output can include the probability that the pixel is part of a depiction of necrotic tissue cells. Alternatively, the output can be the coordinates of a rectangular region predicted to depict necrosis. Another component of the ML execution processor 240 or analysis system 205 can post-process the output. For example, if a probability is output for each pixel representing the probability that the pixel is part of a depiction of necrotic tissue cells, filtering can be applied to smooth the values. The ML execution processor 240 can output the prediction for subsequent analysis.
[0057] Mask generator 245 can use predictions to generate masks for multiplex immunofluorescence images. Where the output from the necrotic tissue detection machine learning model includes the probability that each pixel represents a portion of necrotic tissue cells, mask generation can include comparing each probability to a predefined threshold (e.g., 50%) to convert each probability into a binary value. The mask can be a binary image, where each value of 1 represents a prediction that the corresponding pixel represents a portion of necrotic cells, and each value of 0 represents a prediction that the corresponding pixel does not represent a portion of necrotic cells (or that the corresponding pixel does not represent a portion of a tissue slice). The mask can be used in subsequent image processing to remove or otherwise segment predicted necrotic tissue regions in the multiplex immunofluorescence image.
[0058] In some cases, subsequent image processing is performed by an image processor (not shown). Subsequent image processing may be based on predictions of depicting necrotic tissue from a multiplex immunofluorescence image. For example, subsequent image processing may include determining that at least one metric corresponding to a specific portion (e.g., one or more pixels) of the multiplex immunofluorescence image exceeds a predefined threshold. This determination may be made for each of one or more regions of the multiplex immunofluorescence image, where each of the one or more regions is predicted to depict necrotic tissue. Examples of metrics may include the cumulative size of the necrotic tissue (e.g., the number of pixels depicting at least a portion of the necrotic tissue region), the number of necrotic tissue regions, and / or the probability that the multiplex immunofluorescence image includes at least one necrotic tissue region. For example, the predefined threshold for the probability of depicting necrotic tissue for a specific portion may be 70%. Subsequent image processing may be based on determining that a specific portion of the multiplex immunofluorescence image has a probability greater than 70% of depicting necrotic tissue.
[0059] In some cases, an image processor can generate a modified version of a multiplex immunofluorescence image based on predictions. Generating a modified version may include applying a mask to the multiplex immunofluorescence image or applying other image enhancement techniques to each predicted specific portion of the multiplex immunofluorescence image depicting necrotic tissue cells. Applying a mask may include multiplying the mask value by the value of the multiplex immunofluorescence image for each pixel in the image. The modified version of the multiplex immunofluorescence image can then be processed using another machine learning model.
[0060] In some cases, subsequent image processing includes modifying one or more additional images based on the location in a multiplex immunofluorescence image predicted to depict a specific portion of necrotic tissue. These additional images may correspond to the same or different slices of the sample. Modifying these additional images may include applying masks, filtering, or segmenting the images at locations in other slices of the sample that correspond to the location of the specific portion in a mask-based multiplex immunofluorescence image. One or more machine learning models can then be used to process these additional images to determine the predicted characterization of the sample.
[0061] In some cases, image characterizer 250 identifies predicted characterizations for multiplex immunofluorescence images based on the execution of subsequent image processing. The execution itself may produce results that include characterizations, or the execution may include results that classifier 250 can use to determine the predicted characterization of a sample. For example, subsequent image processing may include detecting a depiction of a set of tumor cells. The results may characterize the presence, number, and / or size of a set of tumor cells. Image characterizer 250 may apply rules and / or transformations to map probabilities and / or confidence levels to characterizations. As an illustration, if the result including tumor corresponds to a greater than 50% probability that the multiplex immunofluorescence image includes a set of tumor cells, a second characterization may be assigned in another manner.
[0062] The communication interface 255 can collect results and transmit one or more results (or a processed version thereof) to a user device or other system. For example, the communication interface 255 can generate output identifying the presence, number, and / or size of a group of tumor cells. This output can then be presented and / or transmitted, which may facilitate the display of the output data, such as on a display of a computing device.
[0063] IV. Exemplary Process
[0064] Figure 3 A block diagram illustrating a process for predicting necrotic tissue areas according to some aspects of this disclosure is shown. This can be achieved from a provider system or imaging device (such as...) Figure 2 The provider system 230 or imaging device 235 receives multiplex immunofluorescence images 305. Multiplex immunofluorescence images 305 can depict a sample slice including at least two channels. The channels can include at least nuclear markers and epithelial tumor markers.
[0065] In some cases, the multiplex immunofluorescence image is preprocessed 310. Preprocessing 310 may involve, for example, performing gamma correction on the multiplex immunofluorescence image 305 to bring the intensity range of pixels in the multiplex immunofluorescence image 305 into a predefined range (e.g., between 0 and 255). Additionally, preprocessing 310 may involve normalizing the intensity values by dividing the intensity of each gamma-corrected pixel by the upper limit of the predefined range (e.g., 255), resulting in pixel intensity ranges between 0 and 1. Preprocessing 310 may also include normalizing and / or standardizing values (in the absence of gamma correction), cropping the multiplex immunofluorescence image, or filtering the image (e.g., to depict the intensity of a subset across wavelengths or colors).
[0066] The preprocessed multiplex immunofluorescence images can then be classified 315. Classification 315 may involve feeding the preprocessed immunofluorescence images into a trained necrotic tissue detection machine learning model (e.g., U-Net), which can generate predictions of specific portions of necrotic tissue depicted in sample slices. The predictions can be the probability of depicting necrotic tissue for each pixel. Alternatively, the predictions can be a binary classification performed for each pixel. Classification 315 can be performed by an ML execution processor (such as...) Figure 2 The ML execution processor 240 in the middle performs the execution.
[0067] In some cases, masking 320 can be performed after classification 315 of the preprocessed multiplex immunofluorescence image. Masking 320 may include generating a mask for the multiplex immunofluorescence image 305 based on predictions from a machine learning model for necrotic tissue detection. The mask may have different values for regions predicted to depict necrotic tissue and regions predicted not to depict necrotic tissue. The mask can then be used in subsequent image processing to further classify the multiplex immunofluorescence image 305, such as by identifying the presence and characteristics of tumor cells in sample slices.
[0068] V. Exemplary U-Net Model
[0069] like Figure 4 As shown, U-Net 400 can include a shrinking path 405 (encoder) and an expanding path 410 (decoder), giving U-Net a U-shaped architecture. The shrinking path 405 is a CNN network that includes repeated applications of convolutions (e.g., 3x3 convolutions (unpadded convolutions)), each followed by a Rectified Linear Unit (ReLU) and a max-pooling operation for downsampling (e.g., 2x2 max-pooling with a stride of 2). The number of feature channels can be doubled in each downsampling step or pooling operation. During shrinking, the spatial information of the image data decreases, while the feature information increases. The expanding path 410 is a CNN network that combines the features and spatial information (upsampling of the feature maps from the shrinking path 405) from the shrinking path 405. The upsampling of the feature map is followed by a series of upconvolutions (upsampling operators) that halve the number of channels, concatenation with the corresponding cropped feature map from the contraction path 405, repeated application of convolutions (e.g., two 3x3 convolutions) followed by each rectified linear unit (ReLU), and a final convolution (e.g., a 1x1 convolution) to generate a two-dimensional non-target region mask. For localization, the high-resolution features from the contraction path 405 are combined with the upsampled output from the expansion path 410.
[0070] In various embodiments, U-Net 400 implements a generally reduced number of channels compared to the conventional U-Net architecture. Specifically, the number of channels in the intermediate activated output layers is reduced by a predetermined factor, such as two or four (see, for example, the bold numbers within shrinking path 405 and expanding path 410). For instance, in the second layer, the number of channels is reduced by a factor of four from 64 to 16, and the maximum number of channels is also reduced by a factor of four to 256, instead of the 1024 used in the conventional U-Net architecture. This reduction in channels is implemented to reduce computational cost and model complexity. This compressed U-Net architecture provides better results compared to the conventional U-Net architecture with a maximum of 1024 channels and no reduction in channel number. Furthermore, U-Net 400 includes spatially randomized deactivation 415 to combat overfitting. Spatially randomized deactivation 415 can be implemented in the last few layers of shrinking path 405 (i.e., the last layer, the second layer, the third layer, or the fourth layer). Spatial random deactivation 415 deactivates the entire 2D feature map, rather than deactivating individual elements as conventional deactivation would. For example, if neighboring pixels in the feature map are strongly correlated (as is common in early convolutional layers), conventional deactivation does not regularize the activations, which would otherwise lead to a decrease in the effective learning rate. In contrast, spatial random deactivation 415 will regularize the activations and help promote independence between feature maps, which would otherwise lead to an increase in the effective learning rate.
[0071] VI. Exemplary Preprocessing Techniques
[0072] Figure 5 Figure 500 illustrates gamma correction according to some aspects of this disclosure. In digital imaging, there is a linear relationship between the luminance input and the detected light output, as shown by line 510. For example, when a digital imaging system receives three times the number of photons impacting a sensor, the detected light is three times brighter. In contrast, the eye perceives only a fraction of the three times the light as bright, as shown by line 520. Gamma correction can be performed to convert the bit usage of the image between the eye's sensitivity to light and the sensitivity of the digital imaging system.
[0073] Gamma correction can be defined as luminance 出 =Brightness 入 γ A γ of 0.5 can be used for multiplex immunofluorescence images with a 16-bit image range to achieve an 8-bit image range, where pixel intensities range from 0 to 255. Smaller luminance values can be assigned higher weights, and higher luminance values can be assigned lower weights. For example, 25% of the input luminance can be mapped to 50% of the detected output light, and using a γ of 0.5, 75% of the input luminance can be mapped to 86% of the detected output light.
[0074] Figure 6 Exemplary results of preprocessed multiplex immunofluorescence images according to some aspects of this disclosure are shown. Image 605 (DAPI channel) and image 610 (PanCK channel) are shown. Figure 5 As described, gamma correction is performed on each of the DAPI channel image 605 and the PanCK channel image 610, resulting in gamma-corrected images 615 and 620. Further normalization preprocessing can then be applied to the gamma-corrected images 615 and 620 to generate normalized images 625 and 630. Normalized images 625 and 630 can have pixel intensity values ranging from 0 to 1. Normalized images 625 and 630 can then be fed into a necrotic tissue detection machine learning model to predict image portions depicting necrotic tissue regions. Ground truth mask images 635 and 640 represent images manually generated by a pathologist that separate necrotic tissue depictions from non-necrotic tissue depictions. Brighter portions of ground truth mask images 635 and 640 can represent depictions of necrotic tissue regions, and darker portions can represent depictions of non-necrotic tissue regions, and vice versa.
[0075] Figure 7 Examples of image patch generation according to some aspects of this disclosure are shown. In some cases, a whole-slide multiplex immunofluorescence image can be divided into image patches. Image patch generation can be performed during preprocessing. Image patches with a fixed size of 705 (e.g., 512x512 pixels) can be used. Image patches can also be overlapping, with a set stride of 710. A sliding window can be used to generate image patches of a fixed size of 705. After generating an image patch, the sliding window can be moved by the set stride of 710 to position the next image patch. During the training of a necrosis tissue detection machine learning model that predicts the depiction of necrotic tissue regions in multiplex immunofluorescence images, image patches and corresponding labels can be randomly selected and used as training images. The trained necrosis tissue detection machine learning model can receive image patches instead of whole slide images to reduce the computational requirements of the trained necrosis tissue detection machine learning model.
[0076] Figure 8 Exemplary input images 800 and 802, and corresponding necrosis masks 825 and 830, are shown according to some aspects of this disclosure. Input images 800 and 802 include DAPI channel images 805 and 810 and PanCK channel images 815 and 820, respectively. The necrosis mask 825 is generated by a trained necrosis tissue detection machine learning model to predict the depiction of necrotic tissue regions. Brighter portions of the necrosis masks 825 and 830 may represent depictions of necrotic tissue regions, and darker portions may represent depictions of non-necrotic tissue regions, and vice versa.
[0077] VII. Exemplary Use Cases
[0078] Figure 9 Exemplary images are shown during the process of identifying a delineation of necrotic tissue areas according to some aspects of this disclosure. These images can be generated by an analysis system (e.g., Figure 2 The analysis system 205 accesses the full slide image 905. The full slide image 905 can be preprocessed with gamma correction and / or normalization to reduce the pixel intensity range of the full slide image 905. In some cases, the preprocessing also involves tile processing 910 on the full slide image 905. Tile processing 910 may involve dividing the full slide image 905 into non-overlapping portions of a fixed size (e.g., 1000 pixels × 1000 pixels). Image patches can be derived from each patch, and each image patch can have a fixed size smaller than the fixed size of the patch. For example, each image patch can be 512 pixels × 512 pixels.
[0079] The tiles can then be individually input into the U-Net model 915. The U-Net model 915 can be trained to predict the location of necrotic tissue depicted in multiplex immunofluorescence images. In some cases, the U-Net model 915 can receive only tile image patches to be processed. The U-Net model 915 can output predictions of specific portions of the necrotic tissue depicted in the tile (or image patch). Mask generators (e.g., Figure 2 The mask generator 245 can receive the output of the U-Net model 915 and generate predictive mask patches 920, which include a first binary value for the portion of the patch predicted to depict necrotic tissue and a second binary value for the portion of the patch predicted not to depict necrotic tissue. The mask generator can then stitch the predictive mask patches 920 together to generate a full slide image mask 925 corresponding to the full slide image 905. The full slide image mask 925 can then be used to remove or modify the portions of the full slide image 905 predicted to depict necrotic tissue during the identification and classification of target regions in the full slide image 905.
[0080] Figure 10 An exemplary procedure 1000 is illustrated using a necrotic tissue detection machine learning model to identify depictions of necrotic tissue regions, according to some aspects of this disclosure. At box 1005, a multiplex immunofluorescence image of a sample slice is accessed. The multiplex immunofluorescence image may correspond to a whole slide image or an image patch of a whole slide depicting one or more necrotic tissue regions. The multiplex immunofluorescence image may include at least two channels (e.g., DAPI and PanCK) for determining the characterization of a group of tumor cells. Preprocessing of the multiplex immunofluorescence image, including gamma correction and normalization, may be performed.
[0081] At box 1010, the necrotic tissue detection machine learning model processes multiplex immunofluorescence images. The necrotic tissue detection machine learning model can be trained using multiplex immunofluorescence images of sample slices, including slices with and without necrotic tissue. The necrotic tissue detection machine learning model can be a deep neural network and / or a convolutional neural network. The necrotic tissue detection machine learning model can include a U-Net model.
[0082] At box 1015, the output of the necrotic tissue detection machine learning model is received. The output of the necrotic tissue detection machine learning model can be a prediction of a specific portion (e.g., one or more pixels) of necrotic tissue depicted in a multiplex immunofluorescence image. In addition, the output can include the determination of a metric corresponding to the specific portion of the multiplex immunofluorescence image (e.g., the cumulative size of the necrotic tissue, the probability of depicting necrotic tissue). This determination can be made for each of one or more regions of the multiplex immunofluorescence image.
[0083] At box 1020, a mask is generated based on the output of the necrotic tissue detection machine learning model. The mask can be an image with different values associated with pixels that have a prediction of necrotic tissue with a value above a threshold (e.g., 50%) and other pixels that have a prediction of necrotic tissue with a value below the threshold. The mask can be binary, or it can include various values corresponding to different predictions of necrotic tissue.
[0084] At box 1025, an output mask is used for subsequent image processing. One or more trained machine learning models or other image analysis techniques may be used for subsequent image processing. Subsequent image processing may include: modifying a multiplex immunofluorescence image of a sample slice or another multiplex immunofluorescence image, and processing a modified version of the multiplex immunofluorescence image. In some cases, subsequent image processing may include detecting a depiction of a group of tumor cells. The results of subsequent image processing may be transferred to another device (e.g., associated with a care provider) and / or displayed. The results may correspond to a predicted characterization of the sample. The results may characterize the presence, number, and / or size of a group of tumor cells. For multiplex immunofluorescence images, the results may characterize more than one group of tumor cells. The results may be used for diagnostic or treatment assessment of the subject.
[0085] Figure 10 This illustrates an exemplary process for using a machine learning model to identify necrotic tissue regions in a sample slice. Other examples may include more steps, fewer steps, different steps, or steps in a different order. For instance, necrotic tissue identification may occur after tumor detection and classification, as a post-processing step to exclude identified necrotic tissue regions.
[0086] VIII. Additional Considerations
[0087] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of the one or more methods and / or part or all of the one or more processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform part or all of the one or more methods and / or part or all of the one or more processes disclosed herein.
[0088] The terms and expressions used are descriptive rather than restrictive, and their use is not intended to exclude any equivalents of the features or portions thereof shown and described, but it should be recognized that various modifications can be made within the scope of the claimed invention. Therefore, it should be understood that while the claimed invention has been specifically disclosed by way of examples and optional features, modifications and variations of the concepts disclosed herein can be made by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined in the appended claims.
[0089] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of preferred exemplary embodiments will provide those skilled in the art with a feasible description for implementing various embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.
[0090] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.
Claims
1. A method for identifying necrotic tissue in digital pathology images, comprising: Access multiple immunofluorescence images of sample sections including a first channel targeting nuclear markers and a second channel targeting epithelial tumor markers, wherein the sample sections include one or more necrotic tissue regions; The multiple immunofluorescence images are provided to the machine learning model; The output of the machine learning model is received, and the output corresponds to the following prediction: the multiplex immunofluorescence image includes one or more necrotic tissue regions at one or more specific portions of the multiplex immunofluorescence image; Based on the output of the machine learning model, a mask is generated for subsequent image processing of the multiplex immunofluorescence images; as well as The mask is output for the subsequent image processing. The subsequent image processing includes: A modified version of the multiplex immunofluorescence image is generated by applying the mask to the multiplex immunofluorescence image; The modified version of processing the multiplex immunofluorescence images; and The output is a modified version of the processed multiplex immunofluorescence image, the result corresponding to a depiction of the detection of a group of tumor cells in the sample slice, wherein the result of subsequent image processing characterizes the presence, number and / or size of the group of tumor cells.
2. The method according to claim 1, wherein the nuclear marker comprises 4',6-diamidinyl-2-phenylindole (DAPI), and the epithelial tumor marker comprises broad-spectrum cytokeratin (PanCK).
3. The method according to claim 1, wherein the machine learning model includes the U-Net model.
4. The method of claim 1, further comprising: Before feeding the multiplex immunofluorescence images to the machine learning model, the multiplex immunofluorescence images are preprocessed using gamma correction and normalization.
5. The method of claim 1, further comprising: Multiple patch images are generated from the multiplex immunofluorescence images; as well as The multiple patch images are provided to the machine learning model as the multiplex immunofluorescence images.
6. The method of claim 1, wherein the subsequent image processing comprises: Access the second image of the sample slice; Apply the mask to the second image; The presence, number, and / or size of a group of tumor cells in the sample slice are determined by processing the second image with the mask; as well as Output the presence, number, and / or size of the group of tumor cells in the sample slice.
7. A system for identifying necrotic tissue in digital pathology images, comprising: One or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations including: Receive multiple immunofluorescence images of sample slices including at least two channels, wherein the multiple immunofluorescence images include one or more necrotic tissue regions; The multiplex immunofluorescence images were preprocessed using gamma correction; Preprocessed multiplex immunofluorescence images are fed into a machine learning model to identify the one or more necrotic tissue regions; Based on the output of the machine learning model, a mask is generated for subsequent image processing of the multiplex immunofluorescence images; as well as The mask is output for the subsequent image processing. The subsequent image processing includes: A modified version of the multiplex immunofluorescence image is generated by applying the mask to the multiplex immunofluorescence image; The modified version of processing the multiplex immunofluorescence images; and The output is a modified version of the processed multiplex immunofluorescence image, the result corresponding to a depiction of the detection of a group of tumor cells in the sample slice, wherein the result of subsequent image processing characterizes the presence, number and / or size of the group of tumor cells.
8. The system of claim 7, wherein the at least two channels comprise a 4',6-diamidinyl-2-phenylindole (DAPI) channel and a broad-spectrum cytokeratin (PanCK) channel.
9. The system of claim 7, wherein the machine learning model includes the U-Net model.
10. The system of claim 7, wherein the non-transitory computer-readable storage medium further comprises instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations including: Before feeding the multiplex immunofluorescence images to the machine learning model, the multiplex immunofluorescence images are preprocessed using gamma correction and normalization.
11. The system of claim 7, wherein the non-transitory computer-readable storage medium further comprises instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations including: Generate multiple patch images from the multiplex immunofluorescence images; and The multiple patch images are provided to the machine learning model as the multiplex immunofluorescence images.
12. The system of claim 7, wherein the subsequent image processing comprises: Access the second image of the sample slice; Apply the mask to the second image; The presence, number, and / or size of a group of tumor cells in the sample slice are determined by processing the second image with the mask; as well as Output the presence, number, and / or size of the group of tumor cells in the sample slice.
13. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform operations including: Access to multiplex immunofluorescence images of sample slices including a first channel targeting nuclear markers and a second channel targeting epithelial tumor markers, wherein the multiplex immunofluorescence images include one or more necrotic tissue regions; The multiplex immunofluorescence image is provided to a machine learning model, wherein the output of the machine learning model corresponds to the following prediction: the multiplex immunofluorescence image includes one or more necrotic tissue regions at one or more specific portions of the multiplex immunofluorescence image; Based on the output of the machine learning model, a mask is generated for subsequent image processing of the multiplex immunofluorescence images; as well as The mask is output for the subsequent image processing. The subsequent image processing includes: A modified version of the multiplex immunofluorescence image is generated by applying the mask to the multiplex immunofluorescence image; The modified version of processing the multiplex immunofluorescence images; and The output is a modified version of the processed multiplex immunofluorescence image, the result corresponding to a depiction of the detection of a group of tumor cells in the sample slice, wherein the result of subsequent image processing characterizes the presence, number and / or size of the group of tumor cells.
14. The computer program product of claim 13, wherein the nuclear marker comprises 4',6-diamidinyl-2-phenylindole (DAPI), and the epithelial tumor marker comprises broad-spectrum cytokeratin (PanCK).
15. The computer program product of claim 13, wherein the machine learning model includes the U-Net model.
16. The computer program product of claim 13, further comprising instructions configured to cause one or more data processors to perform operations including: Before feeding the multiplex immunofluorescence images to the machine learning model, the multiplex immunofluorescence images are preprocessed using gamma correction and normalization.
17. The computer program product of claim 13, further comprising instructions configured to cause one or more data processors to perform operations including: Generate multiple patch images from the multiplex immunofluorescence images; and The multiple patch images are provided to the machine learning model as the multiplex immunofluorescence images.
18. The computer program product of claim 13, wherein the subsequent image processing comprises: Access the second image of the sample slice; Apply the mask to the second image; The presence, number, and / or size of a group of tumor cells in the sample slice are determined by processing the second image with the mask; as well as Output the presence, number, and / or size of the group of tumor cells in the sample slice.