Determining tissue characteristics using multiplexed immunofluorescence imaging

Abstract

To provide determination of tissue characteristics using multiplexed immunofluorescence imaging.SOLUTION: Techniques for processing multiplexed immunofluorescence (MxIF) images include: obtaining at least one MxIF image of the same tissue sample; obtaining information indicative of locations of cells in the at least one MxIF image; identifying multiple groups of cells in the at least one MxIF image at least in part by determining feature values for at least some of the cells using the at least one MxIF image and the information indicative of locations of the at least some cells in the at least one MxIF image and grouping the at least some of the cells into the multiple groups using the determined feature values; and determining at least one characteristic of the tissue sample using the multiple cell groups.SELECTED DRAWING: None

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application was filed on March 6, 2020, the entire contents of which are incorporated herein by reference. U.S. Provisional Application No. 62 / 986,010, entitled "DETERMINING TISSUE CHARACTERISTICS USING MULTI 119(e) of the patent "PLEXED IMMUNOFLUORESCENCE IMAGING" This is what is done. [Background technology]

[0002] Multiplexed immunofluorescence (MxIF) imaging is a method for imaging multiplexed immunofluorescence (MMF) in a single biological sample. Image multiple fluorescent cells and / or histological markers in a sample (e.g., a tissue sample) MxIF imaging is a technique for staining, imaging, and dye chemical inactivation. Repeated activation (e.g., bleaching) and re-imaging several times to visualize multiple fluorescent markers in the biosample. The fluorescence of the markers then forms an image. MxIF imaging is used to visualize multiple different markers in a single tissue sample. This allows imaging of large numbers of markers (e.g., 30 to 100 markers), This allows more information to be gleaned incrementally from a single section of tissue.

[0003] As part of MxIF imaging, the cell membrane, cytoplasm, and Different types of markers can be used, including nuclear markers that bind within the nucleus and nuclear regions, respectively. The resulting images therefore allow tissue analysis at the subcellular level. [Preliminary Technology Documents] [Non-licensed literature]

[0004] [Non-licensed Document 1] Yury Goltsevら, "Deep Profiling of Mouse Splenic Architecture with CODEX Multiplexed Imaging", PMID: 30078711 (August 2018) [Non-licensed Document 2] Stuart Bergら, "ilastik: interactive machine learning for (bio) image analysis" (September 2019) [Non-licensed Document 3] Peter Bankhead, "QuPath: Open source software for digital pathology image analysis" (December 2017) [Non-licensed Document 4] Kaiming He "Deep Residual Learning for Image Recognition", arXiv:1512.03385v1 (December 2015) [Non-licensed Document 5] Mingxing TanおよびQuoc Le "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks", arXiv:1905.11946v5 (September 2020) [Non-licensed Document 6] Kaiming Heら, "Mask R-CNN", arXiv:1703.06870 (January 2018) [Non-licensed Document 7] Petar Velickovic, "Deep Graph Infomax", ICLR 2019 Conference Blind Submission (September 27, 2018) [Non-licensed Document 8] arXiv:1809.10341 [Non-Patent Document 9] Thomas Kipf and Max Welling, "Semi-Supervised Classification with Graph Convolutional Networks," arXiv1609.02907 (February 2017) [Non-Patent Document 10] William Hamilton et al., "Inductive Representation Learning on Large Graphs," arXiv1706.02216 (September 2018) [Non-Patent Document 11] Navaneeth Bodla, “Improving Object Detection With One Line of Code”, arXiv:1704.04503v2 (August 2017) [Non-Patent Document 12] Alexander Kirillov, "Panoptic Segmentation", arXiv:1801.00868 (April 2019) Summary of the Invention [Means for solving the problem]

[0005] Some embodiments utilize at least one computer hardware processor. To do this, acquire at least one multiplex immunofluorescence (MxIF) image of the same tissue sample and obtaining information indicating the arrangement of cells in at least one MxIF image; and The groups of cells in the image are at least partially identified by at least one MxIF image and at least one Also, using information indicating the arrangement of cells within a single MxIF image, at least some of the cells are identified. determining feature values ​​for some of the cells and using the determined feature values ​​to identify a subset of those cells; Identifying at least some cells by grouping them into groups, and and determining at least one characteristic of the tissue sample using the group of Provide a method for

[0006] Some embodiments include at least one computer hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions; the processor-executable instructions are executed by at least one computer hardware processor. When executed by the processor, it is executed by at least one computer hardware processor. acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; and obtaining information indicating the arrangement of cells in at least one MxIF image; The plurality of groups of cells in the sample are, at least in part, analyzed using at least one MxIF image and at least one Using information indicating the arrangement of cells within the MxIF image, we identify at least some of those cells. determining feature values ​​for a number of cells and using the determined feature values ​​to identify a number of those cells; Identifying at least some cells by grouping them into groups; and determining at least one characteristic of the tissue sample using the set of numbers. Provides systems.

[0007] Some embodiments include at least one non-transitory memory device that stores processor-executable instructions. A computer-readable storage medium, the processor-executable instructions being stored in at least one computer When executed by a computer hardware processor, The hardware processor is then loaded with at least one multiplexed immunofluorescence (MxIF) image of the same tissue sample. and obtaining information indicating the arrangement of cells within at least one MxIF image. and determining, at least in part, the plurality of groups of cells in at least one MxIF image by at least one Using two MxIF images and information indicating the arrangement of cells within at least one MxIF image, determining feature values ​​for at least some of the cells of the and grouping at least some of the cells into groups using a and determining at least one characteristic of the tissue sample using the plurality of groups. The present invention provides a non-transitory computer-readable storage medium for causing a computer to perform the steps of:

[0008] Some embodiments utilize at least one computer hardware processor. acquiring at least one multiplex immunofluorescence (MxIF) image of the tissue sample; obtaining information indicative of the location of at least one cell in at least one MxIF image; a plurality of markers expressed in at least one MxIF image and a plurality of markers expressed in at least one MxIF image; Determining marker expression signatures for cells in a tissue sample based on information indicating cell arrangement and analyzing the marker expression signatures for at least one of a plurality of different cell types. cell-to-cell comparison with cell typing data containing a single marker expression signature determining the type; and performing the steps.

[0009] Some embodiments include at least one computer hardware processor and at least one When executed by at least one computer hardware processor, and a computer hardware processor for processing at least one MxIF image of the tissue sample. and obtaining information indicative of the location of at least one cell within at least one MxIF image. and obtaining a plurality of markers and a minority of markers expressed in at least one MxIF image. MxIF images are used to identify cells in a tissue sample based on information indicating the arrangement of cells in at least one MxIF image. determining a marker expression signature and comparing the marker expression signature to a plurality of different types of cell typing data comprising at least one marker expression signature for the cells of and determining a cell type for the cell. and at least one non-transitory computer-readable storage medium storing instructions. provide.

[0010] Some embodiments include at least one non-transitory memory device that stores processor-executable instructions. A computer-readable storage medium, the processor-executable instructions being stored in at least one computer When executed by a computer hardware processor, acquiring at least one MxIF image of the tissue sample; obtaining information indicative of a location of at least one cell within at least one MxIF image; A plurality of markers expressed in at least one MxIF image and at least one MxIF A marker expression signature for cells in a tissue sample is generated based on information indicating the arrangement of cells within the image. The aim is to determine the marker expression signatures and to characterize them by using a small number of different cell types. Compared with cell typing data containing at least one marker expression signature, and determining a cell type to be treated. Also provided is a non-transitory computer-readable storage medium.

[0011] Various aspects and embodiments are described with reference to the following figures, which are not necessarily to scale. It should be understood that items that appear in multiple figures are not necessarily depicted in the figures. In all figures, the same or similar reference numerals will be used. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 pictorially illustrates an exemplary system for multiplex immunofluorescence (MxIF) imaging, in accordance with some embodiments of the techniques described herein. [Figure 2] 1A-1C illustrate an MxIF image and associated data that may be generated by processing the MxIF image, according to some embodiments of the techniques described herein. [Figure 3] FIG. 1 illustrates exemplary components of a pipeline for processing MxIF images of a tissue sample to determine characteristics of cells in the tissue sample, according to some embodiments of the techniques described herein. [Figure 4A] 4 illustrates an example processing flow for an MxIF image using some of the components of FIG. 3, in accordance with some embodiments of the techniques described herein. [Figure 4B] FIG. 4B illustrates an exemplary processing flow for the MxIF image of FIG. 4A including an optional tissue degradation check component, according to some embodiments of the techniques described herein. [Figure 4C] FIG. 10 illustrates a patch mask generated based on tissue degradation information determined by comparing two nuclear marker images of the same tissue sample, according to some embodiments of the techniques described herein. [Figure 4D]FIG. 4D illustrates the use of the patch mask from FIG. 4C to filter regions of a tissue sample for processing, according to some embodiments. [Figure 5A] FIG. 10 is a flowchart illustrating an exemplary computerized process for processing an MxIF image of a tissue sample based on cell arrangement data to group cells of the tissue sample to determine at least one characteristic of the tissue, according to some embodiments of the technology described herein. [Figure 5B] FIG. 10 illustrates an example of feature values ​​for cell location associated with cell location data, according to some embodiments of the technology described herein. [Figure 6A] 1 is a flowchart illustrating an exemplary computerized process for processing MxIF images of a tissue sample to predict cell types based on cell location data of cells in the tissue sample, according to some embodiments of the technology described herein. [Figure 6B] FIG. 1 shows examples of immunofluorescence images and cell location data used for cell typing, according to some embodiments. [Figure 6C] FIG. 1 shows an example of using a neural network to generate expression data that is compared with cell typing data to determine predicted cell types, according to some embodiments of the technology described herein. [Figure 6D] FIG. 10 illustrates an example of using a neural network to generate a probability table that is compared to a cell typing table to determine predicted cell types, according to some embodiments of the technology described herein. [Figure 6E] FIG. 10 illustrates an example of using a neural network to generate a probability table that is compared to a cell typing table to determine predicted cell types, according to some embodiments of the technology described herein. [Figure 7A] 1 is a flowchart illustrating an exemplary computerized process for processing MxIF images of a tissue sample to cluster cells of the tissue sample into multiple cell groups based on cell location data, according to some embodiments of the technology described herein. [Figure 7B] FIG. 1 is a flowchart illustrating an exemplary computerized process for processing a first set of cellular features (local cellular features) to identify one or more communities of cells using a graph neural network, according to some embodiments of the technology described herein. [Figure 7C] 1A-1C show examples of images of tissue contours shaded based on cell type and images of the same tissue with contours shaded based on cell clusters, according to some embodiments. [Figure 8] FIG. 10 illustrates an exemplary MxIF image and manual cell placement / segmentation data for the MxIF image, according to some embodiments of the techniques described herein. [Figure 9-1] FIG. 10 shows example images that can be used to train a neural network to identify cell location information, according to some embodiments of the technology described herein. [Figure 9-2] This is a figure showing a continuation of Figure 9-1. [Figure 10] FIG. 10 pictorially illustrates an example of using a convolutional neural network model to process acquired MxIF images of a tumor to generate cell segmentation data, in accordance with some embodiments of the techniques described herein. [Figure 11] FIG. 10 pictorially illustrates another exemplary use of neural networks to process immunofluorescence images to generate cell location / segmentation data, in accordance with some embodiments of the techniques described herein. [Figure 12] 1A-1C illustrate MxIF images and cell segmentation data generated based on the MxIF images, according to some embodiments of the techniques described herein. [Figure 13] 1A-1C illustrate a composite fluorescent image and cell segmentation data generated based on the composite fluorescent image, according to some embodiments of the techniques described herein. [Figure 14]FIG. 10 illustrates exemplary cell segmentation data for an exemplary acquired MxIF image of kidney tissue, according to some embodiments of the techniques described herein. [Figure 15] FIG. 1 shows an MxIF image of clear cell renal cell carcinoma (CCRCC) and corresponding cell segmentation data, according to some embodiments of the techniques described herein. [Figure 16] FIG. 1 shows an MxIF image of CCRCC and corresponding cell segmentation data, according to some embodiments of the technology described herein. [Figure 17] FIG. 1 illustrates a convolutional network architecture for predicting a noise subtraction threshold for subtracting noise from raw immunofluorescence images, according to some embodiments of the techniques described herein. [Figure 18] 10A-10C illustrate exemplary tissue characteristics that may be determined by processing MxIF images, according to some embodiments of the techniques described herein. [Figure 19A] FIG. 10 is an illustration of stromal and acinar masks generated by processing immunofluorescence images according to some embodiments of the techniques described herein. [Figure 19B] FIG. 10 illustrates generating an object mask using features of a tissue sample, according to some embodiments of the techniques described herein. [Figure 20-1] 1A-1C show examples of measuring the shape, area, and perimeter of an acinus according to some embodiments of the techniques described herein. [Figure 20-2] This is a figure showing a continuation of Figure 20-1. [Figure 21-1] 1A-1C illustrate examples of spatial distribution characteristics according to some embodiments of the techniques described herein. [Figure 21-2] This is a figure showing a continuation of Figure 21-1. [Figure 21-3] This is a figure showing a continuation of Figure 21-1. [Figure 21-4] This is a figure showing a continuation of Figure 21-1. [Figure 22-1]1A-1C illustrate examples of spatial organization properties according to some embodiments of the techniques described herein. [Figure 22-2] This is a figure showing a continuation of Figure 22-1. [Figure 23-1] FIG. 1 shows an example of cell contact information from immunofluorescence images for two different patients, according to some embodiments of the technology described herein. [Figure 23-2] This is a figure showing a continuation of Figure 23-1. [Figure 24-1] FIG. 24 shows example information related to cell neighborhood information for two different patients from FIG. 23, according to some embodiments of the technology described herein. [Figure 24-2] This is a figure showing a continuation of Figure 24-1. [Figure 25] 1A-1C show two example MxIF images and corresponding stroma segmentation masks, according to some embodiments of the techniques described herein. [Figure 26] 1A-1C illustrate exemplary MxIF images, segmentation masks, and corresponding cell groups, according to some embodiments of the techniques described herein. [Figure 27] FIG. 1 shows an example of a complete MxIF slide processing to generate cell populations, according to some embodiments of the technology described herein. [Figure 28] 1A-1C are diagrams of reconstructed cell configurations generated by processing immunofluorescence images according to some embodiments of the techniques described herein. [Figure 29] FIG. 1 shows a 4′,6-diamidino-2-phenylindole (DAPI) stained immunofluorescence image and two images of different cell populations relative to the DAPI image, according to some embodiments of the technology described herein. [Figure 30] FIG. 1 shows cell populations for different CCRCC tissue samples, according to some embodiments of the technology described herein. [Figure 31] FIG. 15 shows exemplary cell populations for an exemplary MxIF image acquired of kidney tissue from FIG. 14, in accordance with some embodiments of the techniques described herein. [Figure 32] FIG. 1 shows a set of cell population images for different clear cell renal cell carcinoma (CCRCC) tissue samples, according to some embodiments of the technology described herein. [Figure 33] 1A-1C show a series of images of cell populations for different CCRCC tissue samples, according to some embodiments of the technology described herein. [Figure 34-1] FIG. 10 shows the results of an analysis of two different MxIF images of a CCRCC tissue sample according to some embodiments of the techniques described herein. [Figure 34-2] This is a figure showing a continuation of Figure 34-1. [Figure 35-1] FIG. 1 shows the results of analysis of MxIF images of CCRCC tissue samples according to some embodiments of the techniques described herein. [Figure 35-2] This is a continuation of Figure 35-1. [Figure 36A] FIG. 1 illustrates cell amounts and ratios according to some embodiments of the technology described herein. [Figure 36B-1] FIG. 1 illustrates cell amounts and ratios according to some embodiments of the technology described herein. [Figure 36B-2] This is a figure showing a continuation of Figure 36B-1. [Figure 36B-3] This is a figure showing a continuation of Figure 36B-1. [Figure 37] 10A-10C illustrate cell distribution characteristics according to some embodiments of the technology described herein. [Figure 38A-1] FIG. 10 illustrates a percentage heat map and distribution density of histological features according to some embodiments of the technology described herein. [Figure 38A-2] This is a figure showing a continuation of Figure 38A-1. [Figure 38B-1] FIG. 10 illustrates a percentage heat map and distribution density of histological features according to some embodiments of the technology described herein. [Figure 38B-2] This is a figure showing a continuation of Figure 38B-1. [Figure 39-1]FIG. 1 illustrates cell neighborhood information and cell contact characteristics, according to some embodiments of the technology described herein. [Figure 39-2] This is a continuation of Figure 39-1. [Figure 39-3] This is a continuation of Figure 39-1. [Figure 40-1] FIG. 1 shows an example of a tSNE plot of a profile of marker expression, according to some embodiments of the technology described herein. [Figure 40-2] This is a figure showing a continuation of Figure 40-1. [Figure 41-1] FIG. 10 is another illustrative diagram showing an example of a tSNE plot of a profile of marker expression, according to some embodiments of the technology described herein. [Figure 41-2] This is a continuation of Figure 41-1. [Figure 41-3] This is a continuation of Figure 41-1. [Figure 41-4] This is a continuation of Figure 41-1. [Figure 42] FIG. 10 pictorially illustrates the use of a convolutional neural network to determine cell segmentation data for 4′,6-diamidino-2-phenylindole (DAPI) heterogeneously stained immunofluorescence images, in accordance with some embodiments of the techniques described herein. [Figure 43] FIG. 10 is a pictorial illustration of a first cell mask generated based on a combination of a DAPI-stained immunofluorescence image of a tissue sample and a CD3 cell marker image, according to some embodiments of the techniques described herein. [Figure 44] FIG. 44 is a pictorial illustration of a second cell mask generated based on a combination of the DAPI-stained immunofluorescence image of FIG. 43 and the CD21 cell marker image, according to some embodiments of the technology described herein. [Figure 45] FIG. 44 is a pictorial illustration of a third cell mask generated based on a combination of the DAPI-stained immunofluorescence image of FIG. 43 and the CD11c cell marker image, according to some embodiments of the techniques described herein. [Figure 46]FIG. 44 is a pictorial illustration of a vascular mask generated based on the MxIF image of the tissue sample of FIG. 43, in accordance with some embodiments of the techniques described herein. [Figure 47] FIG. 47 is a pictorial illustration of a cell population generated using the masks of FIGS. 43-46, in accordance with some embodiments of the techniques described herein. [Figure 48] FIG. 1 illustrates a set of cell populations for a prostate tissue sample and malignant site, according to some embodiments of the technology described herein. [Figure 49] FIG. 49 is a magnified view of a portion of the prostate tissue sample of FIG. 48, according to some embodiments of the technology described herein. [Figure 50] FIG. 1 shows a set of cell populations from prostate tissue samples taken from four different patients, according to some embodiments of the technology described herein. [Figure 51] FIG. 1 is a diagram of an exemplary implementation of a computer system that may be used in connection with any of the embodiments of the technology described herein. [Figure 52] 1A-1C illustrate two exemplary comparisons of cell location information generated using implementations of the techniques described herein and conventional techniques, according to some embodiments. [Figure 53] 1A-1C illustrate two exemplary comparisons of cell location information generated using implementations of the techniques described herein and conventional techniques, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present inventors have, for example, diagnosed or suspected of having cancer or another disease. Multiplexing of biological samples, such as multiplex immunofluorescence (MxIF) imaging of tissue samples from individuals at risk for or with HIV We have developed new image processing and machine learning techniques to process images. For example, labeled antibodies can be used as labeling agents. The antibody may be labeled with various types of labels, such as a label, an enzyme label, and / or the like.

[0014] The techniques developed by the present inventors allow for the determination of the cellular composition (e.g., cell types, cellular morphology) and / or tissue organization (e.g., cellular localization, multicellular structural localization, etc.) It provides robust information about various medically relevant properties of tissue samples that can be identified. Examples of tissue sample characteristics include, but are not limited to, information about the types of cells in the tissue sample, morphological information, spatial information, and some tissue structures (e.g., acini, stroma, tumor, blood vessels, etc.) The information identifying the placement of the

[0015] The inventors have identified medically relevant characteristics of tissue samples that include any of the characteristics described above. It is desirable to be able to distinguish between different cell populations in a tissue sample that can be used to determine their respective characteristics. However, conventional techniques for processing MxIF images are limited to the cell population. In particular, the prior art is not fully automated. MxIF data can be analyzed to determine tissue properties of interest. It is not possible to accurately determine the cell type, cell population, and / or other information of the material. Thus, conventional techniques have been limited to identifying cells and / or other aspects of tissue samples, and even Various MxIF image processing procedures, including configuring parameters for the semi-automated part of the process Requires manual intervention for steps. Such manual intervention results in wasted time. tedious analytical work (e.g., image quality is poor since such images typically have a large number of cells) (Manually annotating the data can be a time-consuming and significant task), The process is subject to human error and / or is not easily repeatable between different samples. (e.g., resulting in inconsistencies in the MxIF tissue analysis procedure and / or MxIF tissue analysis data) will provide inconsistent results (which may result in a match).

[0016] The present inventors have investigated the information indicative of cell arrangement (e.g., information from which cell boundaries can be derived, cell boundaries, etc.) boundaries, and / or masks) are applied to the pixel values ​​of those cells in one or more MxIF images. can be used to determine feature values ​​for individual cells based on novel image processing techniques (by determining one or more feature values ​​for a cell) The feature values ​​were then used to group cells into groups. This can be used to determine cellular properties of interest. In an embodiment, the feature values ​​are calculated for each channel of at least one MxIF image for one cell. This indicates how much a gene is expressed in each cell, and therefore the markers for the cells in the MxIF image. It can be used to determine the CAR expression signature.

[0017] In some embodiments, the cell populations are determined by dividing the tissue sample based on the feature values ​​for the individual cells. Determine the cell types in the sample (e.g., to group cells by cell type) This can be determined by performing a cell typing process (which can be used in some cases). In an embodiment, the feature values ​​are based on traditional chemical staining of different cell and / or tissue structures. For example, in some embodiments, the feature value for a cell may be determined based on The gene expression signature may include a gene expression signature for each of one or more cells (e.g., a gene expression signature for each of one or more cells). Marker expression signatures can be calculated for each cell type (e.g., For example, the cells may be acinar cells, macrophages, myeloid cells, T cells, B cells, or endothelial cells. , and / or any other cell type). The cells are then grouped based on their cell type, with each group containing cells of a particular type. may be used (e.g., a population containing acinar cells for various cell types of interest, A marker expression signature is further defined herein as a group comprising a population of cells, such as a group comprising macrophage cells. Additionally or alternatively, cell populations can be determined based on feature values. (e.g., cells of the same and / or different cell types may be identified based on having similar feature values) For example, in some embodiments, the feature values ​​of a cell are: Any information indicating the expression levels of various markers in a cell may be included. For example, The feature value for a cell is the pixel location in at least one MxIF image of a particular cell. The feature value may be determined using the pixel values ​​for the cells. It may be determined by averaging or calculating the median of the cells, which is It may represent the mean or median expression level of the marker in the cells. and determining one or more characteristics of each cell type in the plurality of populations. In some embodiments, the group comprises different They may represent different parts of an organizational structure.

[0018] Thus, some embodiments provide for (1) a method for detecting cancer cells from the same biological sample (e.g., a cancer-bearing or cancer-bearing individual) and (2) a method for detecting cancer cells from the same biological sample (e.g., a cancer-bearing individual). at least one tissue sample from a subject suspected of having or at risk of having cancer (2) acquiring an MxIF image and (3) information indicating the arrangement of cells in the MxIF image (e.g., cell boundaries) , cell boundaries can be derived (e.g., cell boundaries in an image, and / or masks) By applying machine learning models to identify cell arrangement information (e.g., Compute and / or access such cell placement information (e.g., cell placement masks) (3) obtaining a plurality of groups of cells in at least one MxIF image by at least in part, by (a) using the MxIF image and information indicative of the arrangement of at least some of the cells; and determining feature values ​​for at least some of the cells (e.g., as indicated by information indicating the cellular arrangement of at least some of those cells. The cell is then analyzed by calculating the mean or median pixel value of the pixels within the boundary of that cell. Calculate feature values ​​for the cells, calculate marker expression signatures as described herein, (b) calculating feature values ​​for the cells by using the determined feature values ​​to Grouping at least some of the cells into groups (e.g., dividing the cells into types) Group by average or mean using any suitable clustering algorithm (4) identifying the cells by grouping them based on median pixel value, etc. The plurality of groups are used to determine at least one characteristic of the tissue sample (e.g., the cell type of each group). Determine the class, determine the cell mask, determine the cell community, determine the cell population of multiple groups Determine statistical information about distribution, determine spatial distribution of cell types, determine morphological information and (e.g., performing a step of the process of generating a plurality of images).

[0019] In some embodiments, the information indicative of cellular disposition is chemical or physical information of cells or cell structures. In some embodiments, the chemical staining can be obtained by a fluorescent signal (e.g., In some embodiments, a label (e.g., a fluorescent Photolabeled (photolabeled) antibodies target specific intracellular, membrane, and / or extracellular locations in cell or tissue samples. In some embodiments, a labeled antibody may be used alone to detect In some embodiments, the labeled antibody may be coupled to one or more other stains (e.g., other It can be used with fluorescent stains.

[0020] In some embodiments, acquiring at least one MxIF image is performed on the same tissue. acquiring a single multi-channel image of the sample, A channel may be formed by multiple markers (e.g., as described in the "Description of the Invention" section). any one or more of the markers described herein, including markers In some embodiments, at least one Acquiring one MxIF image involves acquiring multiple immunofluorescence images of the same tissue sample. At least one of the multiple immunofluorescence images is associated with each marker. The immunofluorescence images may include a single channel image containing at least one One can also include multi-channel images, where the channels in a multi-channel image are A plurality of markers (e.g., the markers described in the Detailed Description section) Each marker in any one or more of the markers described herein, including In some embodiments, at least one of the tissue samples is associated with a MxIF images are captured in vitro.

[0021] In some embodiments, the feature values ​​for the cells are determined from one or more of the MxIF images. For example, in some embodiments, the cell may include values ​​of the pixels of the first and determining the feature value includes determining a location of the first cell in the at least one MxIF image. a first feature for a first cell using at least one pixel value associated with the In some embodiments, determining a characteristic value of at least one of the cells. and some of the cells include a second cell, and determining the feature value includes determining the feature value from at least one MxIF. using at least one pixel value associated with the placement of the second cell within the image. In some embodiments, determining a second characteristic value for the first cell. Determining the first feature value for the cell includes: using the pixel values ​​associated with each location of the first cell in the Includes:

[0022] In some embodiments, at each position of the first cell in the plurality of channels Using the associated pixel values, for each of the plurality of channels: (a) Pixels for the first cell are calculated using information indicating the location of the first cell in that channel. (b) identifying a set of pixels; and (b) determining a first cell based on the values ​​of pixels in the set of pixels. and determining a feature value for the first cell. The information indicates the location of the boundary of the first cell and identifies a set of pixels for the first cell. The separating may be performed by at least partially (e.g., partially or completely) separating the first cell from the first cell boundary. This involves identifying the pixels that are inside.

[0023] In some embodiments, determining the first characteristic value comprises determining the first characteristic value from at least one MxIF image. One or more of the markers on the image (e.g., at least 1, at least 3, at least 5) , at least 10, at least 15, between 1 and 10, between 5 and 20, or or any other suitable number, range, or value within the range), These markers include ALK, BAP1, BCL2, BCL6, CAIX, CCASP3, CD10, CD106, CD11b, and CD1 1c, CD138, CD14, CD16, CD163, CD1, CD1c, CD19, CD2, CD20, CD206, CD209, CD21, CD 23, CD25, CD27, CD3, CD3D, CD31, CD33, CD34, CD35, CD38, CD39, CD4, CD43, CD44, CD45, CD49a, CD5, CD56, CD57, CD66b, CD68, CD69, CD7, CD8, CD8A, CD94, CDK1, CDX 2, Clec9a, chromogranin, collagen IV, CK7, CK20, CXCL13, DAPI, DC-SIGN, Death Min, EGFR, ER, ERKP, fibronectin, FOXP3, GATA3, GRB, granzyme B, H3K36T M, HER2, HLA-DR, ICOS, IFNg, IgD, IgM, IRF4, Ki67, KIR, lumican, Lyve-1, mamma Globin, MHCI, p53, NaKATPase, PanCK, PAX8, PCK26, CNAP, PBRM1, PD1, PDL1, Par Lecan, PR, PTEN, RUNX3, S6, S6P, SMA, SMAa, SPARC, STAT3P, TGFb, Va7.2, and It is vimentin.

[0024] In some embodiments, multiple markers are present in a single channel, and / or can be detected. For example, signals from chemical stains (e.g., DAPI) can be detected by Also used to provide immunofluorescence signals (e.g., to detect cell localization information) In some embodiments, the images may be in the same image of the tissue sample (or channel). There is only one marker in a single channel and / or image.

[0025] In some embodiments, grouping the cells into a plurality of cell populations comprises clustering. A clustering algorithm is used to cluster cells based on their feature values. For example, centroid-based clustering algorithms (e.g., K-means), Distribution-based clustering algorithms (e.g., clustering using Gaussian mixture models) clustering), density-based clustering algorithms (e.g., DBSCAN), hierarchical clustering Rastering algorithms, principal component analysis (PCA), independent component analysis (ICA), and / or any Any suitable clustering algorithm, including other suitable clustering algorithms Any algorithm may be used, as the embodiments described herein are not limited in this respect.

[0026] In some embodiments, at least some of the cells are divided into multiple groups. The grouping is performed by analyzing the determined feature values ​​and identifying at least some of the cells. Determining the relationships between cells and separating groups into groups The cells are determined based on the relationships determined so that each cell has a characteristic value that indicates the relationship between the cells in the group. In some embodiments, determining at least some of the cells. Determining the relationships between several cells requires identifying at least some of those cells. In some embodiments, determining the similarity between the feature values ​​of the cells. Determining the relationships between at least some of the cells in the Compared with the existing cell typing data, at least some of those cells This involves determining the cell type for each of the cells.

[0027] In some embodiments, these techniques involve cell typing based on feature values ​​of individual cells. In some embodiments, the feature values ​​include cell type and and / or channels that can be used to determine cell populations (e.g., cell clusters). channel contribution (e.g., the average channel contribution, which indicates how much each channel contributes to the cell). The inventors have found that the average channel contribution can be used for cell typing, but In some cases, exploiting the average channel contribution in a cell can be useful for segmentation. may be affected by the quality of the sample, cell size, cell shape, and / or tissue staining. For example, such techniques are prone to variations in marker intensity. It is possible that the marker emerges from the contours of nearby cells (e.g., due to segmentation errors). Add expression and / or create additional clusters of cells with intermediate marker expression This can complicate the cell type discrimination process. understood that channel contribution does not take into account information about the intracellular signal localization. (e.g., can be useful to help distinguish real signal data from noise) As a further example, we have calculated the value of the channel contribution for a particular type of cell. Without the ability to set a range of cells, it is difficult to purposefully search for cell types of interest. We also understand that this can be difficult (e.g., clustering may be used instead). (This may result in missing cells of interest.) Therefore, we We understand that average channel contributions may not provide stable cell typing. Given such potential problems, cell typing results are manually checked. This may increase delays and affect automation.

[0028] Therefore, the present inventors developed a technology for cell typing that utilizes machine learning. In some embodiments, cell typing involves determining the identity of each of one or more cells. Using a trained neural network to determine marker expression signatures in The marker expression signature for a cell can be performed by: For each particular marker in one or more of the markers, The marker expression signature of a cell can then be expressed as , cell types (e.g., marker expression signatures) for each cell type, e.g., For example, compare the results to a previously determined marker expression signature correlated by a pathologist. Such machine learning techniques can be used to identify (e.g., by (compared to traditional techniques, which may require the user to manually adjust settings) In some embodiments, the present invention provides an automated method for typing trained keywords. The central network can be configured to transmit one or more channels (e.g., a separate 1-channel MxIF image , 2-channel MxIF images, 3-channel MxIF images, and / or the like) It can also take as input one MxIF image and cell positioning data for the cell of interest. The trained neural network not only analyzes marker expression intensity but also the detected cell shape, Other data such as cell texture, location of marker expression, etc. can also be used. The neural network calculates the number of cells for each channel and its associated marker. Outputting a likelihood (e.g., ranging from 0 to 1) of having a signal of appropriate strength and shape The likelihood determined by the neural network for each marker is These are combined to generate a marker expression signature for the cells of interest. This can be compared with cell typing data to determine predicted cell types for the cells of interest. The type of signal can be determined by using trained neural networks to detect differences in signal level distributions. Robustness of the network (e.g., training the network using heterogeneous training data) , by utilizing additional features as described both above and herein. By virtue of this, the cell typing approach described herein allows for the identification of a specific locus for each cell. It provides automated cell type detection that results in robust signal presence determination.

[0029] In some embodiments, the plurality of channels may include a plurality of markers. First cells in multiple channels of at least one MxIF image associated with a marker determining a first feature value using pixel values ​​associated with each of the locations of the means, for each particular marker of one or more of the plurality of markers, A marker expression signature comprising the likelihood that the particular marker is expressed in the first cell. and determining a population of cells, wherein at least some of the cells are divided into a plurality of groups. Grouping was performed using marker expression signatures and cell typing data. determining a predicted cell type for the first cell; and and associating the first cell with one of a plurality of groups. and determining feature values ​​for at least some of the cells, A marker expression signature is generated for each specific cell of a plurality of cells in at least one MxIF image. determining a marker expression signature, the marker expression signature being one or more of the plurality of markers; For each specific marker or markers, the specific marker is expressed in a specific cell. and determining the likelihood of the feature values ​​being expressed using the determined feature values. Grouping at least some of the cells into groups is a way of identifying specific cells. Using marker expression signatures and cell typing data for cells, multiple cells were identified. Determining the predicted cell type for each specific cell and then performing a cellular analysis based on the predicted cell type. and grouping the plurality of cells into a plurality of groups. It contains at least one of the markers described.

[0030] In some embodiments, the cell typing data includes a sequence for each of a plurality of cell types. and a marker expression signature for each specific cell of the plurality of cell types. At least one marker expression signature for a cell type is determined by which of the multiple markers It includes data showing whether it is expressed in cells of a particular cell type.

[0031] In some embodiments, the method is configured to determine a marker expression signature. The marker expression signatures were first obtained using a pre-trained neural network. In some embodiments, the first trained neural network further comprises determining The network comprises at least 1 million parameters. In some embodiments, The method includes as input images training immunofluorescence images of tissue samples for associated cell types. A set of images and information indicating marker expression in the input images for the associated cell types. and associated output data including the information. In some embodiments, the method further comprises training the at least one providing two MxIF images as inputs to a first trained neural network; Obtain marker expression signatures as outputs from the trained neural network of This includes the following.

[0032] In some embodiments, the method comprises: determining a plurality of channels of cells from the information indicative of the cell arrangement; and generating information indicating the arrangement of the first cell in at least some of the channels of the first training sample. This includes providing the data as part of the input to a trained neural network. A neural network can include multiple convolutional layers.

[0033] In some embodiments, the cell typing data includes a sequence for each of a plurality of cell types. and a plurality of marker expression signatures, the plurality of marker expression signatures including at least one marker expression signature that Determining the predicted cell type of the first cell includes determining a marker expression signature of the first cell. and associating the marker expression signature with at least one marker expression signature of the plurality of marker expression signatures. In some embodiments, the method further comprises comparing the predicted cell type. The marker expression signature of a cell is determined by comparing the marker expression signature of a cell with at least one of the plurality of marker expression signatures. Comparison with another marker expression signature is performed using a distance measure. In some embodiments, the distance measures include cosine distance, Euclidean distance, and In some embodiments, the method is at least one of: Manhattan distance ,select the comparison metric with the lowest or highest value among the calculated comparison metrics. determining predicted cell types by

[0034] In some embodiments, the at least one characteristic of the tissue sample is determined by the cellular composition of the tissue sample. In some embodiments, the composition of the tissue sample is characterized, the tissue composition of the tissue sample is characterized, or both. Determining the at least one characteristic determines information about cell types in the tissue sample. For example, in some embodiments, information about cell types in a tissue sample may be collected. Determining the information identifies one or more cell types present in a tissue sample. In some embodiments, the cell types include endothelial cells, epithelial cells, macrophages, , T cells, malignant cells, NK cells, B cells, and acinar cells. In some embodiments, the T cells are selected from the group consisting of CD3+ T cells, CD4+ T cells, and CD8+ T cells. In some embodiments, information regarding cell types in a tissue sample is provided. Determining the percentage of one or more cell types in a tissue sample. In some embodiments, the cell types may be enriched with various information and / or techniques. In some embodiments, cell types can be determined based on their size, shape, Histologically determined based on appearance and / or staining (e.g., using different chemical stains) Additionally or alternatively, immunofluorescence signals can be used to measure cell size and shape. Additionally or alternatively, the method may be used to assess the state of the cells and determine the cell type. Cell-specific markers (e.g., proteins) are used to determine cell type (e.g., target It can be used (by using a specific antibody).

[0035] Any of a number of varieties of tissue properties have been developed by the inventors and are described herein. For example, in some embodiments, at least Determining another characteristic relates to the distribution of at least a portion of the cells in the plurality of populations of cells. For example, in some embodiments, the method may include determining statistical information relating to the number of cells. Determining statistical information regarding the distribution of at least some of the cell types in a tissue sample In some instances, determining the spatial distribution of one or more cell types in the Determining statistical information about the distribution of at least some of the cells includes: This includes determining the distribution among different cell types.

[0036] As another example, in some embodiments, determining the at least one characteristic determining spatial information regarding the arrangement of at least some cells of the plurality of groups of cells; In some embodiments, determining spatial information includes determining spatial information of a plurality of groups of cells. In some embodiments, determining spatial information includes determining the distance between cells. The method comprises: extracting a tissue sample containing one or more cells of one of the plurality of populations of cells; In some instances, determining at least some of the regions Determining spatial information regarding the arrangement of cells can be performed by analyzing the spatial distribution of cells in one or more cell types of a tissue sample. determining spatial organization (e.g., determining the cellular organization in a tissue sample by cell type) (including information about the cell structure, compared to other cell types, and / or similar). In some instances, determining spatial information about the arrangement of at least some cells is One or more regions of the tissue sample containing one or more cell types (e.g., regions exceeding a threshold value) The method involves determining a region containing a large number of cells of one or more cell types.

[0037] As another example, in some embodiments, determining the at least one characteristic determining morphological information for at least some cells of the plurality of groups of cells; In some instances, determining morphological information about at least some of the cells may be necessary. 2. Characterize the cell and / or tissue sample, including cell shape, structure, morphology, and / or size information about the morphology and / or structure of the This includes determining the information.

[0038] As another example, in some embodiments, determining the at least one characteristic , for at least some cells of a plurality of groups of cells (e.g., cells of one group, multiple determining physical information (for a group of cells, etc.), the physical information including cell area, determining at least one of the following: cell perimeter, cell size.

[0039] In some embodiments, the determined characteristics include morphological information, spatial information, tissue structure information, and the like. The present inventors have found that the tissue sample may be fused to a tissue sample. by creating one or more masks that can be used to analyze the cellular structure. Such properties can be determined, the analysis of which can then be used to determine properties of interest. However, as pointed out above, typically The mask can be used to automatically identify cells and / or cell groups in a tissue sample using computerized techniques. They cannot be distinguished and must be created manually (e.g., using signal thresholding). Cell information such as cell segments and / or cell groups must be automatically determined. By utilizing feature values ​​to identify the characteristics of tissue samples, such masks can be used to identify the characteristics of tissue samples. Analyzing different cells and / or cellular structures in a tissue sample may allow for the determination of For example, these techniques can be used to identify cell types (e.g., T Identify T cells (interstitial and / or non-interstitial regions of the tissue sample) and use the stromal mask to identify whether T cells are present in the interstitial and / or non-interstitial regions of the tissue sample. It is possible to identify whether the area is in the quality region.

[0040] In some embodiments, the mask may be a binary mask. 0 or 1 (or any other suitable) for at least some of the pixels in the MxIF image It may be a pixel-level binary mask containing binary values ​​of the type.

[0041] In some embodiments, determining at least one characteristic comprises determining a portion of the tissue sample. determining one or more acinar masks that indicate the arrangement of acini within at least one multiplexed immunofluorescence image; In some embodiments, the acinar mask for the MxIF image may include: A binary mask may be used to represent at least some of the pixels in the MxIF image. The binary value for a pixel may include a binary value for the pixel represented in the MxIF image. This indicates whether the nucleus is located within the acini.

[0042] As another example, in some embodiments, determining the at least one characteristic , one or more stromal masks indicating the location of stroma within at least one MxIF image of the tissue sample. In some embodiments, the stromal mask for the MxIF image includes determining: It may be a binary mask, and at least some of the pixels in the MxIF image may be The binary value for a pixel may include the value at which the pixel is represented in the MxIF image. Indicates whether the target is located within the interstitium.

[0043] In another example, determining the at least one characteristic comprises determining at least one Mx of the tissue sample. determining one or more tumor masks indicative of the location of the tumor within the IF image. In some embodiments, the tumor mask for the MxIF image may be a binary mask, and may include binary values ​​for at least some of the pixels in the F image, The binary value for a pixel indicates whether the pixel is located within the tumor shown in the MxIF image. Indicates whether

[0044] The present inventors have demonstrated that some of the cellular structures in tissue samples, such as those indicative of cancer (e.g., breast cancer, renal cancer, etc.), We further understand that it may be desirable to explore some cellular structures. Some previous approaches to perform cell clustering into communities have reconstructed Use information from neighbors in the published cell contact graph (e.g., https: / / pubmed.nc Yury Goltsev et al., “Deep Profiling of Mice,” available at bi.nlm.nih.gov / 30078711 / e Splenic Architecture with CODEX Multiplexed Imaging”, PMID:30078711 (August 2018 However, such an approach does not allow for the use of the method described in the present invention. It does not incorporate other information that may be relevant to the clustering process.

[0045] We used a graph neural network that exploits the characteristics of cells to We developed a technique to identify clusters or communities. Cell features are the characteristics of each cell. cell type, cell neighbors, neighbor cell types, neighbor distance data, and / or It may also include other data as further described in the specification. The technique identifies cellular communities in tissue samples by utilizing such cellular features. and identifying information about those cells, such as cell type and distance (e.g., sparse (to provide information about other cell clusters, nearby cell clusters, etc.) Such a mask can therefore be used to detect signals that would otherwise be difficult to detect. Complex structures within an organization that are difficult to analyze (e.g., compared to using thresholding approaches) It allows for the discovery of clusters and provides automatic clustering at scale. Some embodiments apply graph neural networks to identify cell clusters or performs community detection.

[0046] In some embodiments, determining the at least one characteristic comprises determining the at least one characteristic based on a plurality of groups. determining one or more cell clusters of the tissue sample based on the and determining a cell cluster mask indicative of the arrangement of several cell clusters. In an embodiment, one or more cell clusters generate a first set of cell features ( For example, we triangulate the image to generate a graph, which can be used to calculate cell neighborhoods, cell neighborhood distances, and so on. (determine at least some of the cell features, such as separation, etc.) We embed the first set of cell features into a higher dimensional space using a network. By identifying communities of cells by clustering observed features is determined.

[0047] In some embodiments, determining one or more cell clusters in a tissue sample This means that there is a node for each of at least some cells, and a node for each of the nodes in the graph. The method involves generating a graph containing edges between nodes that determine the features to be used, and then finding the embeddings in the latent space. in the graph as input to the graph neural network to obtain embedded features. features for the nodes and embeddings to obtain clusters of nodes. Clustering the features and using the clusters of nodes to identify one or more cells and determining clusters.

[0048] In some embodiments, each cell cluster of the one or more cell clusters comprises a plurality of In some embodiments, each cell cluster comprises a number of cell types. In some embodiments, the tissue structure represents at least a portion of the mantle tissue, interstitial tissue, or the like. The tissue may include stromal tissue, tumors, follicles, blood vessels, or any combination thereof.

[0049] In some embodiments, the method includes a first determination of cellular characteristics of at least some of the cells. The method further comprises determining one or more cell clusters based on the set. A node for each of at least some cells in the sample, and edges between the nodes. determining a first set of cell features for each cell by generating a graph including In some embodiments, the method may further include triangulation (e.g., , Delaunay triangulation and / or any other type of triangulation) to obtain a small number of tissue samples. The method further includes generating a graph based on at least some of the cells.

[0050] In some embodiments, the method comprises: The edge lengths of the nodes in the graph and the cells for each node in the graph based on the mask data In some embodiments, the method further comprises determining the first set of features. In some embodiments, the method further comprises encoding the graph into a sparse adjacency matrix. The method further includes encoding the graph into an adjacency list of the edges of the graph.

[0051] In some embodiments, the method uses a graph as input to train a graph neural network. The network is provided with the graph neural network, and the graph neural network provides the and obtaining a set of feature embeddings. A trained graph neural network contains one or more convolutional layers. In an embodiment of the present invention, the set of feature embeddings is generated by a trained graph neural network In some embodiments, the activations of the last graph convolution layer of Determining one or more cell clusters is based on a set of feature embeddings for each node. In some embodiments, determining one or more cell clusters based on the determining one or more cell clusters based on a set of feature embeddings for each node; is a method for clustering cells in a tissue sample based on a set of feature embeddings for each node. This includes:

[0052] In some embodiments, the MxIF images are pre-processed before feature values ​​for the cells are determined. In some embodiments, machine learning techniques may be used to preprocess the MxIF images to generate machine learning images. It is applied to remove artifacts and / or identify information indicative of cell placement. The inventors believe that MxIF images are generated by, for example, a microscope imaging tissue. Noise, noise from surrounding cells in the tissue sample (e.g., fluorescence noise), antibodies in the tissue sample noise from the image, and / or any other type of artifacts that may be present in the MxIF image. It is understood that there may be artifacts introduced during imaging. For example, low signal-to-noise levels may affect cell clustering and therefore , raw MxIF images and / or data generated by the cell segmentation process data may not be sufficient for automated cell clustering. Process the immunofluorescence images, including performing background subtraction to remove noise. We have developed a technique to remove artifacts.

[0053] Thus, in some embodiments, background subtraction is performed on one of the MxIF images. or more than one. Thus, in some embodiments, a tissue sample The acquiring of at least one MxIF image includes acquiring a first channel of the at least one MxIF image. In some embodiments, performing background subtraction on the Performing background subtraction is configured to perform background subtraction. A second trained neural network model (e.g., a convolutional neural network) The network is a convolutional neural network with a U-net architecture. In some embodiments, the method further comprises providing a first channel. The neural network includes at least 1 million parameters. The method includes receiving a set of noisy training immunofluorescence images as input images and receiving at least Associated output data, including associated images that do not contain any noise and further comprising training a second trained neural network using the While pre-processing of MxIF images is possible, aspects of the technology described herein are not limited in this respect. Therefore, in addition to (in some embodiments) background subtraction, or (in some embodiments) It should be understood that other types of pre-treatment may be included instead (in embodiments). For example, preprocessing can include filtering, noise suppression, artifact removal, smoothing, etc. Different areas for performing filtering, sharpening and pre-processing operations (e.g. wavelet transforms to the image domain (time domain, Fourier domain, short-time Fourier domain), and back to the image domain, and / or any other suitable type of pretreatment. ,These techniques are useful for image reconstruction because some regions may show more noise than others. Additionally or alternatively, some In some embodiments, the image may be processed on a per-channel basis (e.g., different markers may be present). (Denoise per marker as each may exhibit different noise). Some embodiments In this study, the trained machine learning model processes the immunofluorescence images and thresholds them. Perform background subtraction by:

[0054] We have repeatedly stained the same image with different markers in the MxIF image acquisition process. This requires local tissue damage each time the marker is washed out of the tissue sample. As a result, individual cells and / or cell populations can be may be damaged, such as washed away and / or shifted with respect to their original placement in the material. The use of such unintentionally modified parts of the tissue specimen may result in poor imaging performance. This can cause undesirable effects in the pipeline. As a result of the damage, inaccurate information about the damaged cells is generated in the feature values, cell groupings, and If implemented at all, the prior art requires manual Therefore, we performed an autopsy on the immunofluorescence images. Develop an automated tissue degradation check (e.g., nuclei acquired over multiple staining steps) Check and / or identify areas of tissue damage (by comparing markers) The identified regions indicate that only the intact parts of the tissue sample are included in the image processing pipeline. In order to ensure that the data is analyzed by the subsequent steps (e.g., the The cell typing component 340 and the cell morphology evaluation component of the image processing pipeline 300 are Steps performed by component 350 and / or characterization component 360 ) to skip damaged areas from processing. A mask may be used to prevent tissue samples with severe damage (such as damaged tissue). The techniques described herein (as compared to conventional techniques that would not be able to process tissue samples) This may allow the tissue to be treated using techniques.

[0055] In some embodiments, at least one MxIF image comprises a plurality of immunofluorescence images. Obtaining information indicating the arrangement of cells within at least one MxIF image is useful for multiple immune responses. The method includes analyzing the fluorescent image to identify one or more damaged portions of the tissue sample.

[0056] In some embodiments, analyzing the plurality of immunofluorescence images includes analyzing the plurality of immunofluorescence images. A third trained neural network is used that is configured to identify differences between In some embodiments, the method includes processing the plurality of immunofluorescence images. a training set of immunofluorescence images containing pairs of force images and immunofluorescence images of the same marker and tissue sample associated output data containing information indicating whether at least a portion of the and further comprising training a third trained neural network using the data.

[0057] In some embodiments, the method comprises: The corresponding portion of each immunofluorescence image is input into a third trained neural network. wherein at least two immunofluorescence images contain images of the same marker. and determining from the third trained neural network at least one of the portions of the tissue sample. In some embodiments, the method further comprises obtaining a classification of at least one of the A classification comprises one set of classifications. In some embodiments, the set of classifications comprises a The first category is without cells, the second category is undamaged cells, and the third category is damaged cells. In some embodiments, the classification includes: For each classification, the set includes an associated confidence that the classification applies to the part and a confidence and selecting a final classification from the set of classifications for the portion based on the reliability.

[0058] In some embodiments, the method comprises the steps of: final classification of the portion and at least two immunological generating a patch mask indicative of the final classification for a plurality of other portions of the immunofluorescence image. In some embodiments, the method comprises: Remove a portion of the mentation mask and extract the tissue sample associated with the removed portion. The method further includes preventing the cells from being included in multiple cell populations.

[0059] In some embodiments, the third trained neural network comprises one or more In some embodiments, the third trained neural net includes a number of convolutional layers. The work includes at least 5 million parameters. The neural network generates signals that represent different parts of the tissue sample, each containing at least one classification. In some embodiments, at least one classification is configured to output a reliability. In some embodiments, the set of classes includes a set of classes, such as cells. The first category of damaged cells, the second category of undamaged cells, and the third category of damaged cells. classes, or any combination thereof.

[0060] We have investigated the conventional cell segmentation algorithms used to process MxIF images. We understand the deficiencies of the approach. To process MxIF images, the prior art typically In this section, cell segmentation is performed based on nucleus thresholding, allowing users to select different images. Manually adjusting settings and / or identifying cell membranes (e.g., if the user is looking for a nucleus) (by expanding each nucleus by contouring it with the desired number of pixels) An example of such a conventional approach is CellProfiler (compared to CellProfiler). To illustrate the improvements to the cell segmentation techniques described herein, the following will be described with reference to Figures 52-53. further explained below), Ilastik (e.g., https: / / www.nature.com / articles / s41 Stuart Berg et al., "ilastik: interactive machine learning," available at http: / / www.stuartberg.com / , 592-019-0582-9. learning for (bio) image analysis” (September 2019), and QuPath (e.g., Peter Bank, available at https: / / www.nature.com / articles / s41598-017-17204-5 head et al., “QuPath: Open source software for digital pathology image analysis” (20 Some conventional approaches use neural networks to Although neural networks can be used, the use of such networks is typically limited to nuclear detection. Therefore, conventional approaches to determine cell segmentation information As a result, we have achieved a high level of accuracy compared to conventional cell segmentation methods. The drawbacks of mobile technology are that it requires manual user input and is often poorly and inconsistently used. cell segmentation results (which are useful for cell typing, cell morphology assessment, and specific Further downstream steps that utilize cell segmentation data, such as sex determination I understand that there are various flaws, including the possibility that

[0061] Some embodiments use a trained neural network to generate cell segmentation data. The trained neural network is then run on the We can train the network using images with different densities to adapt it to MxIF images. As a result, these techniques make the trained model robust to signal variations. This can be costly, requiring the user to manually adjust settings for each individual immunofluorescence image. Therefore, such a technique offers the ability to rapidly process large numbers of immunofluorescence images. can be used (e.g., requiring manual adjustment and curation for each step) (Compared to the prior art)

[0062] In some embodiments, the method further comprises: providing information indicative of the arrangement of cells within at least one MxIF image; Obtaining a fourth trained image in at least one channel of at least one MxIF image This involves applying a neural network to generate information indicative of cell placement. For example, in some embodiments, a neural network model (e.g., a convolutional The neural network (NN) is used to identify cell boundaries in one or more immunofluorescence images. The output of the neural network can be applied to, for example, generate peaks that indicate cell boundaries. and / or at least in part (e.g., partially) by identifying the by identifying pixels that are within the cell boundary (or entirely within the cell boundary) in any suitable manner. To identify cell boundaries, the neural network model , can be applied to one or more immunofluorescence images generated using membrane markers. In some embodiments, the fourth trained neural network is based on the U-Net architecture. Implemented using a neural network architecture or a region-based convolutional neural network architecture. In some embodiments, the fourth trained neural network comprises at least Both contain 1 million parameters.

[0063] In some embodiments, a fourth trained neural network (e.g., a CNN) is a set of training immunofluorescence images of a tissue sample as input images and the location of cells in the input images. and associated output images containing information indicating Any suitable training technique for training a neural network may be used, including those described herein. Aspects of the technology are not limited in this respect and may be applied.

[0064] It should be understood that the embodiments described herein can be implemented in any number of ways. Examples of specific implementations are provided below for illustrative purposes only. The embodiments and features / functionality provided are not intended to limit the scope of the technology described herein. So they can be used individually, all together, or in any combination of two or more. For example, the various figures may be used in accordance with the techniques described herein. It describes the various processes and sub-processes that can be performed by , can be used individually and / or in combination with each other.

[0065] Multiple processes and sub-processes may each be performed on the same MxIF image and / or This may be performed using one or more types of information determined based on the image. The characteristics described in can be obtained from the same set of MxIF images and / or the same tissue sample.

[0066] The techniques developed by the present inventors offer advantages over conventional techniques for processing MxIF images. As described herein, some of these techniques are mechanically Using learning methods to identify cell types in tissue cells, cell segmentation data data (e.g., cell arrangement in a tissue sample, cell size and / or shape, cell segmentation, etc.) Determining the data (e.g., data showing the mask) and determining the tissue damage during the MxIF staining process Checking for defects (e.g., by analyzing images taken at different staining steps) by identifying cellular communities (e.g., representing different structures in a tissue sample and different To identify a group of cells that may be of a particular cell type, and / or to identify a group of cells that may be of a particular cell type for MxIF image processing. To remove noise from MxIF images (e.g., to remove background noise) This involves performing one or more of the following:

[0067] These machine learning methods allow us to achieve results that would otherwise be impossible with conventional techniques. In addition, a fully automated MxIF image processing pipeline is realized. ,machine learning models are pre-trained neural networks to implement such methods. Neural networks are trained on a large training dataset containing specific input data. The associated output data discovered and evaluated by the inventors is described in the present specification. For example, machine learning techniques can be trained to perform the methods described in this document. The authors propose that a neural network receives as input (a) an image containing nuclear information (e.g., DAPI image), (b) immunofluorescence image (e.g., cell membrane, cytoplasm, etc.), and (c) cells within the image. It receives information about the arrangement of markers in the input image and calculates the likelihood that the cells are expressed by the markers. We have discovered that the likelihood can be trained to serve as a force for the cell under analysis. These may be used to generate marker expression signatures that can then predict cell types. As another example, the inventors have compared the data with cell typing data to estimate the The neural network receives as input a graph with nodes representing tissue cells. This outputs higher dimensional data that can be used to group cells into communities. We discovered and understood that a graph can be trained to do the following for each node: For example, the cell belongs to one of several groups, the cell location information, and the node in the graph. Data for associated tissue cells, such as the typical edge length for the edges of the grid Graph neural networks cluster cells into cell communities. Embedding node data into a higher dimensional space that can be used to rasterize As a result, the trained neural network can perform better than other models using conventional techniques. These and other examples are provided in this specification. This is further explained in the book.

[0068] A neural network is trained using such training data, and the trained neural network A neural network is a network that allows a neural network to perform its associated function. Such a machine learning model determines the final parameters of the network. As described in , and / or have a huge number of parameters, such as hundreds of millions of parameters. As also further described herein, the trained neural network , capable of performing tasks in an automated, repeatable manner and with a high degree of precision ( For example, a trained neural network can be used to train a model using imaging data. Robust enough to be immune to imaging noise and / or signal level distribution differences within the (Because it is trained using training data that makes it robust).

[0069] 1 and 2 are diagrams showing an overview of the technology described in this specification. 1 illustrates an exemplary system 100 for MxIF image processing, in accordance with some embodiments of the described technology. The microscope 110 and the computing device 112 are In the example, anti-human CD31 antibody, a fluorescent marker for CD8 T cells, CD68 antibody, and NaKATPase is used to acquire a set of one or more MxIF images 102 of the tissue sample, including: In some embodiments, different markers are displayed in different colors in the MxIF image 102. For example, CD31 antibodies can be shown in red and CD8 T cells can be shown in green. , CD68 antibody can be shown in magenta, and NaKATPase can be shown in gray. The computing device 112 transmits the MxIF image 102 to a computing device via a network 114. The received signal is transmitted to the audio device 116.

[0070] Tissue samples include, but are not limited to, blood, one or more body fluids, one or more cells, , one or more tissues, one or more organs (which may, for example, include multiple tissues), and / or any other biological sample obtained from the subject, including any other biological sample from the subject. A tissue sample can be a sample containing cellular material (e.g., one or more cell types). one or more cells) and / or extracellular material (e.g., extracellular material connecting cells matrix, cell-free DNA in the extracellular components of tissue, etc. Tissue samples were analyzed in vitro at the tissue and / or cellular level as described above. obtain.

[0071] The MxIF immunofluorescence images described herein may be one or more immunofluorescence images of the same tissue sample. For example, MxIF images may be taken of the same tissue sample. a single multi-channel image (e.g., each channel associated with a different marker) As another example, the MxIF images may be multiple images of the same tissue sample (e.g., Each of which may have one or more channels. For example, one or more Referring to the MxIF image 102, in some embodiments, the MxIF image 102 contains CD31, CD8 In a single image with multiple channels for each of the markers: CD68, NaKATPase, and CD68. (e.g., so that the MxIF image 102 is a four-channel image). 2. There are four one-channel images, each containing a separate immunofluorescence image for each marker. As a further example, the MxIF image 102 may include one or more immunofluorescence images having multiple channels. Immunofluorescence images (e.g., two MxIF images with two channels each, and / or similar It may contain

[0072] Although not shown in Figure 1, there are several methods for identifying cellular structures or subcellular compartments. Any marker known in the art (e.g., membrane marker, cytoplasmic marker, nuclear marker) may be used. Various other types of markers can be used, including genes (such as For example, it may be a DNA or RNA encoding a protein, or a protein. Markers include intracellular markers (e.g., intracellular proteins), membrane markers (e.g., , membrane proteins), extracellular markers (e.g., extracellular matrix proteins), or It may also be a combination of two or more of them. Another example of a marker is PCK26 antibody. , DAPI (for DNA), carbonic anhydrase IX (CAIX), S6, CD3, and / or similar. Further examples of markers include genes (and the proteins encoded by such genes). proteins), namely ALK, BAP1, BCL2, BCL6, CAIX, CCASP3, CD10, CD106, and CD11b. , CD11c, CD138, CD14, CD16, CD163, CD1, CD1c, CD19, CD2, CD20, CD206, CD209, CD2 1, CD23, CD25, CD27, CD3, CD3D, CD31, CD33, CD34, CD35, CD38, CD39, CD4, CD43, C D44, CD45, CD49a, CD5, CD56, CD57, CD66b, CD68, CD69, CD7, CD8, CD8A, CD94, CDK1 , CDX2, Clec9a, chromogranin, collagen IV, CK7, CK20, CXCL13, DC-SIGN, Desmi EGFR, ER, ERKP, fibronectin, FOXP3, GATA3, GRB, granzyme B, H3K36TM , HER2, HLA-DR, ICOS, IFNg, IgD, IgM, IRF4, Ki67, KIR, Lumican, Lyve-1, Mamma Globin, MHCI, p53, NaKATPase, PanCK, PAX8, CNAP, PBRM1, PD1, PDL1, perlecan , PR, PTEN, RUNX3, S6, S6P, SMA, SMAa, SPARC, STAT3P, TGFb, Va7.2, vimentin, and / or other markers. The markers may be any marker that selectively binds to the marker of interest. The antibody or other labeled binding agent can be used to detect the antibody or other labeled binding agent. The agent (e.g., labeled antibody) may be a luminescent (e.g., fluorescent), chemical, enzymatic, or other label. As a result, the tissue images described herein can be labeled with fluorescent markers, chemical markers, or other labels. The signals from various types of markers, including markers, and / or enzyme markers, are used. Thus, in some embodiments, the images described herein (e.g., For example, MxIF images can contain information from non-fluorescent as well as fluorescent markers. However, in some embodiments, the MxIF image may be The image contains only information from the fluorescent signal. In some embodiments, the fluorescent information is Information from fluorescent stains (e.g., DAPI) in addition to information from fluorescent antibodies against specific markers This may include information.

[0073] In some embodiments, immunofluorescence signals from tissues or cells are detected using fluorescently labeled antibodies. contacting the body with tissue or cells (e.g., fixed and / or sectioned tissue or cells); , detecting a fluorescent signal (e.g., using a fluorescent microscope) and detecting one or more It is obtained by determining the presence, location, and / or level of markers. In some embodiments, the fluorescently labeled antibody is a primary antibody that binds directly to the marker of interest. In some embodiments, the fluorescently labeled antibody binds directly to the marker of interest. A secondary antibody is a secondary antibody that binds to an unlabeled primary antibody. For example, a secondary antibody is a It can bind to the primary antibody when raised against the antibody's host species. Different techniques can be used for fixing and / or sectioning the cells. Alternatively, the cells may be fixed with formaldehyde or other reagents. In some cases, tissues can be fixed by vascular perfusion with a fixative. The tissues or cells may also be fixed by immersion in a fixative solution. In this method, fixed tissues or cells are dehydrated and embedded in a material such as paraffin. However, in some embodiments, the tissue or cells are preserved in paraffin or other media. As opposed to being embedded in a material, tissue can be frozen to preserve its morphology. In embodiments of the present invention, tissues or cells (e.g., fixed, embedded, and / or Frozen tissue or cells can be sectioned, for example, using a microtome. In some embodiments, the sectioned tissue or cells are subjected to microscopic examination using a microscope. The tissue or cells may be mounted on a slide or other suitable support. (e.g., mounted sectioned tissue or cells) are incubated with one or more primary antibodies and and / or contacted with a secondary antibody (e.g., several incubations, blocking After the incubation (with subsequent incubation and / or washing steps), labeled tissue or cells can be obtained. do.

[0074] The computing device 116 processes the MxIF image 102. The device 116 processes the MxIF image 102 to determine the location of cells in the tissue sample (e.g., the location of the image of the tissue sample). (by segmenting the image into cells) and distinguishing different types of cells in tissue samples. In some embodiments, the computing The device 116 identifies the plurality of populations of cells in the tissue sample at least in part by (a) an MxIF image 102 and and information indicating the location of at least some of those cells is used to determine feature values ​​for at least some of the cells (e.g., a cell mask For at least some of the cells identified by the cell location information, (b) determining feature values ​​for at least one of those cells using the determined feature values; and by grouping some cells into groups.

[0075] The computing device 116 may analyze one or more of the tissue samples using the plurality of cell populations. In some embodiments, the computing device 116 determines a characteristic of Determine information about cell types in a tissue sample. For example, by using a computing device 116 can determine the cell types of cells in multiple cell populations in a tissue sample. Examples of cell types determined by the routing device 116 are endothelial cells, epithelial cells, macrophages, and phages, T cells (e.g., CD3+ T cells, CD4+ T cells, or CD8+ T cells), malignant cells, NK The cells may include one or more of: cells, B cells, and acinar cells. The imaging device 116 may select cell types based on user input and / or using artificial intelligence techniques. For example, the computing device 116 may determine one or more Multiple pre-trained neural networks are used to process multiple cell populations, and the cells within each cell population are analyzed. For example, as described herein, cell type can be determined by cell group information about the cell, as well as other relevant information (e.g., cell shape / size, neighbors in the graph) The predictions are based on the elements, and are processed by a neural network to produce the predicted details. As further described herein, the cell type can be determined. A neural network may be trained using a set of training data, for example. Each set consists of one or more cell populations and / or other relevant input data (e.g., The input data includes the cell shape, mask, etc., and the details of each cell group in the input data. Specify associated output data that identifies the cell type. The computing device 116 processes the plurality of cell populations based on the user input and generates a The cell types within the population can be determined. For example, user input can be Manual identification of visible cells, possible cell types of grouped cells, and / or similar It can contain things.

[0076] In some embodiments, the computing device 116 may include one or more The percentage of multiple cell types can be determined. The imaging device 116 detects endothelial cells, epithelial cells, macrophages, T cells, malignant cells, etc. in a tissue sample. Determine the percentage of one or more of: cells, NK cells, B cells, and acinar cells. It is possible.

[0077] The computing device 116 may collect cell location, cell type, and / or other information (e.g., For example, information about the physical parameters of cells (e.g., cell area, density, etc.) to determine a characteristic 106 of the tissue sample, which may be a function of adjacent cells in the tissue sample (e.g., adjacent This includes determining information about the cell type and / or tissue composition of the cells. For example, The computing device 116 determines neighboring cell types of cells of the cell type of interest. Such neighboring cell type information can be used, for example, to identify cells of the cell type of interest. At least some of the (a) closely clustered within one or more clusters (b) whether the cells of interest are mostly adjacent to each other (e.g., if the cells of interest are mostly adjacent to each other); whether the cells of interest are distributed throughout the body (e.g., whether the cells of interest are distributed among other types of cells in the tissue sample) (c) grouped with one or more other cell types in the tissue sample; Whether the cells of interest are grouped (e.g., whether the cells of interest are grouped with one or more other cells in the tissue sample) cell types), and / or other cell neighbor information. Cut.

[0078] The computing device 116 may collect statistical information (e.g., cell distribution information), spatial information (e.g., MxIF analysis, such as distance between cell types, morphological information, and / or the like. image, information 104 (e.g., cell type and / or cell location), and / or information 106 (e.g., one or more further analyses of the tissue sample based on the cell-neighboring elements and / or cell organization) For example, the statistical information may include a characteristic 108 of at least some of the plurality of cell types. The distribution information can include information about the distribution of cells, for example, the distribution of different cell types. distribution between cells (e.g., two cell types distributed near each other, or in one or more regions) are mixed with each other within the area, separated by distances, and / or similar the distribution of one or more cell types in a tissue sample (e.g., high or low in cells); information about one or more regions within a tissue sample, and / or other distribution information. It can be done.

[0079] As another example, in some embodiments, spatial information is obtained from at least one of a plurality of groups of cells. Both of these may contain information about the arrangement of some cells. For example, spatial information may be Information about the spatial organization of one or more cell types in a tissue sample (e.g., cell type information about the cellular composition of tissue samples, comparison with other cell types, and / or As another example, the spatial information may include one or more One or more regions of the tissue sample containing cell types (e.g., one or more regions exceeding a threshold value) The information may include information about a region containing a large number of cells of a given cell type.

[0080] By way of further example, in some embodiments, the morphological information may include the morphology of cells and and / or structure (e.g., cell shape, structure, morphology, and / or size), tissue samples. Contains information regarding the form and / or structure of, and / or the like.

[0081] The computing devices 112 and 116 may be laptop computers, desktop computers, laptops, smartphones, cloud computing devices, and / or or any other computing device capable of executing the techniques described herein. The network 114 may be any computing device, including: Local Area Network (LAN), Wide Area Network (WAN), Internet, and / or the like, including wired and / or wireless network connections , can be any type of network connection. Although two computing devices 112 and 116 are shown communicating, one a computing device or configuration having two or more computing devices, No network (for example, if you only use one computing device) Alternatively, configurations with multiple networks and / or other configurations may be used. should be understood.

[0082] FIG. 2 illustrates an MxIF image and a MxIF image according to some embodiments of the techniques described herein. FIG. 200 shows related data that can be generated by processing an image. The xIF image 202 shows the results of PCK26 antibody, CD8 T cells, CD31 antibody, CD68 antibody, DAPI (for DNA), and and / or various markers such as those described herein. The image 202 can be used to determine various information about the tissue sample (e.g., as described herein). The information can be processed (using AI-based techniques) to identify the arrangement of cells in the tissue sample. The information may include segmentation information 204. The information may include information regarding the location of the stroma in the tissue sample. a stromal mask containing information about the tumor cells in the tissue sample and / or a tumor mask containing information about the tumor cells in the tissue sample One or more masks 206 that may be applied to the image to identify tissue features, such as a mask The information may also include information 208 about cell location and / or cell neighborhood. The information may include information about the elements of the cell. The information may be relevant to the analysis of cell types and cell subpopulations. Such information may further include information 210, as shown at 212, Raw MxIF images 202 are processed to identify endothelial cells, macrophages, T cells, and malignant cells in tissue samples. can be used to identify cellular and / or histological aspects of tissue samples, including sexual cells. do.

[0083] FIG. 3 illustrates processing an MxIF image of a tissue sample according to some embodiments of the techniques described herein. Exemplary components of a pipeline for processing and characterizing cells in tissue samples 3 is a diagram 300 showing an MxIF image, generally referred to herein as an MxIF image 310. As explained above, each MxIF The image can be an image of the fluorescence of one or more markers applied to the tissue sample. The pipeline is an MxIF image processor that performs one or more preprocessing steps on the MxIF image. Pre-processing includes a pre-processing component 320. Pre-processing includes background subtraction (e.g., MxIF image (subtract background noise from the image), changing the image resolution, downsampling rendering, resizing, filtering, and / or other types of The pipeline can include one or more of the following image pre-processing steps: Information about the arrangement of cells within the cell (e.g., cell segmentation masks, Cell segmentation (by segmenting the image to identify the location) It also includes a configuration component 330.

[0084] The pipeline uses cell location information from the cell segmentation component 330 and and cell typing ( For example, to perform cell type determination and / or to separate cells in a tissue sample into multiple different groups. The pipeline also includes a cell typing component 340 that groups the cells into cell segments. Cell morphology assessment component 3 using data from the annotation component 330 The cell morphology evaluation component 350 includes cell area, cell perimeter, cell size, and The pipeline determines cellular parameters, such as cell populations and / or the like. Also included is a component 360, which includes the cell typing component 340 and cell morphology evaluation. The characterization component 360 uses data from both the cell information about the distribution of the cells, the distance between cells, and other information as described herein. One or more properties of the textile sample can be determined. In some embodiments, Distance information is used to determine the distance between two cells that does not cross through another cell or structure (or part of it). For example, the distance information may be the length of the shortest path along a portion of the tissue sample. For example, if two cells are separated by an acinus, the distance may be a measure of the distance around the acinus (rather than the distance through the acinus).

[0085] FIG. 4A illustrates a block diagram of the components of FIG. 3 in accordance with some embodiments of the techniques described herein. 4A illustrates an example processing flow 400 for an MxIF image 300 using some of the As shown on the left, an MxIF image 310 is processed using the MxIF image preprocessing component 320. In this example, the MxIF image preprocessing component 320 preprocesses each processed M xIF images 410A, 410B, 410C, through 410N (collectively referred to herein as processed MxIF images 410) The present invention generates a processed MxIF image for each of the MxIF images 310, which is shown as a For example, the processed MxIF image 410 may undergo background subtraction, as described herein. The processed MxIF image may be subjected to processing such as removing noise. 410 is then provided to the cell typing component 340 and / or any other FIG. 4A shows the image data processed by the MxIF image preprocessing component 320. While each of the MxIF images 310 shown is for illustrative purposes and not limiting, and only one or more of the MxIF images 310 are processed by the MxIF image pre-processing component. The data can be processed by the server 320.

[0086] In some embodiments, the computing device may store the separate processed MxIF images. The image 410 is used to generate a combined image used by the cell clustering component 116. Generate a multi-channel processed MxIF image with different markers (e.g., each channel has a different For example, as explained above, each marker image is The image is processed to perform background subtraction independently for each marker in the image. For example, each marker may be used to subtract the noise associated with that marker. (e.g., because noise may be different for each marker). The computing device is configured to allow the cell typing component 340 to Cell clustering can be performed using a single image containing channels for each of the Additionally or alternatively, a combined image can be generated such that The typing component 340 performs cell clustering using multiple images. It is possible.

[0087] As also shown in FIG. 4A, the cell segmentation component 330 At least some of the MxIF images 310, shown as MxIF images 310A and 310C in 42, which shows the arrangement of cells in the tissue sample captured by immunofluorescence images using 0. The MxIF image 310 is preprocessed using the MxIF image preprocessing component 320. (used by the cell segmentation component 330, not shown in FIG. 4A) (To generate a processed MxIF image that can be used for cell segmentation). The location information 420 may include cell segmentation data, such as a location mask. is provided to the cell typing component 340 and / or otherwise utilized. It is made possible.

[0088] It should be understood that FIG. 4A is intended to illustrate an exemplary process flow only. One or more of the components shown in Figure 4A may be optional. (e.g., MxIF image preprocessing 320, cell segmentation 330, etc.). 320 is shown on both the left and right sides of FIG. 4A, but may be used in either or both orientations. Furthermore, Figure 4A shows a cell segmentation control using two MxIF images. Although a cell segment component 330 is shown, this is for illustrative purposes only. The commenting component 330 processes all of the MxIF images and / or the MxIF image preprocessing components. Any number of MxIF images 310 may be processed by the same number of MxIF images processed by the component 320. It can be used.

[0089] Each of the MxIF images 300 may be imaged using a different marker (e.g., tissue may be imaged using a different antibody marker). The tissue samples may be differently stained with Each immunofluorescence image is captured when the marker is applied. As a result, the MxIF staining process The method may be periodic, staining the tissue sample for imaging, preparing the subsequent staining, and The present inventors have found that the marker is not a target for tissue analysis. I understand that local tissue damage may occur whenever the material is washed away from the skin. For example: Some staining methods, such as cyclic immunofluorescence (CyCIF) staining, (e.g., the use of hydrogen peroxide) It can be destructive (depending on the application) and cause some localized tissue damage. As a result, individual cells and / or cell groups are washed away with respect to their original location in the sample and / or Such damage may result in injury or damage to the Through the image processing pipeline, starting from the previous step and then going through the subsequent steps, This may give erroneous information about damaged cells that are processed (e.g., further damaged cells (Potentially exacerbated by M Check for tissue degradation during one or more steps of the xIF imaging process This may include:

[0090] FIG. 4B illustrates a tissue degradation check component in accordance with some embodiments of the technology described herein. 4B illustrates the exemplary process flow 400 of FIG. 4A, which also includes a component 430. To avoid affecting the tissue analysis, the tissue degradation check component 430 checks for damage. Cells can be detected and excluded from tissue analysis. In this study, cell markers in immunofluorescence images from different stages were compared, thereby revealing the tissue structure. Changes in structure can be detected (e.g., changes in the cell nucleus, cell boundaries, etc. over time). For example, by comparing the expression of nuclear markers (e.g., DAPI) from different stages. Immunofluorescence images were compared to identify changes in cell nuclei arrangement in tissue samples over time. It can be monitored.

[0091] In some embodiments, the tissue degradation check component 430 is a trained neural network. Using neural networks to compare markers from different immunofluorescence imaging stages Neural network models can be used, for example, as described in the entirety of this application. Kaiming He et al., incorporated herein by reference, available at https: / / arxiv.org / abs / 1512.03385. "Deep Residual Learning for Image Recognition", arXiv:1512.03385v1 (December 2015) It can be implemented based on the ResNets model described in [1]. The states can contain a variable number of parameters. Such a model requires at least 500,000 parameters. Many parameters, such as 1,000,000 parameters, or even more. In some embodiments, such a model may include several thousand Million parameters (e.g., 10 million parameters, 25 million parameters, 50 million parameters) It can contain a parameter (or more parameters). For example, The number of parameters can range from 11 million to 55 million parameters, depending on the implementation. In some embodiments, the parameters are at least 100 million parameters (e.g., For example, at least 100 million parameters, between 1 million and 100 million parameters, hundreds of millions parameters, at least 1 billion parameters, and / or any suitable number or range of parameters As another example, a neural network model can include parameters within a range: , see, e.g., https: / / arxiv.org / abs / 1905.1 , which is incorporated herein by reference in its entirety. Mingxing Tan and Quoc Le, “EfficientNet: Rethinking Model Sc,” available from 1946. "Aligning for Convolutional Neural Networks," arXiv:1905.11946v5 (September 2020) It can be implemented based on the EfficientNet model. The form may be 500,000 parameters, at least 1 million parameters, or several million parameters ( 5 million parameters), tens of millions of parameters (e.g., 10 million parameters), data, 25 million parameters, 50 million parameters, or more Any number of parameters can be included. For example, the number of parameters can be 5 million. The number of parameters may range from 0 to 60 million. ,parameters are at least 100 million parameters (e.g., at least 100 million parameters). parameters, between 1 million and 100 million parameters), hundreds of millions of parameters, at least 1 billion parameters parameters, and / or parameters within any suitable number or range. do.

[0092] In some embodiments, the trained neural network A set of immunofluorescence marker images (e.g., DAPI marker images) of the same tissue sample taken A set of immunofluorescence marker images can be taken as input. Multiple immunofluorescence imaging loops, such as the base loop and test loop, were used. In some embodiments, the trained neural network ,can take as input a portion of an immunofluorescence marker image.,For example, some In this embodiment, the set of immunofluorescent marker images is processed in smaller portions. The immunofluorescence images can be processed using a sliding window across the image to identify the specific regions. A window can be specified as a specific width (e.g., 128 pixels, 256 pixels, etc.) and height (e.g., 128 pixels, 256 pixels, etc.), and the number of channels representing the number of markers (for example, 2, 3, 4 The sliding window can be a pre-configured pattern. to move across the immunofluorescence image and process the complete content of the immunofluorescence image. For example, the sliding window starts from the top left corner of the first line of immunofluorescence images. and move horizontally across the immunofluorescence image until you reach the right side of the immunofluorescence image. Go down to the second line of the fluorescent image and work again from left to right until you reach the bottom right of the immunofluorescent image. This can be repeated with . Thus, the trained neural network is It can be used as a convolution filter for various image processing. (part of) may be normalized, such as by using z-normalization and / or any other normalization technique as appropriate. and can be normalized.

[0093] In some embodiments, the output of the neural network comprises at least one A value indicating whether the dot is associated with damaged tissue (e.g., a binary value and / or The probability that the window is associated with damaged tissue can be expressed as: In some embodiments, the output may include a value for each of multiple classes, At least one of the number classes is associated with an area showing signs of tissue damage. For example, if there are three classes (e.g., OK, EMPTY, and DAMAGED regions of a tissue sample), In this case, the output is the probability for each corresponding class of each window in the immunofluorescence image, In some embodiments, for each window, the maximum The class with the highest probability is selected for that window, A final class or classification (e.g., damaged tissue or not) is selected. The final output of the check process combines the classes determined for each window, as shown in Figure 4C. Patch masses reflecting various classes of relatedness, as further explained in As a result, in some embodiments, the patch mask can be The granularity given by the window size indicates whether the tissue is damaged.

[0094] In some embodiments, the annotated dataset (e.g., one or more pathologies) (annotated by scholars) are used to train neural networks, This allows for classification of portions of the tissue sample into a set of classes. The dataset used to train the neural network is 2500 annotated DA datasets. PI marker images (although the same marker may be used in multiple steps of the imaging process) Any marker can potentially be used as long as it is used in the group or step. (It should be understood that the images can be used to train neural networks.) It is annotated with a set of classes that can be used to classify new data into classes. (e.g., determining the likelihood that new data corresponds to each of the classes) As described herein, in some embodiments, The neural network then applies new data to, for example, empty parts of the tissue (e.g., cells). The first class for the intact parts, the second class for the damaged parts, a third class for parts that are watched over a period of time, a fourth class for parts that are watched over a period of time, and / or The images can be trained to classify similar ones. The images can be annotated using a tissue sampler, whereby the images are line-by-line annotated. Contains annotations for the windowed areas traversed (e.g., the image is essentially a grid (The data is divided into grids, and each block of the grid is annotated with a class.) To annotate the dataset, in one example, the Universal Data Tool (UDT) is used, and each image The windowed sub-portions of the image were classified into three classes: OK, EMPTY, and DAMAGED. In some embodiments, the input image is transformed by a number of transformations, such as an affine transformation and / or other transformations. The training process may be augmented by using one or more transformations as appropriate. This can be a process.

[0095] In some embodiments, the tissue degradation check component 430 detects the loss of a tissue sample. A patch mask of the tissue sample representing the injured and / or uninjured areas can be generated. For example, a patch mask is a mask that shows the parts of a tissue sample that are classified by a neural network. The parts that represent the different classes processed by the neural network (e.g., The patch mask can be used to mask damaged areas of the tissue sample. Some and / or all of the classified parts may be used to prevent further analysis of the parts. For example, a patch mask can be used to filter out the Segmentation contours (e.g., cell segmentation module) are generated from the classified regions. 330).

[0096] FIG. 4C shows two nuclei from the same tissue sample according to some embodiments of the techniques described herein. The combination determined by comparing marker images 440A and 440B (collectively stained section 440) 10 is a diagram showing a patch mask 444 generated based on fabric degradation information. Tissue degradation, such as the process performed by tissue degradation check component 430 of 4B. In this example, each of the two nuclear marker images 440 Use the DAPI marker during different steps or cycles of the MxIF imaging process The window of the marker image 440A is generated by imaging the tissue sample with the The window 442A of the marker image 440B and the window 442B of the marker image 440 are the windowing parts of the marker image 440. By comparing the time points, tissue disturbance across MxIF imaging steps can be checked. A window is moved across the marker image 440 to iteratively process the marker image 440. The windowed portion of the marker image 440 is depicted as described herein. ,The trained neural model classifies the patches of tissue samples into three,classes (empty, damaged, and intact). The classes with the highest probability are compared using a neural network and assigned to each window. The resulting patch mask 444 is mostly or completely empty. Section 444A to indicate the windowed portion of the tissue sample classified as Assemble these parts sufficiently intact to be OK for further analysis. Section 444B for windowed sections of tissue samples containing tissue, and Section 444C There are sufficient injuries so that section 444C should be excluded from further analysis. cells (e.g., one or more damaged cells, a number of damaged cells above a predetermined threshold, etc.) Including section 444C.

[0097] FIG. 4D illustrates a method for filtering a region of a tissue sample to be processed, according to some embodiments. 4C illustrates the use of patch mask 444 to isolate damaged cells. This can be used to filter out section 444C, which contains As shown, patch mask 444 is used to filter out section 444C. A segmentation mask 446 is applied to the cell type as described herein. In the image processing pipeline, including segmentation and further tissue analysis A filtered segmentation mask that can be used (instead of segmentation mask 446) 448. By using the filtered segmentation mask 448 , image processing pipeline (e.g., cell typing component 340 in Figure 3, cell morphology The evaluation component 350 and / or the characterization component 360) may include cell typing. and / or not process Section 444C that has been removed from the mask for characterization. .

[0098] The tissue degradation check module 430 is similar to the cell segmentation module 3 in FIG. 30, but it is understood that this is for illustrative purposes only. For example, the tissue degradation check should be performed by the cell segmentation module 330. As part of the MxIF image pre-processing module 320, It may be performed at any other point in the process. In this embodiment, the tissue degradation check is performed by performing background subtraction on the MxIF images. Furthermore, the tissue degradation check module 430 is optional and may be performed after Therefore, tissue degradation checks do not need to be performed as part of the image processing pipeline. It should be understood that

[0099] The cell typing component 340 uses the processed MxIF image 410 and the alignment information 420 to and / or (e.g., based on cells exhibiting similar feature values) The cells are grouped into groups, which may be one or more of the tissue samples. This can be used by the characterization component 360 to determine the characteristics of the Determining at least one property of tissue according to some embodiments of the techniques described herein. MxIF analysis of tissue samples based on cell location data to group cells in the tissue samples for 5 is a flowchart illustrating an exemplary computerized process 500 for processing an image. The computerized process 500 may be implemented, for example, by the computing device 116 of FIG. The computing device 5100 of FIG. 51 may be implemented as a 3-4B. The device may be configured to:

[0100] In step 502, the computing device performs at least one multiplexing of the same tissue sample. An immunofluorescence image (e.g., MxIF image 310 described in conjunction with FIG. 3) is acquired. Finally, one multiplexed immunofluorescence image was obtained from a single multi-channel image of the same tissue sample, as described in Single immunofluorescence image and / or multiple immunofluorescence images of the same tissue sample (each with one or more In some embodiments, the computer The imaging device acquires multiplexed immunofluorescence images by acquiring previously generated images. (e.g., stored using one or more computer-readable storage devices) access one or more images stored in at least one communication network; For example, MxIF images can be obtained using a microscope. and imaging the tissue sample using the The information may be stored for later access and accessed during activity 502. Activity 502 may include capturing an image. The tissue sample may be taken from a patient (e.g., a patient with cancer). previously obtained from patients with, or suspected of having, or at risk of having This is an in vitro sample.

[0101] We have demonstrated that some of the processes described herein remove noise from MxIF images. It is understood that it may be desirable to process MxIF images, for example , since noise cannot affect the system's ability to determine the arrangement of cells in a tissue sample. The system performs cell segmentation (e.g., cellular segmentation) on MxIF images without removing noise. For example, cell segmentation 330 in FIG. 3 may be performed. However, noise can be a factor in cell clustering (e.g., cell typing 34 in Figure 3). 0) can cause problems, for example, low signal-to-noise The level may affect cell clustering and therefore the raw MxIF images and and / or the data generated by the cell segmentation process are used to identify cell clusters. Rings may not be enough.

[0102] In some embodiments, these techniques include at least one of immunofluorescence imaging. to generate a corresponding processed image. can include performing background subtraction. , for example, at least some noise can be removed. Noise in the image caused by the microscope that captured the image, and noise caused by surrounding cells Noise caused by aspects of the tissue, such as noise (e.g., fluorescence noise), In some embodiments, the noise may include noise caused by the noise generated by the noise source, and / or similar noise. In the image processing, different regions within the image can be processed. For example, some Areas may show more noise than others, so it is important to remove noise from specific areas in the image. Additionally or alternatively, in some embodiments, The image may be processed image by image and / or channel by channel. For example, each marker channel Noise can also be removed on a per-marker basis, as different channels may exhibit different noise.

[0103] In some embodiments, these techniques remove at least some noise. This involves applying a trained neural network model to each of the immunofluorescence images to FIG. 17 shows a live immune response according to some embodiments of the technology described herein. Noise for subtracting noise from fluorescence images (e.g., for background subtraction) Convolution to implement a trained neural network capable of predicting the subtraction threshold 17 is a diagram of a convolutional neural network architecture 1700. The architecture 1700 may be implemented using the MxIF image preprocessing 120 of FIG. 1 and / or the MxIF image preprocessing 120 shown in FIGS. 3-4B. Implemented and / or performed as part of MxIF image preprocessing, such as part of image preprocessing 320. The convolutional neural network architecture 1700 may be implemented in conjunction with other architectures described herein. It may be implemented and / or performed as part of a process and sub-process. For example, architecture 1700 may include, as part of steps 502, 504, and / or 506 of FIG. 5A: As another example, the architecture 1700 may be used to filter out noise. may be used to remove noise as part of steps 612, 614, and / or 616. As a further example, a trained neural network implemented using architecture 1700 may be The network may be configured to detect noise as part of steps 704, 706, and / or step 708 of FIG. 7A. can be used to remove

[0104] Neural network implemented according to the Convolutional Network Architecture 1700 is trained to predict a threshold image 1706 based on a raw input image 1708. As shown, the convolutional network architecture 1700 includes convolutional layers 1702A, 170 2B and 1702C are first (along the downsampling path) a sequence of raw image 1708 data. The resulting image is then applied to a lower resolution version of the sequence, and then an upsampling layer is applied. The raw images 1704A, 1704B, and 1704C (along the upsampling path) are shown as 8 with a "U" structure applied to a sequence of successively higher resolution versions of the data. In some embodiments, the convolutional network architecture 1700 may include Although not shown, the resolution of the data can be improved by using one or more pooling layers (e.g. , along the downsampling path) and one or more corresponding unpooling Such a technique can be augmented using layers (e.g., along an upsampling path). Neural network implementations can include a variety of numbers of parameters. Such models have at least 500,000 parameters, 1 million parameters, 2 million parameters, data, 5 million parameters, or even more. In some embodiments, such models can have tens of millions of parameters. For example, the number of parameters may be based on the implementation and may include at least 10 million parameters, 20 million parameters, 25 million parameters, 50 million parameters In some embodiments, the parameters may include at least At least 100 million parameters (e.g., at least 100 million parameters, between 1 million and 100 million) parameters), hundreds of millions of parameters, at least a billion parameters, and / or arbitrary Any suitable number or range of parameters may be included.

[0105] In some embodiments, the model may be pre-trained using the denoising information. In some embodiments, the model is configured to use a threshold to remove noise. In such an embodiment, the model can be trained to an appropriate threshold for noise removal. The training data may be used to train the model using threshold data, such as thresholded images representing values. Immunofluorescence images as input images so that we can learn to generate thresholded images from raw images. and the corresponding thresholded image (with noise removed) as the output image. For example, with further reference to FIG. 17, the training data may include raw images 1708 and corresponding thresholds. The image may include a quantized image 1706, which when compared have a threshold difference 1710. In this embodiment, thresholding is performed across the immunofluorescence image (e.g., using a Gaussian smoothing step). In some embodiments, the smoothing step may be performed globally (after a smoothing step such as a smear). These techniques involve dividing immunofluorescence images into sub-images for noise reduction. For example, the intensity of the markers varies across the immunofluorescence image, and therefore The threshold used for one part of the image is different from the threshold used for another part of the image. These techniques can vary, for example, to generate immunofluorescence images in 256x256, 512x512, 256x5 12, and / or into sets of sub-images of similar size. It can include.

[0106] Still referring to FIG. 5A, at step 504, the computing device Information indicating the arrangement of cells within the fluorescent image (e.g., the cells described in conjunction with Figures 4A-4B) The information indicating the arrangement of cells is obtained by acquiring arrangement information 420) of some of the cells in the immunofluorescence image. In some embodiments, the data may include data indicating the location of all the Therefore, cell location information includes cell boundary information, information from which cell boundary information can be derived (e.g., immune Fluorescence images of, for example, one or more markers expressed by the cell membrane and / or nucleus In some embodiments, the expression level of the target gene may be a marker, and / or a mask. In this case, the cellular localization for some and / or all of these cells can be determined by immunofluorescence. For example, the immunofluorescence images may all be of the same tissue sample. In some embodiments, the cell arrangement may be the same in each image. In this case, cell positioning information can be specified using a cell segmentation mask, This is applied to one or more of the immunofluorescence images, thereby determining the concentration of the cells in the immunofluorescence image. The arrangement can be identified.

[0107] In some embodiments, the mask is a binary mask and / or a multi-valued mask. The binary mask may be, for example, a mask representing one or more cells in the imaged tissue sample. , tissue structure, and / or pixel binary values ​​(e.g., presence or absence) The multi-value mask can be, for example, a representation of pixels, fine lines, or regions in the imaged tissue sample. A range of values ​​for cells, and / or tissue structures can be shown. For example, multi-value mass The markers are based on the partial presence of tissue components (e.g., components that are completely present or absent). In addition, multiple aspects of the tissue (e.g., the presence of different cells, tissue structures, etc.), and / or It may also be used to indicate other non-binary information. The mask can be created based on cell boundary information obtained from immunofluorescence images. In an embodiment, the binary cell boundary mask is used to determine the presence (e.g., presence) of detected cell boundaries in the tissue sample. for example, through white pixels) or absence (for example, through black pixels or other non-white pixels) It indicates either

[0108] In some embodiments, the computing device accesses the cell location information. For example, the cell location information may be generated by a separate computing device. and connected to a computing device (e.g., via a wired and / or wireless network) memory accessible to a computing device, transmitted over a network communications link The data may be stored in a library and / or similarly manipulated.

[0109] In some embodiments, the cell location information is generated manually and is stored in a manner consistent with the techniques described herein. FIG. 8 illustrates some of the techniques described herein. 8 shows an exemplary MxIF image 800 and manual cell placement / segmentation for an MxIF image according to an embodiment of the present invention. 8 shows an example of station data 802.

[0110] In some embodiments, the computing device comprises at least one multiple Immunofluorescence images are used to generate cell location information. For example, see Figures 3-4B. The computing device may include one of the cell segmentation components 330 or The system may be configured to perform several aspects to generate cell location information. In the method, the computing device performs one or more operations to generate cell location information. A set of channels for multiplexed immunofluorescence images is selected, and each selected channel provides information on the cell structure. For example, selected channels may include marker data that indicates cell membrane, cell type, or cell This may include data on nuclei, vesicles, and / or the like, which may generate cell location information. For example, cell nuclei can be stained with DAPI, H3K36™, and / or other markers. As another example, cell surface markers can be used to identify CD20, CD3, CD4 , CD8, and / or the like. As a further example, cytoplasmic markers In some embodiments, the spool may include S6 and / or the like. A plurality of markers are used to generate cell location information. For example, The immunofluorescence markers used to generate the staining are DAPI as a cell nucleus marker, and cell membrane Using NaKATPase as a marker, and S6 as a cytoplasmic marker of cells It can include.

[0111] In some embodiments, the computing device uses machine learning techniques. For example, the computing device may generate cell location information by performing convolutional neural network analysis. A neural network model is applied to one or more of the immunofluorescence image channels to estimate cell alignment. In some embodiments, computing devices can generate positional data. The device selects channels with cell structure information and applies a convolutional neural network model. The model is applied to a selected subset of immunofluorescence images to generate cell location information.

[0112] In some embodiments, the convolutional neural network model is a U-Net architecture. A neural network having a "U" shape, such as the architecture described in conjunction with FIG. 17, It can include a network model.

[0113] In some embodiments, the neural network architecture is a Mask R-CN Including region-based convolutional neural network (R-CNN) architectures such as N An example of Mask R-CNN can be found at https: / / Kaiming He et al., "Mask R-CNN," arXiv: 1703.06870 (January 2018). Such a model uses a large number of parameters. For example, such a model may contain at least 500,000 parameters, At least 1 million parameters, multiple million parameters (e.g., at least 1 million parameters, 2 million parameters, 3 million parameters, etc.), and / or tens of millions parameters (for example, at least 10 million parameters, 25 million parameters, 1 For example, such a model can contain 40 million parameters. to 45 million parameters (e.g., 40 million parameters, 42 million parameters, and / or 45 million parameters). The data must contain at least 100 million parameters (e.g., at least 100 million parameters, 1 million between 100 million and 100 million parameters), hundreds of millions of parameters, at least 1 billion parameters, and / or any suitable number or range of parameters. The input to the delta is multiple channels containing a nuclear marker and one or more membrane markers. For example, the input may include a nuclear marker (e.g., DA PI), a membrane marker present on most cells for the second channel (e.g., CD45 or or NaKATPase), and an additional membrane marker with sufficient staining for the third channel ( For example, three channels (including CD3, CD20, CD19, CD163, CD11c, CD11b, CD56, CD138, etc.) In some embodiments, the input channels may be normalized (e.g., For example, in the range 0 to 1).

[0114] In some embodiments, an input image (or multiple input images) may be intersected by positive The image is divided into rectangles (for example, 128x128 pixels, 256x256 pixels, etc.) and the window In such an embodiment, the network output is processed on a cell-by-cell basis. In some embodiments, the output may be an array of mask proposals for each window. Additionally or alternatively, the output may be a binary mask for a given pixel. An image with a value (e.g., a value in the range 0 to 1) representing the probability that a cell is part of a mask of cells For such an embodiment, the pixels of the output image may be a set of The mask values ​​may be thresholded and / or selected using various techniques to determine the final mask value. For example, if each pixel contains two probabilities that add up to 1, then the output pixel The pixels are thresholded using a value of 0.5 to obtain the final pixel for the binary mask. It is possible.

[0115] In some embodiments, the prediction windows include overlapping data. For example, windows may share data and / or cells may share the edge of a window. Such redundancy can occur if the This can be avoided by treating only cells from some parts, rather than all parts. For example, the center, top corner, and right corner pixels of each image are processed for each window. can only be the last window from the right or bottom (e.g., if the current window is the last window from the right or bottom). (The bottom and right parts of the image are processed only if they are black). The resulting The cell segments can be aggregated into a final output mask (e.g., integer values ​​are used to represent individual cell instances). (representing a chest of drawers).

[0116] In some embodiments, the architecture (e.g., U-Net, R-CNN, etc.) is Encoder heads such as ResNets models (e.g., ResNet-50) described in the specification (e.g., , as the first layer or layers of the model). The segmentation network is similar and / or or may be made by the same encoder head.

[0117] In some embodiments, the convolutional neural network model is a set of training immunofluorescence images as the training image, and the associated cell segmentation images as the output image. It is trained using mentation images: about 100 images, about 200 images, about 300 images, and about 2,000 images. images, approximately 3,000 images, approximately 4,000 images, approximately 5,000 images, approximately 6,000 images, approximately 10,000 images, and / or or a similar number of images, various training set sizes can be used to test neural network models. In some embodiments, the training set size can be used to train the The set size depends on the input image size, which can range from about 4,000 to 5,000 images. In some embodiments, the image squares are obtained by subtracting the original full image (e.g., randomly) sampled (e.g., thus requiring fewer training images) The training immunofluorescence images may be of several tissue samples, and the training immunofluorescence images may be of several tissue samples. The cell segmentation image associated with the optical image provides information about the arrangement of cells in the tissue sample. In some embodiments, the neural network model may include ,training using multi-channel images and / or single-channel images as input images. It can be refined.

[0118] The training images may contain multiple markers, one marker per channel. For example, a three-channel image (or multiple images) may be used as described herein. The first channel is a nuclear marker and the second channel is a marker for the tissue type of interest. is a membrane marker expressed by most cells, and the third channel is an additional membrane marker. For example, a three-channel image may be created using DAPI, NaKATPase, and S6 markers. The corresponding output images used for training can be generated using manually generated samples. A raw output image (e.g., a cell segmentation mask with cell contour outlines) In some embodiments, the training set (such as For example, input images and / or associated output images) are used for training. For example, a preprocessing step may be performed to identify boundary cells (e.g., 3 pixels of the cell boundary). It can include cells that have a number of intersecting pixels greater than a threshold, such as This can be performed on the training output images to detect the According to some embodiments of the technology described herein, a method for identifying cell location information is provided. The following shows examples of images that can be used to train a neural network: images 902, 904, and 905. Each set of 6 and 908 contains a composite immunofluorescence image, a corresponding neural network of cell arrangements, and predicted image, a training image of the cell arrangement used to train the neural network , and images of cell boundary contacts (e.g., where cells contact other cells).

[0119] Neural network models are trained using various backpropagation algorithms. In some embodiments, the convolutional neural network is configured with a learning rate of 0.00 It is trained using the backpropagation algorithm with the ADAM optimizer, where σ is 1. At each training step, the neural network model computes the input image For example, in some embodiments, neural networks are trained to generate cell location information. The network model is trained using a 3-channel input image and associated A two-channel output image (e.g., a segmentation map) that exactly matches the training output image. one channel with the boundary cell contact map and the other channel with the boundary cell contact map) Neural network models can be trained using a variety of loss functions. For example, the model is performing a pixel classification task, so a categorical cross- experiment An entropy loss function may be used.

[0120] FIG. 10 shows MxIF images obtained of a tumor according to some embodiments of the technology described herein. A trained convolutional neural network is used to process the image and generate cell segmentation data 1002. FIG. 10 is a pictorial illustration of an example of using the neural network model 1000. In the example of FIG. The MxIF image 1010 includes a first marker image 1012 and a second marker image 1014. The computing device uses the trained neural network model 1000 to The first marker image 1012 and the second marker image 1014 are processed to obtain cell location / segmentation. The application information 1002 is generated.

[0121] As explained in conjunction with Figure 17, the convolutional neural network model uses convolutions. The congested layers are bars of successively lower resolution of the data along the downsampling path 1000A. is applied to the upsampling path 1000B, and then the data is successively upsampled along the upsampling path 1000B. It has a "U" structure that applies to different resolution versions. Then, the resolution of the data is downsampled using one or more pooling layers. The resolution of the data can be reduced by 1000A, as indicated by the green arrow. Along the upsampling path 1000B using one or more corresponding pooling layers It can be raised.

[0122] FIG. 11 shows a processed immunofluorescence image according to some embodiments of the techniques described herein. A trained neural network is used to generate cell location / segmentation data 1102. FIG. 11 pictorially illustrates another exemplary use of the MxIF image 1100. In this example, 10 includes a DAPI marker image 1112, and a NaKATPase marker image 1114, where DAPI is a fluorescent DNA DAPI, NaKATPase, and / or other markers are chromosome and membrane markers, respectively. It should be understood that other markers may be used, for example, cytoplasmic Marker S6, membrane marker PCK26, carbonic anhydrase IX (CAIX), CD3, and / or similar The computing device may include a trained neural network. 1100 was used to process the DAPI marker image 1112 and the NaKATPase marker image 1114. Cell placement / segmentation information 1102 is generated.

[0123] 12-16 show immunofluorescence images and and associated cell positioning information (e.g., in this example, the cell segmentation matrix). The cell location data is accessed and / or stored by the system. or may be generated by a system such as the cell segmentation module 330 of FIG. FIG. 12 illustrates an MxIF image 1200 and an MxIF image 1210 in accordance with some embodiments of the techniques described herein. FIG. 13 shows cell segmentation data 1202 generated based on the image 1200. 13A and 13B show composite fluorescent image 1300 and composite fluorescent image 1301 according to some embodiments of the described technology. FIG. 14 shows cell segmentation data 1302 generated based on the 10. For example MxIF images acquired of kidney tissue according to some embodiments of the techniques described in FIG. 1 shows exemplary cell segmentation data. The orientation data 1402 is determined based on the MxIF images 1404 and 1406 and is shown in the exemplary cell section. Segmentation data 1408 was determined based on MxIF images 1410 and 1412. Image 1404 and 1410 contain DAPI, CAIX, PCK26, and ki67 protein markers. and 1412 contain CD31, CD8, CD68, and NaKATPase markers.

[0124] FIG. 15 illustrates a clear cell renal cell carcinoma (CCRCC) tumor cell line in accordance with some embodiments of the technology described herein. FIG. 16 shows an MxIF image 1500 and corresponding cell segmentation data 1502. , MxIF image 1600 of CCRCC and corresponding 16 shows cell segmentation data 1602.

[0125] With further reference to FIG. 5A, the computing device may perform, at least in part, steps Steps 506-508 are performed to identify groups of cells in a tissue sample. 6, the computing device receives the immunofluorescence images and the The information indicating the location of at least some of the cells is used to identify at least some of the cells. In some embodiments, these techniques determine feature values ​​that represent the Determine feature values ​​for most of the cells and / or all of the cells in the immunofluorescence image This may include:

[0126] In some embodiments, the techniques include detecting a signal in at least one of the immunofluorescence images. a pixel value associated with the location of the cell; As further described herein, Then, the feature values ​​are multi-immunity profiles for the configuration of at least some of the cells. The values ​​of pixels in the fluorescence image (e.g., pixels at or near the location of a cell) For example, multiple immunofluorescence images may be obtained by multiple When a single-channel image is included, these techniques can be used to visualize the number of cells in a single-channel image. pixel values ​​at each location (e.g., different images depending on how the image was acquired) (which may be the same layout and / or different images across As another example, if an image contains one or more multi-channel images, The placement in the channel image may be the same for each channel. In this case, the feature value includes one or more values ​​derived from the values ​​of the pixels in the cell. For example, the feature values ​​may include contributions determined for each of the channels of an immunofluorescence image. The contribution can be, for example, how much a pixel associated with a cell contributes to that channel. For example, the contribution may be 0%. for each channel ranging from (e.g., no contribution) to 100% (e.g., full contribution). The contribution can be determined by a representative value across multiple pixels in the cell location, a Average value across pixels in the cell, ratio of positive pixels in the cell to negative pixels in the cell and / or the like. It can be determined according to a variety of techniques.

[0127] In some embodiments, these techniques allow for the location of cells within an immunofluorescence image. The immunofluorescent pixel values ​​at or near the cell are used to determine the feature value for the cell. FIG. 5B shows a cell location data set according to some embodiments of the technology described herein. 6 shows an example of feature values ​​600 for associated cell configurations of data 602. For example, it can be determined by cell typing component 340 of Figure 3. In the example of Figure 5B, the feature The value 600 contains the percentage for each channel 1, channel 2 through channel N, where N is the number of cells. An integer representing the number of channels associated with the arrangement. Each channel can be, for example, 1 one or more immunofluorescent markers, one or more immunofluorescent images, and / or similar Feature value 600 indicates that channel 1 has a 25% contribution (for example, For example, channel 1 has a 40% contribution towards the cell configuration, and channel N has a 10% contribution towards the cell configuration. It is shown to have an 85% contribution towards cell alignment.

[0128] Still referring to FIG. 5A, in step 508, the computing device The determined feature values ​​are used to group the cells into groups. -Processing MxIF images of tissue samples based on cell location data according to some embodiments of the technique A computerized process for clustering cells of a tissue sample into multiple cell groups using the method of claim 1. 700. The computerized process 700 is shown, for example, in conjunction with FIG. For example, the method may be performed by the computing device 116 described in FIG. The computing device 5100 of FIG. 3-FIG. 4B may include the cell typing component 340. The computer may be configured to perform one or more aspects described in conjunction with the present invention. The data processing 700 may be used as part of other processes and sub-processes described herein. For example, the computerized process 700 may be implemented to group cells into multiple cell populations. This may be performed as part of step 508 of FIG. 5A to perform the filtering.

[0129] In step 702, the computing device performs one or more In step 704, the computing device In step 706, the computing device selects the immunofluorescence image. Identifying a set of pixels at or near the location of the selected cell in the optical image. For example, these techniques involve the detection of cells that are at least partially within the cell boundary (e.g., This may include identifying a set of pixels that are within the image (either partially or completely).

[0130] In step 708, the computing device performs a In some embodiments, these techniques involve the use of pictorial features to determine feature values ​​for the cells. The method may include determining a feature value based on pixel values ​​of the set of cells. , a representative value over a set of pixels, or one or more criteria (e.g., the presence of markers, fluorescence values ​​associated with each marker higher than a threshold, etc. ) In some embodiments, the characteristic values ​​may include values ​​described herein. Thus, one or more immunofluorescent markers and / or one or more immunofluorescent In some embodiments, the contribution of each cell in the image may be represented. The computing device generates a feature based on how the pixel values ​​contribute to the cells. For example, the computing device may determine a pixel value. The pixel values ​​of the selected set are within a certain percentage of the cell arrangement (e.g., 3 0%, 40% of cell placements, etc.) and / or It can be determined that the fluorescence present above a certain threshold is present.

[0131] In step 710, the computing device performs an additional For example, a computing device is a specific number of immunofluorescent markers (e.g., one marker, two markers, marker all of the channels, etc.), a certain number of immunofluorescence images (e.g., having one or more channels) get, one image, two images, all of the images, etc.), and / or the like As another example, the computing device may be configured to determine the characteristic value. Immunofluorescent markers and / or images associated with some feature (e.g., cells Immunofluorescent markers and / or staining associated with some markers indicating structure The computer may be configured to determine a feature value for each of a particular set of images (e.g., images). The imaging device analyzes one or more additional immunofluorescence images in step 710. If so, the computing device returns to step 704 to Select the immunofluorescence image.

[0132] The computing device analyzes the selected cells in step 710. If the computing device determines that there are no further immunofluorescence images to be acquired, Proceed to step 712 and determine whether there are one or more cell configurations for which feature values ​​should be determined. For example, the computing device may determine a particular set of cells (e.g., a particular a certain number of cells, cells in one or more arrangements, and / or the like) and / or The computing device may be configured to determine feature values ​​for all of the cells. In step 712, the method determines the feature values ​​for one or more additional cells. If so, the computing device returns to step 702 to select another cell. Select.

[0133] If the computing device determines that there are no more cells to analyze, The computing device proceeds to step 714 and performs cell clustering (e.g., For example, as part of cell typing 340, the cells may be classified into one or more subgroups based on the determined characteristic values. In some embodiments, these techniques use feature values ​​to group the cells into a number of groups. The analysis can be used to identify relationships between cells (e.g., similar marker expression, known Marker expression consistent with published cell typing data, cells are of the same type, and and / or analysis of the probability that each cell in the cell population has similar characteristics. The cells are grouped based on the relationships determined to have characteristic values ​​that indicate the relationships between cells within the cell group. This includes grouping.

[0134] These techniques use one or more methods to identify relationships between cells based on determined feature values. Multiple clustering algorithms (e.g., unsupervised clustering algorithms) This may include applying a clustering algorithm that can be used to group cells. Some examples of rhythms are hierarchical clustering, density-based clustering, k-mean s-clustering, and / or any other suitable unsupervised clustering algorithm , Self-organizing map clustering algorithm, Minimum spanning tree clustering algorithm algorithms, and / or the like, and aspects of the technology described herein may be In some embodiments, these techniques include, but are not limited to, self-assembling matrix techniques. The data can be analyzed using the FlowSOM algorithm to analyze cell clusters. Rastering can be performed.

[0135] In step 510, the computing device generates a tissue using the plurality of cell populations. and determining at least one characteristic of the sample. In some embodiments, these techniques include , generating a report indicating the at least one characteristic. , information about a plurality of groups and / or at least one MxIF image as described herein The report may also include other information, such as any other information determined regarding For example, reports may be provided to users via a graphical user interface. (for example, web-based or device-based application programs) You can view the report (in the program), save it as an electronic file (e.g., a PDF file or any the transmission of the report to the user (file in any suitable format) and / or It may be provided by any other sufficient technique for providing to a user.

[0136] In some embodiments, these techniques involve analyzing the immunofluorescence images obtained in step 502. The image is used in conjunction with the cell grouping information obtained in step 504 to identify the cells in the tissue sample. determining one or more characteristics of the tissue sample. the cellular composition of the material (e.g., cell type, cell morphology, etc.) and / or the tissue composition of the tissue sample (e.g., For example, cellular localization, multicellular structural localization, etc.) may be characterized. Additionally, the one or more characteristics may be, for example, a characteristic of a cell type (e.g., a characteristic of a cell population) in a tissue sample. In some embodiments, the marker may include a marker that is associated with a cell type. The computing device identifies the cell type of individual cells in the tissue sample. In some embodiments, the computing device performs a small number of individual cells in the tissue sample. Identify at least a threshold percentage of cell types. For example, the threshold percentage may be: At least 25%, 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, It could be 95%, 99%, etc.

[0137] As another example, the one or more characteristics may be a distribution of one or more cell types in a tissue sample. (e.g., distribution among cells of a group), distribution among different cell types (e.g., different groups of cells , distribution between , and / or similar statistical information about the distribution of cells. In another example, a computing device can determine which types of cells are different from other types of cells. Cell neighbor information indicating which types of cells are neighbors and / or what types of cells are neighbors of cells 1. Cell contact information for one or more cell types (or tissue types) of a tissue sample, including cell contact information indicating whether the cell types are in contact with other types For example, the spatial organization of the cells (where each cell type is associated with a different population) Spatial information about the arrangement of cells can be determined. The cell size, shape, and structure, and / or cell area, Morphology with respect to other aspects of the tissue, such as cell perimeter, cell size, and / or the like Any of the characteristics described herein can be used to determine biological information about a tissue sample. It should be understood that the above may be determined in some embodiments. These techniques provide information on cell types, cell distribution, spatial information, morphological information, and multicellular structure organization. Multiple characteristics of the tissue sample, such as the texture and / or any other characteristics described herein. This includes determining the characteristics of the

[0138] In some embodiments, these techniques use probabilistic analysis based on feature values. Cell typing is performed by determining the relationships between cells based on their feature values, such as by As described in conjunction with FIGS. 5A-5B, the channel contribution (e.g., For example, the average channel contribution can be used to determine cell type, which However, in some circumstances, the inventors showed that such channel contributions may not provide stable cell typing. For example, we understand that channel contributions affect the quality of segmentation, This may depend on cell size, cell shape, tissue staining conditions, and / or the like. As another example, the present inventors have found that channel contribution is related to intracellular signal localization. understand that the actual signal data is not taken into account (This may be useful to help distinguish between different sizes.) The inability to set a range of values ​​for channel contribution for a particular type of cell is a major concern. We also understand that it may be difficult to purposefully probe certain cell types (and For example, clustering may be used instead, which may not find cells of interest. As an additional example, given such potential problems, Results may need to be checked manually, which increases delays and impacts automation. This can have an impact.

[0139] To address this potential problem, we utilized marker expression signatures to We have developed techniques for cell typing using the ELISA kit. Figure 6A shows some of the techniques described herein. According to some embodiments, an MxIF image of a tissue sample is obtained based on the cell arrangement data of cells in the tissue sample. 6 is a flow diagram illustrating an exemplary computerized process 610 for processing and predicting cell type. The computerized process 610 may, for example, be a computerized system that implements the cell typing components of FIG. The computerized process 610 may be performed by the computerized component 340. The computerized process 610 may be implemented as part of the process and sub-processes of may be performed, for example, as part of steps 506 and / or 508 of FIG. 5A .

[0140] In step 612, the computing device uses MxIF imaging to For example, a small portion of the obtained tissue sample (as described in conjunction with step 502 of FIG. 5A) may be At least one multiplexed immunofluorescence image is acquired. As described herein, the marker image is Provided as separate images and / or as a combined image with multiple channels At least one immunofluorescence image may be obtained by analyzing nuclear markers (e.g., FOXP3, DAPI), membrane markers (e.g., IgG, IgE, IgM ... markers (e.g., CD3e), cytoplasmic markers (e.g., CD68), and / or the like. In step 614, the computer The imaging device obtains information indicative of the arrangement of cells within at least one multiplexed immunofluorescence image. For example, the placement information may be based on cell boundaries as described in conjunction with step 504 of FIG. 5A. information that allows for determining cell boundaries (e.g., marker expression), and / or One or more cell masks (e.g., for one or more of the received immunofluorescence images) The mask may include a cell mask.

[0141] FIG. 6B shows immunofluorescence images and a graph that can be used for cell typing, according to some embodiments. FIG. 6B shows an example of a first data set for a first cell 622A. and data for a second data set for a second cell 622B (collectively referred to as cells 622). For each cell 622, the data includes a nuclear marker (DAPI) image 624 and a cell segmentation image. A binary mask 626 and a marker image 62 that can be used to explore different cell expressions. In this example, the marker image 628 is a set of 8 markers based on the location of the marker image. Intracellular marker image 628A (FOXP3 image), membrane marker image 628B (CD3e and CD68 image), and cellular marker image 628C (CD3e and CD68 image). It can be broadly classified as cytoplasmic image 628C (CD68).

[0142] In step 616, the computing device For each specific type of marker in at least one multiplexed immunofluorescence image, determining a marker expression signature including the likelihood of each of the markers being expressed in the first cell; Therefore, cell location information (e.g., cell mask) can be used to determine the cell location in the immunofluorescence image. In some embodiments, the computing The device uses a trained neural network to determine marker expression signatures. The neural network can be used to determine whether the marker signal is related to the cell to determine the likelihood of a gene being present and expressed (or not). The neural network model can be trained, for example, as described herein. Such a model may be implemented based on a ResNets model. As described herein, , which may include a large number of parameters. For example, such a model may have at least 50 Ten thousand parameters, at least a million parameters, several million parameters (e.g., at least 1 million parameters, 2 million parameters, 3 million parameters, etc.), and / or tens of millions of parameters (e.g., at least 10 million parameters, 1500 It can contain millions of parameters, 25 million parameters, 50 million parameters, etc. For example, such a model can have 40 to 45 million parameters (e.g., 4 0 million parameters, 42 million parameters, and / or 45 million parameters) In some embodiments, the parameters may include at least 100 million parameters ( For example, at least 100 million parameters, between 1 million and 100 million parameters), hundreds of millions parameters, at least 1 billion parameters, and / or any suitable number or It can include parameters within a range.

[0143] The inputs to the model are separate 1-channel MxIF images, 2-channel MxIF images, and 3-channel MxIF images. Each of the images may be a multiple immunofluorescence image of at least one of different markers, such as an image. Immunofluorescence images are scaled to fit the height (e.g., 128 pixels, 256 pixels, etc.) and width (e.g., 1 In some embodiments, the first channel may have a pixel count of 28 pixels, 256 pixels, etc. The image or image may be, for example, a DAPI image (e.g., centered by the cell of interest). The second channel or image can be another immunofluorescent marker image (cropped as shown). For example, this may be a segmented image (which is also cropped), and a third channel or image may be segmented. The segmentation mask can be a segmentation mask of only one cell of interest. Referring to B, for example, for each cell 622, the input to the model is a nuclear marker image 624 , segmentation mask 626, and one of the marker images 628 (e.g., intranuclear image 628A , membrane image 628B and cytoplasm image 628C). In this case, the input image (e.g., MxIF image or channel) may be normalized. The pixel values ​​of the image are normalized to fall within the range 0 to 1, which corresponds to the bit depth of the image. For example, values ​​in an 8-bit image can be divided by 255, and values ​​in a 16-bit image can be divided by 255. may be divided by 65535 and / or a similar operation may be performed. Such normalization is The image pre-processing performed by the MxIF image pre-processing module 320 described in conjunction with FIGS. 3-4B. It may be used in conjunction with other processes and sub-processes described herein, such as part of image pre-processing. It is possible.

[0144] Neural networks can be used for binary classification (e.g., 0 or 1 for classification) and / or The cell is associated with an input marker image that contains signals of appropriate intensity and shape (e.g., For example, it can output a likelihood of 0 to 1. Referring to FIG. 6B, for dataset 622A, nuclear marker image 624, cell segmentation For input of the fusion mask 626 and the nuclear image 628A (FOXP3), the output is the nuclear image 628A. Therefore, the appropriate intensity and shape of the cells that will be expressed are taken as the likelihood that the nuclear image 628A has. It is possible.

[0145] The neural network is generated from at least one multiplexed immunofluorescence image (or a portion thereof) and For example, neural networks can be trained to process cell and spatial information. At least one multiplex immunofluorescence image containing cells in the center of the image as identified by a mask It may be configured to compare a portion of the image (e.g., 128x128 pixels). As described above, immunofluorescence images can be used to visualize the nuclear marker images of the region, as well as cell outline images or Other expression images may include membrane images, nuclear images, cytoplasmic images, and / or the like. A sample dataset with such information can be used to visualize the actual cell structure of the marker image. The marker image shows whether it is likely to be expressed (or not). Used to train a neural network to discriminate based on intensity and shape It is possible.

[0146] A neural network can be trained on a library of images. The network is based on immunofluorescence images and / or can be trained on cell location data. In an illustrative example, which is not intended to be limiting, Three different training sets were used: four sets for nuclear localization markers (e.g., Ki67); The first one has 2,186 images (training set) and 403 images (test set) of markers. Set,9,485 images of 28 markers for membrane-localized markers (e.g., CD3, CD20) (training set) and a second set with 1,979 images (e.g., validation set). 898 images (training set) of 8 markers with defined localization (e.g., CD68) A third set with 427 images (validation set) and 427 images (validation set) can be used. As a result, the neural network can predict whether the marker image of a cell is the actual marker expression. It can be trained to provide a likelihood of whether

[0147] In some embodiments, these techniques are used to determine marker expression signatures. This can include running multiple trained neural networks to For example, different neural networks can be trained and used for different markers. In some embodiments, different neural networks generate different cellular localization information. For example, a nuclear, membrane, or cytoplasmic marker may be used. For example, a first neural network can be trained to detect expression in nuclear images, and a second The neural network can be trained to detect the expression of the cell membrane image, and the third model The model can be trained to detect expression in cytoplasmic images and / or similarly trained models. It can be done.

[0148] In some embodiments, the marker expression signature for a tissue sample comprises at least We also calculated the probability or probability that each marker in the immunofluorescence image indicates whether the cell expressed the marker. In some embodiments, the likelihood values ​​are 0 (for example, Each likelihood can range from 0 (e.g., no expression) to 1 (e.g., expression). In addition to being based on marker intensity, other information that can be determined based on the cell mask (e.g. , cell morphology, pixel intensity over the cell area, etc.) can also be used to determine the cell size.

[0149] In step 618, the computing device converts the marker expression signature into a cell type. In some embodiments, the cell typing data is compared to the cell typing data for each cell. For each cell type, a set of markers (e.g., markers in at least one multiplexed immunofluorescence image) It may contain a set of known marker signature entries for Each known marker signature determines whether the marker is expressed on a cell. Binary data (e.g., 0 or 1) indicating whether the (meaning that expression will be found in the cell type to which it is attached) In some embodiments, certain markers are ambiguous and / or unrelated. non-informative and therefore not necessarily expressive (e.g. (The marker may or may not be expressed on the cell), and Data can be represented as such (e.g., by including both 0s and 1s). As an example of cell typing data, the cell typing table is available from Sino Biological Cluster of Differentiation available from the company (e.g., https: / / www.sinobiological. com / areas / immunology / cluster-of-differentiation) and / or from Bio-Rad. human immune cell markers (e.g., https: / / www.bio-rad-antibodies.com / human-immune provided in the literature and / or databases, such as (e.g., ne-cell-markers-selection-tool.html) These can be generated based on known marker expression of the cells as identified.

[0150] In some embodiments, the marker expression signature is compared to cell typing data. The marker expression signatures are then compared to known sequences of cells in the cell typing data. Comparison data can be generated that can be compared to the expression signatures of the target gene. The data is a set of probabilities for each marker expression signature and each probability for each cell in the cell typing data. Showing a comparison between the associated marker values ​​of a known marker signature The comparison data can be used to measure the distance between expression signatures, the percentage of overlap of expression signatures, and The similarity scores between the expression signatures and / or the marker expression signatures are used to Any cell signatures that can be used to compare with known cell signatures in the cell typing data. It can include comparison metrics, such as other applicable comparisons. For example, The distance between the current signature and known marker expression signatures from cell typing data Distance can be cosine distance, Euclidean distance, Manhattan distance, and / or similar In step 620, the computing device Based on this comparison, the device determines a predicted cell type. See, e.g., Figure 6B. and the first set of data 622A can be processed to predict regulatory T cell types, the data The second set 622B can be processed to predict macrophage CD68+ cell type.

[0151] In some embodiments, the computing device The calculated comparison results for each cell type are analyzed to select the top candidate cell type. For example, when calculating distance, the computing device may It is possible to select from a variety of cell types by selecting a cell type having To try to have only one top candidate, the cell typing data is It may be configured to include a unique entry for each cell type (e.g., The comparisons made result in the same value for multiple cell types within the cell typing data. (So ​​that it doesn't happen).

[0152] FIG. 6C shows cell typing data 63 according to some embodiments of the techniques described herein. 8 to generate a marker expression signature 636 that determines the predicted cell type 640. 6 illustrates an example of using a neural network 634 for In this case, a marker expression signature can be a list, array, table, or expression of the probability for each marker. The cell typing data may be in any suitable data structure, such as a database. Similarly, a list, array, table, or other data structure of binary values ​​for each marker can be FIG. 6D illustrates a predicted variance in a ... Cell typing data (in this example, cell typing data) is used to determine the cell types involved. The expression signatures are shown as 654 and contain known expression signatures for each cell type. The set of probabilities to be calculated (shown in this example as probability table 652) is generated. 6 shows an example of using a neural network 650. The neural network 650 Images 656A (DAPI marker), 656B (FOXP3 marker), 656C (CD19 marker), 656D (CD11c marker) 656E (CD3e marker), and segmentation mask 656F for cells. 6. Process input 656, including a separate neural network, as described above. may be used to process the applicable marker images. For this example, The neural network was run on (a) images 656B (FOXP3 marker), 656C (CD19 marker), and 65 One of 6D (CD11c marker) and 656E (CD3e marker) was used for the CD4 and CD50 markers not shown. (b) Image 656A (DAPI marker) and (c) cell segmentation data along with the HLA-DR image. The first trained image may be processed in conjunction with the image processing mask 656F. A neural network was used to process the nuclear FOXP3 marker images, and the second The trained neural network is then used to identify membrane marker images (e.g., CD3e image 656E and CD1 1c image 656D), etc., so that separate Neural networks can be used to process images based on different locations of markers. Therefore, although only one neural network 650 is shown in FIG. 6D, the associated Several different pre-trained neural networks are used to process the attached marker images. A workpiece may be used.

[0153] In this example of FIG. 6D, neural network 650 has FOXP3 in probability table 652. Probability 0.96 for marker, probability 0.54 for CD3e marker, probability 0.66 for CD4, CD1 Generates probability 0.003 for 9, probability 0.002 for HLA-DR, and probability 0.0005 for CD11c (The closer the probability is to 0, the lower the probability of expression; the closer it is to 1, the higher the probability of expression for the marker.) rate will be higher).

[0154] The computing device compares the probabilities in table 652 with the markers for each cell type. Calculate the cosine distance between the marker and the value of the marker. In this example, a "+" indicates that the marker is "-" means that the marker is not expressed by the cells. As a result of the comparison, the cosine distance value was 0.028 for the T-reg cell type and 0.048 for the CD4 T cell type. 0.339 for the B cell type, 0.99 for the myeloid cell type, and 0.99 for the myeloid cell type. In the example, the smaller the distance, the more likely the associated cell type is to be the cell under analysis. As a result, the computing device Select T-reg (lowest distance value 0.028) as the cell type to be analyzed.

[0155] FIG. 6E shows a method for determining predicted cell types according to some embodiments of the technology described herein. Cell typing data (shown in this example as Cell Typing Table 664) is used to determine the to generate a set of probabilities (shown in this example as probability table 662) that are compared against 6 shows an example of using a neural network 660 for The image 660 shows MxIF images 666A (DAPI marker), 666B (CD68 marker), 666C (CD19 marker), and 6 66D (CD11c marker) and 666E (CD3e marker), as well as segmentation of cells 6. Process input 666, including a mask 666F. The neural network then maps each of the images 666B, 666C, 666D, and 666E (as well as the DAPI images 666A and and segmentation mask 666F). , the neural network 660 calculates the probability for the CD68 marker in the probability table 652. 0.63, probability 0.0006 for CD3e marker, probability 0.01 for CD4, probability 0.001 for CD19 , yielding a probability of 0.0004 for HLA-DR and a probability of 0.69 for CD11c (again, The closer the probability is to 0, the lower the probability of expression; the closer it is to 1, the higher the probability of expression for the marker. (It becomes more difficult.)

[0156] The computing device compares the probabilities in table 662 with the markers for each cell type. In this example, a "+" indicates that the marker is located in a specific cell. "-" means that the marker is expressed for a specific cell type. "+-" means that the marker may or may not be expressed for the cell. In this example, the comparison results in a cosine distance of 0.001 for macrophage CD68+ cell types, 0.992 for CD4 T cell types, and 0.992 for B cell types 0.999 for the myeloid cell type and 0.478 for the myeloid cell type. In this example, again, the distance The smaller the value, the more likely it is that the associated cell type is the cell type of the analyzed cell. As a result, the computing device is more likely to Select macrophages CD68+ (lowest distance value 0.001) as the target.

[0157] In some embodiments, these techniques identify clusters or regions of cells based on cell characteristics. For example, cancer (e.g., breast cancer, renal It is desirable to search for some cellular structures in a tissue sample, such as those indicative of cancer. These techniques are useful for identifying cell communities in tissue samples and for Identifying information about those cells, such as type, distance, etc. (e.g., sparsely-occurring cells) clusters, adjacent cell clusters, etc.) The community of cells can include different types of cells. In some embodiments, In the method, the cell clusters represent at least a portion of the tissue structure of the tissue sample. For example, mantle tissue, stromal tissue, tumor, follicle, blood vessel, and / or any other tissue structure. It can include.

[0158] FIG. 7B shows a first set of cell features ( To identify communities by acquiring and processing local cellular features 7B is a flowchart illustrating an exemplary computerized process 750. The computerized process 750 uses a graph neural network 772 to analyze the obtained station. The computer processes the localized cellular features to identify one or more communities 778 of cells. The data typing process 750 may be performed, for example, by the cell typing component 340 of FIG. The computerized process 750 may be implemented in conjunction with other processes and sub-processes described herein. For example, the computerized process 700 may be performed as part of a process for As part of step 508 of FIG. 5A, the cells are grouped into groups of cells, and / or tissue samples are A step is performed to determine at least one characteristic of the material (e.g., the community of cells). In step 752, the computing device generates local cell features, which are generated in step 752A by using cell data 774 (e.g., acquiring cell location data (e.g., cell placement, cell mask, and / or other cell localization data); and triangulating the cell data 774 to generate a graph 776. The device uses triangulation techniques (e.g., Delaunay triangulation) to For example, by constructing a configuration of cells 774 (based on the center of gravity of the cells) and creating a graph representation 776 of the cellular structure of the tissue. In some embodiments, the graph 776 may be configured to obtain the detected It contains a number of nodes equal to the number of cells. In some embodiments, the graph generation aspect is For example, in some embodiments, the edge of each edge in the graph 776 The length of the edge is the upper limit (based on pixel distance, for example, 200 pixels, 300 pixels, etc.). The number of edges in the graph 776 may be limited to a threshold value below which the number of edges may be determined. These techniques involve pruning edges that are smaller than a given length threshold. may include:

[0159] The computing device then generates a graphical representation of the organization 776 in step 752B. The local cell features are calculated based on the cell data 774 and / or The graph 776 may contain information about the cell that can be determined based on the graph 776. For example, The local cell features are determined for each cell based on the cell type, the edge of the graph 776, Cell neighbors, neighbor cell types, neighbor distance data (e.g., median distance to neighbors, Mask-related data (e.g., marker masks under each cell (e.g., CD31 markers for blood vessels) the percentage of the area filled with positive pixels relative to the area of ​​interest (e.g., a mask), and / or can contain similar. Thus, each node can contain local data point associations. It is possible to have a set (e.g., represented as a vector) of In embodiments of the present invention, node data may be encoded using multiple variables. For example, if there are seven discovered cell types in a tissue sample: , "Cell Type 6" may be encoded as [0, 0, 0, 0, 0, 1, 0]. In this case, the node data includes the median length of all node edges for a cell. In some embodiments, the node data can be calculated based on a given mask for the cell. The percentage of positive pixels in each of the multiple masks (present In some embodiments, the data The data is the percentage of cells that lie within one or more masks of selected markers. The percentage of the area of ​​the cell mask that is filled with positive cells may be included. Such mask-based data can be used by a computing device to Exploiting information about cells and / or structures that may otherwise be difficult to segment As a result, in some embodiments, The total number of data points is L, which is determined by (1) the number of cell types and (2) the number of masks to consider (if any). ), and (3) the sum of the values ​​for the median distance of the edges of a given node.

[0160] The graph 776 may be encoded for input to the graph neural network 772. In some embodiments, the node information may be stored in a matrix having dimensions n×L. where n is the number of nodes and L is the number of node features. A graph can be represented as a sparse adjacency matrix (e.g., with dimensions n×n nodes), an adjacency list of edges, and / or the like.

[0161] In step 754, the computing device generates a graph (e.g., an organizational graph) 776) into a graph neural network, which Graph neural networks embed cell features in a high-dimensional space. The graph structure containing the nodes and functions is used to process the input graph. The network may have different architectures. In this paper, the graph neural network 772 is an unsupervised convolutional graph neural network. For example, a graph neural network can be any suitable type of graph. Deep Graph I uses rough convolutional network layers to perform embedding within a graph. The Deep Graph Infomax architecture can be implemented using See, e.g., https: / / openreview.net / for Petar Velickovic et al., "Deep Graph Infomax," available at um?id=rklz9iAcKQ. ICLR 2019 Conference Blind Submission (September 27, 2018), and / or arXiv:1809.1 0341. Examples of different types of convolutional layers that can be used in graph neural networks and https: / / arxiv.org / abs / 16, both of which are incorporated herein by reference in their entirety. Thomas Kipf and Max Welling, "Semi-Supervised Class," available from sification with Graph Convolutional Networks,” arXiv1609.02907 (February 2017) GCN, such as, or available from https: / / arxiv.org / abs / 1706.02216, William H amilton et al., “Inductive Representation Learning on Large Graphs”, arXiv1706.0221 6 (September 2018). In addition to graph convolutional layers, graph neural networks are also used. A neural network can have a discriminative layer that is used in training. The implementation of the work is at least 500,000 parameters, 1 million parameters, 2 million parameters parameters, 5 million parameters, or even more. In some embodiments, such models may include meters. parameters (e.g., at least 10 million parameters, 1500 million parameters, depending on the implementation) Million parameters, 20 million parameters, 25 million parameters, 50 million parameters In some embodiments, the parameters may include at least one billion parameters (e.g., at least 100 million parameters, between 1 million and 100 million parameters) parameters), hundreds of millions of parameters, at least a billion parameters, and / or any Any suitable number or range of parameters may be included.

[0162] Graph neural networks use a variant of noise contrast estimation to estimate each node embedding. The neural network may be trained to reconstruct ,correct representation (e.g., based on real data) and incorrect,representation (e.g., based on noise representation). The system essentially learns an internal representation of nodes (e.g., cells) to distinguish between nodes (based on the Rough neural networks are based on the idea that graph neural networks can distinguish cell labels, neighboring cells, Information including cell distance, mask size (e.g., of some radius), and / or other inputs Since both local and global features of the tissue structure can be expressed from the information, Therefore, the nodes generated by the graph neural network Embeddings are based on global features, not just local node or cell neighborhood information. The resulting feature embeddings can also be It can contain the same predetermined number of features for each cell (e.g., a fixed-size vector (represented using rules, arrays, and / or other data structures).

[0163] In some embodiments, the output of the neural network is The activations of the graph convolution layers of the Since we obtain (e.g., GCN, SAGEConv, etc. as explained above), the output dimensionality is , 16 dimensions, 32 dimensions, etc. Each dimension of the output has a higher dimension. It can be an embedding of a space into a lower dimension, and in general an organizational structure for representation. Thus, for example, the embedding values ​​and the specific cell type In some embodiments, there may be at least some correlation between the class composition and the Clustering is performed using these embeddings, as further explained in ,Clusters can be described in terms of the predominant cell types within them.

[0164] In some embodiments, for each of the plurality of training tissue samples, a graph of the tissue sample is representation, the local cells calculated as described above for steps 752A and 752B features (e.g., for each node, cell type, mask relation data, and edge center) A dataset containing the metric (value distance) is generated and used to train a graph neural network 772. In some embodiments, loss=log(P i )+log(1-P' i ) is a loss function such as , may be maximized, and P i is the node (obtained through cell segmentation) is the probability that a cell (which is a node in the graph) is similar to all the graph nodes, and P' i is permuted is the probability that a given node is similar to all graph nodes. The code contains associated information (e.g., cell type assignment, cell subdivisions to neighboring elements). median distance, mask information, etc.), which can take the form of a feature vector. In an embodiment, the Deep Graph Infomax architecture calculates all the nodes in a given sample graph. We use a summary vector, which can be the average value of the feature vectors for all nodes. The classifier layer of the graph model consists of (a) a given node feature vector that is classified as belonging to the graph summary vector; and (b) permutations that are classified as not belonging to the graph summary vector (e.g., The embedding between the node feature vectors (shuffled in place) is used to distinguish can be trained to.

[0165] In step 756, the cell embedding (including the neighborhood data) is analyzed to generate one or more clusters. To determine the clusters, the various classes are clustered. For example, as described herein, these techniques may be used to Mind-based clustering algorithms (e.g., K-means), distribution-based clustering clustering algorithms (e.g., clustering using Gaussian mixture models), density-based Clustering algorithms (e.g., DBSCAN), hierarchical clustering algorithms , PCA, ICA, and / or any other suitable clustering algorithm. For each determined cluster, the percentage of cells in the cluster and The associated mask is the explanatory data for each cluster, as shown in 778. can be used to generate

[0166] FIG. 7C illustrates tissue contours shaded based on cell type, according to some embodiments. 7B, and cell clusters (e.g., as described in conjunction with FIG. 7B) for the same tissue sample. 7 shows an example of an image 782 in which the contours are shaded based on the contours (determined to be Image 782 shows that the cell clusters are slightly different in comparison to the shading based on cell type in image 780. The tissue structure of the spleen (e.g., dark zone, light zone, and / or mantle zone follicles) can be more clearly identified. This shows how it can be used to visualize

[0167] FIG. 18 illustrates processing an MxIF image in accordance with some embodiments of the techniques described herein. FIG. 18 is a diagram 1800 illustrating exemplary tissue properties that may be determined by one of the tissue properties or The plurality may be determined, for example, by the characteristic determination module 360 ​​of FIG. 3. The tissue characteristics identified are part of the various processes and sub-processes described herein. For example, one or more of the tissue characteristics may be determined as part of step 510 of FIG. As shown, the characteristics may be determined using a radius check 1804 and / or a triangular to include and / or determine cellular spatial co-occurrence 1802, which may be used for angular surveying 1806; The characteristics may be used to generate one or more masks 1808. The sex may be used to contain and / or classify the cell population 1810. In morphology, these techniques use graph neural networks to classify cell populations. This may include:

[0168] In some embodiments, cell grouping information is obtained by immunofluorescence analysis to indicate tissue aspects. The masks may be used to determine one or more masks that may be applied to the light image. One example is a tumor mask, which contains data representing tumor cells in a tissue sample. is an acinar mask containing data indicating the spacing between cells in the tissue sample, which is Ducts forming the gaps / spaces between the alveoli can be identified. For example, acinar masks It can show the secretory ducts of tumors and therefore provide information about the ductal geometry. (e.g., because different tumors may have vessels of different shapes / sizes). An example of such a mask is a stromal mask containing information indicative of supporting tissue and / or stromal cells in a tissue sample. is.

[0169] In some embodiments, the mask is used to distinguish cells in different regions of the tissue sample. For example, masks of acini, tumor, and stroma can be created to identify T cells, etc. can be used to understand where some cells are located in a tissue sample. For example, a mask can be used to distinguish T cells within stromal and / or non-stromal regions. do.

[0170] FIG. 19A shows a method for processing an immunofluorescence image 1900 according to some embodiments of the techniques described herein. 19. The stromal mask 1902 and the acinar mask 1904 are generated by processing the As shown, the stromal mask 1902 includes data representing the supporting tissue of the tissue sample, including the acini. The mask 1904 includes data indicative of the spacing of cells in the tissue sample. Therefore, these techniques are useful for detecting cytoplasmic markers (e.g., most, if not all, cells). A stromal mask was generated using a marker expressed in cytoplasm (S6) to identify cytoplasmic regions in tissue samples. and the identified regions using epithelial markers (e.g., PCK26 epithelial marker). In some embodiments, the cytoplasmic marker fraction may be removed. The image may be smoothed to perform noise removal, as described herein, and / or In some embodiments, the epithelial marker image may be thresholded to identify the filled holes. (e.g., by which the interior portions of adjacent shapes are filled in and thus and removed from the final stromal mask). In some embodiments, these techniques Generate an acinar mask using an epithelial marker (e.g., a smoothed PCK26 mask) This can include removing the stromal zone to create the final acinar mask, which can be reversed. Epithelial markers can be generated using tumor masks (using the smoothed PCK26 mask). It can also be used to generate other masks, such as

[0171] In some embodiments, the object is based on cell community information (e.g., in conjunction with FIG. 7B). For example, the object mask may be Each of the one or more cell communities identified in the cell community information is a mass The cell community information is used to create a cluster of related objects in the cluster. Therefore, the objects in the object mask are obtained by dividing the individual objects identified in the cell community information. It can correspond to cells and / or multicellular structures. Generating an object mask 1956 using features of the tissue sample, according to some embodiments of The object mask 1956 is a diagram showing the cell coordinates, as shown at 1950 and 1952. Cells of a tissue sample, such as cell labels (e.g., community information), and image shape data In particular, image 1950 in this example is generated based on the type of community. Showing shaded cell contours, this example image 1952 shows the cells belonging to the identified community. One or more object detection algorithms process the cell features to determine the area ( For example, using Voronoi tessellation), which can be used to count objects For example, object detection algorithms generate cell labels, count the cell labels, and then identify tissues. A mosaic of a sample (e.g., a randomly generated mosaic of a tissue sample) It can generate noise regions / cells. The resulting density heatmap 1954 is used for object detection. The density heat map 1954 can be generated based on an algorithm. Cells are shaded dark or light based on the number of cells detected. The imaging device creates an object mask 1956 of objects with a high density of cells having the selected label. As an illustrative example, lymph node follicles may be Community-based detection of dark, light, and mantle zones in node tissue samples. can.

[0172] FIG. 20 illustrates a comparison of acinar shape, area, and 20 is a diagram 2000 showing an example of measuring the characteristics of the perimeter and the distance between the object and the object. The diagram 2000 shows three images 2002, 2004, and 2006. 2006. Diagram 2000 shows the corresponding acinar masses for images 2002, 2004, and 2006, respectively. As shown in FIG. 20, each acinar mask 2008, 2010, and 2012 is 2010 and 2012 are pixel-wise measures of the area of ​​acinar structures in the acinar mask, A measure of the area of ​​acinar structures as a percentage of the total number of image pixels (e.g., the percentage of mask pixels that are masked), and a pixel-wise measure of the perimeter of the acinar structures 2020, 2022, and 2024, which include a corresponding set of parameters 2020, 2022, and 2024. The meter can provide information about acinar structure, for example, acinar area divided by acinar perimeter. Comparison with the length (e.g., as a ratio) can provide information about the overall structure of the acinar. The diagram 2000 shows the corresponding fibrosis images for images 2002, 2004, and 2006, respectively. Also shown are masks 2014, 2016 and 2018. The fibrotic masks represent connective tissue, which is the interstitial mask and therefore can be produced as described herein for the stromal mask. Each fibrosis mask 2014, 2016, and 2018 contains fibrosis filled with interstitial tissue. A measure of the area of ​​the mask, and a measure of the variance of the distribution of interstitial tissue (e.g., whether interstitial tissue is present in the mask) parameters 2026, 2028, including parameters 2026, 2028 (which may provide information about how widespread the , and 2030 corresponding sets.

[0173] FIG. 21 shows examples of spatial distribution characteristics according to some embodiments of the techniques described herein. FIG. 21 includes examples 2102 measuring fibrosis distribution and 2104 measuring T cell distribution. Fibrosis heatmap 2102A and t cell heatmap 2104A show the relationship between fibrosis and t cell density. For example, darker colors (e.g., dark red) may be shaded to indicate It can indicate areas of highest density, and brighter colors (e.g. bright red) indicate areas of next highest density. A different color (e.g., light blue) can indicate areas of high density, and then another color (e.g., light blue) can indicate areas of even lower density. a different color (e.g., dark blue) can indicate the least dense areas ( For example, you can use a different darker shade to improve contrast from denser areas. In some embodiments, the analysis is performed to provide concentration measurements in the interstitial tissue. Fabrics may be used, for example, to distribute the distribution of interstitial tissue (e.g., as described in conjunction with FIG. 20). (such as) can be determined based on the standard deviation of the interstitial distribution. ,For example, the stroma distribution divides the stroma mask image into equal-sized squares (e.g. , using tessellation), and the representative value is compared to other squares For example, the percentage of pixels that represent stromal tissue can be determined for each square. The page can be used as a distribution from which the standard deviation can be calculated. These techniques can measure how homogeneously the stroma is distributed within the mask. For example, the apical fibrosis distribution 2106 indicates that the mask does not contain large areas without stroma. 2108 has a uniform distribution as shown, but the bottom fibrosis distribution 2108 is not uniform and therefore the mask Indicates that it contains areas with no quality.

[0174] FIG. 22 shows examples of spatial organization properties according to some embodiments of the techniques described herein. Figure 22 shows the positive areas (inside malignant cells) within the tumor mask and the negative areas within the tumor mask. Example 2202 measuring the distribution of endothelial cells in the malignant compartment (non-malignant compartment) and malignant compartment. Examples 2204 include measuring the distribution of T cells in malignant and non-malignant compartments.

[0175] FIG. 23 shows a comparison of the results for two different patients according to some embodiments of the techniques described herein. 1 shows an example of cell contact information in an immunofluorescence image obtained by the method described herein. The contact information may be one of the determined characteristics of the tissue sample, for example, cell contact information. may be determined as part of step 510 of FIG. 5A and may be determined for a tissue sample. The number of groups may be determined based on, for example, each group being associated with a different cell type. Fluorescence image 2302 is of a tissue sample from patient 1, and immunofluorescence image 2304 is of a tissue sample from patient 2. Images 2302 and 2304 are both of CD8, CD68, CD31, and PCK26. The characteristic information 2306 includes the T markers that are in contact with the blood vessel for each of the patients 1 and 2. Information on the percentage of cells and the percentage of macrophages in contact with blood vessels. Information on the number of T cells in contact with macrophages and the percentage of T cells in contact with macrophages and information 2312 relating to the

[0176] FIG. 24 illustrates a comparison of two different patients from FIG. 23 in accordance with some embodiments of the technology described herein. 1 illustrates an example of information about cell neighbors relative to a cell, as described herein. The neighbor information may be one of the determined properties of the tissue sample, such as which types of cells Cell neighbors can indicate whether a cell type is adjacent to another cell type. Both cell types in contact, as well as those not in contact but between them, Figure 24 shows the immunofluorescence image 2302 from Figure 23 and 2034. FIG. 24 shows the number of T cell neighbors for each of the immunofluorescence images 2302 and 2034. It also includes characteristic information 2306 indicating:

[0177] FIG. 25 illustrates MxIF images 2502, 2504 and 2506 in accordance with some embodiments of the techniques described herein. Two examples of the stromal segmentation masks 2506, 2508 and the corresponding stromal segmentation masks are shown.

[0178] 26-35 show cell populations and associated properties determined using the techniques described herein. 26 illustrates various aspects of information, according to some embodiments of the techniques described herein. Exemplary MxIF images 2602 and 2604, segmentation masks 2606 and 2608, and The corresponding cell populations 2610 and 2612 are shown.

[0179] FIG. 27 illustrates a method for generating cell populations according to some embodiments of the techniques described herein. An example of a complete MxIF slide process is shown in Figure 27. The cell population information view 2702 and the A blow-up view 2704 of a portion 2706 of the sera is shown. The cell populations are CD3 T cells, CD4 T cells, CD5 T cells, and Includes T cells, macrophages, blood vessels, and malignant zone cells.

[0180] FIG. 28 illustrates processing of an immunofluorescence image 2804 according to some embodiments of the techniques described herein. 28 is a diagram of a restored cell arrangement 2802 generated by , representing cells in a tissue sample (e.g., where each cell type corresponds to a different group of cells) . Restored cell arrangement configuration 2802 includes acinar cells, CD4 FOXP3 T cells, CD4 T cells, CD8 T cells, endothelial cells, K167 and PCK26 cells, macrophages, and unsorted cells are shown.

[0181] FIG. 29 illustrates a method for producing 4′,6-diamidino-2-phenylindole-1-one according to some embodiments of the technology described herein. DAPI stained immunofluorescence image 2902 and two comparisons of different cell groups against the DAPI image. Images 2904 and 2096 are shown. Image 2904 shows markers for CD31, CD21, CD3, and CD68. Image 2906 also includes markers for CD31, CD21, CD3, and CD68, and CD20 Further includes:

[0182] FIG. 30 shows a different CCRCC tissue sample 300 according to some embodiments of the technology described herein. Cell populations for 2, 3004, and 3006 are shown.

[0183] FIG. 31 illustrates the removal of kidney tissue from FIG. 14 according to some embodiments of the techniques described herein. 14 shows exemplary cell populations for the exemplary MxIF images 1404, 1406, 1410, and 1412 obtained. The cell group 3102 is determined based on the MxIF images 1404 and 1406, and the cell group 3104 is determined based on the MxIF images 1404 and 1406. The cell populations were determined based on 10 and 1412. The cell populations were T cells, macrophages, blood vessels, and malignant Includes the genital tract.

[0184] FIG. 32 shows the results of different clear cell renal cell carcinoma (CCR) tumors in accordance with some embodiments of the technology described herein. 32 shows a set of cell population images 3200 for a CRCC tissue sample. The images 3200 include T cells, NK cells, Figure 33 shows the cell populations for B cells, macrophages, blood vessels, and malignant zones. A series of cell populations for different CCRCC tissue samples according to some embodiments of the techniques described in Similar to FIG. 32, image 3300 shows T cells, NK cells, B cells, macrophages, and , blood vessels, and malignant zones. For example, it may be possible to distinguish between both T cells and malignant zones in a tissue sample. do.

[0185] FIG. 34 shows two different chromatin-enriched chromatin (chromatin-enriched chromatin) analyses of CCRCC tissue samples according to some embodiments of the technology described herein. FIG. 34 shows the results of analyzing the MxIF image. 15 regions 3412 in image 3404 (e.g., determined as part of step 510 of FIG. 5A) CCRCC tissue samples 3402 and 3404 and corresponding characteristic information 3406 and The characteristic information includes, for each region 3410 and 3412, the B cells, CD4 T cells, and cells, CD8 T cells, endothelial cells, epithelial cells, macrophages, macrophages 206, macrophage The percentages of phage 68, NK cells, tumor cells, tumor K167+, and unsorted cells are shown. The characteristic information classifies the 15 regions 3410 into either fibrous / normal regions or tumor regions. The region 3412 is classified as a fibrous / tumor region, a normal region, or a tumor region.

[0186] FIG. 35 shows the MxI of a CCRCC tissue sample 3502 according to some embodiments of the technology described herein. The analysis information 3504 shows the results of the analysis of the F image. B cells, CD4 T cells, CD8 T cells, endothelial cells, epithelial cells, macrophages, and myeloid cells in the area. Macrophage 206, macrophage 68, NK cells, tumor cells, tumor K167+, and unclassified cells Figure 35 also shows a blow-up view 3508 of the exemplary region 9. vinegar.

[0187] 36A-41 generally illustrate tissue cell properties determined using the techniques described herein. 36A-36B show examples of cell FIG. 36A illustrates the amounts and ratios of different exemplary patients (e.g., patient RP83). 93, etc.), unclassified cells, endothelial cells, macrophages, CD8 T cells, CD4 T cells, tumor Figure 36B includes a feature 3602 showing the percentage of tumor K167+ cells and tumor cells. Characteristics of CD8 T cells 3604, characteristics of endothelial cells 3606, characteristics of macrophage cells 3608, characteristics about tumor cells 3610, and characteristics about tumor cells 3612. nothing.

[0188] FIG. 37 illustrates cell distribution characteristics according to some embodiments of the techniques described herein. Figure 37 shows a tissue image 3702 and corresponding stromal mask 3704 for a first patient. In both cases, T cells are shown in red. Figure 37 shows tissue image 3706 and and the corresponding stromal mask 3708, both with macrophage cells shown in red. Feature 3710 indicates that for images 3702 and 3704, just under 90% of the T cells are in the interstitium, with the remaining percentage Feature 3712 shows that the macro is located within the tumor for images 3706 and 3708. The majority of phage cells are located within the stroma, with a very small percentage in the tumor. It shows.

[0189] 38A-38B show a histological characterization of a sample in accordance with some embodiments of the technology described herein. The percentage heat map and distribution density are shown. The heat map is based on step 50 in Figure 5A. 8 and 510. Each block in the heatmap can be generated based on a cell population. represents the percentage of blocks corresponding to the positive interstitial mask. The corresponding stromal mask 3802, the corresponding stromal heat map 3804, and the stromal distribution density graph 3806 are shown. 38A shows a stromal mask 3808 and corresponding stromal heatmap 3810 for a second patient. , and a graph 3812 of the interstitial distribution density. FIG. 38B shows the interstitial mask 3818 and 38B includes corresponding graphs of x-sum intensity 3814 and y-sum intensity 3816. Also included is a quality mask 3824 and corresponding graphs of x-sum intensity 3820 and y-sum intensity 3822.

[0190] FIG. 39 illustrates a cell-neighborhood element and a cell, according to some embodiments of the technology described herein. FIG. 5B illustrates contact characteristics. Cell contact characteristics may be determined, for example, in step 510 of FIG. 5A. Figure 39 shows the results of the immunization of CD8 T cells, endothelial cells, macrophages, tumor cells, and unsorted cells ( Here, neighboring cells are similarly CD8 T cells, endothelial cells, macrophages, tumor cells, and For the first patient, the percentage of adjacent cells (which may be unclassified cells) is shown. The first table 3902 shows the same neighbor cell percentages for various cell types. FIG. 39 includes an associated pie chart 3904 showing the distribution of CD8 T cells, endothelial cells, macrophages, and A second graph showing the percentage of adjacent cells relative to the number of site, tumor, and unclassified cells. A second table 3906 for the patient, as well as the same neighboring cell patterns for various cell types, 3908 showing the percentages.

[0191] FIG. 40 illustrates a tS analysis of a profile of marker expression according to some embodiments of the technology described herein. This figure shows examples of NE plots 4002 to 4016. In particular, tSNE plot 4002 shows CD45 marker expression. tSNE plot 4004 is for CAIX marker expression, and tSNE plot 4005 is for CAIX marker expression. Lot 4006 is for CD163 marker expression and tSNE plot 4008 is for CD206 marker expression, and tSNE plot 4010 is for CD31 marker expression, and tS NE plots are for CD3E marker expression and tSNE plots 4014 are for PBRM1 marker expression and tSNE plot 4016 is for PCK26 marker expression. SNE plots were generated using marker expression and showed that cells with high marker expression levels Different colors and regions are used to indicate the correspondence between cells and their assigned cell types. By using shading and / or by using a marker in the cell, two types of information can be obtained, as shown in Figure 40. To display the current intensity (each point is a cell), and the cell type in the graph 4020, The characteristics may be, for example, those determined in step 510 of FIG. 5A, as described herein. It is possible.

[0192] FIG. 41 shows a graph of tS of a characteristic of marker expression according to some embodiments of the technology described herein. Another diagram showing examples of NE plots 4102-4148. In particular, tSNE plot 4102 shows the CD138 marker. - expression, and tSNE plot 4104 is for IgD marker expression, and t SNE plot 4106 is for CD31 marker expression, and tSNE plot 4108 is for SPARC marker expression. tSNE plots are for KAr expression and for BCL2 marker expression. tSNE plot 4112 is for CD10 marker expression, and tSNE plot 4114 is for CD20 marker expression. tSNE plot 4116 is for CD3 marker expression. tSNE plot 4118 is for collagen marker expression, and tSNE plot 4120 is for CD163 marker expression, and tSNE plot 4122 is for HLA-DR marker expression. The tSNE plot is for CD11c marker expression. Plot 4126 is for CD21 marker expression, and tSNE plot 4128 is for CD68 marker expression. tSNE plot 4130 is for Ki-67 marker expression and tSNE Plot 4132 is for CD25 marker expression and tSNE plot 4134 is for FOXP3 marker. - expression, and tSNE plot 4136 is for CD8 marker expression, and t SNE plot 4138 is for CD44 marker expression, and tSNE plot 4140 is for CD35 marker expression. tSNE plots are for CAR expression and for CD4 marker expression. tSNE plot 4144 is for CD45 marker expression, and tSNE plot 4146 is for BCL6 marker expression. tSNE plots are for marker expression and for IRF4 marker expression. do.

[0193] Figures 42-47 generally show 4',6-diamidino-2-phenylindole (DAPI) heterogeneous markers. It relates to using the techniques described herein in the context of Kerr staining. 4',6-diamidino-2-phenylindole (DAPI), according to some embodiments of the described technology To determine cell segmentation data 4204 for a heterogeneously stained immunofluorescence image 4206 FIG. 4 is a pictorial illustration of the use of a convolutional neural network 4202.

[0194] FIG. 43 illustrates a DAPI stained immunosorbent assay of a tissue sample according to some embodiments of the technology described herein. A first cell mask generated based on a combination of the fluorescence image 4304 and the CD3 cell marker image 4306. 4302. The cell mask may be used, for example, in conjunction with step 504 of FIG. 5A. The cell matrix in Figure 43 can be obtained as part of the information showing the arrangement of cells as explained in As shown in the example mask, cell mask 4302 is used to identify the expression of markers for cells in a tissue sample. It can be applied to different immunofluorescence images to convey the results.

[0195] FIG. 44 is a DAPI stained immunofluorescence image of FIG. 43 according to some embodiments of the techniques described herein. A second cell mask 440 generated based on a combination of image 4304 and a CD21 cell marker image 4404. FIG. 45 is a pictorial illustration of some embodiments of the techniques described herein. Based on the combination of the DAPI stained immunofluorescence image 4304 and the CD11c cell marker image 4504 in Figure 43, 45 is a pictorial illustration of a third cell mask 4502 generated.

[0196] FIG. 46 shows an MxIF image of the tissue sample of FIG. 43 according to some embodiments of the technology described herein. A vascular mask 4602 generated based on image 4604 (image 4606 is an enlarged view of a portion of image 4604) is illustrated. The vascular mask 4602 shows the placement of blood vessels within the tissue. To emphasize that the calculated mask 4602 corresponds to blood vessels in the tissue, a synthetic pseudocolor - In image 4604, the CD31 marker is first placed on top of the DAPI nuclear marker in a second shade (e.g., blue). As described herein, the vascular mask may be For example, it can be created as part of step 510 of FIG. 5A.

[0197] FIG. 47 illustrates the mask 4302 of FIGS. 43-46 in accordance with some embodiments of the techniques described herein. 4 is a pictorial illustration of a cell population 4702 generated using 4402, 4502, and 4602. Cell population 4702 includes T cells (green), follicular dendritic cells (dark orange), CD11c+ cells (blue), and Includes blood vessels (red).

[0198] FIG. 48 illustrates a prostate tissue sample 4804 and a Figure 48 shows a set of cells 4802 for the malignant site 4806. Surrounding the affected dense tumor area, with the majority of the tumor still segmented FIG. 49 shows a graph of a 3D image of a human ovarian tumor to highlight immune infiltration, according to some embodiments of the technology described herein. FIG. 50 shows an enlarged view of a portion 4902 of a cell population 4802 of the prostate tissue sample 4804 of FIG. 48. According to some embodiments of the technology described herein, the results are shown in Table 1. The set of cell populations 5002, 5004, 5006, and 5008 for the visualized prostate tissue sample is shown. Each cell in Figures 48 to 50 is colored according to its cell type, and the blood vessel mask is The binary black and white images in Figures 48 and 49 are colored red. Figure 10 shows a full-slide segmentation mask of fibrosis obtained using the CT scan.

[0199] A computer that may be used in connection with any of the disclosed embodiments provided herein An exemplary implementation of a computer system 5100 is shown in FIG. The computer system 5100 may include a computing device 112 and / or a computing The computer system 5100 may be used to implement the computer device 116 shown in FIGS. , one or more of the MxIF image processing pipelines shown in FIGS. 5A, 6A, and / or 7A-7B. The computer system 5100 may be used to implement multiple aspects. a plurality of computer hardware processors 5102 and a non-transitory computer-readable storage medium; (e.g., memory 5104 and one or more non-volatile storage devices 5106) The processor 5102 may comprise one or more articles of manufacture. writing and reading data to and from memory 5104 and non-volatile storage device 5106; The processor may control the reading of data to perform any of the functions described herein. The processor 5102 includes a non-uniform memory that stores processor-executable instructions for execution by the processor 5102. One or more non-transitory computers that can act as temporary computer-readable storage media. One or more processor executable programs stored in a readable storage medium (e.g., memory 5104) A possible instruction can be executed.

[0200] The computer system 5100 includes a processor 5102, a memory 5104, and non-volatile storage. The device 5106 may be any type of computing device. For example, the computer system 5100 may be a server, a desktop computer, a laptop, In some embodiments, the computer may be a tablet or a smartphone. The data system 5100 may include a cluster of computing devices, virtual computing devices, and / or multiple computers, such as cloud computing devices The present invention may be implemented using a computing device.

[0201] 52-53 illustrate how the techniques described herein can be used to diagnose cell placement and cell division, according to some embodiments. We present two comparisons to show how our method outperforms conventional techniques in both cell shape reconstruction and cell shape reconstruction. The ground truth data for both Figures 52 and 53 are from different non-small cell lung cancers. Tissue samples and markers included DAPI, NaKATPase, PCK26, CD3, and CD68. To guide manual segmentation by pathologists on the end-truth data To provide an example of the prior art, ground truth data is available at https: / / www. CellProfiler, available from the Broad Institute, at http: / / cellprofiler.org / releases The cells were segmented using CellPro 4.1.3 (for ease of explanation, we refer to this paper as "CellPro"). (hereinafter referred to as "filer").

[0202] To demonstrate the techniques described herein, cell segmentation was performed using trained convolutional This is performed using neural networks, thereby generating cell segmentation data. (For ease of explanation, the examples in Figures 52 and 53 are referred to as "Mask R-CNN" In particular, the pre-trained model used in this example uses a ResNet-50 encoder head. We implemented a Region-Based CNN architecture using the CNN. This model has approximately 43 million parameters. Had a meter.

[0203] The input to Mask R-CNN is three channels, which includes nuclear markers for the first channel. - (including DAPI), almost all cells of a particular tissue type for a second marker The third marker was found to be a membrane marker (including CD45 or NaKATPase) present on the surface of the cells, and the third marker was found to be a membrane marker (including CD45 or NaKATPase) present on the surface of the cells. One additional membrane marker with colorimetric activity (CD3, CD20, CD19, CD163, CD11c, CD11b, CD56, CD All channels are normalized to fall within the range 0 to 1 and For the match, the maximum expression value at each pixel among all selected markers is used. was done.

[0204] In this implementation, the Mask R-CNN segmentation process is performed by the forward network. This includes a post-processing step to generate the final cell segmentation data. The input image is divided into intersecting squares of size 256x256 pixels. The output is an array of mask proposals for each distinct cell. In this implementation, the output is The output is not a value image, but rather represents the probability that a given pixel is part of the cell's mask. The output image was a set of images with values ​​ranging from 0 to 1. The output image was thresholded using a value of 0.5. The final binary mask is obtained by multiplying the masks by 1. Some of the masks intersect with each other. If so, a non-maximum suppression algorithm was implemented (e.g., this is Available as https: / / arxiv.org / pdf / 1704.04503.pdf, which is incorporated herein by reference. Navaneeth Bodla, "Improving Object Detection With One Line of Code" e,” arXiv:1704.04503v2 (August 2017). For example, the prediction window is Such intersections may contain redundant data, such as due to cells between the edges of windows. This was avoided by processing only cells from the center, top corners, and right corners of the image (e.g. For example, if the current window is the last window from the right or bottom, (The bottom and right parts of the image are processed only if the The segments are represented in the final output mask by integer values ​​that represent individual cell instances. and converted into cell outlines.

[0205] The training dataset was pruned for training with a pruning size of 300 × 300 pixels. It contained 10 images. Dropout, rotation, elastic transformation, Gaussian blur, contrast Adjusting the reflections, adding Gaussian noise, reducing the membrane channel intensity (sometimes even to zero), All included images were enhanced and further cropped to a fixed size of 256 × 256 pixels. For the first channel, the nuclear marker is DAPI, and for the second channel, CD45 or is NaKATPase, and for the third channel, various additional markers (CD16, CD163, CD45 ( If the marker in the second channel is NaKATPase), CD206, CD8, CD11c, CD20, CD56, All labels were obtained through human annotation. Mask R-CNN model uses backpropagation and the Adam optimizer to generate randomized images of annotated images. Trained on pruning.

[0206] In FIG. 52, image 5202A shows a first ground truth cell image for a first tissue sample. In these examples, the segmentation does not necessarily involve cell boundaries. Since the contour is predicted without including the boundary, the cell segments are close to each other (or or overlapping), which allows you to see different segmented cells in image 5202A. Therefore, image 5202B shows an example of various segmented cells. First ground toe shaded using different shading to help show Image 5204A shows different cell segments from the loose data. 1 shows the determined cell segmentation information for the ground truth data of Image 5204B shows the different segmented cells as determined by the cell profiler. The various segmented cells are shown shaded to help illustrate them. Image 5206A shows the cell alignment of the first ground truth data determined using Mask R-CNN. Image 5206B shows the position information determined for the first ground truth data. The different segmented cells are shown shaded.

[0207] Table 1 below shows the comparison of CellProfiler with ground truth data for Figure 52. Comparison metrics for Mask R-CNN segmentation are shown. [Table 1]

[0208] Panoptic Quality (PQ) is used as a performance metric, e.g., http Available at: s: / / arxiv.org / abs / 1801.00868.,Alexander Kirillov, “Panoptic S "Egmentation," arXiv:1801.00868 (April 2019).

[0209] The Jaccard coefficient indicates the accuracy of cell detection and cell shape reproduction. Cell segmentation using end-truth data and the Hungarian algorithm The cell is determined by matching cells between the predicted values ​​(each cell is either 0 or 1). For each matched pair, the jacks between them The final result is the sum of the values ​​for the maximum number of cells (ground truth mask or is calculated as the number of cells in the prediction mask divided by the number of cells in the prediction mask.

[0210] F1, Precision, and Recall are used in machine learning to measure the accuracy of a given ground truth (GT). Used to check the quality of the prediction compared to the object (or segment in these examples) is the metric used, where Precision=TPS / GTS, Recall=TPS / PS, and F1=2*Precision*R TPS (true positive segments) is the number of segments from GT with Jaccard ≥ 0.5. represents the number of segments from the neural network prediction with the reference segment, GTS is the number of all segments from GT, and PS is the number of all segments from prediction. do.

[0211] In FIG. 53, image 5302A shows a second ground truth cell image for a second tissue sample. Image 5302B shows segmentation information relative to the second ground truth data. Image 5304A uses shading to show the different cell segmentations. Cell segmentation determined against second ground truth data using profiler Image 5304B shows the different segmentation information determined via CellProfiler. Image 5306A shows the cells that were detected using Mask R-CNN with a second ground truth filter. Image 5306B shows cell segmentation information determined for the loose data. The different segmentations determined on the second ground truth data using k R-CNN The irradiated cells are shown shaded.

[0212] Table 2 below shows the comparison of CellProfiler with ground truth data for Figure 53. Comparison metrics for Mask R-CNN segmentation are shown. [Table 2]

[0213] As shown by the images in Figures 52-53 and the comparative data in Tables 1 and 2, As mentioned above, Mask R-CNN is better at detecting cell contours than CellProfiler, but this is due to , because this model was trained for cell segmentation (while Cel lProfiler is generally used for cell segmentation based on nuclear thresholding, and for Cell membrane identification by enlarging the nucleus so that it is outlined with a selected number of pixels Mask R-CNN has fewer false positives of cells than CellProfiler (for example, CellProfiler Reproduced cell shapes closer to the original state (e.g., higher PQ and Jaccard coefficient metrics). Therefore, Mask R-CNN is a promising candidate for cell segmentation. This represents a significant improvement over CellProfiler and other similar previous approaches. Such improved cell segmentation can then be used to Subsequent aspects performed by the image processing pipeline used can be improved. , that further analysis can be performed on cell segmentation data as described herein. This is because the .NET Framework may use and / or rely on the .NET Framework.

[0214] The terms "program" and "software" are used in their general sense in this specification. and a computer or a processor configured to implement various aspects of the embodiments as described above. Any kind of processor that can be used to program other processors (physical or virtual) It is used to refer to computer code or any set of processor-executable instructions. Additionally, by one aspect, the methods of the disclosures provided herein when implemented One or more computer programs that implement the method may be used on a single computer or processor. Although not necessarily resident on a processor, various aspects of the disclosure provided herein may be implemented. may be distributed in modules among different computers or processors to implement .

[0215] Processor-executable instructions are instructions executed by one or more computers or other devices. The program may take many forms, such as a program module, executed by a A module is a set of routines that perform a particular task or implement a particular abstract data type. , programs, objects, components, data structures, etc. The functionality of the program modules may be combined or distributed.

[0216] The data structures may also be stored in one or more non-transitory computer-readable storage media in any suitable format. For ease of illustration, the data structure may be stored in a readable storage medium. These relationships can be shown to have related fields through their placement within the Similarly, a field may be defined as an arrangement in a non-transitory computer-readable medium that conveys the relationship between fields. However, this can be achieved by allocating storage for The mechanism is a pointer, tag, or other mechanism that establishes relationships between data elements. Used to establish relationships between information in fields of a data structure, such as through the use of It can be done.

[0217] Various inventive concepts may be embodied as one or more processes, examples of which include The activities performed as part of each process may be ordered in any suitable manner. Thus, embodiments in which activities are performed in a different order than illustrated may be possible. may be configured, which is shown in the exemplary embodiment as a sequential activity. Even if a task is performed, it may involve performing several activities simultaneously.

[0218] As used in this specification and claims, a reference to a list of one or more elements means The phrase "at least one" in a reference means any one of the elements in a list of elements. means at least one element selected from one or more, and not necessarily within a list of elements does not contain at least one of every element specifically listed in It should be understood that this definition does not exclude any combination of elements in the list. The element is, optionally, in the list of elements to which the phrase "at least one" refers Other than the elements specifically identified, whether related to the elements specifically identified or unrelated Therefore, for example, "A and B's at least one" (or equivalently, "at least one of A or B" or etc., but "at least one of A and / or B" is, in one embodiment, optional. There is at least one A and no B (and optionally, more than one B) In another embodiment, at least one B is optionally and A is not present (and optionally contains elements other than A), and In the form, at least one A, optionally including multiple, and at least one A, optionally including multiple, may be taken to mean that there is at least one B (and optionally other elements), etc. stomach.

[0219] As used in this specification and claims, the phrase "and / or" The clause does not apply to "either or both" of the elements so combined, i.e., the elements are In some cases, it is understood to mean conjunctive existence, and in other cases, disjunctive existence. Multiple elements listed with "and / or" should be treated in the same way, i.e. i.e., to be interpreted as "one or more" of the elements so joined. The elements of are optionally and specifically identified by the "and / or" clause. They may be present whether related or unrelated to the identified element. Thus, as a non-limiting example, a reference to "A and / or B" may be used in conjunction with an open When used in conjunction with the end phrase, in one embodiment, it refers to A only (optionally in another embodiment, it refers only to B (optionally including elements other than A); In yet another embodiment, it may refer to both A and B (optionally including other elements), etc. stomach.

[0220] In the scope of a claim, use of terms such as "first," "second," "third," etc. to modify claim elements is prohibited. The use of ordinal numbers does not in itself imply a priority or precedence of one claim element over another. It does not imply a sequence or the chronological order in which the activities of the method are performed. Such terms distinguish one claim element with a particular name from another element with the same name. They are only used as labels to distinguish between different classes (except for the use of ordinal numbers). The phraseology and terminology used herein is for purposes of description and not of limitation. "including," "comprises," "constitutes," "has," "contains," "includes," "contain ... "comprising," "having," "containing," "involving" The use of "" and variations thereof herein are intended to be illustrative of the terms and phrases used in the following paragraphs. and additional items.

[0221] Although several embodiments of the technology described herein have been described in detail, various modifications and Modifications will readily occur to those skilled in the art. Such modifications and improvements are within the scope of the present disclosure. It is intended that the foregoing description be taken as an example only and is within the spirit and scope of the present disclosure. These techniques are described in the following claims and their equivalents: It is limited only to definition by equivalents. [Explanation of symbols]

[0222] 1, 2 patients 100 systems 102 MxIF images 104 Information 110 Microscope 112 Computing Devices 114 Network 116 Computing Devices 200 Figures 202 raw MxIF images 204 Segmentation Information 206 Mask 208 Information 210 Information 300 Image Processing Pipeline 300 Figures 310 MxIF images 310A, 310B, 310C, to 310N images 320 MxIF Image Preprocessing Components 330 Cell Segmentation Component 340 Cell Typing Components 350 Cell Morphology Evaluation Component 360 Characterization Component 400 Processing Flow 410 processed MxIF images 410A, 410B, 410C, to 410N processed MxIF images 420 Placement information 430 Tissue Deterioration Check Component 440 Dyeing Department 440A and 440B nuclear marker images 442A Window 442B Window 444 Patch Mask Sections 444A, 444B, and 444C 446 Segmentation Mask 448 Filtered Segmentation Mask 500 Computerized Processes 502 Activities 600 feature values 602 Cell Arrangement Data 610 Computerized Processes 622 cells 622A First Cell 622B Second Cell 624 Nuclear marker (DAPI) images 626 Cell Segmentation (Binary) Mask 628 marker images 628A Nuclear marker image 628B Membrane Marker Image 628C Cytoplasm Image 634 Neural Networks 636 marker expression signature 638 cell typing data 640 cell types 650 Neural Networks 652 Probability Table 654 Cell Typing Table 656 input 656A, 656B, 656C, 656D, 656E Images 656F Segmentation Mask 660 Neural Networks 662 Probability Table 664 Cell Typing Table 666 input 666A, 666B, 666C, 666D, and 666E MxIF images 666F Segmentation Mask 700 Computerized Processes 750 Computerized Processes 774 cell data 776 graphs 772 Graph Neural Networks 778 Community 780 images 782 images 800 MxIF images 802 manual cell placement / segmentation data 1000 pre-trained convolutional neural network models 1002 cell segmentation data 1010 MxIF images 1012 First marker image 1014 Second marker image 1100 trained neural networks 1102 Cell placement / segmentation information 1110 MxIF images 1112 DAPI marker images 1114 NaKATPase marker image 1200 MxIF images 1202 Cell Segmentation Data 1300 composite fluorescent images 1302 Cell Segmentation Data 1402 cell segmentation data 1404 and 1406 MxIF images 1408 Cell Segmentation Data 1410 and 1412 MxIF images 1500 MxIF images of clear cell renal cell carcinoma (CCRCC) 1502 cell segmentation data MxIF image of 1600 CCRCC 1602 cell segmentation data 1700 Convolutional Neural Network Architecture 1702A, 1702B, and 1702C convolutional layers 1704A, 1704B and 1704C upsampling layers 1706 Threshold Image 1708 raw input images 1710 Threshold Difference 1800 Figures 1802 Cellular spatial co-occurrence 1804 Radius Check 1806 Triangulation 1808 Mask 1810 cell group 1900 immunofluorescence images 1902 Interstitial Mask 1904 Acinar Mask 1950 images 1952 images 1956 Object Mask 2000 Figure 2002, 2004 and 2006 images 2008, 2010 and 2012 Acinar Mask 2014, 2016 and 2018 Fibrosis Masks 2026, 2028, and 2030 parameters 2102 Examples 2104 Examples 2106 Apical fibrosis distribution 2108 Fundal fibrosis distribution 2202 Examples 2204 Examples 2302 Immunofluorescence images 2304 Immunofluorescence images 2306 Characteristics Information 2308 Information 2310 Information 2312 Information 2502, 2504 MxIF images 2506, 2508 Stroma segmentation mask 2602 and 2604 MxIF images 2606 and 2608 Segmentation Masks 2610 and 2612 cell groups 2702 views 2704 Blow-up View 2706 parts 2802 Restored cell arrangement 2804 Immunofluorescence images 2902 4',6-diamidino-2-phenylindole (DAPI) stained immunofluorescence image 2904 and 2906 images 3002, 3004, and 3006 CCRCC tissue samples 3200 images 3300 images 3402 images 3404 images 3402 and 3404 CCRCC tissue samples 3406 and 3408 characteristic information 3410 area 3412 area 3502 CCRCC tissue samples 3508 Blow-up View 3602 Characteristics 3604 Characteristics 3606 Characteristics 3608 Characteristics 3610 Characteristics 3612 Characteristics 3702 Tissue Images 3704 Interstitial mask 3706 Tissue Images 3708 Interstitial mask 3710 Characteristics 3712 Characteristics 3802 Interstitial mask 3804 Stromal Heatmap 3806 Graph of interstitial distribution density 3808 Interstitial mask 3810 Stromal Heatmap 3812 Graph of interstitial distribution density 3814 x-sum strength 3816 y-sum intensity 3818 Interstitial Mask 3902 First Table 3904 pie chart 3906 Second Table 3908 pie chart 4002~4016 tSNE plot 4102~4148 tSNE plot 4202 Convolutional Neural Network 4204 Cell Segmentation Data 4206 4',6-diamidino-2-phenylindole (DAPI) heterogeneous staining immunofluorescence image 4302 First Cell Mask 4304 DAPI stained immunofluorescence image 4306 CD3 cell marker image 4402 Second Cell Mask 4404 CD21 cell marker image 4502 Third Cell Mask 4504 CD11c cell marker image 4602 Vascular Mask 4604 MxIF images 4606 images 4702 Cell group 4802 cell group 4804 Prostate tissue samples 4806 Malignant site 4902 Part 5100 Computing devices and computer systems 5102 Computer Hardware Processor 5104 memory 5106 Non-volatile storage device 5202A Images 5202B Images 5204A Images 5204B Images 5206A Images 5206B Images 5302A Images 5302B Images 5304A Images 5304B Images 5306A Images 5306B Images

Claims

1. A method implemented by at least one computer, comprising performing the following steps: acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; obtaining information indicative of an arrangement of cells within said at least one MxIF image; wherein said information indicative of the arrangement of cells includes information indicative of cell boundaries of at least some of said cells; Identifying a plurality of groups of cells in said at least one MxIF image, said plurality of groups of cells comprising, at least in part, identifying pixel intensity values ​​for at least some of the cells using the at least one MxIF image and the information indicative of cell location; grouping said at least some of said cells into said plurality of groups, at least in part by for each particular cell of at least some of the cells, using information indicative of cell boundaries of at least some of the cells, identifying pixels within a boundary of the particular cell; calculating at least one feature value using the identified pixel intensity values ​​for the identified pixels within the boundary of the particular cell; grouping said at least some of said cells into said plurality of groups by identifying a plurality of groups of cells in the at least one MxIF image by: Determining at least one characteristic of the tissue sample using the plurality of groups.

2. The method described in claim 1, wherein the step of determining at least one characteristic of the tissue sample includes determining information about cell types in the tissue sample, determining a cell mask, and / or determining the spatial distribution of the cell types.

3. The method described in claim 1, wherein the step of obtaining information indicating the arrangement of cells within the at least one MxIF image includes using a neural network.

4. The neural network is implemented using a U-Net architecture or a region-based convolutional neural network architecture; The method of claim 3 , wherein the neural network comprises at least 1 million parameters.

5. The method of claim 3, wherein grouping at least some of the cells includes clustering using a graph neural network different from the neural network.

6. The method described in claim 5, wherein the clustering includes a step of performing hierarchical clustering, density-based clustering, k-means clustering, self-organizing map clustering, or minimum spanning tree clustering.

7. The method described in claim 1, wherein the at least one MxIF image includes multiple channels associated with each marker in the multiple markers.

8. at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions, the processor-executable instructions, when executed by the at least one computer hardware processor, causing the at least one computer hardware processor to: acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; obtaining information indicative of an arrangement of cells within said at least one MxIF image; wherein said information indicative of the arrangement of cells includes information indicative of cell boundaries of at least some of said cells; Identifying a plurality of groups of cells in said at least one MxIF image, said plurality of groups of cells comprising, at least in part, identifying pixel intensity values ​​for at least some of the cells using the at least one MxIF image and the information indicative of cell location; grouping said at least some of said cells into said plurality of groups, at least in part by for each particular cell of at least some of the cells, using information indicative of cell boundaries of at least some of the cells, identifying pixels within a boundary of the particular cell; calculating at least one feature value using the identified pixel intensity values ​​for the identified pixels within the boundary of the particular cell; grouping said at least some of said cells into said plurality of groups by identifying a plurality of groups of cells in the at least one MxIF image by: determining at least one characteristic of the tissue sample using the plurality of groups.

9. The system described in claim 8, wherein the step of determining at least one characteristic of the tissue sample includes determining information about cell types in the tissue sample, determining a cell mask, and / or determining the spatial distribution of the cell types.

10. The system described in claim 8, wherein the step of obtaining information indicating the arrangement of cells within the at least one MxIF image includes using a neural network.

11. The method of claim 10, wherein the neural network is implemented using a U-Net architecture or a region-based convolutional neural network architecture; The system of claim 10 , wherein the neural network includes at least 1 million parameters.

12. The system of claim 10, wherein grouping at least some of the cells includes clustering using a graph neural network different from the neural network.

13. The system of claim 12, wherein the clustering includes performing hierarchical clustering, density-based clustering, k-means clustering, self-organizing map clustering, or minimum spanning tree clustering.

14. The system described in claim 8, wherein the at least one MxIF image includes multiple channels associated with each marker in the multiple markers.

15. At least one non-transitory computer-readable storage medium storing processor-executable instructions, the processor-executable instructions, when executed by at least one computer hardware processor, causing the at least one computer hardware processor to: acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; obtaining information indicative of an arrangement of cells within said at least one MxIF image; wherein said information indicative of the arrangement of cells includes information indicative of cell boundaries of at least some of said cells; Identifying a plurality of groups of cells in said at least one MxIF image, said plurality of groups of cells comprising, at least in part, identifying pixel intensity values ​​for at least some of the cells using the at least one MxIF image and the information indicative of cell location; grouping said at least some of said cells into said plurality of groups, at least in part by for each particular cell of at least some of the cells, using information indicative of cell boundaries of at least some of the cells, identifying pixels within a boundary of the particular cell; calculating at least one feature value using the identified pixel intensity values ​​for the identified pixels within the boundary of the particular cell; grouping said at least some of said cells into said plurality of groups by identifying a plurality of groups of cells in the at least one MxIF image by: determining at least one characteristic of the tissue sample using the plurality of groups.

16. A non-transitory computer-readable storage medium as described in claim 15, wherein the step of determining at least one characteristic of the tissue sample includes determining information about cell types in the tissue sample, determining a cell mask, and / or determining the spatial distribution of the cell types.

17. The method of claim 17, wherein the step of obtaining information indicative of an arrangement of cells in the at least one MxIF image comprises using a neural network, the neural network being implemented using a U-Net architecture or a region-based convolutional neural network architecture; 16. The non-transitory computer-readable storage medium of claim 15, wherein the neural network includes at least 1 million parameters.

18. The non-transitory computer-readable storage medium of claim 17, wherein grouping at least some of the cells includes clustering using a graph neural network different from the neural network.

19. The non-transitory computer-readable storage medium of claim 18, wherein the clustering includes performing hierarchical clustering, density-based clustering, k-means clustering, self-organizing map clustering, or minimum spanning tree clustering.

20. A non-transitory computer-readable storage medium as described in claim 15, wherein the at least one MxIF image includes multiple channels associated with each marker in the multiple markers.

Images