Determination of tissue characteristics using multiplexed immunofluorescence imaging

Abstract

To provide determination of tissue characteristics using multiplexed immunofluorescence imaging.SOLUTION: Techniques for processing multiplexed immunofluorescence (MxIF) images include: obtaining at least one MxIF image of the same tissue sample; obtaining information indicative of locations of cells in the at least one MxIF image; identifying multiple groups of cells in the at least one MxIF image at least in part by determining feature values for at least some of the cells using the at least one MxIF image and the information indicative of locations of the at least some cells in the at least one MxIF image and grouping the at least some of the cells into the multiple groups using the determined feature values; and determining at least one characteristic of the tissue sample using the multiple cell groups.SELECTED DRAWING: None

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application was filed on March 6, 2020, the entire contents of which are incorporated herein by reference. No. 62 / 986,010, entitled "DETERMINING TISSUE CHARACTERISTICS USING MULTI-LAYER SYSTEMS" 119(e) of the U.S. Patent Act, which claims the benefit of the U.S. Patent No. 5,399,626, issued to the U.S. District Court for the District of Columbia for the District of Columbia District of Columbia for the filing of the same suit. This is what we have in mind. [Background technology]

[0002] Multiplexed immunofluorescence (MxIF) imaging is a method for detecting multiplexed immunofluorescence in a single biological sample. Image multiple fluorescent cells and / or histological markers in a sample (e.g., tissue sample) MxIF imaging is a technique for detecting and analyzing the staining, imaging, and chemical inactivation of dyes. Repeated activation (e.g., bleaching) and reimaging several times was used to detect multiple fluorescent markers in the biosample. The fluorescence of the markers then forms an image. MxIF imaging is used to detect multiple different markers in a single tissue sample. This allows imaging of a large number of markers (e.g., 30 to 100 markers), This allows more information to be gleaned incrementally from a single section of tissue.

[0003] As part of MxIF imaging, the cell membrane, cytoplasm, and Different types of markers can be used, including nuclear markers that bind in the nuclear region, respectively. Thus, the resulting images allow tissue analysis at a subcellular level.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 6

Non-Patent Document 7

Non-Patent Document 8

[0005] Some embodiments use at least one computer hardware processor. To this end, at least one multiplex immunofluorescence (MxIF) image of the same tissue sample is acquired, and at least obtaining information indicative of the arrangement of cells in at least one MxIF image; A plurality of groups of cells in an image are identified, at least in part, by at least one MxIF image and at least one Also, information indicating the arrangement of cells in a single MxIF image is used to identify at least some of the cells. determining feature values ​​for a number of cells and identifying a number of those cells using the determined feature values; To identify at least some cells by grouping them into groups, and to identify multiple groups. and determining at least one characteristic of the tissue sample using the set of This provides a method for

[0006] Some embodiments include at least one computer hardware processor and at least one non-transitory computer-readable storage medium storing processor-executable instructions; the processor executable instructions executing at least one computer hardware processor; At least one computer hardware processor is responsible for executing the program. acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; and obtaining information indicative of an arrangement of cells within at least one MxIF image; The plurality of groups of cells in the at least one MxIF image and at least one Using information indicating the arrangement of cells in the MxIF image, determining feature values ​​for a number of cells and using the determined feature values ​​to identify one or more of the cells; Identifying at least some of the cells by grouping them into a plurality of groups; and determining at least one characteristic of the tissue sample using the set of numbers. Provide the system.

[0007] Some embodiments include at least one non-transitory memory device that stores processor-executable instructions. A computer-readable storage medium, the processor-executable instructions being stored in at least one computer When executed by a computer hardware processor, The hardware processor is then fed with at least one multiplexed immunofluorescence (MxIF) image of the same tissue sample. and obtaining information indicative of an arrangement of cells within at least one MxIF image. and determining whether or not the plurality of groups of cells in at least one MxIF image is at least partially based on at least one Using two MxIF images and information indicating the arrangement of cells in at least one MxIF image, determining feature values ​​for at least some of the cells of the and grouping at least some of the cells into groups using and determining at least one characteristic of the tissue sample using the plurality of groups. The present invention provides a non-transitory computer-readable storage medium for causing a computer to perform the steps of:

[0008] Some embodiments use at least one computer hardware processor. acquiring at least one multiplex immunofluorescence (MxIF) image of the tissue sample; obtaining information indicative of a location of at least one cell within at least one MxIF image; A plurality of markers expressed in at least one MxIF image and Determining a marker expression signature for cells in a tissue sample based on information indicative of cell location and analyzing the marker expression signatures for at least one of a plurality of different cell types. Cell-to-cell comparison of marker expression signatures determining a type of the stimuli; and performing

[0009] Some embodiments include at least one computer hardware processor and at least one When executed by at least one computer hardware processor, and a computer hardware processor for processing at least one MxIF image of the tissue sample. and obtaining information indicative of a location of at least one cell within the at least one MxIF image. and determining whether a plurality of markers and a minority of markers are expressed in at least one MxIF image. A mapping of cells in a tissue sample based on information indicating the arrangement of cells in at least one MxIF image. determining a marker expression signature and comparing the marker expression signature to a plurality of different types of and cell typing data comprising at least one marker expression signature for the cells of the and determining a cell type for the cell. and at least one non-transitory computer-readable storage medium storing the instruction. provide.

[0010] Some embodiments include at least one non-transitory memory device that stores processor-executable instructions. A computer-readable storage medium, the processor-executable instructions being stored in at least one computer When executed by a computer hardware processor, acquiring at least one MxIF image of the tissue sample; obtaining information indicative of a location of at least one cell within at least one MxIF image; A plurality of markers expressed in at least one MxIF image and at least one MxIF A marker expression signature for cells in a tissue sample is generated based on information indicating the arrangement of cells in the image. The aim of the present study is to determine the marker expression signatures for multiple different cell types. marker expression signatures, determining a cell type to be detected; Also provided is a non-transitory computer-readable storage medium.

[0011] Various aspects and embodiments are described with reference to the following figures, which are not necessarily to scale. It should be understood that items not depicted in the figures are necessarily representative of the parts shown. In all the figures, the same or similar reference numerals refer to the same or similar components. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 pictorially illustrates an exemplary system for multiplex immunofluorescence (MxIF) imaging, in accordance with some embodiments of the techniques described herein. [Diagram 2] 1A-1C are diagrams illustrating an MxIF image and associated data that may be generated by processing the MxIF image, according to some embodiments of the techniques described herein. [Diagram 3] FIG. 1 illustrates exemplary components of a pipeline for processing MxIF images of a tissue sample to determine characteristics of cells in the tissue sample, according to some embodiments of the techniques described herein. [Figure 4A] A diagram showing an example processing flow of an MxIF image using some of the components of FIG. 3 according to some embodiments of the techniques described herein. [Figure 4B] FIG. 4B illustrates an example processing flow for the MxIF image of FIG. 4A including an optional tissue degradation check component, according to some embodiments of the techniques described herein. [Figure 4C] FIG. 13 illustrates a patch mask generated based on tissue degradation information determined by comparing two nuclear marker images of the same tissue sample, according to some embodiments of the techniques described herein. [Figure 4D]FIG. 4D illustrates the use of the patch mask from FIG. 4C to filter areas of a tissue sample for processing, according to some embodiments. [Figure 5A] FIG. 13 is a flowchart illustrating an exemplary computerized process for processing MxIF images of a tissue sample based on cell arrangement data to group cells of the tissue sample to determine at least one characteristic of the tissue, according to some embodiments of the technology described herein. [Figure 5B] FIG. 13 illustrates an example of feature values ​​for cell location associated with cell location data, according to some embodiments of the technology described herein. [Figure 6A] 1 is a flow chart illustrating an exemplary computerized process for processing MxIF images of a tissue sample to predict cell type based on cell location data of cells in the tissue sample, according to some embodiments of the technology described herein. [Figure 6B] FIG. 1 shows examples of immunofluorescence images and cell location data used for cell typing, according to some embodiments. [Figure 6C] FIG. 1 shows an example of using a neural network to generate expression data that is compared to cell typing data to determine predicted cell types, according to some embodiments of the technology described herein. [Figure 6D] FIG. 13 illustrates an example of using a neural network to generate a probability table that is compared to a cell typing table to determine a predicted cell type, according to some embodiments of the technology described herein. [Figure 6E] FIG. 13 illustrates an example of using a neural network to generate a probability table that is compared to a cell typing table to determine a predicted cell type, according to some embodiments of the technology described herein. [Figure 7A] 1 is a flow chart illustrating an exemplary computerized process for processing MxIF images of a tissue sample to cluster cells of the tissue sample into multiple cell groups based on cell location data, according to some embodiments of the techniques described herein. [Figure 7B] FIG. 1 is a flowchart illustrating an example computerized process for processing a first set of cellular features (local cellular features) to identify one or more communities of cells using a graph neural network, according to some embodiments of the technologies described herein. [Figure 7C] FIG. 1 shows an example image of a tissue contour that is shaded based on cell type, and an image of the same tissue with contours shaded based on cell clusters, according to some embodiments. [Figure 8] FIG. 1 illustrates an exemplary MxIF image and manual cell placement / segmentation data for the MxIF image, according to some embodiments of the techniques described herein. [Figure 9-1] FIG. 13 shows example images that can be used to train a neural network to identify cell location information, in accordance with some embodiments of the technology described herein. [Figure 9-2] This figure shows a continuation of Figure 9-1. [Figure 10] FIG. 1 pictorially illustrates an example of using a convolutional neural network model to process acquired MxIF images of a tumor to generate cell segmentation data, in accordance with some embodiments of the techniques described herein. [Figure 11] FIG. 10 pictorially illustrates another exemplary use of neural networks to process immunofluorescence images to generate cell location / segmentation data, in accordance with some embodiments of the techniques described herein. [Figure 12] 1A-1C show MxIF images and cell segmentation data generated based on the MxIF images, according to some embodiments of the techniques described herein. [Figure 13] 1A-1C show a composite fluorescent image and cell segmentation data generated based on the composite fluorescent image, according to some embodiments of the techniques described herein. [Figure 14]FIG. 13 illustrates exemplary cell segmentation data for an exemplary MxIF image acquired of kidney tissue, according to some embodiments of the techniques described herein. [Figure 15] FIG. 1 shows an MxIF image and corresponding cell segmentation data of clear cell renal cell carcinoma (CCRCC), according to some embodiments of the technology described herein. [Figure 16] FIG. 1 shows an MxIF image and corresponding cell segmentation data of CCRCC, according to some embodiments of the technology described herein. [Figure 17] FIG. 1 illustrates a convolutional network architecture for predicting a noise subtraction threshold for subtracting noise from raw immunofluorescence images, according to some embodiments of the techniques described herein. [Figure 18] 14A-14C illustrate example tissue characteristics that may be determined by processing an MxIF image, according to some embodiments of the techniques described herein. [Figure 19A] FIG. 13 illustrates stromal and acinar masks generated by processing an immunofluorescence image, according to some embodiments of the techniques described herein. [Figure 19B] 1A-1C illustrate generating an object mask using features of a tissue sample, according to some embodiments of the techniques described herein. [Figure 20-1] FIG. 13 shows examples of measuring acinar shape, area, and perimeter according to some embodiments of the technology described herein. [Figure 20-2] This figure shows a continuation of Figure 20-1. [Figure 21-1] 1A-1C are diagrams illustrating examples of spatial distribution characteristics according to some embodiments of the techniques described herein. [Figure 21-2] This figure shows a continuation of Figure 21-1. [Figure 21-3] This figure shows a continuation of Figure 21-1. [Figure 21-4] This figure shows a continuation of Figure 21-1. [Figure 22-1]1A-1C are diagrams illustrating examples of spatial organization properties according to some embodiments of the techniques described herein. [Figure 22-2] This is a continuation of Figure 22-1. [Figure 23-1] FIG. 1 shows an example of cell contact information of immunofluorescence images for two different patients, according to some embodiments of the technology described herein. [Figure 23-2] This figure shows a continuation of Figure 23-1. [Figure 24-1] FIG. 24 shows an example of information regarding cell neighborhood information for two different patients from FIG. 23, according to some embodiments of the technology described herein. [Figure 24-2] This figure shows a continuation of Figure 24-1. [Diagram 25] 1A-1C show two example MxIF images and corresponding stromal segmentation masks, according to some embodiments of the techniques described herein. [Figure 26] 1A-1C show exemplary MxIF images, segmentation masks, and corresponding cell groups according to some embodiments of the techniques described herein. [Figure 27] FIG. 1 shows an example of a complete MxIF slide processing to generate cell populations, according to some embodiments of the technology described herein. [Figure 28] 1A-1C are diagrams of reconstructed cell configurations generated by processing immunofluorescence images according to some embodiments of the techniques described herein. [Figure 29] FIG. 1 shows a 4′,6-diamidino-2-phenylindole (DAPI) stained immunofluorescence image and two images of different cell populations relative to the DAPI image, according to some embodiments of the technology described herein. [Diagram 30] FIG. 1 shows cell populations for different CCRCC tissue samples according to some embodiments of the technology described herein. [Diagram 31] FIG. 15 shows exemplary cell populations for an exemplary MxIF image acquired of kidney tissue from FIG. 14, in accordance with some embodiments of the techniques described herein. [Diagram 32] FIG. 1 shows a set of cell population images for different clear cell renal cell carcinoma (CCRCC) tissue samples, according to some embodiments of the technology described herein. [Diagram 33] FIG. 1 shows a series of images of cell populations for different CCRCC tissue samples, according to some embodiments of the technology described herein. [Figure 34-1] FIG. 13 shows the results of an analysis of two different MxIF images of a CCRCC tissue sample according to some embodiments of the techniques described herein. [Figure 34-2] This figure shows a continuation of Figure 34-1. [Figure 35-1] FIG. 1 shows the results of analysis of MxIF images of CCRCC tissue samples according to some embodiments of the techniques described herein. [Figure 35-2] This figure shows a continuation of Figure 35-1. [Figure 36A] FIG. 1 illustrates cell amounts and ratios according to some embodiments of the technology described herein. [Figure 36B-1] FIG. 1 illustrates cell amounts and ratios according to some embodiments of the technology described herein. [Figure 36B-2] This is a continuation of Figure 36B-1. [Figure 36B-3] This is a continuation of Figure 36B-1. [Figure 37] FIG. 1 illustrates cell distribution characteristics according to some embodiments of the technology described herein. [Figure 38A-1] FIG. 13 illustrates a percentage heat map and distribution density of histological features according to some embodiments of the technology described herein. [Figure 38A-2] This is a continuation of Figure 38A-1. [Figure 38B-1] FIG. 13 illustrates a percentage heat map and distribution density of histological features according to some embodiments of the technology described herein. [Figure 38B-2] This is a continuation of Figure 38B-1. [Figure 39-1]FIG. 1 illustrates cell neighborhood information and cell contact characteristics, according to some embodiments of the technology described herein. [Figure 39-2] This figure shows a continuation of Figure 39-1. [Figure 39-3] This figure shows a continuation of Figure 39-1. [Figure 40-1] FIG. 1 shows an example of a tSNE plot of a profile of marker expression, according to some embodiments of the technology described herein. [Figure 40-2] This is a continuation of Figure 40-1. [Figure 41-1] FIG. 2 is another illustrative diagram showing an example of a tSNE plot of a profile of marker expression, according to some embodiments of the technology described herein. [Figure 41-2] This is a continuation of Figure 41-1. [Figure 41-3] This is a continuation of Figure 41-1. [Figure 41-4] This is a continuation of Figure 41-1. [Diagram 42] FIG. 1 pictorially illustrates the use of a convolutional neural network to determine cell segmentation data for 4′,6-diamidino-2-phenylindole (DAPI) heterogeneously stained immunofluorescence images, in accordance with some embodiments of the technology described herein. [Diagram 43] FIG. 13 is a pictorial illustration of a first cell mask generated based on a combination of a DAPI stained immunofluorescence image of a tissue sample and a CD3 cell marker image, according to some embodiments of the technology described herein. [Diagram 44] FIG. 44 is a pictorial illustration of a second cell mask generated based on a combination of the DAPI stained immunofluorescence image of FIG. 43 and the CD21 cell marker image, according to some embodiments of the technology described herein. [Diagram 45] FIG. 44 is a pictorial illustration of a third cell mask generated based on a combination of the DAPI stained immunofluorescence image of FIG. 43 and the CD11c cell marker image, according to some embodiments of the techniques described herein. [Figure 46]FIG. 44 is a pictorial illustration of a vascular mask generated based on the MxIF image of the tissue sample of FIG. 43, in accordance with some embodiments of the techniques described herein. [Figure 47] FIG. 47 is a pictorial illustration of a cell population generated using the masks of FIGS. 43-46, according to some embodiments of the techniques described herein. [Figure 48] FIG. 1 illustrates a set of cell populations relative to a prostate tissue sample and malignant site, according to some embodiments of the technology described herein. [Figure 49] FIG. 49 is a magnified view of a portion of the prostate tissue sample of FIG. 48 according to some embodiments of the techniques described herein. [Figure 50] FIG. 1 shows a set of cell populations from prostate tissue samples taken from four different patients, according to some embodiments of the technology described herein. [Figure 51] FIG. 1 is a diagram of an exemplary implementation of a computer system that may be used in association with any of the embodiments of the technology described herein. [Figure 52] FIG. 1 shows two exemplary comparisons of cell location information generated using implementations of the techniques described herein and conventional techniques, according to some embodiments. [Figure 53] FIG. 1 shows two exemplary comparisons of cell location information generated using implementations of the techniques described herein and conventional techniques, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The inventors have, for example, diagnosed, or are suspected of having, cancer or another disease. Multiplexing of biological samples, such as multiplex immunofluorescence (MxIF) imaging of tissue samples from individuals with or at risk for HIV We have developed new image processing and machine learning techniques to process images. Such images are For example, labeled antibodies can be used as labeling agents. Each antibody can be fluorescently labeled, chemically labeled, or The antibodies may be labeled with various types of labels, such as markers, enzyme labels, and / or the like.

[0014] The techniques developed by the present inventors allow the determination of the cellular composition (e.g., cell types, cellular Characterize the cellular morphology (e.g., cellular morphology) and / or tissue organization (e.g., cellular localization, multicellular structure localization, etc.) It provides robust information about a variety of medically relevant properties of tissue samples that can be Examples of tissue sample characteristics include, but are not limited to, information about the types of cells in the tissue sample, morphological information, spatial information, and several tissue structures (e.g., acini, stroma, tumor, blood vessels, etc.) The information identifying the placement of the

[0015] The present inventors have determined that a tissue sample containing any of the characteristics described above is medically relevant. It is desirable to be able to distinguish different cell populations in a tissue sample that can be used to determine the characteristics of the cell population. However, conventional techniques for processing MxIF images are limited to the cell population. In particular, the prior art techniques are not fully automated. The MxIF data can be analyzed to determine tissue properties of interest. It is not possible to accurately determine the cell type, cell population and / or other information of the material. Thus, conventional techniques are unable to identify cells and / or other aspects of a tissue sample, and even Various MxIF image processing procedures, including configuring parameters for the semi-automated parts of the process Requires manual intervention for steps that result in wasted time. The tedious analytical work (e.g., image quantification, since such images typically have a large number of cells) (Manually annotating the data can be a time-consuming and significant task), The process is subject to human error and / or is not easily repeatable between different samples. and (e.g., resulting in inaccuracies in the MxIF tissue analysis procedure and / or the MxIF tissue analysis data. will provide inconsistent results (which may result in a match).

[0016] The present inventors have proposed to obtain information indicative of cell arrangement (e.g., information from which cell boundaries can be derived, cell boundaries The boundaries and / or masks are applied to the pixel values ​​of those cells in one or more MxIF images. can be used to determine feature values ​​for individual cells based on By determining one or more feature values ​​for a given cell, A bline was developed. The feature values ​​were then used to group cells into groups. This can be used to determine cellular properties of interest. In an embodiment, the feature values ​​are calculated for each channel of at least one MxIF image for one cell. This indicates how much a particular gene is expressed in each cell, and therefore the marker for the cells in the MxIF image. It can be used to determine the CAR expression signature.

[0017] In some embodiments, the cell population is determined by dividing the cell population into groups based on feature values ​​for individual cells in the tissue sample. Determine the cell types in the sample (e.g., to group cells by cell type) This can be determined by performing a cell typing process (which can be used in some cases). In an embodiment, the feature values ​​are based on traditional chemical staining of different cellular and / or tissue structures. For example, in some embodiments, the feature value for a cell may be determined based on The expression signature may include a marker expression signature for each of one or more cells. A marker expression signature can be calculated for each cell type (e.g. For example, the cells may be acinar cells, macrophages, myeloid cells, T cells, B cells, or endothelial cells. , and / or any other cell type). The cells are then grouped based on their cell type, such that each group contains cells of a particular type. may be used (e.g., a population containing acinar cells for various cell types of interest, A marker expression signature is further defined herein as a Additionally or alternatively, cell populations may be determined based on feature values. (e.g., cells of the same and / or different cell types may be identified based on having similar feature values) For example, in some embodiments, the feature values ​​of a cell can be: Any information indicating the expression levels of various markers in a cell may be included. For example, The feature value for a cell is the pixel value of the location in at least one MxIF image of a particular cell. The feature value may be determined using pixel values ​​for the cells. It may be determined by averaging or calculating the median value of the cells, which is It may represent the mean or median expression level of the marker in the cells. and determining the one or more characteristics includes determining a cell type for each of the plurality of populations. In some embodiments, the group comprises different They may represent different parts of an organizational structure.

[0018] Thus, some embodiments provide for (1) detecting a cancer-related event in a biological sample from the same biological sample (e.g., a cancer-related event in a biological sample from ... At least one sample of a tissue from a subject suspected of having or at risk of having cancer (2) acquiring an MxIF image; and (3) determining whether the cell is located within the MxIF image (e.g., cell boundaries). ,information from which cell boundaries can be derived, and / or masks (e.g., the boundaries of cells in an image By applying machine learning models to identify cell arrangement information (e.g. For example, a cell positioning mask) and / or access such cell positioning information. (3) obtaining a plurality of groups of cells in at least one MxIF image by performing a subsample of the plurality of groups of cells in the at least one MxIF image. At least in part, (a) using the MxIF image and information indicative of the arrangement of at least some of the cells. and determining feature values ​​for at least some of the cells (e.g., as indicated by information indicating the cellular disposition of at least some of those cells. The cell size is calculated by calculating the mean or median pixel value of the pixels within the boundary of the cell. Calculate feature values ​​for the cells, calculate marker expression signatures as described herein, (b) calculating feature values ​​for the cells by Group at least some of the cells into groups (e.g., classify the cells into groups of different types) 2. Group by average or mean using any suitable clustering algorithm (4) identifying the cells by grouping them based on median pixel value, etc. The plurality of groups are used to determine at least one characteristic of the tissue sample (e.g., the cell type of each group). Determine the class, determine the cell mask, determine the cell community, determine the cell communities of multiple groups of cells Determine statistical information about the distribution; Determine spatial distribution of cell types; Determine morphological information and (e.g., performing a step of detecting a plurality of pixels in a plurality of pixels).

[0019] In some embodiments, the information indicative of the location of a cell includes chemical or physical properties of a cell or a cell structure. In some embodiments, the chemical staining can be obtained by a fluorescent signal (e.g., In some embodiments, a label (e.g., a fluorescent Photolabeled (photolabeled) antibodies target specific intracellular, membrane, and / or extracellular locations in cell or tissue samples. In some embodiments, a labeled antibody may be used alone to detect In some embodiments, the labeled antibody may be coupled to one or more other stains (e.g., other Fluorescent stains may also be used.

[0020] In some embodiments, acquiring at least one MxIF image is performed on a same tissue. acquiring a single multi-channel image of the sample; A channel may be formed by multiple markers (e.g., as described in the Detailed Description section). any one or more of the markers described herein, including markers In some embodiments, at least one Acquiring one MxIF image involves acquiring multiple immunofluorescence images of the same tissue sample. At least one of the multiple immunofluorescence images is associated with each marker. A single channel image of at least one of the immunofluorescence images may be included. One can also include a multi-channel image, where the channels in the multi-channel image are A plurality of markers (e.g., the markers described in the Detailed Description section) Each marker in any one or more of the markers described herein, including In some embodiments, at least one of the tissue samples is associated with MxIF images are captured in vitro.

[0021] In some embodiments, the feature values ​​for the cells are determined from one or more of the MxIF images. For example, in some embodiments, the cell may include values ​​of pixels of the first determining a feature value for the first cell in the at least one MxIF image; A first feature for a first cell using at least one pixel value associated with the In some embodiments, determining a characteristic value of at least one of the cells. At least some of the cells include a second cell, and determining the feature value includes detecting at least one MxIF a second pixel value associated with the location of the second cell in the image; In some embodiments, determining a second characteristic value for the first cell. Determining a first feature value for the cell includes determining a first feature value for the cell based on a plurality of channels of at least one MxIF image. using pixel values ​​associated with each location of the first cell in the Includes.

[0022] In some embodiments, at each position of the first cell in the plurality of channels, Using the associated pixel values ​​includes, for each of the plurality of channels: (a) Pixels for the first cell are determined using information indicating the location of the first cell in that channel. (b) identifying a set of pixels for a first cell based on values ​​of pixels within the set of pixels; and determining a feature value for the first cell. The information indicative of the first cell indicates a boundary location of the first cell and identifies a set of pixels for the first cell. The separating includes at least partially (e.g., partially or completely) separating the first cell from the first cell. This involves identifying the pixels that are inside.

[0023] In some embodiments, determining the first characteristic value comprises: One or more of the markers on the image (e.g., at least 1, at least 3, at least 5 at least 10, at least 15, a number between 1 and 10, a number between 5 and 20, or or any other suitable number, range, or value within the range. These markers are ALK, BAP1, BCL2, BCL6, CAIX, CCASP3, CD10, CD106, CD11b, CD1 1c, CD138, CD14, CD16, CD163, CD1, CD1c, CD19, CD2, CD20, CD206, CD209, CD21, CD 23, CD25, CD27, CD3, CD3D, CD31, CD33, CD34, CD35, CD38, CD39, CD4, CD43, CD44, CD45, CD49a, CD5, CD56, CD57, CD66b, CD68, CD69, CD7, CD8, CD8A, CD94, CDK1, CDX 2, Clec9a, chromogranin, collagen IV, CK7, CK20, CXCL13, DAPI, DC-SIGN, Death Min, EGFR, ER, ERKP, fibronectin, FOXP3, GATA3, GRB, granzyme B, H3K36T M, HER2, HLA-DR, ICOS, IFNg, IgD, IgM, IRF4, Ki67, KIR, lumican, Lyve-1, mamma Globin, MHCI, p53, NaKATPase, PanCK, PAX8, PCK26, CNAP, PBRM1, PD1, PDL1, Par Lecan, PR, PTEN, RUNX3, S6, S6P, SMA, SMAa, SPARC, STAT3P, TGFb, Va7.2, and It is vimentin.

[0024] In some embodiments, multiple markers are present in a single channel, and / or may be detected. For example, a signal from a chemical stain (e.g., DAPI) may be detected by Also used to provide immunofluorescence signals (e.g., to detect cellular location information) In some embodiments, the images may be taken of the same tissue sample (or channel). There is only one marker in a single channel and / or image.

[0025] In some embodiments, grouping the cells into a plurality of cell populations comprises clustering. A clustering algorithm is used to cluster cells based on their feature values. For example, centroid-based clustering algorithms (e.g., K-means), Distribution-based clustering algorithms (e.g., clustering using Gaussian mixture models) clustering), density-based clustering algorithms (e.g., DBSCAN), hierarchical clustering Rastering algorithms, Principal Component Analysis (PCA), Independent Component Analysis (ICA), and / or any Any suitable clustering algorithm, including other suitable clustering algorithms of Any algorithm may be used, as the aspects described herein are not limited in this respect.

[0026] In some embodiments, at least some of the cells are divided into multiple populations. The grouping is carried out by analyzing the determined feature values ​​to identify at least some of the cells. Determining the relationship between cells and dividing the population into groups The cells are then assigned a characteristic value that indicates the relationship between the cells in the group. In some embodiments, determining whether at least some of the cells Determining the relationships between several cells requires identifying at least some of those cells. In some embodiments, the method further comprises determining the similarity between the feature values ​​of the cells. Determining the relationships between at least some of the cells of the set of cells includes determining the feature values. Compared with the cell typing data available, the cellular typing of at least some of these cells This involves determining the cell type for each of the cells.

[0027] In some embodiments, these techniques involve cell typing based on feature values ​​of individual cells. In some embodiments, the characteristic values ​​include cell type and and / or a channel that can be used to determine cell populations (e.g., cell clusters). channel contribution (e.g., the average channel contribution that shows how much each channel contributes to a cell) The inventors have demonstrated that the average channel contribution can be used for cell typing, but In some cases, exploiting the average channel contribution in a cell can be useful for segmentation. may be affected by the quality of the extraction, cell size, cell shape, and / or tissue staining. For example, such techniques are prone to fluctuating marker intensities. It is possible that the marker may be missing from the contours of nearby cells (e.g., due to segmentation errors). Add expression and / or create additional clusters of cells with intermediate marker expression As another example, the inventors They understood that channel contribution does not take into account information about intracellular signal localization. (e.g., they can be useful in helping to distinguish real signal data from noise. As a further example, we have found that the value of the channel contribution for a particular type of cell Without the ability to set ranges, it would be difficult to purposefully search for cell types of interest. We also understand that it can be difficult to (This may result in missing cells of interest.) Therefore, we We understand that equal channel contributions may not provide stable cell typing. Given such potential problems, cell typing results are manually checked. This may increase delays and affect automation.

[0028] Therefore, the present inventors developed a technique for cell typing that utilizes machine learning. In some embodiments, cell typing comprises typing each of one or more cells. Using a trained neural network to determine marker expression signatures in The marker expression signature for a cell can be performed by: For each particular marker of one or more markers, The marker expression signature of a cell can then include the likelihood that the marker is expressed in the cell. , cell types (e.g., marker expression signatures for each cell type, e.g., For example, the expression signatures may be compared to previously determined marker expression signatures correlated by a pathologist. Such machine learning techniques can be used to identify (compared to conventional techniques, which may require the user to manually adjust settings) In some embodiments, the present invention provides an automated method for typing trained neurites. The global network can be configured with one or more channels (e.g., a separate 1-channel MxIF image , 2-channel MxIF image, 3-channel MxIF image, and / or the like) It can also take as input one MxIF image and cell location data for the cell of interest. The trained neural network can predict not only the marker expression intensity but also the detected cell shape, Other data such as cell texture, location of marker expression, etc. can also be used. The neural network calculates the number of cells for each channel and its associated marker. Outputting the likelihood (e.g., between 0 and 1) of having a signal of appropriate strength and shape The likelihood determined by the neural network for each marker is These are then combined to generate a marker expression signature for the cells of interest. This can be compared to cell typing data to determine predicted cell types for the cells of interest. The type of signal can be determined by using trained neural networks to distinguish between different signal level distributions. Robustness of the network (e.g., training the network using heterogeneous training data) , by utilizing additional features as described both above and herein. By virtue of this, the cell typing approach described herein allows for the identification of a specific locus for each cell. It provides automated cell type detection that results in robust signal presence determination.

[0029] In some embodiments, the multiple channels include a plurality of markers. First cells in multiple channels of at least one MxIF image associated with a marker determining a first feature value using pixel values ​​associated with each of the locations of the first feature; means, for each particular marker of one or more of the plurality of markers, A marker expression signature that includes the likelihood that the particular marker is expressed in the first cell. and dividing at least some of the cells into a plurality of groups. Grouping was performed using marker expression signatures and cell typing data. determining a predicted cell type for the first cell; and and associating the first cell with one of a plurality of groups. and determining feature values ​​for at least some of the cells, A marker expression signature is generated for each specific cell of a plurality of cells in at least one MxIF image. determining a marker expression signature, the marker expression signature being one or more of the plurality of markers; For each particular marker or markers, the specific marker is expressed in a particular cell. determining the likelihood that the feature values ​​are expressed in the feature sets; Grouping at least some of the cells into groups indicates that a particular cell Using marker expression signatures and cell typing data for cells, Determining the predicted cell type for each particular cell and classifying the cells based on the predicted cell type and grouping the plurality of cells into a plurality of populations. Contains at least one of the markers described.

[0030] In some embodiments, the cell typing data includes a set of data for each of a plurality of cell types. and detecting at least one marker expression signature for each particular cell of the plurality of cell types. At least one marker expression signature for a cell type is determined by determining which of a plurality of markers It includes data showing whether it is expressed in cells of a particular cell type.

[0031] In some embodiments, the method is configured to determine a marker expression signature. We first used a pretrained neural network to generate marker expression signatures. In some embodiments, the first trained neural network further comprises determining In some embodiments, the network includes at least 1 million parameters. The method includes training immunofluorescence images of tissue samples for associated cell types as input images. A set of images and information indicating marker expression in the input images for the associated cell types. and associated output data including the information about the first trained neural network. In some embodiments, the method further comprises training at least one providing the MxIF images as inputs to a first trained neural network; Obtain marker expression signatures as output from a trained neural network of This includes the following.

[0032] In some embodiments, the method further comprises: determining a number of channels of a cell from the information indicative of the cell arrangement; The first training information indicates the arrangement of the first cell in at least some of the channels. As part of the input to a trained neural network. A neural network can include multiple convolutional layers.

[0033] In some embodiments, the cell typing data includes a set of data for each of a plurality of cell types. The method includes the steps of: Determining the predicted cell type of the first cell includes determining a marker expression signature of the first cell. The marker expression signature is associated with at least one of the plurality of marker expression signatures. In some embodiments, the method further comprises comparing the predicted cell type to determine the predicted cell type. The marker expression signature of a cell is determined to be at least one of a plurality of marker expression signatures. Comparison to another marker expression signature is performed using a distance measure. In some embodiments, the distance measures include cosine distance, Euclidean distance, and In some embodiments, the method is at least one of the Manhattan distance. ,select the comparison metric that has the lowest or highest value among the calculated comparison metrics. determining a predicted cell type by

[0034] In some embodiments, the at least one characteristic of the tissue sample is determined by analyzing the cellular composition of the tissue sample. In some embodiments, the composition of the tissue sample is characterized, or the tissue composition of the tissue sample is characterized, or both. Determining the at least one characteristic determines information about cell types in the tissue sample. For example, in some embodiments, information regarding cell types in a tissue sample may be collected. Determining the information identifies one or more cell types present in the tissue sample. In some embodiments, the cell types include endothelial cells, epithelial cells, macrophages, The cells include one or more of: T cells, malignant cells, NK cells, B cells, and acinar cells. In some embodiments, the T cells are selected from the group consisting of CD3+ T cells, CD4+ T cells, and CD8+ T cells. In some embodiments, information regarding cell types in a tissue sample is provided. Determining the percentage of one or more cell types in a tissue sample. In some embodiments, the cell types include In some embodiments, cell types can be determined based on their size, shape, Histologically determined based on appearance and / or staining (e.g., using different chemical stains) Additionally or alternatively, immunofluorescence signals can be used to assess cell size and shape. Additionally or alternatively, the method may be used to assess cell type and cell state. Cell-specific markers (e.g., proteins) are used to determine cell type (e.g., target By using a specific antibody,

[0035] Any of a number of varieties of tissue characteristics have been developed by the inventors and are described herein. For example, in some embodiments, at least Determining the further characteristic relates to a distribution of at least a portion of the cells in the plurality of populations of cells. For example, in some embodiments, the method includes determining statistical information relating to a plurality of cells. Determining statistical information regarding the distribution of at least some of the types of cells in a tissue sample In some instances, determining the spatial distribution of one or more cell types in the Determining statistical information regarding the distribution of at least some of the cells of the This includes determining the distribution among different cell types.

[0036] As another example, in some embodiments, determining the at least one characteristic determining spatial information regarding the arrangement of at least some of the cells of the plurality of groups of cells; In some embodiments, determining spatial information includes determining spatial information of a plurality of groups of cells. In some embodiments, determining spatial information includes determining the distance between cells. The method includes: obtaining a sample of a tissue containing one or more cells of a group of the plurality of groups of cells; In some examples, determining at least some of the fine areas includes determining a number of fine areas. Determining spatial information regarding the arrangement of cells can be performed by determining the spatial arrangement of cells in one or more cell types of a tissue sample. Determining spatial organization (e.g., determining the spatial organization of cells in a tissue sample by cell type) (including information about the composition of the cell, compared to other cell types, and / or similar). In some instances, determining spatial information about the arrangement of at least some cells is One or more regions of the tissue sample containing one or more cell types (e.g., The method includes determining a region that contains a large number of cells of one or more cell types.

[0037] As another example, in some embodiments, determining the at least one characteristic determining morphological information for at least some of the cells of the plurality of populations of cells; In some instances, determining morphological information about at least some of the cells may be performed. Characterize the cell and / or tissue sample, including cell shape, structure, morphology, and / or size information on the morphology and / or structure of the This includes determining the amount of information required to

[0038] As another example, in some embodiments, determining the at least one characteristic , for at least some cells of a plurality of groups of cells (e.g., cells of one group, a plurality of determining physical information (for a group of cells, etc.), the physical information including cell area, The method includes determining at least one of: cell circumference, cell size.

[0039] In some embodiments, the determined characteristics include morphological information, spatial information, tissue structure information, and the like. The present inventors have also found that the tissue samples may include a microarray of a tissue structure, a microarray of a tissue sample, and / or the like. by creating one or more masks that can be used to analyze the cellular structure. Such characteristics can be determined, the analysis of which can then be used to determine properties of interest. However, as pointed out above, typically such The masks can be used to automatically identify cells and / or cell groups in tissue samples using computerized techniques. They cannot be distinguished and must be created manually (e.g., using signal thresholding). Automatically determine cell information such as cell segments and / or cell groups. By utilizing feature values ​​to identify the characteristics of tissue samples, such masks can be used to identify the characteristics of tissue samples. Analyzing different cells and / or cellular structures in a tissue sample may enable one to determine For example, these techniques can be used to identify cell types (e.g., T Identify T cells (interstitial and / or non-interstitial regions of the tissue sample) and use the stromal mask to determine whether T cells are present in the interstitial and / or non-interstitial regions of the tissue sample. It is possible to identify whether the defect is in the quality region.

[0040] In some embodiments, the mask may be a binary mask. 0 or 1 (or any other suitable) for at least some of the pixels in the MxIF image It may be a pixel-level binary mask containing the binary values ​​of the type.

[0041] In some embodiments, determining at least one characteristic comprises: determining one or more acinar masks that indicate the arrangement of acini in at least one of the multiplexed immunofluorescence images; In some embodiments, the acinar mask for the MxIF image may include: A binary mask may be used to represent at least some of the pixels in the MxIF image. The binary value for a pixel may include a binary value for the pixel represented in the MxIF image. This indicates whether the nucleus is located within the acini where the nucleus is located.

[0042] As another example, in some embodiments, determining the at least one characteristic , one or more stromal masks indicative of the location of stroma in at least one MxIF image of the tissue sample. In some embodiments, the stromal mask for the MxIF image includes determining It may be a binary mask, and at least some of the pixels in the MxIF image may be The binary value for a pixel may include a value indicating whether the pixel is to be represented in the MxIF image. The term indicates whether the endothelial cell is located within the interstitium.

[0043] As another example, determining the at least one characteristic comprises determining at least one Mx determining one or more tumor masks indicative of the location of the tumor within the IF image. In some embodiments, the tumor mask for the MxIF image may be a binary mask, and a binary value for at least some of the pixels in the F image; The binary value for a pixel indicates whether the pixel is located within the tumor shown in the MxIF image. This indicates whether

[0044] The present inventors have demonstrated that some of the cellular structures in tissue samples, such as those indicative of cancer (e.g., breast cancer, renal cancer, etc.), We further understand that it may be desirable to explore some cellular structures. Several previous approaches to perform cell clustering into a single unity have been proposed, including the reconstitution of Use information from neighbors in the cell contact graph (e.g., https: / / pubmed.nc Yury Goltsev et al., “Deep Profiling of Mice,” available at bi.nlm.nih.gov / 30078711 / e Splenic Architecture with CODEX Multiplexed Imaging”, PMID:30078711 (August 2018 However, such an approach does not allow for the use of the method described in the present invention. The present invention does not incorporate other information that may be relevant to the clustering process.

[0045] We use a graph neural network that exploits the features of cells to We developed a technique to identify clusters or communities. Cell features are For this purpose, cell types, cell neighbors, neighboring cell types, neighbor distance data, and / or It may also include other data as further described in the specification. The technique identifies cellular communities in tissue samples by exploiting such cellular features. and identifying information about those cells, such as cell type and distance (e.g., sparse (to provide information about cell clusters present in the target cell population, adjacent cell clusters, etc.) Such a mask can therefore be used to detect signals that would otherwise be difficult to detect. Complex structures within tissues, which can be difficult to identify (e.g., compared to using thresholding approaches) It allows for the discovery of new classes of clusters and provides automated clustering at scale. Some embodiments apply a graph neural network to identify cell clusters or performs community detection.

[0046] In some embodiments, determining the at least one characteristic comprises determining the at least one characteristic based on a plurality of groups. determining one or more cell clusters of the tissue sample based on the determined cell clusters; and determining a cell cluster mask indicative of an arrangement of several cell clusters. In an embodiment, the one or more cell clusters generate a first set of cell features ( For example, we triangulate the image to generate a graph and use it to estimate cell neighborhoods, cell neighborhood distances, and (Determine at least some of the cell features, such as separation, etc.) We embed the first set of cell features into a higher dimensional space using a network. By identifying communities of cells by clustering observed features is determined.

[0047] In some embodiments, determining one or more cell clusters in a tissue sample This means that there is a node for each of at least some of the cells, and a node for each of the nodes in the graph. The method involves generating a graph with edges between nodes that determine the features to be used, and then finding the filling factor in the latent space. In order to obtain the embedded features, we use the graph as input to the graph neural network. We provide features for the nodes in the cluster, and embedding them to obtain clusters of nodes. and clustering the features obtained by the clustering of nodes into one or more cells. and determining the clusters.

[0048] In some embodiments, each cell cluster of the one or more cell clusters comprises a plurality of In some embodiments, each cell cluster comprises a number of cell types. In some embodiments, the tissue structure represents at least a portion of the mantle tissue, interstitial tissue, The tissue may include stromal tissue, tumors, follicles, blood vessels, or any combination thereof.

[0049] In some embodiments, the method includes a first determination of a cellular characteristic of at least some of the cells. The method further comprises determining one or more cell clusters based on the set. A method for determining whether a sample contains a node for each of at least some of the cells in the sample, and edges between the nodes. determining a first set of cell features for each cell by generating a graph including the In some embodiments, the method may further include triangulation (e.g., , Delaunay triangulation and / or any other type of triangulation) to determine the size of the tissue sample. The method further includes generating a graph based on at least some of the cells.

[0050] In some embodiments, the method further comprises: The edge lengths of the nodes in the graph and the cells for each node in the graph based on the mask data. In some embodiments, the method further comprises determining the first set of features. In some embodiments, the method further comprises encoding the graph into a sparse adjacency matrix. The method further includes encoding the graph into an adjacency list of edges of the graph.

[0051] In some embodiments, the method includes: The network is provided with a graph neural network, which then calculates the number of nodes for each node. and obtaining a set of feature embeddings. A trained graph neural network contains one or more convolutional layers. In one embodiment, the set of feature embeddings is generated using a trained graph neural network. In some embodiments, the activations of the final graph convolution layer of Determining one or more cell clusters is based on a set of feature embeddings for each node. In some embodiments, the method further comprises determining one or more cell clusters based on the determined cell clusters. determining one or more cell clusters based on a set of feature embeddings for each node; is a method for clustering cells in a tissue sample based on a set of feature embeddings for each node. This includes:

[0052] In some embodiments, the MxIF images are pre-processed before feature values ​​for the cells are determined. In some embodiments, machine learning techniques may be used to pre-process the MxIF images to machine learn. These techniques are applied to remove artifacts and / or identify information indicative of cell placement. The inventors have demonstrated that MxIF images are generated by, for example, a microscope imaging tissue. Noise, noise from surrounding cells in the tissue sample (e.g., fluorescence noise), antibodies in the tissue sample noise from the image, and / or any other type of artifacts that may be present in the MxIF image. It is understood that any imaging may have artifacts that are introduced during imaging. For example, low signal-to-noise levels may affect cell clustering and therefore , raw MxIF images and / or data generated by the cell segmentation process The data may not be sufficient for automated cell clustering. Process the immunofluorescence images, including performing background subtraction to remove noise. We developed a technique to remove artifacts.

[0053] Thus, in some embodiments, background subtraction is performed on one of the MxIF images. Or multiple samples may be performed. Thus, in some embodiments, tissue samples The acquiring of the at least one MxIF image includes acquiring a first channel of the at least one MxIF image. In some embodiments, the method further comprises performing background subtraction on the The performing background subtraction is configured to perform background subtraction. A second trained neural network model (e.g., a convolutional neural network) The input to the network is a convolutional neural network with a U-net architecture. In some embodiments, the method further comprises providing a first channel. The neural network includes at least 1 million parameters. The method includes receiving a set of noisy training immunofluorescence images as input images and receiving at least Associated output data, which contains the associated image without any noise and training a second trained neural network using the trained neural network. While pre-processing of MxIF images is possible, aspects of the technology described herein are not limited in this respect. Therefore, in addition to (in some embodiments) background subtraction, or (in some embodiments) It should be understood that in embodiments, other types of pre-treatment may instead be included. For example, preprocessing can include filtering, noise suppression, artifact removal, smoothing, etc. Different areas for performing filtering, sharpening and pre-processing operations (e.g. wavelet Transforms to the image domain (time domain, Fourier domain, short-time Fourier domain), and back to the image domain. and / or any other suitable type of pretreatment. These techniques are useful because some regions of the image may show more noise than others. Additionally or alternatively, some In some embodiments, the images may be processed on a per-channel basis (e.g., with different markers). (Denoise for each marker, as each marker may exhibit different noise). In, the trained machine learning model processes the immunofluorescence images and thresholds the immunofluorescence images. Perform background subtraction by:

[0054] We have demonstrated that the MxIF image acquisition process repeatedly stains the same images with different markers. This requires that the marker be washed out of the tissue sample, resulting in local tissue damage. As a result, individual cells and / or cell populations may be may be damaged, such as washed away and / or shifted with respect to their original placement in the material. The use of such unintentionally modified parts of the tissue specimen may result in poor imaging performance. This can cause undesirable effects in the pipeline. For example, As a result of the damage, inaccurate information about the damaged cells is generated in the feature values, cell groupings, and If implemented at all, the prior art requires manual Therefore, we performed an autopsy on the immunofluorescence images. Develop an automated tissue degradation check (e.g., nuclei acquired across multiple staining steps) markers) to check and / or identify areas of tissue damage The identified regions indicate that only the undamaged parts of the tissue sample are included in the image processing pipeline. In order to ensure that the data are analyzed by the subsequent steps (e.g., the The cell typing component 340 and the cell morphology evaluation component of the image processing pipeline 300 are Steps performed by the component 350 and / or the characterization component 360 ) to skip damaged areas from processing. A mask may be used to mask out tissue samples with severe damage (such as The techniques described herein (compared to conventional techniques that would not be able to process tissue samples) This may allow the tissue to be treated using techniques.

[0055] In some embodiments, at least one MxIF image comprises a plurality of immunofluorescence images. Obtaining information indicative of the arrangement of cells in at least one MxIF image is a method for detecting multiple immune cells. The method includes analyzing the fluorescent image to identify one or more damaged portions of the tissue sample.

[0056] In some embodiments, analyzing the plurality of immunofluorescence images includes analyzing the plurality of immunofluorescence images. A third trained neural network is used that is configured to distinguish between the differences In some embodiments, the method further comprises processing the plurality of immunofluorescence images by using an input A training set of immunofluorescence images containing pairs of force images and immunofluorescence images of the same marker and tissue sample associated output data that contains information about whether at least a portion of the The method further includes training a third trained neural network using the data.

[0057] In some embodiments, the method comprises: Input the corresponding portion of each of the immunofluorescence images into a third trained neural network. wherein at least two immunofluorescence images include images of the same marker. and determining from the third trained neural network at least one of the portions of the tissue sample. In some embodiments, the at least one A classification comprises a set of classifications. In some embodiments, the set of classifications comprises The first category is without cells, the second category is undamaged cells, and the third category is damaged cells. In some embodiments, the classification includes a set of classifications, For each classification, the set includes an associated confidence that the classification applies to the part, and a confidence and selecting a final classification from the set of classifications for the portion based on the reliability.

[0058] In some embodiments, the method further comprises the steps of: generating a patch mask indicative of a final classification for a plurality of other portions of the immunofluorescence image. In some embodiments, the method comprises: Remove a portion of the servicing mask to obtain the tissue sample associated with the removed portion. The method further includes preventing the cell from being included in multiple cell populations.

[0059] In some embodiments, the third trained neural network comprises one or more In some embodiments, the third trained neural net includes a number of convolutional layers. The work includes at least 5 million parameters. In some embodiments, the training The neural network generates signals that contain at least one classification for different parts of the tissue sample. In some embodiments, the at least one classification is configured to output a reliability. In some embodiments, the set of classes includes a set of classes, such as cells. The first category is for damaged cells, the second category is for undamaged cells, and the third category is for damaged cells. classes, or any combination thereof.

[0060] We have used conventional cell segmentation algorithms to process MxIF images. We understand the deficiencies of the approach to processing MxIF images. In this paper, we perform cell segmentation based on nucleus thresholding, and allow users to select the best fit for different images. Manually adjusting settings and / or identifying cell membranes (e.g., if the user is looking for a nucleus) (by expanding each kernel by contouring it with the desired number of pixels) An example of such a conventional approach is CellProfiler ( To illustrate the improvements in the cell segmentation techniques described herein, the following is provided with reference to Figures 52-53. (further described below), Ilastik (e.g., https: / / www.nature.com / articles / s41 Stuart Berg et al., "ilastik: interactive machine learning," available at 592-019-0582-9 learning for (bio) image analysis” (September 2019), and QuPath (e.g., Available at: https: / / www.nature.com / articles / s41598-017-17204-5. Peter Bank head et al., “QuPath: Open source software for digital pathology image analysis” (20 Some conventional approaches use neural networks to Although such neural networks can be used, their use is typically limited to nuclear detection. Therefore, conventional approaches use As a result, we have achieved a high degree of accuracy compared to conventional cell segmentation techniques. The drawbacks of mobile app technologies are that they require manual input from users and are often poor and inconsistent. Cell segmentation results without defects (which are useful for cell typing, cell morphology assessment, and characterization) Further downstream steps that utilize cell segmentation data, such as sex determination I understand that there are various deficiencies, including the possibility that

[0061] Some embodiments use a trained neural network to generate cell segmentation data. The trained neural network includes a set of cells. We train the network using images with different densities to adapt it to MxIF images. As a result, these techniques make the trained models robust to signal variations. This can be costly, requiring the user to manually adjust settings for each individual immunofluorescence image. Therefore, such a technique offers the ability to rapidly process large numbers of immunofluorescence images. (e.g., each step requires manual adjustment and curation) (Compared to the prior art.)

[0062] In some embodiments, the method further comprises: providing information indicative of the arrangement of cells within at least one MxIF image; The obtaining includes: applying a fourth trained filter to at least one channel of at least one MxIF image; This involves applying a neural network to generate information indicative of cell placement. For example, in some embodiments, a neural network model (e.g., convolutional The neural network (RNN) is used to identify cell boundaries in one or more immunofluorescence images. The output of the neural network can be applied to, for example, and / or at least in part (e.g., in part) by identifying by identifying pixels that are within the cell boundary (or entirely within the cell boundary) in any suitable manner. To identify cell boundaries, the neural network model , may be applied to one or more immunofluorescence images generated using membrane markers. In some embodiments, the fourth trained neural network is based on the U-Net architecture. Implemented using a convolutional neural network architecture or a region-based convolutional neural network architecture. In some embodiments, the fourth trained neural network comprises at least Both contain 1 million parameters.

[0063] In some embodiments, a fourth trained neural network (e.g., a CNN) is a set of training immunofluorescence images of a tissue sample as input images, and the location of cells in the input images. and an associated output image that includes information indicative of Any suitable training technique for training the neural network may be used, including those described herein. Aspects of the technology are not limited in this respect and may be applicable.

[0064] It should be understood that the embodiments described herein can be implemented in any number of ways. Examples of specific implementations are provided below for illustrative purposes only. The embodiments and features / functions provided are not intended to limit the scope of the technology described herein. may be used individually, all together, or in any combination of two or more. For example, the various figures may be modified in accordance with the techniques described herein. This document describes the various processes and sub-processes that can be performed by , can be used individually and / or in combination with each other.

[0065] Each of the multiple processes and sub-processes may be performed on the same MxIF image and / or This may be performed using one or more types of information determined based on the image. The characteristics described in can be obtained from the same set of MxIF images and / or the same tissue sample.

[0066] The techniques developed by the present inventors provide a novel advantage over conventional techniques for processing MxIF images. As described herein, some of these techniques are mechanically Using learning methods to identify cell types in tissue cells, cell segmentation data Data (e.g., cell arrangement in a tissue sample, cell size and / or shape, cell segmentation, etc.) The purpose of the MxIF staining process is to determine the data (e.g., data showing tissue damage, etc.) Checking for scratches (e.g., by analyzing images taken at different staining steps) by identifying cellular communities (e.g., representing distinct structures in tissue samples and To identify a group of cells that may be of a certain cell type, and / or to process MxIF images. Removing noise from MxIF images to reduce background noise The method involves performing one or more of the following:

[0067] Using these machine learning methods, we can achieve results that would otherwise be impossible using traditional techniques. In addition, a fully automated MxIF image processing pipeline is realized. A machine learning model is a pre-trained neural network to carry out such a method. Neural networks are trained on a large set of training data that contains specific input data. The associated output data discovered and evaluated by the inventors is described in the present specification. For example, the present invention provides a method for training machine learning techniques to perform the methods described in the present application. The authors propose that the neural network receives as input (a) an image containing nuclear information (e.g., DAPI image), (b) immunofluorescence image (e.g., cell membrane, cytoplasm, etc.), and (c) cells within the image. It receives information about the arrangement of markers in the input image and calculates the likelihood that a cell is expressed by the marker. We have found that the likelihood can be trained to serve as a force for the cell under analysis. These may be used to generate marker expression signatures that can then predict cell types. As another example, the inventors have compared the graphs to cell typing data to estimate the The neural network receives as input a graph having nodes representing tissue cells. outputting higher dimensional data that can be used to group cells into communities We discovered and understood that a graph can be trained to do the following for each node: For example, the cell’s group membership, cell location information, and node number in the graph can be obtained. Data for associated tissue cells, such as the typical edge length for the edges of the Graph neural networks cluster cells into cell communities. Embedding node data into a higher dimensional space that can be used to rasterize As a result, the trained neural network can perform better than other models using conventional techniques. These and other examples are provided herein for purposes of illustration only. This is further explained in the document.

[0068] A neural network is trained using such training data, and the trained neural network is A neural network is a network that performs its associated function. Such a machine learning model determines the final parameters of the network. As described in , and / or have a huge number of parameters, such as hundreds of millions of parameters. As further described herein, the trained neural network may , capable of performing tasks in an automated, repeatable manner and with high precision ( For example, a trained neural network can then feed the model imaging data Robust enough to be immune to imaging noise and / or signal level distribution differences within the (Because it is trained using training data that gives it consistency.)

[0069] 1 and 2 are diagrams showing an overview of the technology described in this specification. FIG. 1 illustrates an example system 100 for MxIF image processing, in accordance with some embodiments of the present technology. The microscope 110 and the computing device 112 are In the example, anti-human CD31 antibody, a fluorescent marker for CD8 T cells, CD68 antibody, and NaKATPase The MxIF image 102 is used to acquire a set of one or more MxIF images 102 of the tissue sample, including: In some embodiments, different markers are displayed in different colors in the MxIF image 102. For example, CD31 antibodies can be shown in red and CD8 T cells can be shown in green. , CD68 antibody can be shown in magenta and NaKATPase can be shown in grey. The computing device 112 computes the MxIF image 102 via the network 114. The received signal is transmitted to the video signal processing device 116.

[0070] Tissue samples include, but are not limited to, blood, one or more body fluids, one or more cells, , one or more tissues, one or more organs (which may, for example, include multiple tissues), and / or any other biological sample obtained from the subject. A tissue sample can be a sample that contains cellular material (e.g., one or more cell types). one or more cells) and / or extracellular material (e.g., extracellular material that connects cells The matrix, cell-free DNA in the extracellular components of the tissue, etc. As described above, tissue samples are analyzed in vitro at the tissue and / or cellular level. obtain.

[0071] The MxIF immunofluorescence images described herein may be one or more immunofluorescence images of the same tissue sample. For example, MxIF images may include images of the same tissue sample. A single multi-channel image (e.g., each channel is associated with a different marker) As another example, the MxIF images may be multiple images of the same tissue sample (e.g., Each of which may have one or more channels. For example, one or more Referring to the MxIF image 102, in some embodiments, the MxIF image 102 is a CD31, CD8 In a single image with multiple channels for each of the markers, CD68, and NaKATPase (For example, MxIF image 102 is a four-channel image). 2. There are four one-channel images, each containing a separate immunofluorescence image for each marker. As a further example, the MxIF image 102 may include one or more immunological images having multiple channels. Immunofluorescence images (e.g., two MxIF images each with two channels, and / or similar It may include

[0072] Although not shown in Figure 1, there are several methods to identify cellular structures or subcellular compartments. Any marker known in the art (e.g., membrane markers, cytoplasmic markers, nuclear markers, Various other types of markers can be used, including genes (such as markers). For example, it may be a DNA or RNA encoding a protein, or a protein. Markers include intracellular markers (e.g., intracellular proteins), membrane markers (e.g., , membrane proteins), extracellular markers (e.g., extracellular matrix proteins), or It may be a combination of two or more of them. Another example of a marker is PCK26 antibody. , DAPI (for DNA), carbonic anhydrase IX (CAIX), S6, CD3, and / or similar. Further examples of markers include genes (and the proteins encoded by such genes). proteins involved in the differentiation of ALK, BAP1, BCL2, BCL6, CAIX, CCASP3, CD10, CD106, and CD11b. , CD11c, CD138, CD14, CD16, CD163, CD1, CD1c, CD19, CD2, CD20, CD206, CD209, CD2 1, CD23, CD25, CD27, CD3, CD3D, CD31, CD33, CD34, CD35, CD38, CD39, CD4, CD43, C D44, CD45, CD49a, CD5, CD56, CD57, CD66b, CD68, CD69, CD7, CD8, CD8A, CD94, CDK1 , CDX2, Clec9a, chromogranin, collagen IV, CK7, CK20, CXCL13, DC-SIGN, Desmi EGFR, ER, ERKP, fibronectin, FOXP3, GATA3, GRB, granzyme B, H3K36TM , HER2, HLA-DR, ICOS, IFNg, IgD, IgM, IRF4, Ki67, KIR, Lumican, Lyve-1, Mamma Globin, MHCI, p53, NaKATPase, PanCK, PAX8, CNAP, PBRM1, PD1, PDL1, perlecan , PR, PTEN, RUNX3, S6, S6P, SMA, SMAa, SPARC, STAT3P, TGFb, Va7.2, vimentin, and / or other markers. The markers may be any marker that selectively binds to the marker of interest. The antibody or other labeled binding agent can be used to detect the antibody or other labeled binding agent. The agent (e.g., a labeled antibody) may be a luminescent (e.g., fluorescent), chemical, enzymatic, or other label. As a result, the tissue images described herein can be labeled with a fluorescent marker, a chemical marker, or a combination of both. The signal from various types of markers, including markers, and / or enzyme markers, is used. Thus, in some embodiments, the images described herein (e.g., For example, MxIF images can contain information from non-fluorescent as well as fluorescent markers. However, in some embodiments, the MxIF image may be The image contains information only from the fluorescent signal. In some embodiments, the fluorescent information is Information from fluorescent stains (e.g., DAPI) in addition to information from fluorescent antibodies against specific markers This may include information.

[0073] In some embodiments, immunofluorescence signals from tissues or cells are detected using fluorescently labeled antibodies. Contacting the body with tissue or cells (e.g., fixed and / or sectioned tissue or cells) Detecting the fluorescent signal (e.g., using a fluorescent microscope) and detecting one or more The present invention is obtained by determining the presence, location, and / or level of a marker. In some embodiments, the fluorescently labeled antibody is a primary antibody that binds directly to the marker of interest. In some embodiments, the fluorescently labeled antibody binds directly to the marker of interest. A secondary antibody is a non-labeled antibody that binds to an unlabeled primary antibody. For example, a secondary antibody is When raised against the antibody of the host species of the antibody, it can bind to the primary antibody. Different techniques can be used for fixing and / or sectioning the cells. Alternatively, the cells may be fixed with formaldehyde or other reagents. In some cases, the tissue may be fixed by vascular perfusion with a fixative. The tissues or cells may also be fixed by immersion in a fixative. In this method, the fixed tissue or cells are dehydrated and embedded in a material such as paraffin. However, in some embodiments, the tissue or cells are preserved in paraffin or other media. As opposed to being embedded in a material, tissue can be frozen to preserve tissue morphology. In embodiments of the present invention, tissues or cells (e.g., fixed, embedded, and / or Frozen tissues or cells can be sectioned, for example, using a microtome. In some embodiments, sectioned tissue or cells are subjected to a microscope for fluorescence microscopy. The tissue or cells may be mounted on a slide or other suitable support. (e.g., mounted sectioned tissue or cells) are incubated with one or more primary antibodies and and / or a secondary antibody (e.g., several incubations, blocking After incubation (with subsequent incubation and / or washing steps), labeled tissue or cells can be obtained. do.

[0074] The computing device 116 processes the MxIF image 102. The device 116 processes the MxIF image 102 to determine the location of cells in the tissue sample (e.g., (by segmenting images into cells) and identifying different types of cells in tissue samples. In some embodiments, the computing The device 116 identifies a plurality of populations of cells in a tissue sample, at least in part, by (a) an MxIF image 102 and and using information indicative of the arrangement of at least some of those cells to determining feature values ​​for at least some of the cells of the For at least some of the cells identified by the cell location information such as (b) determining feature values ​​for at least one of the cells using the determined feature values; and by grouping some cells into groups.

[0075] The computing device 116 may further comprise a computing device for analyzing one or more of the tissue samples using the plurality of cell populations. In some embodiments, the computing device 116 determines a characteristic of Determine information about cell types in a tissue sample. For example, using a computing device The computer 116 can determine the cell type of cells in a plurality of cell populations in a tissue sample. Examples of cell types determined by the routing device 116 are endothelial cells, epithelial cells, macrophages, and phages, T cells (e.g., CD3+ T cells, CD4+ T cells, or CD8+ T cells), malignant cells, NK The subject invention may include one or more of: pulmonary artery disease (PCD) cells, B cells, and acinar cells. The imaging device 116 may determine the cell type based on user input and / or using artificial intelligence techniques. For example, the computing device 116 may determine one or more A number of pre-trained neural networks are used to process multiple cell populations and identify subpopulations within each cell population. For example, as described herein, cell type can be determined by determining the cell group information about the cell, as well as other relevant information (e.g., cell shape / size, neighbors in the graph, The predictions are based on the number of elements, and processed by a neural network to produce the predicted As further described herein, the cell type can be determined. A neural network may, for example, be trained using a set of training data. Each set may consist of one or more cell populations and / or other relevant input data (e.g. Input data including (e.g., cell shape, mask, etc.) and the details of each cell group in the input data Specify associated output data that identifies the cell type. The computing device 116 processes the plurality of cell populations based on the user input and The cell types within the population can be determined. For example, user input can be Manual identification of visible cells, possible cell types of grouped cells, and / or similar It can include things.

[0076] In some embodiments, the computing device 116 detects one or more The percentage of a cell type or a number of cell types can be determined. The imaging device 116 detects endothelial cells, epithelial cells, macrophages, T cells, malignant cells, etc. in a tissue sample. To determine the percentage of one or more of: NK cells, B cells, and acinar cells. It is possible.

[0077] The computing device 116 may store cell location, cell type, and / or other information (e.g., For example, information on the physical parameters of cells (e.g., cell area, density, etc.) to determine a characteristic 106 of the tissue sample, which may be determined by comparing adjacent cells in the tissue sample (e.g., adjacent This includes determining information about the cell type and / or tissue composition of the cells. For example, The computing device 116 determines neighboring cell types of the cell of the cell type of interest. Such neighboring cell type information can be, for example, At least some of the (a) cluster closely together within one or more clusters (b) whether the cells of interest are mostly adjacent to each other (e.g., if the cells of interest are mostly adjacent to each other); whether the cells of interest are distributed throughout the body (e.g., whether the cells of interest are distributed among other types of cells in the tissue sample) (c) grouped together with one or more other cell types in the tissue sample Whether the cells of interest are grouped (e.g., the cells of interest are grouped with one or more other cell types), and / or other cell neighbor information may be provided. Cut.

[0078] The computing device 116 may collect statistical information (e.g., cell distribution information), spatial information ( MxIF images, such as distances between cell types, morphological information, and / or the like The image, information 104 (e.g., cell type and / or cell location), and / or information 106 (e.g., one or more further analyses of the tissue sample based on the cellular neighbors and / or cellular organization For example, the statistical information may include a characteristic 108 that is associated with at least some of the plurality of cell types. The distribution information can include information about the distribution of cells. distribution between (e.g., two cell types are distributed near each other, or in one or more regions) are mixed together within the region, separated by distances, and / or the distribution of one or more cell types in a tissue sample (e.g., high or low in cells); information about one or more regions within a tissue sample, including concentration, and / or other distribution information. It is possible to do so.

[0079] As another example, in some embodiments, spatial information is provided for at least one of a plurality of populations of cells. Both can contain information about the arrangement of some cells. For example, spatial information can be Information about the spatial organization of one or more cell types in a tissue sample (e.g., cell type information about the cellular composition of tissue samples, comparison with other cell types, and / or As another example, the spatial information may include one or more One or more regions of the tissue sample that contain a cell type (e.g., one or more regions that exceed a threshold value The information may include information about a region containing a large number of cells of a cell type.

[0080] By way of further example, in some embodiments, the morphological information includes the morphology and and / or structure (e.g., cell shape, structure, morphology, and / or size, etc.), tissue samples. Includes information regarding the form and / or structure, and / or the like.

[0081] The computing devices 112 and 116 may be laptop computers, desktop laptops, smartphones, cloud computing devices, and / or or any other computing device capable of executing the techniques described herein. The network 114 may be any computing device, including: Local Area Network (LAN), Wide Area Network (WAN), Internet, and / or the like, including wired and / or wireless network connections , can be any type of network connection. FIG. 1 illustrates Although two computing devices 112 and 116 are shown in communication, a computing device or configuration having two or more computing devices, network (for example, if you only have one computing device) Alternatively, configurations with multiple networks and / or other configurations may be used. should be understood.

[0082] FIG. 2 illustrates an MxIF image, and a method for extracting an MxIF image, according to some embodiments of the techniques described herein. FIG. 200 shows related data that may be generated by processing an image. xIF image 202 shows PCK26 antibody, CD8 T cells, CD31 antibody, CD68 antibody, DAPI (for DNA), and and / or various markers such as those described herein. The images 202 can be used to determine various information about the tissue sample (e.g., as described herein). The information can be processed (using AI-based techniques) to identify the arrangement of cells in the tissue sample. Segmentation information 204 may be included. The information may include information regarding the arrangement of stroma in the tissue sample. a stromal mask containing information about the tumor cells in the tissue sample and / or a tumor mask containing information about the tumor cells in the tissue sample One or more masks 206 that may be applied to the image to identify tissue features, such as a mask The information may also include information about cell location 208 and / or cell neighborhood. The information may include information about the analysis of cell types and cell subpopulations. Such information may further include information 210 indicating the The raw MxIF images 202 are processed to identify endothelial cells, macrophages, T cells, and malignant cells in the tissue samples. can be used to identify cellular and / or histological aspects of tissue samples, including sexual cells. do.

[0083] FIG. 3 illustrates a method for processing MxIF images of a tissue sample according to some embodiments of the techniques described herein. Exemplary components of a pipeline for characterizing cells in tissue samples by processing 3 is a diagram 300 showing an MxIF image, generally referred to herein as an MxIF image 310. As explained above, each MxIF The image may be an image of the fluorescence of one or more markers applied to the tissue sample. The pipeline performs one or more pre-processing steps on the MxIF image. A pre-processing component 320 is included. Pre-processing includes background subtraction (e.g., MxIF image subtract background noise from the image), change the image resolution, downsampling shaping, resizing, filtering, and / or other types of The pipeline may include one or more of the following image pre-processing steps: Information about the arrangement of cells within the cell (e.g., cell segmentation masks, Cell segmentation (by segmenting the image to identify the location of the The system also includes a motion component 330.

[0084] The pipeline uses cell location information from the cell segmentation component 330 and and cell typing based on the preprocessed MxIF image from the MxIF image preprocessing component 320 ( For example, to perform cell type determination and / or to separate cells in a tissue sample into multiple different groups. The pipeline also includes a cell typing component 340 that groups cells into cell segmentations. Cell morphology assessment component 3 using data from the mentation component 330 The cell morphology evaluation component 350 includes cell area, cell perimeter, cell size, and The pipeline determines parameters of the cells, such as the cell population and / or the like. Also included is a component 360, which includes the cell typing component 340 and cell morphology evaluation component 360. The characterization component 360 uses data from both the cell information about the distribution of the chromosomes, the distance between cells, and other information as described herein. One or more properties of the textile sample may be determined. In some embodiments, Distance information is the distance between two cells that does not cross through another cell or structure (or part of it). For example, the distance information may be the length of the shortest path along a portion of the tissue sample. For example, if two cells are separated by an acinus, the distance may be a measurement of the distance around the acinus (rather than the distance through the acinus).

[0085] FIG. 4A illustrates a block diagram of the components of FIG. 3 in accordance with some embodiments of the techniques described herein. FIG. 4A illustrates an example processing flow 400 of an MxIF image 300 using some of the As shown on the left, an MxIF image 310 is processed using the MxIF image pre-processing component 320. In this example, the MxIF image preprocessing component 320 preprocesses each processed M xIF images 410A, 410B, 410C, through 410N (collectively referred to herein as processed MxIF images 410 The present invention generates a processed MxIF image for each of the MxIF images 310, which is shown as a For example, the processed MxIF image 410 may include background subtraction, as described herein. The processed MxIF image may be subjected to processing such as removing noise. 410 is then provided to the cell typing component 340 and / or any other FIG. 4A shows the image processed by the MxIF image pre-processing component 320. Although each of the MxIF images 310 shown in FIG. 1 is a single MxIF image, this is for illustrative purposes and is not intended to be limiting. and only one or more of the MxIF images 310 are intended to be processed by the MxIF image pre-processing component. The received signal may be processed by the processor 320.

[0086] In some embodiments, the computing device stores the separate processed MxIF images. The image 410 is used to generate a combined image used by the cell clustering component 116. Generate a multi-channel processed MxIF image with multiple channels (e.g., each channel has a different marker) For example, as explained above, each marker image is The image is processed to perform background subtraction independently for each marker in the image. For example, each marker may be subtracted from the noise associated with that marker. The noise can be treated differently (e.g., because the noise can be different for each marker). The computing device includes a cell typing component 340 that detects the processed marker So that cell clustering can be performed using a single image containing channels for each of the In addition, or alternatively, a combined image can be generated. The typing component 340 performs cell clustering using the multiple images. It is possible.

[0087] As also shown in FIG. 4A, the cell segmentation component 330 At least some of the MxIF images 310, shown as MxIF images 310A and 310C in 42 Using the location information 42, which indicates the location of cells in the tissue sample captured by the immunofluorescence image, 0. The MxIF image 310 is pre-processed using the MxIF image pre-processing component 320. (used by the cell segmentation component 330, not shown in FIG. 4A) (to generate a processed MxIF image that is processed by the cell segmentation). The configuration information 420 may include cell segmentation data, such as a segmentation mask. is provided to the cell typing component 340 and / or utilized in any other manner. It is made possible.

[0088] It should be understood that FIG. 4A is intended to illustrate an example process flow only. One or more of the components shown in FIG. 4A may be optional. (e.g., MxIF image pre-processing 320, cell segmentation 330, etc.). 320 is shown both on the left and right of FIG. 4A, but may be used in either or both configurations. Furthermore, Figure 4A shows a cell segmentation control using two MxIF images. Although a cell segment component 330 is shown, this is for illustrative purposes only. The mentation component 330 processes all of the MxIF images and / or the MxIF image pre-processing components. Any number of MxIF images 310 may be processed by the same number of MxIF images processed by the component 320. It can be used.

[0089] Each of the MxIF images 300 may be imaged using a different marker (e.g., tissue may be imaged using a different antibody marker). by staining with Each immunofluorescence image is captured when the marker is administered. The result is the MxIF staining process. The method may be periodic and may involve staining the tissue sample for imaging, preparing the tissue for subsequent staining, and The present inventors have found that the markers are not stained by the tissue samples. You understand that localized tissue damage can occur whenever fluid is washed away from the skin. For example, Some staining techniques, such as cyclic immunofluorescence (CyCIF) staining, (e.g., the use of hydrogen peroxide) Depending on the application, it can be destructive and cause some localized tissue damage. As a result, individual cells and / or cell groups are washed away with respect to their original location in the sample, and / or Such damage may result in injury or damage to the equipment. Through the image processing pipeline, starting from the step mentioned above and going through the steps that follow, This may give erroneous information about damaged cells that are being treated (e.g., further damaged cells As a result, in some embodiments, these techniques Check for tissue degradation at one or more steps in the xIF imaging process This may include:

[0090] FIG. 4B illustrates a tissue degradation check component according to some embodiments of the techniques described herein. FIG. 4B illustrates the exemplary process flow 400 of FIG. 4A, which also includes a component 430. To avoid affecting the tissue analysis, the tissue degradation check component 430 checks for damage The cells can be detected and excluded from the tissue analysis. In this study, cell markers in immunofluorescence images from different stages were compared, allowing the identification of tissue structures. Changes in structure can be detected (e.g., changes in cell nuclei, cell boundaries, etc. over time). For example, by comparing the expression of nuclear markers (e.g., DAPI) from different stages. Immunofluorescence images were compared to identify changes in cell nuclear arrangement in tissue samples over time. It can be monitored.

[0091] In some embodiments, the tissue degradation check component 430 is a trained neural network. Using neural networks to compare markers from different immunofluorescence imaging stages Neural network models can be used, for example, as described herein in its entirety. Kaiming He et al., available at https: / / arxiv.org / abs / 1512.03385, which is incorporated herein by reference. "Deep Residual Learning for Image Recognition", arXiv:1512.03385v1 (December 2015) The implementation of such a ResNets model can be based on the ResNets model described in The state can include a variable number of parameters. Such a model has at least 500,000 Many parameters, such as 1,000,000 parameters, or even more. In some embodiments, such a model can include several thousand Million parameters (for example, 10 million parameters, 25 million parameters, 50 million parameters, It can contain one or more parameters. For example, The number of parameters can range from 11 million to 55 million parameters, depending on the implementation. In some embodiments, the parameters are at least 100 million parameters (e.g. For example, at least 100 million parameters, between 1 million and 100 million parameters, or hundreds of millions parameters, at least 1 billion parameters, and / or any suitable number or range As another example, a neural network model can include parameters within a range: , see, e.g., https: / / arxiv.org / abs / 1905.1, which is incorporated herein by reference in its entirety. Mingxing Tan and Quoc Le, "EfficientNet: Rethinking Model Sc,, available at: "Aligning for Convolutional Neural Networks," arXiv:1905.11946v5 (September 2020) It can be implemented based on the EfficientNet model. The form may be 500,000 parameters, at least 1 million parameters, or several million parameters ( 5 million parameters), tens of millions of parameters (e.g., 10 million parameters), data, 25 million parameters, 50 million parameters, or more Any number of parameters may be included. For example, the number of parameters may be 5 million. The number of parameters may range from 0 to 60 million. ,The parameters are at least 100 million parameters (e.g., at least 100 million parameters). data, between 1 million and 100 million parameters), hundreds of millions of parameters, at least 1 billion parameters parameters, and / or parameters within any suitable number or range. do.

[0092] In some embodiments, the trained neural network is A set of immunofluorescence marker images (e.g., DAPI marker images) of the same tissue sample taken A set of immunofluorescence marker images can be taken as input. Multiple immunofluorescence imaging loops, including the base loop and test loop, were In some embodiments, the trained neural network is ,It can take as input a portion of an immunofluorescence marker image. For example, some In this embodiment, the set of immunofluorescence marker images is processed in smaller portions. The immunofluorescence images can be processed using a sliding window across them to identify the A window can be a specific width (e.g. 128 pixels, 256 pixels, etc.) and height (e.g. pixels, 256 pixels, etc.), and the number of channels representing the number of markers (for example, 2, 3, 4 The sliding window can be a preconfigured pattern. to move across the immunofluorescence image and process the complete content of the immunofluorescence image. For example, the sliding window starts from the top left corner of the first line of immunofluorescence images. Then move horizontally across the immunofluorescence image until you reach the right side of the immunofluorescence image. Go down to the second line of the fluorescent image and work again from left to right until you reach the bottom right of the immunofluorescent image. This can be repeated with . Thus, the trained neural network is The input marker image (or its (part of) may be normalized, such as by using z-normalization and / or any other normalization technique, as appropriate. and can be normalized.

[0093] In some embodiments, the output of the neural network comprises at least A value indicating whether the dough is associated with damaged tissue (e.g., a binary value and / or The probability that the window is associated with damaged tissue or not can be expressed as In some embodiments, the output may include a value for each of a plurality of classes. At least one of the number classes is associated with an area showing signs of tissue damage. For example, if you have three classes (e.g., OK, EMPTY, and DAMAGED regions of a tissue sample), In this case, the output is the probability for each corresponding class of each window of the immunofluorescence image, In some embodiments, for each window, the maximum The class with the highest probability is selected for that window. A final class or classification (e.g., damaged tissue or not) is selected. The final output of the check process combines the classes determined for each window, as shown in Figure 4C. Patch mass reflecting various classes of relatedness, as further explained in As a result, in some embodiments, the patch mask can be It indicates whether the tissue is damaged with a granularity given the window size.

[0094] In some embodiments, the annotated dataset (e.g., one or more pathologies) The dataset (annotated by scholars) was used to train a neural network. This allows for classification of portions of the tissue sample into a set of classes. The dataset used to train the neural network is a set of 2500 annotated DA PI marker images were included (although the same marker may have been included in multiple loops of the imaging process). Any marker could potentially be used as long as it is used in the group or step. (It should be understood that the images can be used to train neural networks. A set of classes is used to annotate the data, which allows the classification of new data into classes. (e.g., to determine the likelihood that new data corresponds to each of the classes) As described herein, in some embodiments, The neural network then applies new data to, for example, empty parts of tissue (e.g., the empty spaces of cells). The first class is for the damaged parts, the second class is for the unchanged parts, and the third class is for the damaged parts. a third class for parts that are watched over a period of time, and / or The images can be trained to classify similar ones. The images can be annotated using a line-by-line approach, whereby the tissue sample is identified Includes annotations for the windowed areas traversed (e.g., the image is essentially a grid (The data is divided into blocks of the grid, and each block of the grid is annotated with a class.) To annotate the dataset, in one example, the Universal Data Tool (UDT) is used to annotate each image. The windowed sub-portions of the images were classified into three classes: OK, EMPTY, and DAMAGED. In some embodiments, the input image is transformed using a number of transformations, such as an affine transformation and / or other transformations. The training process may be augmented by using one or more transformations as appropriate. This can be a process.

[0095] In some embodiments, the tissue degradation check component 430 detects the loss of a tissue sample. A patch mask of the tissue sample representing the damaged and / or undamaged areas can be generated. For example, a patch mask is a mask that shows the parts of a tissue sample that are classified by a neural network. The parts that represent the different classes processed by the neural network (e.g., The patch mask may include a mask that covers the damaged area of ​​the tissue sample. Some and / or all of the classified parts may be removed to prevent further analysis of the parts. For example, a patch mask can be used to filter out Segmentation contours (e.g., cell segmentation module) are generated from the classified regions. 330)。

[0096] FIG. 4C illustrates two nuclei from the same tissue sample, according to some embodiments of the techniques described herein. A pair of markers determined by comparing marker images 440A and 440B (collectively stained portion 440) FIG. 4 is a diagram showing a patch mask 444 generated based on the texture degradation information. Tissue degradation, such as the process performed by tissue degradation check component 430 of FIG. 4B In this example, each of the two nuclear marker images 440 Use the DAPI marker during different steps or cycles of the MxIF imaging process The window of the marker image 440A is generated by imaging the tissue sample with the marker. The window 442A of the marker image 440B and the window 442B of the marker image 440 are windowed portions of the marker image 440. By comparing the time series, tissue disturbance across MxIF imaging steps can be checked. A window is moved across the marker image 440 to iteratively process the marker image 440. The windowed portion of the marker image 440 is depicted as described herein. A trained neural network classifies patches of tissue samples into three classes (empty, damaged, and undamaged). The classes with the highest probability are compared using a neural network and assigned to each window. The resulting patch mask 444 is mostly or completely empty. Section 444A to show the windowed portion of the tissue sample classified as Assemble these parts sufficiently intact to be OK for further analysis. Section 444B for windowed sections of tissue samples containing tissue, and Section 444C for There is a sufficient number of injuries so that section 444C should be excluded from further analysis. Cells (e.g., one or more damaged cells, a number of damaged cells exceeding a predetermined threshold, etc.) Including section 444C.

[0097] FIG. 4D illustrates a method for filtering an area of ​​a tissue sample to be processed, according to some embodiments. 4C to identify damaged cells. This may be used to filter out section 444C, which includes As shown, patch mask 444 is used to filter out section 444C. A segmentation mask 446 is applied to the result. The results are then processed using cell typing as described herein. In the image processing pipeline, including segmentation and further tissue analysis A filtered segmentation mask that may be used instead of the segmentation mask 446 448. By using the filtered segmentation mask 448 , an image processing pipeline (e.g., cell typing component 340 in FIG. 3, cell morphology The evaluation component 350 and / or the characterization component 360 may include cell typing. and / or not process section 444C that has been removed from the mask for characterization. .

[0098] The tissue degradation check module 430 is a cell segmentation module 3 in FIG. 30, but it will be understood that this is for illustrative purposes only. For example, the tissue degradation check should be performed in the cell segmentation module 330. As part of the MxIF image pre-processing module 320, prior to cell segmentation (e.g., It may be performed as part of the process, and / or at any other point in the process. In one embodiment, the tissue degradation check is performed by performing background subtraction on the MxIF images. Additionally, the tissue deterioration check module 430 is optional and may be performed after the tissue deterioration check module 430 is performed. Therefore, tissue degradation checks do not need to be performed as part of the image processing pipeline. It should be understood that

[0099] The cell typing component 340 uses the processed MxIF image 410 and the alignment information 420 to and / or (e.g., based on cells exhibiting similar feature values) The cells are grouped into populations, which may be one or more of the populations of tissue samples. The present invention may be used by the characterization component 360 to determine the characteristics of the Determining at least one characteristic of tissue according to some embodiments of the techniques described herein. To group cells in a tissue sample for analysis, MxIF analysis of the tissue sample was performed based on cell arrangement data. 5 is a flowchart illustrating an exemplary computerized process 500 for processing an image. The computerized process 500 may be implemented, for example, by the computing device 116 of FIG. The computing device 5100 of FIG. 51 may be implemented as a 3 to 4B. The present invention may be configured to:

[0100] In step 502, the computing device receives at least one multiplex of the same tissue sample. An immunofluorescence image (e.g., MxIF image 310 described in conjunction with FIG. 3) is acquired. The final multiplexed immunofluorescence image was obtained from a single multi-channel image of the same tissue sample, as described in Single immunofluorescence image and / or multiple immunofluorescence images of the same tissue sample (each of one or more In some embodiments, the computing The imaging device acquires multiplexed immunofluorescence images by acquiring previously generated images. (e.g., stored using one or more computer-readable storage devices) access one or more images stored in the at least one communication network; For example, MxIF images can be captured using a microscope. 500. The information may be stored for later access and accessed during activity 502. Activity 502 may include capturing an image. The tissue sample may be taken from a patient (e.g., a patient with cancer). Previously obtained immunoglobulins from patients with, suspected of, or at risk of having This is an in vitro sample.

[0101] We have demonstrated that some of the processes described herein remove noise from MxIF images. It is understood that it may be desirable to process MxIF images, for example by , since noise cannot affect the system's ability to determine the location of cells in a tissue sample. The system performed cell segmentation ( For example, cell segmentation 330 in FIG. 3) may be performed. However, noise can be a factor in cell clustering (e.g., cell typing 34 in FIG. 3). 0) can cause problems, for example, low signal-to-noise The level may affect cell clustering and therefore the results of the raw MxIF images and and / or the data generated by the cell segmentation process may be used to identify cell clusters. Rings may not be enough.

[0102] In some embodiments, these techniques include at least one of immunofluorescence imaging. to generate a corresponding processed image. can include performing background subtraction. , for example, at least some noise can be removed. Noise in the image caused by the microscope that captured the image, caused by surrounding cells Noise caused by aspects of the tissue, such as noise (e.g., fluorescence noise), In some embodiments, the noise may include noise caused by the noise generated by the noise source. In the present invention, the image can be processed at different regions within the image. For example, some Areas may show more noise than others, so it is important to remove noise from specific areas in the image. Additionally or alternatively, in some embodiments, The images may be processed image by image and / or channel by channel. For example, each marker channel may be Noise can also be removed on a per-marker basis, since different channels may exhibit different noise.

[0103] In some embodiments, these techniques remove at least some noise. This involves applying a trained neural network model to each of the immunofluorescence images to FIG. 17 shows a live immune response according to some embodiments of the technology described herein. Noise to subtract noise from fluorescent images (e.g., for background subtraction) Convolution to implement a trained neural network capable of predicting the subtraction threshold A diagram of a convolutional neural network architecture 1700. The architecture 1700 may be implemented using the MxIF image pre-processing 120 of FIG. 1 and / or the MxIF image pre-processing 120 of FIG. 3-FIG. 4B. Implemented and / or performed as part of MxIF image pre-processing, such as part of image pre-processing 320. The convolutional neural network architecture 1700 may be implemented in accordance with other implementations described herein. It may be implemented and / or performed as part of a process or sub-process. For example, the architecture 1700 may include, as part of steps 502, 504, and / or 506 of FIG. As another example, the architecture 1700 may be used to filter out noise from the signal in FIG. As part of steps 612, 614, and / or 616, the method may be used to remove noise. As a further example, a trained neural network implemented using architecture 1700 may be The network may be configured to detect noise as part of steps 704, 706, and / or 708 of FIG. 7A. can be used to remove

[0104] Neural network implemented according to the Convolutional Network Architecture 1700 is trained to predict a threshold image 1706 based on a raw input image 1708. As shown, the convolutional network architecture 1700 includes convolutional layers 1702A, 170 2B and 1702C first extract (along the downsampling path) a sequence of raw image 1708 data. Then, the downsampling layer 17 is applied to the sequence of lower resolution versions. The raw images 1704A, 1704B and 1704C (along the upsampling path) are shown as It has a "U" structure that is applied to a sequence of eight successively higher resolution versions of the data. In some embodiments, the convolutional network architecture 1700 may include Although not shown, the resolution of the data can be improved using one or more pooling layers (e.g. , along the downsampling path) and one or more corresponding unpooling Such a niche can be augmented (e.g., along an upsampling path) using layers. Neural network implementations can include a variety of numbers of parameters. Such models have at least 500,000 parameters, 1 million parameters, 2 million parameters, Many parameters, such as 5 million parameters or more. In some embodiments, such models can be constructed using tens of millions of parameters. For example, the number of parameters may be at least 10 million parameters, 20 million parameters, 25 million parameters, 50 million parameters In some embodiments, the parameters may include at least At least 100 million parameters (for example, at least 100 million parameters, between 1 million and 100 million) parameters), hundreds of millions of parameters, at least a billion parameters, and / or any The parameters may include any suitable number or range.

[0105] In some embodiments, the model may be pre-trained using the denoising information. In some embodiments, the model is adapted to use a threshold to remove noise. In such an embodiment, the model can be trained to determine an appropriate threshold for noise removal. The model may be trained using threshold data, such as a thresholded image representing values. Immunofluorescence images as input images so that we can learn to generate thresholded images from raw images and the corresponding thresholded image (with noise removed) as the output image. For example, with further reference to FIG. 17, the training data may include raw images 1708 and corresponding thresholds. The image may include a filtered image 1706, which when compared have a threshold difference 1710. In this embodiment, thresholding is performed across the immunofluorescence image (e.g., using a Gaussian smoothing step). In some embodiments, the smoothing step may be performed globally (after a smoothing step such as a filter). These techniques involve splitting immunofluorescence images into sub-images for noise reduction. For example, the intensity of the markers may vary across the immunofluorescence image, and therefore The threshold used for one part of the image is different from the threshold used for another part of the image. These techniques can vary, for example, to convert immunofluorescence images to 256x256, 512x512, 256x5 12, and / or into a set of sub-images of equal size, such as similar size. It can include.

[0106] With further reference to FIG. 5A, at step 504, the computing device Information indicating the arrangement of cells in the fluorescent image (e.g., the cells described in conjunction with Figures 4A to 4B) The information indicating the arrangement of the cells is obtained by obtaining some of the cells in the immunofluorescence image. and / or data indicative of all of the locations. In addition, cell location information may include cell boundary information, information from which cell boundary information can be derived (e.g., immune Fluorescence images of, for example, one or more markers expressed by the cell membrane and / or nucleus In some embodiments, the expression level of the gene may include a mask. In some cases, the cellular localization for some and / or all of these cells may be determined by immunofluorescence. For example, the immunofluorescence images may all be of the same tissue sample. In some embodiments, the cell arrangement may be the same in each image. In this case, cell positioning information can be specified using a cell segmentation mask, This is applied to one or more of the immunofluorescence images, thereby determining the concentration of the cells in the immunofluorescence image. The arrangement can be identified.

[0107] In some embodiments, the mask is a binary mask and / or a multi-value mask. The binary mask may, for example, be a representation of one or more cells in the imaged tissue sample. , texture, and / or indicating a binary value (e.g., presence or absence) for a pixel The multi-value mask can be, for example, a representation of pixels, fine lines, or lines in the imaged tissue sample. A range of values ​​for cells, and / or tissue structures can be shown. For example, multi-value mass The markers are the partial presence of tissue components (e.g., components that are completely present or absent in addition to components that are absent). in addition to the above, multiple aspects of the tissue (e.g., the presence of different cells, tissue structures, etc.), and / or It may also be used to indicate other non-binary information. The mask can be created based on cell boundary information obtained from the immunofluorescence images. In an embodiment, the binary cell boundary mask is a mask that represents the presence of detected cell boundaries in a tissue sample (e.g., for example, through white pixels) or absence (for example, through black pixels or other non-white It indicates either the color (through the pixel).

[0108] In some embodiments, the computing device accesses the cell location information. For example, the cell location information may be generated by a separate computing device. and to a computing device (e.g., over a wired and / or wireless network). A memory accessible to a computing device, the memory being transmitted via a network communications link. The information may be stored in a library and / or similarly manipulated.

[0109] In some embodiments, the cell location information is generated manually and is stored in a manner consistent with the techniques described herein. FIG. 8 illustrates some of the techniques described herein. 8 shows an exemplary MxIF image 800 and manual cell placement / segmentation for an MxIF image according to an embodiment of the present invention. 8 shows an example of station data 802.

[0110] In some embodiments, the computing device comprises at least one multiple Immunofluorescence images are used to generate cell location information. For example, referring to Figures 3 to 4B, The computing device may include one of the cell segmentation components 330 or The system may be configured to perform a number of aspects to generate cell location information. In the method, the computing device performs one or more operations to generate cell location information. A set of channels for multiplexed immunofluorescence images is selected, and each selected channel represents the cell structure information. For example, selected channels include marker data that indicates cell membrane, cell type, cell This may include data on nuclei, vesicles, and / or the like, which may generate cell location information. For example, cell nuclei may be stained with DAPI, H3K36TM, and / or other markers. In another example, cell surface markers can be CD20, CD3, CD4 , CD8, and / or the like. As a further example, cytoplasmic markers In some embodiments, the stimulator may include S6 and / or the like. A number of markers are used to generate cell location information. For example, The immunofluorescence markers used to generate the staining were DAPI as a cell nucleus marker, cell membrane Using NaKATPase as a marker, and S6 as a cytoplasmic marker of cells It can include.

[0111] In some embodiments, the computing device uses machine learning techniques. For example, the computing device may generate cell location information by performing a convolutional neural network. A neural network model is applied to one or more of the immunofluorescence image channels to estimate cell alignment. In some embodiments, the computing device may generate positional data. The device selects channels with cell structure information and applies a convolutional neural network model The model is applied to a selected subset of the immunofluorescence images to generate cell location information.

[0112] In some embodiments, the convolutional neural network model is the U-Net architecture. A neural network having a "U" shape as described in conjunction with FIG. 17, such as the architecture This may include a network model.

[0113] In some embodiments, the neural network architecture is a Mask R-CN Includes region-based convolutional neural network (R-CNN) architectures such as N An example of Mask R-CNN can be found at https: / / www.maskr.com / , which is incorporated by reference in its entirety. Kaiming He et al., "Mask R-CNN," arXiv: Such a model is described in 1703.06870 (January 2018). For example, such a model may contain at least 500,000 parameters, At least 1 million parameters, multiple million parameters (e.g., at least 1 million parameters, 2 million parameters, 3 million parameters, etc.), and / or tens of millions parameters (for example, at least 10 million parameters, 25 million parameters, 1 5 million parameters). For example, such a model could contain 40 million to 45 million parameters (e.g., 40 million parameters, 42 million parameters, and / or 45 million parameters). The data must contain at least 100 million parameters (e.g., at least 100 million parameters, 1 million between 100 and 100 million parameters), hundreds of millions of parameters, at least 1 billion parameters, and / or any suitable number or range of parameters. The input to the delta is multiple channels containing a nuclear marker and one or more membrane markers. For example, the input can include a nuclear marker for the first channel (e.g., DA PI), and a membrane marker present on most cells for the second channel (e.g., CD45 or or NaKATPase), and an additional membrane marker with sufficient staining for the third channel ( For example, CD3, CD20, CD19, CD163, CD11c, CD11b, CD56, CD138, etc.) In some embodiments, the input channels may be normalized (e.g., For example, in the range 0 to 1).

[0114] In some embodiments, an input image (or multiple input images) are The image is divided into rectangles (for example, sizes 128x128 pixels, 256x256 pixels, etc.) and the window In such an embodiment, the network output is In some embodiments, the output may be an array of mask proposals for each window. Additionally or alternatively, the output may be a binary mask for a given pixel. An image with a value (e.g., a value in the range 0 to 1) that represents the probability that a cell is part of a mask of cells For such an embodiment, the pixels of the output image may be a set of The mask values ​​may be thresholded and / or selected using various techniques to determine the final mask value. For example, if each pixel contains two probabilities that add up to 1, then the output pixel The pixels are then thresholded using a value of 0.5 to obtain the final pixels for the binary mask. It is possible.

[0115] In some embodiments, the prediction windows include overlapping data. For example, windows may share data and / or cells may share the edges of windows. Such redundancy may occur if the This can be avoided by treating only cells from some parts, rather than all parts. For example, the center, top corner, and right corner pixels of each image are processed for each window. can only be the last window from the right or bottom (for example, the current window is the last window from the right or bottom). (The bottom and right parts of the image are processed only if the The cell segments can be aggregated into a final output mask (e.g., integer values ​​are the individual cell instances). (representing a chest of drawers).

[0116] In some embodiments, the architecture (e.g., U-Net, R-CNN, etc.) is Encoder heads such as ResNets models (e.g., ResNet-50) described in the specification (e.g., , as the first layer or layers of the model. The segmentation network is used to generate similar and / or similar or may be produced by the same encoder head.

[0117] In some embodiments, the convolutional neural network model is A set of training immunofluorescence images as the training images, and associated cell segmentation images as the output images. The model is trained using mentation images. images, about 3,000 images, about 4,000 images, about 5,000 images, about 6,000 images, about 10,000 images, and / or or a similar number of images, various training set sizes are used to test the neural network model. In some embodiments, the training set size can be used to train the Depending on the input image size, the set size can range from about 4,000 to 5,000 images. In some embodiments, the image squares are obtained by subtracting the squares from the original complete image (e.g., Randomly) sampled (e.g., therefore fewer training images are needed) The training immunofluorescence images may be of several tissue samples. The cell segmentation image associated with the optical image provides information about the arrangement of cells in the tissue sample. In some embodiments, the neural network model may include The input images are trained using multi-channel and / or single-channel images. It can be refined.

[0118] The training images contain multiple markers, one for each channel. For example, a three-channel image (or multiple images) may be used as described herein. The first channel is a nuclear marker and the second channel is a marker for the tissue type of interest. A membrane marker expressed by most cells, whereas the third channel is an additional membrane marker. For example, a three-channel image was created using DAPI, NaKATPase, and S6 markers. The corresponding output images used for training can be created using manually generated samples. A sample output image (e.g., a cell segmentation mask with cell contour outlines) In some embodiments, the training set (such as For example, the input images and / or associated output images are For example, the pre-processing step may be to filter out border cells (e.g., 3 pixels of the cell boundary) It may include cells that have a number of intersecting pixels greater than a threshold, such as The training output images can be run to detect the sigma-based feature (Figure 9-1 and Figure 9-2). According to some embodiments of the technology described herein, a novel method for identifying cell location information is provided. The figures show examples of images that can be used to train a neural network. Images 902, 904, and 90 Each set of 6 and 908 contains a synthetic immunofluorescence image, a corresponding neural network of cell arrangements, A training image of the cell arrangement used to train the neural network. , and images of cell boundary contacts (e.g., where cells contact other cells).

[0119] The neural network model is trained using various backpropagation algorithms. In some embodiments, the convolutional neural network has a learning rate of 0.00 It is trained using the backpropagation algorithm with the ADAM optimizer, where σ is the mean squared mean. At each training step, the neural network model computes the For example, in some embodiments, neural networks are trained to generate cell location information. The network model is trained using three-channel input images and associated A two-channel output image (e.g., a segmentation map) that exactly matches the training output image and the other channel with the border cell contact map) Neural network models can be trained using a variety of loss functions. For example, the model is performing a pixel classification task, so a categorical cross-effect An entropy loss function may be used.

[0120] FIG. 10 shows MxIF images obtained of a tumor according to some embodiments of the technology described herein. A trained convolutional neural network is used to process the image and generate cell segmentation data 1002. FIG. 10 pictorially illustrates an example of using the neural network model 1000. In the example of FIG. The MxIF image 1010 includes a first marker image 1012 and a second marker image 1014. The computing device uses the trained neural network model 1000 to The first marker image 1012 and the second marker image 1014 are processed to obtain cell location / segmentation. The application information 1002 is generated.

[0121] As explained in conjunction with Figure 17, the convolutional neural network model uses convolutions. The congestion layers are successively lower resolution versions of the data along the downsampling path 1000A. is applied to the upsampling path 1000B, and then the data is successively moved up along the upsampling path 1000B. The "U" structure is applied to the different resolution versions. As shown by the red arrows, Then, the resolution of the data is downsampled using one or more pooling layers. The resolution of the data can be reduced by 1000A, as shown by the green arrow. Along the upsampling path 1000B using one or more corresponding pooling layers It can be raised.

[0122] FIG. 11 shows a processed immunofluorescence image according to some embodiments of the techniques described herein. A trained neural network is used to generate cell location / segmentation data 1102. FIG. 11 pictorially illustrates another exemplary use of the MxIF image 1100. 10 includes a DAPI marker image 1112, and a NaKATPase marker image 1114, where DAPI is a fluorescent DNA DAPI, NaKATPase, and / or other markers are not included. It should be understood that other markers may be used. For example, cytoplasmic Marker S6, membrane marker PCK26, carbonic anhydrase IX (CAIX), CD3, and / or similar The computing device may include a trained neural network. 1100 was used to process the DAPI marker image 1112 and the NaKATPase marker image 1114. Cell placement / segmentation information 1102 is generated.

[0123] 12-16 show immunofluorescence images and and associated cell positioning information (e.g., in this example, the cell segmentation matrix). The cell location data is accessed by the system and / or or may be generated by a system such as the cell segmentation module 330 of FIG. FIG. 12 illustrates an MxIF image 1200 and an MxIF image 1210 in accordance with some embodiments of the techniques described herein. FIG. 13 shows cell segmentation data 1202 generated based on an image 1200. 1300 and 1301, according to some embodiments of the techniques described herein. FIG. 14 shows cell segmentation data 1302 generated based on the 4 shows an exemplary MxIF image of kidney tissue obtained according to some embodiments of the techniques described in FIG. 1 shows exemplary cell segmentation data. The orientation data 1402 is determined based on the MxIF images 1404 and 1406, and is an exemplary cytocenter. Segmentation data 1408 was determined based on MxIF images 1410 and 1412. and 1410 contain DAPI, CAIX, PCK26, and ki67 protein markers. and 1412 contain the CD31, CD8, CD68, and NaKATPase markers.

[0124] FIG. 15 illustrates a cross-sectional view of a clear cell renal cell carcinoma (CCRCC) according to some embodiments of the technology described herein. FIG. 16 shows an MxIF image 1500 and corresponding cell segmentation data 1502. 16, a MxIF image 1600 of a CCRCC and corresponding 16 shows cell segmentation data 1602.

[0125] With further reference to FIG. 5A, the computing device may be configured to perform, at least in part, steps Steps 506 to 508 are performed to identify groups of cells in a tissue sample. In step 6, the computing device receives the immunofluorescence images and the Using information indicative of the location of at least some of the cells, In some embodiments, the techniques determine feature values ​​that correspond to the Determine feature values ​​for most of the cells and / or all of the cells in the immunofluorescence image This may include:

[0126] In some embodiments, the techniques include detecting a signal in at least one of the immunofluorescence images. a pixel value associated with the location of the cell; As further described herein, Then, the feature value is a multiple immunity to the configuration of at least some of the cells. The value of a pixel in a fluorescent image (e.g., a pixel at or near a cell location) For example, multiple immunofluorescence images may include and / or be determined based on multiple When a single channel image is included, these techniques can be used to measure the number of cells in a single channel image. The pixel values ​​for each of these locations (e.g., different images depending on how the image was acquired) (the same arrangement and / or different images may be used across As another example, if an image contains one or more multi-channel images, The arrangement in the channel image may be the same for each channel. In this case, the feature value includes one or more values ​​derived from the values ​​of the pixels in the cell. For example, the feature values ​​can include contributions determined for each of the channels of an immunofluorescence image. The contribution can be, for example, how much a pixel associated with a cell contributes to that channel. For example, the contribution can be 0%. For each channel ranging from 0% (e.g., no contribution) to 100% (e.g., full contribution), The contribution can be determined by a representative value across multiple pixels in the cell location, Average value across pixels in the cell, percentage of positive pixels in the cell and percentage of negative pixels in the cell and / or the like. This can be determined according to a variety of techniques.

[0127] In some embodiments, these techniques focus on the location of cells within an immunofluorescence image. The pixel values ​​of immunofluorescence at or near the cell are used to determine a feature value for the cell. FIG. 5B shows a cell location data set according to some embodiments of the techniques described herein. 6 shows an example of feature values ​​600 for associated cell configurations of data 602. For example, the feature may be determined by the cell typing component 340 of FIG. 3. In the example of FIG. The value 600 contains the percentages for each channel 1, channel 2 through channel N, where N is the number of cells. An integer representing the number of channels associated with the arrangement. Each channel can be, for example, 1 One or more immunofluorescent markers, one or more immunofluorescent images, and / or similar Feature value 600 may be associated with channel 1 having a 25% contribution (e.g. For example, channel 1 has a 40% contribution towards the cell configuration, channel N has a 10% contribution towards the cell configuration, and channel B has a 10% contribution towards the cell configuration. It is shown to have an 85% contribution towards cell alignment.

[0128] Still referring to FIG. 5A, in step 508, the computing device The determined feature values ​​are used to group the cells into groups. Processing MxIF images of tissue samples based on cell location data according to some embodiments of the technique A computerized process for clustering cells of a tissue sample into a plurality of cell groups according to the present invention. The computerized process 700 is illustrated, for example, in conjunction with FIG. For example, the method may be performed by the computing device 116 described in FIG. The computing device 5100 of FIG. 3-FIG. 4B includes the cell typing component 340. The computer may be configured to perform one or more aspects described in conjunction with the present invention. The data processing 700 may be used as part of other processes and sub-processes described herein. For example, the computerized process 700 may be implemented to group cells into multiple cell populations. This may be performed as part of step 508 in FIG. 5A to group the

[0129] In step 702, the computing device performs one or more In step 704, the computing device In step 706, the computing device selects the immunofluorescence image. Identifying a set of pixels at or near the location of the selected cell in the light image. For example, these techniques involve the detection of cells that are at least partially within the cell boundary (e.g., This may include identifying a set of pixels that are within the image (either partially or completely).

[0130] In step 708, the computing device performs a pixel calculation based on the determined set of pixels. In some embodiments, these techniques use pictorial The method may include determining a feature value based on pixel values ​​of the set of cells. , a representative value over a set of pixels, or one or more criteria (e.g., the presence of markers, the fluorescence value associated with each marker higher than the threshold, etc. ) In some embodiments, the characteristic values ​​may include values ​​as described herein. Thus, one or more immunofluorescent markers and / or one or more immunofluorescent The contribution of each cell in the image may be represented. In some embodiments, The computing device generates a feature based on how the pixel values ​​contribute to the cells. For example, the computing device may determine a pixel value. The pixel values ​​of the set are determined to be within a certain percentage of the cell array (e.g., 3 0%, 40% of the cell configurations, etc.) and / or It can be determined that the fluorescence present is above a certain threshold.

[0131] In step 710, the computing device performs an additional For example, a computing device is a specific number of immunofluorescence markers (e.g., one marker, two markers, marker all of the channels, etc.), a certain number of immunofluorescence images (e.g., having one or more channels 1 image, 2 images, all of the images, etc.) and / or the like As another example, the computing device may be configured to determine a characteristic value. Immunofluorescent markers and / or images associated with some feature (e.g., cells Immunofluorescence markers and / or staining associated with some markers showing structure The computer may be configured to determine feature values ​​for each of a particular set of images (e.g., images). The imaging device may then analyze one or more additional immunofluorescence images in step 710. If so, the computing device returns to step 704 to Select the immunofluorescence image.

[0132] The computing device performs an analysis on the selected cells in step 710. If the computing device determines that there are no further immunofluorescence images to be processed, the computing device Proceed to step 712 to determine whether there are one or more cell configurations for which feature values ​​should be determined. For example, the computing device may determine a particular set of cells (e.g., a certain number of cells, cells in one or more arrangements, and / or the like) and / or The computing device may be configured to determine feature values ​​for all of the cells. The method determines, in step 712, feature values ​​for one or more additional cells. If so, the computing device returns to step 702 and selects another cell. Select.

[0133] If the computing device determines that there are no more cells to analyze, The computing device may proceed to step 714 and perform cell clustering (e.g., For example, as part of cell typing 340, the cells may be classified into one or more subgroups based on the determined feature values. In some embodiments, these techniques use feature values The analysis can be used to determine relationships between cells (e.g., similar marker expression, known Marker expression consistent with published cell typing data, cells are of the same type, and and / or analysis of the probability that each cell in the cell population has a similar characteristic. The cells are grouped based on the relationships determined to have characteristic values ​​that indicate the relationships between the cells within the cell group. This includes grouping the

[0134] These techniques use one or more Multiple clustering algorithms (e.g., unsupervised clustering algorithms) A clustering algorithm that can be used to group cells can be applied. Some examples of rhythms are hierarchical clustering, density-based clustering, k-mean s-clustering, and / or any other suitable unsupervised clustering algorithm , Self-organizing map clustering algorithm, Minimum spanning tree clustering algorithm algorithms, and / or the like, and aspects of the technology described herein may be Without being limited thereto, in some embodiments, these techniques include self-assembling matrix The data can be analyzed using the FlowSOM algorithm to analyze cell clusters. Rastering can be performed.

[0135] In step 510, the computing device generates a tissue using the plurality of cell populations. and determining at least one characteristic of the sample. In some embodiments, these techniques include and generating a report indicative of the at least one characteristic. , information about a plurality of groups and / or at least one MxIF image as described herein. The report may also include other information, such as any other information determined regarding For example, reports may be provided to users via a graphical user interface. (for example, through a web-based or device-run application program) You may view the report in a program, download it as an electronic file (e.g., a PDF file or any to transmit the report to the User (in any preferred format) and / or It may be provided by any other sufficient technique for providing to a user.

[0136] In some embodiments, these techniques involve the use of immunofluorescence images obtained in step 502. The image is used in conjunction with the cell grouping information obtained in step 504 to identify a group of cells in the tissue sample. The method includes determining one or more characteristics of the tissue sample. The at least one characteristic of the tissue sample is The cellular composition of the material (e.g., cell type, cell morphology, etc.) and / or the tissue composition of the tissue sample ( For example, cellular localization, multicellular structural localization, etc., may be characterized. In addition, the one or more characteristics may, for example, be characteristic of cell types (e.g., different populations) in a tissue sample. In some embodiments, the marker may include a marker that is associated with a cell type. The computing device then identifies the cell type of individual cells in the tissue sample. In some embodiments, the computing device performs a step of detecting a small number of individual cells in a tissue sample. Identify at least a threshold percentage of cell types. For example, the threshold percentage may be: At least 25%, 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, It could be 95%, 99%, etc.

[0137] As another example, the one or more characteristics may be a distribution of one or more cell types in a tissue sample. (e.g., distribution among cells of a group), distribution among different cell types (e.g., distribution among different groups of cells , distribution among, and / or similar. In another example, the computing device can determine which types of cells are different from other types of cells. Cell neighbor information indicating which types of cells are neighbors and / or what types of cells are neighbors 1. Detect one or more cell types (or tissue types) of a tissue sample, including cell contact information indicating whether the cell types are in contact with other types For example, the spatial organization of the cells (where each cell type is associated with a different population) Spatial information regarding the arrangement of cells can be determined. The cell size, shape, and structure, and / or cell area, microscopy, and / or microscopy may be used to determine the cell size, shape, and structure. Morphology with respect to other aspects of the tissue, such as cell circumference, cell size, and / or the like Any of the characteristics described herein can be used to determine biological information about a tissue sample. It should be understood that the above may be determined in some embodiments. These techniques provide information on cell types, cell distribution, spatial information, morphological information, and multicellular structure organization. Multiple characteristics of the tissue sample, such as the texture and / or any other characteristics described herein. This includes determining the characteristics of the

[0138] In some embodiments, these techniques use probabilistic analysis based on feature values. Cell typing is performed by determining the relationships between cells based on their feature values, such as by As described in conjunction with FIG. 5A-FIG. 5B, the channel contribution (e.g., For example, the average channel contribution can be used to determine cell type, which However, in some circumstances, the inventors showed that such channel contributions may not provide stable cell typing. For example, we have shown that the channel contributions affect the quality of the segmentation, This may depend on cell size, cell shape, tissue staining conditions, and / or the like. As another example, the present inventors have found that channel contribution is related to intracellular signal localization. understand that the actual signal data is not taken into account (This may be useful to help distinguish from other sizes.) The inability to set a range of values ​​for channel contributions for a particular type of cell is a major problem. We also understand that it may be difficult to purposefully probe certain cell types ( For example, clustering may be used instead, which may not find cells of interest. As an additional example, given such potential problems, Results may need to be checked manually, which increases delays and impacts automation. This could have an impact.

[0139] To address this potential problem, we used marker expression signatures to We have developed techniques for cell typing using the ELISA kit described in this document. According to some embodiments, an MxIF image of a tissue sample is obtained based on the cell arrangement data of cells in the tissue sample. 6 is a flow diagram illustrating an exemplary computerized process 610 for processing and predicting cell type. 6 is a chart. The computerized process 610 may, for example, The computerized process 610 may be performed by the computerized component 340 as described herein. The computerized process 610 may be implemented as part of the process and sub-processes of may be performed, for example, as part of steps 506 and / or 508 of FIG. 5A. .

[0140] In step 612, the computing device uses MxIF imaging to For example, at least one of the tissue samples obtained (as described in conjunction with step 502 of FIG. 5A) may be At least one multiplexed immunofluorescence image is acquired. As described herein, the marker images are Provided as separate images and / or as a combined image with multiple channels At least one immunofluorescence image may be provided, including nuclear markers (e.g., FOXP3, DAPI), membrane markers (e.g., IgG ... markers (e.g., CD3e), cytoplasmic markers (e.g., CD68), and / or the like. In step 614, the computer The imaging device obtains information indicative of the arrangement of cells within at least one multiplexed immunofluorescence image. For example, the placement information may be based on cell boundaries as described in conjunction with step 504 of FIG. information that allows for determining cell boundaries (e.g., marker expression), and / or One or more cell masks (e.g., for one or more of the received immunofluorescence images) The mask may include a cell mask.

[0141] FIG. 6B shows immunofluorescence images and results that can be used for cell typing, according to some embodiments. FIG. 6B shows an example of a first data set for a first cell 622A. and data for a second data set for a second cell 622B (collectively referred to as cells 622). For each cell 622, the data includes a nuclear marker (DAPI) image 624 and a cell segmentation We present a binary mask 626 and a marker image 62 that can be used to explore different representations of cells. In this example, the marker image 628 includes a set of 8 marker images. Intracellular marker image 628A (FOXP3 image), membrane marker image 628B (CD3e and CD68 image), and cellular marker image 628C (CD3e and CD68 image). It can be broadly classified as cytoplasmic image 628C (CD68).

[0142] In step 616, the computing device performs For each specific type of marker in at least one multiplexed immunofluorescence image, determining a marker expression signature comprising a likelihood of each of the markers being expressed in the first cell; Therefore, cell location information (e.g., cell mask) can be used to determine the cell location in the immunofluorescence image. In some embodiments, the computing The device uses a trained neural network to determine marker expression signatures. The neural network can be used to determine whether the marker signal is in the to determine the likelihood of being present and expressed (or not) The neural network model can be trained, for example, as described herein. Such a model may be implemented based on the ResNets model. As described herein, , which may include a large number of parameters. For example, such a model may include at least 50 Ten thousand parameters, at least a million parameters, several million parameters (e.g. at least 1 million parameters, 2 million parameters, 3 million parameters, etc.), and / or tens of millions of parameters (e.g., at least 10 million parameters, 1500 It may contain millions of parameters, 25 million parameters, 50 million parameters, etc. For example, such a model could have 40 to 45 million parameters (say, 4 0 million parameters, 42 million parameters, and / or 45 million parameters) In some embodiments, the parameters may include at least 100 million parameters ( For example, at least 100 million parameters, between 1 million and 100 million parameters, hundreds of millions parameters, at least 1 billion parameters, and / or any suitable number or It may include parameters within a range.

[0143] The inputs to the model are separate 1-channel MxIF images, 2-channel MxIF images, and 3-channel MxIF images. Each of the images may be at least one multiple immunofluorescence image of different markers, such as an image. Immunofluorescence images are scaled according to height (e.g., 128 pixels, 256 pixels, etc.) and width (e.g., 1 In some embodiments, the first channel may have a pixel size of 28 pixels, 256 pixels, etc. The sample or image may be, for example, a DAPI image (e.g., centered by the cell of interest). ) and the second channel or image can be another immunofluorescence marker image ( For example, this may be a single channel or image (also cropped) and a third channel or image may be segmented. The segmentation mask can be a segmentation mask of only one cell of interest. Referring to B, for example, for each cell 622, the input to the model is a nuclear marker image 624 , a segmentation mask 626, and one of the marker images 628 (e.g., intranuclear image 628A , membrane image 628B, and cytoplasmic image 628C). In this case, the input image (e.g., MxIF image or channel) may be normalized. The pixel values ​​of the image are normalized to be in the range 0 to 1, which corresponds to the bit depth of the image. For example, values ​​for an 8-bit image may be divided by 255, values ​​for a 16-bit image may be divided by 65535 and / or a similar operation may be performed. Such normalization is The image pre-processing performed by the MxIF image pre-processing module 320 described in conjunction with FIGS. 3 to 4B. It may be used in conjunction with other processes and sub-processes described herein, such as part of the image pre-processing. It is possible.

[0144] Neural networks can be used for binary classification (e.g., 0 or 1 for a classification) and / or The signal with the appropriate intensity and shape is generated for the input marker image with which the cell is associated ( For example, it can output one of the likelihoods with a mean of 1 (e.g., in the range 0 to 1). For example, Referring to FIG. 6B, for data set 622A, nuclear marker image 624, cell segmentation image 625, and For input of the quantification mask 626 and the nuclear image 628A (FOXP3), the output is the nuclear image 628A. Therefore, the likelihood that the nuclear image 628A has the appropriate intensity and shape of the cell that will be expressed is taken as the likelihood that the nuclear image 628A has the appropriate intensity and shape of the cell that will be expressed. It is possible.

[0145] The neural network is configured to generate at least one multiplexed immunofluorescence image (or a portion thereof) and For example, neural networks can be trained to process cell-specific information. At least one multiplex immunofluorescence image containing cells in the center of the image as identified by a mask It may be configured to compare a portion of the image (e.g., 128×128 pixels). As described above, the immunofluorescence images were used to obtain nuclear marker images of the region, as well as cell outline images or Other expression images may include membrane images, nuclear images, cytoplasmic images, and / or the like. A sample data set with such information can be used to determine whether the marker images are actually of the cells. Marker expression is likely (or not) determined by the Used to train a neural network to discriminate based on intensity and shape. It is possible.

[0146] A neural network can be trained on a library of images. The network is based on immunofluorescence images and / or can be trained on cell location data. In an illustrative example, which is not intended to be limiting, Three different training sets were used: four for nuclear localization markers (e.g., Ki67); The first one had 2,186 images (training set) and 403 images (test set) of markers. Set,9,485 images of 28 markers for membrane-localized markers (e.g., CD3, CD20) (training set) and a second set with 1,979 images (e.g., validation set). A total of 898 images (training set) of 8 markers with defined localization (e.g., CD68) were A third set with 427 images (validation set) and 427 images (validation set) may be used. As a result, the neural network can predict that the marker images of cells are the actual marker expression. It can be trained to provide a likelihood of whether

[0147] In some embodiments, these techniques are used to determine marker expression signatures. For example, the method may include running multiple trained neural networks to For example, different neural networks can be trained and used for different markers. In some embodiments, different neural networks generate different cellular localization information. For example, a marker having a specific function (e.g., a nuclear, membrane, or cytoplasmic marker) may be used. For example, a first neural network can be trained to detect expression in a nuclear image, and a second The neural network can be trained to detect the expression of cell membrane images, and a third model The model can be trained to detect expression in cytoplasmic images and / or similarly trained. It can be done.

[0148] In some embodiments, the marker expression signature for a tissue sample comprises at least We also calculated the probability or probability that a cell expressed each marker in one immunofluorescence image. In some embodiments, the likelihood value can be 0 (for example, Each likelihood can range from 0 (e.g., no expression) to 1 (e.g., expression). In addition to being based on marker intensity, other information can be determined based on the cell mask (e.g. , cell morphology, pixel intensity over the cell area, etc.) can also be used to determine the cell's shape.

[0149] In step 618, the computing device may further include a computing device that can analyze the marker expression signatures. In some embodiments, the cell typing data is compared to the cell typing data for each cell. For each cell type, a set of markers (e.g., markers in at least one multiplexed immunofluorescence image) It may contain a set of known marker signature entries for Each known marker signature determines whether the marker is expressed on a cell. Binary data (e.g., 0 or 1) indicating whether the (meaning that expression will be found in the cell type to which it is attached) In some embodiments, the specific markers may be ambiguous and / or the information may be inaccurate. are non-informative and therefore not necessarily expressive (e.g. (the marker may or may not be expressed on the cell), Data can be represented as such (e.g., by including both 0s and 1s). As an example of cell typing data, the cell typing table is provided by Sino Biological Cluster of Differentiation available from the company (e.g., https: / / www.sinobiological. com / areas / immunology / cluster-of-differentiation) and / or Bio-Rad Human immune cell markers that can be used to detect immune cells (e.g., https: / / www.bio-rad-antibodies.com / human-immunocytes) ne-cell-markers-selection-tool.html) Such antibodies can be generated based on known marker expression of such cells.

[0150] In some embodiments, the marker expression signature is compared to cell typing data. The marker expression signatures are then compared to known expression signatures of cells in the cell typing data. Comparison data can be generated to compare the expression signatures of the target gene with those of the target gene. The data is a set of probabilities for each marker expression signature and each probability for each cell in the cell typing data. Showing a comparison between the associated marker values ​​of a known marker signature The comparison data can be used to measure the distance between expression signatures, the percentage of overlap of expression signatures, and The similarity scores between the expression signatures and / or the marker expression signatures are used to refine the Any available cell signatures that can be compared to known cell signatures in the cell typing data. It may include comparison metrics, such as marker occurrence, and other applicable comparisons. The distance between the current signature and known marker expression signatures from cell typing data Distance can be cosine distance, Euclidean distance, Manhattan distance, and / or similar. In step 620, the computing device Based on this comparison, the device determines a predicted cell type. See, e.g., FIG. 6B. The first set of data 622A can be processed to predict a regulatory T cell type, and the data The second set 622B can be processed to predict macrophage CD68+ cell type.

[0151] In some embodiments, the computing device stores the cell typing data. The calculated comparison results for each cell type are analyzed to select the top candidate cell type. For example, when calculating distance, the computing device may It is possible to select from among various cell types by selecting a cell type having To try to have only one top candidate, the cell typing data is The cell type may be configured to include a unique entry for the cell type (e.g., The comparisons made result in the same value for multiple cell types in the cell typing data. (so that it doesn't happen).

[0152] FIG. 6C illustrates cell typing data 63 according to some embodiments of the techniques described herein. 8 to generate a marker expression signature 636 that determines a predicted cell type 640. 6 illustrates an example of using a neural network 634 for In this case, a marker expression signature can be a list, array, table, or expression of the probability for each marker. The cell typing data may be stored in a database, or the like, or in any suitable data structure. Similarly, you can write a list, array, table, or other data structure of binary values ​​for each marker. FIG. 6D illustrates a predicted variance in accordance with some embodiments of the techniques described herein. Cell typing data (in this example, cell typing tape) is used to determine the cell type being examined. The results are shown as 654 and contain known expression signatures for each cell type. A set of probabilities (shown in this example as a probability table 652) is generated to 6 shows an example of using a neural network 650. The neural network 650 Images 656A (DAPI marker), 656B (FOXP3 marker), 656C (CD19 marker), 656D (CD11c marker -) and 656E (CD3e marker), as well as the segmentation mask 656F for cells. 6. Process input 656, including a separate neural network, as described above. may be used to process the applicable marker images. The neural network was run on (a) images 656B (FOXP3 marker), 656C (CD19 marker), and 65 6D (CD11c marker) and 656E (CD3e marker) were used in combination with CD4 and (b) Image 656A (DAPI marker) and (c) cell segmentation with HLA-DR and HLA-DR images. The first trained image may be processed in conjunction with the training mask 656F. The neural network was then used to process the nuclear FOXP3 marker images, and the second The trained neural network was then used to identify membrane marker images (e.g., CD3e image 656E and CD1 1c image 656D), etc., so that separate news Neural networks can be used to process images based on different locations of markers. Thus, although only one neural network 650 is shown in FIG. 6D, the associated Several different pre-trained neural nets are used to process the attached marker images. A workpiece may be used.

[0153] In the example of FIG. 6D, the neural network 650 has FOXP3 in the probability table 652. Probability 0.96 for markers, probability 0.54 for CD3e markers, probability 0.66 for CD4, and probability 0.76 for CD1 Generates probability 0.003 for 9, probability 0.002 for HLA-DR, and probability 0.0005 for CD11c (The closer the probability is to 0, the lower the probability of expression; the closer it is to 1, the higher the probability of expression for the marker.) rate will be higher).

[0154] The computing device compares the probabilities in table 652 with the markers for each cell type. Calculate the cosine distance between the marker and the value of the marker. In this example, a "+" indicates that the marker is A "-" indicates that the marker is not expressed by the cells. The comparison results showed that the cosine distance value was 0.028 for the T-reg cell type and 0.036 for the CD4 T cell type. 0.339 for the B cell type, 0.99 for the myeloid cell type, and 0.99 for the myeloid cell type. In the example, the smaller the distance, the more likely the associated cell type is to be the cell of interest. As a result, the computing device is more likely to Select T-reg (lowest distance value 0.028) as the cell type to be analyzed.

[0155] FIG. 6E shows a method for determining predicted cell types according to some embodiments of the technology described herein. To determine the cell typing data (shown in this example as cell typing table 664), To generate a set of probabilities (shown in this example as probability table 662) that are compared to FIG. 6 shows an embodiment using a neural network 660 for the purpose of The 660 image shows MxIF images 666A (DAPI marker), 666B (CD68 marker), 666C (CD19 marker), and 6 66D (CD11c marker) and 666E (CD3e marker), as well as segmentation of cells 6. Process input 666, including a filter mask 666F. The neural network then maps each of the images 666B, 666C, 666D, and 666E (as well as the DAPI images 666A and In this example, , the neural network 660 calculates the probability for the CD68 marker in the probability table 652. 0.63, probability 0.0006 for CD3e marker, probability 0.01 for CD4, probability 0.001 for CD19 , generating a probability of 0.0004 for HLA-DR and a probability of 0.69 for CD11c (again, The closer the probability is to 0, the lower the probability of expression; the closer it is to 1, the higher the probability of expression for the marker. (It becomes more difficult.)

[0156] The computing device compares the probabilities in table 662 with the markers for each cell type. Calculate the cosine distance between the marker and the value of the positive and negative values. In this example, a "+" indicates that the marker is in a particular cell. A "-" indicates that the marker is expressed in a particular cell. "+-" means that the marker may or may not be expressed for the cell. In this example, the comparison results in a cosine distance of 0.001 for macrophage CD68+ cell types, 0.992 for CD4 T cell types, and 0.402 for B cell types 0.999 for the myeloid cell type and 0.478 for the myeloid cell type. In this example again, the distance The smaller the value, the more likely it is that the associated cell type is the cell type of the analyzed cell. As a result, the computing device is more likely to Select macrophages CD68+ (lowest distance value 0.001) as the target.

[0157] In some embodiments, these techniques identify clusters or cells based on cell characteristics. For example, cancer (e.g., breast cancer, renal cancer, It is desirable to search for certain cellular structures in a tissue sample, such as cellular structures indicative of cancer. These techniques are useful for identifying cell communities in tissue samples and for Identifying information about those cells, such as type, distance, etc. (e.g., sparsely populated cells) (to provide information about clusters, adjacent cell clusters, etc.) The community of cells can include different types of cells. In some embodiments, In the present invention, the cell clusters represent at least a portion of the tissue structure of the tissue sample. For example, mantle tissue, stromal tissue, tumors, follicles, blood vessels, and / or any other tissue structure. It can include.

[0158] FIG. 7B illustrates a first set of cell features ( To identify communities by acquiring and processing local cellular features 7B is a flowchart illustrating an exemplary computerized process 750. The computerized process 750 uses a graph neural network 772 to compute the obtained station The localized cellular features are processed to identify one or more communities 778 of cells. The data typing process 750 may be performed, for example, by the cell typing component 340 of FIG. The computerized process 750 may include other processes and sub-processes described herein. For example, the computerized process 700 may be performed as part of a process for As part of step 508 of FIG. 5A, the tissue samples may be grouped into groups of cells. A process for determining at least one characteristic of the sample (e.g., the community of cells) is performed. In step 752, the computing device generate local cell features, which are generated in step 752A by acquiring cell location data (e.g., cell configuration, cell mask, and / or other cell localization data); and triangulating the cell data 774 to generate a graph 776. The sensing device may use triangulation techniques (e.g., Delaunay triangulation) to 774) and construct a configuration of the cells (e.g., based on the center of gravity of the cells) to generate a graph representation 776 of the cellular structure of the tissue. In some embodiments, the graph 776 may be configured to obtain the detected The graph includes a number of nodes equal to the number of cells. In some embodiments, the graph generation aspect For example, in some embodiments, the edge of each edge in the graph 776 may be constrained. The length of the edge is the upper limit (based on pixel distance, for example, 200 pixels, 300 pixels, etc.). The number of edges in the graph 776 may be limited to a threshold value below which the number of edges may be determined. These techniques involve pruning edges that are smaller than a given length threshold. may include.

[0159] The computing device then generates a graph of the organization 776 in step 752B. Based on the current state, local cell features are calculated. The local cell features are calculated based on the cell data 774 and / or The graph 776 may include information about the cell that can be determined based on the graph. The local cell features are determined for each cell based on the cell type, the edge of the graph 776, Cell neighbors, neighboring cell types, neighbor distance data (e.g., median distance to neighbors, Mask-related data (e.g., marker masks under each cell (e.g., CD31 markers for blood vessels) the percentage of the area filled with positive pixels relative to the number of pixels in the image, and / or Thus, each node can contain a local data point association A set of vectors (e.g., vectors) can be stored in a In an embodiment of the present invention, the node data may be encoded using multiple variables. For example, if there are seven discovered cell types in a tissue sample: , "cell type 6" may be encoded as [0, 0, 0, 0, 0, 1, 0]. In this paper, the node data includes the median length of all node edges for a cell. In some embodiments, the node data can be calculated based on a given mask for the cell. The percentage of positive pixels in each of the multiple masks can be included. In some embodiments, the data may be expanded to include data for The data is a percentage of cells that are located within one or more masks of selected markers. Positive cell counts (e.g., percentage of the area of ​​the cell mask that is filled with positive cells) Such mask-based data can be used by a computing device to Exploiting information about cells and / or structures that may otherwise be difficult to segment As a result, in some embodiments, The total number of data points is L, which is determined by (1) the number of cell types and (2) the number of masks to consider (if any). ), and (3) the sum of the values ​​for the median distance of the edges of a given node.

[0160] The graph 776 may be encoded for input to the graph neural network 772. In some embodiments, the node information may be stored in a matrix having dimensions n×L. where n is the number of nodes and L is the number of node features. A graph can be represented as a sparse adjacency matrix (e.g., with dimensions n×n nodes), an adjacency list of edges, and / or the like.

[0161] In step 754, the computing device may generate a graph (e.g., an organizational graph). We feed the graph neural network with a set of locally finely-tuned images (an encoded version of 776). Graph neural networks embed cell features in a high-dimensional space. The graph structure containing the nodes and the graph is used to process the input graph. The global network can have different architectures. In this paper, the graph neural network 772 is an unsupervised convolutional graph neural network. For example, a graph neural network can be any suitable type of graph. Deep Graph I uses rough convolutional network layers to perform embedding in a graph The Deep Graph Infomax architecture can be implemented using See, e.g., https: / / openreview.net / for Petar Velickovic et al., "Deep Graph Infomax," available at um?id=rklz9iAcKQ. ICLR 2019 Conference Blind Submission (September 27, 2018), and / or arXiv:1809.1 0341. Examples of different types of convolutional layers that can be used in graph neural networks and https: / / arxiv.org / abs / 16 09.02907, Thomas Kipf and Max Welling, "Semi-Supervised Class sification with Graph Convolutional Networks," arXiv1609.02907 (February 217) William H. amilton et al., “Inductive Representation Learning on Large Graphs”, arXiv1706.0221 6 (September 2018). In addition to graph convolution layers, A neural network can have a discriminative layer that is used in training. The implementation of the work is at least 500,000 parameters, 1 million parameters, 2 million parameters, Various numbers of parameters, such as 100,000,000 parameters, 5 million parameters, or even more In some embodiments, such models may include tens of millions of meters. parameters (e.g., at least 10 million parameters, 1500 million parameters, depending on the implementation) Million parameters, 20 million parameters, 25 million parameters, 50 million parameters In some embodiments, the parameters may include at least one billion parameters (for example, at least 100 million parameters, between 1 million and 100 million parameters) parameters), hundreds of millions of parameters, at least a billion parameters, and / or any Any suitable number or range of parameters may be included.

[0162] Graph neural networks use a variant of noise contrast estimation to estimate each node embedding. The neural network may be trained to reconstruct ,correct representation (e.g., based on real data) and incorrect,representation (e.g., based on noise representation). The system essentially learns an internal representation of the nodes (e.g., cells) to distinguish between the nodes and the Rough neural networks are graph neural networks that can distinguish cell labels, neighboring cells, Information including cell distance, mask size (e.g., of some radius), and / or other inputs Since both local and global features of the tissue structure can be represented from the information, Therefore, the nodes generated by the graph neural network can be used Embedding is based on global covariance through global features, not just local node or cell neighborhood information. The resulting feature embeddings can then be It is possible to include the same number of features for each cell (e.g., a fixed-size vector (represented using tables, arrays, and / or other data structures).

[0163] In some embodiments, the output of the neural network is The activations of the graph convolution layers of the . Different types of convolution layers are used. Since we obtain a linear model (e.g., GCN, SAGEConv, etc., as described above), the output dimensionality is , 16 dimensions, 32 dimensions, etc. Each dimension of the output has a higher dimension. It can be an embedding of a space into a lower dimension, and generally, an organizational structure for the representation. Thus, for example, the embedding values ​​and the specific cell type In some embodiments, there may be at least some correlation between the class composition and the Clustering is then performed using these embeddings, as further explained in ,Clusters can be described in terms of the predominant cell types within them.

[0164] In some embodiments, for each of the plurality of training tissue samples, a graph of the tissue sample is Representation, local cells calculated as described above for steps 752A and 752B Features (e.g., for each node, cell type, mask relation data, and edge center A dataset including the mean distance (value distance) is generated and a graph neural network 772 is trained. In some embodiments, loss=log(P i )+log(1-P' i ) loss function such as , may be maximized, P i is the node (obtained through cell segmentation) is the probability that a given cell (a given cell) is similar to all graph nodes, and P' i is permuted is the probability that a given node is similar to all graph nodes. The code contains associated information (e.g., cell type assignment, cell-level information to neighboring elements). Median distance, mask information, etc.), which may take the form of a feature vector. In an embodiment, the Deep Graph Infomax architecture calculates all the We use a summary vector, which can be the average of the feature vectors for all nodes. The classifier layer of the graph model (a) classifies a given node feature vector as belonging to the graph summary vector. and (b) permutations that are classified as not belonging to the graph summary vector (e.g., To distinguish between the nodes, we use embeddings between the nodes (shuffled in place) and the feature vectors. can be trained to.

[0165] In step 756, the cell embeddings (including the neighborhood data) are analyzed to identify one or more clusters. To determine the clusters, the various classes are clustered. For example, as described herein, these techniques include Mind-based clustering algorithms (e.g., K-means), distribution-based clustering clustering algorithms (e.g., Gaussian mixture model clustering), density-based Clustering algorithms (e.g., DBSCAN), hierarchical clustering algorithms , PCA, ICA, and / or any other suitable clustering algorithm. For each determined cluster, the percentage of cells in the cluster, and The associated masks are the explanatory data for each cluster, as shown in 778. can be used to generate

[0166] FIG. 7C illustrates tissue contours shaded based on cell type, according to some embodiments. 7B , and a cell cluster (e.g., 7 shows an example of an image 782 in which the contours are shaded based on the contours of the object (determined to be the contours of the object). Image 782 shows that cell clusters are somewhat different than the shading based on cell type in image 780. The tissue structure of the follicular tissue (e.g., dark zone, light zone, and / or mantle zone follicles) can be more clearly identified. This shows how the method can be used to visualize

[0167] FIG. 18 illustrates processing an MxIF image in accordance with some embodiments of the techniques described herein. FIG. 18 is a diagram 1800 illustrating exemplary tissue properties that may be determined by The plurality may be determined, for example, by the characteristic determination module 360 ​​of FIG. 3. The tissue characteristics disclosed are part of the various processes and sub-processes described herein. For example, one or more of the tissue characteristics may be determined as part of step 510 of FIG. As shown, the characteristics may be determined using a radius check 1804 and / or a triple check 1805. To include and / or determine cellular spatial co-occurrence 1802, which may be used for angular surveying 1806. The characteristics may be used to generate one or more masks 1808. The sex may be used to contain and / or classify the cell population 1810. In morphology, these techniques use graph neural networks to classify cell populations. This may include:

[0168] In some embodiments, cell grouping information is determined by immunofluorescence to indicate tissue aspects. The masks may be used to determine one or more masks that may be applied to the light image. One example is a tumor mask, which contains data representing cells of a tumor in a tissue sample. is an acinar mask that contains data indicating the spacing between cells in a tissue sample, which is the It is possible to identify the ducts that form the gaps / spaces between the alveoli. For example, the acinar masks are It can show the ducts of the tumor that produce secretions, thus providing information about the ductal geometry. (e.g., because different tumors may have vessels of different shapes / sizes). An example of such a method includes the steps of: It is.

[0169] In some embodiments, the mask is used to distinguish cells in different regions of the tissue sample. For example, masks of acini, tumor, and stroma can be created to identify T cells, etc. can be used to understand where some cells are located in a tissue sample. For example, a mask can be used to identify T cells within stromal and / or non-stromal regions. do.

[0170] FIG. 19A illustrates a method for processing an immunofluorescence image 1900 according to some embodiments of the techniques described herein. FIG. 19 is a diagram of a stromal mask 1902 and an acinar mask 1904 generated by processing the As shown, the stromal mask 1902 includes data indicative of the supporting tissue of the tissue sample, including the acini. The mask 1904 includes data indicative of the spacing of cells in a tissue sample. These techniques are therefore useful for detecting cytoplasmic markers (e.g., most, if not all, cellular A stromal mask was generated using a marker expressed in cytoplasmic cells (S6) to identify cytoplasmic regions in tissue samples. and the regions identified using epithelial markers (e.g., PCK26 epithelial marker). In some embodiments, the cytoplasmic marker fraction can include excluding The image may be smoothed to perform noise removal as described herein, and / or In some embodiments, the epithelial marker image may be thresholded to identify filled holes. (e.g., whereby the interior portions of adjacent shapes are filled in and thus and removed from the final interstitial mask). In some embodiments, these techniques Generate an acinar mask using an epithelial marker (e.g., the smoothed PCK26 mask) This can include removing the interstitial zone to create the final acinar mask, which can be reversed. Epithelial markers can be generated using the tumor mask (using the smoothed PCK26 mask). It may also be used to generate other masks, such as

[0171] In some embodiments, the object is based on cell community information (e.g., FIG. 7B in conjunction with For example, the object mask may be Each of the one or more cell communities identified in the cell community information is a mass The cell community information is used to create a cluster of related objects in the cluster. Therefore, the objects in the object mask are obtained by dividing the objects into groups by the single objects identified in the cell community information. FIG. 19B illustrates a method for detecting a cell in a cell-cell structure using the techniques described herein. Generating an object mask 1956 using features of the tissue sample, according to some embodiments of the present invention. 19. The object mask 1956 is a representation of the cell coordinates, as shown at 1950 and 1952. Cells of a tissue sample, such as cell labels (e.g., community information), and image shape data. In particular, image 1950 in this example is generated based on the type of community. Showing shaded cell contours, image 1952 in this example shows the cells belonging to the identified community. One or more object detection algorithms process the cell features to determine the region ( For example, using Voronoi tessellation), which can be used to count objects For example, an object detection algorithm can generate cell labels, count the cell labels, and then identify the tissue. A mosaic of a sample (e.g. a randomly generated mosaic of a tissue sample) The resulting density heatmap 1954 can be used to detect objects. The density heat map 1954 can be generated based on an algorithm. Cells are shaded dark or light based on the number of cells detected. The imaging device creates an object mask 1956 of objects having a selected label and a high density of cells. As an illustrative example, lymph node follicles may be Community-based detection of dark, light, and mantle zones in node tissue samples. can.

[0172] FIG. 20 illustrates a comparison of acinar shape, area, and FIG. 2000 shows an example of measuring the characteristics of the circumference and the distance between the object and the object. FIG. 2000 shows three images 2002, 2004 and 2006. FIG. 2000 shows the corresponding acinar masses for images 2002, 2004, and 2006, respectively. As shown in FIG. 20, each acinar mask 2008, 2010, and 2012 is 2010 and 2012 are a measure in pixels of the area of ​​acinar structures in the acinar mask, A measure of the area of ​​acinar structures as a percentage of the total number of image pixels (e.g. the percentage of mask pixels that are in the acinar region), and a pixel-wise measure of the perimeter of the acinar structures Such parameters include a corresponding set of parameters 2020, 2022 and 2024. The meter can provide information about acinar structure. For example, the acinar area can be calculated by multiplying the acinar circumference by the acinar area. Comparing the length of the acinar to the total length (e.g., as a ratio) can provide information about the overall structure of the acinar. FIG. 2000 shows corresponding fibrosis images for images 2002, 2004, and 2006, respectively. Also shown are masks 2014, 2016 and 2018. The fibrosis masks show connective tissue, which is the interstitial mask and therefore can be produced as described herein for the interstitial mask. Each Fibrosis Mask 2014, 2016, and 2018 contains fibrosis filled with interstitial tissue. A measure of the area of ​​the mask, and a measure of the variance of the distribution of interstitial tissue (e.g., whether interstitial tissue is in the mask or not). parameters 2026, 2028, including parameters 2026, 2028, which may provide information about how widespread the , and a corresponding set of 2030.

[0173] FIG. 21 shows examples of spatial distribution characteristics according to some embodiments of the techniques described herein. FIG. 21 includes an example 2102 of measuring fibrosis distribution and an example 2104 of measuring T cell distribution. Fibrosis heatmap 2102A and t-cell heatmap 2104A show fibrosis and t-cell density. For example, darker shades (e.g., dark red) may be shaded to indicate It can indicate areas of highest density, while brighter colors (e.g. bright red) indicate areas of next highest density. A color can indicate areas of high density, and then another color (e.g., light blue) can indicate areas of even lower density. A different color (e.g., dark blue) can indicate the least dense areas ( For example, you can use a darker shade to improve contrast from denser areas. In some embodiments, the analysis is performed to provide concentration measurements in interstitial tissue. For example, a distribution of interstitial tissue (e.g., the distribution of (such as) may be determined based on the standard deviation of the interstitial distribution. ,For example, the stroma distribution divides the stroma mask image into equal sized squares (e.g. , using tessellation), and the representative value is compared to other squares For example, the percentage of pixels that represent stromal tissue may be determined for each square. The page can be used as a distribution from which the standard deviation can be calculated. These techniques can measure how homogeneously the stroma is distributed within the mask. For example, the apical fibrosis distribution 2106 indicates that the mask does not contain large areas without stroma. 2108 has a uniform distribution as shown, but the bottom fibrosis distribution 2108 is not uniform and therefore the mask is Indicates that it contains areas of no quality.

[0174] FIG. 22 shows examples of spatial organization properties according to some embodiments of the techniques described herein. FIG. 22 shows positive areas (within malignant cells) in the tumor mask and negative areas in the tumor mask. Example 2202 measuring the distribution of endothelial cells in the malignant compartment (non-malignant compartment) and The present invention includes an example 2204 that measures the distribution of T cells in malignant and non-malignant compartments.

[0175] FIG. 23 shows a comparison of the results of a CT scan performed on two different patients, according to some embodiments of the techniques described herein. FIG. 1 shows an example of cell contact information of an immunofluorescence image obtained by The contact information may be one of the determined characteristics of the tissue sample. For example, cell contact information may be determined as part of step 510 of FIG. 5A, and the number of cells determined for the tissue sample may be determined as part of step 510 of FIG. The number of groups may be determined based on, for example, each group being associated with a different cell type. Fluorescence image 2302 is of a tissue sample from patient 1 and immunofluorescence image 2304 is of a tissue sample from patient 2. Images 2302 and 2304 are both of CD8, CD68, CD31, and PCK26. The characteristic information 2306 includes the T markers in contact with the blood vessel for each of the patients 1 and 2. Information on the percentage of cells and the percentage of macrophages in contact with blood vessels. Information on the number of T cells in contact with macrophages and the percentage of T cells in contact with macrophages and information 2312 relating to the

[0176] FIG. 24 illustrates a comparison of two different patients from FIG. 23, according to some embodiments of the techniques described herein. FIG. 1 shows an example of information about cell neighbors for a cell, as described herein. The neighbor information may be one of the determined characteristics of the tissue sample, such as which type of cell A cell neighbor element can indicate whether a cell type is adjacent to another cell type. Both cell types in contact, as well as other cell types not in contact but in between FIG. 24 shows the immunofluorescence image 2302 from FIG. 23 and the FIG. 24 shows the number of T cell neighbors for each of the immunofluorescence images 2302 and 2034. The characteristic information 2306 also includes:

[0177] FIG. 25 illustrates MxIF images 2502, 2504 and 2506 in accordance with some embodiments of the techniques described herein. 25. Two examples of the stromal segmentation masks 2506, 2508 and the corresponding stromal segmentation masks 2508, 2509, respectively, are shown.

[0178] 26-35 show cell populations and associated properties determined using the techniques described herein. FIG. 26 illustrates various aspects of information, according to some embodiments of the techniques described herein. Exemplary MxIF images 2602 and 2604, segmentation masks 2606 and 2608, and The corresponding cell populations 2610 and 2612 are shown.

[0179] FIG. 27 illustrates a method for generating cell populations according to some embodiments of the techniques described herein. An example of a complete MxIF slide process is shown in FIG. A blow-up view 2704 of a portion 2706 of the retina is shown. The cell populations are CD3 T cells, CD4 T cells, CD5 Includes T cells, macrophages, blood vessels, and malignant zone cells.

[0180] FIG. 28 illustrates processing of an immunofluorescence image 2804 according to some embodiments of the techniques described herein. 28 is a diagram of a restored cell arrangement 2802 generated by , which represent cells in a tissue sample (e.g., where each cell type corresponds to a different population of cells) . Restored cell arrangement configuration 2802 includes acinar cells, CD4 FOXP3 T cells, CD4 T cells, CD8 T cells, endothelial cells, K167 and PCK26 cells, macrophages, and unsorted cells are shown.

[0181] FIG. 29 illustrates a 4′,6-diamidino-2-phenylindole (4′,6-diamino ... DAPI stained immunofluorescence image 2902 and two comparisons of different cell groups against the DAPI image. Images 2904 and 2096 are shown. Image 2904 shows markers for CD31, CD21, CD3, and CD68. Image 2906 also contains markers for CD31, CD21, CD3, and CD68, and CD20 Further includes:

[0182] FIG. 30 illustrates a different CCRCC tissue sample 300 according to some embodiments of the technology described herein. Cell populations for 2, 3004, and 3006 are shown.

[0183] FIG. 31 illustrates a renal tissue harvest from FIG. 14 according to some embodiments of the techniques described herein. 14 shows exemplary cell populations for the exemplary MxIF images 1404, 1406, 1410, and 1412 obtained. The cell group 3102 is determined based on the MxIF images 1404 and 1406, and the cell group 3104 is determined based on the MxIF images 1404 and 1406. The cell populations were determined based on 10 and 1412. The cell populations were T cells, macrophages, blood vessels, and malignant Includes the genital zone.

[0184] FIG. 32 shows a comparison of different clear cell renal cell carcinoma (CCR) samples according to some embodiments of the technology described herein. 32 shows a set of cell population images 3200 for a CRCC tissue sample. The images 3200 include T cells, NK cells, B cells, macrophages, blood vessels, and cell populations relative to the malignant zone are shown in FIG. A series of cell populations for different CCRCC tissue samples according to some embodiments of the technology described in 32, image 3300 shows T cells, NK cells, B cells, macrophages, and , blood vessels, and cell populations relative to malignant zones. For example, it may be possible to identify both T cells and malignant zones in a tissue sample. do.

[0185] FIG. 34 illustrates two different ELISAs of CCRCC tissue samples according to some embodiments of the technology described herein. FIG. 34 shows the results of an analysis of the MxIF image. Fifteen regions 3412 in image 3404 (e.g., determined as part of step 510 of FIG. 5A ) For the CCRCC tissue samples 3402 and 3404 and the corresponding characteristic information 3406 and The characteristic information includes, for each region 3410 and 3412, the B cells, CD4 T cells, and Cells, CD8 T cells, endothelial cells, epithelial cells, macrophages, macrophages 206, macrophages The percentages of phage 68, NK cells, tumor cells, tumor K167+, and unsorted cells are shown. The characteristic information classifies the 15 regions 3410 into either fibrous / normal regions or tumor regions, The regions 3412 are classified as fibrous / tumor regions, normal regions, or tumor regions.

[0186] FIG. 35 shows the MxI of a CCRCC tissue sample 3502 according to some embodiments of the technology described herein. The analysis results of the F image are shown. The analysis information 3504 is for each of the eleven regions 3506 of the tissue sample 3502. The B cells, CD4 T cells, CD8 T cells, endothelial cells, epithelial cells, macrophages, and myeloid cells in the area Macrophage 206, macrophage 68, NK cells, tumor cells, tumor K167+, and unclassified cells FIG. 35 also shows a blow-up view 3508 of the example region 9. vinegar.

[0187] 36A-41 generally illustrate tissue cell properties determined using the techniques described herein. 36A-36B show examples of cell FIG. 36A illustrates the amounts and ratios of different exemplary patients (e.g., patient RP83). 93, etc.), unclassified cells, endothelial cells, macrophages, CD8 T cells, CD4 T cells, tumor FIG. 36B includes a feature 3601 showing the percentage of tumor K167+ cells, and a feature 3602 showing the percentage of tumor cells. Characteristics of CD8 T cells 3604, characteristics of endothelial cells 3606, characteristics of macrophage cells 3608 for a characteristic of a tumor cell, 3610 for a characteristic of a tumor cell, and 3612 for a characteristic of a tumor cell. nothing.

[0188] FIG. 37 illustrates cell distribution characteristics according to some embodiments of the techniques described herein. FIG. 37 shows a tissue image 3702 and a corresponding stromal mask 3704 for a first patient. In both cases, T cells are shown in red. Figure 37 shows tissue images 3706 and 3707 for a second patient. and the corresponding stromal mask 3708, both with macrophage cells shown in red. Characteristic 3710 indicates that for images 3702 and 3704, just under 90% of the T cells are in the interstitium, with the remaining percentage Feature 3712 shows that the macro is located within the tumor for images 3706 and 3708. The majority of phage cells are located in the stroma, with a very small percentage in the tumor. It shows.

[0189] 38A-38B show histological characterization according to some embodiments of the techniques described herein. The percentage heat map and distribution density are shown. The heat map is the same as step 50 in FIG. 5A. 8 and 510, etc., may be generated based on cell populations. represents the percentage of blocks corresponding to the positive interstitial mask. A stromal mask 3802, a corresponding stromal heatmap 3804, and a graph of stromal distribution density 3804 are shown. FIG. 38A shows a stromal mask 3808 and corresponding stromal heatmap 3810 for a second patient. , and a graph 3812 of interstitial distribution density. FIG. 38B shows a third patient's interstitial mask 3818 and FIG. 38B includes corresponding graphs of x-sum intensity 3814 and y-sum intensity 3816. Also included is a quality mask 3824 and corresponding graphs of x-sum intensity 3820 and y-sum intensity 3822.

[0190] FIG. 39 illustrates a cell-neighborhood element and a cell, according to some embodiments of the technology described herein. FIG. 5B illustrates contact characteristics. Cell contact characteristics may be determined, for example, in step 510 of FIG. 5A. FIG. 39 shows the results of the ELISA for CD8 T cells, endothelial cells, macrophages, tumor cells, and unsorted cells ( Here, the neighboring cells are similarly CD8 T cells, endothelial cells, macrophages, tumor cells, and For the first patient, the percentage of adjacent cells (which may be unclassified cells) is shown. The first table 3902 shows the same neighboring cell percentages for various cell types. FIG. 39 includes an associated pie chart 3904 showing the distribution of CD8 T cells, endothelial cells, macrophages, and The second plot shows the percentage of adjacent cells relative to site, tumor, and unclassified cells. A second table 3906 for the patient, as well as the same adjacent cell patterns for various cell types. 3908 showing the percentages.

[0191] FIG. 40 illustrates a tS analysis of a characteristic of marker expression according to some embodiments of the technology described herein. A diagram showing examples of NE plots 4002 to 4016. In particular, tSNE plot 4002 shows CD45 marker expression. tSNE plot 4003 is for CAIX marker expression, and tSNE plot 4004 is for CAIX marker expression. Lot 4006 is for CD163 marker expression and tSNE plot 4008 is for CD206 marker expression, and tSNE plot 4010 is for CD31 marker expression, and tS NE plots are for CD3E marker expression and tSNE plots 4014 are for PBRM1 marker expression and tSNE plot 4016 is for PCK26 marker expression. SNE plots were generated using marker expression and were plotted using the marker expression levels. Different colors and sizes are used to indicate the correspondence between cells and their assigned cell types. By applying shading and / or by applying shading, two kinds of information are displayed as shown in FIG. To display the current intensity (each point is a cell), and the cell type on the graph 4020, The characteristics may be determined, for example, in step 510 of FIG. 5A, as described herein. It is possible.

[0192] FIG. 41 shows a graph of tS of a characteristic of marker expression according to some embodiments of the technology described herein. Another diagram showing examples of NE plots 4102-4148. In particular, the tSNE plot 4102 shows the CD138 marker. - expression, tSNE plot 4104 is for IgD marker expression, t SNE plot 4106 is for CD31 marker expression and tSNE plot 4108 is for SPARC marker expression. tSNE plot 4110 is for marker expression and tSNE plot 4111 is for marker expression tSNE plot 4112 is for CD10 marker expression and tSNE plot 4114 is for CD20 marker expression. tSNE plot 4116 is for CD3 marker expression. tSNE plot 4118 is for collagen marker expression, and tSNE plot 4120 is for collagen marker expression. is for CD163 marker expression, and tSNE plot 4122 is for HLA-DR marker expression. tSNE plot 4124 is for CD11c marker expression, and tSNE plot 4125 is for CD11c marker expression. Plot 4126 is for CD21 marker expression and tSNE plot 4128 is for CD68 marker expression. tSNE plot 4130 is for Ki-67 marker expression, and tSNE plot 4131 is for Ki-67 marker expression Plot 4132 is for CD25 marker expression and tSNE plot 4134 is for FOXP3 marker - expression, tSNE plot 4136 is for CD8 marker expression, t SNE plot 4138 is for CD44 marker expression and tSNE plot 4140 is for CD35 marker expression. tSNE plot 4142 is for marker expression and tSNE plot 4143 is for marker CD4 expression tSNE plot 4144 is for CD45 marker expression and tSNE plot 4146 is for BCL6 marker expression. tSNE plot 4148 is for IRF4 marker expression. do.

[0193] 42-47 generally show 4',6-diamidino-2-phenylindole (DAPI) heterogeneous markers. It is relevant to use the techniques described herein in the context of Kerr staining. According to some embodiments of the described technology, 4',6-diamidino-2-phenylindole (DAPI) To determine cell segmentation data 4204 for a heterogeneously stained immunofluorescence image 4206 FIG. 4 is a pictorial illustration of the use of a convolutional neural network 4202.

[0194] FIG. 43 illustrates a DAPI stained immunofluorescence image of a tissue sample according to some embodiments of the technology described herein. A first cell mask generated based on a combination of the fluorescent image 4304 and the CD3 cell marker image 4306. 4302. The cell mask may be used, for example, in conjunction with step 504 of FIG. 5A. The information may be obtained as part of the information showing the arrangement of cells as described in the cell matrix of FIG. As shown in the example mask, cell mask 4302 is used to identify markers for cells in a tissue sample. It can be applied to different immunofluorescence images to convey the results.

[0195] FIG. 44 is a DAPI stained immunofluorescence image of FIG. 43 according to some embodiments of the techniques described herein. A second cell mask 440 generated based on a combination of the image 4304 and the CD21 cell marker image 4404. FIG. 45 is a pictorial illustration of a method for detecting a stenosis in accordance with some embodiments of the techniques described herein. Based on the combination of the DAPI stained immunofluorescence image 4304 and the CD11c cell marker image 4504 in FIG. 45 is a pictorial illustration of a third cell mask 4502 that has been generated.

[0196] FIG. 46 shows an MxIF image of the tissue sample of FIG. 43 according to some embodiments of the techniques described herein. A vascular mask 4602 generated based on image 4604 (image 4606 is an enlarged view of a portion of image 4604) is illustrated. A vascular mask 4602 shows the arrangement of blood vessels within the tissue. To emphasize that the calculated mask 4602 corresponds to blood vessels in tissue, a synthetic pseudocolor filter is used. - In image 4604, the CD31 marker is first on top of the DAPI nuclear marker of a second shade (e.g., blue). As described herein, the vascular mask may be For example, it can be created as part of step 510 of FIG. 5A.

[0197] FIG. 47 illustrates the mask 4302 of FIGS. 43-46 in accordance with some embodiments of the techniques described herein. FIG. 4 is a pictorial illustration of a cell population 4702 generated using 4402, 4502, and 4602. Cell population 4702 is comprised of T cells (green), follicular dendritic cells (dark orange), CD11c+ cells (blue), and and blood vessels (red).

[0198] FIG. 48 illustrates a prostate tissue sample 4804 and FIG. 48 shows a set of cells 4802 relative to a malignant site 4806. Surrounding the affected dense tumor area, with the majority of the tumor still segmented FIG. 49 shows a method for highlighting immune infiltration, according to some embodiments of the technology described herein. FIG. 50 shows an enlarged view of a portion 4902 of a cell population 4802 of the prostate tissue sample 4804 of FIG. According to some embodiments of the technology described herein, the results are shown in Table 1. A set of cell populations 5002, 5004, 5006, and 5008 for a visualized prostate tissue sample are shown. Each cell in Figures 48 to 50 is colored according to its cell type, and the blood vessel mask is The binary black and white images in Figures 48 and 49 are colored red. 13 is a full-slide segmentation mask of fibrosis obtained using the CT scan.

[0199] Cosmetic compositions that may be used in connection with any of the disclosed embodiments provided herein An exemplary implementation of a computer system 5100 is shown in FIG. The computer system 5100 may include a computing device 112 and / or a computing The computer system 5100 may be used to implement the processing device 116 shown in FIGS. 5A, 6A, and / or 7A-7B. The computer system 5100 may be used to implement multiple aspects. A plurality of computer hardware processors 5102 and a non-transitory computer readable storage medium. (e.g., memory 5104 and one or more non-volatile storage devices 5106) The processor 5102 may include one or more articles of manufacture that include a memory. The memory 5104 and the non-volatile storage device 5106 are used to write and read data. The processor may control the reading of data in order to perform any of the functions described herein. The processor 5102 includes a non-integral memory that stores processor-executable instructions for execution by the processor 5102. One or more non-transitory computer readable storage media that may act as a temporary computer readable storage medium. One or more processor-readable storage media (e.g., memory 5104) that may be used to execute programs. Possible instructions can be executed.

[0200] The computer system 5100 includes a processor 5102, a memory 5104, and non-volatile storage. The computing device 5106 may be any type of computing device. For example, the computer system 5100 may be a server, a desktop computer, a laptop, In some embodiments, the device may be a computer, a tablet, or a smartphone. The data system 5100 may include a cluster of computing devices, a virtual computing devices, and / or cloud computing devices The present invention may be implemented using a computing device.

[0201] 52-53 show that, according to some embodiments, the techniques described herein can be used to align and align cells. We present two comparisons to show how our method outperforms conventional techniques in both cell shape reconstruction and cell shape reconstruction. The ground truth data for both Figures 52 to 53 are from different non-small cell lung cancer Tissue samples and markers included DAPI, NaKATPase, PCK26, CD3, and CD68. To guide manual segmentation by pathologists on the end-of-truth data To provide an example of the prior art, ground truth data was collected from https: / / www. CellProfiler, available from the Broad Institute, at / / cellprofiler.org / releases The cells were segmented using Ofiler 4.1.3 (for ease of explanation, we refer to this paper as CellPro (hereinafter referred to as "filer").

[0202] To demonstrate the techniques described herein, cell segmentation was performed using trained convolutional This is carried out using neural networks, which allows cell segmentation data (For ease of explanation, the examples in Figures 52 and 53 are referred to as "Mask R-CNN" In particular, the pre-trained model used in this example uses a ResNet-50 encoder head. We implemented a Region-Based CNN architecture using the CNN. This model has about 43 million parameters. Had a meter.

[0203] The input to Mask R-CNN is three channels, which includes nuclear markers for the first channel. - (including DAPI), almost all cells of a particular tissue type for a second marker The markers included membrane markers (including CD45 or NaKATPase) present above the markers, and the third marker showed sufficient staining. One additional membrane marker with chromosomes (CD3, CD20, CD19, CD163, CD11c, CD11b, CD56, CD All channels are normalized to be in the range 0 to 1 and As a match, the maximum expression value at each pixel among all selected markers is used. It was done.

[0204] In this implementation, the Mask R-CNN segmentation process is performed by the forward network. This includes a post-processing step that generates the final cell segmentation data. The input image is divided into intersecting squares of size 256 × 256 pixels. The output is an array of mask proposals for each distinct cell. In this implementation, the output is The output is not a value image, but rather represents the probability that a given pixel is part of the cell's mask. The output images were thresholded using a value of 0.5. The masks are then multiplied to obtain the final binary mask. Some of the masks intersect with each other. If so, a non-maximum suppression algorithm was implemented (e.g., this is the case when Available at https: / / arxiv.org / pdf / 1704.04503.pdf, which is incorporated herein by reference. Navaneeth Bodla, "Improving Object Detection With One Line of Code" For example, the prediction window is Such intersections may contain redundant data, such as due to cells between the edges of the windows. This was avoided by processing only cells from the center, top corners, and right corners of the image (e.g. For example, if the current window is the last one from the right or bottom, (The bottom and right parts of the image are processed only if the The segments are represented in the final output mask by integer values ​​that represent individual cell instances. and converted to cell outlines.

[0205] The training dataset was pruned for training with a pruning size of 300 × 300 pixels. It contained 1 image. Dropout, rotation, elastic transformation, Gaussian blur, contrast Adjusting the reflections, adding Gaussian noise, decreasing the membrane channel intensity (sometimes even to zero), All included images were augmented and further cropped to a fixed size of 256 × 256 pixels. For the first channel, the nuclear marker was DAPI and for the second channel, CD45 or is NaKATPase, and the third channel is expressed by various additional markers (CD16, CD163, CD45 ( If the marker in the second channel is NaKATPase), CD206, CD8, CD11c, CD20, CD56, All the labels were obtained through human annotation. Mask R-CNN model We use backpropagation and the Adam optimizer to randomly generate annotated images. Trained on pruning.

[0206] In FIG. 52, image 5202A shows a first ground truth cell image for a first tissue sample. Segmentation information is shown. In these examples, the segmentation does not necessarily follow cell boundaries. Since the contour is predicted without including the boundary, the cell segments are close to each other (or or overlapping), which allows one to see different segmented cells in image 5202A. Thus, image 5202B shows an example of various segmented cells. The first ground toe is shaded using different shading to help indicate Image 5204A shows different cell segments from the loose data. 4 shows the determined cell segmentation information for the ground truth data of Image 5204B shows the different segmented cells as determined by the cell profiler. The various segmented cells are shown shaded to help illustrate. Image 5206A shows the cell alignment of the first ground truth data determined using Mask R-CNN. 5 shows the position information, and image 5206B shows the position information determined for the first ground truth data. The different segmented cells are shown shaded.

[0207] Table 1 below shows the comparison of CellProfiler and ground truth data for Figure 52. We present segmentation comparison metrics for Mask R-CNN. [Table 1]

[0208] Panoptic Quality (PQ) is used as a performance metric, e.g., http Alexander Kirillov, "Panoptic S," available at: https: / / arxiv.org / abs / 1801.00868. "Egmentation," arXiv:1801.00868 (April 2019).

[0209] The Jaccard coefficient indicates the accuracy of cell detection and cell shape reproduction. Cell segmentation using end-truth data and the Hungarian algorithm The cell is determined by matching cells with the predictions of the cell type (each cell is either 0 or 1). For each matched pair, the jackal function between them is The final result is the sum of the values ​​for the maximum number of cells (ground truth mask or is calculated as the number of cells in the prediction mask divided by the number of cells in the prediction mask.

[0210] F1, Precision, and Recall are the metrics used in machine learning to measure the accuracy of a given ground truth (GT) Used to check the quality of the prediction against the object (or segment, in these examples). where Precision=TPS / GTS, Recall=TPS / PS, F1=2*Precision*R ecall / (Precision+Recall). TPS (true positive segments) is the number of segments from GT with Jaccard ≥ 0.5. represents the number of segments from the neural network prediction with the reference segment, GTS is the number of all segments from GT and PS is the number of all segments from prediction. do.

[0211] In FIG. 53, image 5302A shows a second ground truth cell image for a second tissue sample. Image 5302B shows segmentation information against the second ground truth data. Image 5304A uses shading to show the different cell segmentations in CellP Cell segmentation determined against second ground truth data using profiler Image 5304B shows the segmentation information of the different Image 5306A shows the cells that were imaged using Mask R-CNN with a second ground-truth filter. Image 5306B shows cell segmentation information determined for the Mas data. Different segmentations determined on the second ground truth data using k R-CNN The irradiated cells are indicated by shading.

[0212] Table 2 below shows the comparison of CellProfiler and ground truth data for Figure 53. We present segmentation comparison metrics for Mask R-CNN. [Table 2]

[0213] As shown by the images in Figures 52-53 and the comparative data in Tables 1 and 2, As mentioned above, Mask R-CNN is better at detecting cell contours than CellProfiler, but this is due to , because this model was trained for cell segmentation (while Cell lProfiler is generally used for cell segmentation based on nuclear thresholding, and for Cell membrane identification by enlarging the nucleus so that it is outlined with a selected number of pixels. Mask R-CNN has fewer false positives (e.g., more CellProfiler Reproduced cell shapes closer to the state of the cells (e.g., higher PQ and Jaccard coefficient metrics). Therefore, Mask R-CNN is a promising candidate for cell segmentation. This represents a significant improvement over CellProfiler and other similar previous approaches. Such improved cell segmentation can then be used to However, subsequent aspects performed by the image processing pipeline used can be improved. , which allows further analysis to be performed on cell segmentation data as described herein. , because they may use and / or rely on

[0214] The terms "program" and "software" are used in their general sense in this specification. and a computer or a processor configured to implement various aspects of the embodiments as described above. Any kind of program that can be used to program other processors (physical or virtual) It is used to refer to computer code or any set of processor-executable instructions. Additionally, in accordance with one aspect, the disclosures provided herein when implemented One or more computer programs implementing the method may be used on a single computer or processor. Although not necessarily resident on a single processor, various aspects of the disclosure provided herein may be implemented using the same. may be distributed in modules among different computers or processors to implement .

[0215] Processor-executable instructions are instructions executed by one or more computers or other devices. The program may take many forms, such as a program module, executed by a A module is a set of routines that perform a particular task or implement a particular abstract data type. , programs, objects, components, data structures, etc. The functionality of the program modules may be combined or distributed.

[0216] The data structures may also be implemented in one or more non-transitory computer readable storage devices in any suitable format. For ease of illustration, the data structure may be stored in a readable storage medium. Such relationships can be shown to have fields that are related through their placement within the Similarly, a field may refer to an arrangement in a non-transitory computer readable medium that conveys relationships between fields. However, any suitable method may be used to allocate storage for The mechanism is pointers, tags, or other mechanisms that establish relationships between data elements. Used to establish relationships between information in fields of a data structure, such as through the use of It can be done.

[0217] Various inventive concepts may be embodied as one or more processes, examples of which include The activities performed as part of each process may be ordered in any suitable manner. Thus, embodiments in which activities are performed in a different order than illustrated may be possible. may be constructed, which is shown in the exemplary embodiment as a sequential operation. A single action may involve performing several activities simultaneously.

[0218] As used in the specification and claims, a reference to a list of one or more elements means The phrase "at least one" in a reference means any one of the elements in a list of elements. means at least one element selected from one or more, and not necessarily from a list of elements does not contain at least one of the elements specifically listed in It should be understood that this definition does not exclude any combination of elements in the above list. The element may, optionally, be in the list of elements referred to by the phrase "at least one" Other than the elements specifically identified, whether related to the elements specifically identified or unrelated Therefore, for example, "A and B at least one of" (or equivalently, "at least one of A or B" or equivalently etc., but "at least one of A and / or B" is, in one embodiment, optional. There is at least one A and no B (and, optionally, no more than B) In another embodiment, at least one B, optionally including more than one, and A is not present (and optionally contains elements other than A), and In the form, at least one A, optionally including multiple, and at least one B, optionally including multiple may be taken to mean that there is at least one B (and optionally other elements), etc. stomach.

[0219] As used in the specification and claims, the phrase "and / or" The claim states that "either or both" of the elements are so combined, i.e., the elements are In some cases it is understood to mean conjunctive existence, and in other cases disjunctive existence. Multiple elements listed with "and / or" are treated in the same way, i.e. i.e., should be interpreted as "one or more" of the elements so combined. The elements of are optionally specifically identified by the "and / or" clause. They may be present whether or not they are related to the element being identified. Thus, as a non-limiting example, a reference to "A and / or B" may be amended to include an open When used in conjunction with the end phrase, in one embodiment, it refers only to A (optionally in another embodiment, it refers only to B (optionally including elements other than A); In yet another embodiment, it may refer to both A and B (optionally including other elements), etc. stomach.

[0220] In the claims, use of "first," "second," "third," etc. to modify claim elements is permitted. The use of ordinal numbers does not, in and of itself, imply any priority or precedence of one claim element over another. It does not imply a sequence or the chronological order in which the activities of the method are performed. Such terms are used to distinguish one claim element with a particular name from another element with the same name. They are only used as labels to distinguish between different classes (except for the use of ordinal numbers). The phraseology and terminology used herein is for purposes of description and not of limitation. "Including," "comprising," "including," "comprising," "including" "comprising," "having," "containing," "involving" The use of the term "invention" and variations thereof herein are intended to be construed as being in accordance with the following paragraphs and the accompanying drawings. This is meant to include additional items.

[0221] Although several embodiments of the technology described herein have been described in detail, various modifications and Modifications will readily occur to those skilled in the art. Such modifications and improvements are intended to be included within the spirit and scope of the present disclosure. It is intended that the foregoing description be construed as being within the spirit and scope of the present disclosure. These techniques are described in the following claims and their equivalents: It is limited only by the definition of equivalents. [Explanation of symbols]

[0222] 1, 2 patients 100 Systems 102 MxIF images 104 Information 110 Microscope 112 Computing Devices 114 Network 116 Computing Devices 200 Figures 202 Raw MxIF Images 204 Segmentation Information 206 Mask 208 Information 210 Information 300 Image Processing Pipeline 300 Figures 310 MxIF images 310A, 310B, 310C, to 310N Images 320 MxIF Image Pre-Processing Components 330 Cell Segmentation Component 340 Cell Typing Components 350 Cell Morphology Evaluation Component 360 Characterization Component 400 Processing Flow 410 Processed MxIF Images 410A, 410B, 410C, to 410N Processed MxIF images 420 Placement information 430 Tissue Deterioration Check Component 440 Dyeing Department 440A and 440B Nuclear Marker Images 442A Window 442B Window 444 Patch Mask Sections 444A, 444B, and 444C 446 Segmentation Mask 448 Filtered Segmentation Mask 500 Computerized Processes 502 Activities 600 feature value 602 Cell arrangement data 610 Computerized Processes 622 cells 622A First Cell 622B Second Cell 624 Nuclear marker (DAPI) images 626 Cell Segmentation (Binary) Mask 628 Marker Images 628A Nuclear Marker Image 628B Membrane Marker Image 628C Cytoplasmic Image 634 Neural Networks 636 Marker Expression Signature 638 Cell Typing Data 640 cell types 650 Neural Networks 652 Probability Table 654 Cell Typing Table 656 Input 656A, 656B, 656C, 656D, 656E Images 656F Segmentation Mask 660 Neural Networks 662 Probability Table 664 Cell Typing Table 666 Input 666A, 666B, 666C, 666D, and 666E MxIF Images 666F Segmentation Mask 700 Computerized Processes 750 Computerized Processes 774 Cell Data 776 Graphs 772 Graph Neural Networks 778 Community 780 images 782 images 800 MxIF images 802 Manual cell placement / segmentation data 1000 pre-trained convolutional neural network models 1002 Cell Segmentation Data 1010 MxIF images 1012 First marker image 1014 Second marker image 1100 Trained Neural Networks 1102 Cell arrangement / segmentation information 1110 MxIF images 1112 DAPI marker image 1114 NaKATPase marker image 1200 MxIF images 1202 Cell Segmentation Data 1300 composite fluorescent images 1302 Cell Segmentation Data 1402 Cell Segmentation Data 1404 and 1406 MxIF images 1408 Cell Segmentation Data 1410 and 1412 MxIF Images 1500 MxIF images of clear cell renal cell carcinoma (CCRCC) 1502 Cell Segmentation Data MxIF image of 1600 CCRCC 1602 Cell Segmentation Data 1700 Convolutional Neural Network Architecture 1702A, 1702B and 1702C convolutional layers 1704A, 1704B and 1704C upsampling layers 1706 Threshold Image 1708 raw input images 1710 Threshold Difference 1800 Figures 1802 Cellular spatial co-occurrence 1804 Radius Check 1806 Triangulation 1808 Mask 1810 cell group 1900 Immunofluorescence Images 1902 Interstitial Mask 1904 Acinar Mask 1950 Images 1952 Images 1956 Object Mask 2000 Figure 2002, 2004 and 2006 Images 2008, 2010 and 2012 Acinar Mask 2014, 2016 and 2018 Fibrosis Masks 2026, 2028, and 2030 Parameters 2102 Example 2104 Example 2106 Apical fibrosis distribution 2108 Plantar fibrosis distribution 2202 Example 2204 Examples 2302 Immunofluorescence images 2304 Immunofluorescence images 2306 Characteristics Information 2308 Information 2310 Information 2312 Information 2502, 2504 MxIF images 2506, 2508 Stroma segmentation mask 2602 and 2604 MxIF Images 2606 and 2608 Segmentation Masks 2610 and 2612 cell groups 2702 Views 2704 Blow Up View 2706 parts 2802 Reconstruction of cell arrangement 2804 Immunofluorescence images 2902 4',6-diamidino-2-phenylindole (DAPI) stained immunofluorescence image 2904 and 2906 Images 3002, 3004, and 3006 CCRCC tissue samples 3200 images 3300 images 3402 images 3404 images 3402 and 3404 CCRCC tissue samples 3406 and 3408 Characteristics Information 3410 area 3412 area 3502 CCRCC tissue sample 3508 Blow Up View 3602 Characteristics 3604 Characteristics 3606 Characteristics 3608 Characteristics 3610 Characteristics 3612 Characteristics 3702 Tissue Images 3704 Interstitial mask 3706 Tissue Images 3708 Interstitial Mask 3710 Characteristics 3712 Characteristics 3802 Interstitial mask 3804 Stroma Heatmap 3806 Graph of interstitial distribution density 3808 Interstitial mask 3810 Stroma Heatmap 3812 Graph of interstitial distribution density 3814 x-sum strength 3816 y-sum intensity 3818 Interstitial Mask 3902 First Table 3904 Pie Chart 3906 Second Table 3908 Pie Chart 4002~4016 tSNE plot 4102~4148 tSNE plot 4202 Convolutional Neural Network 4204 Cell Segmentation Data 4206 4',6-diamidino-2-phenylindole (DAPI) heterogeneous staining immunofluorescence image 4302 First Cell Mask 4304 DAPI stained immunofluorescence image 4306 CD3 cell marker image 4402 Second Cell Mask 4404 CD21 cell marker image 4502 Third Cell Mask 4504 CD11c cell marker image 4602 Vascular Mask 4604 MxIF images 4606 images 4702 Cell group 4802 cell group 4804 Prostate tissue samples 4806 Malignant site 4902 Part 5100 Computing devices, computer systems 5102 Computer Hardware Processor 5104 Memory 5106 Non-volatile storage device 5202A Images 5202B Images 5204A Images 5204B Images 5206A Images 5206B Images 5302A Images 5302B Images 5304A Images 5304B Images 5306A Images 5306B Images

Claims

1. Use at least one hardware processor acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; obtaining information indicative of an arrangement of cells within said at least one MxIF image; The plurality of groups of cells in the at least one MxIF image are at least in part identified by: showing said at least one MxIF image and an arrangement of cells within said at least one MxIF image; determining feature values ​​for at least some of the cells using the information; Determine, The determined characteristic values ​​are used to detect at least some of the cells. into said plurality of groups. and identifying the determining at least one characteristic of the tissue sample using the plurality of groups; How to do it.

2. The step of acquiring at least one MxIF image includes acquiring a single multi-image of the same tissue sample. acquiring a multi-channel image, The method of claim 1, wherein each of the markers is associated with a respective marker in the plurality of markers. Law.

3. The step of acquiring at least one MxIF image may include performing multiple immunofluorescence analysis of the same tissue sample.

3. The method of claim 1 or claim 2, comprising acquiring a fluorescent image.

4. At least one of the plurality of immunofluorescence images is associated with a respective marker. 3, or any one of claims 1 or 2, comprising a single channel image The method according to claim 5.

5. At least one of the plurality of immunofluorescence images comprises a multi-channel image; A channel in a multi-channel image corresponds to each marker in a set of multiple markers.

5. The method of claim 3, or any one of claims 1, 2 or 4, wherein

6. 12. The method of claim 1, wherein the at least one MxIF image of the tissue sample is captured in vitro. or the method according to any one of claims 2 to 5.

7. At least some of the cells include a first cell, and the characteristic value is The determining step associates a location of the first cell within the at least one MxIF image. a first feature value for the first cell using at least one pixel value stored in the 7. The method of claim 1, or any one of claims 2 to 6, comprising a determining step.

8. At least some of the cells include a second cell, and the characteristic value is The determining step associates the second cell with a location within the at least one MxIF image. a second feature value for the second cell using at least one pixel value that has been 8. The method of claim 7 or any one of claims 1 to 6, comprising the step of determining.

9. The step of determining the first feature value for the first cell comprises: associated with respective placement of said first cells in the multiple channels of xIF images. 7 or any of claims 1 to 6 or 8, comprising the step of using pixel values.

3. The method according to claim 1 .

10. The step of determining the first characteristic value may include determining the first characteristic value from ALK, BAP1, BCL2, BCL6, CAIX, CCASP3, CD10, , CD106, CD11b, CD11c, CD138, CD14, CD16, CD163, CD1, CD1c, CD19, CD2, CD20, CD2 06, CD209, CD21, CD23, CD25, CD27, CD3, CD3D, CD31, CD33, CD34, CD35, CD38, CD39 , CD4, CD43, CD44, CD45, CD49a, CD5, CD56, CD57, CD66b, CD68, CD69, CD7, CD8, CD 8A, CD94, CDK1, CDX2, Clec9a, chromogranin, collagen IV, CK7, CK20, CXCL13, D API, DC-SIGN, Desmin, EGFR, ER, ERKP, Fibronectin, FOXP3, GATA3, GRB, G Ranzyme B, H3K36TM, HER2, HLA-DR, ICOS, IFNg, IgD, IgM, IRF4, Ki67, KIR, Lumi Kan, Lyve-1, Mammaglobin, MHCI, p53, NaKATPase, PanCK, PAX8, PCK26, CNAP, PB RM1, PD1, PDL1, perlecan, PR, PTEN, RUNX3, S6, S6P, SMA, SMAa, SPARC, STAT3P, Determine feature values ​​for one or more of the following markers: TGFb, Va7.2, and vimentin 10. The method of claim 9 or any one of claims 1 to 8, comprising the step of:

11. the plurality of channels being associated with respective markers in a plurality of markers; The respective ones of the first cell in the plurality of channels of the at least one MxIF image. The pixel values ​​associated with each location are used to determine the first feature value. The step of: detecting each particular marker of one or more of the plurality of markers a marker comprising a likelihood that the particular marker is expressed in the first cell, - determining an expression signature, grouping said at least some of said cells into said plurality of populations. The top The marker expression signature and cell typing data are used to identify the first cell. determining a predicted cell type for the Associating the first cell with one of the plurality of groups based on the predicted cell type. The method according to claim 9 or any one of claims 1 to 8 or 10, comprising the steps of: How to.

12. determining said characteristic values ​​for said at least some of said cells; Pu is marker expression for each particular cell of the plurality of cells in said at least one MxIF image determining a signature, the marker expression signature being a set of markers For each particular marker of the one or more markers, is expressed in said particular cell, Predicting the at least some of the cells using the determined characteristic values. The step of grouping into a plurality of groups includes: The marker expression signature and cell typing data for the particular cell. determining a predicted cell type for each particular cell of said plurality of cells using and, A process for grouping the plurality of cells into the plurality of groups based on the predicted cell type. The method of claim 1 or any one of claims 2 to 11, comprising the steps of:

13. The plurality of markers is at least one of the markers according to claim 10.

13. The method of claim 12 or any one of claims 1 to 11, comprising:

14. The cell typing data includes at least one marker for each of a plurality of cell types. - expression signatures, the at least one marker for each particular cell type of the plurality of cell types. The expression signature indicates which of the plurality of markers are expressed in cells of the particular cell type. The method of claim 11 or claims 1 to 10 or 12 to 13, comprising data showing whether the 2. The method according to any one of claims 1 to 11.

15. A first trained neural network configured to determine a marker expression signature. determining said marker expression signature using a network.

12. The method of claim 11, or any one of claims 1 to 10 or 12 to 14.

16. The first trained neural network includes at least 1 million parameters. , claim 15, or a method according to any one of claims 1 to 14.

17. As input images, we train immunofluorescence images of tissue samples against associated cell types. and a marker expression in the input image for the associated cell type. and associated output data including information about the first trained neural network. The method of claim 15, or claim 1 to 14 or claim 1 further comprising the step of training the network.

7. The method according to any one of claims 6 to 6.

18. The at least one MxIF image is input to the first trained neural network. and providing The marker expression signature is obtained as an output from the first trained neural network. The method according to claim 15, or any one of claims 1 to 14 or 16, further comprising the step of obtaining a tea.

18. The method according to any one of claims 17 to 17.

19. The information indicating the location of the cells is used to select at least some of the plurality of channels. The information indicating the arrangement of the first cell in the first few channels is transmitted to the first trained neural network. 15, or the method of claim 14, further comprising providing the method as part of an input to a network of 19. The method according to any one of claims 1 to 14 or 16 to 18.

20. 15. The method of claim 14, wherein the first trained neural network includes a plurality of convolutional layers. Or a method according to any one of claims 1 to 14 or 16 to 19.

21. The cell typing data includes at least one marker for each of a plurality of cell types. a plurality of marker expression signatures, The step of determining the predicted cell type of the first cell comprises: The marker expression signature is determined by dividing the marker expression signature by at least one of the plurality of marker expression signatures. comparing said predicted cell type to said marker expression signature.

12. The method according to claim 11, or any one of claims 1 to 10 or 12 to 20.

22. The marker expression signature of the first cell is expressed by the plurality of marker expression signatures. the step of comparing to the at least one marker expression signature comprises using a distance measure 21. The method according to claim 21 or any one of claims 1 to 20, which is carried out using degrees. 。

23. The measure of distance may be selected from the group consisting of cosine distance, Euclidean distance, and Manhattan distance.

22. The method of claim 22, or any one of claims 1 to 21, wherein the at least one of

24. A peak associated with each position of the first cell in the plurality of channels. The step of using the offset value includes, for each of the plurality of channels: The arrangement of the first cells in the channel is used to detect pinpoints for the first cells. identifying a set of interfaces; determining the first characteristic for the first cell based on values ​​of pixels in the set of pixels; and determining one of the characteristic values ​​of the characteristic values. Or a method according to any one of claims 10 to 23.

25. The step of determining the feature value for the first cell comprises:

25. The method of claim 24, further comprising determining an average pixel intensity of said values ​​of said pixels; 24. The method according to any one of claims 1 to 23.

26. the location of the first cell indicates the location of a boundary of the first cell; The step of identifying the set of pixels for the first cell comprises at least in part 25. The method of claim 24, further comprising identifying pixels that are essentially inside a boundary of the first cell. Or a method according to any one of claims 1 to 23 or 25.

27. The at least one characteristic of the tissue sample may include a cellular composition of the tissue sample, or both, characterised by claim 1 or any one of claims 2 to 26 The method according to claim 5.

28. The step of determining at least one characteristic relates to a cell type in the tissue sample. The method according to claim 1 or any one of claims 2 to 27, further comprising the step of determining information. method.

29. The step of determining information about the cell types in the tissue sample comprises: 28, or claim 29, comprising identifying one or more cell types present in the 28. The method of any one of claims 1 to 27.

30. The step of determining the information regarding the cell type may include: and identifying a cell type of an individual cell of at least some of the cells in the 30. The method of claim 29 or any one of claims 1 to 28, comprising the steps:

31. Identifying the cell type of said individual cells of said at least some of said cells. The step of determining at least a threshold percentage of said individual cells in said at least one MxIF image The method of claim 30 or any of claims 1 to 29, further comprising the step of identifying a percentage of cell types. The method according to any one of claims 1 to 5.

32. The cell types include endothelial cells, epithelial cells, macrophages, T cells, malignant cells, NK cells, B 29, or also claims 1 to 28, comprising one or more of the following: Or a method according to any one of claims 30 to 31.

33. The T cells include one or more of CD3+ T cells, CD4+ T cells, and CD8+ T cells. 32, or a method according to any one of claims 1 to 31.

34. The step of determining information about cell types in the tissue sample may further comprise determining information about one or more cell types in the tissue sample. or determining the percentage of a plurality of cell types. The method according to any one of claims 1 to 27 or 29 to 33.

35. The step of determining at least one characteristic includes determining a plurality of characteristics.

35. The method of claim 1 or any one of claims 2 to 34.

36. The step of determining at least one characteristic includes determining the spatial distribution of cell types in the tissue sample. The method according to claim 1 or any one of claims 2 to 35, further comprising a step of determining the fabric. Law.

37. The step of determining at least one characteristic comprises: determining spatial information regarding the location of the portion of the 37. The method according to any one of items 2 to 36.

38. The step of determining spatial information includes determining distances between cells of the plurality of groups of cells.

37. The method of claim 37 or any one of claims 1 to 36, comprising the step of:

39. The step of determining spatial information may include determining spatial information of one or more of a group of the plurality of groups of cells. determining one or more regions of the tissue sample that contain a plurality of cells.

38. The method of claim 38 or any one of claims 1 to 37.

40. The step of determining at least one characteristic comprises determining a characteristic of at least one of the plurality of populations of cells.

2. The method of claim 1, further comprising determining morphological information about a number of cells.

39. The method of any one of claims 1 to 39.

41. The step of determining morphological information of at least some of the cells The structure of the cells and / or the morphology of said at least some of said cells. The method according to claim 40 or any one of claims 1 to 39, further comprising the step of determining information to be Method of posting.

42. The step of determining at least one characteristic comprises determining a characteristic of at least one of the plurality of populations of cells. Determining physical information for a number of cells, said physical information comprising cell area, 2. The method of claim 1, further comprising the steps of: measuring at least one of a cell circumference and a cell size; Or a method according to any one of claims 2 to 41.

43. The step of determining at least one characteristic comprises determining a gland in the at least one MxIF image.

43. The method of claim 1 or claim 2 to 42, further comprising determining an acinar mask indicative of the arrangement of the acinar compartments.

2. The method according to any one of claims 1 to 11.

44. The step of determining at least one characteristic includes determining a characteristic within the at least one MxIF image.

44. The method of claim 1 or claim 2 to 43, further comprising the step of determining a stromal mask indicative of the arrangement of the stroma.

2. The method according to any one of claims 1 to 11.

45. The step of determining at least one characteristic of a tumor in the at least one MxIF image includes: The method of claim 1 or claim 2 to 44, further comprising the step of determining a tumor mask indicative of the location of the tumor.

2. The method according to any one of claims 1 to 11.

46. The step of determining the at least one characteristic comprises: determining one or more cell clusters in the tissue sample based on the plurality of groups; Tep and determining a cell cluster mask indicative of an arrangement of said one or more cell clusters; 46. ​​The method of claim 1, or any one of claims 2 to 45, comprising:

47. Determining the one or more cell clusters in the tissue sample includes: a node for each of said at least some cells, and edges between said nodes; generating a graph including the determining features for nodes in the graph; Towards a graph neural network to obtain embedded features in the latent space providing the features for the nodes in the graph as input to Clustering the embedded features to obtain clusters of the nodes. Steps and The one or more cell clusters are determined using the clusters of nodes.

46. ​​The method of claim 46 or any one of claims 1 to 45, comprising the steps of:

48. each cell cluster represents at least a portion of a tissue structure of said tissue sample; The tissue structure comprises mantle tissue, stromal tissue, tumor, follicle, and / or blood vessel. Item 46. The method of any one of claims 1 to 45 or 47.

49. determining whether said one or more cells are positively associated with said first set of cellular characteristics based on said first set of cellular characteristics of said at least some cells; 46, or any of claims 1 to 45 or any of claims 1 to 46, further comprising the step of determining cell clusters.

49. The method according to any one of claims 47 to 48.

50. a node for each of said at least some cells, and edges between said nodes; determining the first set of cell features for each cell by generating a graph including 49. The method of claim 1, further comprising the step of: 。

51. generating said graph based on said at least some cells using triangulation; 50. The method of claim 50 or any one of claims 1 to 49, further comprising:

52. A group of the plurality of groups including the cell, an edge of the node in the graph The first selection of cell features for each node of the graph based on length and mask data. 50, or the method of claim 1 to 49 or 51, further comprising the step of determining the set 2. The method according to any one of claims 1 to 11.

53. providing said graph as an input to a trained graph neural network; 、 From the trained graph neural network, we obtain a set of feature embeddings for each node. and obtaining a set of the results of the measurement.

52. The method of claim 1, wherein

54. The trained graph neural network includes one or more convolutional layers.

51. A method according to claim 50, or any one of claims 1 to 49 or 51 to 53.

55. The set of feature embeddings is the final graph of the trained graph neural network. Claim 50, or claims 1 to 49 or 5, which are generated based on the activation of a rough convolutional layer 55. The method of any one of 1 to 54.

56. The step of determining one or more cell clusters may further include: determining the one or more cell clusters based on the set. Item 50, or any one of claims 1 to 49 or 51 to 55.

57. determining the one or more cell clusters based on the set of feature embeddings of each node; determining the location of the feature in the tissue sample based on the set of feature embeddings for each node; 56. The method of claim 1, further comprising the step of clustering the cells.

13. The method according to claim 1.

58. The tissue sample may be obtained from a subject having, suspected of having, or at risk of having cancer. A method according to claim 1 or any one of claims 2 to 57, which has been previously obtained from a subject. method.

59. The step of acquiring at least one MxIF image of the tissue sample comprises: A step of performing background subtraction on a first channel of the at least one MxIF image.

59. The method of claim 1, or any one of claims 2 to 58, comprising the steps of:

60. The step of performing background subtraction includes: A second trained neural network configured to perform background subtraction. The method according to claim 59 or any one of claims 1 to 58, comprising the step of providing How to.

61. The second trained neural network includes at least 1 million parameters. , claim 60, or a method according to any one of claims 1 to 59.

62. A set of noisy training immunofluorescence images as input images and a set of images with at least some noise The associated output data, including the associated image, is used to 61. The method of claim 60, further comprising the step of training the second trained neural network. Or a method according to any one of claims 1 to 59 or 61.

63. the at least one MxIF image comprises a plurality of immunofluorescence images; obtaining said information indicative of said location of said cells within said at least one immunofluorescence image; A step of analyzing the plurality of immunofluorescence images to identify one or more damaged areas of the tissue sample. The method according to claim 1 or any one of claims 2 to 62, further comprising the step of identifying the portion How to.

64. The step of analyzing the plurality of immunofluorescence images may include analyzing the plurality of immunofluorescence images to identify differences between the immunofluorescence images. and detecting the plurality of immunofluorescence signals using a third trained neural network configured to:

63. The method according to claim 1, further comprising the step of processing the image. How to.

65. a training set of immunofluorescence images including pairs of immunofluorescence images of the same marker as the input image; The tissue sample further comprises information indicating whether at least a portion of the tissue sample is damaged. training the third trained neural network using the output data.

64. The method of claim 64, or any one of claims 1 to 63, further comprising:

66. a corresponding portion of each of at least two immunofluorescence images of the plurality of immunofluorescence images; into the third trained neural network, and wherein both immunofluorescence images contain images of the same marker. from the third trained neural network, and obtaining a classification of the first to third classes.

66. The method of any one of claims 65 to 65.

67. The at least one classification may include a first classification of no cells, a second classification of undamaged cells, and / or a set of classifications including a third classification of damaged cells. Item 66. The method of any one of claims 1 to 65.

68. The set of classifications includes, for each classification, an association The reliability and selecting a final classification from said set of classifications for said portion based on said confidence.

67. The method of claim 67 or any one of claims 1 to 66, comprising:

69. A final classification of the portion and a plurality of other portions of the at least two immunofluorescence images. 68, or claim 69, further comprising generating a patch mask indicative of said final classification.

68. The method of any one of claims 1 to 67.

70. determining a portion of a segmentation mask for the tissue sample based on the patch mask; and removing said tissue sample cells associated with said removed portions from said plurality of groups. The method of claim 69 or any of claims 1 to 68, further comprising the step of: The method according to any one of claims 1 to 5.

71. The third trained neural network includes one or more convolutional layers.

64. The method of claim 64, or any one of claims 1 to 63 or 64 to 70.

72. The third trained neural network includes at least 5 million parameters. 71, or a method according to any one of claims 1 to 70.

73. The third trained neural network determines whether different portions of the tissue sample are 64, or claim 1, configured to output a confidence score including at least one classification. 63 or 65 to 72.

74. The at least one classification may include a first classification of no cells, a second classification of undamaged cells, a classification that includes a third classification of damaged cells, or some combination of these.

73. The method of claim 73, or any one of claims 1 to 72, comprising a set.

75. obtaining said information indicative of said location of said cells within said at least one MxIF image; The top a fourth trained neural network for at least one channel of the at least one MxIF image; applying a network to generate said information indicative of said location of said cells.

75. A method according to any one of claims 1 or 2 to 74.

76. The fourth trained neural network may be based on the U-Net architecture or domain-based 75, or the method of claim 74, implemented using a convolutional neural network architecture of Or a method according to any one of claims 1 to 74.

77. The fourth trained neural network includes at least 1 million parameters. 76, or a method according to any one of claims 1 to 75.

78. A set of training immunofluorescence images of tissue samples as input images and the location of cells in the input images. and an associated output image including information indicating the location of the fourth trained neural network. 75, or any of claims 1 to 74, further comprising the step of training the neural network. or a method according to any one of claims 76 to 77.

79. grouping said at least some of said cells into said plurality of populations. the at least some of the cells based on the determined characteristic values. performing cell clustering to group the cells into the plurality of groups. 1, or the method of any one of claims 2 to 78.

80. The step of performing cell clustering includes hierarchical clustering, density-based clustering, and the like. clustering, k-means clustering, self-organizing map clustering, or minimum clustering 79 or claim 1 to 78, comprising the step of performing panning tree clustering.

2. The method according to claim 1 .

81. grouping said at least some of said cells into said plurality of populations. The top Analyzing the determined characteristic values ​​of at least some of the cells determining a relationship between The plurality of groups are separated by a step of dividing the plurality of groups into groups, each cell in the group being separated by a step of dividing the plurality of groups into groups, each cell ... and determining based on the determined relationship that the characteristic value indicates the relationship.

81. The method of claim 1, or any one of claims 2 to 80.

82. The step of determining a relationship between at least some of the cells comprises: determining similarity between feature values ​​of at least some of the cells; 81. The method of claim 81, or any one of claims 2 to 80, comprising:

83. The step of determining a relationship between at least some of the cells comprises: comparing said characteristic values ​​to known cell typing data to identify said at least one of said cells; determining the cell type for each of at least some of the cells, 83. The method of any one of claims 2 to 80 or 82.

84. The step of determining at least one characteristic is described in any one of claims 1 to 83.

84. The method of claim 1, or any one of claims 2 to 83, comprising determining a plurality of characteristics of The method described above.

85. at least one computer hardware processor; At least one non-transitory computer-readable storage medium storing processor-executable instructions. and wherein the processor executable instructions are when executed by the at least one computer hardware For the hardware processor, acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; obtaining information indicative of an arrangement of cells within the MxIF image; The plurality of groups of cells in the at least one MxIF image are at least in part identified by: The at least one MxIF image and the information indicative of cell location are used to identify the cells. determining feature values ​​for at least some of the cells in The determined characteristic values ​​are used to detect at least some of the cells. into said plurality of groups. and identifying the determining at least one characteristic of the tissue sample using the plurality of groups; A system that executes the above.

86. At least one non-transitory computer-readable storage medium storing processor-executable instructions. the processor executable instructions are implemented on at least one computer hardware When executed by the processor, said at least one computer hardware To the app processor, acquiring at least one multiplex immunofluorescence (MxIF) image of the same tissue sample; obtaining information indicative of an arrangement of cells within the MxIF image; The plurality of groups of cells in the at least one MxIF image are at least in part identified by: showing said at least one MxIF image and an arrangement of cells within said at least one MxIF image; determining feature values ​​for at least some of the cells using the information; Determine, The determined characteristic values ​​are used to detect at least some of the cells. into said plurality of groups. and identifying the determining at least one characteristic of the tissue sample using the plurality of groups; A non-transitory computer-readable storage medium for causing the execution of the above.

87. Using at least one computer hardware processor, acquiring at least one MxIF image of the tissue sample; Obtaining information indicative of an arrangement of at least one cell within said at least one MxIF image. Steps and A plurality of markers expressed in said at least one MxIF image and said at least and detecting a pre-existing region in the tissue sample based on the information indicating the location of the cells within a single MxIF image. determining a marker expression signature for the cell; The marker expression signature is compared to at least one of a plurality of different cell types. and comparing the cell typing data comprising the marker expression signature to the cell. determining a type of the

88. determining at least one characteristic of said tissue sample based on said cell type of said cells.

88. The method of claim 87, further comprising:

89. at least one computer hardware processor; At least one non-transitory computer-readable storage medium storing processor-executable instructions. and wherein the processor executable instructions are when executed by the at least one computer hardware For the hardware processor, acquiring at least one MxIF image of the tissue sample; Obtaining information indicative of an arrangement of at least one cell within said at least one MxIF image. Steps and A plurality of markers expressed in said at least one MxIF image and said at least and detecting a pre-existing region in the tissue sample based on the information indicating the location of the cells within a single MxIF image. determining a marker expression signature for the cell; The marker expression signature is compared to at least one of a plurality of different cell types. and comparing the cell typing data comprising the marker expression signature to the cell. and determining the type.

90. At least one non-transitory computer-readable storage medium storing processor-executable instructions. the processor executable instructions are implemented on at least one computer hardware When executed by the processor, said at least one computer hardware To the app processor, acquiring at least one MxIF image of the tissue sample; Obtaining information indicative of an arrangement of at least one cell within said at least one MxIF image. Steps and A plurality of markers expressed in said at least one MxIF image and said at least and detecting a pre-existing region in the tissue sample based on the information indicating the location of the cells within a single MxIF image. determining a marker expression signature for the cell; The marker expression signature is compared to at least one of a plurality of different cell types. and comparing the cell typing data comprising the marker expression signature to the cell. and determining the type.

Images