Cross-modality pixel alignment and cell-to-cell registration across various imaging modalities

Abstract

A method implemented by one or more computer devices includes receiving a plurality of images of a set of tissue cells, the plurality of images including a first image including a first visualization modality and a second image including a second visualization modality. The method includes identifying a first tissue cell of the set of tissue cells in the first image and a first tissue cell in the second image, and performing a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image. The cell-to-cell registration process includes matching the first tissue cell identified in the first image to the first tissue cell identified in the second image. The method includes classifying the first tissue cell into a phenotype based on the cell-to-cell registration process, the phenotype being in part indicative of a disease condition.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 407,077, filed September 15, 2022, entitled "Cross-Modality Pixel Alignment and Cell-to-Cell Registration Across Various Imaging Modalities," the entire contents of which are incorporated herein by reference.

[0002] This application relates generally to molecular annotation, and more particularly to resolving molecular annotation discrepancies utilizing cross-modality pixel alignment and inter-cell registration across various imaging modalities. [Background technology]

[0003] Digital pathology typically involves visualizing and analyzing digitized slides to determine whether variations in tissue cells are due to disease, toxicity, and / or natural processes. Visualization of tissue cells can generally involve modalities consisting of dye-based visualization modalities, molecular-based visualization modalities, or probe-based visualization modalities. For example, dye-based visualization, such as hematoxylin and eosin (H&E), can utilize the chemical properties of dye molecules to bind to specific tissue cells, such as mucin, fat, and proteins. Similarly, molecular-based visualization, such as immunohistochemistry (IHC), can utilize antibodies to bind to protein epitope targets with high specificity, for example, by producing a brownish stain using 3,3'-diaminobenzidine or other similar organic compounds. IHC can also include other visualization colors, which can be combined, for example, to visualize multiple different biomarkers.

[0004] Other molecular-based visualization may include, for example, immunofluorescence (IF), which may utilize fluorescent dyes to multiplex and highlight multiple different target cells (e.g., antibodies). Similarly, other molecular-based visualization may include probe-based techniques that may be utilized to visualize messenger RNA (mRNA), microRNA (miRNA), and DNA within one or more tissue cells using, for example, brightfield or IF visualization modalities. Furthermore, more recent visualization modalities may include, for example, hyperspectral imaging, which may be utilized to distinguish between dozens of different biomarkers labeled with different heavy metals and visualized using imaging mass cytometry (IMC) or mass spectrometry (MS).

[0005] Therefore, as can be understood from the above, different visualization modalities may often be deployed to visualize and analyze the same target tissue cells based on the information a pathologist, scientist, or clinician is seeking to ascertain. For example, H&E staining may be well suited to broadly visualizing specific tissue cells, such as cancer cells and proteins, while multiplex IHC (mxIHC) or multiplex IF (mxIF) may be well suited to visualizing and identifying specific proteins and tissue cells. Various tissue cells (or various biological features) appear differently in different visualization modalities. For example, tissue cell images captured using one visualization modality often appear significantly different from images of the same tissue cells captured using another visualization modality, making it difficult or even counterintuitive for either a human or a computationally based model to recognize that the images depict the same tissue cells.

[0006] For example, some visualization modalities may include spatial features for pathologists, scientists, and / or clinicians to easily identify or annotate target tissue cells, while other visualization modalities may require computational-based modeling to identify and classify target tissue cells. However, without verifiable annotations of the target tissue cells for comparison and validation, even computational-based modeling may not accurately identify and classify the target tissue cells. Furthermore, even conventional image registration techniques may be suitable for registering certain two-dimensional (2D) digital images, but such conventional image registration techniques typically perform poorly when utilized to register individual tissue cells and / or other tissue cell features. Furthermore, these conventional image registration techniques may typically require that the 2D images being registered include the same imaging modality, and thus are technically inadequate for registering and tracking individual tissue cells across various imaging or visualization modalities. Therefore, it would be useful to provide an improved technique for tracking target tissue cells across different visualization modalities. Summary of the Invention

[0007] Embodiments of the present disclosure relate to one or more computing devices, methods, and non-transitory computer-readable media that can perform a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities. The identification and phenotyping of tissue cells may use one visualization modality. One visualization modality may be used as verifiable ground truth data for training one or more machine learning models to classify and phenotype tissue cells captured using another visualization modality. That is, according to embodiments disclosed herein, ground truth data for training one or more machine learning models to classify and phenotype tissue cells for a particular visualization modality (e.g., tissue cells that are not readily observable) may be generated by relying on spatial features that may be readily observable by human expert (e.g., pathologists, scientists, clinicians, or other medical and scientific experts) observation for a different visualization modality.

[0008] Indeed, by utilizing the cross-modality cell-to-cell registration process, one or more tissue cells or populations of tissue cells can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists), or other medical and scientific professionals can more easily classify and phenotype immune cells (e.g., cancer cells, macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, natural killer (NK) cells, fibroblasts, etc.). This may result in earlier detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients with both early and advanced stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). This cross-modality cell-cell registration technology may further improve and enhance the efficacy of targeted immunotherapies (e.g., monoclonal antibodies (mAbs), T cell-engaging bispecific antibodies (bsAbs), antibody-drug conjugates (ADCs), etc.) in patients with NHL, including FL and DLBCL.

[0009] In certain embodiments, one or more computing devices may receive several images of a set of tissue cells, where the several images may include at least a first image including a first visualization modality and a second image including a second visualization modality. For example, in certain embodiments, the first visualization modality and the second visualization modality may be independently acquired by a whole-slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality. The whole-slide imaging modality may be selected from bright-field or fluorescence imaging. The microscopy modality may be selected from bright-field microscopy, fluorescence microscopy, confocal microscopy, high-content screening (HCS) microscopy, or composite image generation. The non-optical imaging modality may be selected from imaging mass cytometry (IMC) or myocardial perfusion imaging (MIBI). In some embodiments, the first visualization modality or the second visualization modality includes a dye-based visualization modality. In some embodiments, the dye-based visualization modality may be selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining. In some embodiments, the dye-based visualization modality includes hematoxylin and eosin (H&E) staining. In some embodiments, the first visualization modality or the second visualization modality may include immunostaining.

[0010] In certain embodiments, the one or more computing devices may identify regions of pixels in the first image and the second image, each corresponding to a respective tissue cell of the set of tissue cells. For example, in some embodiments, before identifying the regions of pixels in the first image and the second image, the one or more computing devices may align the images so that at least the first image and the second image are vertically stacked. For example, in some embodiments, identifying the regions of pixels in the first image and the second image may include performing nuclear segmentation of the regions of pixels in the first image and the second image to segment each tissue cell of the set of tissue cells. In certain embodiments, the one or more computing devices perform a cell-to-cell registration process based on the identified regions of pixels, the cell-to-cell registration process including matching a first region of pixels corresponding to the first tissue cell in the first image with a second region of pixels corresponding to the first tissue cell in the second image.

[0011] For example, in certain embodiments, performing the cell-to-cell registration process may include performing a scale-invariant Fourier transform (SIFT) registration of the first image and the second image; performing a tile-level registration of the first image and the second image, where the tile-level registration includes a matrix transformation of the SIFT registration of the first image and the second image; and performing tile-level segmentation of the tile-level registered first image and the second image. In some embodiments, the tile-level segmentation is performed to segment each tissue cell of a set of tissue cells in the first image and the second image. In certain embodiments, performing the cell-to-cell registration process may further include performing object-level cell registration based on the tile-level segmented tissue cells. In some embodiments, the object-level cell registration may be performed to match a first tissue cell in the first image with a first tissue cell in the second image. In certain embodiments, prior to performing the cell-to-cell registration process, one or more computing devices may extract one or more features from the first image and the second image based on the identified regions of pixels. In some embodiments, one or more features are used to identify a first tissue cell in the first image and a first tissue cell in the second image.

[0012] In certain embodiments, the one or more computing devices may then phenotype the first tissue cell based on the cell-cell registration process. In some embodiments, the phenotype may at least partially indicate a disease condition. For example, in some embodiments, phenotyping the first tissue cell may include classifying the first tissue cell as a cancer cell, a plasma cell, a lymphocyte, a macrophage, or a fibroblast based on one or more spatial features. In some embodiments, phenotyping the first tissue cell may include classifying the first tissue cell as a cancer cell, a macrophage, a regulatory T cell (Treg), a CD8 cell, a B lymphocyte, a natural killer (NK) cell, or a fibroblast based on one or more molecular annotations. In some embodiments, the disease condition may include a non-Hodgkin's lymphoma disease condition, which may include follicular lymphoma (FL) or diffuse large B-cell lymphoma (DLBCL). In certain embodiments, the one or more computing devices may then generate a phenotyping table based on the phenotypic classification of the first tissue cell.

[0013] In certain embodiments, during the training phase, the one or more computing devices may determine a phenotypic class label for the first tissue cell based on one or more spatial features associated with a first region of pixels corresponding to the first tissue cell, and determine a correspondence between the first region of pixels corresponding to the first tissue cell and a second region of pixels corresponding to the first tissue cell based on a cell-to-cell registration process. Embodiments of the present disclosure include training a model based on 1) the second region of pixels corresponding to the first tissue cell, and 2) the determined phenotypic class label for the first tissue cell. In certain embodiments, during the inference phase, the one or more computing devices may input a third image of the plurality of images to the trained model. In some embodiments, the third image comprises a second visualization modality. In certain embodiments, the one or more computing devices may utilize the trained model to identify a region of pixels in the third image corresponding to the first tissue cell and output a predicted phenotypic class label for the first tissue cell in the third image. In some embodiments, the predicted phenotypic class label for the first tissue cell may correspond to the determined phenotypic class label for the first tissue cell.

[0014] In some embodiments, the model may include one or more deep neural networks (DNNs). In certain embodiments, the one or more computing devices may determine ground truth data for training the model by mapping a phenotypic classification of a first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image. In some embodiments, at least a subset of the ground truth data may include molecularly annotated ground truth data. In other embodiments, at least a subset of the ground truth data may include human-annotated ground truth data. In certain embodiments, the one or more computing devices may use a phenotypic analysis table to map the phenotypic classification of the first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image. [Brief explanation of the drawings]

[0015] One or more of the drawings included herein are colored in accordance with 37 CFR § 1.84. Color drawings are necessary to illustrate the invention. More specifically, Figures 1B, 1C, 3B, 3D, and 4A-7 are one or more high-resolution tissue cell images captured using various visualization modalities, all of which colors play a major role in enabling one of ordinary skill in the art to understand the invention, and such color drawings are the only practical medium for disclosing patentable subject matter.

[0016] [Figure 1A] 1 illustrates an exemplary network of interacting computer systems including a cell-to-cell registration system.

[0017] [Figure 1B] 1 illustrates an intercellular registration system workflow for performing a cross-modality intercellular registration process.

[0018] [Figure 1C] We demonstrate the implementation of a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities.

[0019] [Figure 2A] 1 shows a flow diagram of a method for providing a cross-modality cell-to-cell registration process for identifying, phenotyping, and tracking tissue cells captured utilizing various visualization modalities.

[0020] [Figure 2B] 1 shows another flow diagram of a method for providing a cross-modality cell-to-cell registration process for identifying, phenotyping, and tracking tissue cells captured utilizing various visualization modalities.

[0021] [Figure 3A] 1 shows a flow diagram of a method for training one or more machine-learning models to classify and phenotype tissue cells in one visualization modality image using class labels from different visualization modality images as ground truth.

[0022] [Figure 3B] We present a practical example of training one or more machine learning models to classify and phenotype tissue cells within a single visualization modality image using class labels from different visualization modality images as ground truth.

[0023] [Figure 3C] FIG. 1 shows a flow diagram of a method for utilizing one or more machine-learning models trained to classify and phenotype tissue cells.

[0024] [Figure 3D] An implementation example is presented that utilizes one or more machine learning models trained to classify and phenotype tissue cells.

[0025] [Figure 4A-4B] Annotation of immune cells on whole slide images in more than five classes of immune cells is shown.

[0026] [Figure 4C-4D] Immune cell annotation and phenotyping table for more than seven classes of immune cells.

[0027] [Figures 5A-5C] 10 illustrates one or more example graphs or implementations of example tissue cell matching.

[0028] [Figure 6] 1 illustrates one or more example graphs or implementations of a cross-modality intercellular registration process.

[0029] [Figure 7] 1 illustrates one or more example graphs or implementations of a cross-modality intercellular registration process.

[0030] [Figure 8] 1 shows a diagram of an exemplary artificial intelligence (AI) architecture included as part of a network of interacting computer systems. DETAILED DESCRIPTION OF THE INVENTION

[0031] Embodiments of the present disclosure relate to one or more computing devices, methods, and non-transitory computer-readable media that can perform a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities. Such a registration process can enable improved visualization of various cells. The disclosed process can further enable tissue cell identification and phenotyping using one visualization modality to be utilized as verifiable ground truth data for training one or more machine learning models to classify and phenotype tissue cells captured using another visualization modality. That is, according to embodiments disclosed herein, ground truth data for training one or more machine learning models to classify and phenotype tissue cells for a particular visualization modality (e.g., tissue cells that cannot be readily ascertained by observation) can be generated, for example, by relying on spatial features that may be readily ascertainable by observation by a human expert (e.g., a pathologist, scientist, clinician, or other medical and scientific expert) for a different visualization modality.

[0032] Indeed, by utilizing the cross-modality cell-to-cell registration process, one or more tissue cells or populations of tissue cells can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists), or other medical and scientific professionals can more easily classify and phenotype immune cells (e.g., macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, natural killer (NK) cells, etc.). This may result in earlier detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients with both early and advanced stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). This cross-modality cell-to-cell registration process may further improve and enhance the efficacy of targeted immunotherapies (e.g., monoclonal antibodies (mAbs), T-cell-engaging bispecific antibodies (bsAbs), antibody-drug conjugates (ADCs), etc.) for NHL patients, including FL and DLBCL.

[0033] As used herein, "phenotype," "phenotyping," or "phenotyped" may be broadly understood to refer to the identity of one or more tissue cells or immune cells, the cellular state of one or more tissue cells or immune cells, the cellular program of one or more tissue cells or immune cells, the spatial location of one or more tissue cells or immune cells, the physical appearance of one or more tissue cells or immune cells, the number of one or more tissue cells or immune cells, or other similar human- or machine-perceivable characteristics or properties of one or more tissue cells or immune cells that can be utilized to classify one or more tissue cells or immune cells across various visualization modalities.

[0034] FIG. 1A illustrates a network 100A of interacting computer systems that may be suitable for performing a cross-modality cell-to-cell registration process for identifying, phenotypic interpretation, and tracking tissue cells captured using various visualization modalities, according to an embodiment of the present disclosure.

[0035] In certain embodiments, the whole slide imaging system 101 may generate one or more whole slide images or histopathology images corresponding to a particular sample. For example, an image generated by the whole slide imaging system 101 may include a stained section of a biopsy sample. As another example, an image generated by the whole slide imaging system 101 may include a slide image of a liquid sample (e.g., a blood film). As another example, an image generated by the whole slide imaging system 101 may include a fluorescence microscopy slide image, such as a slide image showing fluorescence in situ hybridization (FISH) after a fluorescent probe has bound to a target DNA or RNA sequence.

[0036] Some types of samples (e.g., samples containing tissue) may be processed by the sample preparation system 105 to fix and / or embed the sample. The sample preparation system 105 may facilitate infiltrating the sample with a fixating agent (e.g., a liquid fixing agent such as a formaldehyde solution) and / or an embedding substance (e.g., histological wax). For example, the sample fixation subsystem may fix the sample by exposing the sample to a fixative for at least a threshold amount of time (e.g., at least 3 hours, at least 6 hours, or at least 13 hours). The dehydration subsystem may dehydrate the sample (e.g., by exposing the fixed sample and / or portions of the fixed sample to one or more ethanol solutions) and potentially clear the dehydrated sample using a clearing intermediate (e.g., including ethanol and histological wax). The sample embedding subsystem may infiltrate the sample with heated (e.g., therefore liquid) histological wax (e.g., one or more times during a corresponding predefined time period). The histological wax may include paraffin wax and potentially one or more resins (e.g., styrene or polyethylene). The sample and wax may then be cooled, and the wax-infiltrated sample may then be blocked out.

[0037] In certain embodiments, the sample slicer 107 may receive a fixed and embedded sample and generate a set of sections. The sample slicer 107 may cool or expose the fixed and embedded sample to cold temperatures. The sample slicer 107 may then cut the cooled sample (or a trimmed version thereof) to create a set of sections. Each section may have a thickness that is (for example) less than 100 μm, less than 50 μm, less than 10 μm, or less than 5 μm. Each section may have a thickness that is (for example) greater than 0.1 μm, greater than 1 μm, greater than 2 μm, or greater than 4 μm.

[0038] In certain embodiments, the automated staining system 109 may facilitate staining of one or more of the sample sections by exposing each section to one or more stains. Each section may be exposed to a predefined amount of stain for a predefined period of time. In some cases, a single section is exposed to multiple stains simultaneously or sequentially. In certain embodiments, each of the one or more stained sections may be presented to an image scanner 115, which may capture a digital image of the section. The image scanner 115 may include a microscope camera. The image scanner 115 may capture digital images at multiple magnification levels (e.g., using a 10x objective, a 20x objective, a 40x objective, etc.). Image manipulation may be used to capture selected portions of the sample at a desired range of magnifications.

[0039] It will be appreciated that one or more components of the whole slide imaging system 101 may, in some cases, operate in conjunction with a human operator. For example, the human operator may move samples through various subsystems (e.g., of the sample preparation system 105 or of the whole slide imaging system 101) and / or initiate or terminate the operation of one or more subsystems, systems, or components of the whole slide imaging system 101. In certain embodiments, the whole slide imaging system 101 may transmit images generated by the image scanner 115 to the cell-to-cell registration system 121 in accordance with the techniques of the present disclosure. Although not shown, other intermediate devices (e.g., a data store on a server connected to the whole slide imaging system 101 or the whole slide image processing system 103) may also be used.

[0040] Network 100A can be used in a variety of situations where scanning and evaluating histopathology images, such as whole slide images, is an essential component of the work. Whole slide image processing system 103 can process histopathology images, including whole slide images, to classify digital pathology images and generate annotations for the digital pathology images and related outputs. Tile generation module 111 can define a set of tiles or patches for each digital pathology image. To define the set of tiles or patches, tile generation module 111 can segment the digital pathology image into the set of tiles or patches. Tile generation module 111 can further define tile or patch sizes depending on the type of abnormality being detected. For example, tile generation module 111 can be configured with knowledge of the type of tissue abnormality the entire slide image processing system 103 is searching for and can customize tile or patch sizes depending on the tissue abnormality to optimize detection.

[0041] In certain embodiments, the tile embedding module 117 may generate an embedding of each tile or patch in a corresponding feature embedding space. The embedding may be represented by the entire slide image processing system 103 as a feature vector for the tile or patch. The tile embedding module 117 may utilize a neural network (e.g., one or more convolutional neural networks (CNNs)) to generate the feature vector representing each tile or patch of the image. In certain embodiments, the tile embedding neural network may be based on a residual neural network (ResNet) image classification network trained on a dataset based on natural (e.g., non-medical) images, such as the ImageNet dataset. In other embodiments, the tile embedding network utilized by the tile embedding module 117 may be an embedding network customized to process multiple tiles or patches of large-format images, such as digital pathology whole-slide images.

[0042] In certain embodiments, whole slide image access module 113 may manage requests to access whole slide images from other modules of whole slide image processing system 103 and cell-to-cell registration system 121. For example, in some embodiments, whole slide image access module 113 may receive requests to access, for example, multiplex immunofluorescence (mxIF) images, hematoxylin and eosin (H&E) images, or other imaging or visualization modalities for providing to cell-to-cell registration system 121. For example, in one embodiment, output generation module 119 of whole slide image processing system 103 may generate output corresponding to the accessed or requested mxIF and H&E images. Output generation module 119 may then provide the output to cell-to-cell registration system 121 for performing a cross-modality cell-to-cell registration process for identifying, phenotypic interpretation, and tracking tissue cells captured by the mxIF and H&E images, according to embodiments of the present disclosure.

[0043] 1B illustrates a system workflow 100B for performing a cross-modality cell-to-cell registration process for identifying, phenotypic interpreting, and tracking tissue cells captured using various visualization modalities, according to an embodiment of the present disclosure. In certain embodiments, the system workflow 100B may begin with accessing a multiplex immunofluorescence (mxIF) image 102, an mxIF image 104, and a hematoxylin and eosin (H&E) image 106. For example, in certain embodiments, the mxIF image 102, the mxIF image 104, and the H&E image 106 may each include one or more tissue cells. Specifically, according to embodiments of the present disclosure, the mxIF image 102, the mxIF image 104, and the H&E image 106 may each include the same exact tissue cells captured by different visualization modalities. Indeed, while the present cross-modality cell-to-cell registration techniques may be described herein primarily with respect to mxIF and H&E visualization modalities, it should be understood that the present cross-modality cell-to-cell registration techniques may be applied to any of a variety of visualization modalities, such as immunohistochemistry (IHC), fluorescence in situ hybridization (FISH), brightfield microscopy, imaging mass cytometry (IMC), myocardial perfusion imaging (MIBI), among a variety of other visualization modalities.

[0044] For example, in one embodiment, mxIF image 102 (e.g., "IF1") and mxIF image 104 (e.g., "IF2") may each include molecular-based visualization such as multiplexed immunofluorescence (mxIF), which may utilize fluorescent dyes to multiplex and highlight multiple different target cells. For example, in some embodiments, mxIF image 102 (e.g., "IF1") and mxIF image 104 (e.g., "IF2") may each include mxIF images of tissue cells captured utilizing a cyclic immunofluorescence process, which may be utilized to generate highly multiplexed images using a cyclic process (e.g., cycles) in which IF images of the same tissue cells are repeatedly collected and ultimately assembled.

[0045] Thus, in one embodiment, mxIF image 102 (e.g., "IF1") may represent an mxIF image captured in the first cycle of the cyclic process, and mxIF image 104 (e.g., "IF2") may represent an mxIF image captured in the second cycle of the cyclic process. Similarly, in one embodiment, H&E image 106 may include dye-based visualization, such as hematoxylin and eosin (H&E), which may utilize the chemical properties of dye molecules to bind to one or more specific tissue cells. In certain embodiments, as will be further understood below with respect to Figures 3A-3D, mxIF image 102 (e.g., "IF1") and mxIF image 104 (e.g., "IF2") may include one or more spatial features (e.g., any features suitable for informing or confirming the phenotype of a cell by its spatial organization relative to neighboring cells or its location within a tissue or region of a tissue) suitable to enable a pathologist, scientist, or clinician (e.g., oncologist) to manually label one or more matching tissue cells using a cell-to-cell registration process to corresponding one or more tissue cells in H&E image 106, which may not include spatial features.

[0046] In certain embodiments, the mxIF image 102, the mxIF image 104, and the H&E image 106 may then be input to a registration system 108. For example, in certain embodiments, the registration system 108 may include any process that can be utilized to vertically align the mxIF image 102, the mxIF image 104, and the H&E image 106 into a vertical stack 110A (e.g., pixel-to-pixel registration by cross-correlation or whole slide image registration). In some embodiments, the registration system 108 may perform a scale-invariant Fourier transform (SIFT) registration of the mxIF image 102, the mxIF image 104, and the H&E image 106. In another embodiment, the registration system 108 may perform a tile-level registration of the mxIF image 102, the mxIF image 104, and the H&E image 106. In certain embodiments, the tile-level registration may be performed after or in conjunction with the SIFT registration. For example, in one embodiment, the tile-level registration may include a matrix transformation of the SIFT registration of the mxIF image 102 , the mxIF image 104 , and the H&E image 106 .

[0047] In certain embodiments, the system workflow 100B may continue with the H&E image 106 being scaled and rotated. For example, in one embodiment, the scaled and rotated H&E image 106 may be placed on top of the updated vertical stack 110B. In certain embodiments, the system workflow 100B may continue with image segmentation 112 of the updated vertical stack 110B. For example, in some embodiments, the image segmentation 112 may include nuclear segmentation, which may be utilized to segment pixels of the mxIF image 102, the mxIF image 104, and the H&E image 106 in the updated vertical stack 110B, e.g., along the boundaries of the nuclei of individual tissue cells. In another embodiment, the image segmentation 112 may include semantic segmentation (e.g., pixel-by-pixel image segmentation), which may be utilized to segment and annotate pixels of the mxIF image 102, the mxIF image 104, and the H&E image 106 in the updated vertical stack 110B, e.g., on a pixel-by-pixel basis. In certain embodiments, the system workflow 100B may then proceed to perform a cross-modality intercellular registration process 114 in accordance with the techniques of this disclosure.

[0048] In certain embodiments, the cross-modality cell-to-cell registration process 114 may include object-level cell registration to match individual tissue cells across, for example, the mxIF images 102, 104, and the H&E images 106 in the segmented and updated vertical stack 110B. For example, in some embodiments, the object-level cell registration may include, for example, multi-directionally matching polygons (e.g., two-dimensional (2D) or three-dimensional (3D) polygons) based on identifying overlapping and / or intersecting tissue cells across the mxIF images 102, 104, and the H&E images 106 in the segmented and updated vertical stack 110B. In some embodiments, polygons with multiple matches may also be tracked and recorded (e.g., stored in the phenotyping table 118).

[0049] In certain embodiments, based on the cross-modality cell-to-cell registration process 114, the system workflow 100B may then proceed to perform a phenotyping process 116 of one or more matched tissue cells. For example, in some embodiments, the phenotyping process 116 may include classifying the one or more matched tissue cells into one or more classes of immune cells, such as macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, natural killer (NK) cells, etc. In certain embodiments, the system workflow 100B may then proceed to storing the aforementioned data in a phenotyping table 118. For example, in some embodiments, the phenotyping table 118 may include records of the matched and identified tissue cells determined based on the cross-modality cell-to-cell registration process 114, for example. In certain embodiments, the phenotyping table 118 may then be utilized to label images 120 (e.g., H&E images), which may then be utilized in downstream tasks to train one or more machine-learning models and classify various immune cells (e.g., macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, etc.) in accordance with the present embodiments. For example, in one embodiment, the phenotyping table 118 may include immune cell phenotypes, immune cell activation states, image identification or labels, and tissue cell shapes and locations of interest.

[0050] 1C illustrates an example implementation 100C of a cross-modality cell-to-cell registration process for identifying, phenotyping, and tracking tissue or immune cells captured using various visualization modalities, according to an embodiment of the present disclosure. As shown, in one embodiment, the example implementation 100C of the cross-modality cell-to-cell registration process may be described with respect to an mxIF image 122 and an H&E image 124. As illustrated, the mxIF image 122 may be passed to one or more machine learning models 126, which may be utilized, for example, to segment the mxIF image 122, extract one or more features of interest corresponding to one or more tissue cells, and classify and phenotype the one or more tissue cells. For example, in some aspects, the one or more tissue cells may include one or more populations of immune cells (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.). In certain embodiments, one or more classified and phenotyped tissue cells in the mxIF image 122 can then be matched to one or more corresponding tissue cells in the H&E image 124 using a cell-to-cell registration model 128.

[0051] Specifically, according to embodiments disclosed herein, cell-to-cell registration model 128 may perform a cell-to-cell registration process suitable for matching one or more populations of immune cells identified in mxIF image 122 with corresponding immune cells in H&E image 124. H&E image 124 may then be utilized as ground truth data to train one or more machine learning models for classifying immune cells or other tissue cells. Specifically, as will be further understood with respect to FIGS. 3A-3D and 4A-4D , one or more populations of immune cells identified in mxIF image 122 may be classified and phenotyped, for example, by a human annotator (e.g., a scientist, pathologist, clinician, or other medical or scientific professional) based on one or more spatial features (e.g., any feature suitable for informing or confirming the phenotype of a cell by its spatial organization relative to neighboring cells or its location within a tissue or region of a tissue). Thus, by then matching one or more populations of immune cells to corresponding immune cells in the H&E image 124, the classification and phenotyping can be verifiably trusted (e.g., 90%-100% confidence score) as ground truth data for accurately training one or more machine learning models to classify immune cells or other tissue cells in the H&E image 124.

[0052] 2A illustrates a flow diagram of a method 200A for providing a cross-modality cell-to-cell registration process for identifying, phenotypic interpreting, and tracking tissue cells captured using various visualization modalities, according to an embodiment of the present disclosure. The method flow diagram 200A may be implemented using one or more processing device networks 100A, which may include hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or any other processing device suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or any combination thereof.

[0053] Method 200A may begin at block 202 by one or more processing devices receiving a plurality of images of a set of tissue cells, the plurality of images including at least a first image including a first visualization modality and a second image including a second visualization modality. For example, in one embodiment, the first image may include an mxIF visualization modality, and the second image may include an H&E visualization modality. Method 200A may include block 204, in which the one or more processing devices identify a first tissue cell of the set of tissue cells in the first image and a first tissue cell in the second image. Method 200A may also include block 206, in which the one or more processing devices perform a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image. For example, in some embodiments, the cell-to-cell registration process may include matching the first tissue cell in the first image with the first tissue cell in the second image. Method 200A may further include block 208, where the one or more processing devices classify the first tissue cells into a phenotype based on the cell-to-cell registration process, the phenotype being at least partially indicative of a disease condition. For example, in certain embodiments, the cell-to-cell registration process may include matching one or more populations of immune cells identified in the mxIF image with corresponding immune cells in the H&E image.

[0054] As described above, by utilizing a cross-modality cell-to-cell registration process, one or more tissue cells or populations of tissue cells can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists), or other medical and scientific professionals can more easily classify and phenotype immune cells (e.g., macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, natural killer (NK) cells, etc.). This may result in earlier detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients with both early and advanced stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). This cross-modality cell-cell registration technology may further improve and enhance the efficacy of targeted immunotherapies (e.g., monoclonal antibodies (mAbs), T cell-engaging bispecific antibodies (bsAbs), antibody-drug conjugates (ADCs), etc.) for patients with NHL, including FL and DLBCL.

[0055] 2B shows another flow diagram of a method 200B for providing a cross-modality cell-to-cell registration process for identifying, phenotypic interpreting, and tracking tissue cells captured using various visualization modalities, according to an embodiment of the present disclosure. Method 200B can be performed using one or more processing device networks 100A, which can include hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or any other processing device that can be suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or any combination thereof.

[0056] Method 200B may begin at block 210 by one or more processing devices receiving multiple images of a set of tissue cells, the multiple images including at least a first image including a first visualization modality and a second image including a second visualization modality. For example, in one embodiment, the first image may include an mxIF visualization modality, and the second image may include an H&E visualization modality. Method 200B may then continue at block 212 by one or more processing devices identifying regions of pixels in the first and second images, each corresponding to a respective tissue cell of the set of tissue cells. For example, in some embodiments, nuclear segmentation may be utilized to segment pixels in the mxIF and H&E images, e.g., along the boundaries of the nuclei of individual tissue cells. Method 200B may include block 214, in which one or more processing devices perform a cell-to-cell registration process based on the identified pixel regions. For example, in certain embodiments, the cell-to-cell registration process may include matching a first region of pixels corresponding to a first tissue cell in the first image with a second region of pixels corresponding to the first tissue cell in the second image. Method 200B may include block 216, in which one or more processing devices classify the first tissue cell into a phenotype based on the cell-to-cell registration process, the phenotype being at least partially indicative of a disease condition. For example, in certain embodiments, the cell-to-cell registration process may include matching one or more populations of immune cells identified in the mxIF image with corresponding immune cells in the H&E image.

[0057] As described above, by utilizing a cross-modality cell-to-cell registration process, one or more tissue cells or populations of tissue cells can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists), or other medical and scientific professionals can more easily classify and phenotype immune cells (e.g., macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, natural killer (NK) cells, etc.). This may result in earlier detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients with both early and advanced stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). This cross-modality cell-cell registration technology may further improve and enhance the efficacy of targeted immunotherapies (e.g., monoclonal antibodies (mAbs), T cell-engaging bispecific antibodies (bsAbs), antibody-drug conjugates (ADCs), etc.) for patients with NHL, including FL and DLBCL.

[0058] FIG. 3A illustrates a flow diagram of a method 300A for training one or more machine-learning models to classify and phenotype tissue cells (e.g., immune cells) in one visualization modality image using class labels from different visualization modality images as ground truth, according to an embodiment of the present disclosure. The flow diagram 300A may be performed utilizing one or more processing device networks 100A, which may include hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or any other processing device that may be suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0059] Method 300A may begin at block 302 with one or more processing devices determining a phenotypic class label of a first tissue cell based on one or more spatial features associated with a first region of pixels corresponding to the first tissue cell. Method 300A may include at block 304, the one or more processing devices determining a correspondence between the first region of pixels corresponding to the first tissue cell and a second region of pixels corresponding to the first tissue cell based on a cell-to-cell registration process. Method 300A may include at block 306, the one or more processing devices training a machine-learning model based on 1) the second region of pixels corresponding to the first tissue cell and 2) the determined phenotypic class label of the first tissue cell.

[0060] 3B illustrates an example implementation 300B of training one or more machine learning models to classify and phenotype tissue cells (e.g., immune cells) in one visualization modality image using class labels from a different visualization modality image as ground truth, according to an embodiment of the present disclosure. Figure 3B illustrates an example implementation of the process described above with respect to Figure 3A. As shown, in one embodiment, the example implementation 300B of the cross-modality cell-to-cell registration process may be described with respect to an mxIF image 308 and an H&E image 310. However, it should be understood that the present technique may be applied between any of the visualization modalities.

[0061] As shown, one or more tissue cells in the mxIF image 308 may be annotated based on one or more spatial features (e.g., any feature suitable for informing or confirming the phenotype of a cell by its spatial organization relative to adjacent cells or its location within a tissue or region of a tissue) and then matched to one or more corresponding tissue cells in the H&E image 310 (e.g., as shown by the line with the open circle extending between the mxIF image 308 and the H&E image 310). For example, as generally shown by FIG. 3B, a human annotator (e.g., a scientist, pathologist, clinician, or other medical or scientific professional) may view the mxIF image 308 and classify and label one or more tissue cells or populations of tissue cells (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) based on, for example, one or more spatial features (e.g., different colors of the pixels of the tissue cells, the number and density of the tissue cells, the proportion and proximity of the tissue cells, the area and multiplicity of the tissue cells, and / or their spatial organization relative to neighboring cells or other features suitable for informing the phenotype of a cell by its location within a tissue or tissue region).

[0062] In certain embodiments, once one or more tissue cells or populations of tissue cells in the mxIF image 308 have been classified and labeled (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.), the corresponding one or more tissue cells or populations of tissue cells in the H&E image 310 (e.g., the same tissue cells captured using a different visualization modality) can be matched thereto according to the cross-modality cell-to-cell registration process described herein. Specifically, the corresponding one or more tissue cells or populations of tissue cells in the H&E image 310 are labeled based on information known and determined from the mxIF image 308 (e.g., "this tissue cell in the H&E image 310 is a lymphocyte"; "this tissue cell in the H&E image 310 is a fibroblast"; "this other tissue cell in the H&E image 310 is a macrophage"; and so forth).

[0063] In this way, even though the H&E image 310 itself does not contain spatial features for easily determining and classifying tissue cells or populations of tissue cells by matching them with those in the mxIF image 308, the H&E image 310 may be rendered as proficient and accurate ground truth data for training one or more machine learning models 312 to classify and phenotype tissue cells or immune cells (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.). That is, in some embodiments, this cross-modality cell-to-cell registration process may allow the H&E image 310 to be molecularly annotated to become verifiably reliable ground truth data (e.g., as opposed to relying on spatial features observed by a human annotator, especially when certain spatial features may be present in one visualization modality but absent in another).

[0064] In certain embodiments, one or more machine learning models 312 (e.g., a deep neural network (DNN), a convolutional neural network (CNN), a fully connected neural network (FCNN), etc.) may then be trained (e.g., via supervised machine learning). For example, during the training phase, the one or more machine learning models 312 may be provided with a dataset of training images 314. The training images 314 may be images having a first visualization modality or a second visualization modality (e.g., thousands of H&E training images). For example, the training images 314 may be H&E training images. In some embodiments, the dataset of H&E training images 314 may be annotated to train the one or more machine learning models 312 to identify and classify tissue cells based on known class labels determined from the mxIF images 308 (different visualization modalities). In certain embodiments, the one or more machine learning models 312 may generate one or more predictions of class labels of the output images 316. In certain embodiments, one or more predictions of class labels of the output image 316 may then be compared to the H&E image 310 (e.g., ground truth) and utilized to iteratively update one or more machine learning models 312 (e.g., by backpropagation or loss calculated between the one or more predictions of class labels of the output image 316 and the ground truth H&E image 310) until they are sufficiently trained to classify and phenotype the tissue cells in the H&E image (e.g., predicting the class labels of tissue cells with an accuracy of 0.8, 0.9, or better on a scale of 0.0 to 1.0).

[0065] 3C shows a flow diagram of a method 300C for utilizing one or more trained machine-learning models to classify and phenotype tissue cells (e.g., immune cells) according to an embodiment of the present disclosure. Method 300C may be performed utilizing one or more processing device networks 100A, which may include hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or any other processing device that may be suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0066] Method 300C may begin by inputting a third image of the plurality of images into a trained machine learning model at block 318, the third image including a second visualization modality. Flow diagram 300C may then include one or more processing devices utilizing the trained machine learning model to identify regions of pixels in the third image corresponding to first tissue cells at block 320. Flow diagram 300C may also include one or more processing devices utilizing the trained machine learning model to output predicted phenotypic class labels for the first tissue cells in the third image at block 322, the predicted phenotypic class labels for the first tissue cells corresponding to the determined phenotypic class labels for the first tissue cells.

[0067] FIG. 3D illustrates an example implementation 300D that utilizes one or more trained machine learning models to classify and phenotype tissue cells, according to embodiments of the present disclosure. FIG. 3C illustrates an example implementation of the process described above with reference to FIG. 3D. As shown, in certain embodiments, during the inference stage, one or more trained machine learning models 324 (e.g., trained as described above with respect to FIGS. 3A and 3B ) may receive an input H&E image 326. In certain embodiments, based on the input H&E image 326, the one or more trained machine learning models 324 may then generate an output H&E image 328 that includes one or more predicted class labels (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.) for one or more tissue cells or populations of tissue cells in the output H&E image 328.

[0068] 4A and 4B illustrate tissue or immune cell annotations on whole slide images for five or more classes of tissue or immune cells (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) according to embodiments of the present disclosure. For example, as shown, FIG. 4A illustrates immune cell annotations performed by human annotator-based classification and labeling 402 compared to model-based classification and labeling 404. FIG. 4B illustrates an H&E image 406 showing classification of more classes of tissue or immune cells (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) labeled with colors. For example, in certain embodiments, magnified portion 408A of H&E image 406 may show one or more tissue cells or one or more populations of tissue cells as unlabeled, while magnified portion 408B of H&E image 406 may show one or more tissue cells or one or more populations of tissue cells that have been labeled in accordance with the techniques of the present disclosure (e.g., as indicated by individually stained tissue cells corresponding to cancer cells, plasma cells, lymphocytes, macrophages, and fibroblasts, respectively).

[0069] 4C and 4D illustrate tissue or immune cell annotations for seven or more classes of tissue or immune cells (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.) and phenotypic analysis tables according to embodiments of the present disclosure. For example, as shown, FIG. 4C illustrates immune cell annotations performed by human annotator-based classification and labeling 410 compared to model-based classification and labeling 412. FIG. 4D illustrates table 414 showing classifications of more phenotypic classes of tissue or immune cells (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) labeled with colors.

[0070] 5A-5C illustrate one or more example graphs or implementations of example tissue cell matches according to embodiments of the present disclosure. For example, original image 500A of mxIF slide 502A, mxIF slide 506A, and H&E slide 510A may include one or more tissue cells. It should be understood that tissue cell 504A, tissue cell 508A, and tissue cell 512A may be the same exact tissue cell captured by different visualization modalities. For example, as shown by mxIF slide 502A, one or more tissue cells 504A may be in different orientations, alignments, proximity, etc. on one or more of mxIF slide 502A, mxIF slide 506A, and H&E slide 510A.

[0071] Figure 5B shows real-world images 500B of mxIF image 502B corresponding to mxIF slide 502A, mxIF image 506B corresponding to mxIF slide 506A, and H&E image 510B corresponding to H&E slide 510A. Figure 5C shows an example graph of the cell-to-cell registration process described herein. Specifically, as shown by image 500C in FIG. 5C, each of tissue cells 504C, 508C, and 512C (e.g., tissue cell "001," tissue cell "002," tissue cell "003," and tissue cell "004") can be matched and tracked between mxIF slide 502A, mxIF slide 506A, and H&E slide 510A such that the phenotype or classification of the tissue cells in one or more of mxIF slide 502A, mxIF slide 506A, and H&E slide 510A can be assumed to be representative of the others of mxIF slide 502A, mxIF slide 506A, and H&E slide 510A.

[0072] 6 illustrates one or more example graphics or implementations 600 of a cross-modality intercellular registration process according to an embodiment of the present disclosure. Specifically, the cross-modality intercellular registration process as illustrated by the one or more graphics or implementations 600 corresponds to the intercellular registration process 114 as described above with respect to FIG. 1A. For example, the cross-modality intercellular registration process may include performing a scale-invariant Fourier transform (SIFT) alignment of the cross-modality visualization images (e.g., SIFT-based alignment with the downsampled pyramid layer) (602) and performing a tile-level alignment of the cross-modality visualization images (e.g., a matrix transform from the coarse image alignment is used to transform tiles from the full-resolution layer that are re-aligned with SIFT) (604), including a matrix transform of the SIFT alignment of the cross-modality visualization images.

[0073] In certain embodiments, the cross-modality cell-to-cell registration process may further include performing tile-level segmentation of the tile-level registered cross-modality visualization images (606), where the tile-level segmentation may be performed to segment each tissue cell within the tile-level registered cross-modality visualization images. In certain embodiments, the cross-modality cell-to-cell registration process may further include performing object-level cell registration based on the tile-level segmented tissue cells (608), where the object-level cell registration is performed to match individual tissue cells across the cross-modality visualization images. For example, in certain embodiments, the object-level cell registration may include matching polygons (e.g., 2D or 3D polygons) in multiple directions based on, for example, identifying overlapping and / or intersecting tissue cells across the cross-modality visualization images.

[0074] FIG. 7 illustrates an example implementation 700 of a cross-modality cell-to-cell registration process, demonstrating that tissue cells can be accurately matched, for example, between mxIF and H&E images, or across various visualization modalities, according to an embodiment of the present disclosure. For example, referring to FIG. 7 , a first image 704A of the example implementation illustrates object-level cell registration of one or more tissue cells between mxIF and H&E visualization modalities, as indicated by overlapping and / or intersecting polygons and / or contours corresponding to the mxIF and H&E visualization modalities, respectively. For example, in some embodiments, the first image 702A of the example implementation illustrates alignment of segmentation polygons, with H&E segmentation shown in yellow (e.g., filled polygons). Similarly, the first image 704A of the example implementation further illustrates IF nuclei segmentation in the DAPI (4′,6-diamidino-2-phenylindole) channel, shown as contours, with the particular color of the contour indicating the number of matches. Specifically, a first image 702A of the example implementation shows the DAPI channel with a phenotype inset, and a second image 704A of the example implementation shows an H&E image with the same segmentation overlay.

[0075] In certain embodiments, as further shown, a phenotyping table 702B may also be included and associated with the first image 702A in the example implementation. In certain embodiments, the phenotyping table 702B may include a list of potential phenotypes that may be color-coded to correspond to visual overlap and / or intersection polygons and / or contours corresponding to the mxIF and H&E visualization modalities, and may also include a number indicating the number of matches for each different phenotype. Similarly, another phenotyping table 704B may also be included and associated with the first image 704A in the example implementation. In certain embodiments, the phenotyping table 704B may include a list of potential phenotypes that may be color-coded to correspond to visual overlap and / or intersection polygons and / or contours corresponding to the mxIF and H&E visualization modalities, and may also include labels indicating each different phenotype (e.g., "Other," "Pax5+Ki+," "Pax5+Ki-," "all_neg," "CD68+," "CD3+CD3+Ki?", "CD8?CD335+," "CD8?FoxP3+Ki?"). In one embodiment, the "+" or "?" may include an imported symbolic representation that may indicate that additional characters may be included as part of one or more phenotypic labels. It should be further appreciated that smaller images included in the lower, right corners of the first image 702A and the second image 704A, respectively, include overlays of the mapped phenotypes, shown, for example, as colored dots.

[0076] 8 illustrates a diagram 800 of an exemplary artificial intelligence (AI) architecture 802 (which may be included as part of one or more of the networks of interacting computer systems 100A, as described above with respect to FIG. 1 ) that may be utilized to perform a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities, according to embodiments of the present disclosure. The identification and phenotyping of tissue cells may be utilized as verifiable ground truth data for training one or more machine learning models to classify and phenotype tissue cells captured using one visualization modality and another visualization modality, according to embodiments of the present disclosure.

[0077] In particular embodiments, the AI ​​architecture 802 may be implemented utilizing one or more processing devices, which may include, for example, hardware (e.g., a general-purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), and / or other processing device(s) that may be suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions operating / executing on one or more processing devices), firmware (e.g., microcode), or some combination thereof.

[0078] 8, AI architecture 802 may include machine learning (ML) algorithms and functions 804, natural language processing (NLP) algorithms and functions 806, expert systems 808, computer-based vision algorithms and functions 810, speech recognition algorithms and functions 812, planning algorithms and functions 814, and robotics algorithms and functions 816. In particular embodiments, ML algorithms and functions 804 may include any statistically-based algorithms that may be suitable for finding patterns across large amounts of data (e.g., "big data" such as genomics data, proteomics data, metabolomics data, metagenomics data, transcriptomics data, and / or other omics data). For example, in particular embodiments, ML algorithms and functions 804 may include deep learning algorithms 818, supervised learning algorithms 820, and unsupervised learning algorithms 822.

[0079] In particular embodiments, the deep learning algorithm 818 may include any artificial neural network (ANN) that can be utilized to learn deep-level representations and abstractions from large amounts of data. For example, the deep learning algorithm 818 may include ANNs such as perceptrons, multi-layer perceptrons (MLPs), autoencoders (AEs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memories (LSTMs), grated recurrent units (GRUs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks, neural autoregressive distribution estimation (NADEs), adversarial networks (ANs), attention models (AMs), spiking neural networks (SNNs), deep reinforcement learning, etc.

[0080] In particular embodiments, supervised learning algorithm 820 may include any algorithm that can be utilized to apply what has been learned in the past to new data, e.g., using labeled examples to predict future events. For example, starting from an analysis of a known training data set, supervised learning algorithm 820 may create an inferred function to make a prediction about output values. Supervised learning algorithm 820 may also compare its output with the correct intended output and find errors in order to correct supervised learning algorithm 820 accordingly. On the other hand, unsupervised learning algorithm 822 may include, for example, any algorithm that can be applied when the data used to train unsupervised learning algorithm 822 is neither classified nor labeled. For example, unsupervised learning algorithm 822 may study and analyze how a system can infer functions to describe hidden structure from unlabeled data.

[0081] In particular embodiments, NLP algorithms and functions 806 may include any algorithms or functions that may be suitable for automatically manipulating natural language, such as speech and / or text. For example, in some embodiments, NLP algorithms and functions 806 may include content extraction algorithms or functions 824, classification algorithms or functions 826, machine translation algorithms or functions 828, question answering (QA) algorithms or functions 830, and text generation algorithms or functions 832. In particular embodiments, content extraction algorithms or functions 824 may include means for extracting text or images from electronic documents (e.g., web pages, text editor documents, etc.) for use in other applications, for example.

[0082] In particular embodiments, classification algorithm or function 826 may include any algorithm that may utilize a supervised learning model (e.g., logistic regression, naive Bayes, stochastic gradient descent (SGD), k-nearest neighbors, decision trees, random forests, support vector machines (SVMs), etc.) to learn from and make new observations or classifications based on data input into the supervised learning model. Machine translation algorithm or function 828 may include any algorithm or function that may be suitable for automatically converting source text in one language into text in another language, for example. QA algorithm or function 830 may include any algorithm or function that may be suitable for automatically answering questions posed by humans in natural language, such as those performed by a voice-controlled personal assistant device, for example. Text generation algorithm or function 832 may include any algorithm or function that may be suitable for automatically generating natural language text.

[0083] In particular embodiments, expert system 808 may include any algorithms or functions that may be suitable for simulating the judgment and actions of a human or organization with expertise and experience in a particular field (e.g., stock trading, medicine, sports statistics, etc.). Computer-based vision algorithms and functions 810 may include any algorithms or functions that may be suitable for automatically extracting information from images (e.g., photographic images, video images). For example, computer-based vision algorithms and functions 810 may include image recognition algorithms 834 and machine vision algorithms 836. Image recognition algorithms 834 may include any algorithms that may be suitable, for example, for automatically identifying and / or classifying objects, places, people, etc. that may be included in one or more image frames or other display data. Machine vision algorithms 836 may include any algorithms that may be suitable for enabling a computer to "see" or that may be suitable, for example, for relying on image sensor cameras with specialized optics to acquire images in order to process, analyze, and / or measure various data characteristics for decision-making purposes.

[0084] In particular embodiments, speech recognition algorithms and functions 812 may include any algorithms or functions that may be suitable for recognizing and translating spoken language into text, such as through automatic speech recognition (ASR), computer speech recognition, speech-to-text (STT) 838, or text-to-speech (TTS) 840, for computing purposes to communicate with one or more users via voice. In particular embodiments, planning algorithms and functions 814 may include any algorithms or functions that may be suitable for generating a sequence of actions, where each action may include its own set of preconditions to be satisfied before performing the action. Examples of AI planning may include classical planning, reduction to other problems, temporal planning, probabilistic planning, preference-based planning, conditional planning, etc. Finally, robotics algorithms and functions 816 may include any algorithms, functions, or systems that may enable one or more devices to replicate human behavior, for example, through movements, gestures, performance tasks, decision-making, emotions, etc.

[0085] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or clear from the context. Thus, as used herein, "A or B" means "A, B, or both," unless clearly stated otherwise or indicated otherwise by the context. Moreover, "and" is both jointly and severally, unless clearly stated otherwise or indicated otherwise by the context. Thus, as used herein, "A and B" means "A and B jointly or severally," unless clearly stated otherwise or clear from the context.

[0086] As used herein, "automatically" and its derivatives mean "without human intervention" unless expressly indicated otherwise or indicated otherwise by context.

[0087] The embodiments disclosed herein are merely examples, and the scope of the disclosure is not limited thereto. Embodiments according to the present disclosure are disclosed in the appended claims, particularly those directed to methods, storage media, systems, and computer program products. Any feature recited in one claim category, e.g., a method, may also be claimed in another claim category, e.g., a system. Dependencies or references in the appended claims are chosen for formality reasons only. However, just as any combination of a claim and its features may be disclosed and claimed without regard to the dependencies recited in the appended claims, any subject matter resulting from an intentional reference to any preceding claim (e.g., multiple dependencies) may likewise be claimed. Subject matter that may be claimed includes not only combinations of features as recited in the appended claims, but also any other combinations of features within the scope of the claims, and each feature recited in a claim may be combined with any other feature or combination of features within the scope of the claim. Furthermore, any of the embodiments and features described or illustrated in this specification may be claimed in a separate claim and / or in any combination with any of the embodiments or features described or illustrated in this specification or with any of the features of the accompanying claims.

[0088] The scope of the present disclosure encompasses all changes, substitutions, variations, changes, and modifications to the exemplary embodiments described or illustrated herein that would be understood by one skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although the present disclosure describes and illustrates each embodiment herein as including particular components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by one skilled in the art. Furthermore, references in the appended claims to a device or system or a component of a device or system that is arranged, arranged, enabled, configured, enabled, operable, or operates to perform a particular function encompass that device, system, or component, to the extent that the device, system, or component is so arranged, arranged, enabled, configured, enabled, operable, or operates, regardless of whether it or that particular function is activated, turned on, or released. Furthermore, although this disclosure may describe or illustrate particular embodiments as providing certain advantages, the particular embodiments may provide none, some, or all of these advantages.

[0089] Embodiment Among the embodiments provided are the following: 1. A method for classifying tissue cells into one or more phenotypes, the method comprising: receiving, by one or more computing devices, a plurality of images of a set of tissue cells, the plurality of images including at least a first image including a first visualization modality and a second image including a second visualization modality; identifying a first tissue cell of the set of tissue cells in the first image and a first tissue cell in the second image; performing a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image, the cell-to-cell registration process including matching the first tissue cell identified in the first image with the first tissue cell identified in the second image; and classifying the first tissue cell into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease condition. 2. The method of embodiment 1, further comprising generating a phenotyping table based on the phenotypic classification of the first tissue cells. 3. The method of embodiment 2, further comprising utilizing a phenotypic analysis table to map the phenotypic classification of the first tissue cell identified in the first image to the first tissue cell identified in the second image. 4. The method of any one of embodiments 1 to 3, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole-slide imaging modality, a microscope modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality. 5. The method of embodiment 4, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging. 6. The method of embodiment 4 or 5, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic imaging. 7. The method of any one of embodiments 4 to 6, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI). 8. The method of any one of embodiments 4 to 7, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality. 9. The method of embodiment 8, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining. 10. The method of embodiment 9, wherein the histological staining comprises hematoxylin and eosin (H&E) staining or chromogenic staining. 11. The method of any one of embodiments 1 to 10, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining. 12. The method of any one of embodiments 1 to 11, wherein phenotypic classification of the first tissue cell comprises classifying the cellular state of the first tissue cell as an activated immune cell based on one or more molecular annotations. 13. The method of any one of embodiments 1 to 12, wherein the first tissue cell comprises a cancer cell, a plasma cell, a lymphocyte, a macrophage, or a fibroblast. 14. The method of any one of embodiments 1 to 13, wherein phenotypic classification of the first tissue cell comprises classifying the first tissue cell as an immune cell based on one or more molecular annotations. 15. The method of embodiment 14, wherein the immune cells comprise macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, or natural killer (NK) cells. 16. The method of any one of embodiments 1 to 15, wherein the disease condition comprises a non-Hodgkin's lymphoma (NHL) disease condition. 17. The method of embodiment 16, wherein the NHL disease state comprises follicular lymphoma (FL). 18. The method of embodiment 16, wherein the NHL disease state comprises diffuse large B-cell lymphoma (DLBCL). 19. A method for classifying tissue cells into one or more phenotypes, the method comprising: receiving, by one or more computing devices, a plurality of images of a set of tissue cells, the plurality of images including at least a first image including a first visualization modality and a second image including a second visualization modality; identifying regions of pixels in the first image and in the second image, each of the regions of pixels corresponding to a respective tissue cell of the set of tissue cells; performing a cell-to-cell registration process based on the identified regions of pixels, the cell-to-cell registration process including matching a first region of pixels corresponding to the first tissue cell in the first image with a second region of pixels corresponding to the first tissue cell in the second image; and classifying the first tissue cell into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease condition. 20. The method of embodiment 19, further comprising generating a phenotyping table based on the phenotypic classification of the first tissue cells. 21. The method of any one of embodiments 19 to 20, further comprising: determining a phenotypic class label in the first tissue cell based on one or more spatial features associated with a first region of pixels corresponding to the first tissue cell; determining a correspondence between the first region of pixels corresponding to the first tissue cell and a second region of pixels corresponding to the first tissue cell based on a cell-to-cell registration process; and training a model based on 1) the second region of pixels corresponding to the first tissue cell and 2) the determined phenotypic class label in the first tissue cell. 22. The method of embodiment 21, further comprising inputting a third image of the plurality of images into the trained model, the third image comprising a second visualization modality, and utilizing the trained model to identify regions of pixels in the third image that correspond to first tissue cells, and outputting predicted phenotypic class labels for the first tissue cells in the third image, wherein the predicted phenotypic class labels for the first tissue cells correspond to the determined phenotypic class labels for the first tissue cells. 23. The method of embodiment 21 or 22, wherein the model comprises one or more deep neural networks (DNNs). 24. The method of any one of embodiments 21 to 23, further comprising determining ground truth data for training the model by mapping a phenotype classification of the first tissue cell to the first tissue cell in the first image and the first tissue cell in the second image. 25. The method of embodiment 24, wherein at least a subset of the ground truth data includes molecularly annotated ground truth data, or at least a subset of the ground truth data includes human-annotated ground truth data. 26. The method of embodiment 24 or 25, further comprising using a phenotype analysis table to map the phenotype classification of the first tissue cell to the first tissue cell in the first image and the first tissue cell in the second image. 27. The method of any one of embodiments 19 to 26, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality. 28. The method of embodiment 27, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging. 29. The method of embodiment 27 or 28, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic imaging. 30. The method of any one of embodiments 27 to 29, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI). 31. The method of any one of embodiments 27 to 30, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality. 32. The method of embodiment 31, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescence staining. 33. The method of embodiment 32, wherein the histological staining comprises hematoxylin and eosin (H&E) staining or chromogenic staining. 34. The method of any one of embodiments 19 to 33, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining. 35. The method of any one of embodiments 19 to 34, further comprising, prior to identifying the regions of pixels in the first image and the second image, aligning the multiple images so that at least the first image and the second image are stacked vertically. 36. The method of any one of embodiments 19 to 35, wherein performing an intercellular registration process includes performing alignment of the first image with the second image. 37. The method of embodiment 36, wherein performing the intercellular registration process further includes performing tile-level alignment of the first image and the second image, the tile-level alignment including a matrix transformation of the alignment of the first image and the second image. 38. The method of embodiment 37, wherein performing the cell-cell registration process further includes performing tile-level segmentation of the tile-level aligned first image and second image, the tile-level segmentation being performed to segment each tissue cell of the set of tissue cells in the first image and the second image. 39. The method of embodiment 38, wherein performing the cell-to-cell registration process further includes performing object-level cell registration based on the tile-level segmented tissue cells, and the object-level cell registration is performed to match a first tissue cell in the first image with a first tissue cell in the second image. 40. The method of any one of embodiments 19 to 39b, wherein identifying regions of pixels in the first image and the second image includes performing segmentation of regions of pixels in the first image and the second image to segment each tissue cell of the set of tissue cells. 41. The method of any one of embodiments 19 to 40, further comprising extracting one or more features from the first image and the second image based on the identified regions of pixels before performing the cell-to-cell registration process, wherein the one or more features are utilized to identify the first tissue cell in the first image and the first tissue cell in the second image. 42. The method of any one of embodiments 19 to 41, wherein classifying the first tissue cells into a phenotype comprises classifying the first tissue cells into a phenotype based on one or more spatial features. 43. The method of any one of embodiments 19 to 42, wherein classifying the first tissue cells into a phenotype comprises classifying the first tissue cells into a phenotype based on one or more molecular annotations. 44. The method of any one of embodiments 19 to 44, wherein the disease condition comprises a non-Hodgkin's lymphoma (NHL) disease condition. 45. The method of embodiment 44, wherein the NHL disease state comprises follicular lymphoma (FL). 46. ​​The method of embodiment 44, wherein the NHL disease state comprises diffuse large B-cell lymphoma (DLBCL). 47. A system including one or more computing devices comprising one or more non-transitory computer-readable storage media containing instructions and one or more processors coupled to the one or more storage media, wherein the one or more processors are configured to execute the instructions to perform the method of any one of embodiments 1 to 46. 48. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to perform the method of any one of embodiments 1 to 46.

Claims

1. 1. A method for classifying tissue cells into one or more phenotypes, comprising, by one or more computing devices: receiving a plurality of images of a set of tissue cells, the plurality of images including at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; identifying a first tissue cell of the set of tissue cells in the first image from the first tissue cell in the second image; performing an intercellular registration process based on the first tissue cells identified in the first image and the first tissue cells identified in the second image, the intercellular registration process including matching the first tissue cells identified in the first image with the first tissue cells identified in the second image; classifying the first tissue cells into a phenotype based on the cell-cell registration process, the phenotype being at least partially indicative of a disease pathology; and A method comprising:

2. The method of claim 1 , further comprising generating a phenotyping table based on the phenotypic classification of the first tissue cells.

3. 3. The method of claim 2, further comprising utilizing the phenotypic analysis table to map the phenotypic classification of the first tissue cells identified in the first image to the first tissue cells identified in the second image.

4. 2. The method of claim 1, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole-slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

5. 5. The method of claim 4, wherein the whole slide imaging modality is selected from bright field or fluorescent imaging.

6. 5. The method of claim 4, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic imaging.

7. 5. The method of claim 4, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI).

8. The method of claim 4 , wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

9. 9. The method of claim 8, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

10. 10. The method of claim 9, wherein the histological stain comprises a hematoxylin and eosin (H&E) stain or a chromogenic stain.

11. The method of claim 1 , wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

12. 10. The method of claim 1, wherein classifying the phenotype of the first tissue cell comprises classifying a cellular state of the first tissue cell as an activated immune cell based on one or more molecular annotations.

13. The method of claim 1 , wherein the first tissue cells comprise cancer cells, plasma cells, lymphocytes, macrophages, or fibroblasts.

14. 10. The method of claim 1, wherein classifying the first tissue cell into the phenotype comprises classifying the first tissue cell as an immune cell based on one or more molecular annotations.

15. 15. The method of claim 14, wherein the immune cells comprise macrophages, regulatory T cells (Treg), CD8 cells, B lymphocytes, or natural killer (NK) cells.

16. 10. The method of claim 1, wherein the disease state comprises a non-Hodgkin's lymphoma (NHL) disease state.

17. 17. The method of claim 16, wherein the NHL disease state comprises follicular lymphoma (FL).

18. 17. The method of claim 16, wherein the NHL disease state comprises diffuse large B-cell lymphoma (DLBCL).

19. 1. A system including one or more computing devices, one or more non-transitory computer-readable storage media containing instructions; one or more processors coupled to the one or more storage media; wherein the one or more processors: instructions for receiving a plurality of images of a set of tissue cells, the plurality of images including at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; instructions for identifying a first tissue cell of the set of tissue cells in the first image and the first tissue cell in the second image; instructions for performing a cell-to-cell registration process based on the first tissue cells identified in the first image and the first tissue cells identified in the second image, the cell-to-cell registration process including matching the first tissue cells identified in the first image with the first tissue cells identified in the second image; instructions for classifying the first tissue cells into a phenotype based on the cell-cell registration process, the phenotype being at least partially indicative of a disease condition; and A system configured to run

20. 20. The system of claim 19, wherein the instructions further comprise instructions for generating a phenotyping table based on the phenotypic classification of the first tissue cells.

21. 21. The system of claim 20, wherein the instructions further comprise instructions for utilizing the phenotyping table to map the phenotypic classification of the first tissue cell identified in the first image to the first tissue cell identified in the second image.

22. 20. The system of claim 19, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

23. 23. The system of claim 22, wherein the whole slide imaging modality is selected from bright field or fluorescent imaging.

24. 23. The system of claim 22, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic imaging.

25. 23. The system of claim 22, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI).

26. 23. The system of claim 22, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

27. 27. The system of claim 26, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

28. 28. The system of claim 27, wherein the histological stain comprises a hematoxylin and eosin (H&E) stain or a chromogenic stain.

29. 20. The system of claim 19, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

30. 20. The system of claim 19, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying a cellular state of the first tissue cell as an activated immune cell based on one or more molecular annotations.

31. 20. The system of claim 19, wherein the first tissue cell comprises a cancer cell, a plasma cell, a lymphocyte, a macrophage, or a fibroblast.

32. 20. The system of claim 19, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell as an immune cell based on one or more molecular annotations.

33. 33. The system of claim 32, wherein the immune cells comprise macrophages, regulatory T cells (Treg), CD8 cells, B lymphocytes, or natural killer (NK) cells.

34. 20. The system of claim 19, wherein the disease state comprises a non-Hodgkin's lymphoma (NHL) disease state.

35. 35. The system of claim 34, wherein the NHL disease state comprises follicular lymphoma (FL).

36. 35. The system of claim 34, wherein the NHL disease state comprises diffuse large B-cell lymphoma (DLBCL).

37. A non-transitory computer-readable medium containing instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to: receiving a plurality of images of a set of tissue cells, the plurality of images including at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; identifying a first tissue cell in the set of tissue cells in the first image from the first tissue cell in the second image; performing a cell-to-cell registration process based on the first tissue cells identified in the first image and the first tissue cells identified in the second image, the cell-to-cell registration process including matching the first tissue cells identified in the first image with the first tissue cells identified in the second image; classifying the first tissue cells into a phenotype based on the cell-cell registration process, the phenotype being at least partially indicative of a disease pathology. Non-transitory computer-readable medium.

38. 38. The non-transitory computer-readable medium of claim 37, wherein the instructions further comprise instructions for generating a phenotyping table based on the phenotypic classification of the first tissue cells.

39. 39. The non-transitory computer-readable medium of claim 38, wherein the instructions further comprise instructions for utilizing the phenotyping table to map the phenotypic classification of the first tissue cells identified in the first image to the first tissue cells identified in the second image.

40. 38. The non-transitory computer-readable medium of claim 37, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

41. 41. The non-transitory computer-readable medium of claim 40, wherein the whole-slide imaging modality is selected from bright field or fluorescence imaging.

42. 41. The non-transitory computer-readable medium of claim 40, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic imaging.

43. 41. The non-transitory computer-readable medium of claim 40, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI).

44. 41. The non-transitory computer-readable medium of claim 40, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

45. 45. The non-transitory computer-readable medium of claim 44, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

46. 46. ​​The non-transitory computer-readable medium of claim 45, wherein the histological stain comprises a hematoxylin and eosin (H&E) stain or a chromogenic stain.

47. 38. The non-transitory computer-readable medium of claim 37, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

48. 38. The non-transitory computer-readable medium of claim 37, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying a cellular state of the first tissue cell as an activated immune cell based on one or more molecular annotations.

49. 38. The non-transitory computer-readable medium of claim 37, wherein the first tissue cell comprises a cancer cell, a plasma cell, a lymphocyte, a macrophage, or a fibroblast.

50. 38. The non-transitory computer-readable medium of claim 37, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell as an immune cell based on one or more molecular annotations.

51. 51. The non-transitory computer-readable medium of claim 50, wherein the immune cells comprise macrophages, regulatory T cells (Treg), CD8 cells, B lymphocytes, or natural killer (NK) cells.

52. 38. The non-transitory computer-readable medium of claim 37, wherein the disease condition comprises a non-Hodgkin's lymphoma (NHL) disease condition.

53. 53. The non-transitory computer-readable medium of claim 52, wherein the NHL disease state comprises follicular lymphoma (FL).

54. 53. The non-transitory computer-readable medium of claim 52, wherein the NHL disease condition comprises diffuse large B-cell lymphoma (DLBCL).

55. 1. A method for classifying tissue cells into one or more phenotypes, comprising, by one or more computing devices: receiving a plurality of images of a set of tissue cells, the plurality of images including at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; identifying regions of pixels in the first image and the second image, each of the regions of pixels corresponding to a respective tissue cell of the set of tissue cells; performing an intercellular registration process based on the identified regions of pixels, the intercellular registration process including matching a first region of pixels corresponding to a first tissue cell in the first image with a second region of pixels corresponding to the first tissue cell in the second image; classifying the first tissue cells into a phenotype based on the cell-cell registration process, the phenotype being at least partially indicative of a disease pathology; and A method comprising:

56. 56. The method of claim 55, further comprising generating a phenotyping table based on the phenotypic classification of the first tissue cells.

57. determining a phenotypic class label for the first tissue cell based on one or more spatial features associated with the first region of pixels corresponding to the first tissue cell; determining a correspondence between the first region of pixels corresponding to the first tissue cell and the second region of pixels corresponding to the first tissue cell based on the cell-to-cell registration process; training a model based on 1) the second region of pixels corresponding to the first tissue cell, and 2) the determined phenotypic class label of the first tissue cell; 56. The method of claim 55, further comprising:

58. inputting a third image of the plurality of images into the trained model, the third image comprising the second visualization modality; and Using the trained model, identifying a region of pixels in the third image corresponding to the first tissue cells; and outputting predicted phenotypic class labels for the first tissue cells in the third image; wherein the predicted phenotypic class label in the first tissue cell corresponds to the determined phenotypic class label in the first tissue cell.

58. The method of claim 57.

59. 58. The method of claim 57, wherein the model comprises one or more deep neural networks (DNNs).

60. 58. The method of claim 57, further comprising determining ground truth data for training the model by mapping a phenotype classification of the first tissue cells to the first tissue cells in the first image and to the first tissue cells in the second image.

61. 61. The method of claim 60, wherein at least a subset of the ground truth data comprises molecularly annotated ground truth data, or wherein at least a subset of the ground truth data comprises human-annotated ground truth data.

62. 61. The method of claim 60, further comprising utilizing a phenotypic analysis table to map the phenotypic classification of the first tissue cell to the first tissue cell in the first image and the first tissue cell in the second image.

63. 56. The method of claim 55, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

64. 64. The method of claim 63, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

65. 64. The method of claim 63, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic imaging.

66. 64. The method of claim 63, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI).

67. 64. The method of claim 63, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

68. 68. The method of claim 67, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

69. 69. The method of claim 68, wherein the histological stain comprises a hematoxylin and eosin (H&E) stain or a chromogenic stain.

70. 56. The method of claim 55, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

71. 56. The method of claim 55, further comprising, prior to identifying the regions of pixels in the first image and the second image, registering the multiple images to vertically stack at least the first image and the second image.

72. 56. The method of claim 55, wherein performing the cell-to-cell registration process comprises performing alignment of the first image and the second image.

73. 73. The method of claim 72, wherein performing the cell-to-cell registration process further comprises performing a tile-level registration of the first image and the second image, wherein the tile-level registration comprises a matrix transformation of the registration of the first image and the second image.

74. 74. The method of claim 73, wherein performing the cell-to-cell registration process further comprises performing tile-level segmentation of the tile-level aligned first and second images, wherein the tile-level segmentation is performed to segment each tissue cell of the set of tissue cells in the first and second images.

75. 75. The method of claim 74, wherein performing the cell-to-cell registration process further comprises performing object-level cell registration based on the tile-level segmented tissue cells, wherein the object-level cell registration is performed to match the first tissue cells in the first image with the first tissue cells in the second image.

76. 56. The method of claim 55, wherein identifying the regions of pixels in the first image and the second image comprises performing segmentation of the regions of pixels in the first image and the second image to segment each tissue cell of the set of tissue cells.

77. extracting one or more features from the first image and the second image based on the identified regions of pixels before performing the cell-to-cell registration process, wherein the one or more features are utilized to identify the first tissue cell in the first image and the second tissue cell in the second image; 56. The method of claim 55, further comprising:

78. 56. The method of claim 55, wherein classifying the first tissue cells into the phenotype comprises classifying the first tissue cells into the phenotype based on one or more spatial features.

79. 56. The method of claim 55, wherein classifying the first tissue cell into the phenotype comprises classifying the first tissue cell into the phenotype based on one or more molecular annotations.

80. 56. The method of claim 55, wherein the disease state comprises a non-Hodgkin's lymphoma (NHL) disease state.

81. 81. The method of claim 80, wherein the NHL disease state comprises follicular lymphoma (FL).

82. 81. The method of claim 80, wherein the NHL disease state comprises diffuse large B-cell lymphoma (DLBCL).

83. 1. A system including one or more computing devices, one or more non-transitory computer-readable storage media containing instructions; one or more processors coupled to the one or more storage media; wherein the one or more processors: instructions for receiving a plurality of images of a set of tissue cells, the plurality of images including at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; instructions for identifying regions of pixels in the first image and the second image, each of the regions of pixels corresponding to a respective tissue cell of the set of tissue cells; instructions for performing an intercellular registration process based on the identified regions of pixels, the intercellular registration process including matching a first region of pixels corresponding to a first tissue cell in the first image with a second region of pixels corresponding to the first tissue cell in the second image; instructions for classifying the first tissue cells into a phenotype based on the cell-cell registration process, the phenotype being at least partially indicative of a disease condition; and A system configured to run

84. 84. The system of claim 83, wherein the instructions further comprise instructions for generating a phenotyping table based on the phenotypic classification of the first tissue cells.

85. The instruction: instructions for determining a phenotypic class label for the first tissue cell based on one or more spatial features associated with the first region of pixels corresponding to the first tissue cell; instructions for determining a correspondence between the first region of pixels corresponding to the first tissue cell and the second region of pixels corresponding to the first tissue cell based on the cell-to-cell registration process; instructions for training a model based on 1) the second region of pixels corresponding to the first tissue cell, and 2) the determined phenotypic class label of the first tissue cell; 84. The system of claim 83, further comprising:

86. The instruction: instructions for inputting a third image of the plurality of images into a trained model, the third image comprising the second visualization modality; and Using the trained model, identifying a region of pixels in the third image corresponding to the first tissue cell; and outputting a predicted phenotypic class label for the first tissue cell in the third image, the predicted phenotypic class label for the first tissue cell corresponding to the determined phenotypic class label for the first tissue cell.

86. The system of claim 85.

87. 86. The system of claim 85, wherein the model comprises one or more deep neural networks (DNNs).

88. The instruction: instructions for determining ground truth data for training the model by mapping a phenotype classification of the first tissue cells to the first tissue cells in the first image and to the first tissue cells in the second image; 86. The system of claim 85, further comprising:

89. 90. The system of claim 88, wherein at least a subset of the ground truth data comprises molecularly annotated ground truth data, or wherein at least a subset of the ground truth data comprises human-annotated ground truth data.

90. 90. The system of claim 88, wherein the instructions further comprise instructions for utilizing a phenotypic analysis table to map the phenotypic classification of the first tissue cell to the first tissue cell in the first image and the first tissue cell in the second image.

91. 84. The system of claim 83, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

92. 92. The system of claim 91, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

93. 92. The system of claim 91, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic imaging.

94. 92. The system of claim 91, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI).

95. 92. The system of claim 91, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

96. 96. The system of claim 95, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

97. 97. The system of claim 96, wherein the histological stain comprises a hematoxylin and eosin (H&E) stain or a chromogenic stain.

98. 84. The system of claim 83, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

99. The instruction: instructions for registering the images so as to vertically stack at least the first image and the second image before identifying the regions of pixels in the first image and the second image; 84. The system of claim 83, further comprising:

100. 84. The system of claim 83, wherein the instructions to perform the cell-to-cell registration process further comprise instructions to perform alignment of the first image and the second image.

101. 101. The system of claim 100, wherein the instructions to perform the intercellular registration process further comprise instructions to perform a tile-level registration of the first image and the second image, the tile-level registration comprising a matrix transformation of the registration of the first image and the second image.

102. 102. The system of claim 101, wherein the instructions to perform the cell-to-cell registration process further comprise instructions to perform tile-level segmentation of the tile-level aligned first and second images, the tile-level segmentation being performed to segment each tissue cell of the set of tissue cells in the first and second images.

103. 103. The system of claim 102, wherein the instructions to perform the cell-to-cell registration process further comprise instructions to perform object-level cell registration based on tile-level segmented tissue cells, the object-level cell registration being performed to match the first tissue cells in the first image with the first tissue cells in the second image.

104. 84. The system of claim 83, wherein the instructions for identifying the regions of pixels in the first image and the second image further comprise performing segmentation of the regions of pixels in the first image and the second image to segment each tissue cell of the set of tissue cells.

105. The instruction: instructions for extracting one or more features from the first image and the second image based on the identified regions of pixels prior to performing the cell-to-cell registration process, the one or more features being utilized to identify the first tissue cell in the first image and the second tissue cell in the second image; 84. The system of claim 83, further comprising:

106. 84. The system of claim 83, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell into the phenotype based on one or more spatial features.

107. 84. The system of claim 83, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell into the phenotype based on one or more molecular annotations.

108. 84. The system of claim 83, wherein the disease state comprises a non-Hodgkin's lymphoma (NHL) disease state.

109. 109. The system of claim 108, wherein the NHL disease state comprises follicular lymphoma (FL).

110. 109. The system of claim 108, wherein the NHL disease state comprises diffuse large B-cell lymphoma (DLBCL).

111. A non-transitory computer-readable medium containing instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to: receiving a plurality of images of a set of tissue cells, the plurality of images including at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; identifying regions of pixels in the first image and the second image, each of the regions of pixels corresponding to a respective tissue cell of the set of tissue cells; performing a cell-to-cell registration process based on the identified regions of pixels, the cell-to-cell registration process including matching a first region of pixels corresponding to a first tissue cell in the first image with a second region of pixels corresponding to the first tissue cell in the second image; classifying the first tissue cells into a phenotype based on the cell-cell registration process, the phenotype being at least partially indicative of a disease pathology. Non-transitory computer-readable medium.

112. 112. The non-transitory computer-readable medium of claim 111, wherein the instructions further comprise instructions for generating a phenotyping table based on the phenotypic classification of the first tissue cells.

113. The instruction: instructions for determining a phenotypic class label for the first tissue cell based on one or more spatial features associated with the first region of pixels corresponding to the first tissue cell; instructions for determining a correspondence between the first region of pixels corresponding to the first tissue cell and the second region of pixels corresponding to the first tissue cell based on the cell-to-cell registration process; instructions for training a model based on 1) the second region of pixels corresponding to the first tissue cell, and 2) the determined phenotypic class label of the first tissue cell; 112. The non-transitory computer-readable medium of claim 111, further comprising:

114. The instruction: instructions for inputting a third image of the plurality of images into a trained model, the third image comprising the second visualization modality; and Using the trained model, identifying a region of pixels in the third image corresponding to the first tissue cell; and outputting a predicted phenotypic class label for the first tissue cell in the third image, the predicted phenotypic class label for the first tissue cell corresponding to the determined phenotypic class label for the first tissue cell.

114. The non-transitory computer-readable medium of claim 113.

115. 114. The non-transitory computer-readable medium of claim 113, wherein the model comprises one or more deep neural networks (DNNs).

116. The instruction: instructions for determining ground truth data for training the model by mapping a phenotype classification of the first tissue cells to the first tissue cells in the first image and to the first tissue cells in the second image; 114. The non-transitory computer-readable medium of claim 113, further comprising:

117. 117. The non-transitory computer-readable medium of claim 116, wherein at least a subset of the ground truth data comprises molecular-annotated ground truth data, or wherein at least a subset of the ground truth data comprises human-annotated ground truth data.

118. 117. The non-transitory computer-readable medium of claim 116, wherein the instructions further comprise instructions for utilizing a phenotyping table to map the phenotypic classification of the first tissue cell to the first tissue cell in the first image and the first tissue cell in the second image.

119. 112. The non-transitory computer-readable medium of claim 111, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

120. 120. The non-transitory computer-readable medium of claim 119, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

121. 120. The non-transitory computer-readable medium of claim 119, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

122. 120. The non-transitory computer-readable medium of claim 119, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiplexed ion beam imaging (MIBI).

123. 120. The non-transitory computer-readable medium of claim 119, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

124. 124. The non-transitory computer-readable medium of claim 123, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

125. 125. The non-transitory computer-readable medium of claim 124, wherein the histological stain comprises a hematoxylin and eosin (H&E) stain or a chromogenic stain.

126. 112. The non-transitory computer-readable medium of claim 111, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

127. The instruction: instructions for registering the images so as to vertically stack at least the first image and the second image before identifying the regions of pixels in the first image and the second image; 112. The non-transitory computer-readable medium of claim 111, further comprising:

128. 112. The non-transitory computer-readable medium of claim 111, wherein the instructions for performing the cell-to-cell registration process further comprise instructions for performing alignment of the first image and the second image.

129. 129. The non-transitory computer-readable medium of claim 128, wherein the instructions to perform the cell-to-cell registration process further comprise instructions to perform a tile-level alignment of the first image and the second image, the tile-level alignment comprising a matrix transformation of the alignment of the first image and the second image.

130. 130. The non-transitory computer-readable medium of claim 129, wherein the instructions for performing the cell-to-cell registration process further include instructions for performing tile-level segmentation of the tile-level aligned first and second images, the tile-level segmentation being performed to segment each tissue cell of the set of tissue cells in the first and second images.

131. 131. The non-transitory computer-readable medium of claim 130, wherein the instructions for performing the cell-to-cell registration process further include instructions for performing object-level cell registration based on tile-level segmented tissue cells, the object-level cell registration being performed to match the first tissue cells in the first image with the first tissue cells in the second image.

132. 112. The non-transitory computer-readable medium of claim 111, wherein the instructions for identifying the regions of pixels in the first image and the second image further comprise performing segmentation of the regions of pixels in the first image and the second image to segment each tissue cell of the set of tissue cells.

133. The instruction: instructions for extracting one or more features from the first image and the second image based on the identified regions of pixels prior to performing the cell-to-cell registration process, the one or more features being utilized to identify the first tissue cell in the first image and the second tissue cell in the second image; 112. The non-transitory computer-readable medium of claim 111, further comprising:

134. 112. The non-transitory computer-readable medium of claim 111, wherein the instructions for classifying the first tissue cells into the phenotype further comprise instructions for classifying the first tissue cells into the phenotype based on one or more spatial features.

135. 112. The non-transitory computer-readable medium of claim 111, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell into the phenotype based on one or more molecular annotations.

136. 112. The non-transitory computer-readable medium of claim 111, wherein the disease condition comprises a non-Hodgkin's lymphoma (NHL) disease condition.

137. 137. The non-transitory computer-readable medium of claim 136, wherein the NHL disease state comprises follicular lymphoma (FL).

138. 137. The non-transitory computer-readable medium of claim 136, wherein the NHL disease condition comprises diffuse large B-cell lymphoma (DLBCL).

Images