Cross-modality pixel alignment and cell-to-cell registration across various imaging modalities

Abstract

A method implemented by one or more computer devices includes receiving a plurality of images of a collection of tissue cells, the plurality of images including a first image including a first visualization modality and a second image including a second visualization modality. The method includes identifying a first tissue cell in the set of tissue cells in the first image and the first tissue cell in the second image, and performing a cell-to-cell registration process based on the first tissue cells identified in the first image and the first tissue cells identified in the second image. The cell-to-cell registration process includes matching the first tissue cells identified in the first image to the first tissue cells identified in the second image. The method includes classifying the first tissue cells into a phenotype based on the cell-to-cell registration process, the phenotype being indicative in part of a disease pathology.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 407,077, filed on September 15, 2022, and entitled “Cross-Modality Pixel Alignment and Cell-to-Cell Registration Across Various Imaging Modalities,” the contents of which are hereby incorporated by reference in their entirety. Technical Field

[0003] The present application relates generally to molecular annotation, and more particularly to resolving differences in molecular annotation using cross-modality pixel alignment and cell-to-cell registration across various imaging modalities. Background Art

[0004] Digital pathology typically involves visualizing and analyzing digitized slides to determine whether changes occurring in tissue cells are caused by disease, toxicity, and / or natural processes. Visualization of tissue cells can typically include dye-based visualization modalities, molecular-based visualization modalities, or probe-based visualization modalities. For example, dye-based visualization (such as hematoxylin and eosin (H&E)) can utilize the chemical properties of dye molecules to bind to specific tissue cells (such as mucin, fat, and protein). Similarly, molecular-based visualization (such as immunohistochemistry (IHC)) can utilize antibodies to bind to protein epitope targets with high specificity, and can be visualized, for example, using 3,3'-diaminobenzidine or other similar organic compounds to produce brown staining. IHC can also include other visualization colors, which can be combined, for example, to visualize multiple and different biomarkers.

[0005] Other molecular-based visualizations can include, for example, immunofluorescence (IF), which can utilize fluorescent dyes to multiplex and highlight a large number of different target cells (e.g., antibodies). Similarly, other molecular-based visualizations can include probe-based techniques that can be used to visualize, for example, messenger RNA (mRNA), microRNA (miRNA), and DNA in one or more tissue cells using bright field or IF visualization modalities. However, newer visualization modalities can include, for example, hyperspectral imaging, which can be used to distinguish dozens of different biomarkers labeled with different heavy metals and visualized using imaging mass spectrometry (IMC) or mass spectrometry (MS).

[0006] Therefore, it can be understood from the above that different visualization modalities can be deployed generally, to visualize and analyze the same target tissue cells based on the information that pathologists, scientists or clinicians try to determine. For example, H&E staining may be very suitable for widely visualizing specific tissue cells (such as cancer cells and proteins), and multiple IHC (mxIHC) or multiple IF (mxIF) may be very suitable for visualizing and identifying specific proteins and tissue cells. Various tissue cells (or classified biological features) perform differently in different visualization modalities. For example, the tissue cell image captured using a visualization modality may usually show significantly different images from the same tissue cell captured using another visualization modality, and therefore it may be challenging, even counterintuitive, for humans or computationally based models to recognize that these images depict the same tissue cell.

[0007] For example, although some visualization modalities can include the spatial features for making pathologists, scientists and / or clinicians easily identify or annotate target tissue cells, other visualization modalities may need to identify and classify target tissue cells based on computational modeling. However, in the absence of the verifiable annotation of target tissue cells for comparison and verification, even based on computational modeling, target tissue cells may not be accurately identified and classified. However, even if the traditional image registration technology that can be suitable for aligning some two-dimensional (2D) digital images is used, this traditional image registration technology is usually poorly performed when used to align each tissue cell and / or other tissue cell features. In addition, these traditional image registration technologies may require that 2D images are usually aligned to include the same imaging modality, and are therefore not technically suitable for aligning and tracking each tissue cell across various imaging or visualization modalities. Therefore, providing improved technology to track target tissue cells between different visualization modalities may be very useful. Summary of the invention

[0008] Embodiments of the present disclosure relate to one or more computing devices, methods and non-transitory computer-readable media, which can perform cross-modal cell-to-cell registration processes to identify, phenotype and track tissue cells captured using various visualization modalities. The identification and phenotype of tissue cells can use a visualization modality. A visualization modality can be used as verifiable real-value data, which is used to train one or more machine learning models to classify and phenotype the tissue cells captured using another visualization modality. That is, according to the currently disclosed embodiments, the real-value data for training one or more machine learning models to classify and phenotype tissue cells for specific visualization modalities (for example, wherein tissue cells cannot be easily determined by observation) can be generated by relying on, for example, spatial features, which can be easily determined by observation of human experts (for example, pathologists, scientists, clinicians or other medical and scientific experts) relative to different visualization modalities.

[0009] In fact, by utilizing a cross-modal cell-to-cell registration process, one or more tissue cells or tissue cell groups can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists) or other medical and scientific experts can more easily classify and phenotype immune cells (e.g., cancer cells, macrophages, regulatory T cells (Treg), CD8 cells, B lymphocytes, natural killer (NK) cells, fibroblasts, etc.). For example, this may lead to early detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients, including early and advanced stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). The current cross-modal cell-to-cell registration technology can further improve and enhance the efficacy of targeted immunotherapy (e.g., monoclonal antibodies (mAbs), T cell-engaging bispecific antibodies (bsAbs), antibody-drug conjugates (ADCs), etc.) for patients with NHL (including FL and DLBCL).

[0010] In certain embodiments, one or more computing devices may receive a plurality of images of a collection of tissue cells, wherein the plurality of images may include at least a first image comprising a first visualization modality and a second image comprising a second visualization modality. For example, in certain embodiments, the first visualization modality and the second visualization modality may be independently acquired by a whole slide imaging modality, a microscope modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality. The whole slide imaging modality may be selected from bright field or fluorescence imaging. The microscope modality may be selected from a bright field microscope, a fluorescence microscope, a confocal microscope, or a high content screening (HCS) microscope or a synthetic image generation. The non-optical imaging modality may be selected from imaging mass spectrometry flow technology (IMC) or myocardial perfusion imaging (MIBI). In certain embodiments, the first visualization modality or the second visualization modality includes a visualization modality based on a dye. In certain embodiments, the visualization modality based on a dye may be selected from a histological stain, fluorescence in situ hybridization (FISH), or immunofluorescence staining. In certain embodiments, the visualization modality based on a dye includes hematoxylin and eosin (H&E) staining. In certain embodiments, the first visualization modality or the second visualization modality may include immunostaining.

[0011] In some embodiments, one or more computing devices may identify pixel regions in the first image and the second image, wherein each of the pixel regions corresponds to a corresponding tissue cell in the set of tissue cells. For example, in some embodiments, before identifying the pixel regions in the first image and the second image, one or more computing devices may align the multiple images to vertically stack at least the first image and the second image. For example, in some embodiments, identifying the pixel regions in the first image and the second image may include performing nuclear segmentation on the pixel regions in the first image and the second image to segment the corresponding tissue cells in the set of tissue cells. In some embodiments, one or more computing devices perform a cell-to-cell registration process based on the identified pixel regions, and the cell-to-cell registration process includes matching a first pixel region corresponding to a first tissue cell in the first image with a second pixel region corresponding to a first tissue cell in the second image.

[0012] For example, in some embodiments, performing a cell-to-cell registration process may include: performing a scale-invariant Fourier transform (SIFT) alignment of the first image and the second image; performing a tile-level alignment of the first image and the second image, the tile-level alignment including a matrix transformation of the SIFT alignment of the first image and the second image; and performing a tile-level segmentation of the first image and the second image aligned at the tile level. In some embodiments, a tile-level segmentation is performed to segment each tissue cell in the set of tissue cells in the first image and the second image. In some embodiments, performing a cell-to-cell registration process may also include performing an object-level cell registration based on the tissue cells segmented at the tile level. In some embodiments, an object-level cell registration may be performed to match the first tissue cell in the first image with the first tissue cell in the second image. In some embodiments, before performing the cell-to-cell registration process, one or more computing devices may extract one or more features from the first image and the second image based on the identified pixel area. In some embodiments, one or more features are used to identify the first tissue cell in the first image and the first tissue cell in the second image.

[0013] In certain embodiments, one or more computing devices can then classify the first tissue cell into phenotype based on cell to cell registration process.In certain embodiments, phenotype can at least partially indicate disease pathology.For example, in certain embodiments, the first tissue cell is classified into phenotype can include classifying the first tissue cell into cancer cell, plasma cell, lymphocyte, macrophage or fibroblast based on one or more spatial features.In certain embodiments, the first tissue cell is classified into phenotype can include classifying the first tissue cell into cancer cell, macrophage, regulatory T cell (Treg), CD8 cell, B lymphocyte, natural killer (NK) cell or fibroblast based on one or more molecular annotations.In certain embodiments, disease pathology can include non-Hodgkin's lymphoma disease pathology, which can include follicular lymphoma (FL) or diffuse large B cell lymphoma (DLBCL).In certain embodiments, one or more computing devices can then generate phenotypic analysis table based on the phenotypic classification of the first tissue cell.

[0014] In some embodiments, during the training phase, one or more computing devices may determine the phenotypic category label for the first tissue cell based on one or more spatial features associated with the first pixel area corresponding to the first tissue cell, and determine the correspondence between the first pixel area corresponding to the first tissue cell and the second pixel area corresponding to the first tissue cell based on the cell-to-cell registration process. Embodiments of the present disclosure include training the model based on 1) the second pixel area corresponding to the first tissue cell and 2) the determined phenotypic category label for the first tissue cell. In some embodiments, during the inference phase, one or more computing devices may input a third image in a plurality of images into the trained model. In some embodiments, the third image includes a second visualization modality. In some embodiments, one or more computing devices may utilize the trained model to identify the pixel area corresponding to the first tissue cell in the third image, and output the predicted phenotypic category label for the first tissue cell in the third image. In some embodiments, the predicted phenotypic category label for the first tissue cell may correspond to the determined phenotypic category label for the first tissue cell.

[0015] In some embodiments, the model may include one or more deep neural networks. In some embodiments, one or more computing devices may determine the real value data for training the model by mapping the phenotypic classification of the first tissue cell to the first tissue cell in the first image and the first tissue cell mapped to the second image. In some embodiments, at least a subset of the real value data may include the real value data of molecular annotations. In another embodiment, at least a subset of the real value data may include the real value data of human annotations. In some embodiments, one or more computing devices may use a phenotypic analysis table to map the phenotypic classification of the first tissue cell to the first tissue cell in the first image and the first tissue cell mapped to the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] One or more drawings included herein are in color in accordance with 37 CFR § 1.84. Color drawings are necessary to illustrate the present invention. More specifically, Figure 1B , Figure 1C , Figure 3B , Figure 3D and FIG. 4A to FIG. 7 are one or more high-resolution tissue cells captured using various visualization modalities, all of which color images play a major role in enabling those skilled in the art to understand the present invention and such color images are the only practical medium for disclosing the patented subject matter.

[0017] Figure 1A An exemplary network of interacting computer systems including a cell-to-cell registration system is shown.

[0018] Figure 1B The cell-to-cell registration system workflow for performing the cross-modality cell-to-cell registration process is shown.

[0019] Figure 1C Shown is a running example of the cross-modality cell-to-cell registration process for identifying, phenotyping, and tracking tissue cells captured using various visualization modalities.

[0020] Figure 2A A flow chart of a method for providing a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities is shown.

[0021] Figure 2B Another flow chart of a method for providing a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities is shown.

[0022] Figure 3A A flowchart of a method for training one or more machine learning models to classify and phenotype tissue cells in one visualization modality image using class labels from different visualization modality images as true values ​​is shown.

[0023] Figure 3B An example of running one or more machine learning models trained to classify and phenotype tissue cells in one visualization modality image using class labels from different visualization modality images as ground truth values ​​is shown.

[0024] Figure 3C A flow chart of a method for utilizing one or more machine learning models trained to classify and phenotype tissue cells is shown.

[0025] Figure 3D Examples of operations utilizing one or more machine learning models trained to classify and phenotype tissue cells are shown.

[0026] Figure 4A and Figure 4B Immune cell annotations of five or more classes of immune cells on whole-slide images are shown.

[0027] Figure 4C and Figure 4D Immune cell annotations and phenotype tables showing seven or more classes of immune cells.

[0028] FIG. 5A to FIG. 5C One or more diagrams or implementation examples of tissue cell matching examples are shown.

[0029] Figure 6One or more diagrams or implementation examples of a cross-modality cell-to-cell registration process are shown.

[0030] Figure 7 One or more diagrams or implementation examples of a cross-modality cell-to-cell registration process are shown.

[0031] Figure 8 A diagram showing an example artificial intelligence (AI) architecture included as part of a network of interacting computing systems. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure relate to one or more computing devices, methods and non-transitory computer-readable media, which can perform cross-modal cell-to-cell registration process to identify, phenotypic analysis and track tissue cells captured using various visualization modalities. Such registration process can allow the visualization of various cells to be improved. The disclosed process can also allow the use of a visualization modality to identify and phenotypic analysis of tissue cells, which can be used as verifiable real-value data, which can be used to train one or more machine learning models to classify and phenotypic analysis of tissue cells captured using another visualization modality. That is, according to the currently disclosed embodiment, the real-value data for training one or more machine learning models to classify and phenotypic analysis of tissue cells for specific visualization modalities (for example, wherein tissue cells cannot be easily determined by observation) can be generated by relying on, for example, spatial features, which can be easily determined by the observation of human experts (for example, pathologists, scientists, clinicians or other medical and scientific experts) relative to different visualization modalities.

[0033] In fact, by utilizing a cross-modal cell-to-cell registration process, one or more tissue cells or tissue cell populations can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists) or other medical and scientific experts can more easily classify and phenotype immune cells (e.g., macrophages, regulatory T cells (Treg), CD8 cells, B lymphocytes, natural killer (NK) cells, etc.). For example, this may lead to early detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients, including early and advanced stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). The current cross-modal cell-to-cell registration process can further improve and enhance the efficacy of targeted immunotherapy (e.g., monoclonal antibodies (mAbs), T cell-engaging bispecific antibodies (bsAbs), antibody-drug conjugates (ADCs), etc.) for patients with NHL (including FL and DLBCL).

[0034] As used herein, "phenotype," "phenotyping," or "phenotyping" may be broadly understood to refer to the identity of one or more tissue cells or immune cells, the cellular state of one or more tissue cells or immune cells, the cellular program of one or more tissue cells or immune cells, the spatial location of one or more tissue cells or immune cells, the physical appearance of one or more tissue cells or immune cells, the count of one or more tissue cells or immune cells, or other similar human or machine-perceivable features or characteristics of one or more tissue cells or immune cells that may be used to classify one or more tissue cells or immune cells across various visualization modalities.

[0035] Figure 1A A network 100A of interacting computer systems of modalities is shown that may be adapted to perform a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualizations in accordance with presently disclosed embodiments.

[0036] In certain embodiments, the whole slide image generation system 101 can generate one or more whole slide images or histopathology images corresponding to a particular sample. For example, the image generated by the whole slide image generation system 101 can include a stained section of a biopsy sample. As another example, the image generated by the whole slide image generation system 101 can include a slide image of a liquid sample (e.g., a blood smear). As another example, the image generated by the whole slide image generation system 101 can include a fluorescent micrograph, such as a slide image depicting fluorescent in situ hybridization (FISH) after a fluorescent probe has been bound to a target DNA or RNA sequence.

[0037] Some types of samples (e.g., samples including tissues) can be processed by sample preparation system 105 to fix and / or embed the sample. Sample preparation system 105 can facilitate infiltration of samples with fixatives (e.g., liquid fixatives, such as formaldehyde solutions) and / or embedding materials (e.g., histological wax). For example, the sample fixing subsystem can fix the sample by exposing the sample to a fixative for at least a threshold amount of time (e.g., at least 3 hours, at least 6 hours, or at least 13 hours). The dehydration subsystem can dehydrate the sample (e.g., by exposing a portion of the fixed sample and / or the fixed sample to one or more ethanol solutions) and potentially use a clearing intermediate (e.g., which includes ethanol and histological wax) to remove the dehydrated sample. The sample embedding subsystem can infiltrate the sample (e.g., one or more corresponding predefined time periods) with heated (e.g., so in liquid form) histological wax. Histological wax can include paraffin and potentially one or more resins (e.g., styrene or polyethylene). The sample and wax can then be cooled, and the sample infiltrated with wax can then be sealed.

[0038] In certain embodiments, the sample slicer 107 can receive fixed and embedded samples and can produce a set of sections. The sample slicer 107 can expose the fixed and embedded samples to cool or cold temperatures. The sample slicer 107 can then cut the frozen sample (or its trimmed version) to produce a set of sections. Each section can have a thickness of, for example, less than 100 μm, less than 50 μm, less than 10 μm, or less than 5 μm. Each section can have a thickness of, for example, greater than 0.1 μm, greater than 1 μm, greater than 2 μm, or greater than 4 μm.

[0039] In certain embodiments, automated staining system 109 can promote dyeing one or more sample sections in sample section by exposing each section to one or more staining agents.Each section can be exposed to the staining agent of predefined volume for a predefined time period.In some cases, a single section is simultaneously or successively exposed to a variety of staining agents.In certain embodiments, each in one or more dyed sections can be presented to image scanner 115, which can capture the digital image of this section.Image scanner 115 can include a microscope camera.Image scanner 115 can capture digital images at multiple magnification levels (for example, using 10x object lens, 20x object lens, 40x object lens, etc.).The operation of image can be used to capture the selected portion of sample within the desired magnification range.

[0040] It should be understood that in some cases, one or more components of the whole slide image generation system 101 may be operated in conjunction with a human operator. For example, a human operator may move samples across various subsystems (e.g., sample preparation system 105 or subsystems of whole slide image generation system 101) and / or initiate or terminate operation of one or more subsystems, systems, or components of whole slide image generation system 101. In certain embodiments, whole slide image generation system 101 may transmit images generated by image scanner 115 to cell-to-cell registration system 121 in accordance with presently disclosed techniques. Although not shown, other intermediary devices (e.g., data storage areas of servers connected to whole slide image generation system 101 or whole slide image processing system 103) may also be used.

[0041] The network 100A can be used in various situations where scanning and evaluation of histopathology images (such as whole slide images) is an important part of the work. For example, the whole slide image processing system 103 can process histopathology images, including whole slide images, to classify digital pathology images and generate annotations for digital pathology images and related outputs. The tile generation module 111 can define a set of tiles or plaques for each digital pathology image. In order to define the set of tiles or plaques, the tile generation module 111 can segment the digital pathology image into the set of tiles or plaques. The tile generation module 111 can also define the tile or plaque size depending on the type of abnormality detected. For example, the tile generation module 111 can be configured to understand the type of tissue abnormality that the whole slide image processing system 103 will search for and can customize the tile or plaque size according to the tissue abnormality to optimize the detection.

[0042] In some embodiments, the tile embedding module 117 can generate an embedding for each tile or plaque in a corresponding feature embedding space. The embedding can be represented by the whole slide image processing system 103 as a feature vector for the tile or plaque. The tile embedding module 117 can utilize a neural network (e.g., one or more convolutional neural networks (CNNs)) to generate a feature vector representing each tile or plaque of the image. In a particular embodiment, the tile embedding neural network can be based on a residual neural network (ResNet) image classification network trained on a dataset based on natural (e.g., non-medical) images (such as the ImageNet dataset). In other embodiments, the tile embedding network utilized by the tile embedding module 117 can be an embedding network customized to handle a large number of tiles or plaques of large format images (such as digital pathology whole slide images).

[0043] In certain embodiments, the whole slide image access module 113 can manage requests for access to whole slide images from other modules of the whole slide image processing system 103 and the cell-to-cell registration system 121. For example, in some embodiments, the whole slide image access module 113 can receive a request to access, for example, multiplexed immunofluorescence (mxIF) images, hematoxylin and eosin (H&E) images, or other imaging or visualization modalities to provide to the cell-to-cell registration system 121. For example, in one embodiment, the output generation module 119 of the whole slide image processing system 103 can generate outputs corresponding to the accessed or requested mxIF images and H&E images. According to the presently disclosed embodiments, the output generation module 119 can then provide the outputs to the cell-to-cell registration system 121 for use in performing a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured by the mxIF images and H&E images.

[0044] Figure 1BA cell-to-cell registration system workflow 100B according to a currently disclosed embodiment is shown, which is used to perform a cross-modal cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities. In some embodiments, the system workflow 100B can begin with accessing a multiplexed immunofluorescence (mxIF) image 102, an mxIF image 104, and a hematoxylin and eosin (H&E) image 106. For example, in some embodiments, the mxIF image 102, the mxIF image 104, and the H&E image 106 can each include one or more tissue cells. Specifically, according to the currently disclosed embodiment, the mxIF image 102, the mxIF image 104, and the H&E image 106 can each include exactly the same tissue cells captured by different visualization modalities. In fact, although the current cross-modal cell-to-cell registration technology can be discussed in this article mainly with respect to mxIF and H&E visualization modalities, it should be understood that the current cross-modal cell-to-cell registration technology can be applied to any of the various visualization modalities, such as immunohistochemistry (IHC), fluorescence in situ hybridization (FISH), bright field microscopy, imaging mass cytometry (IMC), myocardial perfusion imaging (MIBI), and various other visualization modalities.

[0045] For example, in one embodiment, mxIF image 102 (e.g., "IF1") and mxIF image 104 (e.g., "IF2") can each include a molecular-based visualization, such as multiplexed immunofluorescence (mxIF), which can utilize fluorescent dyes to multiplex and highlight a large number of different target cells. For example, in some embodiments, mxIF image 102 (e.g., "IF1") and mxIF image 104 (e.g., "IF2") can each include an mxIF image of a tissue cell captured using a cyclic immunofluorescence process, which can be used to generate highly multiplexed images using a cyclic process (e.g., cycle) in which IF images of the same tissue cell are repeatedly collected and ultimately assembled.

[0046] Thus, in one embodiment, mxIF image 102 (e.g., "IF1") may represent an mxIF image captured during a first cycle of a cycling process, and mxIF image 104 (e.g., "IF2") may represent an mxIF image captured during a second cycle of a cycling process. Similarly, in one embodiment, H&E image 106 may include a dye-based visualization, such as hematoxylin and eosin (H&E), which may utilize the chemical properties of the dye molecules to bind to one or more specific tissue cells. In certain embodiments, as described below with respect to FIG. 3A to FIG. 3DIt is further understood that the mxIF image 102 (e.g., “IF1”) and the mxIF image 104 (e.g., “IF2”) may include one or more spatial features (e.g., any features suitable for informing or determining the phenotype of a cell by its spatial organization relative to neighboring cells or its location within a tissue or tissue region) suitable for allowing a pathologist, scientist, or clinician (e.g., an oncologist) to manually mark one or more tissue cells to be matched using a cell-to-cell registration process with one or more tissue cells in the H&E image 106 that may not include the spatial features.

[0047] In some embodiments, the mxIF image 102, the mxIF image 104, and the H&E image 106 may then be input into an alignment system 108. For example, in some embodiments, the alignment system 108 may include, for example, a method for vertically aligning (e.g., pixel-to-pixel alignment or whole slide image alignment by cross-correlation) the mxIF image 102, the mxIF image 104, and the H&E image 106 into a vertical stack 110A. In some embodiments, the alignment system 108 may perform a scale-invariant Fourier transform (SIFT) alignment of the mxIF image 102, the mxIF image 104, and the H&E image 106. In another embodiment, the alignment system 108 may perform a tile-level alignment of the mxIF image 102, the mxIF image 104, and the H&E image 106. In some embodiments, the tile-level alignment may be performed after or in conjunction with the SIFT alignment. For example, in one embodiment, the tile level alignment may include a SIFT-aligned matrix transformation of the mxIF image 102 , the mxIF image 104 , and the H&E image 106 .

[0048] In some embodiments, the system workflow 100B can continue, wherein the H&E image 106 can be scaled and rotated. For example, in one embodiment, the scaled and rotated H&E image 106 can be placed on top of the updated vertical stack 110B. In some embodiments, the system workflow 100B can continue with the image segmentation 112 of the updated vertical stack 110B. For example, in some embodiments, the image segmentation 112 may include, for example, nuclear segmentation, which can be used to, for example, segment the pixels of the mxIF image 102, mxIF image 104, and H&E image 106 within the updated vertical stack 110B along the boundaries of the nuclei of each tissue cell. In another embodiment, the image segmentation 112 may include, for example, semantic segmentation (e.g., pixel-by-pixel image segmentation), which can be used to, for example, segment and annotate the pixels of the mxIF image 102, mxIF image 104, and H&E image 106 within the updated vertical stack 110B on a pixel-by-pixel basis. In certain embodiments, the system workflow 100B may then proceed to a cross-modality cell-to-cell registration process 114 in accordance with the presently disclosed techniques.

[0049] In certain embodiments, the cross-modality cell-to-cell registration process 114 may include, for example, object-level cell registration to match individual tissue cells across the mxIF images 102, mxIF images 104, and H&E images 106 within the segmented and updated vertical stack 110B. For example, in some embodiments, the object-level cell registration may include multi-directionally matching, for example, polygons (e.g., two-dimensional (2D) or three-dimensional (3D) polygons) based on the identification of overlapping tissue cells and / or intersecting tissue cells across the mxIF images 102, mxIF images 104, and H&E images 106 within the segmented and updated vertical stack 110B. In some embodiments, polygons with multiple matches may also be tracked and recorded (e.g., recorded in the phenotyping table 118).

[0050] In some embodiments, based on the cross-modal cell-to-cell registration process 114, the system workflow 100B can then proceed to a phenotyping process 116 of one or more matched tissue cells. For example, in some embodiments, the phenotyping process 116 can include classifying one or more matched tissue cells into, for example, one or more categories of immune cells, such as macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, natural killer (NK) cells, etc. In some embodiments, the system workflow 100B can then proceed to store the aforementioned data in a phenotyping table 118. For example, in some embodiments, the phenotyping table 118 can include, for example, a record of matched and identified tissue cells determined based on the cross-modal cell-to-cell registration process 114. In some embodiments, the phenotyping table 118 can then be used to label an image 120 (e.g., a H&E image), which can then be used in a downstream task to train one or more machine learning models to classify various immune cells (e.g., macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, etc.) according to embodiments of the present invention. For example, in one embodiment, the phenotype analysis table 118 may include the immune cell phenotype, immune cell activation state, image identification or label, shape, and location of tissue cells of interest.

[0051] Figure 1C An example of operation 100C of a cross-modal cell-to-cell registration process for identifying, phenotyping, and tracking tissue cells or immune cells captured using various visualization modalities according to the currently disclosed embodiments is shown. As depicted, in one embodiment, an example of operation 100C of a cross-modal cell-to-cell registration process can be described with respect to an mxIF image 122 and an H&E image 124. As shown, the mxIF image 122 can be passed to one or more machine learning models 126, which can be used, for example, to segment the mxIF image 122, extract one or more features of interest corresponding to one or more tissue cells, and classify and phenotype one or more tissue cells. For example, in some embodiments, one or more tissue cells may include one or more immune cell populations (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.). In certain embodiments, the cell-to-cell registration model 128 can then be used to match one or more tissue cells in the mxIF image 122 that have been classified and phenotyped with one or more tissue cells corresponding to the H&E image 124.

[0052] Specifically, according to the presently disclosed embodiments, the cell-to-cell registration model 128 can perform a cell-to-cell registration process suitable for matching one or more immune cell populations identified in the mxIF image 122 with corresponding immune cells in the H&E image 124. The H&E image 124 can then be used as ground truth data to train one or more machine learning models to classify immune cells or other tissue cells. FIG. 3A to FIG. 3D and FIG. 4A to FIG. 4D As further understood, one or more immune cell populations identified in the mxIF image 122 can be classified and phenotyped, for example, by a human annotator (e.g., a scientist, pathologist, clinician, or other medical or scientific expert) based on one or more spatial features (e.g., any feature suitable for informing or determining the phenotype of a cell by its spatial organization relative to neighboring cells or its location within a tissue or tissue region). Thus, by then matching one or more immune cell populations to corresponding immune cells in the H&E image 124, the classification and phenotype analysis can be verifiably trusted (e.g., 90%-100% confidence score) as ground truth data for accurately training one or more machine learning models to classify immune cells or other tissue cells in the H&E image 124.

[0053] Figure 2A A flow chart of a method 200A for providing a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities according to the presently disclosed embodiments is shown. The method flow chart 200A can be performed using a network 100A of one or more processing devices, which can include hardware (e.g., a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or other processing devices suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0054] Method 200A may start at frame 202, where one or more processing devices receive a plurality of images in a collection of tissue cells, the plurality of images including a first image including at least a first visualization modality and a second image including a second visualization modality. For example, in one embodiment, the first image may include an mxIF visualization modality and the second image may include an H&E visualization modality. Method 200A may include frame 204, where one or more processing devices identify a first tissue cell in the collection of tissue cells in the first image and the first tissue cell in the second image. Method 200A may also include frame 206, where one or more processing devices perform a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image. For example, in some embodiments, the cell-to-cell registration process may include matching the first tissue cell in the first image with the first tissue cell in the second image. Method 200A may also include frame 208, where one or more processing devices classify the first tissue cell into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicating a disease pathology. For example, in certain embodiments, the cell-to-cell registration process may include matching one or more immune cell populations identified in the mxIF image with corresponding immune cells in the H&E image.

[0055] As described above, by utilizing a cross-modal cell-to-cell registration process, one or more tissue cells or tissue cell groups can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists) or other medical and scientific experts can more easily classify and phenotype immune cells (e.g., macrophages, regulatory T cells (Treg), CD8 cells, B lymphocytes, natural killer (NK) cells, etc.). For example, this may lead to early detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients, including early and progressive stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). The current cross-modal cell-to-cell registration technology can further improve and enhance the efficacy of targeted immunotherapy (e.g., monoclonal antibodies (mAbs), T cell-engaging bispecific antibodies (bsAb), antibody-drug conjugates (ADCs), etc.) for patients with NHL (including FL and DLBCL).

[0056] Figure 2BAnother flow chart of a method 200B for providing a cross-modality cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities according to the presently disclosed embodiments is shown. The method 200B can be performed using a network 100A of one or more processing devices, which can include hardware (e.g., a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or other processing devices suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0057] Method 200B may start at frame 210, where one or more processing devices receive multiple images of a collection of tissue cells, the multiple images including at least a first image containing a first visualization modality and a second image containing a second visualization modality. For example, in one embodiment, the first image may include an mxIF visualization modality and the second image may include an H&E visualization modality. Method 200B may then continue at frame 212, where one or more processing devices identify pixel regions in the first image and the second image, wherein each of the pixel regions corresponds to a corresponding tissue cell in the collection of tissue cells. For example, in some embodiments, nuclear segmentation may be used to segment pixels of mxIF images and H&E images, for example, along the boundaries of the nuclei of each tissue cell. Method 200B may include frame 214, where one or more processing devices perform a cell-to-cell registration process based on the identified pixel regions. For example, in some embodiments, the cell-to-cell registration process may include matching a first pixel region corresponding to a first tissue cell in the first image with a second pixel region corresponding to a first tissue cell in the second image. The method 200B may include block 216, wherein one or more processing devices classify the first tissue cells into a phenotype based on a cell-to-cell registration process, the phenotype at least partially indicative of a disease pathology. For example, in some embodiments, the cell-to-cell registration process may include matching one or more immune cell populations identified in the mxIF image with corresponding immune cells in the H&E image.

[0058] As described above, by utilizing a cross-modal cell-to-cell registration process, one or more tissue cells or tissue cell groups can be identified, classified, and phenotyped across various visualization modalities. In this way, pathologists, scientists, clinicians (e.g., oncologists) or other medical and scientific experts can more easily classify and phenotype immune cells (e.g., macrophages, regulatory T cells (Treg), CD8 cells, B lymphocytes, natural killer (NK) cells, etc.). For example, this may lead to early detection and diagnosis of non-Hodgkin's lymphoma (NHL) in patients, including early and progressive stages of NHL, such as follicular lymphoma (FL) and diffuse large B-cell lymphoma (DLBCL). The current cross-modal cell-to-cell registration technology can further improve and enhance the efficacy of targeted immunotherapy (e.g., monoclonal antibodies (mAbs), T cell-engaging bispecific antibodies (bsAb), antibody-drug conjugates (ADCs), etc.) for patients with NHL (including FL and DLBCL).

[0059] Figure 3A A flowchart of a method 300A for training one or more machine learning models to classify and phenotype tissue cells (e.g., immune cells) in a visualization modality image using class labels from different visualization modality images as ground truth values ​​is shown according to the presently disclosed embodiments. The flowchart 300A can be performed using a network 100A of one or more processing devices, which can include hardware (e.g., a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or other processing devices suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0060] Method 300A may start at frame 302, where one or more processing devices determine a phenotypic category label of a first tissue cell based on one or more spatial features associated with a first pixel region corresponding to a first tissue cell. Method 300A may include at frame 304, where one or more processing devices determine a correspondence between a first pixel region corresponding to a first tissue cell and a second pixel region corresponding to the first tissue cell based on a cell-to-cell registration process. Method 300A may include at frame 306, where one or more processing devices train a machine learning model based on 1) a second pixel region corresponding to a first tissue cell and 2) a determined phenotypic category label for the first tissue cell.

[0061] Figure 3B An operational example 300B is shown of training one or more machine learning models to classify and phenotype tissue cells (e.g., immune cells) in a visualization modality image using category labels from different visualization modality images as true values ​​according to the presently disclosed embodiments. Figure 3B It is about Figure 3A Operational example of the process shown and described. As depicted, in one embodiment, an operational example 300B of the cross-modality cell-to-cell registration process can be described with respect to an mxIF image 308 and an H&E image 310. However, it should be understood that the techniques of the present invention can be applied between any visualization modalities.

[0062] As shown, one or more tissue cells in mxIF image 308 can be annotated based on one or more spatial features (e.g., any feature suitable for informing or determining the phenotype of a cell by its spatial organization relative to neighboring cells or its location within a tissue or tissue region) and then matched with one or more corresponding tissue cells in H&E image 310 (e.g., as shown by the line with hollow circles extending between mxIF image 308 and H&E image 310). For example, Figure 3B Generally depicted, a human annotator (e.g., a scientist, pathologist, clinician, or other medical or scientific expert) can view the mxIF image 308 and classify and label one or more tissue cells or tissue cell populations (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) based on, for example, one or more spatial features (e.g., different colors of pixels of tissue cells, tissue cell count and density, tissue cell proportion and proximity, tissue cell area and diversity, and / or other features suitable for informing the phenotype of cells by their spatial organization relative to neighboring cells or the location of cells within a tissue or tissue region).

[0063] In certain embodiments, once one or more tissue cells or tissue cell groups in the mxIF image 308 are classified and labeled (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.), the corresponding one or more tissue cells or tissue cell groups in the H&E image 310 (e.g., the same tissue cells captured using different visualization modalities) can then be matched therewith according to the cross-modal cell-to-cell registration process as described herein. Specifically, the corresponding one or more tissue cells or tissue cell groups in the H&E image 310 are labeled based on information known and determined from the mxIF image 308 (e.g., "this tissue cell in the H&E image 310 is a lymphocyte"; "that tissue cell in the H&E image 310 is a fibroblast"; "this other tissue cell in the H&E image 310 is a macrophage", etc.).

[0064] In this way, even though the H&E image 310 itself may not include spatial features for easily determining and classifying tissue cells by matching tissue cells or tissue cell groups with those within the mxIF image 308, the H&E image 310 can be presented as proficient and accurate ground truth data for training one or more machine learning models 312 to classify and phenotype tissue cells or immune cells (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.). That is, in some embodiments, the cross-modal cell-to-cell registration process of the present invention can allow the H&E image 310 to be molecularly annotated so as to be presented as verifiably trustworthy ground truth data (e.g., as opposed to relying on spatial features observed by human annotators, particularly when certain spatial features may be present in one visualization modality and not in other visualization modalities).

[0065] In some embodiments, one or more machine learning models 312 (e.g., a deep neural network (DNN), a convolutional neural network (CNN), a fully connected neural network (FCNN), etc.) can then be trained (e.g., by supervised machine learning). For example, during the training phase, a dataset of training images 314 can be provided to one or more machine learning models 312. The training images 314 can be images with a first visualization modality or a second visualization modality (e.g., thousands of H&E training images). For example, the training images 314 can be H&E training images. In some embodiments, the dataset of H&E training images 314 can be annotated for training one or more machine learning models 312 to identify and classify tissue cells based on class labels determined and known from the mxIF images 308 (different visualization modalities). In some embodiments, one or more machine learning models 312 can generate one or more predictions for the class labels of the output images 316. In some embodiments, one or more predictions of class labels for the output image 316 can then be compared to the H&E image 310 (e.g., the true value) and used to iteratively update one or more machine learning models 312 (e.g., via back propagation or a loss calculated between one or more predictions of class labels for the output image 316 and the true value H&E image 310) until sufficiently trained (e.g., predicting class labels for tissue cells with an accuracy of 0.8, 0.9, or more on a range of 0.0 to 1.0) to classify and phenotype tissue cells in the H&E image.

[0066] Figure 3C A flow chart of a method 300C for utilizing one or more machine learning models trained to classify and phenotype tissue cells (e.g., immune cells) according to the presently disclosed embodiments is shown. The method 300C may be performed utilizing a network 100A of one or more processing devices, which may include hardware (e.g., a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), a neuromorphic processing unit (NPU), or any other processing device that may be suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0067] Method 300C may begin at box 318, where one or more processing devices input a third image of a plurality of images into a trained machine learning model, the third image comprising a second visualization modality. Flowchart 300C may then include at box 320, where one or more processing devices utilize the trained machine learning model to identify a pixel region in the third image corresponding to a first tissue cell. Flowchart 300C may include at box 322, where one or more processing devices utilize the trained machine learning model to output a predicted phenotypic class label for a first tissue cell in the third image, the predicted phenotypic class label for the first tissue cell corresponding to a determined phenotypic class label for the first tissue cell.

[0068] Figure 3D An example of operation 300D utilizing one or more machine learning models trained to classify and phenotype tissue cells according to the presently disclosed embodiments is shown. Figure 3C It is about Figure 3D As depicted, in some embodiments, during the inference phase, one or more trained machine learning models 324 (e.g., as described above with respect to Figure 3A and Figure 3B In some embodiments, the one or more trained machine learning models 324 may receive an input H&E image 326. In some embodiments, based on the input H&E image 326, the one or more trained machine learning models 324 may then generate an output H&E image 328 that includes one or more predicted class labels for one or more tissue cells or tissue cell populations (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.) within the output H&E image 328.

[0069] Figure 4A and Figure 4B Tissue cell or immune cell annotations of five or more categories of tissue cells or immune cells (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) on a whole slide image according to the presently disclosed embodiments are shown. For example, as depicted, Figure 4A Annotation of immune cells by human annotator-based classification and labeling 402 compared to model-based classification and labeling 404 is shown. Figure 4BAn H&E image 406 is shown, showing classification of more categories of tissue cells or immune cells (e.g., cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) marked with colors. For example, in some embodiments, according to the presently disclosed technology, an enlarged portion 408A of the H&E image 406 may show one or more tissue cells or a group of one or more tissue cells that are not marked, while an enlarged portion 408B of the H&E image 406 may show one or more tissue cells or a group of one or more tissue cells that are marked (e.g., as shown by the respective colored tissue cells corresponding to cancer cells, plasma cells, lymphocytes, macrophages, and fibroblasts, respectively).

[0070] Figure 4C and Figure 4D The tissue cell or immune cell annotation and phenotype table of seven or more categories of tissue cells or immune cells (e.g., cancer cells, macrophages, Tregs, CD8 cells, B lymphocytes, NK cells, fibroblasts, etc.) according to the presently disclosed embodiments are shown. For example, as depicted, Figure 4C Annotation of immune cells by human annotator-based classification and labeling 410 compared to model-based classification and labeling 412 is shown. Figure 4D Table 414 is shown showing the classification of more phenotypic classes of tissue cells or immune cells (eg, cancer cells, plasma cells, lymphocytes, macrophages, fibroblasts, etc.) coded with colors.

[0071] FIG. 5A to FIG. 5C One or more graphics or implementation examples of tissue cell matching examples according to the currently disclosed embodiments are shown. For example, the original image 500A of mxIF slide 502A, mxIF slide 506A and H&E slide 510A may include one or more tissue cells. It should be understood that tissue cell 504A, tissue cell 508A and tissue cell 512A may be identical tissue cells captured by different visualization modalities. For example, as depicted by mxIF slide 502A, one or more tissue cells 504A may be in different orientations, alignments, proximity, etc. on one or more of mxIF slide 502A, mxIF slide 506A and H&E slide 510A.

[0072] Figure 5B Real world image 500B is shown of mxIF image 502B corresponding to mxIF slide 502A, mxIF image 506B corresponding to mxIF slide 506A, and H&E image 510B corresponding to H&E slide 510A. Figure 5C , a graphical example of a cell-to-cell registration process as described herein is shown. Figure 5C As depicted in image 500C, each of tissue cells 504C, 508C, and 512C (e.g., tissue cell "001," tissue cell "002," tissue cell "003," and tissue cell "004") can be matched and tracked between mxIF slide 502A, mxIF slide 506A, and H&E slide 510A, such that it can be assumed that the phenotypic analysis or classification of tissue cells on one or more of mxIF slide 502A, mxIF slide 506A, and H&E slide 510A is representative of the other slides of mxIF slide 502A, mxIF slide 506A, and H&E slide 510A.

[0073] Figure 6 One or more graphics or implementation examples 600 of a cross-modal cell-to-cell registration process according to the presently disclosed embodiments are shown. Specifically, the cross-modal cell-to-cell registration process as shown by one or more graphics or implementation examples 600 corresponds to the above description of Figure 1A The cell-to-cell registration process 114 discussed. For example, the cross-modal cell-to-cell registration process includes performing (602) scale-invariant Fourier transform (SIFT) alignment of the cross-modal visualization images (e.g., alignment with downsampled pyramid layers based on SIFT), and performing (604) tile-level alignment of the cross-modal visualization images, including matrix transformation of the SIFT alignment of the cross-modal visualization images (e.g., the matrix transformation from the coarse image alignment is used to transform the tiles from the full-resolution layer that are again aligned with SIFT).

[0074] In some embodiments, the cross-modal cell-to-cell registration process may also include performing (606) tile-level segmentation of the tile-level aligned cross-modal visualization image, wherein tile-level segmentation may be performed to segment each tissue cell in the tile-level aligned cross-modal visualization image. In some embodiments, the cross-modal cell-to-cell registration process may also include performing (608) object-level cell registration based on the tile-level segmented tissue cells, wherein object-level cell registration is performed to match each tissue cell in the cross-modal visualization image. For example, in some embodiments, object-level cell registration may include multi-directionally matching, for example, polygons (e.g., 2D polygons or 3D polygons) based on the identification of overlapping tissue cells and / or intersecting tissue cells in the cross-modal visualization image.

[0075] Figure 7 An implementation example 700 of a cross-modality cell-to-cell registration process according to the presently disclosed embodiments is shown, thereby illustrating that tissue cells can be accurately matched, for example, between mxIF images and H&E images or across any of a variety of visualization modalities. Figure 7, the implementation example first image 704A shows the object-level cell registration of one or more tissue cells between the mxIF visualization modality and the H&E visualization modality, as shown by overlapping and / or intersecting polygons and / or contours corresponding to the mxIF visualization modality and the H&E visualization modality, respectively. For example, in some embodiments, the implementation example first image 702A shows the alignment of the segmentation polygons, wherein the H&E segmentation is shown in yellow (e.g., a filled polygon). Similarly, the implementation example first image 704A further shows the IF nuclear segmentation of the DAPI (4', 6-diamidino-2-phenylindole) channel shown as an outline, wherein the specific color of the outline indicates the number of matches. Specifically, the implementation example first image 702A shows a DAPI channel with a phenotype illustration, and the implementation example second image 704A shows an H&E image with the same segmentation superimposed.

[0076] In some embodiments, as further depicted, a phenotypic analysis table 702B may also be included and associated with the implementation example first image 702A. In some embodiments, the phenotypic analysis table 702B may include a list of potential phenotypes, which may be color-coded to correspond to visual overlaps and / or intersecting polygons and / or contours corresponding to the mxIF and H&E visualization modalities, and may also include a number indicating the number of matches for each different phenotype. Similarly, another phenotypic analysis table 704B may also be included and associated with the implementation example first image 704A. In some embodiments, the phenotypic analysis table 704B may include a list of potential phenotypes, which may be color-coded to correspond to visual overlaps and / or intersecting polygons and / or contours corresponding to the mxIF and H&E visualization modalities, and may also include a label indicating each different phenotype (e.g., "other", "Pax5+Ki+", "Pax5+Ki-", "all_neg", "CD68+", "CD3+CD3+Ki?", "CD8?CD335+", "CD8?FoxP3+Ki?"). In one embodiment, the "+" or "?" may include an imported symbolic representation, which may indicate additional characters that may be included as part of one or more phenotype labels. It should also be understood that a smaller image is included in the lower right corner of the implementation example first image 702A and the implementation example second image 704A, respectively, including, for example, an overlay of a mapped phenotype viewed as a colored dot.

[0077] Figure 8A diagram 800 is shown of an example artificial intelligence (AI) architecture 802 (which may be included as part of one or more of the network 100A of interactive computer systems as discussed above with respect to FIG. 1 ) in accordance with an embodiment of the present disclosure that may be used to perform a cross-modal cell-to-cell registration process to identify, phenotype, and track tissue cells captured using various visualization modalities. According to the presently disclosed embodiments, identification and phenotype analysis of tissue cells may use one visualization modality and be used as verifiable ground truth data for training one or more machine learning models to classify and phenotype tissue cells captured using another visualization modality.

[0078] In certain embodiments, the AI ​​architecture 802 may be implemented using, for example, one or more processing devices, which may include hardware (e.g., a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a vision processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), a deep learning processor (DLP), a tensor processing unit (TPU), and / or other processing devices that may be suitable for processing various omics data and making one or more decisions based thereon), software (e.g., instructions running / executed on one or more processing devices), firmware (e.g., microcode), or some combination thereof.

[0079] In certain embodiments, such as Figure 8 As depicted, AI architecture 802 may include machine learning (ML) algorithms and functions 804, natural language processing (NLP) algorithms and functions 806, expert systems 808, computer-based vision algorithms and functions 810, speech recognition algorithms and functions 812, planning algorithms and functions 814, and robotics algorithms and functions 816. In certain embodiments, ML algorithms and functions 804 may include any statistically based algorithms that may be suitable for finding patterns in large amounts of data (e.g., “big data” such as genomics data, proteomics data, metabolomics data, metagenomics data, transcriptomics data, and / or other omics data). For example, in certain embodiments, ML algorithms and functions 804 may include deep learning algorithms 818, supervised learning algorithms 820, and unsupervised learning algorithms 822.

[0080] In some embodiments, the deep learning algorithm 818 may include any artificial neural network (ANN) that can be used to learn deep representations and abstract concepts from large amounts of data. For example, the deep learning algorithm 818 may include ANNs such as perceptrons, multi-layer perceptrons (MLPs), autoencoders (AEs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTMs), gated recurrent units (GRUs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs) and deep Q-networks, neural autoregressive distribution estimation (NADEs), adversarial networks (ANs), attention models (AMs), spiking neural networks (SNNs), deep reinforcement learning, etc.

[0081] In some embodiments, the supervised learning algorithm 820 may include any algorithm that can be used to apply what has been learned in the past, for example, to new data to predict future events using labeled examples. For example, starting from an analysis of a known training data set, the supervised learning algorithm 820 can generate an inference function to make predictions about output values. The supervised learning algorithm 620 can also compare its output with the correct and expected output and find errors so that the supervised learning algorithm 820 can be modified accordingly. On the other hand, the unsupervised learning algorithm 822 may include any algorithm that can be applied, for example, when the data used to train the unsupervised learning algorithm 822 is neither classified nor labeled. For example, the unsupervised learning algorithm 822 can study and analyze how the system infers functions that describe hidden structures from unlabeled data.

[0082] In some embodiments, NLP algorithms and functions 806 may include any algorithm or function suitable for automatically manipulating natural language (such as speech and / or text). For example, in some embodiments, NLP algorithms and functions 806 may include content extraction algorithms or functions 824, classification algorithms or functions 826, machine translation algorithms or functions 828, question answering (QA) algorithms or functions 830, and text generation algorithms or functions 832. In some embodiments, content extraction algorithms or functions 824 may include means for extracting text or images from electronic documents (e.g., web pages, text editor documents, etc.) for use in, for example, other applications.

[0083] In some embodiments, the classification algorithm or function 826 may include any algorithm that can learn from the data input of the supervised learning model using a supervised learning model (e.g., logistic regression, naive Bayes, stochastic gradient descent (SGD), k-nearest neighbor, decision tree, random forest, support vector machine (SVM), etc.) and make new observations or classifications accordingly. The machine translation algorithm or function 828 may include any algorithm or function that can be suitable for automatically converting source text in one language into text in another language, for example. The QA algorithm or function 830 may include any algorithm or function that can be suitable for automatically answering questions asked by humans in, for example, natural language, such as algorithms or functions performed by voice-controlled personal assistant devices. The text generation algorithm or function 832 may include any algorithm or function that can be suitable for automatically generating natural language text.

[0084] In some embodiments, the expert system 808 may include any algorithm or function that may be suitable for simulating the judgment and behavior of a human or organization with professional knowledge and experience in a particular field (e.g., stock trading, medicine, sports statistics, etc.). The computer-based vision algorithm and function 810 may include any algorithm or function that may be suitable for automatically extracting information from an image (e.g., a photo image, a video image). For example, the computer-based vision algorithm and function 810 may include an image recognition algorithm 834 and a machine vision algorithm 836. The image recognition algorithm 834 may include any algorithm that may be suitable for automatically identifying and / or classifying, for example, objects, places, people, etc. that may be included in one or more image frames or other display data. The machine vision algorithm 836 may include any algorithm that may be suitable for allowing a computer to "see" or, for example, rely on an image sensor camera with dedicated optical elements to acquire images for processing, analysis, and / or measurement of various data characteristics for decision-making purposes.

[0085] In some embodiments, speech recognition algorithms and functions 812 may include any algorithm or function that may be suitable for recognizing spoken language and translating it into text, such as through automatic speech recognition (ASR), computer speech recognition, speech to text (STT) 838, or text to speech (TTS) 840, for example, in order to compute communication with one or more users via speech. In some embodiments, planning algorithms and functions 814 may include any algorithm or function that may be suitable for generating a sequence of actions, where each action may include its own set of prerequisites to be satisfied before performing the action. Examples of AI planning may include classical planning, reduction to other problems, temporal planning, probabilistic planning, preference-based planning, conditional planning, and the like. Finally, robotics algorithms and functions 816 may include any algorithm, function, or system that may enable one or more devices to replicate human behavior through, for example, motion, posture, performance tasks, decision making, emotions, and the like.

[0086] As used herein, "or" is inclusive and not exclusive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Furthermore, as used herein, "and" means both jointly and severally, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0087] As used herein, “automatically” and its derivatives mean “without human intervention” unless expressly indicated otherwise or the context indicates otherwise.

[0088] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Embodiments according to the present disclosure are particularly disclosed in the attached claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) can be claimed for protection in another claim category (e.g., system). Dependents or references in the attached claims are selected only for formal reasons. However, any subject matter resulting from intentional reference to any preceding claim (particularly multiple dependents) may also be claimed, so any combination of claims and their features may be disclosed and claimed regardless of the dependents selected in the attached claims. The subject matter that may be claimed includes not only a combination of features as listed in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. In addition, any embodiment and feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or with any feature of the attached claims.

[0089] The scope of the present disclosure covers all changes, substitutions, variations, alterations and modifications to the exemplary embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the corresponding embodiments herein as including specific parts, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any parts, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, in the appended claims, references to devices or systems or parts of devices or systems adapted to, arranged to, capable of, configured to, enabled to, operable to or operated to perform specific functions, cover the device, system, component, whether or not it or the specific function is activated, turned on or unlocked, as long as the device, system or component is so adapted, arranged, capable of, configured to, enabled to, operable or operated. In addition, although the present disclosure describes or illustrates the specific advantages provided by certain embodiments, certain embodiments may not provide, partially provide or provide all of these advantages.

[0090] Example

[0091] The examples provided are:

[0092] 1. A method for classifying tissue cells into one or more phenotypes, the method comprising: receiving, by one or more computing devices, a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; identifying a first tissue cell in the collection of tissue cells in the first image and the first tissue cell in the second image; performing a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image, the cell-to-cell registration process comprising matching the first tissue cell identified in the first image with the first tissue cell identified in the second image; and classifying the first tissue cell into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicating a disease pathology.

[0093] 2. The method according to Example 1 further comprises generating a phenotypic analysis table based on the phenotypic classification of the first tissue cells.

[0094] 3. The method according to embodiment 2 further comprises using the phenotyping table to map the phenotypic classification of the first tissue cells identified in the first image to the first tissue cells identified in the second image.

[0095] 4. A method according to any one of embodiments 1 to 3, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

[0096] 5. The method of embodiment 4, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

[0097] 6. The method of embodiment 4 or 5, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy or synthetic image generation.

[0098] 7. The method according to any one of embodiments 4 to 6, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

[0099] 8. The method of any one of embodiments 4 to 7, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

[0100] 9. The method of embodiment 8, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescence staining.

[0101] 10. The method of embodiment 9, wherein the histological staining comprises hematoxylin and eosin (H&E) staining or chromogenic staining.

[0102] 11. The method of any one of embodiments 1 to 10, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

[0103] 12. The method of any one of embodiments 1 to 11, wherein classifying the first tissue cell into the phenotype comprises classifying the cell state of the first tissue cell into an activated immune cell based on one or more molecular annotations.

[0104] 13. The method of any one of embodiments 1 to 12, wherein the first tissue cells comprise cancer cells, plasma cells, lymphocytes, macrophages, or fibroblasts.

[0105] 14. A method according to any one of embodiments 1 to 13, wherein classifying the first tissue cell into the phenotype includes classifying the first tissue cell into an immune cell based on one or more molecular annotations.

[0106] 15. The method of embodiment 14, wherein the immune cells comprise macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes or natural killer (NK) cells.

[0107] 16. The method of any one of embodiments 1 to 15, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

[0108] 17. The method of embodiment 16, wherein the NHL disease pathology comprises follicular lymphoma (FL).

[0109] 18. The method of embodiment 16, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

[0110] 19. A method for classifying tissue cells into one or more phenotypes, the method comprising: receiving, by one or more computing devices, a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; identifying pixel regions in the first image and the second image, wherein each of the pixel regions corresponds to a corresponding tissue cell in the collection of tissue cells; performing a cell-to-cell registration process based on the identified pixel regions, the cell-to-cell registration process comprising matching a first pixel region corresponding to a first tissue cell in the first image with a first tissue cell corresponding to the second image; and classifying the first tissue cell into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicating a disease pathology.

[0111] 20. The method according to embodiment 19 further comprises generating a phenotypic analysis table based on the phenotypic classification of the first tissue cells.

[0112] 21. The method according to any one of Examples 19 to 20 further comprises: determining a phenotypic category label for the first tissue cell based on one or more spatial features associated with the first pixel region corresponding to the first tissue cell; determining a correspondence between the first pixel region corresponding to the first tissue cell and the second pixel region corresponding to the first tissue cell based on the cell-to-cell registration process; and training a model based on 1) the second pixel region corresponding to the first tissue cell and 2) the determined phenotypic category label for the first tissue cell.

[0113] 22. The method according to embodiment 21 further comprises: inputting a third image among the multiple images into a trained model, the third image comprising the second visualization modality; utilizing the trained model to: identify a pixel region in the third image corresponding to the first tissue cell; and outputting the predicted phenotypic category label for the first tissue cell in the third image, the predicted phenotypic category label for the first tissue cell corresponding to the determined phenotypic category label for the first tissue cell.

[0114] 23. A method according to embodiment 21 or 22, wherein the model includes one or more deep neural networks (DNNs).

[0115] 24. The method according to any one of embodiments 21 to 23 further comprises: determining real value data for training the model by mapping the phenotypic classification of the first tissue cells to the first tissue cells in the first image and mapping to the first tissue cells in the second image.

[0116] 25. The method of embodiment 24, wherein at least a subset of the real-value data comprises molecularly annotated real-value data, or wherein at least a subset of the real-value data comprises human-annotated real-value data.

[0117] 26. The method according to embodiment 24 or 25 further comprises mapping the phenotypic classification of the first tissue cells in the first image to the first tissue cells in the second image using the phenotypic analysis table.

[0118] 27. A method according to any one of embodiments 19 to 26, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

[0119] 28. The method of embodiment 27, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

[0120] 29. The method of embodiment 27 or 28, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

[0121] 30. The method of any one of embodiments 27 to 29, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

[0122] 31. The method of any one of embodiments 27 to 30, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

[0123] 32. The method of embodiment 31, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescence staining.

[0124] 33. The method of embodiment 32, wherein the histological staining comprises hematoxylin and eosin (H&E) staining or chromogenic staining.

[0125] 34. The method of any one of embodiments 19 to 33, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

[0126] 35. The method according to any one of embodiments 19 to 34 further comprises: before identifying the pixel region in the first image and the second image, aligning the multiple images to vertically stack at least the first image and the second image.

[0127] 36. The method of any one of embodiments 19 to 35, wherein performing the cell-to-cell registration process comprises performing alignment of the first image with the second image.

[0128] 37. A method according to embodiment 36, wherein the cell-to-cell registration process further includes performing a tile-level alignment of the first image and the second image, and the tile-level alignment includes a matrix transformation of the alignment of the first image and the second image.

[0129] 38. A method according to embodiment 37, wherein the cell-to-cell alignment process further includes performing tile-level segmentation of the first image and the second image that are aligned at the tile level, and performing the tile-level segmentation to segment each tissue cell in the set of tissue cells in the first image and the second image.

[0130] 39. A method according to embodiment 38, wherein the cell-to-cell registration process further includes performing object-level cell registration based on tissue cells segmented at the tile level, and performing the object-level cell registration to match the first tissue cell in the first image with the first tissue cell in the second image.

[0131] 40. A method according to any one of embodiments 19 to 39, wherein identifying the pixel regions in the first image and the second image includes segmenting the pixel regions in the first image and the second image to segment corresponding tissue cells in the set of tissue cells.

[0132] 41. The method according to any one of Examples 19 to 40 further comprises: before performing the cell-to-cell registration process, extracting one or more features from the first image and the second image based on the identified pixel areas, and using the one or more features to identify the first tissue cell in the first image and the first tissue cell in the second image.

[0133] 42. A method according to any one of embodiments 19 to 41, wherein classifying the first tissue cell into the phenotype includes classifying the first tissue cell into the immune cell based on one or more spatial features.

[0134] 43. A method according to any one of embodiments 19 to 42, wherein classifying the first tissue cell into the phenotype includes classifying the first tissue cell into the immune cell based on one or more molecular annotations.

[0135] 44. The method of any one of embodiments 19 to 44, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

[0136] 45. The method of embodiment 44, wherein the NHL disease pathology comprises follicular lymphoma (FL).

[0137] 46. ​​The method of embodiment 44, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

[0138] 47. A system comprising one or more computing devices, the system comprising: one or more non-temporary computer-readable storage media comprising instructions; and one or more processors coupled to the one or more storage media, the one or more processors being configured to execute the instructions to perform a method according to any one of embodiments 1 to 46.

[0139] 48. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to perform the method of any one of embodiments 1 to 46.

Claims

1. A method for classifying tissue cells into one or more phenotypes, the method comprising: receiving a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; Identify a first tissue cell in the set of tissue cells in the first image and the first tissue cell in the second image; performing a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image, the cell-to-cell registration process comprising matching the first tissue cell identified in the first image with the first tissue cell identified in the second image; as well as The first tissue cells are classified into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease pathology. 2 . The method according to claim 1 , further comprising generating a phenotypic analysis table based on the phenotypic classification of the first tissue cells. 3 . The method of claim 2 , further comprising mapping the phenotypic classification of the first tissue cell identified in the first image to the first tissue cell identified in the second image using the phenotyping table.

4. The method of claim 1, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

5. The method of claim 4, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

6. The method of claim 4, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

7. The method of claim 4, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

8. The method of claim 4, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

9. The method of claim 8, wherein the dye-based visualization modality is selected from histological staining, fluorescence in situ hybridization (FISH), or immunofluorescence staining.

10. The method of claim 9, wherein the histological staining comprises hematoxylin and eosin (H&E) staining or chromogenic staining.

11. The method of claim 1, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining. 12 . The method of claim 1 , wherein classifying the first tissue cell into the phenotype comprises classifying a cell state of the first tissue cell as an activated immune cell based on one or more molecular annotations.

13. The method of claim 1, wherein the first tissue cell comprises a cancer cell, a plasma cell, a lymphocyte, a macrophage, or a fibroblast.

14. The method of claim 1, wherein classifying the first tissue cell into the phenotype comprises classifying the first tissue cell as an immune cell based on one or more molecular annotations.

15. The method of claim 14, wherein the immune cells comprise macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, or natural killer (NK) cells.

16. The method of claim 1, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

17. The method of claim 16, wherein the NHL disease pathology comprises follicular lymphoma (FL).

18. The method of claim 16, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

19. A system comprising one or more computing devices, the system comprising: one or more non-transitory computer-readable storage media including instructions; as well as One or more processors coupled to the one or more storage media, the one or more processors configured to execute the instructions to: receiving a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; Identify a first tissue cell in the set of tissue cells in the first image and the first tissue cell in the second image; performing a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image, the cell-to-cell registration process comprising matching the first tissue cell identified in the first image with the first tissue cell identified in the second image; as well as The first tissue cells are classified into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease pathology.

20. The system of claim 19, wherein the instructions further comprise instructions for generating a phenotyping table based on the phenotypic classification of the first tissue cells.

21. The system of claim 20, wherein the instructions further comprise instructions for mapping the phenotypic classification of the first tissue cell identified in the first image to the first tissue cell identified in the second image using the phenotyping table.

22. The system of claim 19, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

23. The system of claim 22, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

24. The system of claim 22, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

25. The system of claim 22, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

26. The system of claim 22, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

27. The system of claim 26, wherein the dye-based visualization modality is selected from histological staining, fluorescence in situ hybridization (FISH), or immunofluorescence staining.

28. The system of claim 27, wherein the histological stain comprises hematoxylin and eosin (H&E) staining or chromogenic staining.

29. The system of claim 19, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

30. The system of claim 19, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying a cell state of the first tissue cell as an activated immune cell based on one or more molecular annotations.

31. The system of claim 19, wherein the first tissue cell comprises a cancer cell, a plasma cell, a lymphocyte, a macrophage, or a fibroblast.

32. The system of claim 19, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell as an immune cell based on one or more molecular annotations.

33. The system of claim 32, wherein the immune cells comprise macrophages, regulatory T cells (Tregs), CD8 cells, B lymphocytes, or natural killer (NK) cells.

34. The system of claim 19, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

35. The system of claim 34, wherein the NHL disease pathology comprises follicular lymphoma (FL).

36. The system of claim 34, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

37. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to: receiving a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; Identify a first tissue cell in the set of tissue cells in the first image and the first tissue cell in the second image; performing a cell-to-cell registration process based on the first tissue cell identified in the first image and the first tissue cell identified in the second image, the cell-to-cell registration process comprising matching the first tissue cell identified in the first image with the first tissue cell identified in the second image; as well as The first tissue cells are classified into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease pathology.

38. The non-transitory computer-readable medium of claim 37, wherein the instructions further comprise instructions for generating a phenotyping table based on the phenotypic classification of the first tissue cells.

39. The non-transitory computer-readable medium of claim 38, wherein the instructions further comprise instructions for mapping the phenotypic classification of the first tissue cell identified in the first image to the first tissue cell identified in the second image using the phenotyping table.

40. The non-transitory computer-readable medium of claim 37, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

41. The non-transitory computer readable medium of claim 40, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

42. The non-transitory computer readable medium of claim 40, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

43. The non-transitory computer readable medium of claim 40, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

44. The non-transitory computer-readable medium of claim 40, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

45. The non-transitory computer readable medium of claim 44, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

46. ​​The non-transitory computer readable medium of claim 45, wherein the histological stain comprises hematoxylin and eosin (H&E) stain or a chromogenic stain.

47. The non-transitory computer-readable medium of claim 37, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

48. The non-transitory computer-readable medium of claim 37, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying a cell state of the first tissue cell as an activated immune cell based on one or more molecular annotations.

49. The non-transitory computer-readable medium of claim 37, wherein the first tissue cell comprises a cancer cell, a plasma cell, a lymphocyte, a macrophage, or a fibroblast.

50. The non-transitory computer-readable medium of claim 37, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell as an immune cell based on one or more molecular annotations.

51. The non-transitory computer-readable medium of claim 50, wherein the immune cell comprises a macrophage, a regulatory T cell (Treg), a CD8 cell, a B lymphocyte, or a natural killer (NK) cell.

52. The non-transitory computer readable medium of claim 37, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

53. The non-transitory computer readable medium of claim 52, wherein the NHL disease pathology comprises follicular lymphoma (FL).

54. The non-transitory computer readable medium of claim 52, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

55. A method for classifying tissue cells into one or more phenotypes, the method comprising: receiving a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; Identifying pixel regions in the first image and the second image, wherein each of the pixel regions corresponds to a corresponding tissue cell in the set of tissue cells; performing a cell-to-cell registration process based on the identified pixel regions, the cell-to-cell registration process comprising matching a first pixel region corresponding to a first tissue cell in the first image with a second pixel region corresponding to the first tissue cell in the second image; as well as The first tissue cells are classified into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease pathology.

56. The method of claim 55, further comprising generating a phenotypic analysis table based on the phenotypic classification of the first tissue cells.

57. The method of claim 55, further comprising: determining a phenotypic class label for the first tissue cell based on one or more spatial features associated with the first pixel region corresponding to the first tissue cell; Determining a correspondence between the first pixel region corresponding to the first tissue cell and the second pixel region corresponding to the first tissue cell based on the cell-to-cell registration process; as well as The model is trained based on 1) the second pixel region corresponding to the first tissue cell and 2) the determined phenotypic class label for the first tissue cell.

58. The method of claim 57, further comprising: inputting a third image of the plurality of images to the trained model, the third image comprising the second visualization modality; as well as Use the trained model to: Identifying a pixel region in the third image corresponding to the first tissue cell; as well as A predicted phenotypic class label for the first tissue cell in the third image is output, the predicted phenotypic class label for the first tissue cell corresponding to the determined phenotypic class label for the first tissue cell.

59. The method of claim 57, wherein the model comprises one or more deep neural networks (DNNs).

60. The method of claim 57, further comprising: Real value data for training the model is determined by mapping the phenotypic classification of the first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image.

61. The method of claim 60, wherein at least a subset of the real-value data comprises molecularly annotated real-value data, or wherein at least a subset of the real-value data comprises human-annotated real-value data.

62. The method of claim 60, further comprising mapping the phenotypic classification of the first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image using a phenotyping table.

63. The method of claim 55, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

64. The method of claim 63, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

65. The method of claim 63, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

66. The method of claim 63, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

67. The method of claim 63, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

68. The method of claim 67, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescence staining.

69. The method of claim 68, wherein the histological staining comprises hematoxylin and eosin (H&E) staining or chromogenic staining.

70. The method of claim 55, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

71. The method of claim 55, further comprising: Prior to identifying the pixel regions in the first and second images, the plurality of images are aligned to vertically stack at least the first and second images.

72. The method of claim 55, wherein performing the cell-to-cell registration process comprises performing an alignment of the first image with the second image.

73. The method of claim 72, wherein performing the cell-to-cell registration process further comprises performing a tile-level alignment of the first image with the second image, the tile-level alignment comprising a matrix transformation of the alignment of the first image with the second image.

74. The method of claim 73, wherein the cell-to-cell registration process further comprises performing tile-level segmentation of the first image and the second image that have been aligned at the tile level, and performing the tile-level segmentation to segment each tissue cell in the set of tissue cells in the first image and the second image.

75. The method of claim 74, wherein the cell-to-cell registration process further comprises performing object-level cell registration based on tissue cells segmented at the tile level, and performing the object-level cell registration to match the first tissue cell in the first image with the first tissue cell in the second image.

76. The method of claim 55, wherein identifying the pixel regions in the first image and the second image comprises performing segmentation of the pixel regions in the first image and the second image to segment corresponding tissue cells in the set of tissue cells.

77. The method of claim 55, further comprising: Before performing the cell-to-cell registration process, one or more features are extracted from the first image and the second image based on the identified pixel regions, and the one or more features are used to identify the first tissue cell in the first image and the first tissue cell in the second image.

78. The method of claim 55, wherein classifying the first tissue cell into the phenotype comprises classifying the first tissue cell into the phenotype based on one or more spatial features.

79. The method of claim 55, wherein classifying the first tissue cell into the phenotype comprises classifying the first tissue cell into the phenotype based on one or more molecular annotations.

80. The method of claim 55, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

81. The method of claim 80, wherein the NHL disease pathology comprises follicular lymphoma (FL).

82. The method of claim 80, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

83. A system comprising one or more computing devices, the system comprising: one or more non-transitory computer-readable storage media including instructions; as well as One or more processors coupled to the one or more storage media, the one or more processors configured to execute the instructions to: receiving a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; Identifying pixel regions in the first image and the second image, wherein each of the pixel regions corresponds to a corresponding tissue cell in the set of tissue cells; performing a cell-to-cell registration process based on the identified pixel regions, the cell-to-cell registration process comprising matching a first pixel region corresponding to a first tissue cell in the first image with a second pixel region corresponding to the first tissue cell in the second image; as well as The first tissue cells are classified into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease pathology.

84. The system of claim 83, wherein the instructions further comprise instructions for generating a phenotypic analysis table based on the phenotypic classification of the first tissue cells.

85. The system of claim 83, wherein the instructions further comprise instructions to: determining a phenotypic class label for the first tissue cell based on one or more spatial features associated with the first pixel region corresponding to the first tissue cell; Determining a correspondence between the first pixel region corresponding to the first tissue cell and the second pixel region corresponding to the first tissue cell based on the cell-to-cell registration process; as well as The model is trained based on 1) the second pixel region corresponding to the first tissue cell and 2) the determined phenotypic class label for the first tissue cell.

86. The system of claim 85, wherein the instructions further comprise instructions to: inputting a third image of the plurality of images to the trained model, the third image comprising the second visualization modality; as well as Use the trained model to: Identifying a pixel region in the third image corresponding to the first tissue cell; as well as A predicted phenotypic class label for the first tissue cell in the third image is output, the predicted phenotypic class label for the first tissue cell corresponding to the determined phenotypic class label for the first tissue cell.

87. The system of claim 85, wherein the model comprises one or more deep neural networks (DNNs).

88. The system of claim 85, wherein the instructions further comprise instructions to: Real value data for training the model is determined by mapping the phenotypic classification of the first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image.

89. The system of claim 88, wherein at least a subset of the real-value data comprises molecularly annotated real-value data, or wherein at least a subset of the real-value data comprises human-annotated real-value data.

90. The system of claim 88, wherein the instructions further comprise instructions for mapping the phenotypic classification of the first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image using a phenotyping table.

91. The system of claim 83, wherein each of the first visualization modality and the second visualization modality is independently acquired by a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

92. The system of claim 91, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

93. The system of claim 91, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

94. The system of claim 91, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

95. The system of claim 91, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

96. The system of claim 95, wherein the dye-based visualization modality is selected from histological staining, fluorescence in situ hybridization (FISH), or immunofluorescence staining.

97. The system of claim 96, wherein the histological stain comprises hematoxylin and eosin (H&E) staining or chromogenic staining.

98. The system of claim 83, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

99. The system of claim 83, wherein the instructions further comprise instructions to: Prior to identifying the pixel regions in the first and second images, the plurality of images are aligned to vertically stack at least the first and second images.

100. The system of claim 83, wherein the instructions for performing the cell-to-cell registration process further comprise instructions for performing alignment of the first image with the second image.

101. The system of claim 100, wherein the instructions for performing the cell-to-cell registration process further comprise instructions for performing tile-level alignment of the first image with the second image, the tile-level alignment comprising a matrix transformation of the alignment of the first image with the second image.

102. A system according to claim 101, wherein the instructions for performing the cell-to-cell alignment process further include instructions for performing tile-level segmentation of the first image and the second image that are aligned at the tile level, and the tile-level segmentation is performed to segment each tissue cell in the set of tissue cells in the first image and the second image.

103. A system according to claim 102, wherein the instructions for performing the cell-to-cell registration process further include instructions for performing object-level cell registration based on tissue cells segmented at the tile level, and performing the object-level cell registration to match the first tissue cell in the first image with the first tissue cell in the second image.

104. A system according to claim 83, wherein the instructions for identifying the pixel regions in the first image and the second image further include performing segmentation of the pixel regions in the first image and the second image to segment corresponding tissue cells in the set of tissue cells.

105. The system of claim 83, wherein the instructions further comprise instructions to: Before performing the cell-to-cell registration process, one or more features are extracted from the first image and the second image based on the identified pixel regions, and the one or more features are used to identify the first tissue cell in the first image and the first tissue cell in the second image.

106. The system of claim 83, wherein the instructions for classifying the first tissue cells into the phenotype further comprise instructions for classifying the first tissue cells into the phenotype based on one or more spatial features.

107. The system of claim 83, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell into the phenotype based on one or more molecular annotations.

108. The system of claim 83, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

109. The system of claim 108, wherein the NHL disease pathology comprises follicular lymphoma (FL).

110. The system of claim 108, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

111. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more processors to: receiving a plurality of images of a collection of tissue cells, the plurality of images comprising at least a first image comprising a first visualization modality and a second image comprising a second visualization modality; Identifying pixel regions in the first image and the second image, wherein each of the pixel regions corresponds to a corresponding tissue cell in the set of tissue cells; performing a cell-to-cell registration process based on the identified pixel regions, the cell-to-cell registration process comprising matching a first pixel region corresponding to a first tissue cell in the first image with a second pixel region corresponding to the first tissue cell in the second image; and The first tissue cells are classified into a phenotype based on the cell-to-cell registration process, the phenotype at least partially indicative of a disease pathology.

112. The non-transitory computer-readable medium of claim 111, wherein the instructions further comprise instructions for generating a phenotyping table based on the phenotypic classification of the first tissue cells.

113. The non-transitory computer readable medium of claim 111, wherein the instructions further comprise instructions to: determining a phenotypic class label for the first tissue cell based on one or more spatial features associated with the first pixel region corresponding to the first tissue cell; Determining a correspondence between the first pixel region corresponding to the first tissue cell and the second pixel region corresponding to the first tissue cell based on the cell-to-cell registration process; as well as The model is trained based on 1) the second pixel region corresponding to the first tissue cell and 2) the determined phenotypic class label for the first tissue cell.

114. The non-transitory computer readable medium of claim 113, wherein the instructions further comprise instructions to: inputting a third image of the plurality of images to the trained model, the third image comprising the second visualization modality; as well as Use the trained model to: Identifying a pixel region in the third image corresponding to the first tissue cell; as well as A predicted phenotypic class label for the first tissue cell in the third image is output, the predicted phenotypic class label for the first tissue cell corresponding to the determined phenotypic class label for the first tissue cell.

115. The non-transitory computer-readable medium of claim 113, wherein the model comprises one or more deep neural networks (DNNs).

116. The non-transitory computer readable medium of claim 113, wherein the instructions further comprise instructions to: Real value data for training the model is determined by mapping the phenotypic classification of the first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image.

117. The non-transitory computer-readable medium of claim 116, wherein at least a subset of the real-value data comprises molecularly annotated real-value data, or wherein at least a subset of the real-value data comprises human-annotated real-value data.

118. The non-transitory computer-readable medium of claim 116, wherein the instructions further comprise instructions for mapping the phenotypic classification of the first tissue cell to the first tissue cell in the first image and to the first tissue cell in the second image using a phenotypic analysis table.

119. The non-transitory computer-readable medium of claim 111, wherein each of the first visualization modality and the second visualization modality is independently acquired via a whole slide imaging modality, a microscopy modality, a non-optical imaging modality, or a spatial transcriptomics (ST) imaging modality.

120. The non-transitory computer readable medium of claim 119, wherein the whole slide imaging modality is selected from bright field or fluorescence imaging.

121. The non-transitory computer readable medium of claim 119, wherein the microscopy modality is selected from bright field microscopy, fluorescence microscopy, confocal microscopy, high content screening (HCS) microscopy, or synthetic image generation.

122. The non-transitory computer readable medium of claim 119, wherein the non-optical imaging modality is selected from imaging mass cytometry (IMC) or multiple ion beam imaging (MIBI).

123. The non-transitory computer-readable medium of claim 119, wherein at least one of the first visualization modality and the second visualization modality comprises a dye-based visualization modality.

124. The non-transitory computer readable medium of claim 123, wherein the dye-based visualization modality is selected from histological staining, fluorescent in situ hybridization (FISH), or immunofluorescent staining.

125. The non-transitory computer readable medium of claim 124, wherein the histological stain comprises hematoxylin and eosin (H&E) stain or a chromogenic stain.

126. The non-transitory computer-readable medium of claim 111, wherein at least one of the first visualization modality or the second visualization modality comprises immunostaining.

127. The non-transitory computer readable medium of claim 111, wherein the instructions further comprise instructions to: Prior to identifying the pixel regions in the first and second images, the plurality of images are aligned to vertically stack at least the first and second images.

128. The non-transitory computer-readable medium of claim 111, wherein the instructions for performing the cell-to-cell registration process further comprise instructions for performing alignment of the first image with the second image.

129. The non-transitory computer-readable medium of claim 128, wherein the instructions for performing the cell-to-cell registration process further comprise instructions for performing tile-level alignment of the first image with the second image, the tile-level alignment comprising a matrix transformation of the alignment of the first image with the second image.

130. A non-temporary computer-readable medium according to claim 129, wherein the instructions for performing the cell-to-cell alignment process further include instructions for performing tile-level segmentation of the first image and the second image that are aligned at the tile level, and performing the tile-level segmentation to segment each tissue cell in the set of tissue cells in the first image and the second image.

131. A non-temporary computer-readable medium according to claim 130, wherein the instructions for performing the cell-to-cell registration process further include instructions for performing object-level cell registration based on tissue cells segmented at the tile level, and performing the object-level cell registration to match the first tissue cell in the first image with the first tissue cell in the second image.

132. The non-transitory computer-readable medium of claim 111, wherein the instructions for identifying the pixel regions in the first image and the second image further comprise segmenting the pixel regions in the first image and the second image to segment corresponding tissue cells in the set of tissue cells.

133. The non-transitory computer readable medium of claim 111, wherein the instructions further comprise instructions to: Before performing the cell-to-cell registration process, one or more features are extracted from the first image and the second image based on the identified pixel regions, and the one or more features are used to identify the first tissue cell in the first image and the first tissue cell in the second image.

134. The non-transitory computer-readable medium of claim 111, wherein the instructions for classifying the first tissue cells into the phenotype further comprise instructions for classifying the first tissue cells into the phenotype based on one or more spatial features.

135. The non-transitory computer-readable medium of claim 111, wherein the instructions for classifying the first tissue cell into the phenotype further comprise instructions for classifying the first tissue cell into the phenotype based on one or more molecular annotations.

136. The non-transitory computer readable medium of claim 111, wherein the disease pathology comprises a non-Hodgkin's lymphoma (NHL) disease pathology.

137. The non-transitory computer readable medium of claim 136, wherein the NHL disease pathology comprises follicular lymphoma (FL).

138. The non-transitory computer readable medium of claim 136, wherein the NHL disease pathology comprises diffuse large B-cell lymphoma (DLBCL).

Images