Multi-modal subcellular segmentation method and system

Abstract

Systems and methods for multi-modal subcellular segmentation using photolysable biomarkers and / or transcriptomic readout density maps are disclosed. The systems and methods improve the accuracy of cell segmentation of the nucleus, cytoplasm, and cell membrane regions by using optical and bleach correction from a variety of photolysable morphological markers in combination with high quality 3D images acquired with high dynamic range scans and spatial transcriptomic readout density maps.

Description

[0001] Cross-references This application claims the benefit of U.S. Provisional Application No. 63 / 507,824, filed June 13, 2023, which is incorporated herein by reference in its entirety, and claims priority to the aforementioned application pursuant to Title 35, Section 120 of the United States Code. Background Technology

[0002] Cell segmentation is a crucial step in many biological and medical analyses, providing an important foundation for spatial analysis that utilizes the spatial distribution and characteristics of cells within tissue samples. Precise cell segmentation enables the study of spatial relationships and patterns, thus contributing to understanding tissue composition, cell communication, or disease progression. However, existing cell segmentation methods suffer from low accuracy and are highly dependent on certain experimental conditions and tissue types. Summary of the Invention Cell segmentation is a visual and image processing technique that divides an image or digital representation of a cell into individual regions or segments, each corresponding to a specific cell or cellular component. In space biology, cell segmentation enables a variety of specific analyses utilizing the spatial organization and characteristics of cells within tissue samples, including spatial transcriptomics, spatial analysis, spatial clustering and spatial interaction analysis, spatial single-cell analysis, spatial co-expression analysis, spatial visualization, and data integration. However, cell boundary information is subject to interference due to expression differences in the morphological markers used. In some cases, expression levels may be absent or saturated, leading to missed cells or incorrect segmentation. In current spatial transcriptomics detection, the lack of reliable cell membrane markers is a common challenge for achieving robust cell segmentation.

[0003] Existing cell segmentation methods typically employ limited morphological staining, generally falling between single-channel nuclear imaging and RGB imaging with up to three channels of nuclear and membrane / cytoplasmic staining. This invention relates to a novel method for segmenting cells from microscopic images of tissues. Photodegradable markers can provide an unrestricted source input channel by repeatedly re-staining and removing the markers within the tissue using UV irradiation and chemical washing. An additional number of morphological markers can significantly increase the information content regarding cell boundaries within the tissue. This invention also provides a novel method that utilizes spatial density generated from readout density obtained from spatial transcriptomics testing as a morphological image of the cell body, and performs cell segmentation on the generated readout density map.

[0004] On one hand, the system disclosed in this invention includes at least one processor and instructions executable by the at least one processor to provide multimodal segmentation applications. The system includes: (a) a software module configured to retrieve 3D scanned images of biological samples in high dynamic range (HDR) mode, wherein the biological samples are labeled with one of a plurality of morphological markers; (b) a software module configured to convert the 3D scanned images into 2D images to obtain an optimal focused region in a z-slice; or utilizing the entire 3D stacking volume; (c) a software module configured to retrieve readout density maps from transcriptomic analysis of the biological samples; and (d) a software module configured to perform subcellular segmentation of the biological samples based on the plurality of morphological markers or the readout density maps. In some embodiments, the morphological markers include fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents, or nucleic acid probes. In some embodiments, the images are microscopic imaging images, including images from an optical microscope, an electron microscope, or a scanning probe microscope. In some embodiments, the images may include images from a space molecular imager. In some embodiments, the images are applied for bleaching correction. In some embodiments, the transcriptomics assay includes gene expression assays using fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), capping analysis of gene expression, or single-cell RNA sequencing (scRNA-seq).

[0005] In some embodiments, the software module is configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than 200 images of the biological sample. In some embodiments, the biological sample is obtained at least partially through one or more of the following methods: biopsy, surgical resection, xenograft, animal model, fine-needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, lesion tissue sampling, and transplanted tissue sampling. In some embodiments, the biological sample comprises cells or tissue. In some embodiments, the cells include primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells. In some embodiments, the subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation. In some embodiments, the subcellular segmentation includes training a machine to learn an algorithm and / or applying a machine learning algorithm.

[0006] On the other hand, the present invention discloses a non-transitory computer-readable storage medium encoded with instructions executable by one or more processors to provide multimodal segmentation applications, the medium comprising: (a) a software module configured to retrieve 3D scan images of biological samples in high dynamic range (HDR) mode, wherein the biological samples are labeled with one of a variety of morphological markers; (b.1) an optional software module configured to convert the 3D scan images into 2D images, which utilizes techniques such as maximum intensity projection, focus stacking, and extended depth of field to preserve the optimal focus region from each z-slice; (b.2) or, a software module configured to deconvolve the 2D images to improve image sharpness and clarity while maintaining the z-position of these images in their stack of 3D scan images; (b.3) or, a software module configured to directly use each deconvoluted 2D slice to form a 3D volume stack for 3D cell segmentation; (c) a software module configured to retrieve readout density maps from transcriptomic analysis of biological samples; and (d) A software module configured to perform subcellular segmentation of biological samples based on multiple morphological markers or the readout density map. In some embodiments, the morphological markers include fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents, or nucleic acid probes. In some embodiments, the images are microscopic imaging images, including images from optical microscopes, electron microscopes, or scanning probe microscopes. In some embodiments, the images may include images from a spatial molecular imager. In some embodiments, the images are applied for bleaching correction. In some embodiments, the transcriptomics assay includes gene expression assays using fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), capping analysis of gene expression, or single-cell RNA sequencing (scRNA-seq). In some embodiments, the software module is configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more images of the biological sample. In some embodiments, the biological sample is obtained at least partially through one or more of the following methods: biopsy, surgical resection, xenograft, animal model, fine-needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, diseased tissue sampling, and transplanted tissue sampling.In some embodiments, the biological sample comprises cells or tissues. In some embodiments, the cells include primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells. In some embodiments, the subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, protrusion segmentation, or extracellular segmentation. In some embodiments, the subcellular segmentation includes training a machine to learn an algorithm and / or applying a machine learning algorithm.

[0007] In another aspect, the computer implementation method disclosed in this invention includes: (a) retrieving 3D scan images of biological samples in high dynamic range (HDR) mode by computer, wherein the biological samples are labeled with one of a variety of morphological marker species; (b.1) an optional software module configured to convert the 3D scan images into 2D images, which utilizes techniques such as maximum intensity projection, focus stacking, and extended depth of field to retain the best focus area from each z-slice; (b.2) or, a software module configured to deconvolve the 2D images to improve image sharpness and clarity while maintaining the z-position of these images in their stack of 3D scan images; (c) retrieving readout density maps from transcriptomic detection of the biological samples by computer; and (d) performing image preprocessing and segmentation of the biological samples based on multiple morphological markers or the readout density maps. In some embodiments, the morphological markers include fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-encoding tags, magnetic resonance imaging (MRI) contrast agents, or nucleic acid probes.

[0008] In some embodiments, the image is a microscopic imaging image, including images from an optical microscope, electron microscope, or scanning probe microscope. In some embodiments, the image may include an image from a space molecular imager. In some embodiments, the image is applied for bleaching correction. In some embodiments, the transcriptomics assay includes gene expression assays using fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), capping analysis of gene expression, or single-cell RNA sequencing (scRNA-seq). In some embodiments, the software module is configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more images of the biological sample. In some embodiments, the biological sample is obtained at least partially through one or more of the following methods: biopsy, surgical resection, xenograft, animal model, fine-needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, diseased tissue sampling, and transplanted tissue sampling. In some embodiments, the biological sample comprises cells or tissue. In some embodiments, the cells include primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells. In some embodiments, the subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation. In some embodiments, the subcellular segmentation includes training a machine to learn an algorithm and / or applying a machine learning algorithm.

[0009] The systems, media, and methods disclosed in this invention include combining morphological staining with proteomics and transcriptomics data to provide rich information about cell structures, including the nucleus, cytoplasm, and cell membrane regions. The systems, media, and methods disclosed in this invention include integrating protein images expressing similar structures and / or cell types into a multi-channel input morphological image. The systems, media, and methods disclosed in this invention include segmentation of extracellular processes. Various attributes and statistics can be calculated based on each cell, enabling researchers to study specific structures or regions of interest. These attributes and statistics can support subsequent analysis utilizing the cell location produced during the segmentation process, as well as the fluorescence and shape attributes of each cell. They can also serve as benchmark performance for various sample types, and as an indicator for evaluating ground-value-free segmentation on new datasets. In some embodiments, the cell segmentation performance is evaluated based on direct comparison. In some embodiments, the cell segmentation performance is evaluated by projecting statistics from a new dataset into an embedding space created from statistics from a reference dataset of similar sample types.

[0010] In some implementations, the original image is preprocessed before being input into at least one cell segmentation module. In some embodiments, image preprocessing is based on conventional image processing techniques such as denoising, deblurring, and image enhancement. In some embodiments, various denoising algorithms can be performed, including filter-based (Wiener, Gaussian, median), wavelet transform-based, total variational (TV) denoising, and deep learning-based methods. In some implementations, various deblurring algorithms can be performed, including nearest-neighbor deconvolution, Richardson-Lucy deconvolution, and total variational (TV) regularization. In some implementations, various image enhancement algorithms can be performed, including histogram equalization and contrast-limited adaptive histogram equalization (CLAHE), contrast stretching, gamma correction, and desharpening masks. In some implementations, cell segmentation is based on algorithms including Cellpose deep learning algorithms, adaptive thresholding, watershed algorithms, thresholding, region-based segmentation, active contours, convolutional neural networks (CNNs), level set methods, graph-based algorithms, fuzzy c-means clustering, deep watersheds, image morphology, or Markov random fields. In some implementations, nucleus segmentation, cytoplasm segmentation, protrusion segmentation, and extracellular segmentation are based on the same algorithm. In some implementations, nucleus segmentation, cytoplasm segmentation, protrusion segmentation, and extracellular segmentation are based on different algorithms.

[0011] In various embodiments, image segmentation software based on machine learning (ML) algorithms can be applied to create cell boundaries from fluorescence images obtained from protein assays. In some embodiments, the protein assay may include protein antibodies that bind to membrane proteins. In some embodiments, the ML algorithm applied to image segmentation may include semantic segmentation, instance segmentation, or a generative network for segmentation. In some embodiments, the image segmentation software may include ImageJ, CellProfiller, Cellpose, Ilastik, or QuPath, or any combination thereof. In various implementations, applications and usage scenarios include, but are not limited to, discovering and mapping cell types and cell states, phenotypes of tissue microenvironments, differential expression of cell types based on spatial context, quantifying subcellular expression, and identifying spatially resolved biomarkers. Attached Figure Description

[0012] The invention or application documents contain at least one color drawing. Upon request and payment of the necessary fees, the Patent Office will provide a copy of the invention or patent application publication with the color drawing.

[0013] The features and advantages of the present invention will be better understood by referring to the following detailed description of specific embodiments, in conjunction with the accompanying drawings, wherein: Figure 1 A non-limiting example of a computing device is shown; in this example, the device has one or more processors, memory, storage devices, and network interfaces; Figure 2 A non-limiting example of a web / mobile application delivery system is shown; in this example, the system provides a browser-based and / or native mobile user interface. Figure 3 A non-limiting example of a cloud-based web / mobile application delivery system is shown; in this example, the system includes elastic load balancing, auto-scaling web server and application server resources, and synchronously replicated databases; Figure 4 A-4C illustrates a non-limiting example of a multimodal cell acquisition and segmentation system. Figure 4 A illustrates an exemplary protein acquisition and 3D morphological sample collection. Figure 4 B illustrates an exemplary RNA acquisition and detection process. Figure 4 C illustrates an exemplary subcellular segmentation; Figure 5 A non-limiting example of a high dynamic range (HDR) imaging method is shown; Figure 6 A non-limiting example of a two-step morphological scan is shown; Figure 7A non-limiting example of an image affected by photobleaching due to adjacent exposures is shown; Figures 8A-8C A non-restrictive example of scaling and normalization is shown; Figure 9 A non-limiting example of color difference is shown; Figure 10 A non-limiting example of color difference correction is shown; Figure 11 A non-limiting example of point density generation is shown; Figure 12 A non-limiting example of generating density using a grid is shown; Figure 13A and 13B A non-limiting example of segmentation results based on morphological staining and point readout density is shown. Figure 13A Non-limiting examples of several protein channels are shown. Figure 13B A non-limiting example of a segmented processing channel is shown; Figure 14 A non-limiting example of a multimodal cell segmentation map is shown; Figure 15 A non-limiting example of processing cell segmentation using protein and point density inputs is shown; Figure 16 A non-limiting example of generating a new N-channel model is shown; Figure 17A and 17B A non-limiting example comparing the outputs of 2-channel and 5-channel models is shown. Figure 17A Non-limiting examples of various 2-channel models are shown. Figure 17B Non-limiting examples of various 5-channel model outputs are shown; Figures 18A-18D A non-limiting example of dense neuronal cells in the human brain is shown. Figure 18A Neuronal cells with nuclear markers are shown. Figure 18B Neuronal cells with membrane markers are shown. Figure 18C Astrocytes with GFAP markers were shown. Figure 18D The cell nucleus with DAPI markers is shown; Figure 19 A non-limiting example of extracellular process segmentation is shown; Figure 20 A-20C shows a non-limiting example of cell segmentation results. Figure 20 A shows a non-limiting example of an overlay of segmentation results of cellular and extracellular processes in the human brain. Figure 20 B shows a non-limiting example of the corresponding cell segmentation mask. Figure 20C illustrates a non-limiting example of detecting extracellular cell segmentation masks; Figure 21 A non-limiting example of cell segmentation output providing subcellular and extracellular masks and statistical data is shown; Figure 22 A non-limiting example of the objective cone angle used for the blur function is shown; Figure 23 A non-limiting example of a 3D segmentation process is shown; Figure 24 This demonstrates image preprocessing techniques such as image sharpening and enhancement; Figure 25 A non-limiting example of an image acquired using High Dynamic Range (HDR) settings is shown; Figure 26 A non-limiting example of the output produced by the nearest neighbor (NN) deconvolution method is shown; Figure 27 A non-limiting example of cell protrusion merging is shown; Figure 28 A non-limiting example of the cross-union ratio (IoU) merging results between nuclear segmentation output and membrane-plus nuclear segmentation is shown; Figures 29A-29C A non-restrictive example of intersection analysis is shown. Figure 29A A non-limiting example of IoU merging results in 2D image slices is shown. Figure 29B A non-limiting example of IoU merging results within a 3D volume is shown. Figure 29C A non-limiting example of a single cell labeled across all z-slices in the entire Z-stack is shown; and Figure 30 Non-limiting examples of various tissue cell segmentations are shown. Detailed Implementation

[0014] In some embodiments, the present invention discloses a system comprising at least one processor and instructions executable by the at least one processor to provide multimodal segmentation applications, the system comprising: (a) a software module configured to retrieve 3D scan images of biological samples in high dynamic range (HDR) mode, wherein the biological samples are labeled with one of a plurality of morphological markers; (b) a software module configured to convert the 3D scan images into 2D images to obtain optimal focused regions in z-slices; or to utilize the entire 3D stack; wherein each 2D z-slice has been deconvolved to enhance features and reduce blur; (c) a software module configured to retrieve readout density maps from transcriptomic analysis of the biological samples; and (d) a software module configured to perform subcellular segmentation of the biological samples based on the plurality of morphological markers or the readout density maps. In some embodiments, the morphological markers include one or more of the following: fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents, or nucleic acid probes, or any combination thereof.

[0015] In some embodiments, the image is a microscopic imaging image. In some embodiments, the microscopic imaging image includes images from an optical microscope, an electron microscope, or a scanning probe microscope. In some embodiments, the image may include an image from a space molecular imager. In some embodiments, the image is applied for bleaching correction. In some embodiments, the transcriptomics assay includes gene expression assays using fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), capping analysis of gene expression, or single-cell RNA sequencing (scRNA-seq), or any combination thereof. In some embodiments, the software module is configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than 200 images of the biological sample. In some embodiments, the biological sample is obtained at least partially through one or more of the following methods or any combination thereof: biopsy, surgical resection, xenograft, animal model, fine-needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, diseased tissue sampling, and transplanted tissue sampling. In some embodiments, the biological sample comprises cells or tissues. In some embodiments, the cells include one or more of primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells, or any combination thereof. In some embodiments, the subcellular segmentation includes one or more of nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation, or any combination thereof. In some embodiments, the subcellular segmentation includes training a machine to learn an algorithm or applying a machine learning algorithm, or both.

[0016] In some embodiments, the present invention describes a non-transitory computer-readable storage medium encoded with instructions executable by one or more processors to provide a multimodal segmentation application comprising: (a) a software module configured to retrieve 3D scan images of a biological sample in high dynamic range (HDR) mode, wherein the biological sample is labeled with one of a plurality of morphological markers; (b.1) an optional software module configured to convert the 3D scan images into 2D images, which utilizes techniques such as maximum intensity projection, focus stacking, and extended depth of field to preserve optimal focus regions from different z-slices; (b.2) or, a software module configured to deconvolve the 2D images to improve image sharpness and clarity while maintaining the z-positions of these images in their stack of 3D scan images; (c) a software module configured to retrieve readout density maps from transcriptomic analysis of the biological sample; and (d) a software module configured to perform subcellular segmentation of the biological sample based on the plurality of morphological markers or the readout density maps. In some embodiments, the morphological markers include one or more of fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents, or nucleic acid probes, or any combination thereof. In some embodiments, the images are microscopic images. In some embodiments, the microscopic images include images from an optical microscope, an electron microscope, or a scanning probe microscope, or any combination thereof. In some embodiments, the images may include images from a spatial molecular imager. In some embodiments, the images are applied for bleaching correction. In some embodiments, the transcriptomics assays include one or more of gene expression detection using fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), capping analysis of gene expression, or single-cell RNA sequencing (scRNA-seq), or any combination thereof. In some implementations, the software module is configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than 200 images of the biological sample.In some embodiments, the biological sample is obtained at least in part through one or more of the following methods: biopsy, surgical resection, xenotransplantation, animal model, fine-needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, lesion tissue sampling, and transplanted tissue sampling, or any combination thereof. In some embodiments, the biological sample comprises cells or tissue. In some embodiments, the cells comprise primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells, or any combination thereof. In some embodiments, subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation; or any combination thereof. In some embodiments, subcellular segmentation includes training a machine to learn an algorithm or applying a machine learning algorithm; or both.

[0017] On the other hand, the present invention also describes a computer-implemented method that may include: (a) retrieving 3D scan images of a biological sample in high dynamic range (HDR) mode by computer, wherein the biological sample is labeled with one of a plurality of morphological marker species; (b.1) an optional software module configured to convert the 3D scan images into 2D images, which utilizes techniques such as maximum intensity projection, focus stacking, and extended depth of field to preserve the optimal focus area from different z-slices; (b.2) or, a software module configured to deconvolve the 2D images to improve image sharpness and clarity while maintaining the z-position of these images in their stack of 3D scan images; (c) retrieving readout density maps from transcriptomic analysis of the biological sample by computer; and (d) segmenting the biological sample based on a plurality of morphological markers or the readout density maps. In some embodiments, the morphological markers include one or more of fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents, or nucleic acid probes, or any combination thereof. In some embodiments, the images are microscopic images. In some embodiments, the microscopic images include images from an optical microscope, an electron microscope, or a scanning probe microscope; or any combination thereof. In some embodiments, the images may include images from a spatial molecular imager. In some embodiments, the images are applied for bleaching correction. In some embodiments, the transcriptomics assay includes gene expression assays, which include one or more of fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), capping analysis of gene expression, or single-cell RNA sequencing (scRNA-seq); or any combination thereof. In some embodiments, the software module is configured to retrieve at least 1 image, at least 3 images, at least 5 images, at least 10 images, at least 15 images, at least 20 images, at least 30 images, at least 35 images, at least 40 images, at least 45 images, at least 50 images, at least 55 images, at least 60 images, at least 65 images, at least 70 images, at least 80 images, at least 90 images, at least 100 images, at least 120 images, at least 150 images, at least 200 images, or more than 200 images of the biological sample. In some embodiments, the biological sample is obtained at least partially through one or more of the following: biopsy, surgical resection, xenograft, animal model, fine-needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, diseased tissue sampling, and transplanted tissue sampling, or any combination thereof. In some embodiments, the biological sample comprises cells or tissues.In some embodiments, the cells include primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells; or any combination thereof. In some embodiments, the subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation; or any combination thereof. In some embodiments, the subcellular segmentation includes training a machine to learn an algorithm or applying a machine learning algorithm, or both. definition Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0018] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include their plural references unless the context clearly specifies otherwise. Unless otherwise stated, any reference to “or” in this invention is intended to include “and / or.”

[0019] Throughout this specification, the terms "some embodiments," "other embodiments," or "specific embodiments" all refer to features, structures, or characteristics described in connection with that embodiment being included in at least one embodiment. Therefore, the expressions "in some embodiments," "in other embodiments," or "in specific embodiments" appearing in different locations in this specification do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Multimodal cell acquisition and segmentation system In some embodiments, the systems, media, and methods disclosed in this invention may include combining morphological staining with proteomics and transcriptomics data. In some embodiments, the systems, media, and methods disclosed in this invention may also include receiving information about cell structures. In some embodiments, cell structures may include the nucleus, cytoplasm, or cell membrane regions, or any combination thereof. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images of proteins expressing similar structures into a multi-channel input morphological image. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images of proteins expressing similar cell types into a multi-channel input morphological image. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images of proteins expressing similar cell types and similar structures into a multi-channel input morphological image. In some implementations, the multi-channel input image may include at least 1 channel input morphological image, at least 3 channel input morphological image, at least 5 channel input morphological image, at least 10 channel input morphological image, at least 15 channel input morphological image, at least 20 channel input morphological image, at least 30 channel input morphological image, at least 35 channel input morphological image, at least 40 channel input morphological image, at least 45 channel input morphological image, at least 50 channel input morphological image, at least 55 channel input morphological image, at least 60 channel input morphological image, at least 65 channel input morphological image, at least 70 channel input morphological image, at least 80 channel input morphological image, at least 90 channel input morphological image, at least 100 channel input morphological image, at least 120 channel input morphological image, at least 150 channel input morphological image, or at least 200 channel input morphological image, or more than 200 channel input morphological images, including the incremental values ​​contained therein.

[0020] In some embodiments, the systems, media, and methods disclosed in this invention may include a multi-step segmentation process. In some embodiments, the multi-step segmentation process may include segmenting the nucleus region. In some embodiments, the multi-step segmentation process may include segmenting the cytoplasm region. In some embodiments, the multi-step segmentation process may include segmenting the cell membrane region. In some embodiments, the systems, media, and methods disclosed in this invention may include segmenting extracellular material to segment regions. In some embodiments, the segmented region may be a segmented region of an organ or tissue. In some embodiments, the segmented region may be a segmented region of the brain. In some embodiments, the segmented region may be located distal to the cell body or not connected to the cell body, or both. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images expressing similar structures into one or more input morphological images. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images expressing similar cell types into one or more input morphological images. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images expressing similar structures into one or more input morphological images, and integrating images expressing similar cell types into one or more input morphological images. In some embodiments, the input morphological images may be multi-channel input morphological images. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images expressing similar structures into a 5-channel input morphological image. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images expressing similar cell types into a 5-channel input morphological image. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating images expressing both similar structures and similar cell types into a 5-channel input morphological image.

[0021] In some embodiments, the systems, media, and methods disclosed in this invention may include integrating one or more protein images. In some embodiments, the number of integrated protein images may be about 1, about 2, about 3, about 4, about 5, about 10, about 20, about 30, about 40, about 50, about 60, about 70, about 80, about 90, about 100, about 120, about 140, about 160, about 180, about 200, or more than about 200 protein images. In some embodiments, the systems, media, and methods disclosed in this invention may include additional input channels for processing the raw RNA point readout density to supplement signal-deficient regions of morphological markers.

[0022] In some embodiments, the systems, media, and methods disclosed in this invention may include the segmentation of extracellular processes. In some embodiments, the systems, media, and methods disclosed in this invention may also include one or more images identifying each cell with a unique ID. In some embodiments, the systems, media, and methods disclosed in this invention may also include one or more images identifying each connected process with a unique ID. In some embodiments, the systems, media, and methods disclosed in this invention may also include one or more images identifying each cell and each connected process with a unique ID. The systems, media, and methods disclosed in this invention may also include identifying different portions of each cellular region with "region tags". As a non-restrictive example, Figure 4 A-4C illustrates a multimodal cell acquisition and segmentation system. For example, such as... Figure 4 As shown in Figure A, protein acquisition and 3D morphological sample collection are performed. The protein sample and morphological markers, such as photodegradable markers, are incubated together and scanned into 3D images in high dynamic range (HDR) mode. The markers are then removed, for example, using UV irradiation and histochemical washing. For example, the markers are repeatedly restained and removed using UV irradiation and histochemical washing. As an example, RNA acquisition and detection are performed as follows... Figure 4 As shown in B. Next, for example, after scanning the RNA reporter hybridization sample, RNA spots are detected. Then, for example, markers are removed using UV irradiation and sample washing.

[0023] In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images from one or more tissue scans. In some embodiments, one or more tissue scans may include one or more scans of one or more tissues stained with morphological markers. In some embodiments, one or more tissue scans may include one or more scans of one or more tissues incubated with a protein reporter gene. In some embodiments, one or more tissue scans may include one or more scans of one or more tissues incubated with an RNA reporter gene. In some embodiments, the systems, media, and methods disclosed in this invention may include removing one or more reporter genes from one or more tissue samples. In some embodiments, one or more reporter genes in one or more tissue samples may be reduced. In some embodiments, one or more reporter genes in one or more tissue samples may be bleached, lysed, or both. In some embodiments, reporter genes in one or more tissue samples can be reduced by applying one or more of the following: UV light, heat, protease, endonuclease, nuclease, esterase, ribonuclease, ribonuclease A (RNase A), ribonuclease T1 (RNase T1), ribonuclease H (RNase H), disulfide reducing agents (e.g., dithiothreitol, tris(2-carboxyethyl)phosphine), salt buffer, base, hydrogen-bonded unstable solvent (e.g., formamide, DMSO), or any combination thereof.

[0024] In some non-limiting examples, after processing images of one or more proteins using morphological and density measurements based on RNA detection, subcellular segmentation is performed, such as... Figure 4 As shown in C. Figure 4 As shown in C, the cell segmentation process includes performing, for example, background subtraction, normalization, nuclear region segmentation, cytoplasmic region segmentation, cell membrane region segmentation, or any combination thereof. In some embodiments, different portions of each cell region can be identified by a unique "region label." In some embodiments, various attributes and statistics for each cell can be calculated. In some embodiments, the various attributes and statistics calculated for each cell can provide information about a specific structure or region of interest, or both. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of biological samples. In some embodiments, the retrieved images of biological samples may be labeled with morphological markers. In some embodiments, the morphological markers may include one or more of the following, or any combination thereof, of fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, genetically encoded tags, magnetic resonance imaging (MRI) contrast agents, or nucleic acid probes.

[0025] In some embodiments, the systems, media, and methods disclosed in this invention may include generating readout density maps using transcriptomic detection of biological samples. In some embodiments, the density map may be a heatmap, hierarchical statistical map, nuclear density map, point density map, contour map, or scale symbol map. In some cases, the heatmap may also include a graphical representation of data using color to represent numerical values. In some cases, the heatmap can be used to visualize the density or intensity of a specific phenomenon in a two-dimensional space. In some cases, the two-dimensional space may be a geographic region, an image, or a grid, or any combination thereof. In some embodiments, the density map may display the density or concentration of a specific attribute or event within a given region. In some embodiments, the density map may provide visual information about the distribution and intensity of data points or events in a spatial domain.

[0026] In some implementations, transcriptomics assays may include one or more of the following: gene expression assays using fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), capping analysis of gene expression, or single-cell RNA sequencing (scRNA-seq), or any combination thereof.

[0027] In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of biological samples. In some embodiments, the biological samples may be obtained at least in part through one or more or any combination of biopsy collection, surgical resection, xenotransplantation, animal models, fine-needle aspiration, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, lesion tissue sampling, or transplanted tissue sampling.

[0028] In some embodiments, the biological sample may include cells or tissues, or both. In some embodiments, the cells may include one or more of primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells, or any combination thereof.

[0029] In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of spatially resolved high-gravity gene expression data from tissues. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of spatially resolved high-gravity protein data from tissues. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of RNA detection. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of RNA detection to analyze the entire transcriptome. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of RNA detection to analyze the entire transcriptome of tissue on a single formalin-fixed paraffin-embedded (FFPE) sample. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of RNA detection to analyze the entire transcriptome of tissue on a fresh-frozen (FF) sample slide. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images of protein detection to generate quantitative analysis, spatial analysis, or both of multiple proteins. In some embodiments, quantitative analysis, spatial analysis, or both of multiple proteins can be generated from a single FFPE or FF sample slide. In some embodiments, FFPE or FF tissue slides may be stained with barcode in situ hybridization probes that bind to endogenous mRNA transcripts. In some implementations, the user can select a region of interest (ROI) for analysis. In some implementations, each ROI fragment can be further subdivided into illuminated areas (AOIs) based on tissue morphology. In some implementations, a spatial molecular imager can optically cleave each AOI fragment separately. In some implementations, the spatial molecular imager can collect expression tags or barcodes for each AOI fragment separately. In some implementations, the tags or barcodes can be used for subsequent sequencing or data processing, or both.

[0030] Computing System refer to Figure 1 The diagram illustrates a block diagram of an exemplary machine including a computer system 100 (e.g., a processing or computing system) within which a set of instructions is executable to cause a device to perform or implement any one or more aspects and / or methods of the static code scheduling of the present invention. The components described herein are merely illustrative and do not limit the scope or functionality of any hardware, software, embedded logic components, or combinations of two or more such components used in implementing specific embodiments.

[0031] The computer system 100 may include one or more processors 101, memory 103, and storage devices 108, which communicate with each other and with other components via a bus 140. The bus 140 may also connect to a display 132, one or more input devices 133 (e.g., keypad, keyboard, mouse, stylus, etc.), one or more output devices 134, one or more storage devices 135, and various tangible storage media 136. All these components may be connected to the bus 140 directly or via one or more interfaces or adapters. For example, various tangible storage media 136 may be connected to the bus 140 via a storage media interface 126. The computer system 100 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (e.g., mobile phones or PDAs), laptops or notebooks, distributed computer systems, computing grids, or servers.

[0032] Computer system 100 includes one or more processors 101 (e.g., a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), or a quantum processing unit (QPU)) that perform functions. Processor 101 optionally includes a cache unit 102 for temporarily storing instructions, data, or computer addresses locally. Processor 101 is configured to assist in executing computer-readable instructions. Computer system 100 may be... Figure 1 The components shown provide functionality as a result of processor 101 executing non-transitory processor-executable instructions contained in one or more tangible computer-readable storage media (e.g., memory 103, storage device 108, storage device 135, and / or storage medium 136). The computer-readable medium may store software implementing a particular embodiment, and processor 101 may execute that software. Memory 103 may read the software from one or more other computer-readable media (e.g., mass storage devices 135, 136) or one or more other sources via a suitable interface, such as network interface 120. The software may cause processor 101 to perform one or more processes or steps of one or more processes described or illustrated in this invention. Performing such processes or steps may include defining data structures stored in memory 103 and modifying said data structures according to instructions from the software.

[0033] The memory 103 may include various components (e.g., machine-readable media), including but not limited to random access memory components (e.g., RAM 104) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phase-change random access memory (PRAM), etc.), read-only memory components (e.g., ROM 105), and any combination thereof. ROM 105 can be used to unidirectionally transfer data and instructions to processor 101, while RAM 104 can be used to bidirectionally transfer data and instructions with processor 101. ROM 105 and RAM 104 may include any suitable tangible computer-readable media described below. In one example, the memory 103 may store a basic input / output system 106 (BIOS), which includes basic routines that facilitate the transfer of information between elements of computer system 100, for example, during startup.

[0034] Fixed memory 108 is optionally bidirectionally connected to processor 101 via storage control unit 107. Fixed memory 108 provides additional data storage capacity and may also include any suitable tangible computer-readable medium described below. Memory 108 can be used to store operating system 109, executable files 110, data 111, applications 112, etc. Memory 108 may also include optical disc drives, solid-state storage devices (e.g., flash-based systems), or any combination thereof. Where appropriate, information in memory 108 may be incorporated into memory 103 as virtual memory. In one example, storage device 135 may be removably connected to computer system 100 via storage device interface 125 (e.g., via an external port connector (not shown)). Specifically, storage device 135 and associated machine-readable medium may provide non-volatile and / or volatile storage for machine-readable instructions, data structures, program modules, and / or other data to computer system 100. In one example, software may reside wholly or partially on machine-readable medium on storage device 135. In another example, software may reside wholly or partially in processor 101. Bus 140 connects multiple subsystems. In this invention, references to a bus, where appropriate, include one or more digital signal lines used to provide common functions. Bus 140 can be any of a variety of bus structures, including but not limited to memory buses, memory controllers, peripheral buses, local buses, and any combination thereof, using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Enhanced ISA (EISA) buses, Micro Channel Architecture (MCA) buses, Video Electronics Standards Association Local Bus (VLB), Peripheral Component Interconnect (PCI) buses, PCI-Express (PCI-X) buses, Accelerated Graphics Port (AGP) buses, HyperTransport (HTX) buses, Serial Advanced Technology Attachment (SATA) buses, and any combination thereof.

[0035] Computer system 100 may also include input device 133. In one example, a user of computer system 100 may input commands and / or other information into computer system 100 via input device 133. Examples of input device 133 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices (e.g., mice or touchpads), touchpads, touchscreens, multi-touch screens, joysticks, styluses, game controllers, audio input devices (e.g., microphones, voice response systems, etc.), optical scanners, video or still image acquisition devices (e.g., cameras), and any combination thereof. In some embodiments, the input device is a Kinect, Leap Motion, etc. Input device 133 may be connected to bus 140 via any of a variety of input interfaces 123 (e.g., input interface 123), including but not limited to serial interfaces, parallel interfaces, game ports, USB, FireWire interfaces, Thunderbolt interfaces, or any combination thereof.

[0036] In a particular embodiment, when computer system 100 is connected to network 130, computer system 100 can communicate with other devices connected to network 130, particularly mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, etc. Communication to and from computer system 100 can be sent via network interface 120. For example, network interface 120 can receive incoming communication (e.g., requests or responses from other devices) in the form of one or more data packets (e.g., Internet Protocol (IP) packets) from network 130, and computer system 100 can store the incoming communication in memory 103 for processing. Similarly, computer system 100 can store outgoing communication (e.g., requests or responses to other devices) in memory 103 in the form of one or more data packets and transmit them to network 130 via network interface 120. The processor 101 can access these communication data packets stored in memory 103 for processing.

[0037] Examples of network interface 120 include, but are not limited to, network interface cards, modems, and any combination thereof. Examples of network 130 or network segment 130 include, but are not limited to, distributed computing systems, cloud computing systems, wide area networks (WANs) (e.g., the Internet, corporate networks), local area networks (LANs) (e.g., networks associated with offices, buildings, campuses, or other relatively small geographical areas), telephone networks, direct connections between two computing devices, peer-to-peer networks, and any combination thereof. Networks (e.g., network 130) may employ wired and / or wireless communication modes. Generally, any network topology can be used.

[0038] Information and data can be displayed via display 132. Examples of display 132 include, but are not limited to, cathode ray tube (CRT), liquid crystal display (LCD), thin film transistor liquid crystal display (TFT-LCD), organic liquid crystal display (OLED) (e.g., passive matrix OLED (PMOLED) or active matrix OLED (AMOLED) displays), plasma display, and any combination thereof. Display 132 can be connected to processor 101, memory 103, and fixed storage 108, and other devices (e.g., input device 133) via bus 140. Display 132 is connected to bus 140 via a video interface, and data transmission between display 132 and bus 140 can be controlled by graphics controller 121. In some embodiments, the display is a video projector. In some embodiments, the display is a head-mounted display (HMD), such as a VR headset. In other embodiments, suitable VR head-mounted devices include, but are not limited to, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOYE VR, Zeiss VR One, Avegant Glyph, Freefly VR head-mounted devices, etc. In other embodiments, the display is a combination of devices disclosed in this invention.

[0039] In addition to the display 132, the computer system 100 may also include one or more other peripheral output devices 134, including but not limited to audio speakers, printers, storage devices, and any combination thereof. Such peripheral output devices may be connected to the bus 140 via an output interface 124. Examples of the output interface 124 include, but are not limited to, serial ports, parallel connections, USB ports, firewire ports, thunderbolt ports, and any combination thereof.

[0040] Alternatively or concurrently, the computer system 100 may provide functionality, either in place of or in conjunction with software, by means of hard-wired or otherwise embodied logic in the circuitry, to perform one or more processes or steps of one or more processes described or illustrated in this invention. Software as referred to in this invention may encompass logic, and the logic mentioned may encompass software. Furthermore, where appropriate, the computer-readable medium mentioned may encompass circuitry (e.g., an IC) storing software for execution, circuitry embodying logic for execution, or both. This invention covers any suitable combination of hardware, software, or both. Those skilled in the art will understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed in this invention can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above in a general functional manner.

[0041] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed in this invention may be implemented or performed by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof intended to perform the functions described in this invention. The general-purpose processor may be a microprocessor, but alternatively, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0042] The steps of the methods or algorithms described in conjunction with the embodiments disclosed in this invention may be embodied directly in hardware, or in a software module executed by one or more processors, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and storage medium may reside in an ASIC, which may reside in a user terminal. Alternatively, the processor and storage medium may reside as discrete components in the user terminal.

[0043] According to the present invention, suitable computing devices include, but are not limited to, distributed computing systems, cloud computing platforms, server clusters, server computers, desktop computers, laptops, notebook computers, ultrabooks, netbooks, network tablets, handheld computers, internet devices, mobile smartphones, and tablet computers. Those skilled in the art will also recognize that televisions, video players, and digital music players with optional computer network connectivity are also suitable for the systems described in this invention. In various embodiments, suitable tablet computers include manual, board-type, and convertible tablet computers known to those skilled in the art.

[0044] In some embodiments, the computing device includes an operating system configured to execute executable instructions. The operating system is, for example, software that includes programs and data, manages the device's hardware, and provides services for the execution of applications. Those skilled in the art will recognize that suitable server operating systems include, but are not limited to, FreeBSD, OpenBSD, and NetBSD. ® Linux, Apple ® Mac OS X Server ® Oracle ® Solaris ® Windows Server ® and Novell ® NetWare ® Those skilled in the art will also recognize that suitable personal computer operating systems include, but are not limited to, Microsoft. ® Windows ® Apple ® Mac OS X ® UNIX ® and UNIX-like operating systems, such as GNU / Linux ® In some implementations, the operating system is provided by cloud computing. Those skilled in the art will also recognize that suitable mobile smartphone operating systems include, but are not limited to: Nokia... ® Symbian ® OS, Apple ® iOS ® Research in Motion ® BlackBerry OS ® Google ® Android ® Microsoft ® Windows Phone ® OS, Microsoft ® Windows Mobile ® OS, Linux ® and Palm ® WebOS ® .

[0045] Non-transitory computer-readable storage medium In some embodiments, the platforms, systems, media, and methods disclosed in this invention include one or more non-transitory computer-readable storage media encoded with programs comprising instructions executable by an operating system of an optional networked computing device. In other embodiments, the computer-readable storage medium is a tangible component of a computing device. In other embodiments, the computer-readable storage medium may optionally be removable from the computing device. In some embodiments, the computer-readable storage medium includes, but is not limited to, CD-ROMs, DVDs, flash memory devices, solid-state storage, disk drives, magnetic tape drives, optical disc drives, distributed computing systems including cloud computing systems and services, etc. In some cases, programs and instructions are permanently, substantially permanently, semi-permanently, or non-transitory encoded on the medium. Computer program In some embodiments, the platforms, systems, media, and methods disclosed in this invention include at least one computer program or its use. A computer program includes a series of instructions executable by one or more processors of a computing device's CPU to perform a specified task. The computer-readable instructions can be implemented as program modules that perform a specific task or implement a specific abstract data type, such as functions, objects, application programming interfaces (APIs), computational data structures, etc. As will be appreciated by those skilled in the art based on the disclosure of this invention, computer programs can be written in various versions of various languages.

[0046] The functionality of computer-readable instructions can be combined or distributed in various environments as needed. In some embodiments, a computer program includes a sequence of instructions. In some embodiments, a computer program includes multiple sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from multiple locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plugins, extensions, add-ons, or additional components, or combinations thereof.

[0047] Web Applications In some embodiments, the computer program includes a web application. As will be appreciated by those skilled in the art based on the disclosure of this invention, in various embodiments, the web application utilizes one or more software frameworks and one or more database systems. In some embodiments, the web application is based on, for example, Microsoft... ®The web application is created using the .NET or Ruby on Rails (RoR) software framework. In some implementations, the web application utilizes one or more database systems, including but not limited to relational, non-relational, object-oriented, associational, XML, and document-oriented database systems. In other implementations, suitable relational database systems include, but are not limited to, Microsoft... ® SQL Server, MySQL™ and Oracle ® Those skilled in the art will also recognize that, in various embodiments, the web application is written in one or more versions of one or more languages. The web application can be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, the web application is written to some extent in a markup language, such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or Extensible Markup Language (XML). In some embodiments, the web application is written to some extent in a presentation definition language, such as Cascading Style Sheets (CSS). In some embodiments, the web application is written to some extent in a client-side scripting language, such as Asynchronous JavaScript and XML (AJAX), Flash... ® ActionScript, JavaScript or Silverlight ® In some implementations, the web application is written to some extent in a server-side coding language, such as ActiveServer Pages (ASP) or ColdFusion. ® Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python™, Ruby, Tel, Smalltalk, WebDNA ® Or Groovy. In some implementations, the web application is written to some extent in a database query language, such as Structured Query Language (SQL). In some implementations, the web application integrates with enterprise server products, such as IBM... ® Lotus Domino ® In some implementations, the web application includes a media player component. In various other implementations, the media player component utilizes one or more of a variety of suitable multimedia technologies, including but not limited to Adobe... ® Flash ® HTML5, Apple ®QuickTime ® Microsoft ® Silverlight ® Java™ and Unity ® .

[0048] refer to Figure 2 In a specific embodiment, the application providing system includes one or more databases 200 accessed by a relational database management system (RDBMS) 210. Suitable RDBMSs include Firebird, MySQL, PostgreSQL, SQLite, Oracle Database, Microsoft SQL Server, IBM DB2, IBM Informix, SAP Sybase, Teradata, etc. In this embodiment, the application providing system also includes one or more application servers 220 (such as Java servers, .NET servers, PHP servers, etc.) and one or more web servers 230 (such as Apache, IIS, GWS, etc.). The web servers optionally expose one or more web services through an application programming interface (API) 240. The system provides a browser-based and / or mobile native user interface over a network, such as the Internet.

[0049] refer to Figure 3 In a specific embodiment, the application providing system alternatively has a distributed cloud-based architecture 300, and includes elastically load-balanced, auto-scaling web server resources 310 and application server resources 320, as well as synchronously replicated database 330.

[0050] standalone application In some implementations, a computer program may include a standalone application that runs as a separate computer process, rather than an add-in to an existing process, such as a non-plugin. Those skilled in the art will recognize that standalone applications are typically compiled. A compiler is a computer program that translates source code written in a programming language into binary object code, such as assembly language or machine code. Suitable compilation programming languages ​​include, but are not limited to, C, C++, Objective-C, COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB.NET, or combinations thereof. Compilation is typically performed, at least partially, to create an executable program. In some implementations, a computer program includes one or more executable compiled applications.

[0051] Software Module In some embodiments, the platforms, systems, media, and methods disclosed in this invention include software, server, and / or database modules, or their use. Given the content of this disclosure, the software modules are created using techniques known to those skilled in the art, employing machines, software, and languages ​​known in the art. The software modules disclosed in this invention can be implemented in various ways. In various embodiments, the software module may include files, code segments, programming objects, programming structures, distributed computing resources, cloud computing resources, or combinations thereof. In other embodiments, the software module may include multiple files, multiple code segments, multiple programming objects, multiple programming structures, multiple distributed computing resources, multiple cloud computing resources, or combinations thereof. In various embodiments, one or more software modules include, but are not limited to, web applications, mobile applications, standalone applications, and distributed or cloud computing applications. In some embodiments, the software module resides in a single computer program or application. In other embodiments, the software module exists in multiple computer programs or applications. In some embodiments, the software module is hosted on a single machine. In other embodiments, the software module is hosted on multiple machines. In further embodiments, the software module is hosted on a distributed computing platform, such as a cloud computing platform. In some embodiments, the software module is hosted on one or more machines at a specific location. In other implementations, the software module is hosted on one or more machines in multiple locations.

[0052] database In some embodiments, the systems, media, and methods disclosed in this invention include one or more databases, or utilization thereof. In view of the disclosure of this invention, those skilled in the art will recognize that many databases are suitable for storing and retrieving user information, research information, slide information, field of view (FoV) information, flow cell information, image information, genomic information, transcriptomics information, and proteomics information. In various embodiments, suitable databases include, but are not limited to, relational databases, non-relational databases, object-oriented databases, object databases, entity-relational model databases, association databases, XML databases, document-oriented databases, and graph databases. Other non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, Sybase, and MongoDB. In some embodiments, the database is Internet-based. In other embodiments, the database is Web-based. In still other embodiments, the database is cloud-based. In a particular embodiment, the database is a distributed database. In other embodiments, the database is based on one or more local computer storage devices.

[0053] In some embodiments, the data stored in the database may include biological image data. In some embodiments, biological image data may include, but is not limited to, microscopic images (e.g., photomicrographs) of formalin-fixed paraffin-embedded (FFPE) samples of cells, microscopic images (e.g., photomicrographs) of formalin-fixed paraffin-embedded (FFPE) samples of tissues, microscopic images (e.g., photomicrographs) of fresh-frozen (FF) samples of cells, or microscopic images (e.g., photomicrographs) of fresh-frozen (FF) samples of tissues, or any combination thereof. In some embodiments, data from a single slide may be divided into two datasets. In some cases, these two datasets may include an RNA detection dataset and a protein detection dataset. In some embodiments, data from a single slide may be merged into a single dataset. In some embodiments, a dataset may include RNA detection data and protein detection data. In some embodiments, image data may be two-dimensional image data. In some embodiments, image data may be three-dimensional image data. In some embodiments, data includes, but is not limited to, "omics" data, such as genomic data, proteomics data, metabolomics data, metagenomic data, phenomics data, or transcriptomics data, or any combination thereof. In some embodiments, omics data may be associated with image data. In some implementations, omics data may be spatially associated with image data. In some cases, omics data may be spatially associated with image data in two dimensions, three dimensions, or both. In some implementations, omics data may be associated with image data as an overlay of metadata or images, or both. In some implementations, the data may include, but is not limited to, patient data, demographic data, diagnostic data, disease data, treatment data, research data, or any combination thereof. Image acquisition The systems, media, and methods disclosed in this invention may include retrieving images from one or more tissue scans of tissue stained with morphological markers. In some embodiments, the tissue may be incubated with a protein reporter gene. In some embodiments, the tissue may be incubated with an RNA reporter gene. In some embodiments, the systems, media, and methods disclosed in this invention may include removing the reporter gene from the tissue. In some embodiments, the reporter gene may be bleached. In some embodiments, the reporter gene may be cleaved. In some embodiments, bleaching, cleavage, or both may be performed by applying one or more of the following: UV light, heat, protease, endonuclease, nuclease, esterase, ribonuclease, RNase A, RNase T1, RNase H, disulfate bond reducing agents (e.g., dithiothreitol, tris(2-carboxyethyl)phosphine), salt buffer, base, hydrogen bond destabilizing solvent (e.g., formamide, DMSO), or any combination thereof.

[0054] In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving images. In some embodiments, the images may include images imaged by a microscope. In some embodiments, the microscope images may include images imaged by an optical microscope, an electron microscope, or a scanning probe microscope, or any combination thereof. In some embodiments, the images may include images from a space molecular imager.

[0055] In some embodiments, morphological signals are acquired. In some embodiments, RNA protein signals are acquired. In some embodiments, both morphological and RNA protein signals are acquired. In some embodiments, morphological and RNA protein signals are acquired using high dynamic range imaging (HDR) mode. In some embodiments, HDR mode is used to acquire signals to improve the dynamic range of the signals. Figure 5 As shown, in a non-limiting example, image 1 was obtained using an exposure time of 0.0125 seconds, where details are located in bright areas; while image 2 was obtained using an exposure time of 0.1 seconds, where details are located in dark areas. In some implementations, the resulting image can be generated by dividing the accumulator by a counter. Figure 5 As shown in the non-limiting example, the final intensity level is equivalent to an exposure time of 0.0125 seconds, which is 1 / 8 of the nominal exposure time.

[0056] In some embodiments, the systems, media, and methods disclosed herein may include integrating images of proteins expressing similar structures or cell types, or both, into a multi-channel input morphological image. In some embodiments, the systems, media, and methods disclosed herein may include adding additional input channels. In some embodiments, the additional input channels may include raw RNA dot density input to compensate for signal loss from morphological markers. In some embodiments, a spatial molecular imager may be used to stain and image cell nuclei and DAPI. In some cases, the spatial molecular imager may be a CosMx™ instrument. In some embodiments, the systems, media, and methods disclosed herein may include biological samples labeled with photolyzable morphological markers. In some embodiments, DAPI channel acquisition in one field of view may result in the cleavage or bleaching of reporter genes in adjacent fields of view, or both. In some embodiments, the order of image or channel acquisition, or both, is determined. In some cases, the order of image or channel acquisition is determined to prevent visible and irrecoverable signal loss at the edges of the field of view, particularly in sequential scans. In some embodiments, the systems, media, and methods disclosed in this invention may include integrated multi-channel input morphological images. In some embodiments, the morphological images may be used to represent different markers of the cell nucleus, cytoplasm, or cell membrane, or any combination thereof. In a non-limiting example, the 5-channel morphological image used for segmentation may include blue (B), green (G), yellow (Y), red (R), and UV (U). In some embodiments, the systems, media, and methods disclosed in this invention further include a multi-step image acquisition process. In certain circumstances, a multi-step image acquisition process may be performed to avoid signal loss. In some embodiments, the systems, media, and methods disclosed in this invention may include software modules configured to convert 3D scanned images into 2D images. In certain circumstances, converting 3D scanned images into 2D images may obtain the optimal focus area in a z-slice. In some embodiments, the systems, media, and methods disclosed in this invention may include software modules configured to perform deconvolution processing on 2D images. In certain circumstances, deconvolution processing of 2D images may obtain the optimal focus area of ​​the image while preserving the z-position of the image within its stack of 3D scanned images. In some embodiments, deconvolution processing of 2D images may obtain optimal image sharpness while preserving the z-position of the image within its stack of 3D scanned images. Figure 6 As shown in the non-limiting example, image acquisition is performed in two steps, P01 and P02, to avoid signal loss when using ultraviolet light for DAPI acquisition. In step P01, the system obtains the four channels of B, G, Y, and R in 3D high dynamic range (HDR) z-stacked for the entire tissue for the selected FOV. For example, in Figure 6 In step P02, the system separates the DAPI from the other two morphological channels to align it with the preceding step P01 in 3D coordinate space. For example, once step P01 is completed, the FOV position is revisited to obtain a new Z-stack with the DAPI and two additional morphological channels (e.g., B and Y). In a non-limiting example, the B and Y morphological channels are intended to align the P01 and P02 z-stacks to the same 3D coordinate space, such as... Figure 6 The registration steps are shown. In a non-limiting example, once the stacks are aligned, they are combined to generate a complete 5-channel Z-stack, as shown. Figure 6 Step P99 is shown. In some embodiments, once a morphological scan is performed, protein images of all FOVs can be acquired, followed by RNA acquisition. In some embodiments, in each cycle, reference markers placed on the tissue can be used to register the images to the same 3D coordinate space. In some embodiments, the systems, media, and methods disclosed in this invention may include using a multi-step image acquisition process to acquire time-lapse images of the sample of interest.

[0057] Image enhancement and bleaching correction In some implementations, photodegradable markers are used, and the markers in the tissue are repeatedly restained and removed using ultraviolet irradiation and chemical washing to obtain an unlimited set of source input channels. In some cases, repeated scanning of tissue areas may result in some photobleaching and / or darkening at the FOV edges. In a non-limiting example, Figure 7 The image shows dark bands on the right and bottom sides, indicating reduced cell signal intensity and detectability in these areas. In some cases, the closer the FOVs are and the more repeated the acquisition cycles, the greater the effect of photobleaching. Figure 7 A non-restrictive example is shown. In some embodiments, the systems, media, and methods disclosed in this invention may further include normalizing the intensity of darkened regions using the most recently unbleached portion of an image to correct edge artifacts. In some embodiments, the location of bleaching can be measured by downsampling the image and dividing it into smaller blocks, such as... Figure 8A As shown. In some implementations, a transition smoothing function can be applied during the scaling calculation, such as... Figure 8B As shown. In a non-restrictive example, Figure 8C The normalization and scaling factor calculation scheme is illustrated. In some implementations, the lower 25th percentile and the higher 75th percentile values ​​for each block are calculated, such as... Figure 8C As shown. In some implementations, the ratio of the lower 25th percentile to the higher 75th percentile can be calculated, such as... Figure 8C As shown. In some implementations, a lower intensity quantile ratio can be used, by applying, for example... Figure 8B The scaling factor shown is used to normalize the intensity drop at that location. In some embodiments, higher quantiles can be used to determine the high intensity value for each block. In some embodiments, a sudden drop in the high intensity quantile between adjacent blocks can be used to determine the location where bleaching begins. In some embodiments, the systems, media, and methods disclosed in this invention may also include bleaching correction for morphological and protein image acquisition.

[0058] In some implementations, images can be corrected using conventional image enhancement techniques, including denoising, deblurring, or image sharpening, or any combination thereof. In some implementations, one or more denoising algorithms can be applied to one or more images. These denoising algorithms include filter-based methods (Wiener filtering, Gaussian filtering, median filtering), wavelet transform, total variational (TV) denoising, and deep learning-based methods. Deblurring algorithms include nearest-neighbor deconvolution, Richardson-Lucy deconvolution, and total variational (TV) regularization, or any combination thereof. In some implementations, image enhancement algorithms may include histogram equalization, contrast-limited adaptive histogram equalization (CLAHE), contrast stretching, gamma correction, desharpening masks, or any combination thereof. In some implementations, nearest-neighbor deconvolution can be used. In some cases, nearest-neighbor deconvolution may include eliminating blurred signals using adjacent z-planes. In some embodiments, nearest-neighbor deconvolution does not require point spread function (PSF) data. In some implementations, images from the previous and next z-positions can be used. and To perform nearest neighbor deconvolution to correct the image at position z, I, as described below:

[0059] Where A is the multiplier factor between 0 and 1, and G is the Gaussian kernel. For the optical numerical aperture (NA) and ( The sampling rate is determined.

[0060] In some implementations, the relationship between NA and the width of the blur function can be described using the objective cone angle, i.e., NA = n sin(a), as shown below. Figure 22 As shown, where n is the refractive index. Therefore, in some embodiments, a = arcsin(NA / n). In some non-limiting examples, using a water-based objective lens, n = 1.33333 and NA = 1.10, it can be concluded that... = 55.59 degrees. For example, using Figure 22 The Gaussian kernel width can be derived as follows: .

[0061] RNA dot detection and density map generation In some embodiments, RNA molecules can be detected using single-molecule fluorescent barcodes, which appear as visible dots in an image. In some embodiments, combinations of dots may correspond to certain RNA transcripts encoded in the chemical reagents used. In some embodiments, the systems, media, and methods disclosed herein may also include applying the location and density of the original dots to infer additional cell morphology. In some embodiments, the systems, media, and methods disclosed herein may also include applying the location and density of the original dots to supplement additional cell morphology. In some embodiments, each RNA dot can be detected from an image slice obtained from a tissue section. In some embodiments, a Laplace-Gaussian (LoG) bandpass filter can be used for detection. In some embodiments, the LoG bandpass filter can be designed to match the morphology of the reporter signal in each channel. In some embodiments, a two-dimensional parabolic fit can be used to estimate the precise location of the dots. In some embodiments, chromatic aberration of the optics can be taken into account to ensure spatial alignment between each channel. In some embodiments, the lens does not focus all wavelengths (channels) to the same point. In a non-limiting example, Figure 9 The diagram illustrates the aberrations produced axially (middle image) and laterally (right image) and compares them to an ideal lens on the left. In some embodiments, the systems, media, and methods disclosed in this invention may also include applying a unique calibration to determine the z-off for each wavelength during acquisition to correct for axial chromatic aberration. In some embodiments, the systems, media, and methods disclosed in this invention may further include performing lateral chromatic aberration correction on each multi-channel image. In some embodiments, a pre-calibration transform may be used to perform lateral chromatic aberration correction. In some non-limiting examples, such as... Figure 10 As shown, each FOV detects over 10 million high-confidence (high-signal) point locations, achieving a resolution of one-tenth of a pixel (-12nm). For example, as... Figure 10 As shown, when high-density points are plotted in image space, they correspond to cellular structures and can be used to infer and enhance cell morphology. In some embodiments, the point locations can be compartmentalized into a lower-resolution pixel space. For example, as... Figure 10 As shown, this compartmentalization generates detailed morphological images that can be successfully segmented using a cell segmentation pipeline. In some embodiments, the systems, media, and methods disclosed in this invention may further include using a two-dimensional histogram of the compartmentalized point locations as an additional input segmentation channel. Figure 11 As shown, as a non-limiting example, the original points are detected at a resolution of 0.1 pixels to form a detailed morphological density image. In some non-limiting examples, the points of each FOV are resolved to one-tenth of a pixel, as shown in Table 1. As a non-limiting example, X and Y are in decipixels, ranging from [0, 42560].

[0062] Table 1. Examples of spatial coordinates of detected points

[0063] For example, such as Figure 12 As shown, a 1330 x 1330 grid is overlaid on the range [0, 42560]. For example, the total number of points falling into each box of the grid is counted to determine the grayscale values ​​in the generated point density map image. For example, the density image is resized to a 4256 x 4256 FOV to reduce noise in the density map image. For example, this image is subsequently used as a standard morphological channel obtained by fluorescence staining in a cell segmentation workflow.

[0064] Multimodal cell segmentation In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving five morphological channels. In some embodiments, retrieving five morphological channels may provide more cell boundary description than using nuclear channels alone. In some embodiments, the systems, media, and methods disclosed in this invention may include retrieving 64 or more distinct protein channels. In some embodiments, about 1, 2, 3, 4, 5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, or more than 100 distinct protein channels may be retrieved. For example, Figure 13A Multiple protein pathways are shown, including histone staining, DAPI staining, and somatic rRNA staining pathways. For example, Figure 13B Multiple segmentation processing channels are shown, including reading out density heatmaps, segmentation based on all three morphological staining images, and segmentation based on density heatmaps and histone staining. In some implementations, the number of different protein channels retrieved can be determined by the number of protein markers, scan cycle parameters, other transcript-based information using dot density maps, or any combination thereof, such as Figure 14As shown. In some embodiments, the systems, media, and methods disclosed in this invention may include integrating morphological and protein images of the same morphological type into a smaller set of input channels. In some embodiments, each protein may be annotated to describe its morphology. In some embodiments, the annotation may include cytoplasmic or nuclear expression, or both. In some embodiments, the annotation may describe different cell type expressions. In some cases, cell type expression may include glial cells, astrocytes, and other neuronal cell types, or any combination thereof. In some embodiments, these annotations may be used to integrate each protein image based on cells of the same morphology or the same cell type, or both. In some embodiments, projection methods may be used to merge multiple protein images of the same morphology into a 5-channel image. In some embodiments, the projection method may be one or more of maximum intensity projection, principal component analysis, singular value decomposition, multiple reference alignment, density map averaging, or any combination thereof. In some embodiments, readout or point density images, or both, may be added as additional channels. In some embodiments, background subtraction and normalization may be performed on all input channels prior to the segmentation step. In some embodiments, cell segmentation may be one or more of nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation, or any combination thereof. In some implementations, the output of each cell segmentation can be combined as the final segmentation output. In some implementations, statistics can be performed for each cell and each region. In some implementations, the raw image may be preprocessed before processing by the cell segmentation module to minimize imaging artifacts and improve image quality. In some implementations, image preprocessing techniques may be used to process transcription point density heatmaps. In some implementations, preprocessing may be used to improve the signal-to-noise ratio between intracellular and extracellular regions. Figure 24 A non-limiting example of using denoising preprocessing on a point density heatmap is shown. In some implementations, the point density heatmap can be used without further preprocessing. In some implementations, the image preprocessing step may employ a combination of conventional image processing techniques, such as one or more of deconvolution, Gaussian blur, normalization, feature extraction, Fourier transform, linear filtering, contrast enhancement, binarization, or any combination thereof. In some implementations, the image preprocessing step may utilize machine learning methods, such as the Noise2Self and Cellpose 3.0 image restoration models.

[0065] In some implementations, nuclear channels may be segmented to achieve subcellular segmentation. In some implementations, nuclear channels may be segmented in addition to other channels containing cell membrane and cytoplasmic contents. In some implementations, multi-step segmentation can produce subcellular results that define the separation of nuclear and cytoplasmic regions. In some implementations, the intersection-over-union (IoU) score may be used to measure overlap between cells. For example, in some embodiments, when the IoU score is low, overlapping regions may be assigned to segments with higher confidence signals. In some implementations, nuclear channels may have stronger signals. In some cases, low-intersection pixels may be designated as nuclear segments. In some implementations, nuclear segments are not cytoplasm. In some embodiments, when the intersection is high, two segmentation results may be merged to combine the nucleus and cytoplasm. In some implementations, the segmentation system may generate masks defining regions. In some cases, these regions may be defined as either the nucleus or the cytoplasm. In some implementations, regions may be defined as regions of each cell detected in the image. In some implementations, these masks may be used to measure and statistically analyze each cell. In some implementations, specific cell membrane and extracellular regions may provide biological information.

[0066] In some implementations, for example, such as Figure 23 As shown, a combination method based on Intersection over Union (IoU) can be used to combine multiple segmentation results. In some implementations, the IoU-based combination method can be executed stepwise. In some implementations, the stepwise execution of the IoU-based combination method can combine multiple morphological modalities to obtain the final result. In some implementations, the IoU-based combination method can integrate 2D cell segmentation results from a single 2D image into a 3D cell segmentation volume result of a corresponding stack of 3D scan images. In some implementations, correlation may include labeling each cell at all z-positions using the maximum IoU score for consistent mapping. In some implementations, other measurements, such as cell focus distribution across z-slices, can be used to determine the continuity of cell labeling across z-slices. In some implementations, cell centroid and shape changes across z-slices can be analyzed to label cells in the 3D stack. In some implementations, a machine learning model that accepts 3D stack input can be used to directly segment the 3D stack. In some implementations, such as... Figure 23 As shown, in this arrangement, extending 2D to 3D may include performing 2D multimodal segmentation on each z-slice, followed by 3D cell marker stitching to associate cells in each z-slice to form 3D cell markers. Figure 23 A non-limiting example of a 3D segmentation process is shown.

[0067] In some implementations, the maximum IoU score can be used to map the location of intracellular objects, such as the cell nucleus, to the cell body outline. In some implementations, the maximum IoU score can be used to map cell locations in different source images, such as images taken at different z-coordinates or time points, to each other. In some implementations, the locations of each different source image taken at different z-coordinates or time points are mapped together to form a stacked image, and the maximum IoU score is obtained when a match is found. In some implementations, point images can be registered to the same reference datum as morphologically stained images. In some implementations, time-lapse images of the sample of interest can be captured. In some implementations, IoU-based ensemble methods can be applied to integrate cell segmentation results of the same sample at different time points to track the behavior of the same cells over time. In some implementations, extracellular regions may be detected. In some implementations, extracellular regions may be included as part of the segmentation result. In some cases, extracellular regions may include one or more structures, such as those in the extracellular space of the brain. In some cases, extracellular structures may contain one or more vesicles. In some implementations, each connectivity component may be detected and masked. In some implementations, automatic thresholding techniques may be used to perform masking. In some implementations, automatic thresholding techniques may include finding high-intensity regions in the input morphological channel or using a neural network model trained to detect cell membranes or specific extracellular regions.

[0068] In a non-restrictive example, such as Figure 15 As shown, Ibal markers were used to bind all proteins expressing microglia types to microglia morphology channels. Figure 15 As shown, the input image channels include morphological channels and dot density map channels for the same biological sample. In some implementations, the maximum intensity pixel value can be projected onto all input morphological channels. In some implementations, the readout or dot density image, or both, can be added as a separate channel. In some implementations, nucleus, cytoplasm, or extracellular segmentation can be performed on each input channel. In some implementations, the cell segmentation algorithm can be selected based on one or more input channels. In some implementations, cell segmentation can be based on one or more algorithms. In some cases, one or more algorithms may include Cellpose deep learning algorithms, adaptive thresholding, watershed algorithms, thresholding, region-based segmentation, active contours, convolutional neural networks (CNNs), level set methods, graph-based algorithms, fuzzy c-means clustering, deep watersheds, image morphology, or Markov random fields, or any combination thereof. In some implementations, nucleus segmentation, cytoplasm segmentation, and extracellular segmentation can be based on the same algorithm. In some implementations, nucleus segmentation, cytoplasm segmentation, and extracellular segmentation can be based on different algorithms. For example, as... Figure 15As shown, the selected algorithm is used for the neural network model, and the model is generated using a ground truth dataset. In some implementations, a model suitable for a specific input morphology can be selected by comparing the image and the model style vector. In some implementations, after neural network inference, the output can be generated by measuring image attributes within each cell or cell region mask area. In some implementations, the outputs of all models can be combined into a single final output. In some implementations, statistics are performed for each cell or each region, or both.

[0069] In some embodiments, the systems, media, and methods disclosed in this invention may include multiple input channels in a model. Figure 16 A non-limiting example of generating a new N-channel model to accommodate a new number of morphological channels is shown. In some embodiments, the systems, media, and methods disclosed in this invention may include the application of transfer learning techniques. In some embodiments, transfer learning techniques can be used to reuse pre-existing trained deep learning model weights in higher dimensions. In some embodiments, such as Figure 16 The weight transfer block is shown in the diagram, where a predefined 2-channel model is used as the starting model, and new channels can be added by copying the weights of the original 2 channels. In some implementations, a new model can be trained based on the newly defined weights and a ground truth dataset. In some implementations, a new N-channel model can then be generated. Figures 17A-17B Various new 2-channel and 5-channel outputs are shown, among which Figure 17A A non-limiting example of the new 2-channel model is shown. Figure 17B A non-limiting example of the new 5-channel model is shown.

[0070] In some embodiments, nuclear segmentation and cytoplasmic segmentation can be combined separately. In some embodiments, the combination of nuclear and cytoplasmic segmentation ensures segmentation output of tissues with complete nuclear and membrane signal sets. In some embodiments, the combination of nuclear and cytoplasmic segmentation ensures segmentation output of regions with weaker membrane signals. In some embodiments, segmenting the nucleus and cytoplasm provides subcellular resolution and accuracy. Figures 18A-18D As shown in the non-limiting example, a cell nucleus expression image is selected as the input for cell nucleus segmentation, and all morphological input channels, including the cell nucleus, are used as the input for cytoplasm segmentation. Figures 18A-18D As shown, the results of cell nucleus and cytoplasm segmentation are combined by analyzing the overlap and intersection (IoU) between segmentation results. Various markers are applied to the cell segmentation results, for example, Figure 18A The image shows neuronal cells labeled with histone H3 in the cell nucleus. Figure 18B Neuronal cells with rRNA membrane markers are shown. Figure 18CAstrocytes with GFAP markers were shown. Figure 18D The cell nucleus with DAPI markers applied is shown.

[0071] In some embodiments, the systems, media, and methods disclosed in this invention may further include segmenting extracellular processes as a third segmentation step. In some embodiments, the extracellular region may be a structure present in an image but not containing nucleic acid information. In some embodiments, the systems, media, and methods disclosed in this invention may further include using a filter to track each of one or more neuronal processes. In some embodiments, the filter may include a Laplace-Gaussian (LoG) filter. In some embodiments, the filter may be followed by a connected component algorithm to separate the detected process into individual components, such as... Figure 19 As shown. In some implementations, the LoG filter can extract fine neuronal processes from normalized brain-specific markers. In some cases, brain-specific markers may include astrocytes or microglia, or input channels of both. In some implementations, a threshold may be applied to the LoG output to create a mask. In some implementations, the mask is then enlarged to match the size of the neurons. In some implementations, standard thresholding techniques may be used to detect each process. In some cases, standard thresholding techniques may include Otsu's method, Moments, Li's method, Huang's method, and Bernsen's local thresholding method, or any combination thereof. In some implementations, a connected component algorithm may be used to number each connected neuron. In some implementations, a connected component algorithm may be used to number cells segmented from an image. In some implementations, a machine learning model trained to detect cellular structures may be used to detect each process.

[0072] In some implementations, the process of completely encapsulating the cell body may be merged with the nuclear fragment and labeled as a cell. In some implementations, processes unrelated to the cell body may be labeled as objects, such as... Figure 20 As shown in A-20C. For example. Figure 20 A shows cells and such Figure 20 Figure B shows an overlay of extracellular processes that have been masked to isolate extracellular fragments, such as... Figure 20 As shown in Figure C. In some implementations, machine learning algorithms trained to perform merging operations can be used to perform the merging between cell processes and the cell nucleus.

[0073] In some implementations, because each neuronal process can be individually identified as one or more objects, transcriptomics, cell typing, and other spatial data analyses can be performed in the same manner as single-cell analysis. In some implementations, all three segmentation steps can generate single-cell tags using cellular measurements and statistics, as well as measurements of individual objects. In some implementations, cell tags can be further segmented into nucleal and cytoplasmic regions. In a non-limiting example of brain tissue, each individual cell type is labeled and identified in different channels, such as... Figure 21 As shown.

[0074] Single-cell statistics In some implementations, the segmentation output can provide statistical data for each detected cell. The statistical data may include statistics describing fluorescence properties, spatial location, or characteristics, or any combination thereof. In some implementations, these properties may include average, median, and maximum fluorescence intensity. In some implementations, these properties may include various shape descriptors. In some cases, shape descriptors may include roundness, compactness, aspect ratio, convexity, robustness, eccentricity, and perimeter.

[0075] These shape descriptors are defined as follows: Aspect Ratio =

[0076] Among them, W bb H is the width of the bounding box. bb This represents the height of the bounding box.

[0077] In some implementations, to limit the aspect ratio to 0–1, the width can be specified as a shorter value in the bounding box dimensions. In some implementations, to limit the aspect ratio to 0–1, the height can be specified as a longer value in the bounding box dimensions.

[0078] Roundness =

[0079] Where A is the area of ​​the cell, P conv The perimeter is convex to avoid concave irregularities.

[0080] Tightness =

[0081] Where A is the cell area and P is the normal cell perimeter.

[0082] Convexity =

[0083] Eccentricity =

[0084] Among them, L minor It is the length of the minor axis, Lmajor It is the length of the major axis.

[0085] Durability =

[0086] Where A is the cell area, A conv Let be the area of ​​the convex surface.

[0087] In some implementations, indicators can be calculated for each region, including the nucleus and cytoplasm. In some implementations, these attributes can be used to analyze cell subpopulations. In some implementations, other attributes, such as texture, can also be combined. In some implementations, when imaging a small portion of a tissue sample, some cells may split into different fields of view (FOVs). In some implementations, when used in the context of single-cell transcriptomics analysis, these partial cell fragments at FOV intersections may produce incomplete masks and lead to incorrect analysis. In some implementations, segmentation output measurements can identify cells splitting at image boundaries so that they can be filtered out in subsequent analyses.

[0088] Machine Learning In some embodiments, the systems, media, and methods disclosed in this invention may include subcellular segmentation, which further includes one or more deep learning models. In some embodiments, images of cells or tissues may be used to train one or more deep learning models. In some embodiments, images of cells or tissues may include microscopic images, morphological images, or fluorescence images, or any combination thereof. In some embodiments, deep learning models may be trained to segment specific tissue types. In some embodiments, deep learning models may be trained to restore image quality, thereby obtaining better cell segmentation results.

[0089] In some embodiments, the systems, media, and methods disclosed in this invention may include training or applying machine learning models, or both. In some embodiments, the machine learning model may perform dimensionality reduction. In some embodiments, dimensionality reduction may be performed using nonlinear dimensionality reduction algorithms. In some embodiments, nonlinear dimensionality reduction algorithms may include Sammon mapping, principal curves and manifolds, Laplacian eigenmaps, isometric mappings, locally linear embeddings, locally tangent space alignment, maximum variance expansion, Gaussian process latent variable models, t-distributed random neighborhood embeddings, relational perspective mappings, contagion mappings, curve component analysis, curve distance analysis, differential homeomorphism dimensionality reduction, manifold alignment, diffusion mappings, locally multidimensional scaling, nonlinear PCA, data-driven high-dimensional scaling, manifold sculpting, RankVisu, topologically constrained isometric embeddings, uniform manifold approximation or projection (UMAP), or any combination thereof. In some embodiments, the UMAP algorithm may apply a feedforward neural network to a subset of data. In some embodiments, the UMAP algorithm may project manifold clustering onto the entire dataset. In some embodiments, the UMAP algorithm may be a feedforward algorithm. In some implementations, a feedforward neural network can be trained to approximate an identity function. In some implementations, the approximate identity function may include mapping from a value vector to the same vector. In some implementations, the feedforward neural network can be used for dimensionality reduction. In some implementations, a hidden layer in the network may be restricted to containing only a small number of network units.

[0090] In some embodiments, the systems, media, and methods disclosed in this invention may include training a machine learning model or applying a machine learning model, or both. In some embodiments, the machine learning model may include an unsupervised machine learning model, a supervised machine learning model, a semi-supervised machine learning model, or a self-supervised machine learning model, or any combination thereof. In some embodiments, the supervised machine learning model may include, for example, random forests, support vector machines (SVMs), neural networks, or deep learning algorithms, or any combination thereof.

[0091] In some embodiments, the platforms, systems, media, and methods disclosed in this invention may include machine learning models utilizing one or more neural networks. In some embodiments, the neural network can learn the relationship between an input dataset and a target dataset. In some embodiments, the neural network may be a software representation of the human nervous system. In some embodiments, the human nervous system may be a cognitive system. In some embodiments, the neural network can acquire the “learning” and “generalization” capabilities used by humans. In some embodiments, the machine learning algorithm may include a neural network comprising a CNN. In some embodiments, the structural components of the machine learning algorithm may include one or more of CNNs, recurrent neural networks, dilated CNNs, fully connected neural networks, deep generative models, converters, or Boltzmann machines, or any combination thereof.

[0092] In some implementations, a neural network comprises a series of layers called “neurons.” In some implementations, a neural network includes an input layer to which data is provided; one or more inner and / or “hidden” layers; and an output layer. In some implementations, neurons may be connected to neurons in other layers via weighted connections, where the weights are parameters controlling the strength of the connections. In some implementations, the number of neurons per layer may be related to the complexity of the problem to be solved. In some implementations, the minimum number of neurons required per layer may be determined by the complexity of the problem, while the maximum number may be limited by the generalization ability of the neural network. In some implementations, input neurons may receive the data being provided and then transmit that data to a first hidden layer via connection weights, where the connection weights are modified during training. The first hidden layer may process the data and transmit its results to the next layer via a second set of weighted connections. In some implementations, each subsequent layer may “pool” the results of the preceding layers into a more complex relationship. In some implementations, unlike traditional software programs that require specific instructions to perform functions, neural networks are programmed to provide desired outputs, such as output values, by being trained using a known set of samples and allowing them to modify themselves during (and after) training. In some implementations, after training, when new input data is provided to the neural network, it is configured to summarize what was “learned” during training and apply what was learned from training to new, previously unseen input data to generate an output associated with that input.

[0093] In some embodiments, the neural network includes an artificial neural network (ANN). In some embodiments, the ANN can be a machine learning algorithm that can be trained to map an input dataset to an output dataset, wherein the ANN includes interconnected groups of nodes organized into multiple layers of nodes. For example, an ANN architecture may include at least an input layer, one or more hidden layers, and an output layer. In some embodiments, the ANN may include any total number of layers and any number of hidden layers, wherein the hidden layers serve as trainable feature extractors to allow mapping a set of input data to an output value or a set of output values. As used in this invention, deep learning algorithms (e.g., deep neural networks (DNNs)) are ANNs containing multiple hidden layers, such as two or more hidden layers. Each layer of the neural network may contain multiple nodes (or "neurons"). In some embodiments, nodes receive input directly from the input data or the output of nodes in previous layers and perform specific operations, such as summation. In some embodiments, the connections from the input to the node are associated with weights or weighting factors. In some implementations, the node may sum the product of all inputs and their associated weights. In some implementations, a bias may be used to offset the weighted sum. In some implementations, the output of the node or neuron may be gated using a threshold or activation function. In some implementations, the activation function may be a linear or nonlinear function. In some implementations, the activation function may be, for example, a rectified linear unit (ReLU) activation function, a LeakyReLU activation function, or other functions such as a saturated hyperbolic tangent, an identity function, a binary step function, a logic function, an arctangent function, a soft sign function, a parametric rectified linear unit function, an exponential linear unit function, a softplus function, a bending identity function, a soft exponential function, a sine function, a sine function, a Gaussian function, or a sigmoid function, or any combination thereof.

[0094] In some implementations, the weighting factors, bias values, thresholds, or other computed parameters of the neural network can be "taught" or "learned" using one or more sets of training data during the training phase. For example, input data from the training dataset and gradient descent or backpropagation methods can be used to train the parameters so that the output values ​​computed by the ANN are consistent with the examples contained in the training dataset.

[0095] The number of nodes used in the input layer of an ANN or DNN can be at least about 10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000, 100,000, or greater than 100,000, including incremental values.

[0096] In other cases, the number of nodes used in the input layer can be approximately 100,000, 90,000, 80,000, 70,000, 60,000, 50,000, 40,000, 30,000, 20,000, 10,000, 9,000, 8,000, 7,000, 6,000, 5,000, 4,000, 3,000, 2,000, 1,000, 900, 800, 700, 600, 500, 400, 300, 200, 100, 50, 10, or less than 10, including increments. In some cases, the total number of layers used in an ANN or DNN (including the input and output layers) can be at least approximately 3, 4, 5, 10, 15, 20, or more, including increments. In other cases, the total number of layers can be approximately 20, 15, 10, 5, 4, 3 or fewer, including incremental values.

[0097] In some cases, the total number of learnable or trainable parameters used in an ANN or DNN, such as weighting factors, biases, or thresholds, can be at least about 10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000, 100,000, or greater than 100,000, including incremental values. In other cases, the number of learnable parameters can be up to approximately 100,000, 90,000, 80,000, 70,000, 60,000, 50,000, 40,000, 30,000, 20,000, 10,000, 9000, 8000, 7000, 6000, 5000, 4000, 3000, 2000, 1000, 900, 800, 700, 600, 500, 400, 300, 200, 100, 50, 10 or less, including incremental values.

[0098] In some embodiments, the machine learning models included in the platforms, systems, media, and methods disclosed in this invention may include neural networks, such as deep CNNs. In some embodiments using CNNs, the network consists of any number of convolutional layers, dilated layers, or fully connected layers. In some embodiments, the number of convolutional layers is between 1 and 10. In some embodiments, the number of dilated layers is between 0 and 10. In some embodiments, the total number of convolutional layers (including input and output layers) may be at least about 1, 2, 3, 4, 5, 10, 15, 20, or more, while the total number of dilated layers may be at least about 1, 2, 3, 4, 5, 10, 15, 20, or more. In some embodiments, the total number of convolutional layers may be at most about 20, 15, 10, 5, 4, 3, or less, and the total number of dilated layers may be at most about 20, 15, 10, 5, 4, 3, or less. In some embodiments, the number of convolutional layers is between 1 and 10, and the number of fully connected layers is between 0 and 10. In some implementations, the total number of convolutional layers (including input and output layers) can be at least about 1, 2, 3, 4, 5, 10, 15, 20 or more, and the total number of fully connected layers can be at least about 1, 2, 3, 4, 5, 10, 15, 20 or more. In some implementations, the total number of convolutional layers can be at most about 20, 15, 10, 5, 4, 3, 2, 1 or less, and the total number of fully connected layers can be at most about 20, 15, 10, 5, 4, 3, 2, 1 or less.

[0099] In some implementations, the machine learning algorithm includes a neural network, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), dilated CNNs, fully connected neural networks, deep generative models and / or depth-restricted Boltzmann machines, or any combination thereof.

[0100] In some implementations, the machine learning model includes one or more CNNs. In some implementations, the CNN may be a deep feedforward ANN. In some implementations, the CNN may be suitable for analyzing visual images. In some implementations, the CNN may include an input layer, an output layer, and multiple hidden layers. In some implementations, the hidden layers of the CNN may include convolutional layers, pooling layers, fully connected layers, and normalization layers. In some implementations, these layers may be organized along three dimensions: width, height, and depth.

[0101] In some implementations, convolutional layers can perform convolution operations on the input and pass the results of the convolution operation to the next layer. In some implementations, for image processing, convolution operations can reduce the number of free parameters, allowing the network to become deeper with fewer parameters. In some implementations of neural networks, each neuron can receive input from multiple locations in the previous layer. In some implementations of convolutional layers, neurons can receive input only from a restricted subregion of the previous layer. In some implementations, the parameters of a convolutional layer can include a set of learnable filters (or kernels). In some cases, learnable filters can have small receptive fields and extend to the entire depth of the input volume. In some cases, during the forward pass, each filter can be convolved across the width and height of the input data volume, the dot product between the filter entry and the input data is calculated, and a two-dimensional activation map of the filter is generated. Therefore, in some implementations, the network can learn which filters will be activated when certain specific types of features are detected at a certain spatial location in the input data.

[0102] In some implementations, the machine learning model includes an RNN. An RNN is a neural network with recurrent connections that can encode and process sequential data. In some implementations, an RNN may include an input layer configured to receive an input sequence. In some implementations, an RNN may also include one or more hidden recurrent layers that maintain the state. In some implementations, at each step, each hidden recurrent layer can compute its output and the next state. In some implementations, the next state may depend on the previous state and the current input. In some implementations, the state can be maintained across steps and dependencies in the input sequence can be captured.

[0103] In some implementations, the RNN may be a Long Short-Term Memory (LSTM) network. In some implementations, the LSTM network may consist of LSTM units. In some implementations, the LSTM unit may include a cell, an input gate, an output gate, and a forget gate. In some implementations, the cell may be responsible for tracking dependencies between elements in the input sequence. In some implementations, the input gate may control the extent to which new values ​​flow into the cell, the forget gate may control the extent to which values ​​are retained in the cell, and the output gate may control the extent to which the values ​​in the cell are used to compute the activation of the LSTM unit output. In some implementations, the neural network may include an attention mechanism (e.g., a transducer). In some implementations, the attention mechanism may focus on or "pay attention to" certain input regions while ignoring others. In some implementations, this can improve model performance because some input regions may be less relevant. In some implementations, at each step, the attention unit may compute the dot product of the context vector with the input of that step, as well as other operations. In some other implementations, the output of the attention unit may define the location of the most relevant information in the input sequence. In some implementations, the attention mechanism may include a visual transducer. In some implementations, the visual transducer may include a natural language model. In some cases, the visual transducer may include a masked autoencoder, a Swin transducer, a vector quantization variational autoencoder, or other types of visual transducers. In some implementations, the visual transducer may be combined with a generative adversarial network.

[0104] In some implementations, pooling layers include global pooling layers. In some implementations, global pooling layers can combine the outputs of a group of neurons in one layer into a single neuron in the next layer. For example, a max pooling layer can use the maximum value from each of the neuron groups in a previous layer; an average pooling layer can use the average value from each neuron group in a previous layer.

[0105] In some implementations, a fully connected layer can connect each neuron in one layer to every neuron in another layer. In some neural networks, each neuron can receive input from multiple locations in the previous layer. In some fully connected layers, each neuron can receive input from every element in the previous layer.

[0106] In some implementations, the normalization layer is a batch normalization layer. In some implementations, batch normalization layers can improve the performance and stability of neural networks. In some implementations, batch normalization layers can provide zero-mean / unit-variance inputs to any layer in the neural network. In some cases, the advantages of using batch normalization layers may include faster network training, higher learning efficiency, easier weight initialization, a wider range of viable activation functions, and simpler procedures for building deep networks.

[0107] In some implementations, the trained algorithm may be configured to accept multiple input variables and generate one or more output values ​​based on the multiple input variables. In some implementations, the trained algorithm may include a classifier such that each of the one or more output values ​​may include one of a fixed number of possible values. In some cases, the possible values ​​may include a linear classifier or a logistic regression classifier. In some cases, the algorithm may instruct the classifier to classify a biological sample or a subject, or both. In some implementations, the trained algorithm may include a binary classifier. In some implementations, each of the one or more output values ​​includes one of two values. In some implementations, these values ​​may include {0, 1}, {positive, negative}, or {high risk, low risk} to represent the classification of a biological sample or a subject, or both, by the classifier. In some implementations, the trained algorithm may be another type of classifier such that each of the one or more output values ​​includes one of more than two values. In some implementations, these values ​​may include {0, 1, 2}, {positive, negative, or uncertain}, or {high risk, moderate risk, or low risk} to represent the classification of a biological sample and / or a subject by the classifier. In some implementations, output values ​​may include descriptive labels, numerical values, or combinations thereof. In some implementations, some output values ​​may include descriptive labels. In some implementations, some output values ​​may include numerical values, such as binary values, integers, or continuous values. Such binary output values ​​may include, for example, {0, 1}, {positive, negative}, or {high risk, low risk}. Such integer output values ​​may include, for example, {0, 1, 2}. Such continuous output values ​​may include, for example, probability values ​​that are at least 0 and no greater than 1. Such continuous output values ​​may include, for example, non-normalized probability values ​​that are at least 0. Such continuous output values ​​may indicate the prognosis of a subject's cancer-related category. Some numerical values ​​may be mapped to descriptive labels, for example, mapping 1 to "positive" and 0 to "negative".

[0108] In some implementations, if a sample is not classified as “positive,” “negative,” “1,” or “0,” the sample classification may assign an output value of “uncertain” or 2. In some cases, a set of two cutoff values ​​can be used to classify a sample into one of three possible output values. Examples of cutoff value sets may include {1%, 99%}, {2%, 98%}, {5%, 95%}, {10%, 90%}, {15%, 85%}, {20%, 80%}, {25%, 75%}, {30%, 70%}, {35%, 65%}, {40%, 60%}, and {45%, 55%}. In some implementations, a set of cutoff values ​​can be used to classify a sample into one of n+1 possible output values, where n is any positive integer.

[0109] Visualization In some embodiments, the systems, media, and methods disclosed in this invention may further include generating data summaries, results, or visualizations, or any combination thereof. In some embodiments, visualization may include graphs, charts, and overlaying transcript or cell annotation points onto a tissue image. In some embodiments, visualization may query a Seurat / tileDB object to retrieve relevant data for display. In other embodiments, relevant data may include transcript locations and / or cell annotations. In some embodiments, visualization may display transcript locations as points, cell annotations (e.g., cell types) as points, box plots, violin plots, dot plots, or any combination thereof. In some embodiments, visualization of cell segmentation statistics may be displayed as raw values ​​or points of projected coordinates in an embedding space to evaluate the segmentation result of interest relative to any other segmentation result of the same sample. In some embodiments, visualization may overlay segmentation boundaries, transcripts, and subsequent data analysis outputs. In some cases, subsequent data analysis outputs include cell types. In some cases, subsequent data analysis outputs may include one or more, or any combination thereof, of cell state, phenotype of the tissue microenvironment, differential expression of cell types based on spatial context, quantitative expression of subcellular cells, and spatially resolved biomarker identification. In some implementations, visualizations may come from different segmentation configurations or methods, or from the segmentation results of a reference sample.

[0110] In some implementations, a visual converter can be used to perform visualization. In some embodiments, the visual converter can process an input image into a series of patches. In some cases, the visual converter can serialize each patch into a vector. In some implementations, the visual converter can match each vector to a smaller dimension via matrix multiplication. In some implementations, a converter encoder can process the visual converter data. In some implementations, the converter can measure the relationship between one or more input tags via the converter encoder. In some implementations, the visualization can be trained in a mask encoder, a Swing converter, a vector quantization variational autoencoder, or any combination thereof. Example The following exemplary embodiments represent examples of the software applications, systems, and methods described in this invention and are not intended to limit them in any way. Example 1: Color Difference Correction Lateral chromatic aberration correction is performed on each multi-channel image using a pre-calibrated transform. For example... Figure 10 As shown, the original image on the left has chromatic aberration and has been corrected. During the correction process, the aligned reference points in all channels can be clearly observed in the corrected image on the right. The parameters for chromatic aberration correction are listed in the correction transformation table. Example 2: Point Density Heatmap like Figure 13A and Figure 13B As shown, reasonable segmentation results were obtained, and cellular structures that were not shown in the original segmentation when morphological markers were used alone were revealed. Figure 13A The three columns show morphological images of brain tissue, with image channels including histone staining, DAPI staining, and cell somatic RNA staining. Figure 13B The three columns show the readout density heatmap, segmentation based on all three morphological staining images, and segmentation based on both the density heatmap and histone staining. Figure 13A Compared to the rRNA somatic staining morphology image in the third column of Figure 13B, the readout density heatmap in the first column shows a more detailed outline of the cell body with a higher signal-to-noise ratio. Therefore, Figure 13B The second and third columns show the cell segmentation results based on a combination of all three morphological staining images and a combination of readout density heatmaps and histone staining, respectively.

[0111] like Figure 24 As shown, image preprocessing techniques such as image sharpening and enhancement or machine learning denoising models are applied to the original image to improve image quality. In a non-limiting example, Figure 24 The image shows a comparison between the original image of the readout density heatmap and the image recovery output after processing with the Cellpose 3.0 denoising model. Example 3: Comparison of 2-channel and 5-channel model outputs Figure 17A The initial network output using the Cellpose algorithm is shown. The gradient flow output is generated using a 2-channel model. Ground truth annotations are produced based on the 2-channel model as initial ground truth, using segmentation results generated by an existing custom cell segmentation algorithm; this process is called bootstrapping. The annotations are then reviewed and corrected to ensure the quality of the training data. A new 5-channel network model is initialized using the parameters of the existing 2-channel model, which helps reduce the training burden and achieve good initial segmentation performance. Based on the annotated ground truth, the network is trained and fine-tuned to optimize the parameters of all channels. Once the specified learning rate is satisfied and a specific cost function is minimized, the parameter set is finally determined. Figure 17B The output gradient flow results of a trained 5-channel model initialized from transfer learning are shown. The trained and validated neural network can be exported as an Open Neural Network Exchange Format (ONNX) model to achieve interoperability across different frameworks and deployments.

[0112] Example 4: Image Acquisition: HDR Settings Figure 25 A non-limiting example of an image acquired using High Definition Range (HDR) settings is shown. Figure 25 In the non-limiting example, two non-HDR modes were used to acquire cell nuclear staining images, one with a higher exposure time setting and the other with a lower exposure time. Figure 25 In the non-limiting example shown, DAPI nuclear staining was used, and emission at 512 nm was detected in the blue channel under UV excitation at 385 nm. In the non-limiting example, a low-exposure scan was obtained at 3 ms and 3% power (A), followed by a scan at 24 ms and 4% power (B). Figure 25 As shown, some cells in B are already saturated. Figure 25 As shown, (C) illustrates an image acquired using two combinations of exposure times in HDR mode to provide greater dynamic range while reducing intensity saturation. Figure 25 As shown, (A) is an image acquired at 3ms (low); (B) is an image acquired at 24ms (high); and (C) is an image acquired using HDR mode.

[0113] For example, the morphology channels describing the cell membrane used the acquisition settings in Table 2 below. Table 2 lists the channels / dyes used, along with the excitation / emission wavelengths and exposure times. For example, using HDR with a factor of 8 means that two images are acquired for each channel, one with the specified exposure time and the other with an exposure time one-eighth of the initial image.

[0114] Table 2 Morphological Channels and Settings in Image Acquisition

[0115] Example 5: Image Preprocessing: Nearest Neighbor Deconvolution Figure 26 A non-restricted example of the output produced by the nearest neighbor (NN) deconvolution method is shown. For example, in Figure 26 In this model, based on the estimation described above, a Gaussian filter with a sigma value of 9.7 pixels was used, and the image was sampled at 800 nm at each z-sampling. For example, a multiplier A = 0.75 was used to ensure that most of the blurred signal was removed while still preserving dark cells. Figure 26 As shown, the circled areas mark the cell regions that were previously blurred due to the out-of-focus signal (left image). These regions were restored by nearest neighbor deconvolution, thus revealing the clear boundaries between cells that were previously difficult to see (right image).

[0116] Example 6: Intersection over Union (IoU) Analysis For example, such as Figure 27As shown, improved segmentation is achieved by integrating multiple segmentation steps from various modalities. For example, in some embodiments, combining separate nuclear segmentation with cytoplasmic and nuclear segmentation can produce enhanced results. For instance, some cells may lack sufficient membrane staining, relying solely on nuclear staining, or vice versa. In some embodiments, this multimodal segmentation method can be used to track brain cells. In some embodiments, different segmentation steps may be required to identify elongated processes, which are then integrated with nuclear and cell body data. In some embodiments, the criteria for merging or segmenting are determined through calculations involving cross-union ratios and area measurements that typically have minimal overlap. Figure 27 A non-limiting example of cell protrusion merging is shown.

[0117] Figure 28 shows a non-limiting example of the IoU merging results between the nuclear segmentation output and the membrane plus nuclear segmentation. Figure 28 The circled area indicates cell merging. Figure 28 The merged example in the upper right corner shows that the cell mask was detected only in the cell nucleus segmentation step (A) and not in the cytoplasm / membrane segmentation step (B). Figure 28 The second merge example in the lower left shows that when the two masks in (A) and (B) overlap, the masks are merged, preserving the shapes of (A) and (B).

[0118] Similarly, such as Figures 29A-29C As shown, this intersection analysis can be extended to associate cells in different z-slices to perform 3D cell segmentation. Figure 29A A non-limiting example of IoU merging results to correlate cell IDs across multiple 2D image slices is shown. Figure 29B A non-limiting example of IoU merging results within a 3D volume is shown. Figure 29C A non-restrictive example of a single cell labeled across all z-slices is shown to demonstrate that the cell was segmented across the entire z-stack. This data was generated using the NanoString Whole Transcriptome (Wtx) Pancreas Public Dataset.

[0119] Example 7: Segmentation results of different tissue types Multimodal segmentation systems have been tested on a variety of tissue samples, including those commonly used in immuno-oncology research, such as normal and diseased human breast, colon, lung, kidney, liver, and tonsils. Figure 30 As shown, it successfully segmented brain tissue and other tissues such as ovaries, osteosarcoma, skin, and muscle in humans and mice. Example 8: Statistical Table of Segmentation Mask Measurement Using a segmentation mask, various measurements of the cells were generated, including intensity and shape measurements. Tables 3 and 4 below show examples of intensity and shape attributes for a subset of the cell population, such as 20 out of 5309 cells. In some implementations, hundreds of images may be acquired.

[0120] Table 3. Strength and shape properties of cell subsets

[0121] Table 4. Area and shape properties of cell subsets

[0122] In some implementations, in addition to location, intensity, and shape attributes, such cell segmentation measurements may include other attributes such as cell fluorescence texture / entropy and neighborhood / inter-cell analysis. For example, to label cells dividing across multiple FOVs, calculations are performed by finding all cell IDs intersecting the FOV boundaries. Furthermore, for these dividing cells, the ratio between these cells and the local mean can be measured. In some implementations, if only a small fraction of cell fragments exist in the current FOV, they can be further filtered out during analysis. Table 5 shows examples of cell segmentation measurements. In Table 5, values ​​close to 1 indicate that those cells at the boundaries can be retained because their size is similar to the mean, and only a small fraction of dividing cells are located in adjacent FOVs.

[0123] Table 5 Cell division ratio

Claims

1. A computer-implemented system comprising a computing device, the computing device including at least one processor and instructions executable by the at least one processor to provide a multimodal subsegmentation application; It includes: (a) A software module configured to retrieve 3D scan images of biological samples in high dynamic range (HDR) mode, wherein the biological samples are labeled with one of a variety of morphological markers; (b) An optional software module configured to convert the 3D scanned image into a 2D image and obtain the optimal focus region in the z-slice therefrom; or, configured to deconvolve each 2D image to obtain the optimal focus region using a stack of 3D scanned images of the 2D images. (c) A software module configured to perform image preprocessing to correct and enhance images prior to segmentation, wherein the software module is further configured to perform subcellular segmentation of the biological sample based on the plurality of morphological markers; (d) A software module configured to integrate different imaging modalities, wherein the imaging modalities describe the nuclear modality, the cytoplasmic modality, and the cell membrane modality; (e) A software module configured to perform cell segmentation on 2D, 3D, and time-lapse images; (f) is a software module configured to generate single-cell attributes and measurement data for one or more images in (c)-(e), including fluorescence intensity and shape information as part of the processing results.

2. The system according to claim 1, characterized in that: It also includes a software module configured to retrieve readout density maps from transcriptomic analysis of the biological sample.

3. The system according to claim 2, characterized in that: The imaging modality includes the readout density map or the fluorescence image, or both.

4. The system according to claim 1, characterized in that: The morphological markers include fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents or nucleic acid probes, or any combination thereof.

5. The system according to claim 1, characterized in that: The images are microscope images, including images from optical microscopes, electron microscopes, or scanning probe microscopes.

6. The system according to claim 1, characterized in that: The image is then subjected to bleaching correction and deconvolution processing.

7. The system according to claim 1, characterized in that: The transcriptomics assays include gene expression assays, which use fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), gene expression cap analysis, or single-cell RNA sequencing (scRNA-seq), or any combination thereof.

8. The system according to claim 1, characterized in that: Color difference correction is performed on the image.

9. The system according to claim 1, characterized in that: The software module is configured to retrieve at least 1, at least 3, at least 5, at least 10, at least 15, at least 20, at least 30, at least 35, at least 40, at least 45, at least 50, at least 55, at least 60, at least 65, at least 70, at least 80, at least 90, at least 100, at least 120, at least 150, at least 200 or more images of the biological sample.

10. The system according to claim 1, characterized in that: The results of one or more subcellular segmentations are merged before (f).

11. The system according to claim 1, characterized in that: The biological sample is obtained at least in part by one or more of the following methods: biopsy, surgical resection, xenotransplantation, animal model, fine needle biopsy, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, lesion tissue sampling, and transplanted tissue sampling, or by any combination thereof.

12. The system according to claim 1, characterized in that: The biological sample includes cells or tissues.

13. The system according to claim 12, characterized in that: The cells include primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells.

14. The system according to claim 1, characterized in that: The subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation.

15. The system according to claim 1, characterized in that: The subcellular segmentation includes training the machine to learn algorithms and / or applying machine learning algorithms.

16. A non-transitory computer-readable storage medium encoded with instructions executable by one or more processors to provide multimodal segmentation applications, comprising: (a) A software module configured to retrieve 3D scan images of biological samples in high dynamic range (HDR) mode, wherein the biological samples are labeled with one of a variety of morphological markers; (b) A software module configured to convert the 3D scanned image into a 2D image and obtain the optimal focus region of a z-slice therefrom; or, a software module configured to deconvolve the 2D image to obtain the optimal focus region while preserving the z-position of the 2D image within its stack of 3D scanned images; and (c) A software module configured to perform subcellular segmentation of the biological sample based on the multiple morphological markers.

17. The non-transitory computer-readable storage medium according to claim 16, characterized in that: It also includes a software module configured to retrieve readout density maps from transcriptomic analysis of the biological sample.

18. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The morphological markers include fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents or nucleic acid probes, or any combination thereof.

19. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The images are microscope images, including images from optical microscopes, electron microscopes, or scanning probe microscopes.

20. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The image is then bleached and corrected.

21. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The transcriptomics assays include gene expression assays, which use fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), gene expression cap analysis, or single-cell RNA sequencing (scRNA-seq), or any combination thereof.

22. The non-transitory computer-readable storage medium according to claim 17, characterized in that: The software module is configured to retrieve at least 1, at least 3, at least 5, at least 10, at least 15, at least 20, at least 30, at least 35, at least 40, at least 45, at least 50, at least 55, at least 60, at least 65, at least 70, at least 80, at least 90, at least 100, at least 120, at least 150, at least 200 or more images of the biological sample.

23. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The biological sample is obtained at least in part by one or more of the following methods: biopsy, surgical resection, xenotransplantation, animal model, fine needle biopsy, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, lesion tissue sampling, and transplanted tissue sampling, or by any combination thereof.

24. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The biological sample includes cells or tissues.

25. The non-transitory computer-readable storage medium according to claim 24, characterized in that: The cells include primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells.

26. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation.

27. The non-transitory computer-readable storage medium according to claim 16, characterized in that: Merge one or more image results from the subcellular segmentation.

28. The non-transitory computer-readable storage medium according to claim 16, characterized in that: The subcellular segmentation includes training the machine to learn algorithms and / or applying machine learning algorithms.

29. A computer-implemented method, comprising: (a) Acquiring 3D scan images of biological samples in high dynamic range (HDR) mode by computer, wherein the biological samples are labeled with one of a variety of morphological markers; (b) Convert the 3D scanned image into a 2D image and obtain the optimal focus region of the z-slice therefrom; or, perform deconvolution processing to generate a 2D image for optimal contrast while preserving the z-position of the 2D image in its corresponding stack of 3D scanned images; and (c) Segmenting the biological sample based on the aforementioned multiple morphological markers.

30. The method according to claim 29, characterized in that: It also includes obtaining readout density maps from transcriptomic analysis of the biological samples.

31. The method according to claim 29, characterized in that: The morphological markers include fluorescent dyes, nuclear staining agents, fluorescently labeled antibodies, immunohistochemical (IHC) staining agents, photodegradable morphological markers, gene-coding tags, magnetic resonance imaging (MRI) contrast agents or nucleic acid probes, or any combination thereof.

32. The method according to claim 29, characterized in that: The images are microscope images, including images from optical microscopes, electron microscopes, or scanning probe microscopes.

33. The method according to claim 29, characterized in that: The image is then bleached and corrected.

34. The method according to claim 29, characterized in that: The transcriptomics assays include gene expression assays that use fluorescently labeled probes, RNA sequencing (RNA-seq), microarray analysis, reverse transcription polymerase chain reaction (RT-PCR), gene expression cap analysis, or single-cell RNA sequencing (scRNA-seq), or any combination thereof.

35. The method according to claim 29, characterized in that: The software module is configured to retrieve at least 1, at least 3, at least 5, at least 10, at least 15, at least 20, at least 30, at least 35, at least 40, at least 45, at least 50, at least 55, at least 60, at least 65, at least 70, at least 80, at least 90, at least 100, at least 120, at least 150, at least 200 or more images of the biological sample.

36. The method according to claim 29, characterized in that: The results of one or more subcellular segmentations are merged.

37. The method according to claim 29, characterized in that: The biological sample is obtained at least in part by one or more of the following methods: biopsy, surgical resection, xenotransplantation, animal model, fine needle biopsy, peripheral blood collection, bone marrow biopsy, healthy tissue sampling, tumor tissue sampling, malignant tissue sampling, lesion tissue sampling, and transplanted tissue sampling, or by any combination thereof.

38. The method according to claim 29, characterized in that: The biological sample includes cells or tissues.

39. The method according to claim 38, characterized in that: The cells include primary cells, stem cells, immune cells, epithelial tumor cells, sarcoma cells, lymphoma cells, melanoma cells, cancer cells, or tumor cells.

40. The method according to claim 29, characterized in that: The subcellular segmentation includes nuclear segmentation, cytoplasmic segmentation, or extracellular segmentation.

41. The method according to claim 29, characterized in that: The subcellular segmentation includes training the machine to learn algorithms and / or applying machine learning algorithms.

Images