Domain swap and artificially generated virtual images

Abstract

This disclosure relates to a domain swap by accessing images from a first domain, which is processed using a machine learning model to generate a virtual composite image in a second domain. This technique can eliminate or reduce the need to separately collect images in the second domain, saving time and cost. It leverages tools available in the second domain to perform image processing on the virtual composite image. The results or analysis from the image processing in the second domain can then be directly applied to the first domain to further evaluate the image. Since the spatial reference points (size, scale, view, etc.) are the same, the pixels that identify the boundaries of the regions depicted in the virtual composite image are the same pixels in the first domain.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority and interest to U.S. Provisional Patent Application No. 63 / 499,083, “Domain Swap and Artificial Generated Virtual Images,” filed on 28 April 2023, which is incorporated herein by reference in its entirety for all purposes. [Background technology]

[0002] background Digital pathology may involve the interpretation of digital images to accurately diagnose a subject and guide treatment decisions. Digital pathology solutions can establish image analysis workflows to automatically detect or classify biological objects of interest, such as positive or negative tumor cells. An exemplary digital pathology solution workflow includes acquiring tissue slides, scanning a pre-selected region or the entire tissue slide with a digital image scanner (e.g., a whole-slide image (WSI) scanner) to acquire digital images, performing image analysis on the digital images using one or more image analysis algorithms, and potentially detecting and quantifying each object of interest (e.g., counting or identifying object-specific regions or cumulative regions) based on the image analysis (e.g., quantitative or semi-quantitative scoring such as positive, negative, moderate, weak).

[0003] Digital pathology can utilize singleplex or multiplex techniques. Singleplex uses staining for only one biomarker and also a reference stain. Multiplex, or MPX, on the other hand, involves staining for two or more biomarkers (in addition to the reference stain) on a single slide. Thus, the multiplex technique supports the simultaneous detection and co-expression of multiple biomarkers at the single-cell level. However, it is difficult for pathologists to annotate tumors on multiplex slides. Cross-validation between biomarkers involves comparing the status of biomarkers against morphological features. Morphological features can be identified by using other stains such as hematoxylin and eosin (H&E). H&E is absorbed by the nucleus, extracellular matrix, and cytoplasm.

[0004] Multiplex slides themselves cannot be chemically stained with H&E because absorption of H&E can make it difficult or impossible to reliably and precisely detect biomarker signals. Therefore, the workflow traditionally involved staining sections adjacent to the multiplex section with H&E (see Figure 1). However, this requires pathologists to manually create and annotate extra slides (H&E slides), resulting in extra time, extra glass usage, cost, and effort, as well as the need to collect more tissue for evaluation. Traditional multiplex analysis requires spatial alignment to produce analytical results from H&E serial sections used against the multiplex image. However, this labor-intensive spatial alignment process is never perfectly accurate, and artifacts on either slide can impair the analysis. This is because sections are often 4 microns thick, while cells are often 4–20 microns in diameter (and potentially highly asymmetrical), meaning that a given cell may not be visible on two adjacent slides.

[0005] Furthermore, developing multiplex algorithms requires the use of aligned H&E images to confirm, for example, tumor or stromal segmentation. Multiplex analysis requires cross-validation of multiple biomarkers in relation to H&E morphology. Therefore, developing multiplex algorithms is very complex and requires a sophisticated alignment process that cannot be perfectly accurate. Thus, it is advantageous that reference features can be accurately and reliably identified to reduce the amount of tissue, time, and cost required. [Overview of the project]

[0006] summary Some embodiments of this disclosure relate to the use of generative AI models for transforming images from a first domain to a second domain, leveraging already developed technologies or tools in the second domain, and transferring the results back to the first domain. A computer implementation method includes accessing a first image from the first domain. The first domain may correspond to one or more specific imaging modalities. If the first image is a digital pathology image, the first domain may further correspond to one or more specific stains.

[0007] The method may further include generating a virtual composite image by processing a first image using a machine learning model. The virtual composite image may belong to a second domain corresponding to a different imaging modality or different staining relative to the first domain. Image processing tools configured to process images in the second domain may be accessed. The image processing tools may use segmentation, classification, or object detection models. Thus, processing the virtual composite image using the image processing tools may generate one or more annotations of the virtual composite image. One or more annotations of the virtual composite image may be transferred to the first image. Furthermore, analysis using the first image and the transferred one or more annotations may be performed in the first domain.

[0008] According to some embodiments, the machine learning model may include cycleGAN. In some cases, cycleGAN may be trained and configured to generate virtual H&E based on multiplexed images. The first domain may include dark-field multiplexing, and the second domain may include H&E. The first domain may include immunohistochemistry, and the second domain may include H&E. Additionally or alternatively, the first or second domain may include magnetic resonance imaging or other radiographic imaging. One or more annotations may include annotations of tumor, stromal, or artifact regions. One or more annotations may also include segmentation of each of the cell aggregates.

[0009] The various techniques disclosed herein can be used to leverage existing tools or models within a well-developed imaging domain and transfer them to other or new domains where such tools do not exist or where it is difficult to develop them.

[0010] In some embodiments, a system is provided that includes one or more data processors and a non-temporary computer-readable storage medium containing instructions that, when executed by the one or more data processors, cause the one or more data processors to perform some or all of the methods disclosed herein.

[0011] In some embodiments, a computer program product is provided which includes instructions tangibly embodied in a non-temporary machine-readable storage medium and configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0012] In some embodiments, a system is provided which includes one or more means for performing some or all of one or more methods or processes disclosed herein.

[0013] The terms and expressions used are for illustrative purposes only, not limitation, and in using such terms and expressions there is no intention to exclude equivalents or parts of the features shown and described, however it is acknowledged that various modifications are possible within the scope of the invention as described in the claims. Accordingly, while the invention as described in the claims is specifically disclosed by embodiments and optional features, modifications and variations of the concepts disclosed herein may be used by those skilled in the art, and it should be understood that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawing]

[0014] The patent or application file shall contain at least one drawing made in color. A copy of this patent or patent application publication containing the color drawing shall be provided by the Patent Office upon request and payment of the required fee. This disclosure shall be described in conjunction with the following attached drawings.

[0015] [Figure 1] Figure illustrating an example of traditional multiplex imaging analysis, which requires extra hematoxylin and eosin (H&E) slides from adjacent sections to confirm tumor and / or stromal segmentation.

[0016] [Figure 2] A block diagram illustrating an exemplary outline of a system for performing a domain swap by generating a virtual composite image, according to one embodiment of the present disclosure.

[0017] [Figure 3] An exemplary block diagram illustrating the implementation of a method for domain swapping and subsequent processing using existing tools for a second domain, according to one embodiment of the present disclosure.

[0018] [Figure 4]A diagram showing an exemplary network of the digital pathology image generation system of FIG. 2.

[0019] [Figure 5] A diagram showing an exemplary workflow of virtual staining by generating a synthetic virtual H&E image based on a swap from a multiplexed image to a bridged domain according to an exemplary embodiment of the present disclosure.

[0020] [Figure 6] A diagram showing an example of a CycleGAN (Cycle Generative Adversarial Network) for generating a synthetic virtual H&E image from a multiplexed digital pathology image according to an exemplary embodiment of the present disclosure.

[0021] [Figure 7] A diagram showing examples of biomarkers having different colors in individual channels of an MPX fluorescence image.

[0022] [Figure 8] A diagram showing examples of tumor prediction and artifact detection based on an H&E image.

[0023] [Figure 9] An exemplary diagram of tumor-stroma separation in real and synthetic H&E images according to an exemplary embodiment of the present disclosure.

[0024] [Figure 10] A diagram showing an exemplary method of cell segmentation using an H&E image according to some embodiments of the present disclosure.

[0025] [Figure 11] An exemplary flowchart of a system for performing domain swap and subsequent processing using existing tools in a second domain.

[0026] [Figure 12]A figure illustrating a parallel comparison of a synthetic H&E image with a synthetic or multiplexed image and an adjacent real H&E image according to an exemplary embodiment of the present disclosure.

[0027] [Figure 13A] A figure illustrating a parallel comparison of a synthetic H&E image and a segmented tumor within the synthetic H&E image, according to an exemplary embodiment of the present disclosure.

[0028] [Figure 13B] A figure illustrating an example of cells detected in a synthetic H&E image having a segmented tumor and an enlarged cell patch, as detected according to an exemplary embodiment of the present disclosure.

[0029] [Figure 14A] A figure illustrating an example of a multiplex or composite image having segmented tumors and cells according to an exemplary embodiment of the present disclosure.

[0030] [Figure 14B] This figure shows an enlarged patch of the multiplexed image of Figure 14A having segmented nuclei in the dapi channel and the synthesized image, according to an exemplary embodiment of the present disclosure. [Modes for carrying out the invention]

[0031] Some embodiments of this disclosure relate to the use of generative AI models for transferring images from a first domain to a second domain, leveraging previously developed techniques or tools in the second domain, and transferring the results back to the first domain for further processing of the images. The first or second domain may correspond to different imaging modalities (e.g., radiographic imaging, bright-field imaging, etc.) or different staining in digital pathology (e.g., H&E staining, multiplex staining, etc.). According to some embodiments, this disclosure provides a technical solution to the technical problem of transferring tools and / or techniques to different imaging modalities.

[0032] As used herein, the term “domain” (first domain or second domain) may refer to a representation of an image using a particular staining or imaging modality. Exemplary domains include H&E staining of digital pathology slides, multiplex staining of digital pathology slides (using any combination of stains), singleplex staining of digital pathology slides (using any given stain), and immunohistochemistry (using any given antibody). Exemplary domains may further include imaging modalities such as dark-field microscopy, bright-field microscopy, fluorescence microscopy, or radiographic imaging (e.g., magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET)).

[0033] As used herein, the term “same space” refers to a space having the same spatial reference point within the same source image. Different domains may exist or be represented within the same space. For example, a first domain may include a singleplex domain using a first specific stain, and a second domain may include a different singleplex domain using a different specific stain. In another example, a section (tissue slice) may be stained with a first dye or stain and scanned to obtain a singleplex image in the first domain. The same exact section may then be successively stained with another different dye or stain and scanned to obtain a duplex image in the second domain. Both of these images (singleplex and duplex) may belong to different domains but are represented within the same space. Similarly, two different stains or labelings within exactly the same section may produce images within the same space, such as successively imaging the same immunofluorescence histochemistry sample for two fluorophores. Multiple different images with multiple different spatial reference points may represent the same domain (for example, two H&E images within a set of consecutive or adjacent intersections).

[0034] In some embodiments of this disclosure, images from a first domain are processed using a machine learning model (e.g., cycleGAN model, Pix2pix GAN, generative pre-trained transformer model) to generate a virtual composite image in a second domain. This technique can eliminate or reduce the need to separately collect images in the second domain, saving time and cost. Tools available in the second domain can be used to perform image processing on the virtual composite image. For example, segmentation and / or categorization may be performed. Thus, the results from image processing can be easily and simply used to evaluate the images from the first domain, given that they have the same size, scale, view, etc. Thus, pixels that identify the boundaries of regions depicted in the virtual composite image are the same pixels in the images from the first domain. The disclosed techniques can be used across various combinations of domains.

[0035] In this disclosure, the term “singleplex” may refer to an image displaying a single staining component or marker. This term is often used in contrast to “multiplex” or “MPX” images, which include the simultaneous visualization of multiple staining components in a single cell or tissue sample. The term “sample” may be understood as material derived from a bioorganism, including but not limited to hair, skin samples, tissue samples, cultured cells, cell culture media, and biological fluids. The term “tissue” refers to a mass of interconnected cells derived from a human or other animal (e.g., central nervous system (CNS) tissue, liver tissue, or eye tissue). A sample may also include cell-related connective materials and fluids, such as blood samples. In the context of histopathology, the term “slide” refers to a glass microscope slide carrying a thin section of tissue stained for microscopic examination. The term sample may also include a culture medium containing isolated cells. Those skilled in the art can determine the amount of sample required to obtain a reaction using standard laboratory techniques. Furthermore, the terms adjacent slide or sequential slide refer to a slide containing subsequent sequential tissue slices from the same sample used to prepare the original slide. These adjacent slides can be used for cross-validation or as a reference to the original slide analysis, allowing researchers to interpret the results. For example, the original slide may be stained with a given set of dyes or markers (e.g., multiplex), and the next slide may be stained with a different set of dyes or markers (e.g., H&E).

[0036] As used herein, the term “biomarker” refers to tissue characteristics, including but not limited to the presence of certain cell types, such as immune cells, particularly those indicating a medical condition. Identification of a biomarker may include the presence of certain molecules, such as proteins, within the tissue characteristics.

[0037] The term "marker" is defined herein as a stain, dye, or tag used to distinguish a biomarker from surrounding tissue or other biomarkers. Tags may include antibodies, specifically those that exhibit high affinity for proteins associated with a particular biomarker, but can be employed for labeling purposes. Markers may exhibit high affinity for specific biomarkers, such as specific molecules or proteins associated with a disease. The biomarkers associated with a marker may be different from or exclusive to each other. Furthermore, dye-based markers may impart color to tissue, thereby indicating the presence of a biomarker in the tissue. Various stains and dye-based markers may appear in different colors in a sample, and multiple markers can be used in combination.

[0038] Similarly, in the context of immunohistochemistry (IHC), an IHC marker refers to an antibody specifically designed to bind to a target protein or antigen within a tissue section. IHC markers can be conjugated to various tags (e.g., chromogens, quantum dots, or fluorophores) to facilitate the visualization and identification of specific cellular components or biomolecules within the tissue. Thus, IHC markers aid in the characterization and diagnosis of various medical conditions or research objectives.

[0039] Differential staining is fundamental to pathology and encompasses the staining of cytoplasm, organelles including the nucleus, and markers associated with specific proteins. A prime example is hematoxylin-eosin (H&E) staining, where hematoxylin (blue) primarily stains the cell nucleus, while eosin (magenta-red) acts as a cytoplasmic stain. Differential staining can increase contrast in a sample, allowing for easy identification of cellular components. The ratio of hematoxylin to eosin staining in the cytoplasm can also provide insight into its basophilic or eosinophilic properties. Another common application of differential staining involves IHC staining, which can highlight the presence of specific epitopes based on antigen-antibody binding. With IHC staining, unique, highly specific antibodies can be developed for almost any target (or biomarker) that can also be conjugated with various tags. Because finding a stain or dye to label any two or three random target proteins with different colors can be extremely difficult, IHC staining is commonly used for differential and / or multiplex staining. Pathologists frequently utilize IHC techniques for cancer diagnosis to identify immune cells, assess the expression of tumor and cell proliferation markers, and detect conditions such as degenerative disorders and infections.

[0040] In immunohistochemistry, the term "labeling" refers to attaching a tag or marker to an antigen to aid in its detection. There are two main types of IHC labeling: fluorescent and chromogenic. Fluorescent labeling uses compounds called fluorophores, which produce a fluorescent signal when excited by light (e.g., ultraviolet (UV) or visible light). Recently, quantum dot (QD) labeling has attracted considerable attention. Quantum dots (QDs) are semiconductor nanocrystalline fluorophores with extremely high fluorescence efficiency and low photobleaching. Due to their quantum and size effects, QDs have a constant excitation wavelength, along with a sharp, symmetric, tunable emission spectrum. Fluorescence microscopy can be used to observe fluorescently labeled slides or specimens. Fluorescence imaging can typically be performed using a monochrome camera combined with multiple filter sets matched to the absorbance and emission characteristics of each fluorophore. Furthermore, dark-field microscopy is a technique that utilizes oblique illumination to improve the contrast of specimens that are not well imaged under normal illumination conditions. For example, cellular structures that may appear transparent under bright-field illumination can be observed with better contrast and detail using dark-field microscopy. On the other hand, colorimetric labels produce pigment deposits based on enzyme / substrate reactions, which can be observed using a bright-field microscope. Some chromogens include, but are not limited to, di-amino-benzidine (DAB), amino-ethyl-carbazole (AEC), Bajoran Purple (trademark), Vina Green (trademark), and Fast Red (FR).

[0041] Immunohistochemical techniques can be used to study multiple biomarkers or antigens in the same tissue section. IHC techniques can provide comprehensive information on various cell interactions, tissue heterogeneity, antigen localization and co-localization, functional state, antigen distribution, and relative concentration. In addition, multiplex staining saves the cost, time, and effort of preparing multiple slides for each stain or biomarker and allows for joint or relative analysis of different cell populations on the same tissue section. Multiplex IHC staining involves multiple primary antibodies, each recognizing a specific target. Subsequently, two or more secondary antibody molecules can be bound to each primary antibody, so corresponding secondary antibodies can also be applied to enhance signal amplification. A chromogen or fluorophore can be conjugated to a primary antibody (direct method) or a secondary antibody (indirect method) to label the antigen.

[0042] When fluorophores are used to visualize IHC targets, this technique may be referred to as fluorescence immunohistochemistry (f IHC) or immunofluorescence staining (IF). Multiplex fluorescence immunohistochemistry (mf IHC) can be used to label multiple targets in the same sample by conjugating either a primary or secondary antibody with fluorophores having different absorption and emission spectra. Different fluorophores can be imaged simultaneously or sequentially, for example, by using fluorescence imaging. Each fluorophore may correspond to a specific channel representing the location of a target antigen or biomarker. The channels can then be combined into a single synthetic image or observed separately.

[0043] One exemplary embodiment of this disclosure relates to generating a synthetic reference image (e.g., a synthetic virtual H&E stained image) by processing a multiplex image using a machine learning model (e.g., a cycleGAN model, a Pix2pix GAN model, or a generative pre-trained transformer model). The multiplex image is transferred from a multiplex domain (where multiple biomarkers are stained) to a virtual H&E domain. This technique can eliminate the need to stain, image, and process sections with reference stains, resulting in resource savings. Furthermore, processing tools developed in the H&E domain can be used to analyze the virtual synthetic reference image (e.g., for segmenting tumor and / or stromal regions, detecting artifacts, segmented cells, etc.). Because the reference points are the same, boundaries and / or regions (e.g., tumor regions, stromal regions, artifact depictions, and / or one or more cells) can be easily mapped to the multiplex domain and the multiplex image.

[0044] Another exemplary embodiment relates to the use of synthetically generated images that may be used for validation purposes for model execution and / or for fine-tuning of virtual image results. For example, if an algorithm or tool is developed in the IHC domain, virtual stained H&E images may be generated using IHC images. Tools available in the IHC domain can then be transferred to the H&E domain and applied directly. Thus, it will be understood that tools and algorithms developed in one domain may be transferred using virtual slides generated from other domains.

[0045] The various techniques disclosed herein can be used to leverage existing tools or models within a well-developed imaging domain and transfer them to other or new domains where such tools do not exist or where it is difficult to develop them.

[0046] Figure 2 is a block diagram illustrating an exemplary overview of a system for performing a domain swap by generating a virtual composite image, according to one embodiment of the present disclosure. The exemplary system 200 may include an image generation system 205 connected to one or more computer systems 215 via a network 210. The described technique of domain swapping, which creates a virtual composite image in a second domain by applying a generation AI model to images from a first domain, may be performed on the computer system 215. The computer system 215 may also include user input and output devices (not shown), such as a keyboard, mouse, stylus, and display / touchscreen. The computer system 215 may receive one or more images from the first domain from the image generation system 205 and generate an image of the second domain. In addition, the computer system 215 may be used to run image processing tools for the second domain and store the results or images in one or more databases 220.

[0047] An exemplary computer system 215 of system 200 may include a processing system having one or more high-speed central processing units (CPUs), processors, and one or more memories. The computer system 215 may also include memory for storing processing modules or logical instructions executed by one or more combined processors. The computer memory for storing data may also be maintained on computer-readable media including magnetic disks, optical disks, organic memory, and any other volatile (e.g., random access memory (RAM)) or non-volatile (e.g., read-only memory (ROM), flash memory, etc.) mass storage systems readable by the CPU. The computer-readable media may include collaborative or interconnected computer-readable media that may reside exclusively on the processing system or be distributed among multiple interconnected processing systems that may be local or remote to the processing system.

[0048] Network 210 may include the Internet, intranet, wired LAN (Local Area Network), wireless LAN (Wi-Fi), WAN (Wide Area Network), MAN (City-Scale Network), PSTN (Public Switched Telephone Network), and other types of communication networks. Network 210 may further include one or more communication devices such as gateways, routers, or bridges. As just one example, Network 210 may have one or more servers and one or more websites accessible by users to send and receive information usable by the computer system 215. Network 210 may be any type of network well known to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc.

[0049] An exemplary system 200 may further include one or more databases 220 for processing and storing data (e.g., histopathology images). One or more databases 220 may be integrated with the memory system on the computer, or in secondary storage devices such as hard disks, floppy disks, optical disks, or other non-volatile mass storage devices. The computer system 215 may include one or more servers, or client terminals that communicate with personal information terminals / data assistants (PDAs), laptop computers, mobile computers, internet appliances, one-way or two-way pagers, mobile phones, or other similar desktop, mobile, or handheld electronic devices.

[0050] For example, the computer system 215 may provide means for inputting image data depicting one or more scanned digital pathology slides from the image generation system 205 into memory. The image data may include data relating to color channels (RGB) for bright-field imaging. In fluorescence imaging, the image data may include data relating to several distinct channels. Each channel may correspond to or play a role in capturing a specific spectral range or (signal) wavelength emitted from a fluorophore. Thus, each channel provides the image with a representation of a specific biomarker. For example, a biological specimen, such as a tissue section, may need to be stained by applying a staining assay to highlight one or more different biomarkers associated with chromogenic stains for bright-field imaging or fluorophores for fluorescence imaging. The staining assay may use chromogenic stains for bright-field imaging, organic fluorophores, quantum dots, or a combination of organic fluorophores and quantum dots for fluorescence imaging, or any other combination of staining and observation or imaging devices. In the analysis of a biological specimen, such as cancer tissue, different stains are specified to identify one or more types of biomarkers, such as immune cells.

[0051] Figure 3 shows an exemplary block diagram of the implementation of a method for domain swapping and subsequent processing using existing tools in the second domain. In some embodiments, an image D1 305 from the first domain is processed using a generative model 310 (e.g., a cycleGAN model, Pix2pix GAN, or a generative pre-trained transformer model) to generate a composite image D2 315 in the second domain. For example, the image D1 305 from the first domain may be a multiplexed IHC image, and the composite image D2 315 in the second domain may represent H&E. This technique can eliminate or reduce the need to separately collect images in the second domain, while still allowing the user to employ tools developed for use in the second domain. This technique can save time and cost. The composite image D2 315 may be processed by an image processing model 320. The image processing model 320 may include models or tools that are available in the second domain and can be used to perform image analysis on the composite image D2 315. For example, segmentation and / or categorization may be performed by the image processing model 320 to generate annotation 325. The result from the image processing model 320 or annotation 325 can be mapped to image D1 305 by the analyzer 330. Since image D1 305 and composite image D2 315 have the same size, scale, and view, the mapping can be easily performed. Thus, the pixels that identify the boundaries of the regions depicted in composite image D2 315 are the same pixels in image D1 305 of the first domain. After mapping annotation 325 to image D1 305, the analyzer 330 may perform additional analysis on image D1 305 based on annotation 325 and use it to generate output image D1 335 of the first domain.

[0052] According to several aspects of this disclosure, the disclosed techniques may be used across various combinations of domains. A domain as used herein (e.g., a first domain or a second domain) refers to a representation of an image using a particular stain or imaging modality. Exemplary domains include, but are not limited to, H&E staining, multiplex staining (using any combination of stains), singleplex staining of digital pathology slides (using any given stain), and labeling via immunohistochemistry (using any given antibody). Exemplary domains further include imaging modalities such as dark-field microscopy, bright-field microscopy, fluorescence microscopy, or even radiographic scanning such as MRI, CT, X-ray, and PET. Different domains may exist or be represented in the same space. Image D1 305 of the first domain and a composite image D2 315 of the second domain both represent the same spatial reference point and may be considered in the same space. For example, the first domain may include an image with multiplex staining, and the second domain may include an image with H&E staining.

[0053] Figure 4 shows an exemplary network of the image generation system 205 of Figure 2 for generating digital pathology images. The image generation system 205 may include a fixation / embedding system 405, a tissue slicer 410, a staining system 415, and an imaging system 420. The fixation / embedding system 405 fixes and / or embeds tissue samples (e.g., liquid fixatives such as formaldehyde solution) and / or embedding materials (e.g., histological waxes such as paraffin wax and / or one or more resins such as styrene or polyethylene). Each slice may be fixed by dehydrating the slice after exposing it to the fixative for a predetermined period (e.g., at least 3 hours) (e.g., via exposure to an ethanol solution and / or a clearing intermediate). The embedding material can penetrate the slice when it is in a liquid state (e.g., when heated).

[0054] Next, the tissue slicer 410 slices the fixed and / or embedded tissue sample (e.g., tumor sample) to obtain a series of sections, each section having a thickness of, for example, 4-5 microns. Such sectioning may be performed by first cooling the sample and then slicing the sample in a warm water bath. The tissue may be sliced ​​using (e.g.) a vibratome or compressstorm.

[0055] Since tissue sections and the cells within them are nearly transparent, slide preparation generally involves staining the tissue sections (e.g., automatically) to make the relevant structures more visible. In some cases, staining is performed manually. In other cases, staining is performed semi-automatically or automatically using the staining system 415.

[0056] Staining may involve exposing individual tissue sections to one or more different stains (e.g., sequentially or simultaneously) to express different characteristics of the tissue. For example, each section may be exposed to a predetermined amount of stain for a predetermined period of time. The stains may include (e.g.) RNA probes, protein probes (e.g., nuclear protein probes or cytoplasmic protein probes), immunohistochemical stains, secretion probes, etc. In some cases, the stain may stain KAPPA mRNA or LAMBDA mRNA.

[0057] One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining can be used to show general aspects of tissue morphology and / or cellular microanatomy (e.g., distinguishing the cell nucleus from the cytoplasm, showing lipid droplets, etc.). An example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stain (e.g., Masson's trichrome), Schiff periodate (PAS), silver stain, and iron stain. The molecular weight of histochemical staining reagents (e.g., dyes) is generally about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian blue, phosphomolybdic acid (PMA)) can have molecular weights up to 2000 or 3000 kD. An example of a high molecular weight histochemical staining reagent is α-amylase (approximately 55 kD), which is sometimes used to indicate glycogen.

[0058] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that specifically binds to a target antigen (biomarker) of interest. IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) binds to the primary antibody. Because antibodies have a molecular weight of approximately 150 kD or more, the molecular weight of IHC reagents is much larger than that of histochemical staining reagents.

[0059] The sections may then be individually mounted on corresponding slides. The imaging system 420 may then scan the slides to scan digital pathology images 425a-n. Each section may be mounted on a slide, and the slides may then be scanned to create a digital image, which may then be examined by digital pathology image analysis and / or by a human pathologist (e.g., using image viewer software). The imaging system 420 may digitize pathology slides (entire slide or section) using bright-field imaging, dark-field imaging, or fluorescence imaging. The imaging system 420 may include, but is not limited to, a microscope with a digital camera, a robotic microscope, or a WSI scanner such as a Ventana iScan HT, Ventana DP 200, or Ventana DP 600.

[0060] In some cases, the pathologist may review the digital images of the slides and manually annotate them (e.g., tumor area, necrosis, etc.). Annotation of regions of interest may be performed automatically using computer vision techniques. Digital pathology images 425a-n may be converted to other domains for further processing.

[0061] Digital histopathology images (e.g., 425a) typically contain an array of pixels, usually a rectangular matrix. Each “pixel” is a single pixel, a digital quantity representing some property of the image at a location in the array corresponding to a specific location in the image. Typically, in a continuous-tone black and white image, the pixel value represents a grayscale value. Pixel values ​​in a digital image typically correspond to a specified range. For example, each array element may be one byte (i.e., 8 bits) representing a pixel value in the range of 0 to 255. In a grayscale image, “255” may represent absolute white, and 0 (“0”) may represent absolute black (or vice versa). Color images may generally contain a tricolor plane corresponding to red, green, and blue (RGB). For a given pixel, there is one value in each of these color planes (i.e., a value representing the red component, a value representing the green component, and a value representing the blue component). By varying the intensity of these three components, typically all colors of the color spectrum are created. Specimens stained by multiplex IHC may be sequentially irradiated with multiple light channels matching the absorption bands of the chromogen to capture bright-field images. In the case of multiplex immunofluorescence, fluorescence microscopy using different filters may be used to capture fluorescence or emission from fluorophores associated with each biomarker.

[0062] Figure 5 illustrates an exemplary workflow for virtual staining by generating a synthetic H&E image based on multiplexed images and bridged domain swaps. Conventionally, serial tissue slices from a sample are stained with different dyes or techniques for disease-specific analysis. For example, multiplex IHC staining may be performed on a tissue slice to yield an image such as slice 1 - MPX 505 in Figure 5. Since H&E staining provides better morphological information, serial slices of a sample may be stained with H&E, as shown by serial slice 2 - H&E 510. This staining of serial slice 2 - H&E 510 requires additional processing such as acquisition of serial tissue slices, preparation of glass slides, staining, or scanning, resulting in additional time, cost, and effort. Furthermore, due to the typical thickness of tissue slices (3-4 microns) compared to the typical cell size (4-20 microns), a given cell may not be visible in two serial slices of the sample. As a result, spatial alignment of reference points between two images (e.g., slice 1 - MPX 505 and continuous slice 2 - H&E 510) becomes difficult, and artifacts on the slide or images may impair the analysis.

[0063] In some embodiments of this disclosure, multiplex images may be employed to generate highly realistic corresponding virtual H&E images. When these virtual H&E images are analyzed using algorithms designed for H&E, they yield results that closely reflect the results obtained from the analysis of actual H&E images, achieving a high level of accuracy. For example, generation model 310 may be used to create slice 1-virtual H&E 520 image based on slice 1-MPX 505 image. Image processing model 320, which may include tumor segmentation 530 or cell segmentation 535, may then be utilized on slice 1-virtual H&E 520 image or a patch 525 thereof to generate virtual H&E segmentation 540. Analyzer 330 may utilize virtual H&E segmentation 540 to generate multiplex segmentation by mapping tumor or cells on slice 1-MPX 505 as indicated by image patch 545 of the multiplex image. This mapping by analyzer 330 is possible because the spatial reference point is the same between slice 1-virtual H&E 520 (second domain) and slice 1-MPX 505 (first domain). Thus, according to the disclosed virtual staining and domain swapping techniques, annotations and other data collected in one domain can be seamlessly transferred to another domain.

[0064] Figure 6 shows an example of a cycle-generative adversarial network (CycleGAN) 600 for generating a synthetic virtual H&E image from an MPX digital pathology image, according to an exemplary embodiment of the present disclosure. Figure 6 shows how the CycleGAN model is used as a deep learning technique for transforming an image from a first domain to a second domain. In the illustrated example, the first domain is a 6-channel multiplex domain, and the second domain is the H&E domain. The 6-channel multiplex domain may be represented by a 6-channel multiplex image, also hereafter referred to as a synthetic image or simply a multiplex image. The multiplex image may highlight six different stains or biomarkers. In some cases, the 6-channel multiplex domain may also be represented by MPX fluorescence images or MPX dark-field images associated with individual channels (e.g., a 6-channel fluorescence-MPX dark-field image). Each image in the dark-field image may correspond to a different channel and depict a different biomarker.

[0065] CycleGAN 600 is a type of GAN specifically designed for non-paired image-to-image transformations. It is commonly used for tasks such as style transfer, image colorization, and image transformation. A key innovation of CycleGAN 600 is its ability to learn mappings between two domains (e.g., multiplexed images to H&E images, horses to zebras, or noisy to denoised images) without requiring paired data samples from both domains during training. CycleGAN 600 has two GANs, 605 and 610, for each domain. Each GAN within CycleGAN 600, like other GAN architectures, may further consist of two main components: a generator network (e.g., 620 and 650) and a discriminator network (e.g., 630 and 660). For example, a generator G with an X→Y mapping to transform an image from domain X or a first domain (multiplex) to domain Y or a second domain (H&E), and vice versa. X620 Network and Y→X Inverse Mapping Generator G Y 650 can each be dedicated.

[0066] Each generator can capture one or more images from its respective domain. For example, generator G X 620 can obtain the real multiplexed image (X) 615a as input and a converted image, TIFF2026517721000002.tif5170 This converted image can be similar to the target domain, i.e., TIFF2026517721000003.tif5170 For example, it can be similar to a dataset of hematoxylin or eosin stained images. In CycleGAN 600, there are D Y 630 and D X 660, one discriminator for each domain, as shown. These discriminators aim to distinguish between real images from the target domain and fake or synthetic images generated by the generator. For example, discriminator D Y 630 can aim to distinguish between real H&E images (Y) 635a~r and X from generator G TIFF2026517721000004.tif5170. Similarly, discriminator D X 660 can be trained to distinguish between real multiplexed images (X) 615a~n and fake or Y from generator G TIFF2026517721000005.tif5170. In each GAN (605 and 610), the generator and the discriminator are trained adversarially, which involves a competitive process between the two networks.

[0067] In CycleGAN 600, the generator and discriminator facilitate the transformation of images between two domains while preserving semantic content. The generator may employ a convolutional neural network (CNN), a transformer-based architecture, or a deep neural network architecture such as a residual network (ResNet), which can extract and transform features at different levels of abstraction by leveraging multiple layers. For example, the generator may have encoder-decoder components, where the encoder extracts high-level features from the input image and the decoder reconstructs these features into a target domain. The discriminator, on the other hand, is a binary classifier that evaluates the authenticity of an image as real or fake. Both discriminators may have different or similar architectures that utilize neural networks such as CNNs to analyze the features of a synthetic image and compare them to the features of a real image in their respective domains. It can be understood that the network architectures of both generators and both discriminators can be variations of neural networks. Both generators may have similar or different network architectures that are trained adversarially while maintaining cycle consistency.

[0068] The generator's goal or focus is to produce a composite image that is indistinguishable from a real image. The discriminator's goal or focus is to correctly classify real images as genuine and composite images as fake. The adversarial objective or loss (e.g., 640a and 640b) is one of the main components of the GAN and is responsible for training the generator to produce images that look realistic. Real multiplex images (X) 615a~n Generator G should convert to TIFF2026517721000006.tif5170 X 620 and The adversarial loss 640a for training TIFF2026517721000007.tif5170 can be formulated as follows:

number

[0069] In the case of reverse mapping, TIFF2026517721000011.tif5170 Generator G should convert to TIFF2026517721000012.tif5170 Y 650 and The adversarial loss 640b for training TIFF2026517721000013.tif5170 can be formulated as follows:

number

[0070] The principle of CycleGAN 600 is to transform an image from one domain to the other and back in one cycle. Therefore, between the original input (real multiplex images 615a~n) and the final composite image (composite multiplex images 655a~n) TIFF2026517721000017.tif6170 can be calculated with the goal of achieving consistency across both domains. The cycle consistency loss 645 can ensure that when an input image from domain X is transformed into domain Y and then returned to domain X, it will be highly similar to the input image from domain X. The cycle consistency loss 645 can help maintain mapping between different domains and prevent information loss during transformation. The cycle consistency loss is calculated as follows: TIFF2026517721000018.tif10170 can be achieved.

number

[0071] In CycleGAN 600, both generators can also be forced to preserve the color composition between their respective domains. To pursue this, images are fed from each domain through both the above generators and their reverse (i.e., generators from the opposite domain), as given by the following equation, and then Identity loss can be calculated by calculating the difference between TIFF2026517721000021.tif5170 and the original image. Identity loss can operate within the same domain and may focus on maintaining the identity of individual images. Cycle consistency loss 645 can operate across different domains and may focus on maintaining the consistency of mappings between domains.

number

[0072] Finally, the objective function is calculated by summing up all loss terms as follows: It can be formed by weighting by TIFF2026517721000023.tif6170.

number

[0073] In some other embodiments, other deep learning models may also be used to perform domain swapping. For example, a contrasting non-corresponding image-to-image transformation (CUT) model may be used. Non-corresponding image-to-image transformation may be based on patch-by-patch contrastive and adversarial learning. Compared to CycleGAN, CUT may learn to perform stronger distributional matching. Furthermore, for lighter weight (requiring less memory) and faster training, the FastCUT technique, a variation of CUT, may be used as an alternative to CycleGAN. In some cases, when paired image datasets of domain 1 and domain 2 are available, the pix2pix GAN may be used.

[0074] Figure 7 shows an example of a biomarker that has different colors in individual channels of an MPX fluorescence image. Fluorescence imaging can be performed using a monochrome camera combined with multiple filter sets matched to the absorbance and emission characteristics of each fluorophore. This allows imaging and separation of the same number of fluorophores as spectral separation allows, but with the disadvantage that it takes time to change between filter sets and separately acquire and process different filtered images. Different filtered images or MPX fluorescence images (e.g., six images as shown in Figure 7) can be combined to produce a composite image or multiplex image (e.g., a 6-channel multiplex image). The composite image is produced by integrating pseudocolors from the six different channels of MPX: red for Ki67, cyan for PD1, blue for DAPI, yellow for CD8, green for panCK, and pink for CD3.

[0075] Figure 8 shows an example of tumor prediction and artifact detection based on H&E images. In digital pathology, machine learning models are often trained and configured on H&E images of various clinical use cases, primarily used to support clinical diagnosis and prognosis. For example, tumor lesion detection algorithms may be used to identify and delineate areas within tissue corresponding to tumor lesions (or malignant growths). In some cases, tumor lesion detection algorithms can be deep learning-based models such as convolutional neural networks (CNNs) that can be trained to recognize tumor-specific patterns such as irregular cell shapes, increased cell density, and nuclear atypia. In other examples, methods based on intensity-based thresholding or machine learning techniques with domain-specific extracted features such as texture, shape, and intensity features from H&E image patches may be used to discriminate tumors. As an example, the performance of a tumor lesion detection algorithm on H&E-stained WSI 805 is shown in predicted labeled image 810.

[0076] Furthermore, tumor-stromal separation algorithms can also be used to distinguish between tumor epithelium (cancer cells) and stroma (surrounding connective tissue) within the same H&E image. Tumor-stromal separation algorithms can be based on deep learning techniques such as metastasis learning, which classifies epithelial and stromal regions by leveraging features learned by a pre-trained CNN. In some other examples, machine learning models using extracted features (e.g., texture, color, shape, spatial cell arrangement-related) can be used to segment tumor-stromal regions. Additionally, the tumor-stromal ratio (TSR), a prognostic factor for survival in various types of cancer, can also be calculated later. For illustrative purposes, Figure 8 shows that the predicted overlay image patch 820 is generated after processing a patch of the H&E image 815 via the tumor-stromal separation algorithm.

[0077] Similarly, in digital pathology, deep learning-based models can be trained and configured to remove artifacts from digitized slides (or entire slide images - WSI). Artifacts in digital pathology images can obscure important tissue areas and affect diagnostic accuracy. Common types of artifacts include out-of-focus areas, tissue folds, ink marks, dust particles, pen marks, or air bubbles. Other forms of artifacts may include, but are not limited to, necrosis and collapse. Necrosis can represent a broader category of cell death and can result from a variety of factors, including ischemia, physical drugs, chemical drugs, or immunological damage. Collapse, on the other hand, involves mechanical compression and tissue deformation. As an example, the automated artifact detection 825 in Figure 8 highlights collapsed and necrotic areas in an H&E image.

[0078] Figure 9 illustrates a typical illustration of tumor-stromal separation in real and synthetic H&E images according to an exemplary embodiment of the present disclosure. A comparison of the performance of the tumor-stromal separation algorithm shows that the results from applying the algorithm to a real H&E image 905 (real H&E prediction overlay 910) are largely preserved in the results of applying the same algorithm to a synthetic H&E image 915 (synthetic H&E prediction overlay 920). This example demonstrates that H&E domain tools configured to segment tumor and / or stromal regions and / or detect artifacts can also be applied to synthetic virtual staining images, and that the results are very similar.

[0079] Figure 10 illustrates exemplary methods of cell segmentation using H&E images according to several embodiments of the present disclosure. A first machine learning model (e.g., Cycle-GAN) may be used to transform an image of a first domain into a second domain, the second domain being the H&E domain. A tool trained on the second domain may perform cell segmentation using a process as shown in Figure 10. The cell boundaries are then mapped to the first domain and used for evaluations such as cell counting, tumor cell density calculation, and / or biomarker identification.

[0080] The cell segmentation pipeline 1005 may include converting the entire slide image 1010 into image patches 1015a-n and obtaining cell annotations 1020a-n. A cell segmentation model 1025, such as a U-Net architecture, may be used to generate binary cell segmentation mask images 1030a-n. For example, the cell segmentation model 1025 may be trained with cell annotations 1020a-n on the image patches 1015a-n of the original H&E images and their corresponding binary mask images. Annotation may be performed manually by a pathologist.

[0081] After obtaining binary cell segmentation mask images 1030a-n from the cell segmentation model 1025, visualization of cell segmentation 1035 can be easily performed. To visualize cell boundaries, the binary cell mask (or images from binary cell segmentation mask images 1030a-n) can be mapped to corresponding H&E image patches (or images from image patches 1015a-n) to generate H&E image patches with segmented cells. This exemplary technique, shown in Figure 10, can automate cell segmentation to reveal cell boundaries on H&E images and can facilitate several downstream tasks, including basic cell counting, tumor cell density calculation, and biomarker identification.

[0082] Figure 11 shows an exemplary flowchart of a system that performs domain swapping and subsequent processing using existing tools of the second domain. The blocks in the flowchart are shown in a specific order, but the order can be changed; for example, some blocks may be performed before others, and some blocks may be performed simultaneously. The blocks can be performed by hardware, software, or a combination thereof. The process in block 1105 may include accessing a first image from the first domain corresponding to one or more specific imaging modalities such as multiplex digital pathology images, MPX fluorescence microscopy, MPX bright-field microscopy, MRI, CT, or PET. If the first image is a digital pathology image, the first domain may correspond to one or more specific stains (e.g., mIHC, mfIHC, H&E). For example, a slide having slices of tissue sample may be stained using multiple IHC markers, resulting in a multiplex digital pathology image which may be a duplex, triplex, or fourplex image. If only one IHC marker is used, it produces a singleplex image containing one color / stain.

[0083] In block 1110, a virtual composite image in a second domain is generated by processing a first image using a machine learning model. The second domain corresponds to a different imaging modality or different staining relative to the first domain. The machine learning model can be trained on a dataset containing images from the first and second domains. In block 1115, an image processing tool configured to process images in the second domain can be accessed. For digital pathology images, the image processing tool may include tumor, stroma, or cell segmentation.

[0084] In block 1120, one or more annotations of the virtual composite image may be generated by processing the virtual composite image using an image processing tool. One or more annotations may include segmented tumor, stroma, or cells within the virtual composite image of the second domain. In block 1125, one or more annotations of the virtual composite image may be transferred to the first image of the first domain. Finally, in block 1130, further analysis may be performed using the first image and the transferred one or more annotations.

[0085] Examples: Exemplary embodiments of the disclosed technique are provided for domain swapping using a 6-channel multiplex (mfIHC) as the first domain and H&E as the second domain. More specifically, mfIHC samples were prepared by labeling sections with six different tags or markers corresponding to the biomarkers shown in Figure 7. MPX fluorescence images were obtained using fluorescence microscopy with six channels. The MPX fluorescence images were synthesized to generate a composite image or 6-channel multiplex (hereinafter also referred to as a multiplex image).

[0086] Figure 12 shows a parallel comparison of a synthetic H&E image 1220 with a synthetic or multiplexed image 1210 and an adjacent real H&E image 1230 according to an exemplary embodiment of the present disclosure. The synthetic H&E image 1220 was generated by using the multiplexed image 1210 as input to the CycleGAN method introduced earlier in Figure 6. The multiplexed image 1210 represents the synthetic or multiplexed image. The adjacent real H&E image 1230 represents a true H&E image created by a pathologist by staining an adjacent slice. The similarity between the synthetic H&E image 1220 and the adjacent real H&E image 1230 provides confidence in the synthetic result. Importantly, there is a stronger higher-order match between the synthetic H&E image 1220 and the multiplexed image 1210 than the higher-order match between the adjacent real H&E image 1230 and the multiplexed image 1210. This suggests that the virtual synthetic image may be more valuable than the adjacent slide for reference purposes.

[0087] Figure 13A shows a parallel comparison of a synthetic H&E image 1220 and a segmented tumor 1310 within the synthetic H&E image, according to an exemplary embodiment of the present disclosure. Tumor segmentation in the synthetic H&E image 1220 was achieved by utilizing existing tools developed for the H&E domain, as discussed in Figure 9.

[0088] Figure 13B shows an example of detected cells 1320 in a synthetic H&E image 1220 with a segmented tumor 1310 and an enlarged patch 1330 of detected cells. In the image of detected cells 1320 in the segmented tumor 1310, green circles indicate the segmented boundaries of tumor cells, and green dots indicate predicted tumor nuclei.

[0089] Figure 14A shows an example of a synthetic or multiplexed image or composite image having segmented tumors and cells according to an exemplary embodiment of the present disclosure. By processing the synthetic H&E image 1220, tumor and cell segmentation, as identified in the H&E domain, was mapped to the multiplex domain to generate a composite or multiplexed image as shown in Figure 14A. The multiplexed image having segmented tumor and cell information may be further analyzed to determine a specific problem or disease.

[0090] Figure 14B shows an enlarged patch of the composite or multiplexed image from Figure 14A, segmented using the H&E domain analysis results. The image on the left shows an enlarged patch with segmented nuclei in a given channel (e.g., dapi channel 1410) of the multiplexed image. Similarly, the image on the right represents an enlarged patch of the multiplexed image 1420 with segmented results, transferred from the H&E domain. These figures demonstrate that the boundaries identified by processing the virtual composite H&E image appear to map well to the multiplexed domain.

[0091] Therefore, the various techniques disclosed herein can be used to transfer existing tools or models to different imaging modalities or domains. Similar to the embodiments described above, a synthetic virtual H&E image was generated from multiplex images by using CycleGAN. This synthetic virtual H&E image can act as a bridge (domain swap), eliminating the need for staining (e.g., H&E) and processing of serial slices while saving resources (e.g., time, cost, effort). Through the synthetic virtual H&E image, the multiplex domain is transferred to the H&E domain. Subsequently, by leveraging previously developed tools or deep learning algorithms in the H&E domain, tumor and / or stromal tissue segmentation, artifact detection, or cell segmentation can be facilitated, and cell boundaries on the H&E can be well depicted. The disclosed approach improves accuracy and facilitates multiplex algorithm development by transferring results such as tumor mapping and cell segmentation in the multiplex domain. Since the spatial reference point is the same, results, annotations, detections, or segmentations in the H&E domain can be seamlessly transferred to the multiplex domain.

[0092] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-temporary computer-readable storage medium containing instructions that, when executed by one or more data processors, cause one or more data processors to execute some or all of one or more of the methods disclosed herein and / or some or all of one or more processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-temporary machine-readable storage medium, which contains instructions configured to cause one or more data processors to execute some or all of the methods disclosed herein and / or some or all of one or more processes.

[0093] The terms and expressions used are for illustrative purposes only, not limitation, and in using such terms and expressions there is no intention to exclude equivalents or parts of the features shown and described, however it is acknowledged that various modifications are possible within the scope of the invention described in the claims. Accordingly, although the claimed invention is specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be used by those skilled in the art, and such modifications and variations will be considered to fall within the scope of the invention as defined by the appended claims.

[0094] The description presents only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the description of preferred exemplary embodiments presents to those skilled in the art how to realize various embodiments. It will be understood that the function and arrangement of the elements can be varied without departing from the idea and scope described in the appended claims.

[0095] In the following description, specific details are given to provide a comprehensive understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid unnecessarily obscuring the embodiments with excessive detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

Claims

1. A computer implementation method, Accessing a first image from a first domain, wherein the first domain corresponds to one or more specific imaging modalities, and if the first image is a digital pathology image, accessing a first image wherein the first domain further corresponds to one or more specific stains. A virtual composite image is generated by processing the first image using a machine learning model, wherein the virtual composite image is located in a second domain corresponding to a different imaging modality or different staining relative to the first domain. Accessing an image processing tool configured to process images in the second domain, The process involves using the aforementioned image processing tool to process the virtual composite image, thereby generating one or more annotations for the virtual composite image. Transferring one or more annotations of the virtual composite image to the first image, A computer implementation method comprising performing an analysis using the first image and the one or more transmitted annotations.

2. The method according to claim 1, wherein the machine learning model includes cycleGAN.

3. The method according to claim 1, wherein the first domain is a dark-field multiplex and the second domain is H&E.

4. The method according to claim 1, wherein the first domain is immunohistochemistry and the second domain is H&E.

5. The method according to claim 1, wherein the first domain or the second domain is magnetic resonance imaging.

6. The method according to claim 1, wherein the one or more annotations include annotations of tumor, stromal, or artifact regions.

7. The method according to claim 1, wherein the one or more annotations include segmentation of each of the cell aggregates.

8. It is a system, One or more data processors, A non-temporary computer-readable storage medium, which, when executed by the one or more data processors, includes instructions causing the one or more data processors to execute a set of operations, wherein the set of operations is Accessing a first image from a first domain, wherein the first domain corresponds to one or more specific imaging modalities, and if the first image is a digital pathology image, accessing a first image wherein the first domain further corresponds to one or more specific stains. A virtual composite image is generated by processing the first image using a machine learning model, wherein the virtual composite image is located in a second domain corresponding to a different imaging modality or different staining relative to the first domain. Accessing an image processing tool configured to process images in the second domain, The process involves using the aforementioned image processing tool to process the virtual composite image, thereby generating one or more annotations for the virtual composite image. Transferring one or more annotations of the virtual composite image to the first image, A system comprising performing an analysis using the first image and the one or more transmitted annotations.

9. The system according to claim 8, wherein the machine learning model includes cycleGAN.

10. The system according to claim 8, wherein the first domain is a dark-field multiplex and the second domain is H&E.

11. The system according to claim 8, wherein the first domain is immunohistochemistry and the second domain is H&E.

12. The system according to claim 8, wherein the first domain or the second domain is magnetic resonance imaging.

13. The system according to claim 8, wherein the one or more annotations include annotations of tumor, stromal, or artifact regions.

14. The system according to claim 8, wherein the one or more annotations include segmentation of each of the cell aggregates.

15. A computer program product tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to execute a set of operations, wherein the set of operations is Accessing a first image from a first domain, wherein the first domain corresponds to one or more specific imaging modalities, and if the first image is a digital pathology image, accessing a first image wherein the first domain further corresponds to one or more specific stains. A virtual composite image is generated by processing the first image using a machine learning model, wherein the virtual composite image is located in a second domain corresponding to a different imaging modality or different staining relative to the first domain. Accessing an image processing tool configured to process images in the second domain, The process involves using the aforementioned image processing tool to process the virtual composite image, thereby generating one or more annotations for the virtual composite image. Transferring one or more annotations of the virtual composite image to the first image, A computer program product comprising performing an analysis using the first image and the one or more transmitted annotations.

16. The computer program product according to claim 15, wherein the machine learning model includes cycleGAN.

17. The computer program product according to claim 15, wherein the first domain is a dark-field multiplex and the second domain is H&E.

18. The computer program product according to claim 15, wherein the first domain is immunohistochemistry and the second domain is H&E.

19. The computer program product according to claim 15, wherein the first domain or the second domain is magnetic resonance imaging.

20. The computer program product according to claim 15, wherein the one or more annotations include annotations of tumor, stromal, or artifact regions.

Images