Synthetic Singleplex from Multiplex Brightfield Imaging Using Generative Adversarial Networks

A generative adversarial network addresses the challenges of biomarker alignment and estimation in MPX IHC by generating synthetic singleplex images, enhancing accuracy and robustness across diverse contexts.

JP2024544291AActive Publication Date: 2024-11-28VENTANA MEDICAL SYSTEMS INC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2024535862
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-15
Filing Date
2022-12-12
Publication Date
2024-11-28
Estimated Expiration
2042-12-12

AI Technical Summary

Technical Problem

Existing multiplex bright field immunohistochemistry (MPX IHC) imaging methods face challenges in accurately estimating biomarker expression levels and colocalization due to imperfect color separation and alignment of multiple biomarkers, which are context-dependent and require time-consuming parameter tuning.

Method used

A generative adversarial network (GAN), such as CycleGAN or Pix2Pix, is trained using multiplex and singleplex images to generate synthetic singleplex images, eliminating the need for traditional unmixing and alignment algorithms by learning context-specific parameters for stain types, scanners, and sites.

Benefits of technology

The GAN model robustly generates accurate synthetic singleplex images across various contexts, improving the estimation of biomarker expression levels and colocalization without the limitations of traditional methods, even when training data is incomplete or non-existent.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544291000001_ABST
    Figure 2024544291000001_ABST
Patent Text Reader

Abstract

Accessing multiplex images depicting a particular slice of a particular sample stained with two or more dyes. Using a generating network, generating predicted singleplex images depicting a particular slice of a particular sample stained with each of the expressed biomarkers. The generating network may be trained by training a machine learning model using a set of training multiplex images and a set of training singleplex images. Each set of training multiplex images depicted a slice of a sample stained with two or more dyes. Each set of training singleplex images depicted a slice of a sample stained with a single dye. The machine learning model included a discrimination network configured to discriminate as to whether a given image was generated by the generating network or was a singleplex image of an actual slide. The method further includes outputting the predicted singleplex images.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 289,867, filed December 15, 2021, the entire contents of which are incorporated herein by reference for all purposes. [Background technology]

[0002] In digital pathology, it is often important to identify the relative spatial location of multiple different biomarkers. One approach to assessing multiple biomarkers is to stain each slice of a sample with a single chromogen, but alternate which chromogen is used for each slice. In this way, each slide will show a single biomarker, and successive slides will show different biomarkers. To assess the relative location information, a registration step may then be performed in an attempt to align the successive slides, and the biomarker information from the successive slides may then be overlaid on top of each other. However, the registration step may be imperfect, and the biological attributes of the sample may differ in adjacent slices, so that the overlaid biomarkers may present an unrealistic representation.

[0003] Multiplex bright-field immunohistochemistry (MPX IHC) imaging offers the advantage of providing images showing multiple biomarkers on a single slide. In this way, a given image can be used to simultaneously analyze multiple biomarkers (e.g., identify the location of one type of biomarker relative to another). For example, Figure 1A-1C show three slides from three duplex assays in which slices were stained with two biomarkers with a counterstain (hematoxylin). Figure 1A shows a color-corrected duplex ER / PR image from the DP200 for visualization purposes, where the purple signal represents the progesterone receptor (PR) and the yellow signal represents the estrogen receptor (ER). Figure 1B shows a raw image generated by scanning with the DP200, where the purple signal represents PDL1 and the yellow signal represents pan-cytokeratin (PanCK), which is typically used as a tumor marker. FIG. 1C shows the raw image generated by scanning with iscanHT, where the purple signal represents Ki67 and the yellow signal represents CD8 cells.

[0004] Additionally, Figures 1D-1F show slides from a multiplex assay where slices were stained with three biomarkers with counterstain. Figure 1D shows a slide containing ER, PR, and Her2 dyes (Her2 dye stains HER2 protein). Figure 1E shows a slide containing PDL1, cMET, and EGFR dyes (cMET dye stains c-MET protein and EGFR dye stains EGFR protein). Figure 1F shows a slide containing CD8, CD3, and BCL2 dyes (CD8 and CD8 dyes stain CD3, CD8, and BCL-2 proteins, respectively).

[0005] Pathologists score biomarker expression levels by estimating them from the intensity levels appearing on the slide. However, it is a difficult task for pathologists to estimate biomarker expression levels, especially the co-localization of multiple biomarkers, using multiplexed images. Therefore, images of single biomarkers with their corresponding counterstains in multiplexed images are necessary for the pathologist's scoring task. Single biomarker images may be called singleplex images and can be obtained by unmixing multiplexed images and remixing (i.e., reconstructing) the unmixed single biomarkers with the hematoxylin channel into an image called a composite singleplex.

[0006] Color separation can be performed as a pre-processing step that breaks down a multiplexed brightfield image into separate color image channels. The separated color channels of biomarkers can be remixed with a counterstain to generate a composite singleplexed (simplexed) image for pathologists to score or perform automated image analysis. Color separation can use color deconvolution methods to break down an RGB image into its individual component chromogens for each biomarker. However, color separation is generally imperfect. Standard imaging generally has three color channels (e.g., red, green, and blue channels), so in situations where slices are stained with four or more dyes (which may include a first dye staining the nucleus and at least three other dyes staining the other three types of biomarkers), the imperfection of color separation is amplified. This situation can result in a huge number of color separation solutions.

[0007] Figure 2 outlines some of the advantages and disadvantages of two techniques that use singleplex images to analyze multiple biomarkers. Singleplex images can be obtained: (I) by staining individual slices with multiple stains, performing color separation, and then remixing to generate a composite singleplex image for detecting distinct signals, and (II) by staining adjacent slices, each with a single stain, to generate multiple actual singleplex stained images. Figure 3 illustrates variables that can affect the performance of unmixing or registration algorithms, such as multiple staining protocols, scanners, sites, etc.

[0008] The first approach, based on unmixing and remixing, results in perfect tissue matching, reduces the required tissue (compared to the adjacent slide approach) and does not require registration, but unmix parameter values ​​must be identified. Parameter values ​​are also context specific, so different parameter values ​​are likely to be required for the following examples: Different staining protocols are used Different tissue types are evaluated -Different scanners are used Equipment from different manufacturers is used The samples are processed at a different site, and / or -Different pre-analysis conditions exist

[0009] Unlike unmixing, adjacent staining techniques can be affected by the performance of the registration protocol when performing tissue analysis on multiplex images. Although the biomarker intensities in adjacent slides are actual biological stains, a registration algorithm is required to align tissue regions with the multiplex images for tissue analysis in order to locate the same tissue regions for each singleplex image. However, a given registration algorithm may perform well when used for a first type of tissue and poorly when used for a second type of tissue. Thus, registration parameter values ​​may be trained for multiple contexts, but this effort is time-consuming and expensive. Without training separate parameter values ​​for different contexts, the algorithm may be non-robust and inaccurate. Summary of the Invention

[0010] In some embodiments, a computer-implemented method is provided that includes accessing multiplex images depicting a particular slice of a particular sample stained with two or more dyes (e.g., two or more chromogens) and using a generative network to generate predicted singleplex images depicting a particular slice of a particular sample stained with only one of the two or more dyes. The generative network may have been trained by training a machine learning model using a set of training multiplex images and a set of training singleplex images, each of which depicts a slice of a sample stained with two or more dyes and each of which depicts a slice of a sample stained with a single dye. The machine learning model included a discrimination network configured to discriminate as to whether a given image was generated by the generative network or was a singleplex image of an actual slide. The method further includes outputting the predicted singleplex images.

[0011] Each of the set of training singleplex images may be a composite image generated by processing a corresponding training multiplex image from the set of training multiplex images using an unmix and remix algorithm configured for the context from which the corresponding training multiplex image was obtained.

[0012] The machine learning model may include a Pix2Pix model or a BicycleGAN.

[0013] Each of the set of training singleplex images may be a real image depicting a corresponding slice that was not depicted in any of the set of training multiplex images.

[0014] The machine learning model may include CycleGAN, which included a separate generative network configured to generate a predicted multiplex image for each received singleplex image, and a separate discrimination network configured to discriminate as to whether a given image was generated by the separate generative network or was a multiplex image of an actual slide.

[0015] The method may further include performing training of a machine learning model prior to generating the predicted singleplex image.

[0016] The multiplex image may have been generated at the first site using a first scanner, and the method may further include accessing another multiplex image depicting another specific slice of another specific sample stained with the two or more dyes, generating another predicted singleplex image depicting another specific slice stained with only one of the two or more dyes using a generative network configured with the same parameter values ​​at the time the predicted singleplex image was generated and at the time the another predicted singleplex image was generated, and outputting the another predicted singleplex image.

[0017] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0018] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0019] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0020] The terms and expressions used are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features or portions thereof shown and described, recognizing that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be reclassified by those skilled in the art, and such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0021] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of the various embodiments will become more apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings, in which: FIG. [Brief description of the drawings]

[0023] [Figure 1A] FIG. 1 shows slides from a multiplex assay where tissue slides were stained with two dyes. [Figure 1B] FIG. 1 shows slides from a multiplex assay where tissue slides were stained with two dyes. [Figure 1C] FIG. 1 shows slides from a multiplex assay where tissue slides were stained with two dyes. [Figure 1D] FIG. 1 shows slides from a multiplex assay where tissue slides were stained with two dyes. [Figure 1E] FIG. 1 shows slides from a multiplex assay where tissue slides were stained with two dyes. [Figure 1F] FIG. 1 shows slides from a multiplex assay where tissue slides were stained with two dyes. [Diagram 2] FIG. 1 provides an overview of some of the advantages and disadvantages of two techniques for analyzing multiple biomarkers. [Diagram 3] FIG. 1 illustrates a diagram showing the context that can affect the performance of an unmixing or alignment algorithm. [Figure 4] FIG. 1 shows an exemplary network for generating a composite singleplex image, where the image is generated by an image generation system. [Diagram 5] FIG. 1 illustrates the architecture and training of CycleGAN. [Figure 6] FIG. 1 shows another exemplary network for generating a composite singleplex image. [Figure 7] FIG. 1 illustrates an example Pix2Pix GAN model that the model training system may train. [Figure 8] FIG. 1 shows an example input image, a predicted singleplex image by the generative network, and a comparison singleplex image generated by the unmixing algorithm. [Figure 9] FIG. 13 shows an additional exemplary input image, a predicted singleplex image by the generative network, and a comparison singleplex image generated by the unmixing algorithm. [Figure 10] FIG. 13 shows an additional exemplary input image, a predicted singleplex image by the generative network, and a comparative singleplex image obtained by imaging an adjacent slide stained with a single dye (e.g., a biomarker with a counterstain). [Figure 11] FIG. 1 shows two comparisons of the input multiplex IHC image (left side of pair) and the predicted composite singleplex image for each biomarker in the multiplex IHC image. [Figure 12A] FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 12B]FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 12C] FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 12D] FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 13] FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. [Figure 14] FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. [Figure 15] FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. [Figure 16A] FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 16B] FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 16C] FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 16D] FIG. 1 shows exemplary images demonstrating the performance of two techniques for generating a composite singleplex image. [Figure 17] FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. [Figure 18] FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. [Figure 19]FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. [Figure 20] FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. [Figure 21] FIG. 13 shows example images demonstrating the performance of a network trained using the CycleGAN technique to generate synthetic singleplex images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. If only a first reference label is used herein, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label.

[0025] In some embodiments, a machine learning model is trained or has been trained and used to generate a composite singleplex image from multiplex (MPX) immunohistochemistry (IHC) images. Thus, neither traditional unmixing algorithms (e.g., based on deconvolution) nor alignment are required to identify and evaluate the relative spatial location of multiple biomarkers. While traditional unmixing algorithms fall short when applied to different protocols, tissues, subjects, cancers, or sites, the machine learning models identified herein can robustly generate singleplex images across different contexts.

[0026] The machine learning model may include a generator that receives a real image and generates a predicted singleplex image. The real image may be a multiplex image, or may be a remix image or an image corresponding to adjacent slices from an unmixing algorithm. The generator may be configured with parameter values ​​learned as a result of training a larger model (e.g., including a discriminator), such as a Pix2Pix model, Pix2PixHD, or a GAN model (e.g., a CycleGAN model or BicycleGAN). The generative network may be specific to the stain used for the input image, but may be general enough to apply (for example) across different subjects, across different protocols for applying the stain, across different tissue types, across different equipment parts (e.g., individual scanners), across different equipment manufacturers, across different sites where the slides were made, and / or across different pre-analysis conditions.

[0027] FIG. 4 illustrates an exemplary network for generating a composite singleplex image. The image is generated in an image generation system 405. A fixation / embedding system 410 fixes and / or embeds a tissue sample (e.g., a sample including at least a portion of at least one tumor) using a fixative (e.g., a liquid fixative such as a formaldehyde solution) and / or an embedding substance (e.g., a histology wax such as paraffin wax, and / or one or more resins such as styrene or polyethylene). Each slice may be fixed by exposing the slice to a fixative for a predetermined period of time (e.g., at least 3 hours) and then dehydrating the slice (e.g., via exposure to an ethanol solution and / or a clearing intermediate agent). The embedding substance can penetrate the slice when in a liquid state (e.g., when heated).

[0028] A tissue slicer 415 then slices the fixed and / or embedded tissue sample (e.g., a tumor sample) to obtain a series of sections, each having a thickness of, for example, 4-5 microns. Such sectioning may be performed by first chilling the sample and then slicing the sample in a warm water bath. The tissue may be sliced ​​using (for example) using a vibratome or compressome.

[0029] Because tissue sections and the cells therein are largely transparent, slide preparation typically includes staining (e.g., automatically staining) the tissue sections to make relevant structures more visible. In some instances, staining is performed manually. In some instances, staining is performed semi-automatically or automatically using staining system 420.

[0030] Staining can include exposing individual sections of tissue to one or more different stains (e.g., sequentially or simultaneously) to reveal different properties of the tissue. For example, each section can be exposed to a predetermined amount of stain for a predetermined period of time. A duplex assay includes a procedure in which a slide is stained with two biomarker stains. A singleplex assay includes a procedure in which a slide is stained with a single biomarker stain. A multiplex assay includes a procedure in which a slide is stained with two or more biomarker stains. A triplex assay includes a procedure in which a slide is stained with three biomarker stains (e.g., nuclear staining biomarkers). In either singleplex, duplex, triplex, or multiplex assays, the slide can be further stained with a stain (e.g., Heme stain) that is absorbed by the cell nuclei.

[0031] One exemplary type of tissue stain is a histochemical stain, which uses one or more chemical dyes (e.g., acid dyes, basic dyes, chromogens) to stain tissue structures. Histochemical stains may be used to show general aspects of tissue morphology and / or cellular microanatomy (e.g., to distinguish cell nuclei from cytoplasm, show lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), periodic acid Schiff (PAS), silver stains, and iron stains. The molecular weight of histochemical stains (e.g., dyes) is generally about 500 kilodaltons (kD) or less, although some histochemical stains (e.g., Alcian blue, phosphomolybdic acid (PMA)) may have a molecular weight of up to 2000 or 3000 kD. An example of a high molecular weight histochemical stain is α-amylase (approximately 55 kD) which may be used to demonstrate glycogen.

[0032] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that specifically binds to a target antigen of interest (also called a biomarker). IHC may be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, a primary antibody is first bound to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The molecular weight of the IHC reagents is much larger than that of the histochemical staining reagents, as antibodies have a molecular weight of about 150 kD or more.

[0033] The sections may then be individually mounted on corresponding slides, which the imaging system 425 may then scan or image to generate raw digital pathology images 430a-n. In some cases, adjacent slides are stained with different amounts of stains. For example, every other slide may contain a sample stained only with a first specific stain, or only with a second specific stain (so that imaging the slides produces singleplex images 430a-n), while the remaining slides may each contain a sample stained with both a first specific dye and a second specific dye, such that the multiplex images 435a-n produced when the slides are imaged are duplex images. As another example, every fourth slide from the sample may be stained with three biomarker dyes (so that imaging the slides produces multiplex images 435a-n) and the slide separating these slides of three biomarkers may be stained with only one of the biomarker dyes (e.g., possibly a dye that is absorbed by cell nuclei). It will be appreciated that in addition to the biomarker dye, a counterstain may be present that is used as a location reference. For example, the counterstain may include a stain configured to be absorbed by the cell nucleus (e.g., HTX) or cell membrane.

[0034] In some cases, the singleplex images 430a-n and multiplex images 435a-n may correspond to different samples rather than to adjacent slides. For example, the singleplex images 430a-n may depict slides from one or more first samples stained with either only a first particular dye or only a second particular dye, and the multiplex images 435a-n may depict slides from one or more second samples stained with both the first and second particular dyes.

[0035] In either situation, it is recognized that the singleplex images 430a-n and duplex images 435a-n are actual images that depict actual slides.

[0036] The model training system 445 can use the singleplex images 430a-n and at least some of the multiplex images 435a-n to train a machine learning model (e.g., including a convolutional machine learning model, one or more convolutional layers, U-Net, V-Net, modified U-Net, modified V-Net, etc.) The singleplex images 430a-n and multiplex images 435a-n used to train the machine learning model may include paired and / or registered images, but advantageously are not necessarily those images.

[0037] The machine learning model may include a generative adversarial network (GAN), such as CycleGAN or BicycleGAN. Figure 5 illustrates the architecture and training of a CycleGAN. A GAN may include one or more models having a corresponding model architecture 450. Each of the one or more models may include one or more convolutional layers.

[0038] The GAN includes one or more generative networks including a generator configured to receive one of the multiplex images 435a-n (or a predicted duplex image) and generate a predicted singleplex image. In some cases (e.g., when the machine learning model is CycleGAN), the one or more generative networks further include a generator configured to receive one of the singleplex images 430a-n (or a predicted singleplex image) and generate a predicted duplex image. Each of the one or more generative networks may include (e.g.) a neural network, a deep neural network, a residual neural network, and / or a convolutional neural network (e.g., or a deep residual convolutional neural network, ResNet, UNET, forward propagation network).

[0039] The GAN further includes one or more discrimination networks. Each of the one or more discrimination networks may include (for example) a neural network, a PatchGAN, a deep neural network, and / or a convolutional neural network (e.g., a deep residual convolutional neural network). In some cases, the discrimination network has the same architecture as the corresponding generative network, but in other examples, the architectures are different.

[0040] One of the one or more discrimination networks can be configured to predict whether the predicted singleplex image or singleplex image 430 is a real image (e.g., generated by the image generation system 405) or a predicted image. Another of the one or more discrimination networks can be configured to predict whether the predicted duplex image or duplex image 430 is a real image (e.g., generated by the image generation system 405) or a predicted image. Although only one image generation system 405 is depicted, it will be appreciated that images used to train the model may be generated by multiple image generation systems 405 and / or images processed using the trained generative network may be generated by multiple (different, overlapping, non-overlapping, or the same) image generation systems 405. The different image generation systems 405 may be located (for example) at different sites (e.g., different addresses, cities, etc.).

[0041] The loss (calculated by the model training system 445 and used by the model training system 445 to update the generator parameter values ​​455 and the discriminator parameter values ​​460) may be calculated to depend on a cycle consistency loss, which quantifies how different an original (e.g., singleplex or duplex) image is from a corresponding image processed by two generators. For example, the cycle consistency loss may characterize how different an actual duplex image is from a predicted duplex image generated by one or more first generators converting the actual duplex image into multiple predicted singleplex images, which are then converted by a second generator into predicted duplex images.

[0042] The loss may also, or alternatively, depend on the accuracy of the predictions produced by each of the one or more discrimination networks.

[0043] Once the GAN is trained (e.g., loss falls below a threshold, a prescribed number of training iterations is completed, etc.), a synthetic singleplex generator 465 converts non-training multiplex images 435 (generated by the image generation system) into synthetic singleplex images 470 using the architecture and learned parameter values ​​for the generator configured to convert multiplex images into one or more singleplex images. That is, after the generator is trained (e.g., via training a machine learning model such as CycleGAN), the generator may be separated from the machine learning model and used independently to convert multiplex images into synthetic singleplex images.

[0044] It will be appreciated that the singleplex images 430a-n may include images of slides stained with the same biomarker dye (and counterstain dye, such as a counterstain dye configured to be absorbed by cell nuclei or cell membranes), and the parameter values ​​learned during training may apply to one particular dye of the multiple dyes used to stain the multiplex images 435a-n. In this case, a different set of singleplex images depicting slides stained with different biomarker dyes may then be accessed, and the model training system 445 may then train a model using the model architecture 450 to learn different generator parameter values ​​and different discriminator parameter values. That is, a separate training process may be applied to each of the multiple biomarker dyes depicted in the multiplex images 435a-n, resulting in different generators converting the multiplex images into predicted singleplex images depicting different dyes. To illustrate, in an example where multiplex images 435a-n are triplex images, there may be three different sets of singleplex images 430a-n (corresponding to four different biomarker dyes, including a counterstain dye), which may be used for three independent training processes to create three different generators.

[0045] Figure 6 shows another exemplary network for generating a composite singleplex image. Elements in Figure 6 having the same or similar reference numbers as those in Figure 4 may include the same or similar structures and may implement the same or similar aspects.

[0046] In the illustrated example, during the training phase, the unmixing system 670 uses a conventional unmixing algorithm to generate one or more remixed images 675a-n for each of several multiplex images 635a-n (representing slides stained with two or more or three or more stains). For example, if a multiplex image represents a slide stained with three biomarker stains, the unmixing system 670 may output a single image representing a predicted image of the slide when stained with one of the three biomarker stains, two images each representing a predicted image of the slide when stained with two of the three biomarker stains, or three images each representing a predicted image of the slide when stained with three of the three biomarker stains. That is, the remixed images 675a-n may include images of predicted singleplex slides corresponding to one, more than one, or all of the stains used to prepare the corresponding multiplex image.

[0047] For each image processed, the unmixing system 670 can use an algorithm trained specifically for the context in which the image was acquired. For example, an unmixing algorithm can be selected based on the type of tissue, the type of stain, the location of the site, the piece of equipment used in the corresponding imaging system, etc.

[0048] The unmixed model may use deconvolution techniques, such as those specified in Ruifrok et al., "Quantification of histochemical staining by color deconvolution," Anal Quant Cytol Histol 23:291-299, 2001, which is incorporated by reference in its entirety for all purposes. The unmixed model may alternatively or additionally use non-negative matrix factorization, such as those specified in Miao et al., "Endmember Extraction from Highly Mixed Data Using Minimum Volume Constrained Non-Negative Matrix Factorization," IEEE Transactions on Geoscience and Remote Sensing, vol. 45, no. 3, pp. 765-777, March 2007, doi:10.1109 / TGRS.2006.888466, which is incorporated by reference in its entirety for all purposes.

[0049] The multiplexed image can be paired with one or more predicted singleplexed images. The model training system 645 can use the paired images to train a machine learning model (e.g., a convolutional machine learning model, including one or more convolutional layers, U-Net, V-Net, modified U-Net, modified V-Net, etc.).

[0050] In some cases, the machine learning model includes a conditional adversarial network or Pix2Pix GAN model (i.e., Pix2PixHD) and / or a model configured to perform downsampling followed by upsampling. Figure 7 illustrates an example Pix2Pix GAN model that the model training system 645 may train. A conditional adversarial network or Pix2Pix GAN may include one or more models having a corresponding model architecture 450. Each of the one or more models may include one or more convolutional layers.

[0051] The conditional adversarial network or Pix2Pix GAN includes a generative network including a generator configured to receive one of the multiplex images 635a-n (or a predicted duplex image) and generate a predicted singleplex image. The conditional adversarial network or Pix2Pix GAN can include a downsampling layer and an upsampling layer. For example, the generator can include a U-Net, a V-Net, a modified U-Net, or a modified V-Net.

[0052] The machine learning model may include one or more generative networks and / or one or more discriminatory networks. Each generative network may be configured and trained to receive an image including a representation of a specimen stained with two or more specific stains and generate a predicted image of the specimen stained with only one of the two or more specific stains (another generative network may be configured and trained to receive an image including a representation of a specimen stained with two or more specific stains and generate a predicted image of the specimen stained with only another one of the two or more specific stains). Similarly, each discriminatory network may be trained and configured to predict whether a given image that truly depicts or is predicted (by the generator) to depict only a specific stain is real or fake. Thus, this approach can be used to support generating synthetic singleplex images based on true triplex or N-plex input images.

[0053] The machine learning model may be configured to be trained using paired images. An image may depict a slide in which a sample has been stained with at least two stains or at least three stains; Each of the one or more other images includes an unmixed image (generated for a particular context) that is predicted to depict a single one of the at least two or at least three stains using a context-specific unmixed model.

[0054] The discriminatory network can be configured to predict whether a given image is a fake image generated by the generative network or a real image in the training dataset. The discriminatory network can include a convolutional network and / or one or more convolutional layers.

[0055] The loss (computed by the model training system 645 and used by the model training system 645 to update the generator parameter values ​​655 and the discriminator parameter values ​​660) may be calculated to depend on the accuracy of the predictions generated by the discriminatory network.

[0056] Once the Pix2Pix model is trained (e.g., the loss falls below a threshold, a prescribed number of training iterations is completed, etc.), a synthetic singleplex generator 665 converts the non-training duplex image 635 (generated by the image generation system) into a synthetic singleplex image 665 using the architecture and learned parameter values ​​for the generator configured to convert the multiplex image into one or more singleplex images.

[0057] It will be appreciated that variations of the disclosed embodiments are contemplated. For example, an augmentation of the input image can be applied by perturbing the unmixed image, e.g., the yellow and purple channels, and then the resulting image can be remixed back into a new (i.e., augmented) duplex. The new remixed image can be used as the input (source) image, and the machine learning model can be trained using the perturbed simplex (the synthetic simplex with perturbation in addition to the traditional unmix) as a target. In this way, variations or data augmentation can be introduced into the training data without acquiring additional real data. These variations of the training data can improve the robustness of the deep learning model, and the GAN model can handle more variations of the input image. As a result, it may be unnecessary to generate unmix parameters for each single data set as ground truth data for training the machine learning model (e.g., Pix2Pix model).

[0058] As another example, embodiments may be extended to use similar training and modeling approaches to accurately map biomarker expression levels, i.e., the intensity of marker signals in synthetic singleplexes, as a true representation of biomarker expression, as measured by the intensity of the unmixed image. Additionally, quantified metrics can be used to assess colocalization and biomarker expression levels to improve the robustness of the proposed approach.

[0059] As described and demonstrated herein, the techniques disclosed herein for training and using one or more generative networks have multiple technical advantages. The techniques are robust to a wide range of situations and do not require a specific staining solution, a specific staining protocol, tissue type, scanner, etc., given that the models may be trained using any of a variety of image sets. Furthermore, unlike unmixing and remixing approaches, the techniques disclosed herein can be used in cases where the dyes used to stain the slices do not need to be pure reference colors. In addition, the models disclosed herein can be trained even when the alignment between singleplex and multiplex images in the training set is incomplete, poor, or nonexistent. Finally, the techniques disclosed herein can be used to generate synthetic singleplex images, even when the multiplex images depict multiple biomarker dyes (e.g., three biomarker dyes, four biomarker dyes, five biomarker dyes, or more). On the other hand, the accuracy of other existing techniques for generating composite singleplex images (e.g., non-negative factorization or singular value decomposition) plummets when more than two biomarker dyes are depicted in a multiplex image, given that there can be thousands of combinations of singleplex signals that would result in a given multiplex image. Additional regularization can reduce the number of possible solutions, but such regularization is case-specific (not generalizable across workflows, instruments, and assays) and insufficient to generate highly accurate predictions. EXAMPLES

[0060] Example 1 The Pix2Pix network was trained using MPX IHC / singleplex paired images unmixed with a combination of data from multiple assays, scanners, and cancer indications and subjects. The following data was used for training to generate each GAN in the corresponding Pix2Pix model: - 2 slides duplex ER / PR scanned with DP200 with a total of 1,000 patches of patch size 128x128 - 2 slides duplex PDL1 / PanCK with a total of 1,000 patches of patch size 128x128 scanned with a DP200 - 10 duplex Ki67 / CD8 slides from 10 patients with a total of 1,000 patches of patch size 128x128 scanned with iScanHT

[0061] 8 and 9 show an input image (first column), a predicted singleplex image for one stain (second column), a "true" singleplex image for one stain identified using a context-specific unmixing algorithm (third column), a predicted singleplex image for another stain (fourth column), and a "true" singleplex image for the other stain identified using a context-specific unmixing algorithm (fifth column). The three columns correspond to different stains (and thus different generator models). As shown, the predicted singleplex images are realistic and highly similar to the "true" singleplex images across test images including different biomarkers, scanners, cancer indications, and subjects.

[0062] These results indicate that using standard unmixing techniques, at least three different image analysis algorithms (i.e., ER / PR, PDL1 / PanCK, and Ki67 / CD8) are required to generate those distinct unmixed images. However, using one GAN deep learning network (trained as part of the Pix2Pix model) with the two models, predicted images of very similar quality were generated.

[0063] Example 2 The CycleGAN network was trained using unpaired images of MPX IHC / singleplex adjacent slide images from multiple assays and multiple subjects. The following data was used to train each GAN model as part of the CycleGAN network: - 2 slides duplex ER / PR scanned with DP200, with a total of 510 patches with patch size 128x128 - 2 slides duplex PDL1 / PanCK with a total of 550 patches of patch size 128x128 scanned with a DP200

[0064] In Figure 10, the first column shows example input images fed to a first generator trained as part of a CycleGAN. The second column shows synthetic singleplex images generated by the first generator. The third column shows true singleplex images corresponding to slides adjacent to those input images. Columns 4-6 show data corresponding to the same type as in columns 1-3, but for a different generator (trained as part of a different CycleGAN, corresponding to a different dye, but to generate synthetic images). As can be seen, the predicted images are very similar to the synthetic images.

[0065] Figure 11 shows a comparison of the input multiplex IHC image (left side of the pair) and two predicted composite singleplex images for each biomarker that may be present in the input multiplex IHC image. The image on the left corresponds to a different dye (in terms of prediction) than the image on the right.

[0066] Thus, the output images show realistic and comparable target unmixed images for all test images of different biomarkers, cancer indications, and patients. The output composite image shows aligned structures compared to the input MPX IHC images, and the colors (e.g., yellow, purple) of the output images show similarities to the reference adjacent slides. These results demonstrate that the unpaired image conversion method works effectively to generate both aligned structures and colors of the composite image.

[0067] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0068] The terms and expressions used are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features or portions thereof shown and described, but it is recognized that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be employed by those skilled in the art, and such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0069] The description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the description of the preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0070] In the following description, specific details are given to provide a comprehensive understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0071] Example 3 Assays were performed on multiple samples (36 slides total from 3 subjects) each. The dyes used to stain the slides were PDL1 (TAMRA, membrane), cMET (green, membrane), EGFR (Dabsyl, membrane) and Heme (nucleus). In this way, triplex slides were generated.

[0072] The CycleGAN network was trained three times independently using unpaired images of triplex IHC and singleplex adjacent slide images from multiple assays and multiple subjects corresponding to one of three specific training dyes. The triplex images contained depictions of PDL1, cMET, and EGFR dyes.

[0073] The CycleGAN network included two generative networks with a deep residual convolutional neural network architecture, one configured to convert triplex images to singleplex images and the other configured to convert singleplex images to triplex images, and two discriminative networks with a PatchGAN architecture, one configured to convert singleplex images to triplex images and the other configured to convert triplex images to singleplex images. Each training data image was 256 × 256 pixels in size and corresponded to a patch of a digital pathology slide. During training, an Adam optimizer with a learning rate of 0.0002 was used. We defined a loss function with different loss weights for the generator set to 1, 5, and 10 for adversarial, self, and cycle consistency, respectively. A batch size of 8 was used.

[0074] After training the CycleGAN network for each of the three dyes, another triplex image was then processed using a generator configured to generate a synthetic singleplex image to generate a synthetic singleplex image corresponding to the dye. Figures 12A and 12B show an example triplex image and a synthetic singleplex image for a green dye generated by the generator. The lower portion of Figure 12B further shows a magnified portion of the synthetic singleplex image.

[0075] Actual composite singleplex images from adjacent slides were accessed for comparison purposes (see Figure 12C).

[0076] In addition, non-negative factorization (NMF) with additional regularization was used to generate comparative synthetic singleplex images for the dyes (see Figure 12D).

[0077] The arrow pointing down to the left points to the membrane stain. The arrow pointing up to the left points to the nuclear stain.

[0078] We see that the staining levels of the synthetic singleplexes generated by the generator trained as part of CycleGAN correspond better to the levels of the real adjacent slides than the synthetic singleplexes generated by the NMF technique, which have lower staining levels than those present in the real slides.

[0079] In addition, the synthetic singleplex images generated by the generator trained as part of CycleGAN depict nuclei with high confidence. However, due to the customization required by the NMF technique to reduce stain separation errors, the nuclear stains are first separated from the triplex to reduce the number of stains in the image to three, and then returned to the synthetic singleplex due to the uniqueness of the stain separation results by NMF, which partially leads to the loss of nuclear signal (consider the missing left-upward arrow in Figure 12D).

[0080] Furthermore, in synthetic singleplex images generated by a generator trained as part of CycleGAN, the membrane staining is clear and corresponds to that of the triplex images. However, in synthetic singleplexes generated by the NMF technique, the membrane staining is less clear and corresponds weakly to that of the triplex images.

[0081] FIG. 13 shows another example of a synthetic singleplex Green-cMET image generated by transforming a triplex PDL1-cMET-EGFR image using a generator trained as part of a CycleGAN network. FIG. 14 shows another example of a synthetic singleplex QM-Dabsyl-EGFR image generated by transforming a triplex PDL1-cMET-EGFR image using a generator trained as part of a CycleGAN. FIG. 15 shows another example of a synthetic singleplex Tamra-PDL1 image generated by transforming a triplex PDL1-cMET-EGFR image using a generator trained as part of a CycleGAN. Each of FIGS. 13-15 further shows an example of an actual triplex and singleplex image used to test the network. It can be seen that the color intensity, sharpness, and texture of the synthetic singleplex image generated using the generator trained as part of a CycleGAN is very similar to that of the corresponding image.

[0082] Example 4 Assays were performed on multiple samples (a total of 36 slides from three subjects) for each. The dyes used to stain the slides were CD8 (TAMRA, membrane), Bcl2 (green, membrane), CD3 (Dabsyl, membrane) and Heme (nuclear). In this way, triplex slides were generated.

[0083] The CycleGAN network was trained three times independently using unpaired images of triplex IHC and singleplex adjacent slide images corresponding to one of three specific training dyes from multiple assays and multiple subjects. The triplex images contained depictions of CD8, Bcl2, and CD3 dyes.

[0084] The CycleGAN network included two generative networks with a deep residual convolutional neural network architecture, one configured to convert triplex images to singleplex images and the other configured to convert singleplex images to triplex images, and two discriminative networks with a PatchGAN architecture, one configured to convert singleplex images to triplex images and the other configured to convert triplex images to singleplex images. Each training data image was 256 × 256 pixels in size and corresponded to a patch of a digital pathology slide. During training, an Adam optimizer with a learning rate of 0.0002 was used. We defined a loss function with different loss weights for the generator set to 1, 5, and 10 for adversarial, self, and cycle consistency, respectively. A batch size of 8 was used.

[0085] After training the CycleGAN network for each of the three dyes, the generator configured to generate synthetic singleplex images was then used to process the separate triplex images to generate synthetic singleplex images corresponding to the dyes.

[0086] 16A and 16B show an exemplary triplex image and a composite singleplex image for green dye produced by the generator.

[0087] Actual composite singleplex images from adjacent slides were accessed for comparison purposes (see Figure 16C).

[0088] In addition, non-negative factorization (NMF) with additional regularization was used to generate comparative synthetic singleplex images for the dyes (see Figure 16D).

[0089] It can be seen that the staining levels of the synthetic singleplexes generated by the generator trained as part of CycleGAN correspond to the levels of the real adjacent slides better than the synthetic singleplexes generated by the NMF technique. The staining levels of the synthetic singleplexes generated by the NMF technique are lower than the levels present in the real slides. Furthermore, the membranes of the synthetic singleplexes generated by the generator trained as part of CycleGAN are sharper than those of the synthetic singleplexes generated by the NMF technique.

[0090] FIG. 17 shows another example of a synthetic singleplex Green-Bcl2 image generated by transforming a triplex CD8-Bcl2-CD3 image using a generator trained as part of a CycleGAN. FIG. 18 shows another example of a synthetic singleplex Dabsyl-CD3 image generated by transforming a triplex CD8-Bcl2-CD3 image using a generator trained as part of a CycleGAN. FIG. 19 shows another example of a synthetic singleplex Tamra-CD8 image generated by transforming a triplex CD8-Bcl2-CD3 image using a generator trained as part of a CycleGAN. Each of FIGS. 17-19 further shows exemplary actual triplex and singleplex images used to test the network. It can be seen that the color intensity, sharpness, and texture of the synthetic singleplex images generated using the generator trained as part of a CycleGAN are very similar to those of the corresponding images.

[0091] Example 5 Assays were performed on multiple samples. The dyes used to stain the slides were PR (TAMRA), Her2 (green), ER (Dabsyl), and Heme (nuclear). In this way, triplex slides were generated.

[0092] The CycleGAN network was trained three times independently using unpaired images of triplex IHC and singleplex adjacent slide images from multiple assays and multiple subjects corresponding to one of three specific training dyes. The triplex images contained depictions of PR, Her2, and ER dyes.

[0093] The CycleGAN network included two generative networks with a deep residual convolutional neural network architecture (one configured to convert triplex images to singleplex images and the other configured to convert singleplex images to triplex images) and two discriminative networks with a PatchGAN architecture (one configured to convert singleplex images to triplex images and the other configured to convert triplex images to singleplex images).

[0094] After training the CycleGAN network for each of the three dyes, the generator configured to generate synthetic singleplex images was then used to process the separate triplex images to generate synthetic singleplex images corresponding to the dyes.

[0095] FIG. 20 shows another example of a triplex image, a synthetic singleplex Green-Her2 image generated by transforming a triplex ER-PR-Her2 image using the generator trained as part of CycleGAN, and an adjacent slice stained with Her2 dye.

[0096] FIG. 21 shows another example of a triplex image, a synthetic singleplex Tamra-PR image generated by transforming a triplex ER-PR-Her2 image using a generator trained as part of CycleGAN, and an adjacent slice stained with PR dye.

[0097] It can be seen that the color intensity, sharpness, and texture of the composite singleplex image are very similar to those of the corresponding images.

Claims

1. accessing a multiplexed image depicting a particular slice of a particular sample stained with two or more dyes; generating a predicted singleplex image using a generative network that depicts the particular slice of the particular sample stained with only one of the two or more dyes; the generative network is trained by training a machine learning model using a set of training multiplex images and a set of training singleplex images, each of the sets of training multiplex images depicting slices of a sample stained with two or more dyes, and each of the sets of training singleplex images depicting slices of a sample stained with a single dye; The machine learning model included a discrimination network configured to discriminate as to whether a given image was generated by the generation network or was a singleplex image of an actual slide. generating a predicted singleplex image; outputting the predicted singleplex image; 4. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein each of the set of training singleplex images was a synthetic image generated by processing a corresponding training multiplex image in the set of training multiplex images using an unmix or remix algorithm configured for the context from which the corresponding training multiplex image was obtained.

3. The computer-implemented method of claim 1 , wherein the machine learning model included a Pix2Pix model.

4. 2. The computer-implemented method of claim 1, wherein each of the set of training singleplex images was a real image depicting a corresponding slice not depicted in any of the set of training multiplex images.

5. The machine learning model includes CycleGAN or BicycleGAN, and the CycleGAN or BicycleGAN is another generation network configured to generate a predicted multiplex image for each received singleplex image; a separate discrimination network configured to discriminate as to whether a given image was generated by said separate generation network or was a multiplex image of an actual slide; The computer-implemented method of claim 1 , comprising:

6. 2. The computer-implemented method of claim 1, further comprising performing the training of the machine learning model prior to generating the predicted singleplex image.

7. The multiplexed image is generated using a first scanner at a first site, the method further comprising: accessing another multiplexed image depicting another particular slice of another particular sample stained with the two or more dyes; generating, using the generative network, another predicted singleplex image depicting the other particular slice stained with only one of the two or more dyes, the generative network generating the another predicted singleplex image configured with the same parameter values ​​at the time the predicted singleplex image was generated and at the time the another predicted singleplex image was generated; outputting said another predicted singleplex image; The computer-implemented method of claim 1 , comprising:

8. one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on said one or more data processors, cause said one or more data processors to perform a set of operations; wherein the set of operations comprises: accessing a multiplexed image depicting a particular slice of a particular sample stained with two or more dyes; generating a predicted singleplex image using a generative network that depicts the particular slice of the particular sample stained with only one of the two or more dyes; the generative network is trained by training a machine learning model using a set of training multiplex images and a set of training singleplex images, each of the sets of training multiplex images depicting slices of a sample stained with two or more dyes, and each of the sets of training singleplex images depicting slices of a sample stained with a single dye; The machine learning model included a discrimination network configured to discriminate as to whether a given image was generated by the generation network or was a singleplex image of an actual slide. generating a predicted singleplex image; outputting the predicted singleplex image; Including, the system.

9. 10. The system of claim 8, wherein each of the set of training singleplex images was a synthetic image generated by processing a corresponding training multiplex image from the set of training multiplex images using an unmix or remix algorithm configured for the context from which the corresponding training multiplex image was obtained.

10. The system of claim 8 , wherein the machine learning model included a Pix2Pix model.

11. 9. The system of claim 8, wherein each of the set of training singleplex images was a real image depicting a corresponding slice not depicted in any of the set of training multiplex images.

12. The machine learning model includes CycleGAN or BicycleGAN, and the CycleGAN or BicycleGAN is another generation network configured to generate a predicted multiplex image for each received singleplex image; a separate discrimination network configured to discriminate as to whether a given image was generated by said separate generation network or was a multiplex image of an actual slide; and The system of claim 8, comprising:

13. 10. The system of claim 8, wherein the set of operations further comprises performing the training of the machine learning model prior to generating the predicted singleplex image.

14. the multiplexed image is generated using a first scanner at a first site, and the set of operations further comprises: accessing another multiplexed image depicting another particular slice of another particular sample stained with the two or more dyes; generating, using the generative network, another predicted singleplex image depicting the other particular slice stained with only one of the two or more dyes, the generative network generating the another predicted singleplex image configured with the same parameter values ​​at the time the predicted singleplex image was generated and at the time the another predicted singleplex image was generated; outputting said another predicted singleplex image; The system of claim 8 , comprising:

15. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform a set of operations, the set of operations comprising: accessing a multiplexed image depicting a particular slice of a particular sample stained with two or more dyes; generating a predicted singleplex image using a generative network that depicts the particular slice of the particular sample stained with only one of the two or more dyes; the generative network is trained by training a machine learning model using a set of training multiplex images and a set of training singleplex images, each of the sets of training multiplex images depicting slices of a sample stained with two or more dyes, and each of the sets of training singleplex images depicting slices of a sample stained with a single dye; The machine learning model included a discrimination network configured to discriminate as to whether a given image was generated by the generation network or was a singleplex image of an actual slide. generating a predicted singleplex image; outputting the predicted singleplex image; 2. A computer program product comprising:

16. 16. The computer program product of claim 15, wherein each of the set of training singleplex images was a synthetic image generated by processing a corresponding training multiplex image of the set of training multiplex images using an unmix or remix algorithm configured for the context from which the corresponding training multiplex image was obtained.

17. 16. The computer program product of claim 15, wherein the machine learning model comprises a Pix2Pix model.

18. 16. The computer program product of claim 15, wherein each of the set of training singleplex images was a real image depicting a corresponding slice not depicted in any of the set of training multiplex images.

19. The machine learning model includes CycleGAN or BicycleGAN, and the CycleGAN or BicycleGAN is another generation network configured to generate a predicted multiplex image for each received singleplex image; a separate discrimination network configured to discriminate as to whether a given image was generated by said separate generation network or was a multiplex image of an actual slide; and 16. The computer program product of claim 15, comprising:

20. 16. The computer program product of claim 15, wherein the set of operations further comprises performing the training of the machine learning model prior to generating the predicted singleplex image.

21. the multiplexed image is generated using a first scanner at a first site, and the set of operations further comprises: accessing another multiplexed image depicting another particular slice of another particular sample stained with the two or more dyes; generating, using the generative network, another predicted singleplex image depicting the other particular slice stained with only one of the two or more dyes, the generative network generating the another predicted singleplex image configured with the same parameter values ​​at the time the predicted singleplex image was generated and at the time the another predicted singleplex image was generated; outputting said another predicted singleplex image; 16. The computer program product of claim 15, comprising:

Citation Information

Patent Citations

  • Method for splitting one or more images of a sample into image data split according to dyes, computer program, computer-readable medium storing the program and system for performing the method

    US20210089749A1

  • Three-dimensional forward-looking sonar target recognition with machine learning

    WO2020139442A2

  • Deep learning for optical coherence tomography segmentation

    WO2021124293A1