Digital synthesis of histological coloring using multiplex immunofluorescent imaging

Abstract

To provide a method, a system and a program which facilitate acquisition of a synthesized histological coloring image from a multiplex immunofluorescent (MPX) image.SOLUTION: A method according to the present invention has a step of generating a N-channel input image based on information from each of M channels of a MPX image of a tissue slice (M and N are positive integers and satisfy N≤M), and a step of generating a synthesized image by processing the N-channel input image by using a network of generators trained by using a training data set including a plurality of image pairs. The synthesized image indicates a tissue slice dyed by at least one histological chemical coloring agent. Each of the plurality of image pairs includes the N-channel image generated from a MPX image of a first slice of the tissue and an image of a second slice of the tissue colored by the at least one histological chemical agent.SELECTED DRAWING: Figure 8A

Description

[Technical Field]

[0001] Field The present disclosure relates to digital pathology, and in particular to techniques involving obtaining a composite histological staining image from multiple immunofluorescence images. [Background technology]

[0002] background Histopathology can involve the examination of slides prepared from tissue sections for a variety of reasons, such as diagnosing disease, assessing response to treatment, and / or developing pharmacological agents to combat disease. Because tissue sections and the cells therein are substantially transparent, slide preparation typically involves staining the tissue sections to make relevant structures more visible. Digital pathology can involve scanning the stained slides to obtain digital images, which can then be examined by digital pathology image analysis and / or interpreted by a human pathologist.

[0003] Multiplex immunofluorescence (MPX) staining of tissue sections allows for the simultaneous detection of multiple biomarkers and their co-expression at the single-cell level. MPX allows for characterization of the immune situation in the tumor microenvironment, which may significantly affect response to immunotherapy. For example, the use of MPX can provide detection and colocalization of more biomarkers in a single slide. While MPX promises to be a valuable tool for discovering effective treatments and developing new drugs, several challenges remain regarding accurately correlating MPX results with the underlying tissue structure. Summary of the Invention

[0004] overview In various embodiments, a computer-implemented method of image transformation is provided that includes generating an N-channel input image based on information from each of M channels of multiplex immunofluorescence (MPX) images of a tissue section, where M is a positive integer and N is a positive integer less than or equal to M, and generating a composite image by processing the N-channel input image using a generator network trained using a training dataset including a plurality of image pairs, wherein the composite image depicts a tissue section stained with at least one histochemical stain, and for each pair of images of the plurality of pairs, the pair includes an N-channel image generated from an MPX image of a first section of tissue and an image of a second section of tissue stained with the at least one histochemical stain.

[0005] In some embodiments, generating the N-channel input image includes, for each of M channels of the MPX image of the tissue section, mapping information from the channel to at least one of the N channels of the N-channel image. The mapping may include creating an autofluorescence image based on information from each of multiple channels (e.g., each of M channels) of the MPX image of the tissue section, and the N-channel input image may be based on information from the autofluorescence image. The autofluorescence image may be based on a nonlinear combination of multiple channels of the MPX image of the tissue section. Additionally or alternatively, the autofluorescence image may be based on the spectral distribution of autofluorescence among the multiple channels.

[0006] In some embodiments, the composite image is an N-channel image. Each of the N-channel input image and the composite image may be an RGB image. Additionally or alternatively, the MPX image of the tissue section may be a dark-field image and / or the composite image may be a bright-field image. It may also be an image.

[0007] In some embodiments, the at least one histochemical stain is hematoxylin and eosin.

[0008] In some embodiments, the N-channel image is based on an autofluorescence image and a nuclear counterstain image. For example, the N-channel image may be based on a linear combination of the autofluorescence image and a channel of an MPX image of a tissue section corresponding to the nuclear counterstain. The N-channel image is based on an array of optical density values based on the linear combination.

[0009] In some embodiments, N is equal to 3 and / or M is at least 4.

[0010] In some embodiments, the generator network is trained as part of a generative adversarial network. In some embodiments, the generator network is implemented as a U-Net and / or an encoder-decoder network. The generator network may be updated via an L1 loss measured between the image generated by the generator network and the expected output image.

[0011] In some embodiments, a method is provided that includes determining a diagnosis of a subject by a user based on a composite image generated by an embodiment of the computer-implemented method.

[0012] In some embodiments, a method is provided that includes determining, by a user, a diagnosis of a subject based on a composite image generated by an embodiment of the computer-implemented method, and administering, by the user, treatment with a compound based on (i) the composite image, and / or (ii) the subject's diagnosis.

[0013] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0014] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0015] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0016] The terms and expressions used are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents or portions thereof of the features shown and described, but to permit various modifications within the scope of the invention as defined in the claims. It is recognized, therefore, that although the claimed invention has been specifically disclosed by embodiments and optional features, it is to be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawings]

[0017] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0018] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of various embodiments will become more apparent by way of example only and with reference to the accompanying drawings.

[0019] [Figure 1] 1 shows an exemplary diagram of the workflow of a digital pathology solution. [Figure 2] 1 shows an exemplary diagram of a multiplex immunofluorescence (MPX) imaging workflow and exemplary MPX images. [Figure 3] 1 shows an exemplary MPX image of a slide from an MPX panel. [Figure 4] 10 shows another exemplary MPX image of a slide from an MPX panel. [Figure 5] Shown is an MPX (dark field) image (Figure 5A) registered to an H&E (bright field) image (Figure 5B). [Figure 6] An example workflow for acquiring images of H&E stained samples is shown. [Figure 7A] 1 shows examples of H&E images and mapping images that resemble actual H&E images. [Figure 7B] 1 shows examples of H&E images and mapping images that resemble actual H&E images. [Figure 8A] 1 illustrates a flowchart of an exemplary process according to some embodiments. [Figure 8B] 1 illustrates a flowchart of another exemplary process according to some embodiments. [Figure 9] 1 illustrates a process for generating an autofluorescence image according to some embodiments. [Figure 10A]10 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 10B] 10 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 10C] 10 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 10D] 10 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 11] 1 illustrates a process for generating a three-channel mapped input image (e.g., for training a deep learning network and / or for inference) according to some embodiments. [Figure 12] 10 illustrates another process for generating a three-channel mapped input image according to some embodiments. [Figure 13] 10 illustrates a further process for generating a three-channel mapped input image according to some embodiments. [Figure 14] 1 illustrates an exemplary computing environment according to some embodiments. [Figure 15] 1 illustrates a conditional GAN model according to some embodiments. [Figure 16] 1 illustrates an example of an implementation of a conditional GAN model using Pix2Pix GAN according to some embodiments. [Figure 17] 1 illustrates a flowchart of a process for acquiring training images according to some embodiments. [Figure 18]10 illustrates an example of extraction of tiles and image patches from a full slide image according to some embodiments. [Figure 19] 1 illustrates a representation of networks and network connections in a cycle GAN, according to some embodiments. [Figure 20] 1 illustrates a flow for generating and identifying images using cycle GAN, according to some embodiments. [Figure 21] 1 illustrates examples of input, target, and output images generated in accordance with various embodiments. [Figure 22] 1 illustrates examples of input, target, and output images generated in accordance with various embodiments. [Figure 23] 1 illustrates examples of input, target, and output images generated in accordance with various embodiments. [Figure 24] 1 shows a comparison of the reference spectral distribution of autofluorescence in four different tissue types. [Figure 25] Two example MPX images of unstained tissue samples are shown. [Figure 26] 1 shows an example of MPX images of a pancreatic tissue sample before and after correction. DETAILED DESCRIPTION OF THE INVENTION

[0020] Detailed Description The systems, methods, and software disclosed herein facilitate obtaining a composite histological staining image from multiple immunofluorescence images. While specific embodiments are described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The devices, methods, and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications may be made to the form of the exemplary methods and systems described herein without departing from the scope of protection.

[0021] I. Overview Digital pathology may involve the interpretation of digitized images to accurately diagnose a subject and guide therapeutic decision-making. In a digital pathology solution, an image analysis workflow may be established to automatically detect or classify biological objects of interest (e.g., positive / negative tumor cells, etc.). FIG. 1 shows an exemplary diagram of a workflow 100 of a digital pathology solution. The digital pathology solution workflow 100 includes acquiring a tissue slide at block 110, scanning a preselected area or the entire tissue slide with a digital image scanner (e.g., a whole slide imaging (WSI) scanner) to acquire a digital image at block 120, performing image analysis on the digital image using one or more image analysis algorithms at block 130, and scoring the object of interest based on the image analysis (e.g., quantitative scoring or semi-quantitative scoring such as positive, negative, moderate, weak, etc.) at block 140.

[0022] For example, evaluation of tissue changes caused by disease can be performed by examining thin tissue sections. A tissue sample (e.g., a tumor sample) can be sliced to obtain a series of sections, each having a thickness of, for example, 4 to 5 microns. Because tissue sections and the cells therein are substantially transparent, slide preparation typically involves staining the tissue sections to make relevant structures more visible. For example, different portions of the tissue can be stained with one or more different stains to reveal different characteristics of the tissue.

[0023] Each section may be mounted on a slide, which may then be scanned to create a digital image, which may then be examined by digital pathology image analysis and / or interpreted by a human pathologist (e.g., using image viewer software). The pathologist may review and manually annotate the digital image of the slide (e.g., tumor area, necrosis, etc.) to enable the use of image analysis algorithms to extract meaningful quantitative measurements (e.g., to detect and classify biological objects of interest). Traditionally, a pathologist may manually annotate each successive image of multiple tissue sections from a tissue sample to identify the same aspect on each successive tissue section.

[0024] One type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes) to stain tissue structures. Histochemical stains can be used to reveal general aspects of tissue morphology and / or cellular microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to reveal lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), periodic acid Schiff (PAS), silver stains, and iron stains. Images of histochemically stained samples are typically acquired using brightfield microscopy, and slides of histochemically stained samples are typically scanned at a resolution of 8 bits per pixel per color channel (e.g., each of red, green, and blue (RGB)).

[0025] In H&E staining of tissue samples, hematoxylin stains cell nuclei blue, eosin stains extracellular matrix and cytoplasm pink, and other structures may be stained with different shades, hues, and / or combinations of pink and blue. However, while H&E staining is useful for identifying general tissue and cellular anatomical structures, it cannot provide the specific information needed to support a specific diagnostic evaluation, such as information that can be used to distinguish between various types of cancer (e.g., HER2 scoring), which can be provided by immunohistochemistry as described below.

[0026] Another type of tissue staining is immunohistochemistry (IHC, also known as "immunostaining"), which uses a primary antibody that specifically binds to a target antigen of interest (also called a biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated with a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The use of IHC for tissue staining typically requires the use of very expensive reagents and more complex laboratory equipment and procedures than histochemical staining. Images of samples stained with IHC fluorophores are usually acquired using a dark-field microscope.

[0027] In multiplex immunofluorescence (MPX) imaging, a single slide is stained with multiple IHC fluorophores, with the emission spectra of the various fluorophores being distinguishable from one another. The number of different fluorophores in an MPX panel is typically five or six, but can be fewer or more. In addition to the fluorophores labeling the desired biomarkers, a fluorogenic nuclear counterstain, such as 4',6-diamidino-2-phenylindole ( MPX images are acquired by capturing the emission from each fluorophore into a separate corresponding channel of a dark-field image (typically at a resolution of 16 bits per pixel per channel), and the MPX image is viewed by assigning a different pseudocolor to each channel.

[0028] Figures 2, 3, and 4 show MPX images of three different panels of IHC staining. In the image in Figure 2, the channels corresponding to the biomarkers CD8A, programmed cell death protein 1 (PDCD1), cluster of differentiation 3-epsilon (CD3E), proliferation marker Ki-67 (MKI67), and keratin (KRT) are assigned false colors yellow, light blue, purple, red, and green, respectively. In the image in Figure 3, the channels corresponding to the nuclear counterstain DAPI and the biomarkers PanCK, CD31, fibroblast activation protein alpha (FAP), and major histocompatibility complex 1 (MHC1) are assigned false colors dark blue, light blue, green, red, and pink, respectively. In the image in Figure 4, the channels corresponding to DAPI and the biomarkers CD3 (light blue), programmed death ligand 1 (PD-L1), PD1, PanCK, and granzyme B (Gzmb) are assigned false colors dark blue, light blue, green, yellow, red, and pink, respectively.

[0029] Compared to IHC, detection of multiple biomarkers in a single MPX slide (e.g., as shown in Figures 2-4) provides valuable information regarding biomarker colocalization and coexpression. For example, detecting five biomarkers simultaneously allows for the detection of a large number (e.g., 17 to 90) of phenotypes (coexpression of a defined set of biomarkers) in a single MPX slide.

[0030] Figure 2 also shows an example of an MPX imaging workflow. Such a workflow may begin with algorithm development (e.g., including crosstalk correction, autofluorescence correction, candidate selection, pathologist ground truth, feature machine learning, and / or PD1 intensity classes) and progress through algorithm validation (e.g., comparing results from the algorithm with pathologist results), clinical reporting and analysis, and assay development, followed by sandwich assay validation.

[0031] Unlike histochemically stained (e.g., H&E) images, MPX images typically lack clear delineation of underlying tissue structures. As a result, the task of annotating MPX images to delineate regions of interest (ROIs) for further analysis (e.g., tumor or epitope regions) and / or regions to be excluded from further analysis (e.g., necrotic regions) can be difficult or impossible to achieve alone. For this reason, current techniques for annotating MPX images or confirming diagnoses based on MPX images generally rely on corresponding images of nearby sections of H&E-stained specimens to provide the necessary structural details. The H&E image is registered to the MPX image so that the pathologist can easily switch between the two images on the display (e.g., while performing annotation). Figure 6 shows an example workflow for generating H&E images, which may include operations such as fixation, processing, embedding, sectioning, staining, and microscopy.

[0032] Figure 5A shows an example of an annotated MPX image, and Figure 5B shows an example of a corresponding image of a nearby section of a specimen that has been H&E stained and aligned to the MPX image. Creating a reference H&E image incurs additional costs, and reviewing the two images may require a pathologist to review additional tissue annotations. Aligning one image to the other can be time-consuming and computationally expensive, as unique global and / or local deformations of each section may occur during the slide preparation procedure. Inaccurate alignment can result in inaccurate results when aligned images are superimposed, switched, or otherwise viewed together. The spatial relationship between biomarkers and underlying tissue structures (e.g., cell membranes, nuclei) depicted in the image may also be inaccurate. Furthermore, even when images are optimally registered, the two images do not depict the same tissue space because they are from different sections. Structures in one section may not be present in the other (e.g., structures whose size is comparable to the thickness of each section) or may have a different size and / or morphology in the other section. As a result, overlaying the registered image with the baseline image may not accurately capture the location of the biomarkers relative to the tissue structures, which can create difficulties during annotation and / or review of the MPX images.

[0033] Conventional image processing techniques can be used to create images resembling H&E-stained specimens by mapping one or more channels of an MPX image to the RGB color space. Figure 7A shows an example of such a mapped image, and Figure 7B shows a true H&E-stained image for comparison. While a mapped image such as that shown in Figure 7A may superficially resemble an H&E image, pathologists in routine clinical practice have found that such images exhibit insufficient image reality and quality to support actual diagnostic decisions, for example. For example, by comparing the regions near the right edge of the images in Figures 7A and 7B, it can be seen that the mapped image in Figure 7A lacks much of the structural detail evident at the mid-level values in Figure 7B.

[0034] To overcome these and other limitations, techniques are disclosed herein for generating synthetic (virtually stained) images from MPX images that depict histochemically stained (e.g., H&E-stained) samples with a high degree of structural similarity to actual images of the histochemically stained sample. Generating such synthetic images can assist in avoiding complex and tedious tissue preparation and staining processes and registration operations. Clinically, such a solution may have the advantage of allowing visualization of H&E-stained tissue structures by switching MPX images on and off, which may meet pathologists' standard practice and facilitate clinical adoption of multiplex imaging. Developmentally, such a solution may have the potential to generate ground truth for H&E analysis for algorithm development and enable easier registration of an MPX image to another MPX image, which can aid in biomarker discovery.

[0035] The generation of synthetic images may be performed by a trained generator network, which may include parameters learned during training of a generative adversarial network (GAN). The GAN may further include a discriminator network configured to predict whether an input image is fake (i.e., generated by the generator network) or real (i.e., represents an actual image collected from a subject). Feedback based on the accuracy of these predictions may be provided to the generator network during training.

[0036] One exemplary embodiment of the present disclosure relates to a method of image transformation that includes generating an N-channel input image based on information from each of M channels of an MPX image of a tissue section, where M is a positive integer and N is a positive integer less than or equal to M, and generating a composite image by processing the N-channel input image using a generator network trained using a training dataset including a plurality of image pairs, wherein the composite image depicts a tissue section stained with at least one histochemical stain, and for each pair of images of the plurality of pairs, the pair includes an N-channel image generated from an MPX image of a first section of tissue and an image of a second section of tissue stained with the at least one histochemical stain.

[0037] II. Definition As used herein, if an action is "based on" something, this means that the action is at least partly based on that thing. "based at least in part on" means "based at least in part on"

[0038] As used herein, the terms "substantially," "approximately," and "about" are defined as being largely, but not necessarily entirely, specified (and including) what is specified entirely, as would be understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" may be replaced with "within [percentage]" of what is specified, where percentages include 0.1, 1, 5, and 10%.

[0039] As used herein, the terms "sample," "biological sample," or "tissue sample" refer to any sample containing biological molecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans, veterinary animals such as cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or needle biopsies of tissue), cell samples (such as cytological smears, such as Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components, such as by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucous membranes, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as cheek swabs), or any material containing biological molecules derived from an initial biological sample. In certain embodiments, the term "biological sample," as used herein, refers to a sample prepared from a tumor or portion thereof obtained from a subject (such as a homogenized or liquefied sample).

[0040] III. Techniques for Digital Composition of Histological Staining Using MPX Imaging The techniques for digital synthesis of histological stains using MPX imaging described herein may use multimodality transfer techniques to convert MPX darkfield imaging to H&E brightfield imaging. Such techniques use information from different image channels to obtain intermediate and preprocessed training data for input into a deep learning network (e.g., a generative adversarial network, or GAN). Such techniques may include using information from many or all of the channels of the MPX image to generate the input training dataset.

[0041] 8A shows a flowchart of an example process 800 for converting a source image (e.g., a source image from a set of source images being processed) into a new image (e.g., a new image from a set of new images being generated) having similar characteristics to the target image. Process 800 may be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to FIGS. 14-16, 19, and 20).

[0042] Referring to FIG. 8A , at block 804, an N-channel input image based on information from each of M channels of an MPX image of a tissue section is generated, where M is a positive integer and N is a positive integer less than or equal to M. At block 808, a composite image is generated by processing the N-channel input image using a generator network. The composite image shows a tissue section that has been stained with at least one histochemical stain. The generator network has been trained using a training dataset that includes multiple image pairs, each pair being based on information from each of the M channels of an MPX image of a first section (e.g., ) an N-channel image generated from an MPX image of a first section of tissue and an image of a second section of tissue stained with at least one histochemical stain.

[0043] It may be desirable for the number of channels N of the input image generated in block 804 to be equal to the number of channels of the composite image generated in block 808. For example, it may be desirable to implement block 808 using a generator network such as a Pix2Pix GAN configured to generate an output image having the same number of channels as the input images. The composite image may be a three-channel image, such as an RGB image having a red channel (R), a green channel (G), and a blue channel (B).

[0044] FIG. 8B shows an implementation 812 of process block 804 that includes block 816. In block 816, for each of M channels of the MPX image of the tissue section, information from the channel is mapped to at least one of the N channels of the N-channel image. Such mapping may include linear and / or nonlinear mapping. In this example, block 816 includes block 820. In block 820, an autofluorescence image is generated that is based on information from each of the multiple channels (e.g., from each of the M channels) of the MPX image of the tissue section. In this implementation of process block 804, the N-channel input image is based on information from the autofluorescence image. An instance of block 812 may also be executed to generate each of the input images of the training dataset from a corresponding one of the source set of MPX images.

[0045] In some embodiments of the process 800, the histochemical stain is hematoxylin and eosin.

[0046] In some embodiments of process 800, the generator network is trained as part of a generative adversarial network (e.g., a cGAN, a Pix2Pix GAN, or a cycle GAN).

[0047] The autofluorescence image may be generated (e.g., at block 820) as a nonlinear combination of the multiple channels of the MPX image. In one example, the multiple channels of the MPX image are nonlinearly combined by selecting, for each pixel of the autofluorescence image, the minimum value of the corresponding pixel's values in the multiple channels of the MPX image. This operation may be performed, for example, by calculating b(x,y)=min[a1(x,y), a2(x,y), ..., a M (x, y)] (where a denotes the MPX image and b denotes the autofluorescence image). This operation is not linear. Generating the autofluorescence image may also include performing one or more operations on the image resulting from the nonlinear combination (e.g., a noise reduction operation such as applying a median filter).

[0048] FIG. 9 illustrates an example process for generating an autofluorescence image according to some embodiments. Such processing may be desirable, including equalizing the channels of the MPX image (e.g., to compensate for variations between the levels of tissue autofluorescence recorded in the various channels). In the example illustrated in FIG. 9, M channels of the MPX image are equalized (e.g., by selecting the minimum pixel value as described above) before nonlinearly combining them to generate the autofluorescence image. As illustrated in FIG. 9, the equalization applied to the various channels of the MPX image may be determined according to a reference spectral distribution of autofluorescence. For example, the equalization may be performed by dividing each channel of the MPX image by a corresponding weight (e.g., a scaling factor), and the value of each weight for the various channels may be determined according to a given reference spectral distribution of autofluorescence. The reference spectral distribution may be selected according to the tissue type of the sample in the MPX image; the calculation of the weights for the various channels as elements of the autofluorescence reference vector is described in more detail below.

[0049] In the specific example shown in Figure 9, M equals 6, and each of the six channels corresponds to a different one of six fluorophores (five biomarkers with their corresponding fluorophores and one fluorogenic nuclear counterstain (DAPI)). The staining protocol used to create such an MPX image may also be referred to as a 5-plex assay or panel. In other examples, M may equal 3, 4, or 5 (e.g., corresponding to 2-plex, 3-plex, and 4-plex assays or panels, respectively), or may be greater than 6 (e.g., 7, 8, 9, or 10 or more). In the specific example shown in Figure 9, the channel labeled 4 is the DAPI channel.

[0050] FIG. 24 shows a comparison example of the reference spectral distribution of autofluorescence for four different types of tissue (breast cancer, colorectal cancer (CRC), pancreatic cancer, and gastric cancer) and the average of these distributions for another six-channel system (in this figure, the channel labeled 6 is the DAPI channel). The reference spectral distribution can be obtained from a reference MPX image of an unstained sample of the same type of tissue (e.g., from the same tumor) as the stained sample shown in the MPX image. FIG. 25 shows two example MPX images of unstained tissue samples. The reference MPX image can be captured, for example, using the same imaging system used to capture the MPX image of the stained sample.

[0051] The reference spectral distribution can be obtained as an autofluorescence reference vector (e.g., an M-element reference vector), where each element of the vector corresponds to a different channel of the reference MPX image and represents the relative degree of tissue autofluorescence in that channel. The relative degree can be, for example, linear, polynomial, or exponential (e.g., magnitude, energy, or power). In the example shown in Figure 9, the values of the six elements of the autofluorescence reference vector (which can be applied as weights to the corresponding channels of the MPX image as described above) are [0.0919185036025541 0.147589073045573 0.467635814185659 0.822229790703801 0.195309012162166 0.120506922298875]. The specific region of the reference MPX image for which the relative degree of autofluorescence is determined can be the entire image or a selected region of interest of the image (e.g., tumor region or other annotated region). Each relative degree of autofluorescence can be calculated as the value of a selected statistic (e.g., average of the arithmetic mean, median, etc.) of the values of the corresponding channel over the desired image region.

[0052] It may be desirable to normalize each channel of the reference MPX image across the desired image region before calculating the statistics. In one such example, each channel is separately normalized to the range 0 to 1 by min-max scaling (i.e., by identifying the maximum and minimum pixel values for the channel across the desired image region, subtracting the minimum pixel value, and dividing the result by the difference between the minimum and maximum pixel values to normalize each value for the channel within the region).

[0053] While the number of channels in the source MPX image can be greater than M, it may be desirable to generate an N-channel input image using information from each of the MPX image's channels, and similarly for each of the N-channel training images. In other words, it may be desirable for M to equal the total number of channels in the source MPX image. Maximizing the use of information from all channels of the source MPX image to generate training data may be important to obtain a generator network trained to generate composite images that have maximum structural similarity to the corresponding target image. Similarly, while the number of channels combined to generate an autofluorescence image can be less than M, it may be desirable to combine all M channels to generate the autofluorescence image.

[0054] Examples of mapped input images are now described. Figures 10A-10D show four examples of three-channel (RGB) images, each acquired based on information from six channels of an MPX image of an IHC-stained tissue section, according to a different corresponding mapping according to some embodiments. Each of these four examples is based on a different linear combination of an autofluorescence image and an image from the DAPI (or other nuclear counterstain) channel of the MPX image. In the description herein, the mapping shown in Figure 10A is identified as "AF+AF+DAPI" or AAD, the mapping shown in Figure 10B is identified as "AF+DAPI+DAPI" or ADD, the mapping shown in Figure 10C is identified as "pseudo H&E" or pHE, and the mapping shown in Figure 10D is identified as "pseudo H&E 2" or pHE2.

[0055] Prior to performing the linear combination of the autofluorescence and DAPI images described herein (e.g., with reference to FIGS. 11-13), it may be desirable to rescale the autofluorescence and DAPI images to a specified range (or rescale the autofluorescence and MPX images to a specified range). In one example, rescaling may be normalization, which may be performed by identifying the minimum and maximum pixel values of the image and normalizing the image's pixel values to a range of 0 to 1 (e.g., by min-max scaling as described herein). In another example, rescaling may be performed by applying a desired scaling factor to each pixel value of the image (e.g., rescaling the image's pixel values from a range of 0 to 65535 to a range of 0 to 2000, or a range of 0 to 3000, etc.).

[0056] Before performing the linear combination of the autofluorescence image and the DAPI image described herein (see, e.g., Figures 11-13), it may be desirable to correct the DAPI image (e.g., to compensate for tissue autofluorescence) or correct the MPX image. Such corrections may be performed before or after rescaling the DAPI or MPX image, as described above. Correction of the DAPI image may be performed by multiplying the autofluorescence image by the corresponding element of an autofluorescence reference vector to obtain a weighted autofluorescence image, and then subtracting the corresponding pixel value of the weighted autofluorescence image from each pixel value of the DAPI image to obtain the corrected DAPI image. Correction of each channel of the MPX image may similarly be performed by multiplying the autofluorescence image by the corresponding element of the autofluorescence reference vector to obtain a weighted autofluorescence image, and then subtracting the corresponding pixel value of the weighted autofluorescence image from each pixel value of the channel to obtain the corrected channel image. Figure 26 shows an example of an MPX image of a pancreatic tissue sample before (top left) and after (bottom right) correction.

[0057] As shown in FIGS. 11 and 12, a mapped input image can be generated as a linear combination of an autofluorescence image and a DAPI image. FIG. 11 illustrates an example process for generating a three-channel mapped input image according to some embodiments. In this example (corresponding to the mapped input image “AAD” or “AF+AF+DAPI” as shown in FIG. 10A), the first channel of the input image is an autofluorescence image, the second channel of the input image is also an autofluorescence image, and the third channel of the input image is a DAPI image. The three channels of the AAD image can be paired with the color channels (e.g., RGB) of an actual H&E image as training data for the GAN framework. The three channels of RGB are mapped to the three channels of the AAD image.

[0058] FIG. 12 shows another example of a process for generating a three-channel mapped input image, according to some embodiments. In this example (corresponding to the mapped input image "ADD" or "AF+DAPI+DAPI" as shown in FIG. 10B), the first channel of the input image is an autofluorescence image, the second channel of the input image is a DAPI image, and the third channel of the input image is also a DAPI image. The three channels of the ADD image Each of the vectors can be paired with the corresponding color channels of the actual H&E image as training data for the GAN framework. For example, the three channels of the ADD image can be mapped to the R, G, and B channels of the actual H&E image, respectively.

[0059] Alternatively, the mapped input image may be generated from an optical density array that is a linear combination of the autofluorescence image and the DAPI image. Such calculations may be based, for example, on the Beer-Lambert law, which describes the attenuation of light by the material it travels through. For a given wavelength c and a given stain s, this relationship can be expressed as follows: [Transmitted light intensity at wavelength c] = [Incident light intensity at wavelength c] * exp(-[Optical density of stained tissue at wavelength c]), where ODc = (Amount of stain s in tissue) * (Absorption coefficient of stain s at wavelength c).

[0060] Such an embodiment of process block 804 may include generating an array of optical density values (e.g., an N-channel array of optical density values) as a linear combination of the autofluorescence image and the DAPI image, and applying a non-linear function to the array of optical density values to obtain a mapped input image (also referred to as an "optical density array"). Generating the optical density array as a linear combination of the autofluorescence image and the DAPI image may be performed, for example, as follows: OD_R=DAPIwt[1]*DAPI+afwt[1]*AF; OD_G=DAPIwt[2]*DAPI+afwt[2]*AF; OD_B=DAPIwt[3]*DAPI+afwt[3]*AF; where OD_R, OD_G, and OD_B refer to the red, green, and blue channels of the optical density array, DAPI refers to the DAPI image, AF refers to the autofluorescence image, and DAPIwt and afwt refer to the three-element weight vector. In some cases, it may be desirable to adjust one or more channels of the optical density array (e.g., to subtract bleed-through densities in adjacent spectra).

[0061] The values of DAPIwt and afwt can be selected experimentally (e.g., by visual inspection of the resulting mapped images and comparison of these images with actual H&E images). In one such example, the optical density array of input image pHE, as shown in Figure 10C, is obtained using weight vectors DAPIwt = [29.2500 48.7500 14.6250] and afwt = [0.3900 23.5535 16.6717].

[0062] Alternatively, the values of DAPIwt and afwt can be selected according to the color-dependent absorption characteristics of hematoxylin and eosin. In one such example, the optical density array of input image pHE2, as shown in FIG. 10D, is obtained using weight vectors DAPIwt=w_D*Abs_Hem and afwt=w_A*Abs_Eos, where Abs-Hem is a color vector whose elements indicate the relative absorbance of hematoxylin for red, green, and blue light (e.g., [0.65 0.70 0.29]), Abs_Eos is a color vector whose elements indicate the relative absorbance of eosin for red, green, and blue light (e.g., [0.07 0.99 0.11]), and the sum of scalar weights w_D and w_A is 1 (default values w_D=w_A=0.5). FIG. 13 shows an example of such a process for generating a three-channel mapped input image according to some embodiments. In some cases, it may be desirable to increase the value of w_D relative to w_A (e.g., to increase the influence of cell nuclei in the mapped input image).

[0063] Obtaining a mapped input image from an array of optical density values may include applying a nonlinear function (e.g., an exponential function) to convert the array of optical density values into an array of transmission values (also referred to as a "transmission array"). Such a conversion may be performed, for example, as follows: T_R=exp(-OD_R); T_G=exp(-OD_G); T_B=exp(-OD_B); where T_R, T_G, and T_B denote the red, green, and blue channels of the array of transmission values. In another example of a nonlinear function, the base of the exponential function is 10 rather than Napier's constant e.

[0064] The values of the transmission array may be rescaled to a desired image space to obtain the mapped input image as a bright-field image. In one example, each value of the transmission array is scaled (e.g., multiplied) by 255 to obtain the mapped input image as a 24-bit RGB image. In another example, the values of the transmission array are normalized to a desired range (e.g., a range of 0 to 255) by min-max scaling. Other output image spaces for the mapped input image may include Lab color spaces (e.g., CIELAB, HunterLab, etc.) or YCbCr color spaces (e.g., YCbCr, Y'CbCr, etc.), and generating the mapped input image from the transmission array may include conversion from one color space to another (e.g., from RGB or sRGB to YCrCb or Y'CrCb).

[0065] 14 illustrates an exemplary computing environment 1400 (i.e., a data processing system) for converting MPX images of tissue sections into composite images showing the tissue sections stained with at least one histochemical stain, according to various embodiments. As shown in FIG. 14 , the conversion performed by the computing environment 1400 in this example includes several stages: an image storage stage 1405, a preprocessing stage 1490, a model training stage 1410, a transformation stage 1415, and an analysis stage 1420. The image storage stage 1410 may include one or more digital image scanners or databases 1425 that are accessed (e.g., by the preprocessing stage 1490) to provide a source set of digital images 1430 and a target set of digital images 1435 from preselected regions or the entire biological specimen slide (e.g., a tissue slide).

[0066] The model training stage 1410 builds and trains one or more models 1440a-1440n (where "n" represents any natural number) (which may be referred to herein individually as models 1440 or collectively as models 1440) to be used by other stages. The models 1440 may be machine learning ("ML") models, which may include a convolutional neural network ("CNN"), an initial neural network, a residual neural network ("Resnet"), a U-Net, a V-Net, a single-shot multi-box detector ("SSD") network, a recurrent neural network ("RNN"), a deep neural network, a rectified linear unit ("ReLU"), a long short-term memory ("LSTM") model, a gated recurrent unit ("GRU") model, or the like, or any combination thereof. In various embodiments, the generative model is composed of parameters learned by training a model 1440 that can learn any type of data distribution using unsupervised learning, such as a generative adversarial network (“GAN”), a deep convolutional generative adversarial network (“DCGAN”), a variational autoencoder (VAE), a hidden Markov model (“HMM”), a Gaussian mixture model, a Boltzmann machine, or the like, or a combination of one or more of such techniques, e.g., a VAE-GAN. The computing environment 1400 may use the same or different types of models to convert source images into generated images. In a particular case, the generative model is composed of parameters learned by training a model 1440 that is a GAN constructed with a loss function that attempts to classify an output image as real or fake, while training the generative model to minimize this loss.

[0067] In the exemplary embodiment shown in FIG. 15 , the model 1440 trained to provide learned parameters is part of a conditional GAN ("cGAN") 1500, an extension of the GAN model, which generates images with specific conditions or attributes. The cGAN learns a structured loss that penalizes output combinations. Referring to FIG. 15 , the cGAN 1500 includes a generator 1510 and a discriminator 1515. The generator 1510 is a neural network (e.g., a CNN) that receives a randomly generated noise vector 1520 and a latent feature vector (or one-dimensional vector) 1525 (conditions, e.g., a mapped input image in this example) as input data and feedback from the discriminator 1515, and generates a new image 1530 that is as close as possible to a real target image 1535. The discriminator 1515 is a neural network (e.g., a CNN) configured as a classifier to identify whether the generated image 1530 from the generator 1510 is a real image or a fake image. The latent feature vector 1525 or condition is derived from an input image or set of input images 1540 (e.g., images provided by mapping controller 1593) and encodes a class (e.g., an N-channel image obtained from the M channels of the corresponding MPX image according to a selected mapping) or a set of specific characteristics expected from the input image 1540. The randomly generated noise vector 1520 may be generated from a Gaussian distribution, and the vector space may be composed of latent or hidden variables that are important to the domain but not directly observable. The latent feature vector 1525 and the random noise vector 1520 may be combined as input 1545 to generator 1510. Alternatively or additionally, noise may be added within generator 1510 in the form of dropout (e.g., stochastically dropping inputs down a layer).

[0068] The generator 1510 receives a combined input 1545 and generates an image 1530 based on a latent feature vector 1525 and a random noise vector 1520 within a problem domain (i.e., a domain of features relevant to a histochemically stained target image 1535). The classifier 1515 performs conditional image classification using both the target image 1535 and the generated image 1530 as inputs to predict 1550 the likelihood that the generated image 1530 is a real or false translation of the target image 1535. The output of the classifier 1515 depends on the size of the generated image 1530, but may be a single value or a squared activation map of values. Each value is a probability of the likelihood that a patch in the generated image 1530 is real. These values can be averaged to obtain an overall likelihood or classification score, as desired. The loss functions of both the generator 1510 and the classifier 1515 may be configured such that the loss depends on how well the classifier 1515 does its job of predicting (1550) the likelihood that a generated image 1530 is real or a fake translation of a target image 1535. After sufficient training, the generator 1510 begins to produce generated images 1530 that more closely resemble the target image 1535. Training of the GAN 1500 may proceed for a predefined number of training instances, and the resulting learned parameters may be accepted as long as one or more performance metrics (e.g., accuracy, precision, and / or recall) determined using the training or validation set exceed corresponding thresholds. Alternatively, training of the GAN 1500 may proceed until one or more performance metrics associated with a recent training iteration exceed corresponding thresholds. At this point, the generated image 1530 may be sufficiently similar to the target image 1535 that the classifier can no longer distinguish between real and fake. Once the generator network 1510 is trained, an input set of N-channel images (obtained from the M channels of the corresponding MPX images according to the selected mapping) can be input into the GAN 1500 to transform the input set of images into a new set of generated images that have similar properties to a target set of images obtained from histochemically stained slides (e.g., H&E staining).The newly generated set of images can then be used by the pathologist to visualize tissue structures during annotation of MPX images (e.g., by switching MPX images or composite H&E images on and off), to confirm a diagnosis based on MPX images, and / or to assist in analysis based on a composite panel of biomarkers (e.g., MPX images). It can be used to align an image to another MPX image, etc.

[0069] 14 , to train model 1440 in this example, preprocessing stage 1490 generates samples 1445 by acquiring digital images (a source set of digital images 1430 and a target set of digital images 1435), mapping the source images to N-channel images according to a selected mapping (e.g., by mapping controller 1493), dividing the images into a pairwise subset of images 1445a (at least one pair of a mapped image and a target image) (e.g., 90%) for training and a pairwise subset of images 1445b (e.g., 10%) for validation, preprocessing the pairwise subset of images 1445a and the pairwise subset of images 1445b, expanding the pairwise subset of images 1445a, and optionally annotating the pairwise subset of images 1445a with labels 1450. Pairwise subsets of images 1445a may be acquired from a data storage structure such as a database or an image server, for example. Each image represents a biological sample, such as a tissue.

[0070] The partitioning may be performed randomly or pseudo-randomly (e.g., using 90% / 10%, 80% / 20%, or 70% / 30%), or the partitioning may be performed according to more complex validation techniques, such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation, to minimize sampling bias and overfitting. Preprocessing may include cropping the images so that each image contains only a single object of interest. In some cases, preprocessing may further include standardization or rescaling (e.g., normalization) to bring all features to the same scale (e.g., the same size scale or the same color or saturation scale). In a particular example, the image is resized to a minimum size (width or height) of a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels) while maintaining the original aspect ratio.

[0071] For example, the preprocessing stage 1490 may prepare a plurality of patch images from the source set and the target set as one or more pairwise subsets of images for training data. Preparing the paired images may include accessing matched pairs of source and target images, where the source and target images of each matched pair are from slides of nearby (e.g., adjacent or nearly adjacent) sections of the same biological sample (e.g., a tumor sample), where the sections shown in the source images are stained with multiple (e.g., 2, 3, 4, 5, or 6 or more) selected IHC fluorophores, and the sections shown in the target images are stained with one or more selected histochemical stains. In one non-limiting example, each section in the source images is stained with six fluorophores (including a nuclear counterstain, e.g., DAPI), and each section in the target images is stained with H&E. Figures 5A and 5B show an example of a matched pair of a dark-field MPX image of an IHC-stained tissue section (Figure 5A) and a bright-field image of a nearby (e.g., adjacent) section of the same tissue sample that was H&E-stained (Figure 5B).

[0072] Next, preprocessing stage 1490 may map each of the source images according to a selected mapping (e.g., by mapping controller 1493) to obtain an N-channel input image paired with a corresponding target image, and divide each paired input and target image (e.g., the entire slide image) into a number of patches of a predetermined size (e.g., 128x128, 256x256, or another size) to generate matched pairs of patches for training. For example, it may be desirable to use only patches from regions of interest within the image, such as tumor annotations added by a reviewing pathologist. Preprocessing stage 1490 (e.g., alignment controller 1495) may align and / or unalign the paired input and target images before and / or after the images are divided into patches. The method may perform alignment. Alignment may involve designating one image as a reference image, also called a fixed image, and applying a geometric transformation or local displacement to the other image so that it is aligned with the reference image. Since the histochemically stained image (i.e., the target image) provides ground truth for training the network, it may be desirable to designate the target image as the reference image for purposes of alignment and registration. Pairs of aligned patches from the input set and the target set are selected, and this process results in one or more pairwise subsets of images for training data. A preprocessing stage 1490 may input the patch pairs into a GAN or cGAN to train the deep learning network.

[0073] Referring back to FIG. 14 , the preprocessing stage 1490 may use dilation to artificially expand the size of the pairwise subset of images 1445 a by creating modified versions of images in the dataset. Image data augmentation may be performed by creating transformed versions of images in the dataset that belong to the same class as the original image. Transformations include a range of operations from the field of image manipulation, such as shift, flip, zoom, etc. In some examples, operations include random erasure, shift, brightness, rotation, Gaussian blur, and / or elastic transformation to ensure that the model 1440 can perform under conditions outside those available from the pairwise subset of images 1445 a.

[0074] The training process of model 1440 involves inputting images from pairwise subset 1445a of images into model 1440 and performing iterative operations to find a set of model parameters (e.g., weights and / or biases) that minimize one or more loss or error functions of model 1440 (e.g., a first loss function to train a classifier to maximize the probability of the image training data, and a second loss function to train a classifier to minimize the probability of generated images sampled from a generator and maximize the probability the classifier assigns to its own generated images). Prior to training, model 1440 is configured by a defined set of hyperparameters, which are settings external to the model and can be tuned or optimized to control the behavior of model 1440. In most models, hyperparameters are explicitly defined to control different aspects of the model, such as memory or execution cost. However, additional hyperparameters may be defined to adapt the model to specific scenarios. For example, hyperparameters may include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, or the number of kernels of the model. Each iteration of training may involve finding a set of model parameters for the model 1440 (as constituted by the corresponding defined set of hyperparameters) such that the value of the loss or error function using the set of model parameters is smaller than the value of the loss or error function using a different set of model parameters in the previous iteration. In contrast to hyperparameters, model parameters are variables internal to the model, and their values may be estimated (e.g., learned) from training data. The loss or error function may be constructed to measure the difference between the output inferred using the model 1440 and ground truth target images using labels 1450.

[0075] Once a set of model parameters has been identified, the model 1440 may be trained and validated using a pairwise subset 1445b of images (a test or validation data set). The validation process involves iteratively inputting images from the pairwise subset 1445b of images into the model 1440 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, and nested cross-validation to adjust the hyperparameters and ultimately find an optimal set of hyperparameters. Once an optimal set of hyperparameters is obtained, a reserved test set of images from the subset 1445b of images is input into the model 1440 to obtain an output (in this example, a generated image with similar properties to the target image). The output is evaluated against ground truth target images using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient, and calculating performance metrics such as error, precision, precision, recall, and receiver operating characteristic curves (ROC).

[0076] As should be understood, other training / validation mechanisms are envisioned and may be implemented within computing environment 1400. For example, model 1440 may be trained and hyperparameters may be tuned on images from a pairwise subset of images 1445a, and images from a pairwise subset of images 1445b may be used only to test and evaluate the performance of model 1440.

[0077] The model training stage 1410 outputs trained models, including one or more trained transformation models 1460 and, optionally, one or more image analysis models 1465. In some cases, the first model 1460a has been trained to process an input image obtained from a source image 1430 of a biological specimen (tissue section). The input image is an N-channel image obtained from M channels of an MPX image of the biological sample according to a selected mapping. The input image is obtained by a transformation controller 1470 in the transformation stage 1415. The transformation controller 1470 includes program instructions for transforming the input image into a new image 1475 having characteristics of a target image using one or more trained transformation models 1460. The characteristics of the target image are associated with an image of a tissue section stained with one or more selected histochemical stains (e.g., H&E). The transformation includes inputting the input image into a trained generator model (part of the transformation model 1460) and generating a new image 1475 by the generator model.

[0078] In some cases, the new image 1475 is sent to an analysis controller 1480 within the analysis stage 1420. The analysis controller 1480 includes program instructions for analyzing the biological sample in the new image 1475 using one or more image analysis models 1465 and outputting analysis results 1485 based on the analysis. Analysis of the biological sample in the new image 1475 may include extracting measurements based on areas within the new image 1475, one or more cells within the new image 1475, and / or objects within the new image 1475 other than cells. Area-based measurements may include the most basic assessments, such as quantifying the area (in two dimensions) of a particular stain (e.g., a histochemical stain), the area of fat vesicles, or other events present on a slide. Cell-based measurements aim to identify and enumerate objects, such as cells. This identification of individual cells allows for subsequent evaluation of subcellular compartments. Finally, algorithms may be utilized to evaluate events or objects present on tissue sections that may not be composed of individual cells. In certain examples, the imaging analysis algorithms are configured to provide quantitative representations of cellular staining, morphology, and / or structure that can be used to locate cells or subcellular structures and ultimately aid in diagnosis and prognosis. In some cases, the imaging analysis algorithms are specifically configured for analyzing images having characteristics of a target image (e.g., an image of a histochemically stained section). For example, analysis of the new image 1475 may include providing an ROIS and / or automatic delineation of regions to be excluded relative to a corresponding MPX source image. In another example, analysis of the new image 1475 may include registering the new image 1475 to another new image 1475 and applying the registration results to the corresponding MPX source image.

[0079] Although not explicitly shown, it will be understood that the computing environment 1400 may further include a developer device associated with the developer. Communications from the developer device to components of the computing environment 1400 may include information such as the type of input images to be used for the models, the number and type of models to be used, the hyperparameters of each model, e.g., learning rate and number of hidden layers, how to format data requests, the training data to be used (e.g., and and how to access training data) and validation techniques used, and / or how to configure the controller process.

[0080] One specific example of a cGAN model 1500 that can be used to train a generator network 1510 is a Pix2Pix GAN. FIG. 16 illustrates an example implementation of a cGAN model 1500 that uses a Pix2Pix GAN 1600 to train a generator network 1610 to convert N-channel images, each obtained from M channels of a corresponding MPX image of an IHC-stained tissue section according to a selected mapping, into a composite image of a histochemically stained (e.g., H&E-stained) tissue section. As shown in FIG. 16, the generator network 1610 is implemented using a U-Net architecture, which includes an encoder with a layer that progressively downsamples the input to a bottleneck layer and a decoder with a layer that progressively upsamples the bottleneck output to generate an output. As shown in FIG. 16, the U-Net also includes skip connections between the encoder and decoder layers with feature maps of equal size. These connections connect channels of the encoder layer's feature map with channels of the corresponding decoder layer's feature map. In a particular example, the generator network 1610 is updated via the L1 loss measured between the generated image and the expected output image (e.g., the "Predicted Image" and "Ground Truth" in FIG. 16).

[0081] In general, using a Pix2Pix GAN requires that matched pairs of image patches to be used to train the generator network be aligned (e.g., at the pixel level). FIG. 17 shows a flowchart of an example process 1700 for generating matched, aligned pairs of image patches from the matched image pairs described herein (e.g., for training and / or validation). Process 1700 may be performed by preprocessing stage 1490 (e.g., by alignment controller 1495). Referring to FIG. 17 , at block 1704, low-resolution versions of the matched image pairs are coarsely aligned. In one example, such coarse alignment is performed using a transformation matrix M C N-channel mapping of M channels of MPX images of IHC stained sections (including, for example, translation and / or rotation) Nmap and apply it to a low-resolution version of the image I of a matched (e.g., close) H&E-stained section. H&E The transformation matrix Mc can be calculated automatically, for example, based on the contours of the tissue in each image to be registered.

[0082] In block 1708, tiles are extracted from the region of interest (ROI) of the coarsely aligned image pair (e.g., by projecting a grid onto each image that covers the annotation and extracting corresponding tiles from each image). Nmap Tile T from ROI Nmap (e.g., size 2048x2048 pixels) and image I H&E The corresponding tile T from H&E In block 1712, full-resolution versions of the extracted tile pairs are finely aligned. Such fine alignment can be achieved, for example, by aligning one image (e.g., I Nmap ) to the tiles from the other image (e.g., I H&EThe alignment may involve scaling, deskewing, and / or warping tiles from one image to align them with corresponding tiles from the other image (the reference image described above). In block 1716, the precisely aligned tiles are stitched together to obtain an aligned image pair, and in block 1720, each of the aligned images is sliced into patches (e.g., 128x128 pixels in size, 256x256 pixels, or another size). Then, in block 1724, each aligned image (e.g., patch P as shown in FIG. 18) is aligned. Nmap and P H&E ) are combined (e.g., stitched side-by-side) to obtain a training image. For example, a pair of matching patches, each of size 128x128 pixels, can be stitched together to obtain a training image of size 128x2 Alternatively, training images of size 256 x 256 pixels may be stitched together to obtain a training image of size 256 x 512 pixels. In this manner, the process of stitching precisely aligned tiles of each image together to obtain a pair of aligned images and combining matched patches from each of the aligned images to obtain a set of training images may be implemented to obtain a set of training data for using a Pix2Pix GAN implementation to train a generator network 1510 to convert N-channel mapped images obtained from the M channels of the corresponding MPX image into a composite image of the histochemically stained section.

[0083] Another specific example of a cGAN model 1500 that can be used to train a generator network 1510 is a cycle GAN, which includes multiple generator networks and multiple discriminator networks. X and G Y and the classifier network D X and D YIn this example, the Y region corresponds to an image representing a histochemically stained sample, and the X region corresponds to an N-channel image obtained from the M channels of the corresponding MPX image of an IHC-stained sample according to the selected mapping.

[0084] Figure 20 shows the flow between the generator network and the discriminator network in the application of Cycle GAN as described herein. Cycle GAN is a network of XY generators G Y 2024 (trained as generator network 1510), a YX generator network G configured and trained to convert an image of a histochemically stained sample into an image depicting an N-channel mapping from M channels of an MPX image of an IHC stained sample. X Including 2020. Generator network G X 2020 may include one or more convolutional layers and may include a U-net or a V-net. In some cases, the generator network G X 2020 includes a feature extraction encoder, a transformer, and a decoder, each with one or more convolutional layers. X 2020 and G Y The architecture of the 2024 may be the same.

[0085] Cycle GAN generates a classifier network D that distinguishes between real and fake images (e.g., real N-channel image 2012 and fake N-channel image 2016) depicting N-channel mappings from M channels of MPX images of IHC stained samples. X 2032 and another classifier network D that distinguishes between fake and real images depicting histochemically stained samples (e.g., real histochemically stained image 2004 and fake histochemically stained image 2008). Y 2028. Classifier network D X and D Y may include one or more convolutional layers and activation layers, and the classifier network D X and D YThe architecture may be the same.

[0086] The use of cycle GANs allows for precise alignment of matched image pairs (e.g., image I as described herein with reference to FIGS. 17 and 18 ). Nmap and I H&E This may have the advantage that precise alignment of the images (precise alignment of the images) is not required to generate the training data. However, better results were obtained when training the generator network 1510 on image patches of paired aligned images using a Pix2Pix GAN implementation.

[0087] Methods according to the present disclosure can be implemented to convert MPX images into corresponding composite images of H&E-stained specimens. Such methods can be used, for example, to assist pathologists in annotating MPX source images. Such methods can be implemented as an important part of a rapid screening process to provide a reference image of the underlying tissue structure without performing actual H&E staining. Furthermore, such "virtual staining" techniques can also be used to confirm diagnoses based on corresponding MPX images. Furthermore, the image processing techniques described herein can be used to convert MPX images into corresponding composite images of the H&E-stained specimens. Image transformation methods can be used to assist in the automatic registration of MPX images to enable the analysis (eg, co-localization, co-expression) of an extended panel of biomarkers.

[0088] 21-23 show three different examples of using an implementation of process 800 (using the Pix2Pix GAN described herein) to convert an MPX image into a corresponding composite H&E stained image. Each of FIGS. 21-23 shows the results obtained using each of the four exemplary source image mappings (pHE, pHE2, AAD, ADD) shown in FIGS. 10A-10D and described herein. In each of FIGS. 21-23, the top row shows the N-channel mapped source image ("Approach"), the middle row shows the corresponding generated composite H&E image ("Output"), and the bottom row shows the corresponding actual H&E image ("Target"). It can be seen that the output image is very similar to the target image.

[0089] V. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0090] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0091] The description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0092] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. 1. A method of image transformation, comprising: generating an N-channel input image based on information from each of M channels of a multiplexed immunofluorescence (MPX) image of the tissue section, where M is a positive integer and N is a positive integer less than or equal to M; generating a composite image by processing the N-channel input images using a generator network trained using a training dataset including a plurality of image pairs; Including, the composite image depicts a tissue section stained with at least one histochemical stain; For each image pair of the plurality of image pairs, the pair: an N-channel image generated from the MPX image of the first section of tissue; an image of a second section of the tissue stained with the at least one histochemical stain; A method comprising:

2. 2. The method of claim 1, wherein generating the N-channel input image includes, for each of the M channels of the MPX image of the tissue section, mapping information from the channel to at least one of the N channels of the N-channel image.

3. the mapping includes generating an autofluorescence image based on information from each of a plurality of channels of the MPX image of the tissue section; The method of claim 2 , wherein the N-channel input image is based on information from the autofluorescence image.

4. The method of claim 3 , wherein the autofluorescence image is based on a nonlinear combination of the multiple channels of the MPX image of the tissue section.

5. The method of claim 3 or 4, wherein the autofluorescence image is based on a spectral distribution of autofluorescence among the multiple channels.

6. The method of claim 3 , wherein the plurality of channels of the MPX image are the M channels of the MPX image of the tissue section.

7. 7. The method of claim 1, wherein for each image pair of the plurality of image pairs, the N-channel image of the pair is based on information from each of M channels of the MPX image of the first slice.

8. The method of claim 1 , wherein the composite image is an N-channel image.

9. The method of claim 8 , wherein each of the N-channel input image and the composite image is an RGB image.

10. 10. The method of claim 1, wherein the MPX image of the tissue section is a dark-field image and the composite image is a bright-field image.

11. wherein said at least one histochemical stain is hematoxylin and eosin. Item 11. The method according to any one of items 1 to 10.

12. 12. The method of claim 1, wherein the N-channel image is based on the autofluorescence image and a channel of the MPX image of the tissue section corresponding to a nuclear counterstain.

13. 12. The method of claim 1, wherein the N-channel image is based on a linear combination of the autofluorescence image and a channel of the MPX image of the tissue section corresponding to a nuclear counterstain.

14. The method of claim 1 , wherein the N-channel image is based on an array of optical density values that is based on the linear combination.

15. 15. The method of claim 1, wherein N is equal to 3.

16. 16. The method of any one of claims 1 to 15, wherein M is at least 4.

17. 17. The method of claim 1, wherein the generator network is trained as part of a generative adversarial network.

18. The method of any one of claims 1 to 17, wherein the generator network is implemented as a U-Net.

19. The method of any one of claims 1 to 18, wherein the generator network is implemented as an encoder-decoder network.

20. 20. The method of any one of claims 1 to 19, wherein the generator network is updated via an L1 loss measured between an image produced by the generator network and a predicted output image.

21. 21. A method for determining a diagnosis of a subject by a user, said determining being based on the composite image of any one of claims 1 to 20.

22. 22. The method of claim 21, further comprising administering, by the user, (i) the composite image, and / or (ii) treating the subject with a compound based on the diagnosis.

23. one or more data processors; a non-transitory computer readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform the method of image conversion of any one of claims 1 to 20.

24. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform the method of image conversion according to any one of claims 1 to 20.

Images