Digital merging of histological stains using multiplexed immunofluorescence imaging

Abstract

A technique for obtaining a composite histochemical stained image from a multiplexed immunofluorescence (MPX) image may include generating an N-channel input image based on information from each of M channels of an MPX image of a tissue section, where M and N are positive integers and N is less than or equal to M, and generating a composite image by processing the N-channel input image using a generator network trained using a training dataset including a plurality of image pairs. The composite image shows the tissue section stained with at least one histochemical stain. Each image pair of the plurality of image pairs includes an N-channel image generated from an MPX image of a first section of tissue and an image of a second section of tissue stained with at least one histochemical stain.

Description

[Technical field]

[0001] Field The present disclosure relates to digital pathology, and in particular to techniques involving obtaining a composite histological staining image from multiple immunofluorescence images. [Background technology]

[0002] background Histopathology may involve the examination of slides prepared from tissue sections for a variety of reasons, such as diagnosing disease, evaluating response to treatment, and / or developing pharmacological agents to combat disease. Because tissue sections and the cells therein are substantially transparent, slide preparation typically involves staining the tissue sections to make relevant structures more visible. Digital pathology may involve scanning the stained slides to obtain digital images, which may then be examined by digital pathology image analysis and / or interpreted by a human pathologist.

[0003] Multiplex immunofluorescence (MPX) staining of tissue sections allows for the simultaneous detection of multiple biomarkers and their co-expression at the single cell level. MPX allows for the characterization of the immune landscape in the tumor microenvironment, which may significantly affect the response to immunotherapy. For example, the use of MPX can provide detection and colocalization of more biomarkers in a single slide. MPX promises to be a valuable tool for discovering effective treatments and developing new drugs, but several challenges remain with regard to accurately correlating MPX results with the underlying tissue structure. Summary of the Invention

[0004] overview In various embodiments, a computer-implemented method of image transformation is provided that includes generating an N-channel input image based on information from each of M channels of a multiplexed immunofluorescence (MPX) image of a tissue section, where M is a positive integer and N is a positive integer less than or equal to M, and generating a composite image by processing the N-channel input image using a generator network trained using a training dataset that includes a plurality of image pairs, where the composite image illustrates a tissue section stained with at least one histochemical stain, and for each pair of images of the plurality of pairs, the pair includes an N-channel image generated from an MPX image of a first section of tissue and an image of a second section of tissue stained with at least one histochemical stain.

[0005] In some embodiments, generating the N channel input image includes, for each of the M channels of the MPX image of the tissue section, mapping information from the channel to at least one of the N channels of the N channel image. The mapping may include creating an autofluorescence image based on information from each of the multiple channels (e.g., each of the M channels) of the MPX image of the tissue section, and the N channel input image may be based on information from the autofluorescence image. The autofluorescence image may be based on a non-linear combination of the multiple channels of the MPX image of the tissue section. Additionally or alternatively, the autofluorescence image may be based on a spectral distribution of autofluorescence among the multiple channels.

[0006] In some embodiments, the composite image is an N-channel image. Each of the N-channel input image and the composite image may be an RGB image. Additionally or alternatively, the MPX image of the tissue section may be a darkfield image and / or the composite image may be a brightfield image.

[0007] In some embodiments, the at least one histochemical stain is hematoxylin and eosin.

[0008] In some embodiments, the N channel image is based on an autofluorescence image and a nuclear counterstain image. For example, the N channel image may be based on a linear combination of the autofluorescence image and the channel of the MPX image of the tissue section corresponding to the nuclear counterstain. The N channel image is based on an array of optical density values ​​based on the linear combination.

[0009] In some embodiments, N is equal to 3 and / or M is at least 4.

[0010] In some embodiments, the generator network is trained as part of a generative adversarial network. In some embodiments, the generator network is implemented as a U-Net and / or an encoder-decoder network. The generator network may be updated via an L1 loss measured between the image by the generator network and the expected output image.

[0011] In some embodiments, a method is provided that includes determining, by a user, a diagnosis of a subject based on a composite image generated by an embodiment of the computer-implemented method.

[0012] In some embodiments, a method is provided that includes determining, by a user, a diagnosis of a subject based on a composite image generated by an embodiment of the computer-implemented method, and administering, by the user, a treatment with a compound based on (i) the composite image, and / or (ii) the subject's diagnosis.

[0013] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0014] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0015] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.

[0016] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. [Brief description of the drawings]

[0017] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0018] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of various embodiments will become more apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings, in which: FIG.

[0019] [Figure 1] FIG. 1 shows an exemplary diagram of a workflow for a digital pathology solution. [Diagram 2] 1 shows an exemplary diagram of a multiplex immunofluorescence (MPX) imaging workflow and exemplary MPX images. [Diagram 3] 1 shows an exemplary MPX image of a slide from an MPX panel. [Figure 4] 13 shows another exemplary MPX image of a slide from an MPX panel. [Diagram 5] Shown is an MPX (dark field) image (Figure 5A) registered to an H&E (bright field) image (Figure 5B). [Figure 6] An example of a workflow for acquiring images of H&E stained samples is shown. [Figure 7A] 1 shows examples of H&E images and mapping images similar to actual H&E images. [Figure 7B] 1 shows examples of H&E images and mapping images similar to actual H&E images. [Figure 8A] 1 illustrates a flowchart of an exemplary process according to some embodiments. [Figure 8B] 1 illustrates a flowchart of another example process according to some embodiments. [Figure 9] 1 illustrates a process for generating an autofluorescence image according to some embodiments. [Figure 10A]1 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 10B] 1 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 10C] 1 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 10D] 1 illustrates an example of three-channel (RGB) mapped input images, each obtained from a different corresponding mapping of M channels of an MPX image, according to some embodiments. [Figure 11] 1 illustrates a process for generating a three-channel mapped input image (e.g., for training a deep learning network and / or for inference) according to some embodiments. [Figure 12] 1 illustrates another process for generating a three-channel mapped input image according to some embodiments. [Figure 13] 13 illustrates a further process for generating a three-channel mapped input image according to some embodiments. [Figure 14] 1 illustrates an exemplary computing environment according to some embodiments. [Figure 15] 1 illustrates a conditional GAN ​​model according to some embodiments. [Figure 16] 1 illustrates an example of an implementation of a conditional GAN ​​model using Pix2Pix GAN in accordance with some embodiments. [Figure 17] 1 illustrates a flowchart of a process for acquiring training images according to some embodiments. [Figure 18]13 shows an example of extraction of tiles and image patches from a full slide image according to some embodiments. [Figure 19] 1 illustrates a representation of a network and network connections in a cycle GAN according to some embodiments. [Figure 20] 1 illustrates a flow for generating and identifying images using Cycle GAN, according to some embodiments. [Figure 21] 1 illustrates examples of input, target, and output images generated in accordance with various embodiments. [Figure 22] 1 illustrates examples of input, target, and output images generated in accordance with various embodiments. [Diagram 23] 1 illustrates examples of input, target, and output images generated in accordance with various embodiments. [Figure 24] 1 shows an example comparison of the baseline spectral distribution of autofluorescence in four different tissue types. [Diagram 25] Two example MPX images of unstained tissue samples are shown. [Figure 26] 1 shows example MPX images of a pancreatic tissue sample before and after correction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Detailed Description The systems, methods and software disclosed herein facilitate obtaining a composite histological staining image from multiplexed immunofluorescence images. Although specific embodiments are described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The devices, methods and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions and modifications of the form of the exemplary methods and systems described herein may be made without departing from the scope of protection.

[0021] I. Overview Digital pathology may involve the interpretation of digitized images to accurately diagnose a subject and guide therapeutic decision-making. In a digital pathology solution, an image analysis workflow may be established to automatically detect or classify biological objects of interest (e.g., positive / negative tumor cells, etc.). FIG. 1 shows an exemplary diagram of a workflow 100 of a digital pathology solution. The digital pathology solution workflow 100 includes acquiring a tissue slide at block 110, scanning a preselected area or the entire tissue slide by a digital image scanner (e.g., a whole slide image (WSI) scanner) to acquire a digital image at block 120, performing image analysis on the digital image using one or more image analysis algorithms at block 130, and scoring the object of interest based on the image analysis at block 140 (e.g., quantitative scoring or semi-quantitative scoring such as positive, negative, moderate, weak, etc.).

[0022] Assessment of tissue changes, e.g., caused by disease, may be performed by examining thin tissue sections. A tissue sample (e.g., a tumor sample) may be sliced ​​to obtain a series of sections, each having a thickness of, e.g., 4-5 microns. Because tissue sections and the cells therein are substantially transparent, preparation of slides typically includes staining the tissue sections to make relevant structures more visible. For example, different parts of the tissue may be stained with one or more different stains to reveal different characteristics of the tissue.

[0023] Each section may be mounted on a slide, which may then be scanned to create a digital image, which may then be examined by digital pathology image analysis and / or interpreted by a human pathologist (e.g., using image viewer software). The pathologist may review and manually annotate the digital images of the slide (e.g., tumor areas, necrosis, etc.) to enable the use of image analysis algorithms to extract meaningful quantitative measurements (e.g., to detect and classify biological objects of interest). Traditionally, a pathologist may manually annotate each successive image of multiple tissue sections from a tissue sample to identify the same aspects on each successive tissue section.

[0024] One type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes) to stain tissue structures. Histochemical stains can be used to show general aspects of tissue morphology and / or cellular microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to show lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), periodic acid Schiff (PAS), silver stains, and iron stains. Images of histochemically stained samples are typically acquired using brightfield microscopy, and slides of histochemically stained samples are typically scanned at a resolution of 8 bits per pixel per color channel (e.g., each of red, green, and blue (RGB)).

[0025] In H&E staining of tissue samples, hematoxylin stains cell nuclei blue, eosin stains extracellular matrix and cytoplasm pink, and other structures may be stained to have different tones, hues, and / or combinations of pink and blue. However, while H&E staining is useful for identifying general tissue and cellular anatomy, it cannot provide the specific information needed to support a particular diagnostic evaluation, such as information that can be used to distinguish between various types of cancer (e.g., HER2 scoring), which can be provided by immunohistochemical chemistry as described below.

[0026] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that specifically binds to a target antigen of interest (also called a biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, a primary antibody first binds to the target antigen, and then a secondary antibody conjugated with a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The use of IHC for tissue staining typically requires the use of very expensive reagents as well as more complicated laboratory equipment and procedures than histochemical staining. Images of samples stained by IHC fluorophores are usually acquired using a darkfield microscope.

[0027] In multiplex immunofluorescence (MPX) imaging, a single slide is stained with multiple IHC fluorophores, and the emission spectra of the various fluorophores are distinguishable from one another. The number of different fluorophores in an MPX panel is typically five or six, but can be fewer or more. In addition to the fluorophores that label the desired biomarkers, a fluorogenic nuclear counterstain, such as 4',6-diamidino-2-phenylindole (DAPI), is typically included. MPX images are acquired by capturing the emission from each fluorophore into a separate corresponding channel of a darkfield image (typically with a resolution of 16 bits per pixel per channel), and the MPX image is viewed by assigning a different pseudocolor to each channel.

[0028] Figures 2, 3, and 4 show MPX images of three different panels of IHC staining. In the image in Figure 2, the channels corresponding to the biomarkers CD8A, programmed cell death protein 1 (PDCD1), differentiation cluster 3-epsilon (CD3E), proliferation marker Ki-67 (MKI67), and keratin (KRT) are assigned false color yellow, light blue, purple, red, and green, respectively. In the image in Figure 3, the channels corresponding to the nuclear counterstain DAPI and the biomarkers PanCK, CD31, fibroblast activation protein alpha (FAP), and major histocompatibility complex 1 (MHC1) are assigned false color, dark blue, light blue, green, red, and pink, respectively. In the image in Figure 4, the channels corresponding to DAPI and the biomarkers CD3 (light blue), programmed death ligand 1 (PD-L1), PD1, PanCK, and granzyme B (Gzmb) are assigned false color, dark blue, light blue, green, yellow, red, and pink, respectively.

[0029] Compared to IHC, detection of multiple biomarkers in a single MPX slide (e.g., as shown in Figures 2-4) provides valuable information regarding the colocalization and coexpression of biomarkers. For example, detecting five biomarkers simultaneously allows the detection of a large number (e.g., 17 to 90) of phenotypes (coexpression of a defined set of biomarkers) in a single MPX slide.

[0030] Figure 2 also shows an example of an MPX imaging workflow. Such a workflow may begin with algorithm development (e.g., including crosstalk correction, autofluorescence correction, candidate selection, pathologist ground truth, machine learning on features, and / or PD1 intensity classes) and progress through algorithm validation (e.g., comparing results from the algorithm to pathologist results), clinical reporting and analysis, and assay development followed by sandwich assay validation.

[0031] Unlike histochemically stained (e.g., H&E) images, MPX images typically lack clear delineation of underlying tissue structures. As a result, the task of annotating MPX images to delineate regions of interest (ROIs) for further analysis (e.g., tumor or epitope regions) and / or regions to be excluded from further analysis (e.g., necrotic regions) can be difficult or impossible to do alone. For such reasons, current techniques for annotating MPX images or confirming diagnoses based on MPX images generally rely on corresponding images of nearby sections of H&E stained specimens to provide the necessary structural details. The H&E image is registered to the MPX image so that the pathologist can easily switch between the two images on the display (e.g., while performing annotation). Figure 6 shows an example workflow for generating H&E images, which may include operations such as fixing, processing, embedding, sectioning, staining, and microscopy.

[0032] FIG. 5A shows an example of an annotated MPX image, and FIG. 5B shows an example of a corresponding image of a nearby portion of a specimen that has been H&E stained and registered to the MPX image. Creation of a reference H&E image incurs additional costs, and review of the two images may require a pathologist to review additional tissue annotations. Registering one image to the other can be time-consuming and computationally expensive, as unique global and / or local deformations of each section may occur during the slide preparation procedure. If the registration is inaccurate, the spatial relationships between biomarkers and underlying tissue structures (e.g., cell membranes, nuclei) that are shown when the registered images are superimposed, switched, or otherwise viewed together may also be inaccurate. Furthermore, even if the images are optimally registered, the two images will not depict the same tissue space, since they are from different portions. Structures in one section may not be present in the other section (e.g., structures whose size is comparable through the thickness of each section) or may have a different size and / or morphology in the other section, such that overlay of the alignment image with the baseline image may not accurately capture the location of the biomarker relative to the tissue structures, which may create difficulties during annotation and / or review of the MPX images.

[0033] Conventional image processing techniques can be used to create images resembling H&E stained specimens by mapping one or more channels of an MPX image into RGB color space. FIG. 7A shows an example of such a mapped image, and FIG. 7B shows a true H&E stained image for comparison. Although a mapped image such as that shown in FIG. 7A may superficially resemble an H&E image, pathologists in routine clinical practice have confirmed that such images exhibit insufficient image reality and quality to aid in, for example, practical decisions in diagnosis. For example, by comparing the regions near the right edge of the images in FIG. 7A and FIG. 7B, it can be seen that the mapped image in FIG. 7A lacks much of the structural detail evident in the mid-level values ​​of FIG. 7B.

[0034] To overcome these and other limitations, techniques are disclosed herein for generating synthetic (virtually stained) images from MPX images that depict histochemically stained (e.g., H&E stained) samples with a high degree of structural similarity to actual images of histochemically stained samples. The generation of such synthetic images may aid in avoiding complex and tedious tissue preparation and staining processes as well as registration operations. Clinically, such a solution may have the advantage of allowing visualization of H&E stained tissue structures by switching the MPX image on and off, which may meet the standard practice of pathologists and facilitate clinical adoption of multiplex imaging. Developmentally, such a solution may have the potential to generate ground truth for H&E analysis for algorithm development and may enable easier registration of an MPX image to another MPX image that may aid in biomarker discovery.

[0035] The generation of synthetic images may be performed by a trained generator network, which may include parameters learned during training of a generative adversarial network (GAN). The GAN may further include a discriminator network configured to predict whether an input image is fake (i.e., generated by the generator network) or real (i.e., represents an actual image collected from a subject). Feedback based on the accuracy of these predictions may be provided to the generator network during training.

[0036] One exemplary embodiment of the present disclosure relates to a method of image transformation that includes generating an N-channel input image based on information from each of M channels of an MPX image of a tissue section, where M is a positive integer and N is a positive integer less than or equal to M, and generating a composite image by processing the N-channel input image using a generator network trained using a training dataset that includes a plurality of image pairs, where the composite image shows a tissue section stained with at least one histochemical stain, and for each pair of images of the plurality of pairs, the pair includes an N-channel image generated from an MPX image of a first section of tissue and an image of a second section of tissue stained with at least one histochemical stain.

[0037] II. Definition As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a part of the something.

[0038] As used herein, the terms "substantially," "approximately," and "about" are defined as being largely, but not necessarily completely, specified (and including that which is completely specified), as would be understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" may be replaced with "within [a percentage]" of that which is specified, where the percentage includes 0.1, 1, 5, and 10%.

[0039] As used herein, the terms "sample", "biological sample" or "tissue sample" refer to any sample containing biological molecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as veterinary animals such as humans, cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections or needle biopsies of tissues), cell samples (such as cytological smears such as Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (obtained by lysing cells and separating their components, such as by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucous membranes, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as cheek swabs), or any material containing biological molecules derived from an initial biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or portion thereof obtained from a subject (such as a homogenized or liquefied sample).

[0040] III. Techniques for Digital Composition of Histological Stains Using MPX Imaging The techniques for digital synthesis of histological stains using MPX imaging described herein may use multi-modality transfer techniques to convert from MPX dark-field imaging to H&E stained bright-field imaging. Such techniques use information from different image channels to obtain intermediate and pre-processed training data for input to a deep learning network (e.g., a generative adversarial network or GAN). Such techniques may include using information from many or all of the channels of the MPX image to generate the input training data set.

[0041] 8A shows a flowchart of an example process 800 for converting a source image (e.g., a source image from a set of source images being processed) into a new image (e.g., a new image from a set of new images being generated) having similar characteristics to a target image. Process 800 may be performed using one or more computing systems, models, and networks (e.g., as described herein with respect to FIGS. 14-16, 19, and 20).

[0042] Referring to FIG. 8A, at block 804, an N-channel input image based on information from each of the M channels of an MPX image of a tissue section is generated, where M is a positive integer and N is a positive integer less than or equal to M. At block 808, a composite image is generated by processing the N-channel input image using a generator network. The composite image shows a tissue section that has been stained with at least one histochemical stain. The generator network has been trained using a training dataset that includes a plurality of image pairs, each pair including an N-channel image generated from an MPX image of a first section of tissue (e.g., based on information from each of the M channels of the MPX image of the first section) and an image of a second section of tissue stained with at least one histochemical stain.

[0043] It may be desirable for the number of channels N of the input image generated in block 804 to be equal to the number of channels of the composite image generated in block 808. For example, it may be desirable to implement block 808 using a generator network such as a Pix2Pix GAN configured to generate an output image having the same number of channels as the input images. The composite image may be a three-channel image, such as an RGB image having a red channel (R), a green channel (G), and a blue channel (B).

[0044] FIG. 8B illustrates an implementation 812 of process block 804 that includes block 816. In block 816, for each of the M channels of the MPX image of the tissue section, information from the channel is mapped to at least one of the N channels of the N channel image. Such mapping may include linear and / or non-linear mapping. In this example, block 816 includes block 820. In block 820, an autofluorescence image is generated that is based on information from each of the multiple channels (e.g., from each of the M channels) of the MPX image of the tissue section. In this implementation of process block 804, the N channel input image is based on information from the autofluorescence image. An instance of block 812 may also be executed to generate each of the input images of the training data set from a corresponding one of the source set of MPX images.

[0045] In some embodiments of the process 800, the histochemical stain is hematoxylin and eosin.

[0046] In some embodiments of process 800, the generator network is trained as part of a generative adversarial network (e.g., a cGAN, Pix2Pix GAN, or cycle GAN).

[0047] The autofluorescence image may be generated (e.g., at block 820) as a non-linear combination of the multiple channels of the MPX image. In one example, the multiple channels of the MPX image are non-linearly combined by selecting, for each pixel of the autofluorescence image, the minimum value among the values ​​of the corresponding pixel in the multiple channels of the MPX image. This operation may be performed, for example, by calculating b(x,y)=min[a 1 (x,y), a 2 (x,y), ..., a M(x,y)] (where a denotes the MPX image and b denotes the autofluorescence image). This operation is not linear. Producing the autofluorescence image may also include performing one or more operations on the image resulting from the nonlinear combination (e.g., a noise reduction operation such as applying a median filter).

[0048] FIG. 9 illustrates an example of a process for generating an autofluorescence image according to some embodiments. Such processing may be desirable to include equalizing the channels of the MPX image (e.g., to compensate for variations between the levels of tissue autofluorescence recorded in the various channels). In the example illustrated in FIG. 9, M channels of the MPX image are equalized (e.g., by selecting a minimum pixel value as described above) before nonlinearly combining to generate the autofluorescence image. As illustrated in FIG. 9, the equalization applied to the various channels of the MPX image may be determined according to a reference spectral distribution of autofluorescence. For example, the equalization may be performed by dividing each channel of the MPX image by a corresponding weight (e.g., a scaling factor), and the value of each weight of the various channels may be determined according to a given autofluorescence reference spectral distribution of autofluorescence. The reference spectral distribution may be selected according to the tissue type of the sample in the MPX image, and the calculation of the weights of the various channels as elements of the autofluorescence reference vector is described in more detail below.

[0049] In the particular example shown in FIG. 9, M is equal to 6, and each of the six channels corresponds to a different one of six fluorophores (five biomarkers with their corresponding fluorophores and one fluorogenic nuclear counterstain (DAPI)). The staining protocol used to generate such MPX images may also be referred to as a 5-plex assay or panel. In other examples, M may be equal to 3, 4, or 5 (e.g., corresponding to 2-plex, 3-plex, and 4-plex assays or panels, respectively), or may be greater than 6 (e.g., 7, 8, 9, or 10 or more). In the particular example shown in FIG. 9, the channel labeled 4 is the DAPI channel.

[0050] FIG. 24 shows an example of a comparison between the reference spectral distributions of autofluorescence for four different types of tissue (breast cancer, colorectal cancer (CRC), pancreatic cancer, and gastric cancer) and the average of these distributions for another six-channel system (in this figure, the channel labeled 6 is the DAPI channel). The reference spectral distributions may be obtained from a reference MPX image of an unstained sample that is the same type of tissue (e.g., from the same tumor) as the stained sample shown in the MPX image. FIG. 25 shows two examples of MPX images of unstained tissue samples. The reference MPX images may be captured, for example, using the same imaging system used to capture the MPX images of the stained samples.

[0051] The reference spectral distribution may be obtained as an autofluorescence reference vector (e.g., an M-element reference vector), where each element of the vector corresponds to a different channel of the reference MPX image and represents the relative degree of tissue autofluorescence in that channel. The relative degree may be, for example, linear, polynomial, or exponential (e.g., magnitude, energy, or power). In the example shown in FIG. 9, the values ​​of the six elements of the autofluorescence reference vector (which may be applied as weights to the corresponding channels of the MPX image as described above) are [0.0919185036025541 0.147589073045573 0.467635814185659 0.822229790703801 0.195309012162166 0.120506922298875]. The particular region of the reference MPX image for which the relative degree of autofluorescence is determined may be the entire image or a selected region of interest of the image (e.g., a tumor region or other annotated region). Each relative degree of autofluorescence may be calculated as the value of a selected statistic (e.g., the average of the arithmetic mean, median, etc.) of the values ​​of the corresponding channel over the desired image region.

[0052] It may be desirable to normalize each channel of the reference MPX image over the desired image region before calculating the statistics. In one such example, each channel is separately normalized to the range 0 to 1 by min-max scaling (i.e., by identifying the maximum and minimum pixel values ​​for the channel over the desired image region, subtracting the minimum pixel value, and normalizing each value of the channel within the region by dividing the result by the difference between the minimum and maximum pixel values).

[0053] Although it is possible for the number of channels in the source MPX image to be greater than M, it may be desirable to generate an N-channel input image using information from every one of the channels of the MPX image, and similarly for each of the N-channel training images. In other words, it may be desirable for M to be equal to the total number of channels in the source MPX image. Maximizing the use of information from all channels of the source MPX image to generate training data may be important to obtain a generator network trained to generate a composite image with maximum structural similarity to the corresponding target image. Similarly, it is possible for the number of channels combined to generate an autofluorescence image to be less than M, but it may be desirable to combine all M channels to generate an autofluorescence image.

[0054] Examples of mapped input images are now described. Figures 10A-10D show four examples of three-channel (RGB) images, each obtained based on information from six channels of an MPX image of an IHC-stained tissue section according to a different corresponding mapping according to some embodiments. Each of these four examples is based on a different linear combination of an autofluorescence image and an image from the DAPI (or other nuclear counterstain) channel of the MPX image. In the present description, a mapping as shown in Figure 10A is identified as "AF+AF+DAPI" or AAD, a mapping as shown in Figure 10B is identified as "AF+DAPI+DAPI" or ADD, a mapping as shown in Figure 10C is identified as "pseudo H&E" or pHE, and a mapping as shown in Figure 10D is identified as "pseudo H&E 2" or pHE2.

[0055] It may be desirable to rescale the autofluorescence and DAPI images to a specified range (or rescale the autofluorescence and MPX images to a specified range) prior to performing the linear combination of the autofluorescence and DAPI images described herein (e.g., with reference to FIGS. 11-13). In one example, the rescaling may be normalization, which may be performed by identifying minimum and maximum pixel values ​​of the image and normalizing the image's pixel values ​​to a range of 0 to 1 (e.g., by min-max scaling as described herein). In another example, the rescaling may be performed by applying a desired scaling factor to each pixel value of the image (e.g., rescaling the image's pixel values ​​from a range of 0 to 65535 to a range of 0 to 2000, or a range of 0 to 3000, etc.).

[0056] Prior to performing the linear combination of the autofluorescence and DAPI images described herein (e.g., with reference to FIGS. 11-13), it may be desirable to correct the DAPI image (e.g., to compensate for tissue autofluorescence) and / or correct the MPX image. Such correction may be performed before or after rescaling the DAPI and / or MPX images, as described above. Correction of the DAPI image may be performed by multiplying the autofluorescence image by a corresponding element of an autofluorescence reference vector to obtain a weighted autofluorescence image, and subtracting the corresponding pixel value of the weighted autofluorescence image from each pixel value of the DAPI image to obtain the corrected DAPI image. Correction of each channel of the MPX image may similarly be performed by multiplying the autofluorescence image by a corresponding element of an autofluorescence reference vector to obtain a weighted autofluorescence image, and subtracting the corresponding pixel value of the weighted autofluorescence image from each pixel value of the channel to obtain the corrected channel image. FIG. 26 shows an example of an MPX image of a pancreatic tissue sample before (top left) and after (bottom right) correction.

[0057] As shown in FIG. 11 and FIG. 12, a mapped input image can be generated as a linear combination of an autofluorescence image and a DAPI image. FIG. 11 illustrates an example of a process for generating a three-channel mapped input image according to some embodiments. In this example (corresponding to the mapped input image “AAD” or “AF+AF+DAPI” as shown in FIG. 10A), the first channel of the input image is an autofluorescence image, the second channel of the input image is also an autofluorescence image, and the third channel of the input image is a DAPI image. The three channels of the AAD image can be paired with the color channels (e.g., RGB) of an actual H&E image as training data for the GAN framework. The three channels of RGB are mapped to the three channels of the AAD image.

[0058] FIG. 12 illustrates another example of a process for generating a three-channel mapped input image according to some embodiments. In this example (corresponding to the mapped input image "ADD" or "AF+DAPI+DAPI" as shown in FIG. 10B), the first channel of the input image is an autofluorescence image, the second channel of the input image is a DAPI image, and the third channel of the input image is also a DAPI image. Each of the three channels of the ADD image can be paired with the corresponding color channel of an actual H&E image as training data for the GAN framework. For example, the three channels of the ADD image can be respectively mapped to the R channel, G channel, and B channel of the actual H&E image.

[0059] Alternatively, the mapped input image may be generated from an optical density array that is a linear combination of the autofluorescence image and the DAPI image. Such calculations may be based, for example, on the Beer-Lambert law, which describes the attenuation of light by the material it travels through. For a given wavelength c and a given stain s, this relationship may be expressed as: [transmitted light intensity at wavelength c] = [incident light intensity at wavelength c] * exp(- [optical density of stained tissue at wavelength c]), where ODc = (amount of stain s in tissue) * (absorption coefficient of stain s at wavelength c).

[0060] Such an embodiment of process block 804 may include generating an array of optical density values ​​(e.g., an N-channel array of optical density values) as a linear combination of the autofluorescence image and the DAPI image, and applying a non-linear function to the array of optical density values ​​to obtain a mapped input image (also referred to as an "optical density array"). Generating the optical density array as a linear combination of the autofluorescence image and the DAPI image may be performed, for example, as follows: OD_R=DAPIwt[1]*DAPI+afwt[1]*AF; OD_G=DAPIwt[2]*DAPI+afwt[2]*AF; OD_B=DAPIwt[3]*DAPI+afwt[3]*AF; where OD_R, OD_G, and OD_B refer to the red, green, and blue channels of the optical density array, DAPI refers to the DAPI image, AF refers to the autofluorescence image, and DAPIwt and afwt refer to a three-element weight vector. In some cases, it may be desirable to adjust one or more channels of the optical density array (e.g., to subtract bleed-through densities in adjacent spectra).

[0061] The values ​​of DAPIwt and afwt may be selected experimentally (e.g., by visual inspection of the resulting mapped images and comparison of these images with actual H&E images). In one such example, the optical density array of the input image pHE, as shown in Figure 10C, is obtained using weight vectors DAPIwt = [29.2500 48.7500 14.6250] and afwt = [0.3900 23.5535 16.6717].

[0062] Alternatively, the values ​​of DAPIwt and afwt may be selected according to the color-dependent absorption characteristics of hematoxylin and eosin. In one such example, the optical density array of the input image pHE2 as shown in FIG. 10D is obtained using weight vectors DAPIwt=w_D*Abs_Hem and afwt=w_A*Abs_Eos, where Abs-Hem is a color vector whose elements indicate the relative absorbance of hematoxylin to red, green, and blue light (e.g., [0.65 0.70 0.29]), Abs_Eos is a color vector whose elements indicate the relative absorbance of eosin to red, green, and blue light (e.g., [0.07 0.99 0.11]), and the sum of scalar weights w_D and w_A is 1 (default value w_D=w_A=0.5). FIG. 13 illustrates an example of such a process for generating a three-channel mapped input image according to some embodiments. In some cases, it may be desirable to increase the value of w_D relative to w_A (eg, to increase the influence of cell nuclei in the mapped input image).

[0063] Obtaining the mapped input image from the array of optical density values ​​may include applying a non-linear function (e.g., an exponential function) to convert the array of optical density values ​​into an array of transmission values ​​(also referred to as a "transmission array"). Such a conversion may be performed, for example, as follows: T_R = exp(-OD_R); T_G = exp(-OD_G); T_B = exp(-OD_B); where T_R, T_G, and T_B denote the red, green, and blue channels of the array of transmission values. In another example of a nonlinear function, the exponential function has a base of 10 rather than Napier's constant e.

[0064] The values ​​of the transmission array may be rescaled to a desired image space to obtain the mapped input image as a brightfield image. In one example, each value of the transmission array is scaled (e.g., multiplied) by 255 times to obtain the mapped input image as a 24-bit RGB image. In another example, the values ​​of the transmission array are normalized to a desired range (e.g., a range of 0 to 255) by min-max scaling. Other output image spaces of the mapped input image may include Lab color spaces (e.g., CIELAB, Hunter Lab, etc.) or YCbCr color spaces (e.g., YCbCr, Y'CbCr, etc.), and generating the mapped input image from the transmission array may include a conversion from one color space to another (e.g., from RGB or sRGB to YCrCb or Y'CrCb).

[0065] 14 illustrates an exemplary computing environment 1400 (i.e., a data processing system) for converting MPX images of tissue sections into composite images showing the tissue sections stained with at least one histochemical stain, according to various embodiments. As illustrated in FIG. 14, the conversion performed by the computing environment 1400 in this example includes several stages: an image storage stage 1405, a pre-processing stage 1490, a model training stage 1410, a conversion stage 1415, and an analysis stage 1420. The image storage stage 1410 may include one or more digital image scanners or databases 1425 that are accessed (e.g., by the pre-processing stage 1490) to provide a source set of digital images 1430 and a target set of digital images 1435 from pre-selected regions or the entirety of a biological specimen slide (e.g., a tissue slide).

[0066] The model training stage 1410 builds and trains one or more models 1440a-1440n (where "n" represents any natural number) (which may be referred to herein individually as models 1440 and collectively as models 1440) to be used by other stages. The models 1440 may be machine learning ("ML") models, which may include a convolutional neural network ("CNN"), an initial neural network, a residual neural network ("Resnet"), a U-Net, a V-Net, a single-shot multi-box detector ("SSD") network, a recurrent neural network ("RNN"), a deep neural network, a rectified linear unit ("ReLU"), a long short-term memory ("LSTM") model, a gated recurrent unit ("GRU") model, or the like, or any combination thereof. In various embodiments, the generative model is configured by parameters learned by training a model 1440 that can learn any kind of data distribution using unsupervised learning, such as a generative adversarial network ("GAN"), a deep convolutional generative adversarial network ("DCGAN"), a variational autoencoder (VAE), a hidden Markov model ("HMM"), a Gaussian mixture model, a Boltzmann machine, or the like, or one or more of such techniques, e.g., a combination of VAE-GAN. The computing environment 1400 may use the same kind of model or different kinds of models to convert source images into generated images. In a particular case, the generative model is configured by parameters learned by training a model 1440 that is a GAN constructed with a loss function that tries to classify whether an output image is real or fake, while training the generative model to minimize this loss.

[0067] In the exemplary embodiment shown in FIG. 15, the trained model 1440 to provide learned parameters is part of a conditional GAN ​​("cGAN") 1500, which is an extension of the GAN model, and generates images with certain conditions or attributes. The cGAN learns a structured loss that penalizes the combined configuration of the output. With reference to FIG. 15, the cGAN 1500 includes a generator 1510 and a discriminator 1515. The generator 1510 is a neural network (e.g., CNN) that receives a randomly generated noise vector 1520 and a latent feature vector (or one-dimensional vector) 1525 (conditions, e.g., in this example, the mapped input image) as input data and feedback from the discriminator 1515, and generates a new image 1530 that is as close as possible to a real target image 1535. The discriminator 1515 is a neural network (e.g., CNN) configured as a classifier to discriminate whether the generated image 1530 from the generator 1510 is a real image or a fake image. The latent feature vector 1525 or condition is derived from an input image or set of input images 1540 (e.g., images provided by the mapping controller 1593) and encodes a class (e.g., an N-channel image obtained from the M channels of the corresponding MPX image according to the selected mapping) or a set of specific characteristics expected from the input image 1540. The randomly generated noise vector 1520 may be generated from a Gaussian distribution, and the vector space may be composed of latent or hidden variables that are important to the domain but are not directly observable. The latent feature vector 1525 and the random noise vector 1520 may be combined as an input 1545 to the generator 1510. Alternatively or additionally, noise may be added within the generator 1510 in the form of dropout (e.g., stochastically dropping inputs to a layer).

[0068] The generator 1510 receives a combined input 1545 and generates an image 1530 based on a latent feature vector 1525 and a random noise vector 1520 in a problem domain (i.e., a domain of features relevant to a histochemically stained target image 1535). The classifier 1515 performs conditional image classification on both the target image 1535 and the generated image 1530 as inputs to predict 1550 the likelihood that the generated image 1530 is a real or false translation of the target image 1535. The output of the classifier 1515 depends on the size of the generated image 1530, and may be a single value or a squared activation map of values. Each value is a probability of the likelihood that a patch in the generated image 1530 is real. These values ​​can be averaged to obtain an overall likelihood or classification score, as desired. The loss functions of both the generator 1510 and the discriminator 1515 may be configured such that the loss depends on how well the discriminator 1515 does its job of predicting (1550) the likelihood that a generated image 1530 is real or a fake translation of the target image 1535. After sufficient training, the generator 1510 begins to generate generated images 1530 that are more similar to the target image 1535. Training of the GAN 1500 may proceed for a predefined number of training instances, and the resulting learned parameters may be accepted as long as one or more performance metrics (e.g., accuracy, precision, and / or recall) determined using a training or validation set exceed corresponding thresholds. Alternatively, training of the GAN 1500 may proceed until one or more performance metrics associated with a recent training iteration exceed corresponding thresholds. At this point, the generated image 1530 may be sufficiently similar to the target image 1535 that the discriminator can no longer distinguish between real and fake. Once the generator network 1510 is trained, an input set of N-channel images (obtained from the M channels of the corresponding MPX images according to the selected mapping) may be input into the GAN 1500 to convert the input set of images into a new set of generated images having similar properties to a target set of images obtained from histochemically stained slides (e.g., H&E stained).The newly generated set of images can then be used by a pathologist to visualize tissue structures during annotation of MPX images (e.g., by switching MPX images or composite H&E images on and off), to confirm a diagnosis based on MPX images, and / or to align one MPX image to another (e.g., to assist in analysis based on a composite panel of biomarkers), etc.

[0069] 14, to train the model 1440 in this example, a pre-processing stage 1490 generates samples 1445 by obtaining digital images (a source set of digital images 1430 and a target set of digital images 1435), mapping the source images to N-channel images according to a mapping selected (e.g., by a mapping controller 1493), dividing the images into a pairwise subset of images 1445a (at least one pair of a mapped image and a target image) for training (e.g., 90%) and a pairwise subset of images 1445b for validation (e.g., 10%), pre-processing the pairwise subset of images 1445a and the pairwise subset of images 1445b, expanding the pairwise subset of images 1445a, and possibly annotating the pairwise subset of images 1445a with labels 1450. The pairwise subset of images 1445a may be obtained from a data storage structure, such as, for example, a database or an image server. Each image represents a biological sample, such as a tissue.

[0070] The splitting may be performed randomly or pseudo-randomly (e.g., using 90% / 10%, 80% / 20%, or 70% / 30%), or the splitting may be performed according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, nested cross-validation, etc., to minimize sampling bias and overfitting. Pre-processing may include cropping the images so that each image contains only a single object of interest. In some cases, pre-processing may further include standardizing or rescaling (e.g., normalizing) to bring all features to the same scale (e.g., the same size scale or the same color or saturation scale). In a particular example, the image is resized with a minimum size (width or height) of a given pixel (e.g., 2500 pixels) or a maximum size (width or height) of a given pixel (e.g., 3000 pixels) to maintain the original aspect ratio.

[0071] For example, the pre-processing stage 1490 may prepare a plurality of patch images from the source set and the target set as one or more pairwise subsets of images for training data. Preparing the paired images may include accessing matched pairs of source and target images, where the source and target images of each matched pair are from slides of nearby (e.g., adjacent or nearly adjacent) sections of the same biological sample (e.g., a tumor sample), where the sections shown in the source images are stained with multiple (e.g., 2, 3, 4, 5, or 6 or more) selected IHC fluorophores, and the sections shown in the target images are stained with one or more selected histochemical stains. In one non-limiting example, each section of the source images is stained with six fluorophores (including a nuclear counterstain, e.g., DAPI), and each section of the target images is stained with H&E. Figures 5A and 5B show an example of a matched pair of a darkfield MPX image of an IHC stained tissue section (Figure 5A) and a brightfield image of a nearby (e.g., adjacent) section of the same tissue sample that was H&E stained (Figure 5B).

[0072] The pre-processing stage 1490 may then map each of the source images according to a selected mapping (e.g., by the mapping controller 1493) to obtain an N-channel input image paired with a corresponding target image, and divide each of the paired input and target images (e.g., the entire slide image) into several patches of a predetermined size (e.g., 128×128, 256×256, or another size) to generate matched pairs of patches for training. For example, it may be desirable to use only patches from regions of interest in the images, such as tumor annotations added by a reviewing pathologist. The pre-processing stage 1490 (e.g., the registration controller 1495) may perform registration and / or alignment of the paired input and target images before and / or after the images are divided into patches. Registration may include designating one image as a reference image, also called a fixed image, and applying a geometric transformation or local displacement to the other image such that the other image is aligned with the reference image. Since the histochemically stained image (i.e., the target image) provides a ground truth for training the network, it may be desirable to designate the target image as a reference image for purposes of registration and alignment. Pairs of aligned patches from the input set and the target set are selected, and this process results in one or more pairwise subsets of images for training data. A pre-processing stage 1490 may input the patch pairs into a GAN or cGAN to train the deep learning network.

[0073] Referring back to FIG. 14, the pre-processing stage 1490 may use dilation to artificially expand the size of the pairwise subset of images 1445a by creating modified versions of images in the dataset. Image data augmentation may be performed by creating transformed versions of images in the dataset that belong to the same class as the original image. The transformations include a range of operations from the field of image manipulation such as shift, flip, zoom, etc. In some examples, the operations include random erasure, shift, brightness, rotation, Gaussian blur, and / or elastic transformation to ensure that the model 1440 can perform under circumstances outside those available from the pairwise subset of images 1445a.

[0074] The training process of the model 1440 includes inputting images from the pairwise subset of images 1445a into the model 1440 and performing an iterative operation to find a set of model parameters (e.g., weights and / or biases) that minimize one or more loss or error functions of the model 1440 (e.g., a first loss function to train the classifier to maximize the probability of the image training data, and a second loss function to train the classifier to minimize the probability of the generated images sampled from the generator and maximize the probability that the classifier assigns to its own generated images). Prior to training, the model 1440 is configured by a defined set of hyperparameters, which are settings that are external to the model and can be tuned or optimized to control the behavior of the model 1440. In most models, hyperparameters are explicitly defined to control different aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to a particular scenario. For example, the hyperparameters can include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, or the number of kernels of the model. Each iteration of training may include finding a set of model parameters of the model 1440 (as constituted by the corresponding defined set of hyperparameters) such that the value of the loss or error function using the set of model parameters is smaller than the value of the loss or error function using a different set of model parameters in the previous iteration. In contrast to hyperparameters, model parameters are variables that are internal to the model and their values ​​may be estimated (e.g., learned) from training data. The loss or error function may be constructed to measure the difference between the output inferred using the model 1440 and the ground truth target images using the labels 1450.

[0075] Once a set of model parameters has been identified, the model 1440 may be trained and validated using the pairwise subset 1445b of images (a test or validation data set). The validation process involves iterative operations of inputting images from the pairwise subset 1445b of images into the model 1440 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc., to tune the hyper-parameters and ultimately find an optimal set of hyper-parameters. Once an optimal set of hyper-parameters is obtained, a reserved test set of images from the subset 1445b of images is input into the model 1440 to obtain an output (in this example, a generated image with similar characteristics to the target image), and the output is evaluated against the ground truth target images using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient, and calculating performance metrics such as error, precision, precision, recall, and receiver operating characteristic curves (ROC).

[0076] As should be understood, other training / validation mechanisms are envisioned and may be implemented within computing environment 1400. For example, model 1440 may be trained and hyper-parameters may be tuned on images from the pairwise subset of images 1445a, and images from the pairwise subset of images 1445b may be used only to test and evaluate the performance of model 1440.

[0077] The model training stage 1410 outputs trained models including one or more trained transformation models 1460 and optionally one or more image analysis models 1465. In some cases, the first model 1460a is trained to process an input image obtained from a source image 1430 of a biological specimen (tissue section). The input image is an N-channel image obtained from M channels of an MPX image of the biological sample according to a selected mapping. The input image is obtained by a transformation controller 1470 in the transformation stage 1415. The transformation controller 1470 includes program instructions for transforming the input image into a new image 1475 having the characteristics of a target image using one or more trained transformation models 1460. The characteristics of the target image are associated with an image of a tissue section stained by one or more selected histochemical stains (e.g., H&E). The transformation includes inputting the input image into a trained generator model (part of the transformation model 1460) and generating a new image 1475 by the generator model.

[0078] In some cases, the new image 1475 is sent to an analysis controller 1480 in the analysis stage 1420. The analysis controller 1480 includes program instructions for analyzing the biological sample in the new image 1475 using one or more image analysis models 1465 and outputting an analysis result 1485 based on the analysis. The analysis of the biological sample in the new image 1475 may include extracting measurements based on areas in the new image 1475, one or more cells in the new image 1475, and / or objects in the new image 1475 other than cells. Area-based measurements may include the most basic evaluation, for example, quantifying the area (in two dimensions) of a particular stain (e.g., a histochemical stain), the area of ​​fat vesicles, or other events present on the slide. Cell-based measurements aim at the identification and enumeration of objects, for example, cells. This identification of individual cells allows for the subsequent evaluation of subcellular compartments. Finally, algorithms may be utilized to evaluate events or objects present on tissue sections that may not be composed of individual cells. In certain examples, the imaging analysis algorithms are configured to provide quantitative representations of cell staining, morphology, and / or structure that can be used to locate cells or subcellular structures and ultimately aid in diagnosis and prognosis. In some cases, the imaging analysis algorithms are specifically configured for analysis of images having characteristics of a target image (e.g., an image of a histochemically stained section). For example, the analysis of the new image 1475 may include providing an automatic delineation of a ROIS and / or a region to be excluded relative to a corresponding MPX source image. In another example, the analysis of the new image 1475 may include registering the new image 1475 to another new image 1475 and applying the registration results to the corresponding MPX source image.

[0079] Although not explicitly shown, it will be understood that the computing environment 1400 may further include developer devices associated with the developer. Communication from the developer devices to the components of the computing environment 1400 may indicate the type of input images used in the model, the number and type of models used, the hyperparameters of each model, such as the learning rate and the number of hidden layers, how to format data requests, the training data used (e.g., and how to access the training data) and the validation techniques used, and / or how to configure the controller process.

[0080] A particular example of the cGAN model 1500 that can be used to train the generator network 1510 is the Pix2Pix GAN. FIG. 16 shows an example implementation of the cGAN model 1500 that uses the Pix2Pix GAN 1600 to train the generator network 1610 to convert N-channel images, each obtained from M channels of a corresponding MPX image of an IHC-stained tissue section according to a selected mapping, into a synthetic image of a histochemically stained (e.g., H&E stained) tissue section. As shown in FIG. 16, the generator network 1610 is implemented using a U-Net architecture, which includes an encoder having layers that progressively downsample the input to a bottleneck layer and a decoder having layers that progressively upsample the bottleneck output to generate the output. As shown in FIG. 16, the U-Net also includes skip connections between the encoder layer and the decoder layer having feature maps of equal size. These connections concatenate the channels of the feature map of the encoder layer with the channels of the feature map of the corresponding decoder layer. In a particular example, the generator network 1610 is updated via the L1 loss measured between the generated image and the expected output image (e.g., the "predicted image" and "ground truth" in FIG. 16).

[0081] In general, using a Pix2Pix GAN requires that matched pairs of image patches to be used to train a generator network are aligned (e.g., at the pixel level). FIG. 17 shows a flowchart of an example process 1700 for generating matched and aligned pairs of image patches from matched image pairs described herein (e.g., for training and / or validation). The process 1700 may be performed by the pre-processing stage 1490 (e.g., by the alignment controller 1495). Referring to FIG. 17, at block 1704, low-resolution versions of the matched image pairs are coarsely aligned. In one example, such coarse alignment is performed using a transformation matrix M C Mapping of M channels of MPX images of IHC stained sections (including, for example, translation and / or rotation) into N channels Nmap and apply it to a low-resolution version of the H&E stained image of a matching (e.g., close) section I H&E The transformation matrix Mc can be calculated automatically, for example, based on the contours of tissues in each image to be registered.

[0082] At block 1708, tiles are extracted from the regions of interest (ROIs) of the coarsely aligned image pair (e.g., by projecting a grid onto each image that covers the annotations and extracting corresponding tiles from each image). Nmap Tile T from ROI Nmap (e.g., size 2048x2048 pixels) and image I H&E The corresponding tile T from H&E In block 1712, full resolution versions of the extracted tile pairs are precisely aligned. Such precise alignment can be achieved, for example, by aligning one image (e.g., I Nmap ) to the tiles from the other image (e.g., I H&EThe alignment may involve scaling, deskewing, and / or warping tiles from one image to align them with corresponding tiles from another image (the reference image described above). In block 1716, the precisely aligned tiles are stitched together to obtain an aligned image pair, and in block 1720, each of the aligned images is sliced ​​into patches (e.g., 128x128 pixels in size, 256x256 pixels in size, or another size). Then, in block 1724, each aligned image (e.g., a patch P Nmap and P H&E ) are combined (e.g., stitched side-to-side) to obtain a training image. For example, a pair of matching patches each of size 128×128 pixels may be stitched together to obtain a training image of size 128×256 pixels. Or each of size 256×256 pixels may be stitched together to obtain a training image of size 256×512 pixels. In this manner, the process of stitching together precisely aligned tiles of each image to obtain an aligned image pair and combining matched patches from each of the aligned images to obtain a set of training images may be implemented to obtain a set of training data for using a Pix2Pix GAN implementation to train a generator network 1510 to convert N-channel mapped images obtained from the M channels of the corresponding MPX image into a composite image of the histochemically stained section.

[0083] Another specific example of a cGAN model 1500 that can be used to train the generator network 1510 is a cycle GAN that includes multiple generator networks and multiple discriminator networks. X and G Y and the classifier network D X and D. YIn this example, the Y region corresponds to an image representing a histochemically stained sample and the X region corresponds to an N channel image derived from the M channels of the corresponding MPX image of an IHC stained sample according to the selected mapping.

[0084] FIG. 20 illustrates the flow between the generator network and the discriminator network in the application of cycle GAN as described herein. Cycle GAN is a set of XY generator networks G Y 2024 (trained as a generator network 1510), configured and trained to convert an image of a histochemically stained sample into an image depicting an N-channel mapping from the M channels of an MPX image of an IHC stained sample. X Including 2020. Generator network G X 2020 may include one or more convolutional layers and may include a U-net or a V-net. In some cases, the generator network G X 2020 includes a feature extraction encoder, a transformer, and a decoder, each with one or more convolutional layers. X 2020 and G Y The architecture of the 2024 may be the same.

[0085] Cycle GAN is a classifier network D that distinguishes between real and fake images (e.g., real N-channel image 2012 and fake N-channel image 2016) depicting N-channel mappings from M channels of MPX images of IHC stained samples. X 2032 and another classifier network D that distinguishes between fake and real images depicting histochemically stained samples (e.g., real histochemically stained image 2004 and fake histochemically stained image 2008). Y 2028. Classifier network D X and D. Y Each of the networks D may include one or more convolutional layers and activation layers, and the classifier network D X and D. YThe architecture may be the same.

[0086] The use of cycle GANs allows for precise alignment of matched image pairs (e.g., images I and II as described herein with reference to FIGS. 17 and 18 ). Nmap and I H&E This may have the advantage that precise alignment of the images (precise alignment of the images) is not required to generate the training data. However, better results were obtained when training the generator network 1510 on image patches of paired aligned images using the Pix2Pix GAN implementation.

[0087] The methods of the present disclosure may be implemented to convert images of MPX images into corresponding composite images of H&E stained samples. Such methods may be used, for example, to assist pathologists in annotating MPX source images. Such methods may be implemented as an important part of a rapid screening process to provide reference images of underlying tissue structures without performing actual H&E staining. Furthermore, such "virtual staining" techniques may also be used to confirm diagnoses based on corresponding MPX images. Furthermore, the methods of image conversion described herein may be used to assist in the automatic alignment of MPX images to enable the analysis (e.g., co-localization, co-expression) of an extended panel of biomarkers.

[0088] 21-23 show three different examples of using an implementation of process 800 (using the Pix2Pix GAN described herein) to convert an MPX image into a corresponding synthetic H&E stained image, respectively. Each of FIGS. 21-23 shows the results obtained using each of the four exemplary source image mappings (pHE, pHE2, AAD, ADD) shown in FIGS. 10A-10D and described herein. In each of FIGS. 21-23, the top row shows the N-channel mapped source image ("Approach"), the middle row shows the corresponding generated synthetic H&E image ("Output"), and the bottom row shows the corresponding actual H&E image ("Target"). It can be seen that the output image is very similar to the target image.

[0089] V. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.

[0090] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0091] The description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of the preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0092] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. 1. A method of image transformation comprising the steps of: generating an N-channel input image based on an M-channel multiplexed immunofluorescence (MPX) image of the tissue section, where M is a positive integer and N is a positive integer greater than or equal to 2 and less than or equal to M; generating a composite image by processing the N-channel input images using a generator network trained using a training dataset including a plurality of image pairs; Including, the composite image depicts a tissue section stained with at least one histochemical stain; Each image pair in the plurality of image pairs comprises: an N-channel image generated from the MPX image of the first section of tissue; an image of a second section of the tissue stained with the at least one histochemical stain; A method comprising:

2. 2. The method of claim 1, wherein generating the N-channel input image includes, for each channel of the M-channel MPX image of the tissue section, mapping that channel to at least one of the N channels of the N-channel input image.

3. said mapping includes generating an autofluorescence image based on information from each of a plurality of channels of said MPX image of said tissue section; The method of claim 2 , wherein the N-channel input image is based on information from the autofluorescence image.

4. The method of claim 3 , wherein the autofluorescence image is based on a nonlinear combination of the multiple channels of the MPX image of the tissue section.

5. The method of claim 3 , wherein the autofluorescence image is based on a spectral distribution of autofluorescence among the multiple channels.

6. The method of claim 3 , wherein the multiple channels of the MPX image are the M channels of the MPX image of the tissue section.

7. The method of claim 1 , wherein for each image pair in the plurality of image pairs, the N-channel image is based on each channel of the M-channel MPX image of the first slice.

8. The method of claim 1 , wherein the composite image is an N-channel image.

9. The method of claim 8 , wherein each of the N-channel input image and the composite image is an RGB image.

10. The method of claim 1 , wherein the MPX image of the tissue section is a darkfield image and the composite image is a brightfield image.

11. The method of claim 1 , wherein the at least one histochemical stain is hematoxylin and eosin.

12. The method of claim 3 , wherein the N-channel image is based on the autofluorescence image and a channel of the MPX image of the tissue section corresponding to a nuclear counterstain.

13. The method of claim 3 , wherein the N-channel image is based on a linear combination of the autofluorescence image and a channel of the MPX image of the tissue section corresponding to a nuclear counterstain.

14. The method of claim 13 , wherein the N-channel image is based on an array of optical density values ​​that is based on the linear combination.

15. The method of claim 1 , wherein N is equal to three.

16. The method of claim 1 , wherein M is at least 4.

17. The method of claim 1 , wherein the generator network is trained as part of a generative adversarial network.

18. The method of claim 1 , wherein the generator network is implemented as a U-Net.

19. The method of claim 1 , wherein the generator network is implemented as an encoder-decoder network.

20. The method of claim 1 , wherein the generator network is updated via an L1 loss measured between an image produced by the generator network and a predicted output image.

21. 21. A method of determining, by a user, a diagnosis of a subject, said determining being based on the composite image of any one of claims 1 to 20.

22. 22. The method of claim 21, further comprising administering, by the user, (i) the composite image, and / or (ii) treating the subject with a compound based on the diagnosis.

23. one or more data processors; A system comprising: a storage device having a non-transitory computer readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform the method of image conversion according to any one of claims 1 to 20.

24. A computer program tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform the method of image conversion according to any one of claims 1 to 20.

Images