Transformation of histochemical stain images into composite images for immunohistochemistry (IHC)

Abstract

Diagnosis based on antigen-specific detection can be performed by other types of tissue staining with lower cost and wider availability.SOLUTION: The present disclosure relates to techniques for obtaining composite immunohistochemistry (IHC) images from histochemical stain images. Among other things, aspects of the present disclosure are directed to accessing an input image depicting a tissue section that has been stained with at least one histochemical stain, generating a synthetic image by processing the input image using a trained generator network, and outputting the synthetic image. The composite image depicts a tissue section that has been stained with at least one IHC stain directed to a first antigen, and the techniques may also include receiving an input based on an expression level of the first antigen from the composite image and / or generating a value based on the expression level of the first antigen from the composite image.SELECTED DRAWING: Figure 15A

Description

[Technical Field]

[0001] The present disclosure relates to digital pathology, and in particular to techniques that involve obtaining a composite immunohistochemistry (IHC) image from a histochemical staining image. [Background technology]

[0002] Histopathology can involve the examination of slides made from sections of tissue for a variety of reasons, such as diagnosing disease, determining response to treatment, and / or developing drugs to combat disease. Because tissue sections and their cells are essentially transparent, slide preparation typically involves staining the tissue sections to make relevant structures more visible. Digital pathology can involve scanning the stained slides to obtain digital images, which can then be examined by digital pathology image analysis and / or interpreted by a pathologist.

[0003] Some types of tissue stains are highly specific for and bind only to a particular protein (e.g., an antigen), the presence of which in a sample may indicate a particular condition (e.g., a particular type of cancer). While these stains can provide information essential for diagnosis, they are typically very expensive and require complex laboratory equipment and methods. Other types of tissue stains, which are less costly and more widely available, can provide important general information about a sample, such as the visual contrast between different structures within a cell and / or between tissues in a sample, but it has not been possible to use such stains to make diagnoses based on antigen-specific detection. Summary of the Invention

[0004] In various embodiments, a computer-implemented method of image transformation is provided, comprising: accessing an input image depicting a tissue section stained with at least one histochemical stain; generating a composite image by processing the input image using a generative network, the generative network being trained using a training dataset including a plurality of image pairs; generating the composite image; outputting the composite image; and receiving an input based on the expression level of a first antigen from the composite image, wherein the composite image depicts a tissue section stained with at least one immunohistochemical stain (IHC stain) directed to the first antigen, and each image pair of the plurality of image pairs includes an image of a first section of tissue stained with the at least one histochemical stain and an image of a second section of tissue stained with the at least one IHC stain.

[0005] In some embodiments, the method includes determining a value based on the expression level of the first antigen from the composite image. The determining can be performed, for example, by a trained network.

[0006] In various embodiments, a computer-implemented method of image transformation is provided, comprising: accessing an input image depicting a tissue section stained with at least one histochemical stain; generating a composite image by processing the input image using a generative network, the generative network being trained using a training dataset including a plurality of image pairs; generating the composite image; outputting the composite image; and receiving an input based on the expression level of a first antigen from the composite image, wherein the composite image depicts a tissue section stained with at least one IHC stain targeting the first antigen, and each image pair of the plurality of image pairs includes an image of a first section of tissue stained with at least one histochemical stain and an image of a second section of tissue stained with at least one IHC stain.

[0007] In some embodiments, the histochemical stain is hematoxylin and eosin.

[0008] In some embodiments, the first antigen is a tumor-associated antigen. For example, the first antigen may be human epidermal growth receptor 2 (HER2). In such cases, the received input value and / or the generated value may be a HER2 score.

[0009] In some embodiments, the generative network is trained as part of a generative adversarial network.

[0010] In some embodiments, for each image pair of the plurality of image pairs, the first slice image is stitched to the second slice image, and in such a case, for each image pair of the plurality of image pairs, the first slice image may be registered to the second slice image before being stitched to the second slice image.

[0011] In some embodiments, the computer-implemented method further includes determining, by a user, a diagnosis of the subject based on the composite image.

[0012] In some embodiments, the computer-implemented method further includes administering, by the user, treatment with a compound based on (i) the composite image and / or (ii) the subject's diagnosis.

[0013] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0014] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0015] Some embodiments of the present disclosure include a system including one or more data processors. Some embodiments include a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0016] The terms and expressions employed are used as terms of description and not of limitation, and no attempt is made to use terms and expressions that exclude any synonyms of the shown and described features or portions thereof, and it will be recognized that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0017] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the U.S. Patent and Trademark Office upon request and payment of the necessary fee.

[0018] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of various embodiments will be readily understood by reference to the following detailed description of the preferred embodiments, when read in conjunction with the accompanying drawings, in which: FIG. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an example workflow diagram for a digital pathology solution. [Figure 2] Figure 1 shows an example of paired images of hematoxylin and eosin (H&E) and human epidermal growth factor 2 (HER2) immunohistochemistry (IHC). [Figure 3] FIG. 1 illustrates an exemplary computing environment in accordance with some embodiments. [Figure 4] FIG. 1 illustrates a conditional GAN ​​model according to some embodiments. [Figure 5] FIG. 1 shows an example of a matched pair of H&E and HER2-IHC images. [Figure 6] FIG. 1 shows examples of tumor annotations in HER2-IHC images. [Figure 7] FIG. 1 illustrates an example of an implementation of a conditional GAN ​​model using Pix2Pix GAN, according to some embodiments. [Figure 8] 1 is a flowchart of an exemplary process according to some embodiments. [Figure 9] FIG. 1 illustrates an example of coarse alignment according to some embodiments. [Figure 10] FIG. 10 illustrates an example of extracting tiles from a full slide image according to some embodiments. [Figure 11] FIG. 1 illustrates a process for obtaining training data according to some embodiments. [Figure 12] FIG. 1 illustrates an example set of training data according to some embodiments. [Figure 13] FIG. 1 illustrates a representation of a network and network connections in a Cycle-GAN according to some embodiments. [Figure 14] FIG. 1 illustrates a flow for generating and identifying images using Cycle-GAN according to some embodiments. [Figure 15A] 1 is a flowchart of an exemplary process according to some embodiments. [Figure 15B] 1 is a flowchart of another exemplary process according to some embodiments. [Figure 16] FIG. 1 illustrates an example set of training data according to some embodiments. [Figure 17A-B] 1A-1C illustrate examples of input, target, and output images generated in accordance with various embodiments. [Figure 18A-B] 1A-1C illustrate examples of input, target, and output images generated in accordance with various embodiments. [Figure 19A-B] 1A-1C illustrate examples of input, target, and output images generated in accordance with various embodiments. [Figure 20A-B] 1A-1C illustrate examples of input, target, and output images generated in accordance with various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0020] The systems, methods, and software disclosed herein facilitate obtaining a composite IHC image from a histochemical staining image. While certain embodiments are described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The devices, methods, and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications of the exemplary method and system forms described herein may be made without departing from the scope of protection.

[0021] I. Overview Digital pathology may involve the interpretation of digital images to accurately diagnose a subject and guide treatment decision-making. In a digital pathology solution, an image analysis workflow can be established to automatically detect or classify biological objects of interest, such as positive tumor cells, negative tumor cells, etc. FIG. 1 shows an example diagram of a workflow 100 of a digital pathology solution. The workflow 100 of the digital pathology solution includes obtaining a tissue slide at block 105, scanning a preselected region or the entire tissue slide with a digital image scanner (e.g., a whole slide image (WSI) scanner) to obtain a digital image at block 110, performing image analysis on the digital image using one or more image analysis algorithms at block 115, and scoring the objects of interest based on the image analysis (e.g., quantitative or semi-quantitative scoring such as positive, negative, moderate, weak, etc.).

[0022] For example, evaluation of tissue changes caused by disease may be performed by examining thin tissue sections. A tissue sample (e.g., a tumor sample) may be sliced ​​to obtain a series of sections, each having a thickness of, for example, 4 to 5 microns. Because tissue sections and their cells are virtually transparent, slide preparation typically involves staining the tissue sections to make relevant structures more visible. For example, sections of different tissues may be stained with one or more different stains to represent different characteristics of the tissue.

[0023] After each section is mounted on a slide, it is scanned to create a digital image, which can then be examined by digital pathology image analysis and / or interpreted by a pathologist (e.g., using image viewer software). The pathologist may review and manually annotate the digital image of the slide (e.g., tumor area, necrosis, etc.) to enable the use of image analysis algorithms to extract meaningful quantitative measures (e.g., to detect and classify biological objects of interest). Traditionally, a pathologist may manually annotate each sequential image of multiple tissue sections from a tissue sample to identify the same aspect for each sequential tissue section.

[0024] One type of tissue stain is histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes) to stain tissue structures. Histochemical stains can be used to demonstrate general aspects of tissue morphology and / or cellular histology (e.g., to distinguish cell nuclei from cytoplasm, to show lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), periodic acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of histochemical staining reagents (e.g., dyes) is typically about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian blue, phosphomolybdic acid (PMA)) can have molecular weights of up to 2,000 or 3,000 kD. One example of a high molecular weight histochemical stain is alpha amylase (approximately 55 kD), which can be used to indicate glycogen.

[0025] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which specifically uses a primary antibody that binds to a target antigen of interest (also called a biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The use of IHC for tissue staining typically requires very expensive reagents and more complex laboratory equipment and methods than histochemical staining. The molecular weight of IHC reagents is much higher than that of histochemical staining reagents, as antibodies have a molecular weight of approximately 150 kD or more.

[0026] The expression level of the human epidermal growth factor receptor 2 (HER2) biomarker in tumors is an important biomarker for the diagnosis of several types of cancer, including breast cancer. Whether a tumor is HER2-positive (HER2+) or HER2-negative (HER2-) and how strongly it is expressed can indicate whether a particular drug or other therapy is likely to be effective in treating the cancer. The following criteria are used to assign a HER2 score to slides of sections of tumor samples that have been HER2-IHC-stained: TIFF2026004325000002.tif39170

[0027] Current methods for diagnosing HER2+ breast cancer generally rely on pathological evaluation of H&E-stained slides and multiple IHC-stained slides. To confirm a breast cancer diagnosis, the creation of multiple additional tissue sections for HER2-IHC slides is typically required. Figure 2 shows an example of an image of an H&E-stained tumor section (left) and an image of an adjacent HER2-IHC-stained section (right). Creating multiple stained images can be labor-intensive and incurs additional costs. Furthermore, it requires a pathologist to review additional tissue slides and annotations.

[0028] In many diagnostic scenarios (e.g., cancer diagnosis), it is common for H&E-stained specimens to be prepared for every subject because preparing such specimens is routine, easy, widely practiced, available, and inexpensive. Hematoxylin stains cell nuclei blue, while eosin stains extracellular matrix and cytoplasm pink, and other structures may be stained with different shades, hues, and / or combinations of pink and blue. While H&E staining is useful for general tissue identification and cellular anatomy, it cannot provide the specific information needed to support certain diagnostic assessments, such as distinguishing between different types of cancer (e.g., HER2 scoring), as can be provided by IHC.

[0029] To overcome these and other limitations, techniques are disclosed herein for generating a composite image showing an IHC-stained sample from an image of a histochemically stained sample (e.g., an H&E-stained sample). The generation of such a composite image can support the assessment of biomarker expression levels in a sample without the need to create and image a corresponding IHC-stained sample.

[0030] The generation of synthetic images can be performed by a trained generative network, which can include parameters learned during training of a generative adversarial network (GAN). The GAN can further include a discriminative network configured to predict whether an input image is false (i.e., generated by the generative network) or true (i.e., represents an actual image collected from a subject). Feedback based on the accuracy of these predictions can be provided to the generative network during training.

[0031] One exemplary embodiment of the present disclosure is directed to a method of image transformation, comprising: accessing an input image depicting a tissue section stained with at least one histochemical stain; generating a composite image by processing the input image using a generative network; outputting the composite image; and receiving an input based on an expression level of a first antigen from the composite image, wherein the composite image depicts a tissue section stained with at least one IHC stain targeting the first antigen, and the generative network has been trained using a training dataset comprising a plurality of image pairs, each image pair of the plurality of image pairs comprising an image of a first section of tissue stained with at least one histochemical stain and an image of a second section of tissue stained with at least one IHC stain.

[0032] Another exemplary embodiment of the present disclosure is directed to a method of image transformation, comprising: accessing an input image depicting a tissue section stained with at least one histochemical stain; generating a composite image by processing the input image using a generative network; outputting the composite image; and generating a value from the composite image based on an expression level of a first antigen, wherein the composite image depicts a tissue section stained with at least one IHC stain targeting the first antigen, and wherein the generative network has been trained using a training dataset comprising a plurality of image pairs, each image pair of the plurality of image pairs comprising an image of a first section of tissue stained with at least one histochemical stain and an image of a second section of tissue stained with at least one IHC stain.

[0033] Advantageously, the image conversion methods described herein enable the use of H&E and synthetic IHC data to aid pathologists in the efficient diagnosis of cancer (e.g., breast cancer) subtypes. Such methods can be implemented, for example, as an important part of a rapid screening method that can be used to identify subjects with HER2 3+ tumors (e.g., among subjects with breast cancer) without performing actual IHC staining. Furthermore, such "virtual staining" techniques can also be combined with other artificial intelligence (AI) techniques to increase the veracity of AI systems (e.g., to increase the defensibility and veracity of the output algorithm). Furthermore, the image conversion methods described herein can be used to generate large amounts of imaging data (e.g., numerous synthetic HER2-IHC images) for algorithm validation and training, thereby reducing the cost and time of algorithm development.

[0034] II. Definition As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a part of that something.

[0035] As used herein, the terms "substantially," "approximately," and "about" are defined as largely, but not necessarily entirely, of (and entirely including) what is specified, as would be understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" may be replaced with "within [percentage]" of what is specified, where percentage includes 0.1 percent, 1 percent, 5 percent, and 10 percent.

[0036] As used herein, the terms "sample," "biological sample," or "tissue sample" refer to any sample containing biomolecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans; domestic animals such as cats, dogs, horses, cows, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and needle biopsies of tissue), cell samples (such as cytological smears such as Pap smears, or blood smears, or samples of cells obtained by microdissection), or fractions, fragments, or organelles (such as those obtained by lysing cells and centrifuging or otherwise separating their components). Other examples of biological samples include blood, serum, urine, semen, fecal material, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, a swab (such as a buccal swab), or any substance containing a biological molecule removed from a first biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample (such as a homogenized or liquefied sample) made from a tumor or portion thereof obtained from a subject.

[0037] III. Techniques for digital pathology image conversion Reliable results are important for the availability of cancer treatments, such as HER2-targeted therapy. In pathology, for example, HER2-IHC is known to be sensitive to pre-analytical variables. In one application of the techniques described herein, images of H&E slides and composite IHC images derived from those images are used to support breast cancer diagnosis and immunotherapy by predicting HER2 scores / staining intensity levels using the generated composite IHC images.

[0038] 3 illustrates an exemplary computing environment 300 (i.e., a data processing system) for converting an input image showing a tissue section stained with at least one histochemical stain into a composite image showing the tissue section stained with at least one IHC stain, according to various embodiments. As shown in FIG. 3, the conversion of the input image performed by the computing environment 300 in this example includes several stages: an image storage stage 305, a preprocessing stage 390, a model training stage 310, a transformation stage 315, and an analysis stage 320. The image storage stage 310 may include one or more digital image scanners or databases 325 that are accessed (e.g., by the preprocessing stage 390) to provide a source set of digital images 330 and a target set of digital images 335 from preselected regions across a biological specimen slide (e.g., a tissue slide).

[0039] The model training stage 310 constructs and trains one or more models 340a-340n (where "n" represents any natural number) that are used in other stages (which may be individually referred to herein as models 340 or collectively referred to as models 340). The models 340 may be machine learning ("ML") models that may include convolutional neural networks ("CNNs"), inception neural networks, residual neural networks ("Resnets"), U-Nets, V-Nets, single-shot multibox detection ("SSD") networks, recurrent neural networks ("RNNs"), deep neural networks, rectified linear units ("ReLUs"), long short-term memory ("LSTMs") models, gated recurrent unit ("GRU") models, etc., or any combination thereof. In various embodiments, the generative model is composed of parameters learned by a training model 340 that is capable of learning any type of data distribution using unsupervised learning, such as a generative adversarial network (GAN), a deep convolutional generative adversarial network ("DCGAN"), a variational autoencoder (VAE), a hidden Markov model ("HMM"), a Gaussian mixture model, a Boltzmann machine, or the like, or a combination of one or more of such techniques, e.g., a VAE-GAN. The computing environment 300 may use the same or different types of models to convert source images into generated images. In one particular case, the generative model is composed of parameters learned by training the model 340, which is a GAN constructed with a loss function that attempts to classify whether an output image is real or fake, while simultaneously training the generative model to minimize this loss.

[0040] In the exemplary embodiment shown in FIG. 4 , the model 340 trained to provide learned parameters is a conditional GAN ​​("cGAN") 400, an extension of the GAN model, which generates images with certain conditions or attributes. The cGAN learns a structured loss that penalizes output combinations. Referring to FIG. 4 , the cGAM 400 includes a generator 410 and a discriminator 415. The generator 410 is a neural network (e.g., a CNN) that uses a randomly generated noise vector 420 and a latent feature vector (or one-dimensional vector) 425 (conditions, e.g., a source image in the current example) as input data and feedback from the discriminator 415 and generates a new image 430 that is as close as possible to a true target image 435. The discriminator 415 is a neural network (e.g., a CNN) configured as a classifier to determine whether the generated image 430 from the generator 410 is a real image or a fake image. A latent feature vector 425 or condition is taken from a source image or set of source images 440 (e.g., images from one or more digital scanners, from an image server, etc.) that encodes a class (e.g., histochemically stained images) or a set of specific characteristics expected from the source images 440. A randomly generated noise vector 420 may be generated from a Gaussian distribution, and the vector space may be composed of latent or hidden variables that are domain-important but not directly observable. The latent feature vector 425 and the random noise vector 420 may be combined as input 445 to the generator 410. Alternatively or additionally, noise may be added within the generator 410 in the form of dropout (e.g., stochastic descent input to a layer).

[0041] The generator 410 receives a combined input 445 and generates an image 430 based on a latent feature vector 425 and a random noise vector 420 in a problem domain (i.e., a region of characteristics associated with the IHC-stained target image 435). The classifier 415 performs conditional image classification by taking both the target image 435 and the generated image 430 as input and predicts 450 the likelihood that the generated image 430 is a true or false transformation of the target image 435. The output of the classifier 415 depends on the size of the generated image 430 and may be a single value or a squared activation map of values. Each value is a probability for the likelihood that a patch in the generated image 430 is true. These values ​​may be averaged to provide an overall likelihood or a classification score, as appropriate. The loss functions for both the generator 410 and the classifier 415 may be configured such that the loss depends on how well the classifier 415 does its job 450 of predicting the likelihood that the generated image 430 is a true or false transformation of the target image 435. After sufficient training, the generator 410 will begin to make the generated images 430 look more like the target images 435. Training of the GAN 400 may proceed for a predetermined number of training instances, and the resulting learned parameters may be accepted as long as one or more performance metrics (e.g., accuracy, precision, and / or recall) determined using the training or validation sets exceed corresponding thresholds. Alternatively, training of the GAN 400 may proceed until one or more performance metrics associated with a recent training iteration exceed corresponding thresholds. At this point, the generated images 430 may be sufficiently similar to the target images 435 that the classifier can no longer distinguish between true and false. Once the generative network 410 is trained, a source set of images obtained from slides that have been histochemically stained (e.g., H&E stained) may be input into the GAN 400 to transform the source set of images into a new generated set of images that have properties similar to a target set of images obtained from slides that have been immunostained (e.g., HER2-IHC stained).The resulting set of new images can then be evaluated by a pathologist (e.g., to determine a HER2 score), analyzed using currently available computerized digital pathology image analysis algorithms, and / or used as input to train and / or validate further networks, etc.

[0042] 3 , to train model 340 in this example, preprocessing stage 390 generates samples 345 by obtaining digital images (source set of digital images 330 and target set of digital images 335), dividing the images into pairwise subsets 345a of images (at least one pair of source and target images) for training (e.g., 90%) and pairwise subsets of images (e.g., 10%) for validation, preprocessing pairwise subsets 345a and pairwise subsets 345b of images, and possibly annotating pairwise subsets 345a of images with labels 350. Pairwise subsets 345a of images may be obtained from a data storage structure, such as a database or an image server. Each image represents a biological sample, such as tissue.

[0043] The splitting may be performed randomly or pseudorandomly (e.g., using 90% / 10%, 80% / 20%, or 70% / 30%), or may be performed according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation to minimize sampling bias and overfitting. Preprocessing may include cropping the images so that each image contains only a single object of interest. In some cases, preprocessing may further include standardization or normalization to bring all features to the same scale (e.g., the same size scale or the same color or saturation scale). In certain cases, the images are resized to a minimum size (width or height) or a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels), while maintaining the original aspect ratio.

[0044] For example, the preprocessing stage 390 may create multiple patch images from the source set and the target set as one or more pairwise subsets of images for training data. Creating paired images may include accessing matched pairs of source and target images, where the source and target images are from slides of adjacent sections of the same biological sample (e.g., a tumor sample), with the sections in the source images stained with one or more selected histochemical stains and the sections in the target images stained with one or more selected IHC stains. In one non-limiting example, the sections in each of the source images are stained with H&E, and the sections in each of the target images are stained with HER2-IHC. Figure 5 shows an example of a matched pair of an image of a HER2-IHC-stained slide (left) and an image of an H&E-stained slide (right).

[0045] The preprocessing stage 390 may also divide each pair of images (e.g., a full slide image) into a number of patches of a predetermined size (e.g., 128 x 128, 256 x 256, or another size) to create matched pairs of patches for training. For example, it may be desirable to use only patches that are from regions of interest within the image, such as tumor annotations added by a reviewing pathologist. FIG. 6 shows an example of a matched pair of an image of a HER2-IHC-stained slide (left) and an image of an H&E-stained slide (right), where the image of the HER2-IHC-stained slide is annotated with two curved lines drawn in red. The preprocessing stage 390 may align and / or register the pair of images before and / or after the images are divided into patches. Registration may include designating one image as a reference image, also called a fixed image, and applying a geometric transformation or local displacement to the other image so that the other image is aligned with the reference image. Because the IHC image (i.e., the target image) provides the ground truth for training the network, it may be desirable to designate the target image as a reference image for alignment and registration purposes. Aligned pairs of patches from the source set and the target set are selected, and this process results in one or more pairwise subsets of images for training data. In a preprocessing stage 390, the patch pairs may be input into a GAN or cGAN to train the deep learning network.

[0046] Referring back to FIG. 3, in the preprocessing stage 390, augmentation may be used to artificially expand the size of the pairwise subset of images 345a by creating modified versions of images in the dataset. Image data augmentation may be performed by creating transformed versions of images in the dataset that belong to the same class as the original image. Transformations include a range of operations from the field of image manipulation, such as shifting, flipping, and zooming. In some cases, operations include random erasure, shifting, brightness, rotation, Gaussian blur, and / or elastic transformations that ensure the model 340 can perform under conditions outside of those available from the pairwise subset of images 345a.

[0047] The training process of model 340 involves selecting hyperparameters for model 340 and iterating through inputting images from pairwise subsets of images 345 a into model 340 to find a set of model parameters (e.g., weights and / or biases) that minimize one or more loss or error functions of model 340 (e.g., a first loss function to train the classifier to maximize the probability of the image training data, and a second loss function to train the classifier to minimize the probability of generated images sampled from the generator and maximize the probability the classifier assigns to its own generated images). Hyperparameters are settings that can be adjusted or optimized to control the behavior of model 340. Most models explicitly define hyperparameters that control different aspects of modeling, such as memory or execution costs. However, additional hyperparameters can be defined to adapt the model to specific scenarios. For example, hyperparameters may include the number of hidden units of the model, the learning rate of the model, the width of the convolution kernel, or the number of kernels for the model. Each iteration of training may involve finding a set of model parameters for model 340 (composed of a defined set of hyperparameters) such that the value of the loss or error function using that set of model parameters is less than the value of the loss or error function using a different set of model parameters in the previous iteration. The loss or error function may be constructed to measure the difference between the output inferred using model 340 and the ground truth target image using landmarks 350.

[0048] Once a set of model parameters has been identified, model 340 can be trained and validated using pairwise subset 345b of images (a test or validation dataset). The validation process involves iteratively inputting images from pairwise subset 345b of images into model 340 using a validation technique such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation to adjust the hyperparameters and ultimately find an optimal set of hyperparameters. Once an optimal set of hyperparameters is obtained, a set of reserved test images from subset 345b of images are input into model 345 to obtain output (in this example, generated images with similar characteristics to the target images), which is evaluated against ground truth target images using correlation techniques such as Bland-Altman and Spearman rank correlation coefficients, and calculated performance metrics such as error, precision, precision, recall, and receiver operating characteristic curves (ROC).

[0049] As will be appreciated, other training / validation mechanisms are contemplated and may be implemented within computing environment 300. For example, model 340 may be trained and hyperparameters may be tuned on images from image pairwise subset 345a, and images from image pairwise subset 345b may be used only to test and evaluate the performance of model 340.

[0050] The model training stage 310 outputs trained models, including one or more trained transformation models 360 and, optionally, one or more image analysis models 365. In some cases, a first model 360a is trained to process a source image 330 of a biological specimen. The source image 330 is an image of a section stained with one or more selected histochemical stains. The source image 330 is obtained by a transformation controller 370 within the transformation stage 315. The transformation controller 370 includes program instructions for using the one or more trained transformation models 360 to transform the source image 330 into a new image 375 having properties of the target image. The properties of the target image are associated with the image of the section stained with one or more selected IHC stains. The transformation includes (i) inputting randomly generated noise vectors and latent feature vectors from the source image 330 as input data into a generative model (part of transformation model 360); (ii) generating a new image 375 using the generative model; and (iii) inputting the new image 375 into a discriminative model (another part of model 360), and using the discriminative model to generate a probability (e.g., a number between 1 and 0) of whether the new image 375 is real or fake, where real means that the image has similar properties to those of the target image and fake means that the image does not have similar properties to those of the target image.

[0051] In some cases, the new image 375 is transmitted to an analysis controller 380 within the analysis stage 320. The analysis controller 380 includes program instructions for analyzing the biological sample in the new image 375 using one or more image analysis models 365 and outputting analysis results 385 based on the analysis. Analysis of the biological sample in the new image 375 may include extracting measurements based on regions within the new image 375, one or more cells within the new image 375, and / or objects in the new image 375 apart from cells. Region-based measurements may include the most basic assessment, for example, quantifying the area (two-dimensional) of a particular stain (e.g., chemical or IHC stain) or other event present on a slide. Cell-based measurements aim to identify and enumerate objects, such as cells. Such identification of individual cells allows for subsequent assessment of subcellular compartments. Finally, algorithms are available for assessing events or objects present on tissue sections that may not consist of individual cells. In some cases, the image analysis algorithm is configured to provide a quantitative representation of cellular staining, morphology, and / or architecture that can ultimately be used to discover cellular or subcellular structures and support diagnosis and prognosis. In some cases, the image analysis algorithm is specifically configured for analyzing images having the characteristics of a target image (e.g., an image of an IHC-stained section). For example, analyzing the new image 375 may include calculating, from the new image 375, an expression level of an antigen targeted by at least one IHC stain. In another example, analyzing the new image 375 may include calculating, from the new image 375, a score (e.g., a HER2 score) based on such expression level.

[0052] Although not explicitly shown, it will be understood that computing environment 300 may further include a development device associated with the developer. Communications from the development device to components of computing environment 300 may indicate what types of input images will be used for the models, the number and type of models used, the hyperparameters of each model, e.g., the learning rate and number of hidden layers, how data requests are formatted, what training data will be used (e.g., and how the training data can be accessed), and what validation techniques will be used, and / or how the controller process is configured.

[0053] One specific example of a cGAN model 400 that can be used to train a generative network 410 is a Pix2Pix GAN. FIG. 7 illustrates an example implementation of a cGAN model 400 that uses a Pix2Pix GAN 700 to train a generative network 710 to convert images of H&E-stained tumor sections into synthetic images of IHC-stained (e.g., HER2-IHC-stained) tumor sections. As shown in FIG. 7, the generative network 710 is implemented using a U-Net architecture that includes an encoder with layers that gradually downsample the input to a bottleneck layer and a decoder with layers that gradually upsample the bottleneck output to produce an output. As shown in FIG. 7, the U-Net also includes skip connections with equal-sized feature maps between the encoding and decoding layers, and these connections connect channels of the feature maps of the encoding layer with channels of the feature maps of the corresponding decoding layer. In a particular example, the generative network 710 is updated via the L1 loss measured between the generated image and the expected output image (e.g., the “predicted image” and the “ground truth,” respectively, in FIG. 7).

[0054] In general, the use of Pix2Pix GANs requires that matched pairs of image patches that will be used to train the generative network be registered (e.g., at the pixel level). FIG. 8 shows a flowchart of an example process 800 for generating matched and registered pairs of image patches from matched image pairs as described herein (e.g., for training and / or validation). Process 800 may be performed by pre-processing stage 390. Referring to FIG. 8, at block 804, low-resolution versions of the matched image pairs are coarsely aligned. FIG. 9 shows a diagram illustrating a process for generating a transformation matrix M (e.g., including translation and / or rotation) C Image I shows a low-resolution version of an H&E-stained section. H&E and apply this to annotated images of IHC stained sections, I IHC An example of such a coarse registration is shown below, by aligning the transformation matrix M C can be calculated automatically, for example, based on the contours of the tissue in each of the images to be aligned.

[0055] In block 808, tiles are extracted from the region of interest (ROI) of the coarsely aligned image pair (e.g., by projecting a grid onto each image that covers the annotation and extracting corresponding tiles from each image). IHC A tile of size 2048 x 2048 pixels from the ROI P IHC and Image I H&E The corresponding tile P of size 2048x2048 pixels from H&E In block 812, the extracted tile pairs of the full resolution versions are precisely aligned. Such precise alignment can be achieved, for example, by using one of the images (e.g., I H&E ) to scale, deskew, and / or warp tiles from other images (e.g., I IHCThe finely aligned tiles may be registered to corresponding tiles from other images (reference images discussed above). In block 816, the finely aligned tiles are stitched together to obtain a registered image pair, and in block 820, each of the registered images is sliced ​​into patches (e.g., of size 128x128, 256x256, or another size), and in block 824, matched patches from each registered image are combined to obtain training images. Figure 11 shows, from top to bottom, the process of stitching finely aligned tiles from each image together to obtain a registered image pair and combining matched patches from each of the registered images to obtain a set of training images. Figure 12 shows an example set of training data for using a Pix2Pix GAN implementation to train the generative network 410 to convert images of H&E-stained sections into composite images of HER-IHC-stained sections, where each training image is labeled with the HER2 scores of its component HER-IHC-stained patches.

[0056] Another specific example of a cGAN model 400 that can be used to train a generative network 410 is a Cycle-GAN, which includes multiple generative networks and multiple discriminative networks. X and G Y and identification network D X and D Y In this example, the Y region corresponds to an image showing an IHC stained sample, and the X region corresponds to an image showing a histochemically stained sample.

[0057] 14 shows the flow of the generative and discriminative networks in the application of Cycle-GAN described herein. Cycle-GAN is trained as a generative network 410, which is an XY generative network G Y 1424, and a YX generating network G configured and trained to convert an image of an IHC stained sample into an image of a histochemically stained sample. X 1420. Generative network GX 1420 may include one or more convolutional layers and may include a U-net or a V-net. In some cases, the generative network G X 1420 includes a feature extraction encoder, a transformer, and a decoder, each of which has one or more convolutional layers. X 1420 and G Y The architecture of 1424 may be the same.

[0058] Cycle-GAN is a classification network D that distinguishes between real and fake images (e.g., real histochemically stained image 1412 and fake histochemically stained image 1416) showing histochemically stained samples. X 1432 and another classification network D that distinguishes between fake and real images showing IHC-stained samples (e.g., real IHC-stained image 1404 and fake IHC-stained image 1408). Y 1428 and the identification network D X and D Y may include one or more convolutional layers and activation layers, and the discriminative network D X and D Y The architecture may be the same.

[0059] The use of CycleGAN allows for precise registration of matched image pairs (e.g., image I as described herein with reference to FIG. 8). H&E and I IHC This may have the advantage that precise registration of the paired images (precise registration of the paired images) is not required to generate the training data. However, better results were obtained when using a Pix2Pix GAN implementation to train the generative network 410 on image patches of paired registration images.

[0060] FIG. 15A shows a flowchart of an exemplary process 1500 for converting a source image (e.g., a source image from a set of source images being processed) into a new image (e.g., a new image from a set of generated new images) having similar characteristics to a target image. Process 1500 may be performed using one or more computing systems, models, and networks (e.g., as described herein with reference to FIGS. 3, 4, and 7). Referring to FIG. 15A, at block 1504, an input image illustrating a tissue section stained with at least one histochemical stain is accessed. At block 1512, a composite image is generated by processing the input image using a generative network. The composite image illustrates the tissue section stained with at least one IHC stain directed to a first antigen. The generative network has been trained using a training dataset including a plurality of image pairs, where each pair includes an image of a first section of tissue stained with at least one histochemical stain and an image of a second section of tissue stained with at least one IHC stain. At block 1516, the composite image is output. At block 1520, an input based on the expression level of the first antigen shown in the composite image is received. For example, the input may be received from a user via a keyboard, a touch screen, etc. In some embodiments, process 1500 also includes determining a value based on the expression level of the first antigen from the composite image. The determining may be performed, for example, by a trained network.

[0061] FIG. 15B shows a flowchart of another exemplary process 1502 for converting a source image (e.g., a source image from a set of source images being processed) into a new image (e.g., a new image from a set of generated new images) having similar characteristics to the target image. Process 1500 may be performed using one or more computing systems, models, and networks (e.g., as described herein with reference to FIGS. 3, 4, and 7). Referring to FIG. 15B, at block 1504, an input image illustrating a tissue section stained with at least one histochemical stain is accessed. At block 1512, a composite image is generated by processing the input image using a generative network. The generative network has been trained using a training dataset including a plurality of image pairs. An expression level of an antigen targeted by the at least one IHC stain is determined from the composite image. At block 1516, the composite image is output. At block 1524, a value based on the expression level of the first antigen is generated from the composite image.

[0062] In some embodiments of process 1500 or 1502, the histochemical stain is hematoxylin and eosin.

[0063] In some embodiments of process 1500 or 1502, the first antigen is a tumor-associated antigen. For example, the first antigen may be human epidermal growth receptor 2 (HER2). In such cases, the received input value and / or the generated value may be a HER2 score.

[0064] In some embodiments of process 1500 or 1502, the generative network was trained as part of a generative adversarial network (e.g., a cGAN, Pix2Pix GAN, or CycleGAN).

[0065] In some embodiments of process 1500 or 1502, for each image pair of the multiple image pairs, the first slice image is stitched to the second slice image. In such a case, for each image pair of the multiple image pairs, the first slice image may be registered to the second slice image before being stitched to the second slice image.

[0066] Methods according to the present disclosure may be implemented to convert images of histochemically stained samples, which may be readily available, into synthetic images of IHC-stained samples (which may be more difficult, expensive, and / or time-consuming to obtain non-virtually). Such methods may be used, for example, to enable pathologists to use H&E and synthetic IHC data to aid in the efficient diagnosis of cancer (e.g., breast cancer) subtypes. Such methods may be implemented as an important part of a rapid screening method to identify samples in which specific biomarkers are expressed without performing actual IHC staining. Furthermore, such "virtual staining" techniques can also be combined with other artificial intelligence (AI) techniques to increase the veracity of AI systems (e.g., to increase the defensibility and veracity of the output algorithm). Furthermore, the image conversion methods described herein may be used to generate large amounts of imaging data (e.g., numerous synthetic HER2-IHC images) for algorithm validation and training, thereby reducing the cost and time of algorithm development.

[0067] Figure 16 shows an example of an actual matched pair of images of a tumor section stained with H&E and an adjacent section of the same tumor stained with HER2-IHC and showing the HER2 score, for each of the four HER2 scores shown in the table above. A Pix2Pix implementation of the system described herein was trained and tested using 7,472 pairs of image patches of size 128x128 pixels using four different HER2 scores. The Pix2Pix implementation was also trained and tested using 1,900 pairs of image patches of size 256x256 pixels using four different HER2 scores. The training and testing datasets were split to use 80% and 20%, respectively, and the GAN network parameters were Adam optimizer with a learning rate of 0.0002 and epochs of 100 and 200, respectively.

[0068] Figures 17A and 17B, 18A and 18B, 19A and 19B, and 20A and 20B show the input image (H&E), target image (HER-IHC), and output image (synthetic HER-IHC) for two different image patch sizes for each of the four HER2 scores, respectively. Testing of randomly assembled real images and synthetic HER-IHC images demonstrated that the synthetic images were indistinguishable from real HER-IHC images by pathologists (percentage of test sets correctly identified by pathologists as true or synthetic: 45.8%), and that the synthetic images could represent HER2 scores in different categories, particularly strong HER2 intensity levels, with high agreement with pathologists (agreement with pathologists: 87.5%).

[0069] V. Additional considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0070] The terms and expressions employed are used as terms of description and not of limitation, and no attempt is made to use terms and expressions that exclude any synonyms of the shown and described features or portions thereof, and it will be recognized that various modifications are possible within the scope of the claimed invention. Thus, while the claimed invention has been specifically disclosed by embodiments and optional features, it should be understood that modifications and variations of the concepts disclosed herein may be adopted by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0071] This specification provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0072] In the following description, specific details are set forth to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. 1. A method of image transformation, comprising: accessing an input image showing a tissue section stained with at least one histochemical stain; generating a synthetic image by processing the input image using a generative network, the generative network being trained using a training dataset including a plurality of image pairs; outputting the composite image; receiving an input based on an expression level of a first antigen from the composite image; the composite image shows a tissue section stained with at least one immunohistochemical stain (IHC stain) directed to the first antigen; For each image pair of the plurality of image pairs, the pair comprises: an image of a first section of tissue stained with the at least one histochemical stain; and an image of a second section of the tissue stained with the at least one IHC stain. The method wherein the image of the first slice is registered to the image of the second slice.

2. 10. The method of claim 1, wherein the received input is a human epidermal growth receptor 2 (HER2) score.

3. The method of claim 1 or 2, further comprising determining a value based on the expression level of the first antigen from the composite image.

4. The method of claim 3 , wherein the determining is performed by a trained network.

5. 5. The method of claim 3 or 4, wherein the determined value is a human epidermal growth receptor 2 (HER2) score.

6. 1. A method of image transformation, comprising: accessing an input image showing a tissue section stained with at least one histochemical stain; generating a synthetic image by processing the input image using a generative network, the generative network being trained using a training dataset including a plurality of image pairs; outputting the composite image; generating a value based on the expression level of the first antigen from the composite image; the composite image shows a tissue section stained with at least one immunohistochemical stain (IHC stain) directed to the first antigen; For each image pair of the plurality of image pairs, the pair comprises: an image of a first section of tissue stained with the at least one histochemical stain; and an image of a second section of the tissue stained with the at least one IHC stain. The method wherein the image of the first slice is registered to the image of the second slice.

7. 7. The method of claim 6, wherein the value generated is a human epidermal growth receptor 2 (HER2) score.

8. 8. The method of any one of claims 1 to 7, wherein the histochemical stain is hematoxylin and eosin.

9. 9. The method of claim 1, wherein the first antigen is a tumor-associated antigen.

10. 10. The method of claim 9, wherein the first antigen is human epidermal growth receptor 2 (HER2).

11. The method of claim 1 , wherein the generative network is trained as part of a generative adversarial network.

12. The method of claim 1 , wherein for each image pair of the plurality of image pairs, the image of the first slice is stitched to the image of the second slice.

14. 14. The method of any one of claims 1 to 13, wherein the histochemical staining does not include any antibodies.

15. 15. The method of any one of claims 1 to 14, wherein the histochemical staining does not include any reagent with a molecular weight above 100 kilodaltons.

16. 16. The method of any one of claims 1 to 15, wherein the histochemical staining does not include any reagents with a molecular weight greater than 5 kilodaltons.

17. 17. The method of any one of claims 1 to 16, wherein the histochemical staining does not include any reagents with a molecular weight greater than 2 kilodaltons.

18. 18. The method of any one of claims 1 to 17, wherein the composite image is indistinguishable by a human reader from an image showing a tissue section that has actually been stained with the at least one IHC stain.

19. The method of any one of claims 1 to 18, wherein the generating network is implemented as a U-Net.

20. 20. The method of claim 1, wherein the generating network is implemented as an encoding / decoding network.

21. 21. The method of claim 1, wherein the generative network is updated by an L1 loss measured between an image produced by the generative network and a predicted output image.

22. 22. The method of any one of claims 1 to 21, further comprising determining, by a user, a diagnosis of a subject based on the composite image.

23. 23. The method of claim 22, further comprising administering, by the user, treatment with a compound based on (i) the composite image and / or (ii) the diagnosis of the subject.

24. one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform the method of any one of claims 1 to 6.

25. one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform the method of any one of claims 1 to 23 and 28.

26. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform the method of any one of claims 1 to 6.

27. 29. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform the method of any one of claims 1 to 23 and 28.

28. 24. The method of any one of claims 1 to 23, wherein the generative network is trained as part of a Pix2Pix generative adversarial network.

Images