Label-free virtual immunohistochemistry of tissues using deep learning

Abstract

A deep learning-based virtual HER2 IHC staining method uses a conditional generative adversarial network trained to rapidly convert autofluorescence microscopy images of unlabeled / unlabeled breast tissue sections into brightfield equivalent microscopy images that match standard HER2 IHC staining performed chemically on the same tissue sections. The effectiveness of the staining framework was demonstrated by a quantitative analysis of blindly graded HER2 scores of virtual stained HER2 whole slide images (WSIs) and immunohistochemically stained HER2 whole slide images. Another quantitative blinded study revealed that the virtual stained HER2 images exhibited comparable staining quality in levels of nuclear detail, membrane clarity, and absence of staining artifacts to their immunohistochemically stained counterparts. This virtual staining framework can avoid costly, laborious, and time-consuming IHC staining procedures in laboratories and can be extended to other types of biomarkers to facilitate IHC tissue staining and biomedical workflows.

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 287,006, filed December 7, 2021, which is incorporated by reference in its entirety. Priority is claimed pursuant to 35 U.S.C. § 119 and other applicable law.

[0002] The technical field relates generally to methods and systems used to image unstained (i.e., label-free) tissue, such as, in one embodiment, breast tissue. In particular, the technical field relates to microscopy methods and systems that utilize deep neural network learning to digitally or virtually stain images of unstained or unlabeled tissue that substantially resemble immunohistochemistry (IHC) staining of the tissue. One example includes IHC staining of breast tissue and the human epidermal growth factor receptor 2 (HER2) biomarker.

[0003] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT This invention was made with Government support awarded by the National Science Foundation under Grant No. 1926371. The Government has certain rights in this invention. [Background technology]

[0004] Immunohistochemistry (IHC) staining of tissue sections plays a crucial role in the process of evaluating a wide range of diseases. Since it was first performed in 1941, a wide variety of IHC biomarkers have been validated and used in clinical and research laboratories to characterize specific cellular events, such as the nuclear protein Ki-67 associated with cell proliferation, the cellular tumor antigen P53 associated with tumorigenesis, and human epidermal growth factor receptor 2 (HER2) associated with the development of aggressive breast tumors. IHC staining of tissues has been established as one of the gold standards for tissue analysis and diagnostic decisions, as it can selectively identify targeted biomarkers, leading to the investigation of disease treatment and pathogenesis.

[0005] Although IHC staining of tissues is widely used, it still requires dedicated laboratories and skilled technicians (histotechnologists) to perform laborious tissue preparation steps, and is therefore time-consuming and costly. In recent years, deep learning-based virtual staining techniques have rapidly advanced, providing a promising alternative to traditional histochemical staining workflows. This alternative relies on computationally staining microscopic images captured from unlabeled thin tissue sections, avoiding the laborious and costly chemical staining process. Such label-free virtual staining techniques have been demonstrated using autofluorescence imaging, quantitative phase imaging, light scattering imaging, etc., and have been successfully used to create multiple types of histochemical stains, e.g., hematoxylin and eosin (H&E), Masson's trichrome, and Jones silver stains. For example, Rivenson, Y. et al. disclosed deep learning-based virtual histological structural staining of tissues using the autofluorescence of unlabeled tissues. See Rivenson, Y. et al., Deep learning-based virtual histology staining using auto-fluorescence of label-free tissue. Nat Biomed Eng 3, 466-477 (2019).

[0006] However, these previous studies did not perform any virtual IHC staining and mainly focused on generating structural tissue stains that enhance the contrast of specific morphological features in tissue sections. In related studies, deep learning has also enabled prediction of biomarker status, such as quantification of Ki-67. See Liu, Y. et al., Predict Ki-67 Positive Cells in H&E-Stained Images Using Deep Learning Independently From IHC-Stained Images, Frontiers in Molecular Biosciences 7, (2020). Additional biomarkers studied include predicting tumor prognosis from H&E-stained photomicrographs of various malignancies, including hepatocellular carcinoma, breast cancer, bladder cancer, thyroid cancer, and melanoma. Those studies highlight the possibility of correlation between the presence of certain biomarkers and morphological microscopic changes in tissues. However, they do not provide a substitute for IHC stained tissue images that reveal sub-cellular biomarker information for pathologists to diagnostically examine specific inter- and intra-cellular features such as cytoplasmic and nuclear details.

[0007] IHC staining makes certain proteins or antigens in cells selectively visible brightly through an antigen-antibody binding process. There are various IHC biomarkers (specific proteins detected) that are indicative of various cellular events such as cancer stage, cell proliferation or cell apoptosis. Identification of a specific IHC biomarker (e.g., HER2 protein) allows for direction to molecular targeted therapy and prediction of prognosis. However, IHC staining is often more complex, costly and time-consuming to perform compared to structural stains (hematoxylin and eosin (H&E), Masson's trichrome, Jones silver, etc.). Depending on the tissue type, disease, and cellular event to be evaluated, different IHC stainings may be performed. Unlike IHC staining, structural stains like H&E work in a different way, with hematoxylin staining acidic tissue components (e.g., nuclei) while eosin stains other components (e.g., cytoplasm, extracellular fibers). H&E is used for almost all organ types and allows for a quick overview of tissue morphological features such as tissue structure and nuclear distribution. Unlike IHC, H&E cannot distinguish between specific expressed proteins, for example, HER2-positive cells (cells with overexpressed HER2 protein on their membranes) and HER2-negative cells (cells without HER2 protein on their membranes) appear the same in H&E staining images. Summary of the Invention

[0008] Here, a deep learning-based label-free virtual IHC staining method is disclosed (Figures 1A-1C), which converts autofluorescence microscopy images of unlabeled tissue sections into bright-field equivalent images that are substantially consistent with standard IHC stained images of the same tissue samples. The method of the present invention is particularly focused on IHC staining of HER2, an important cell surface receptor protein involved in regulating cell proliferation and differentiation. Assessment of the level of HER2 expression, i.e., HER2 status, in breast tissues has been routinely performed based on HER2 IHC staining of formalin-fixed paraffin-embedded (FFPE) tissue sections, and helps predict the prognosis of breast cancer and its response to HER2-directed immunotherapy. For example, intracellular and extracellular studies of HER2 have led to the development of pharmacological anti-HER2 agents that are beneficial for the treatment of HER2-positive tumors. Further efforts are being made to develop new pharmacological solutions that can combat HER2-directed drug resistance and improve the treatment outcomes of clinical trials. With numerous animal models established for preclinical and life science related research, a deeper understanding of HER2 oncogenes, biological functionality, and drug resistance mechanisms is being explored. In addition, the HER2 biomarker has also been used as an essential tool in the development and testing of novel biomedical imaging, statistical, and spatial transcriptomics methods.

[0009] In one embodiment, a method for generating a digitally stained immunohistochemistry (IHC) microscopic image of unlabeled tissue revealing features specific to at least one biomarker or antigen of the tissue sample, the method comprising: providing a trained deep neural network executed by image processing software using one or more processors of a computing device, the trained deep neural network being trained on a plurality of matched immunohistochemistry (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample; acquiring one or more autofluorescence images of the unlabeled tissue sample using a fluorescence imaging device; inputting the one or more autofluorescence images of the unlabeled tissue sample into the trained deep neural network; and the trained deep neural network outputting a digitally stained IHC microscopic image of the unlabeled tissue sample revealing features specific to at least one target biomarker, the digitally stained IHC microscopic image appearing to be substantially equivalent to a corresponding image of the same unlabeled tissue sample if the biomarker had been chemically IHC stained.

[0010] In another embodiment, a system for generating digitally stained immunohistochemistry (IHC) microscopic images of unlabeled tissue samples that reveal features specific to at least one biomarker or antigen of the tissue sample, the system includes a computing device having image processing software executed on or by the computing device, the image processing software including a trained deep neural network executed using one or more processors of the computing device, the trained deep neural network being trained on a plurality of matched immunohistochemistry (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample, the image processing software being configured to receive one or more autofluorescence images of the unlabeled tissue sample using a fluorescence imaging device and output a digitally stained IHC microscopic image of the unlabeled tissue sample that reveals features specific to at least one target biomarker, the digitally stained IHC microscopic image appearing to be substantially equivalent to a corresponding image of the same unlabeled tissue sample if the biomarker had been chemically IHC stained.

[0011] The characteristics specific to at least one biomarker or antigen of the tissue sample may include specific subcellular characteristics such as staining intensity and / or distribution in cell membranes, nuclei, or other cellular structures. The characteristics may also include other criteria such as number of nuclei, average nuclear size, connectivity of membrane regions, and area under a characteristic curve (e.g., membrane staining ratio as a function of a saturation threshold). [Brief description of the drawings]

[0012] [Figure 1]FIG. 1A shows a schematic of a system used to generate a digital / virtual HER2 staining output image of an unlabeled breast tissue sample, according to one embodiment. The HER2 staining output image is a brightfield equivalent microscopic image that matches a standard, chemically performed HER2 IHC staining of breast tissue as currently performed. FIGS. 1B-1C show virtual HER2 staining of an unlabeled tissue section via deep learning. The top of FIG. 1B shows that standard immunohistochemistry (IHC) HER2 staining (top) relies on laborious and expensive tissue processing performed by a histotechnologist, which typically takes .about.1 day. The bottom of FIG. 1B shows how a pre-trained deep neural network enables virtual HER2 staining of an unlabeled tissue section. FIG. 1C shows digital or virtual HER2 staining that converts an autofluorescence image of an unlabeled tissue section into a brightfield equivalent image that substantially matches an image of a standard IHC HER2 staining. [Diagram 2] 2A-2B show one embodiment of a digital or virtual HER2 staining neural network. A GAN framework consisting of a generator model and a classifier model was used to train the virtual HER2 staining network. FIG. 2A shows the generator network using an attention-gated U-net structure (with an attention gate (AG) block) to map the label-free autofluorescence image to a bright-field equivalent HER2 image. FIG. 2B shows the classifier network using a CNN consisting of five consecutive two-layer convolutional residual blocks and two fully connected layers. Once the network model has converged, only the generator model (FIG. 2A) is used to infer a virtual HER2 image, which takes ~12 seconds for 1 mm2 of tissue area. [Diagram 3]FIG. 3 shows a comparison of virtual and standard IHC HER2 staining of breast tissue sections at different HER2 scores. Image panels a, b, c1, c2, d1, d2, e1, e2, f, g, h1, h2, i1, i2, j1, j2, k, l, m1, m2, n1, n2, o1, o2, p, q, r1, r2, s1, s2, t1, and t2 are shown. Image panels a, f, k, and p are brightfield WSIs of standard IHC HER2 stained samples at (image panel a) HER2 3+, (image panel f) HER2 2+, (image panel k) HER2 1+, and (image panel 3p) HER2 0. Image panels b, g, l, and q are brightfield WSIs generated by virtual staining and correspond to the same samples as a, f, l, and p, respectively. Image panels c1-e1, c2-e2 are regions of interest enlarged from image panels a, b with a HER2 score of 3+. Image panels h1-j1, h2-j2 are regions of interest enlarged from image panels f, g with a HER2 score of 2+. Image panels m1-o1, m2-o2 are regions of interest enlarged from image panels k, l with a HER2 score of 1+. Image panels r1-t1, h2-t2 are regions of interest enlarged from image panels p, q with a HER2 score of 0. [Figure 4] 4A-4B show a confusion matrix for HER2 scores. Each element of the matrix represents the number of WSIs assessed by board-certified pathologists in which the HER2 score (rows) was based on either Fig. 4A - virtual HER2 staining, or Fig. 4B - standard IHC HER2 staining, compared to the reference (ground truth) HER2 score (columns) provided by the UCLA TPCL. [Diagram 5]Figures 5A-5E: Comparison of image quality of virtual HER2 and standard IHC HER2 staining. Figure 5A: Quality scores of virtual HER2 and standard IHC HER2 images calculated based on four different feature indices: nuclear detail, absence of staining artifacts, absence of excessive background staining, and membrane sharpness. Each value was averaged across all image patches and pathologists. Figures 5B-5E: Detailed comparison of quality scores under each feature indices at different HER2 scores. The graded scale applied for each metric is 1-4: 4 is perfect, 3 is very good, 2 is acceptable, and 1 is unacceptable. Standard deviations are plotted as dashed lines. [Figure 6] Figures 6A and 6B are quantitative assessments based on the features of virtual stained HER2 images and standard IHC HER2 images. Figure 6A: Quantitative comparison of virtual HER2 features and standard IHC HER2 features for HER2-negative cases (N=4,142 images) based on four different indices. Figure 6B: Quantitative comparison of virtual HER2 features and standard IHC HER2 features for HER2-positive cases (N=4,052 images) based on the same indices. [Figure 7] FIG. 7 shows examples of unsuccessful chemical IHC staining. Image panel a shows a pseudo-stained autofluorescence image captured using an unlabeled breast tissue section. Image panel b shows the virtual HER2 staining predicted by the generator network. Image panel c shows the same tissue section that underwent significant tissue damage and loss during standard IHC HER2 staining. Image panel d shows the IHC staining of a serially sliced ​​section from the same specimen block. Image panel e shows a pseudo-stained autofluorescence image captured using another unlabeled breast tissue section. Image panel f shows the virtual HER2 staining predicted by the generator network. Image panel g shows the same tissue section that underwent false-negative IHC HER2 staining (i.e., unsuccessful IHC staining). Image panel h shows the IHC staining of a serially sliced ​​section from the same specimen block. [Figure 8]Figures 8A-8B: HER2 scores corresponding to image patches. Figure 8A: Histograms of graded HER2 scores based on image patches cropped from virtual HER2 WSI and standard IHC HER2 WSI for each patient. Figure 8B: Individual HER2 scores corresponding to image patches graded by three pathologists. [Figure 9] Figures 9A-9B show a comparison of virtual staining network performance with different autofluorescence input channels. Figure 9A: Visual comparison of virtual staining networks trained with one (DAPI), two (DAPI+FITC), three (DAPI+FITC+TxRed), and four (DAPI+FITC+TxRed+Cy5) autofluorescence input channels, showing that the results improve as the number of input channels increases. Figure 9B: Quantitative evaluation of virtual staining networks trained with different numbers of autofluorescence input channels. MSE, SSIM, and SSIM of the membrane color channel (i.e., DAB staining) were calculated using the network output and ground truth images. [Figure 10] Figures 10A-10B: Examples of color deconvolution splitting diaminobenzidine (DAB) and hematoxylin stained channels. Figure 10A: Color deconvolution of HER2 positive regions. Figure 10B: Color deconvolution of HER2 negative regions. [Figure 11] Figure 11 illustrates the image pre-processing and alignment workflow (image panels a-g are shown). Image panel a: Autofluorescence WSI (before IHC staining) and brightfield WSI (after IHC staining) of the same tissue section are stitched together. Image panel b: Global alignment of autofluorescence WSI and brightfield WSI by detecting and matching SURF feature points. Image panel c: Tiles of roughly matched autofluorescence and brightfield images are cut out. Image panel d: A registration model is trained to convert the autofluorescence image to a brightfield image. Image panel e: Registration model output and ground truth image. Image panel f: The ground truth image is aligned to the autofluorescence image using an elastic alignment algorithm. Image panel g: After 3-5 iterative alignments, perfectly matched autofluorescence and brightfield image patches are obtained. [Figure 12] Figure 12: Quantitative comparison of different virtual staining network architectures. Both visual and numerical comparisons reveal the superior performance of the attention-gated GAN used in our work compared to other network architectures. [Figure 13] Figure 13: Comparison of color distributions of output images (with strong HER2 expression) generated by different virtual staining networks. The color distribution of the output images generated by the attention-gated GAN matches closely with the color distribution of standard IHC ground truth images. [Figure 14] Figure 14: Comparison of color distributions of output images (with weak HER2 expression) generated by different virtual staining networks. The color distribution of the output images generated by the attention-gated GAN matches closely with the color distribution of standard IHC ground truth images. [Figure 15] Figure 15: Extraction of nuclear and membrane staining features based on color deconvolution and segmentation algorithms. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] 1A, 1B (bottom), and 1C diagrammatically illustrate one embodiment of a system 2 (FIG. 1A) and method (FIG. 1B (bottom) and FIG. 1C) for outputting a digitally or virtually stained IHC image 40 of unlabeled tissue that reveals at least one biomarker or antigen-specific feature of the tissue sample obtained from one or more input autofluorescence microscopy images 20 of an unlabeled tissue sample 22 (e.g., breast tissue in one particular example). The at least one biomarker or antigen-specific feature may include specific subcellular features such as staining intensity and / or distribution in cell membranes, nuclei, or other cellular structures. The features may also include other criteria such as number of nuclei, average nuclear size, connectivity of membrane regions, and area under characteristic curves (membrane staining ratio as a function of saturation threshold - FIG. 15). As described herein, the one or more input images 20 are autofluorescence image(s) 20 of the unlabeled tissue sample 22. The tissue sample 22 is not stained or labeled with a fluorescent stain or label. That is, the autofluorescence image(s) 20 of the unlabeled tissue sample 22 captures the fluorescence emitted by the unlabeled tissue sample 22 that is the result of one or more endogenous fluorescent molecules or other endogenous emitters of frequency-shifted light contained within that sample. Frequency-shifted light is light emitted at a different frequency (or wavelength) than the incident frequency (or wavelength). Endogenous fluorescent molecules or endogenous emitters of frequency-shifted light may include molecules, compounds, complexes, molecular species, biomolecules, dyes, tissues, etc. In some embodiments, the input autofluorescence image(s) 20 are subjected to one or more linear or nonlinear pre-processing operations selected from contrast enhancement, contrast inversion, image filtering.

[0014] The system 2 includes a computing device 100 including one or more processors 102 therein, and image processing software 104 incorporating a trained deep neural network 10 (e.g., a convolutional neural network as described herein in one or more embodiments). The computing device 100 may include a personal computer, laptop, mobile computing device, remote server, etc., as described herein, although other computing devices may be used (e.g., devices incorporating one or more graphic processing units (GPUs) or other application specific integrated circuits (ASICs) as one or more processors 102). The GPUs or ASICs may be used to accelerate the training as well as the final output image 40. The computing device 100 may be associated with or connected to a monitor or display 106 used to display the digital stained IHC image 40 (e.g., a HER2 image). The display 106 may be used to display a graphical user interface (GUI) used by a user to display and view the digital stained IHC image 40. In one preferred embodiment, the trained deep neural network 10 is a convolutional neural network (CNN).

[0015] For example, in one preferred embodiment as described herein, the trained deep neural network 10 is trained using a GAN model. In a GAN-trained deep neural network 10, two models are used for training. A generative model is used that captures the data distribution (e.g., the generator network of FIG. 2A), while a second model estimates the probability that a sample came from the training data and not from the generative model (e.g., the discriminator network of FIG. 2B). More information on GANs can be found in Goodfellow et al., Generative Adversarial Nets., Advances in Neural Information Processing Systems, 27, pp. 2672-2680 (2014), which is incorporated by reference herein, as is FIG. 2A and the accompanying description herein. Network training of the deep neural network 10 (e.g., GAN) can be performed by the same or a different computing device 100. For example, in one embodiment, a personal computer can be used to train the GAN-based deep neural network 10, although such training can take a significant amount of time. To accelerate this training process, one or more dedicated GPUs can be used for training. As described herein, such training and testing was performed on GPUs obtained from commercially available graphics cards. Once the deep neural network 10 is trained, the generator network portion of the deep neural network 10 can be used or executed on a different computing device 100 that may include one that uses fewer computational resources for the training process (although a GPU can also be integrated into the execution of the trained deep neural network 10).

[0016] The image processing software 104 can be implemented using conventional software packages and platforms, including, for example, MATLAB, Python, and Pytorch. The trained deep neural network 10 is not limited to a particular software platform or programming language, and the trained deep neural network 10 can be executed using any number of commercially available software languages ​​or platforms (or combinations thereof). The image processing software 104 incorporating or cooperating with the trained deep neural network 10 can be executed in a local environment or in a remote cloud-based environment. In some embodiments, some functions of the image processing software 104 can be executed in one particular language or platform (e.g., image pre-processing and alignment), while the trained deep neural network 10 can be executed in another particular language or platform. Nevertheless, both operations are performed by the image processing software 104.

[0017] 1A, in one embodiment, the trained deep neural network 10 receives a single autofluorescence image 20 of an unlabeled tissue sample 22. In other embodiments, for example, when multiple excitation channels are used, there may be multiple autofluorescence images 20 of the unlabeled tissue sample 22 that are input to the trained deep neural network 10 (e.g., one image per channel). These channels may include, for example, DAPI, FITC, TxRed, and Cy5 obtained using different filters / filter cubes of the imaging device 110. For example, multiple autofluorescence images 20 obtained using different filters / filter cubes of the imaging device 110 may be input to the trained deep neural network 10 (e.g., two or more different channels).

[0018] The autofluorescence image 20 can include a wide-field autofluorescence image 20 of a label-free tissue sample 22. Wide-field is meant to indicate that a wide field of view (FOV) can be obtained by scanning or otherwise obtaining a smaller FOV, where a wide FOV can be between 10 and 2,000 mm. 2 For example, the small FOV can be obtained by a scanning fluorescence microscope 110 that digitally stitches together the small FOVs to create a wide FOV using image processing software 104. For example, the wide FOV can be used to obtain a whole slide image (WSI) of the unlabeled tissue sample 22. The autofluorescence image(s) 20 are obtained using a fluorescence imaging device 110. For the fluorescence embodiments described herein, the device can include a fluorescence microscope 110. The fluorescence microscope 110 includes one or more excitation light source(s) that illuminate the unlabeled tissue sample 22, and one or more image sensor(s) (e.g., CMOS image sensors) for capturing the autofluorescence emitted by fluorescent molecules or other endogenous emitters of frequency-shifted light contained in the unlabeled tissue sample 22. The fluorescence microscope 110, in some embodiments, can include the capability of illuminating the unlabeled tissue sample 22 with excitation light of multiple different wavelengths or wavelength ranges / wavelength bands. This can be accomplished using multiple different light sources and / or different filter sets (e.g., standard UV or near-UV excitation / emission filter sets). Additionally, the fluorescence microscope 110, in some embodiments, can include multiple filters capable of filtering different emission bands. For example, in some embodiments, multiple fluorescence images 20 can be captured, each captured in a different emission band using a different filter set (e.g., filter cube). For example, the fluorescence microscope 110 can include different filter cubes for the different channels DAPI, FITC, TxRed, and Cy5.

[0019] The unlabeled tissue sample 22, in some embodiments, can include a portion of tissue disposed on or in a substrate 23. The substrate 23, in some embodiments, can include an optically transparent substrate (e.g., a glass or plastic slide, etc.). The unlabeled tissue sample 22 can include a tissue section cut into thin sections, such as with a microtome apparatus. The unlabeled tissue sample 22 can be imaged with or without a cover glass / cover slip. The unlabeled tissue sample 22 can include a frozen section or a paraffin (wax) section. The unlabeled tissue sample 22 can be fixed (e.g., using formalin) or unfixed. In some embodiments, the unlabeled tissue sample 22 can be fresh or live. The methods described herein can also be used to generate a digitally stained IHC image 40 of the unlabeled tissue sample 22 in vivo.

[0020] As described herein, in one particular embodiment, the unlabeled tissue sample 22 is an unlabeled breast tissue sample, and the digitally or virtually stained IHC image 40 generated is a digitally stained HER2 microscopic image of the unlabeled breast tissue sample 22. It should be understood that other types of tissues other than breast tissue and other types of biomarker targets other than HER2 may be used in conjunction with the system 2 and methods described herein. In IHC staining, a primary antibody is typically used that targets an antigen or biomarker / biomolecule of interest. A secondary antibody is then typically used that binds to the primary antibody. An enzyme such as horseradish peroxidase (HRP) is attached to the secondary antibody and used for binding to a chromogen such as DAB or an alkaline phosphatase (AP) based chromogen. A counterstain such as hematoxylin can be applied after the chromogen to obtain better contrast for visualizing the underlying tissue structure. The methods and systems described herein are used to generate digitally stained IHC images of unlabeled tissue that reveal at least one biomarker or antigen specific feature of the tissue sample. EXAMPLES

[0021] Experimental

[0022] The presented virtual HER2 staining method is based on deep-learnable image-to-image translation using conditional generative adversarial networks (GANs), as shown in Figures 2A-2B. After completing the training phase, two blinded quantitative studies were performed using new breast tissue sections with different HER2 scores to demonstrate the effectiveness of the virtual HER2 staining framework. For this purpose, we used the semiquantitative Dako HercepTest scoring system, which evaluates the percentage of tumor cells exhibiting membranous staining of HER2 along with the intensity of staining. Results are reported as 0 (negative), 1+ (negative), 2+ (weakly positive / indeterminable), and 3+ (positive). In the first study, three expert board-certified breast pathologists blindly evaluated the HER2 scores of the virtual stained HER2 whole slide images 40 (WSIs) and their IHC stained standard counterparts. Results and statistical analysis revealed that the determination of HER2 status based on the virtual HER2 WSIs 40 was as accurate as the standard analysis based on chemically prepared IHC HER2 slides. In the second study, the same pathologists assessed the staining quality of both virtual and standard IHC HER2 images using different metrics, namely nuclear detail, membrane clarity, background staining, and staining artifacts. This study revealed that at least two of the three pathologists agreed that there was no statistically significant difference between the quality of the virtual and standard IHC HER2 stained images in terms of the level of nuclear detail, membrane clarity, and absence of staining artifacts. Additional feature-based quantitative assessments also confirmed a high degree of agreement between the virtually generated HER2 images and their standard IHC stained counterparts for both nuclear and membrane staining features.

[0023] The presented framework achieves the first demonstration of label-free virtual IHC staining, avoiding costly, laborious and time-consuming IHC staining procedures involving toxic chemical compounds. This virtual HER2 staining technique may potentially be extended to virtual staining of other biomarkers and / or antigens, facilitating IHC-based tissue analysis workflows in life science and biomedical applications while increasing the repeatability and standardization of IHC staining.

[0024] result

[0025] Label-free virtual HER2 staining of breast tissue

[0026] Virtual HER2 staining of breast tissue samples 22 was demonstrated by training a deep neural network (DNN) model 10 with a dataset of 25 breast tissue sections taken from 19 unique patients, constituting a total of 20,910 image patches of 1024 × 1024 pixels each. Once trained, the DNN model 10 used autofluorescence microscopy images 20 of unlabeled tissue sections captured with DAPI, FITC, TxRed, and Cy5 filter cubes (see Methods section) to virtually stain them and match them to corresponding brightfield images of the same fields captured after standard IHC HER2 staining. The network training and evaluation process employed a cross-validation approach. Separate network models 10 were trained on different dataset splits to generate 12 virtual HER2 WSIs for the blinded study, i.e., three WSIs at each of the four HER2 scores (0, 1+, 2+, and 3+). Each virtual HER2 WSI corresponds to a unique patient that was not used during the network training phase. Note that all tissue sections22 were obtained from pre-existing tissue blocks with HER2 reference (ground truth) scores provided by the UCLA Translational Pathology Core Laboratory (TPCL) under UCLA IRB 18-001029.

[0027] Figure 3 summarizes the comparison of virtual HER2 images 40 inferred by the DNN model 10 with their corresponding IHC HER2 images captured from the same tissue sections after standard IHC staining. Image panels a-t2 are shown in Figure 3. Both WSI and zoomed in areas show a high degree of agreement between the virtual and standard IHC staining. These results indicate that a well-trained virtual staining network 10 can reliably convert autofluorescence images 20 of unlabeled breast tissue sections 22 into brightfield equivalent virtual HER2 images 40 that match their IHC HER2 stained counterparts across all HER2 statuses, 0, 1+, 2+, and 3+. Upon review, an expert board-certified pathologist confirmed that the comparison between the IHC and virtual HER2 images 40 showed equivalent staining with no significant perceptible differences in subcellular features such as membrane sharpness or nuclear detail. Notably, the virtual staining network 10 clearly produced the expected intensity and distribution of membranous HER2 staining (DAB staining or lack thereof) in tumor cells. In HER2-positive (3+, Figure 3 - image panels a-e2) breast cancers, both virtual and IHC stained images showed strong complete membranous staining in >10% of tumor cells, with indistinct cytoplasmic staining of tumor cells. Neither stromal nor inflammatory cells showed false-positive staining, and nuclear detail of tumor cells was comparable in both panels. In indeterminate (2+, Figure 3f-3j2) tumors, virtual images showed weak to moderate membranous staining in >10% of tumor cells, with membranous staining of an equal amount of tumor cells in the corresponding areas. HER2-negative (1+, Figure 3k-3o2) tumors showed faint membranous staining in >10% of tumor cells. Neither stromal nor inflammatory cells showed faint staining. HER2-negative (0, Figure 3p-3t2) tumors showed no staining of tumor cells.

[0028] Blinded assessment and quantification of virtual HER2 staining

[0029] Next, the efficacy of the virtual HER2 staining framework was evaluated in a quantitative blinded study, in which 12 virtual HER2 WSIs40 and their corresponding standard IHC HER2 WSIs were mixed and presented to three board-certified breast pathologists who graded the HER2 score (i.e., 3+, 2+, 1+, or 0) of each WSI without knowing whether the image was virtual or standard IHC stained. Random image shuffling, rotation, and flipping were applied to the WSIs to facilitate blinding of the evaluation. The HER2 scores of the virtual and standard IHC WSIs graded blindly by the three pathologists are summarized in Figures 4A, 4B and compared with their reference, ground truth scores provided by the UCLA TPCL. The confusion matrices for the virtual HER2 WSI40 (Figure 4A) and IHC HER2 WSI (Figure 4B), each corresponding to N=36 evaluations, reveal that the virtual HER2 staining approach achieved a similar level of accuracy in HER2 status assessment as standard IHC staining. A closer look at these confusion matrices reveals that the sum of the diagonal elements of the virtual HER2-based assessment (22) is higher than that of IHC HER2 (19), indicating that more cases were correctly scored based on the virtual HER2 WSI40 compared to the assessment based on the standard IHC HER2 WSI. Furthermore, the sum of the absolute off-diagonal errors of the virtual HER2-based assessment (14) is smaller than that of the standard IHC HER2 (18). Based on the same confusion matrices shown in Figures 4A and 4B, a chi-square test was performed to compare the degree of agreement between the virtual staining method and the standard IHC staining method in HER2 scoring. The results of the test indicate that there is no statistically significant difference between the two methods (p=0.4752, see Table 1 below). TIFF2025503388000002.tif102170

[0030] In addition to assessing the effectiveness of virtual staining in HER2 scoring, the quality of staining of virtual HER2 images40 was quantitatively assessed and compared to standard IHC HER2 images. In this blinded study, 10 regions of interest (ROIs) were randomly extracted from each of the 12 virtual HER2 WSIs and 10 ROIs were randomly extracted at the same locations from each of their corresponding IHC HER2 WSIs to construct a test set of 240 image patches. Each image patch was 8000 × 8000 pixels (1.3 × 1.3 mm 2), randomly shuffled, rotated, and flipped, and then reviewed by the same three pathologists. The pathologists were asked to grade the image quality of each ROI based on four pre-specified feature indices for HER2 staining: membrane clarity, nuclear detail, absence of excessive background staining, and absence of staining artifacts (Figures 5A-5E). The grade scale for each metric was 1 to 4, with 4 representing perfect, 3 representing very good, 2 representing acceptable, and 1 representing unacceptable. Figure 5A summarizes the staining quality scores (averaged across all image patches and pathologists) of the virtual HER2 and standard IHC HER2 images based on the pre-defined feature indices. Figures 5B-5E further compare the average quality scores in each of the four HER2 conditions under each feature indices. In Figure 5B, the membrane clarity score of the HER2-negative ROI is marked as "not applicable" due to the absence of cell membrane staining in the HER2-negative samples. It is important to emphasize that standard IHC HER2 images were favored in these comparisons because they were preselected. A significant proportion of standard IHC HER2 tissue slides suffered from unacceptable staining quality issues (see Discussion and image panels a-h in Figure 7), and therefore they were excluded from the comparison study at the first stage. Nevertheless, the virtual and standard IHC HER2 staining quality scores were very close to each other and fell within their standard deviation ranges (dashed lines in Figure 5). We also performed one-tailed t-tests for each feature metric assessed by the expert board-certified pathologists to determine whether the standard IHC HER2 images were statistically significantly better in staining quality than the virtual HER2 images40. The t-test results showed that only for the metric “absence of excessive background staining” two of the three pathologists reported a statistically significant improvement in the quality of standard IHC staining compared to virtual staining. For the remaining feature measures (i.e., nuclear detail, membrane clarity, and staining artifacts), at least two of the three pathologists reported that the staining quality of the IHC HER2 images was statistically significantly less good than their virtual HER2 counterparts (Table 2 below).It should also be noted that the virtual stained HER2 image 40 did not lead to false diagnoses at the whole slide level, as analyzed using the confusion matrices shown in Figures 4A and 4B and the chi-square test reported in Table 1. TIFF2025503388000003.tif81170

[0031] Difference = Virtual stained image quality score - IHC stained image quality score

[0032] Null hypothesis: TIFF2025503388000004.tif9170

[0033] Alternative Hypothesis: TIFF2025503388000005.tif9170

[0034] Besides the assessment of the staining quality of each ROI, pathologists also graded the HER2 score for each ROI. The results are reported in Figures 8A-8B. Each histogram in Figure 8A summarizes the HER2 scores of 10 ROIs extracted from each WSI assessed by three pathologists (i.e., N=30 assessments). The reference (ground truth) HER2 score of the corresponding WSI is plotted as a grey vertical dashed line. This analysis reveals that in the majority of patients, there is no discrepancy in the assessed HER2 scores between the virtually generated ROIs and the standard IHC stained ROIs. For cases with discrepancies (e.g., Patient #5 and Patient #11), the histogram of the virtual HER2 scores was centered closer to the reference HER2 scores (dashed line) compared to the histogram of the standard IHC-based HER2 scores. It is also important to note that grading the HER2 score from sub-sampled ROIs versus WSI may yield different results due to the heterogeneous nature of tissue sections.

[0035] Feature-based quantitative assessment of virtual HER2 staining

[0036] In addition to the blinded assessment of virtual staining efficacy and image quality by pathologists, a feature-based quantitative analysis of virtually generated HER2 images in comparison with their IHC stained counterparts was performed. In this analysis, 8194 unique test image patches (each with a size of 1024 × 1024 pixels) were blindly selected for virtual staining. Due to the different staining features of each different HER2 status, these blinded test images were divided into two subsets for quantitative evaluation: one subset containing images from HER2 0 and HER2 1+ (N = 4142) and the other subset containing images from HER2 2+ and HER2 3+ (N = 4052). For each virtual stained HER2 image40 and its corresponding IHC HER2 image (ground truth), four feature-based quantitative evaluation metrics (designed specifically for HER2) were calculated based on the segmentation of nuclear and membranous staining (see Methods section). Quantitative evaluation metrics based on these four features included number of nuclei and average nuclear area (number of pixels) to quantify nuclear staining in each image, and area under the characteristic curve and membrane area connectivity to quantify membrane staining in each image (see Methods section for details).

[0037] The results of these feature-based quantitative assessments of virtual HER2 images 40 compared to their standard IHC counterparts are shown in Figures 6A-6B. This analysis showed that the virtual HER2 staining feature indexes exhibited similar distributions and closely matching mean values ​​(horizontal dashed lines) for both nuclear and membranous staining compared to their standard IHC counterparts. By comparing the assessment results of the HER2-positive group (2+ and 3+) with the HER2-negative group (0 and 1+), a similar distribution of nuclear features (i.e., number of nuclei and average nuclear area) and higher levels of membranous staining were observed, which, as expected, correlates well with higher HER2 scores.

[0038] Consideration

[0039] A deep learning-enabled label-free virtual IHC staining method and system 2 is disclosed herein. By training a DNN model 10, the method of the present invention generated a virtual HER2 image 40 from an autofluorescence image 20 of an unlabeled tissue section 22 that matches a brightfield image captured after standard IHC-staining. Compared to chemically performing IHC staining, the virtual HER2 staining method is quick and easy to operate. Conventional IHC HER2 staining involves laborious sample processing steps that require regular monitoring by a histotechnologist, and the entire process typically takes a day before a diagnostician can review the slide. In contrast, the presented virtual HER2 staining method avoids these laborious and costly steps and generates a brightfield-equivalent HER2 image 40 by computation using an autofluorescence image 20 captured from an unlabeled tissue section 22. After completion of training (a one-time task), the entire inference process using the virtual staining network can be performed using a consumer-grade computer 100 to compute a 1 mm section of tissue. 2 It takes only .about.12 seconds for .gamma., which can be further improved by using a faster hardware acceleration processor 102 / unit.

[0040] Another advantage of the presented method is its ability to minimize staining variability commonly observed with standard IHC staining and produce highly consistent and repeatable staining results. IHC HER2 staining is delicate and laborious, as each tissue processing step requires precise control of time, temperature, and reagent concentration, and in practice, often fails to produce satisfactory staining. In this study, ~30% of specimen slides were discarded due to unsuccessful standard IHC staining and / or significant tissue damage, even though IHC staining was performed by accredited pathology labs. Figure 7 (image panels c, g) shows two examples of standard IHC staining failures experienced, including complete tissue damage and false-negative staining that failed to reflect the correct HER2 score. In contrast, the computational virtual staining approach does not rely on chemical processing of the tissue and produces reproducible results, which is important for standardization of HER2 interpretation by eliminating commonly experienced staining variability and artifacts.

[0041] Since the autofluorescence input images 20 of tissue slices 22 were captured using a standard filter set installed on a conventional fluorescence microscope 110, the presented approach is ready to be implemented on existing fluorescence microscopes 110 without any hardware modifications or customization of optical components. The present results showed that a combination of four commonly used fluorescence filters (DAPI, FITC, TxRed, and Cy5) yielded a very good baseline for virtual HER2 staining performance; see Figure 9A. As an ablation study, the virtual staining network 10 trained with different autofluorescence input channels was quantitatively compared between the network output and ground truth images by calculating the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) (see Figure 9B). Since cell membrane staining is an important evaluation factor in HER2 status assessment, color deconvolution was also performed to split the membrane staining channel (i.e., diaminobenzidine, DAB staining) (Figures 10A-10B), followed by calculating and comparing the SSIM scores (Figures 9A-9B). These analyses revealed that the performance of the virtual staining network 10 partially degraded with a decreasing number of input autofluorescence channels, motivating the use of DAPI, FITC, TxRed, and Cy5 together (Figure 9B).

[0042] The advantages of using an attention-gated GAN structure for virtual HER2 staining were illustrated by an additional comparative study, in which four different network architectures10 were trained and blindly tested, including: 1) the attention-gated GAN structure used herein, 2) the same structure with the residual connections removed, 3) the same structure with the attention gate block removed, and 4) an unsupervised cyclic GAN framework. The train / validation / test datasets and training epochs were the same for all four networks10. After their training, the correlation between the network output and the ground truth images of membrane staining (SSIM DABA quantitative comparison of the networks 10 was performed by calculating PSNR, SSIM, and SSIM of the HER2-expression channels (see Figure 12). Both visual and numerical comparisons revealed that the attention-gated GAN used herein was the only network architecture capable of consistently providing good and accurate virtual staining across various HER2 expression levels, while the other network architectures produced several catastrophic staining errors in one or more test FOVs, making them unacceptable for consistent inference across all HER2 conditions. In Figures 13-14, we further compared the color distributions of the output images 40 generated by the different network architectures against the corresponding ground truth images, including FOVs with strong HER2 expression (Figure 13) and FOVs with weak HER2 expression (Figure 14) (see Methods section). These additional comparisons showed that the color histograms of the output images generated by the framework matched much more closely with the standard IHC ground truth for both membrane and nuclear staining channels, again demonstrating the advantage of using an attention-gated GAN architecture for the trained deep neural network 10 reported herein.

[0043] The success of the virtual HER2 staining method relies on the use of a convolutional neural network 10 to process the spatio-spectral information encoded in the autofluorescence images 20 of unlabeled tissue 22. The presented virtual staining method can potentially be extended to a wide range of other IHC stains. Although the virtual HER2 staining framework was demonstrated based on autofluorescence imaging of unlabeled tissue sections 22, other label-free microscopy modalities such as holography, fluorescence lifetime imaging and Raman microscopy can also be utilized for this task. In addition to generalizing to other types of IHC stains in the evaluation of various biomarkers, the method can also be adapted to unfixed fresh tissue samples or frozen sections, potentially providing real-time virtual IHC images for intraoperative rapid diagnosis during surgery.

[0044] method

[0045] Sample preparation and standard IHC staining

[0046] Unlabeled breast tissue blocks were provided by UCLA TPCL under UCLA IRB 18-001029 and cut into 4 μm thin sections22. The FFPE thin sections22 were then deparaffinized and covered with glass coverslips. After acquiring autofluorescence microscopy images20, the unlabeled tissue sections22 were sent to a certified pathology laboratory for standard IHC HER2 staining, which was performed by UCLA TPCL and the Department of Anatomic Pathology at Cedars-Sinai Medical Center, Los Angeles, USA. The IHC HER2 staining protocol provided by UCLA TPCL is described in IHC HER2 Staining Protocol (Methods).

[0047] Acquisition of image data

[0048] Autofluorescence images of unlabeled tissue sections were captured using a standard fluorescence microscope (IX-83, Olympus) with a ×40 / 0.95NA (UPLSAPO, Olympus) objective. Autofluorescence images were captured at different excitation-emission wavelengths using four fluorescence filter cubes, including DAPI (Semrock DAPI-5060C-OFX, EX377 / 50nm, EM447 / 60nm), FITC (Semrock FITC-2024B-OFX, EX485 / 20nm, EM522 / 24nm), TxRed (Semrock TXRED-4040C-OFX, EX562 / 40nm, EM624 / 40nm), and Cy5 (Semrock CY5-4040C-OFX, EX628 / 40nm, EM692 / 40nm). Twenty autofluorescence images were captured using a scientific complementary metal oxide-semiconductor (sCMOS) image sensor (ORCA-flash4.0 V2, Hamamatsu Photonics) with exposure times of 150 ms, 500 ms, 500 ms, and 1000 ms for DAPI, FITC, TxRed, and Cy5 filters, respectively. Images were normalized at each exposure time for the four channels. Thus, DAPI images (training and testing) were divided by an exposure time of 150 ms. The other channels were normalized at their respective exposure times. The image acquisition process was controlled by the μManager (version 1.4) microscope automation software. After standard IHC HER2 staining was completed, brightfield WSI was obtained using a slide scanner microscope equipped with a ×20 / 0.8NA objective (AxioScan Z1, Zeiss).

[0049] Image pre-processing and alignment

[0050] Matching of autofluorescence images 20 (network input) and brightfield IHC HER2 (network ground truth) image pairs is critical for successful training of the image-to-image conversion network. The image processing workflow for preparing the training dataset for the virtual HER2 staining network is described in Figure 11 and image panels a-g implemented in MATLAB (MathWorks). First, autofluorescence images 20 (before IHC staining) and whole-slide brightfield images (after IHC staining) of the same tissue section were stitched to the WSI (image panel a) and globally co-registered by detecting and matching the speeded up robust features (SURF) points (image panel b). These roughly matched autofluorescence and brightfield WSI were then cropped into pairs of 1024 × 1024 pixel image tiles (image panel c). These image pairs were not matched exactly at the pixel level due to optical aberrations and morphological changes of tissue structures during the standard (labor-intensive) IHC staining procedure. To compute the transformation between the autofluorescence image 20 and its bright-field counterpart using a correlation-based elastic registration algorithm, a registration model needs to be trained to match the style of the autofluorescence image to that of the bright-field image (image panel d). This registration network 50 used the same architecture as the virtual stain network 10. Following the image style transformation using the registration network 50 (image panel e), a pyramidal elastic image registration algorithm was performed to hierarchically match local features of sub-image blocks and compute a transformation map. The transformation map was then applied to correct local wrappings of the ground truth images (image panel f) to better match their autofluorescence counterparts. This training-registration process (image panels d-f) was repeated 3-5 times until the autofluorescence input and bright-field ground truth image patches matched accurately at the single pixel level (image panel g). Finally, a manual data cleaning process was performed to remove image pairs with artifacts such as tissue tears (during the standard chemical staining process) or defocus (during the imaging process).

[0051] Virtual HER2 staining network architecture and training schedule

[0052] As shown in Figures 2A and 2B, a GAN-based network model 10 was used to convert 4-channel label-free autofluorescence images (DAPI, FITC, TxRed, and Cy5) to corresponding bright-field virtual HER2 images. The GAN framework includes (1) a generator network that creates virtual stained HER2 images 40 by learning a statistical transformation between the input autofluorescence image 20 and the corresponding bright-field IHC stained HER2 image (ground truth), and (2) a classifier network that learns to distinguish the virtual HER2 images 40 created by the generator from real IHC stained HER2 images. The generator and classifier were alternatively optimized and improved simultaneously through this competitive training process. Specifically, the generator (G) network and the classifier (D) network were optimized to minimize the following loss function:

[0053] TIFF2025503388000006.tif22170

[0054] TIFF2025503388000007.tif13170

[0055] where G(·) is the generator inference, D(·) is the probability that an image is a real, stained IHC image as predicted by the classifier, and I 入力 indicates the input label-free autofluorescence image, I 標的 indicates the ground truth, standard IHC staining image. 生成器The coefficients (α,λ,γ) of are empirically set as (10, 0.2, 0.5) to balance the pixel-wise smooth L1 error of the generator output against the ground truth, the SSIM loss of the generator output, and the binary cross entropy (BCE) loss of the discriminator prediction of the output image. Compared to using the mean squared error (MSE) loss, the smooth L1 loss is a robust estimator that prevents gradient explosion by using the MSE around zero and the mean absolute error (MAE) in the other parts. Specifically, the smooth L1 loss between two images A and B is defined as:

[0056] TIFF2025503388000008.tif30170

[0057] where m and n are pixel indices and M×N represents the total number of pixels in each image. β was set to 1 in this case.

[0058] The SSIM of two images is defined as:

[0059] TIFF2025503388000009.tif13170

[0060] In the formula, μ A and μ B is the average value of images A and B, σ A 2 and σ B 2 is the variance of images A and B, σ AB is the covariance between images A and B. c1 and c2 are 0.01 and 2 and 0.03 2 was set to.

[0061] The BCE with logit loss for the network is defined as follows:

[0062] TIFF2025503388000010.tif10170

[0063] where p represents the classifier prediction and q represents the actual label (0 or 1).

[0064] As shown in FIG. 2A, the generator network 10 is constructed according to an attention-assisted U-Net architecture with four resolution levels, which can digitally or virtually map the label-free autofluorescence image 20 to the HER2 stained image 40 by learning the transformation of spatial features at different spatial scales, capturing both high-resolution local features at shallower levels and larger-scale global context at deeper levels. The attention-assisted U-Net structure is composed of downsampling and upsampling paths that are symmetrical to each other. The downsampling path includes four downsampling convolution blocks, each of which is composed of a two-layer convolution layer residual block, followed by a leaky regularized linear unit (Leaky ReLU) with gradient 0.1 and a 2×2 max pooling operation with stride size 2 for downsampling. The two-layer convolution layer residual block includes two consecutive convolution layers with kernel size 3×3 and a convolution residual path that connects the input and output tensors of the two convolution layers. The number of input channels and the number of output channels at each level of the downsampling path were set to 4, 64, 128, 256, and 64, 128, 256, 512, respectively.

[0065] Symmetrically, the upsampling path contains four upsampling convolution blocks with the same design as the downsampling convolution block, except that the 2× downsampling operation is replaced by a 2× bilinear upsampling operation. The input of each upsampling block is the concatenation of the output tensor from the previous block and the corresponding feature map at the matched level of the downsampling path, which passes through an attention-gated connection. The attention gate consists of three convolution layers and a sigmoid operation, and outputs an activation weight map that highlights salient spatial features. Notably, an attention gate is added at each level of the U-net skip connection. The attention-gated structure implicitly learns to highlight specific features useful for a particular task while suppressing irrelevant regions in the input image.

[0066] The number of input channels and the number of output channels at each level of the upsampling path were 1024, 1024, 512, 256, and 1024, 512, 256, 128, respectively. Following the upsampling path, a two-layer convolutional layer residual block along with another single convolutional layer reduces the number of channels to three, which matches the number of channels in the ground truth image (i.e., a three-channel RGB image). Furthermore, a two-layer convolutional layer central block was utilized to connect and match the dimensions of the downsampling path and the upsampling path.

[0067] The structure of the classifier network is shown in Figure 2B. An initial block containing one convolutional layer followed by a Leaky ReLU operation first converted the 3-channel generator output or ground truth image into a 64-channel tensor. Then, five consecutive 2-layer convolutional residual blocks were added to perform 2× downsampling to expand the number of channels of each input tensor. The 2× downsampling was enabled by setting the stride size of the second convolutional layer of each block to 2. After passing through the five blocks, the output tensor was averaged and flattened into a one-dimensional vector, and then fed into two fully connected layers to obtain the probability that the input image is a standard IHC stained image.

[0068] The complete image dataset included 25 WSIs from 19 unique patients, producing a set of 20,910 image patches, each with a size of 1024 × 1024 pixels. For training each virtual staining model used in the cross-validation tests, the dataset was divided as follows: (1) test set: images from WSIs of 1-2 unique patients (~10%, no overlap with training or validation patients); after splitting the test set, the remaining WSIs were further divided into (2) validation set: images from 2 of the WSIs (~10%), and (3) training set: images from the remaining WSIs (~80%). The network models were optimized using 256 × 256 pixel image patches randomly cropped from the 1024 × 1024 pixel images in the training dataset. The Adam optimizer with weight decay was used to optimize the generator network with a batch size of 28, with a 1 × 10 -4 , 1×10 for the classifier network -5 We updated the learnable parameters with a learning rate of . We set the generator / discriminator update frequency to 2:1. Finally, we selected the best model based on the best MSE loss, assisted by visual evaluation of validation images. The network converged after ~120 hours of training.

[0069] Execution details

[0070] Image pre-processing was performed with image processing software 104 (i.e., MATLAB using version R2018B (MathWorks)). The virtual staining network 10 was implemented using Python version 3.9.0 and Pytorch version 1.9.0. Training was performed on a desktop computer 100 equipped with an Intel Xeon W-2265 central processing unit (CPU) 102, 64 GB of random access memory (RAM), and an Nvidia GeForce GTX 3090 graphics processing unit (GPU) 102.

[0071] Blinded pathologist evaluation of HER2 images

[0072] For WSI evaluation, 24 high-resolution WSIs were randomly shuffled, rotated, and inverted and uploaded to an online image viewing platform shared with three expert board-certified pathologists who blindly assessed and scored the HER2 status of each WSI using the Dako HercepTest scoring system. For subROI image evaluation, 240 image patches were randomly shuffled, rotated, and inverted and uploaded to the online image sharing platform GIGAmacro (https: / / www.gigamacro.com / ). These 240 image patches used for staining quality evaluation can be accessed at https: / / viewer.gigamacro.com / collections / u08mwIpUDAwfR1vQ?sb=date&sd=asc.

[0073] Blinded assessment by pathologists is shown in Supplementary Data 1.

[0074] statistical analysis

[0075] Chi-square tests (two-sided) were performed to compare the agreement of HER2 scores assessed based on virtual and standard IHC staining. Paired tests (one-sided) were used to compare the image quality of virtual and standard IHC staining. First, the difference in scores between virtual and IHC image patches cropped from the same location was calculated, i.e., the score of each IHC stained image was subtracted from the score of the corresponding virtual stained image. Then, one-sided t-tests were performed to compare the differences to 0 for each feature index and each pathologist. For all tests, a P value of ≤ 0.05 was considered statistically significant. All analyses were performed using SAS v9.4 (SAS Institute, Cary, NC).

[0076] Numerical evaluation of HER2 images

[0077] For feature-based quantitative evaluation of HER2 images (reported in Figures 6A-6B), color deconvolution (Figures 10A-10B) was performed to separate the nuclear staining channel (i.e., hematoxylin staining) and the membrane staining channel (i.e., diaminobenzidine staining, DAB), as shown in Figure 15. The nuclear segmentation map was obtained by Otsu's thresholding method followed by morphological operations (e.g., image erosion, image dilation, etc.) on the hematoxylin channel. Based on the binary nuclear segmentation map, the number of nuclei and the average nuclear area were extracted by counting the number of connected regions and measuring the average region area. For evaluation of membrane staining, the separated DAB image channel was first converted to HSV color space. Then, the segmentation map of membrane staining was obtained by applying threshold(s) to the saturation channel. By gradually increasing the threshold(s) from 0.1 to 0.5 with a step size of 0.02, the ratio of the sum of segmented membrane stained areas to the entire image FOV (i.e., 1024 × 1024 pixels) was calculated to generate a characteristic curve (Figure 15). The area under the characteristic curve can be appropriately extracted to obtain a robust index for evaluating the HER2 expression level. By setting the threshold(s) to 0.25, the ratio of the maximum connected component of the membrane segmentation map to the entire image FOV was also extracted as the connectivity of the membrane area.

[0078] For characterization of the color distributions reported in Figures 13-14, the nuclear and membrane staining channels were segmented using the same color deconvolution method as in Figure 15. For each staining channel, a histogram of all normalized pixel values ​​was created, followed by nonparametric kernel smoothing to fit the distribution profile. The Y-axis (i.e., frequency) of the color histograms shown in Figures 13-14 was normalized by the total number of pixels.

[0079] Data availability

[0080] Data supporting the results demonstrated by this study are available herein. The complete set of images used for the HER2 status and chromatin quality assessment studies can be found in the Supplementary Data 1 file and at https: / / viewer.gigamacro.com / collections / u08mwIpUDAwfR1vQ?sb=date&sd=asc, which is incorporated herein by reference.

[0081] The full pathologist report can be found in the Supplementary Data 1 file. The full statistical analysis report can be found in the Supplementary Data 2 file. Raw WSI corresponding to patient samples were obtained from a UCLA Health private database for this study under UCLA IRB 18-001029 and are therefore not publicly available.

[0082] Code Availability

[0083] All deep learning models used in this study use standard libraries and scripts publicly available in Pytorch. The code used in this manuscript can be accessed through GitHub: https: / / github.com / baibijie / HER2-virtual-staining (incorporated herein by reference).

[0084] IHC HER2 Staining Protocol

[0085] Paraffin-embedded sections were cut at a thickness of 4 μm, the paraffin was removed with xylene, and rehydrated with graded ethanol. Endogenous peroxidase activity was blocked with 3% hydrogen peroxide in methanol for 10 min. Heat-induced antigen retrieval (HIER) was performed on all sections in AR9 buffer (AR9001KT Akoya) using a decloaking chamber (Biocare Medical) at 95°C for 25 min. Slides were then stained with HER2 antibody (Cell Signaling, 4290, 1-200) overnight at 4°C. Signals were detected using DakoCytomation Envision System Labelled Polymer HRP anti-rabbit (Agilent K4003, ready-to-use). All sections were visualized with diaminobenzidine reaction and counterstained with hematoxylin.

[0086] The following publication (including all supplementary materials, supplementary data and code referenced therein): Bai et al., Label-Free Virtual HER2 Immunohistochemical Staining of Breast Tissue using Deep Learning, BME Frontiers, vol. 2022, Article ID 9786242, 15 pages, 2022 is incorporated herein by reference.

[0087] Although embodiments of the present invention have been shown and described, various modifications can be made without departing from the scope of the present invention. For example, while the HER2 biomarker has been specifically considered in label-free virtual immunohistochemical staining of breast tissue, it should be understood that the system 2 and method of the present invention are applicable to other biomarkers and / or antigens. This includes other types of tissue 22 other than breast tissue. Furthermore, while the label-free images have been demonstrated based on autofluorescence imaging of unlabeled tissue sections, it should be understood that other label-free microscopy modalities can be used, such as holography, fluorescence lifetime imaging, Raman microscopy, etc. Thus, the present invention should not be limited, except as set forth in the following claims and their equivalents.

Claims

1. 1. A method for generating digitally stained immunohistochemistry (IHC) microscopy images of unlabeled tissue samples to reveal a specific feature of at least one target biomarker or antigen in said tissue sample, comprising: providing a trained deep neural network executed by image processing software using one or more processors of a computing device, said trained deep neural network being trained on a plurality of matched immunohistochemistry (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of the same tissue sample; acquiring one or more autofluorescence images of the unlabeled tissue sample using a fluorescence imaging device; inputting the one or more autofluorescence images of the unlabeled tissue sample into the trained deep neural network; and the trained deep neural network outputs the digitally stained IHC microscopic image of the unlabeled tissue sample revealing the features specific to the at least one target biomarker, and which further appears substantially equivalent to a corresponding image of the same unlabeled tissue sample if the biomarker had been chemically IHC stained. A method comprising:

2. The method of claim 1 , wherein the trained deep neural network comprises a convolutional neural network.

3. The method of claim 1 , wherein the deep neural network is trained using a generative adversarial network (GAN) model.

4. The method of claim 1 , wherein the tissue comprises breast tissue.

5. 2. The method of claim 1, wherein the at least one target biomarker or antigen of the tissue is human epidermal growth factor receptor 2 (HER2).

6. The method of any of claims 2 to 3, wherein the deep neural network is trained using a generator network configured to learn a statistical transformation between the matched IHC stained images or image patches and autofluorescence images or image patches of the same tissue sample, and a classifier network configured to discriminate between ground truth IHC stained images of the tissue sample and the output digitally stained IHC microscopic image of the tissue sample.

7. The method of claim 2 , wherein the trained deep neural network comprises an attention-gated neural network.

8. The method of claim 1 , wherein the one or more autofluorescence images comprise a plurality of autofluorescence images captured at different excitation-emission wavelengths.

9. 9. The method of claim 8, wherein the plurality of autofluorescence images of the unlabeled tissue sample captured at different excitation-emission wavelengths comprises autofluorescence images obtained in two or more of the following filter channels: DAPI, FITC, TxRed, and Cy5.

10. 2. The method of claim 1, wherein the unlabeled tissue sample comprises an unfixed or fresh tissue sample.

11. 2. The method of claim 1, wherein the unlabeled tissue sample comprises a fixed or frozen tissue sample.

11. 10. The method of claim 1, wherein the unlabeled tissue sample comprises tissue imaged in vivo.

12. 2. The method of claim 1, wherein the multiple matched IHC stained images or image patches and autofluorescence images or image patches of the same tissue sample are subjected to a registration process including passing the multiple matched IHC stained images or image patches and autofluorescence images or image patches through an alignment neural network model that matches local styles / features found in the matched IHC stained images and autofluorescence images prior to training of the deep neural network.

13. The method of claim 12 , wherein registering further comprises registering the IHC stained image or image patch to a respective autofluorescence image or image patch using an elastic registration process.

14. 2. The method of claim 1, wherein the digitally stained IHC microscopic image of the unlabeled tissue sample is output in real-time or near real-time after acquiring the one or more autofluorescence images of the unlabeled tissue sample.

15. The method of claim 1 , wherein the fluorescence imaging device comprises a fluorescence microscope.

16. 1. A system for generating digitally stained immunohistochemistry (IHC) microscopy images of unlabeled tissue samples revealing at least one biomarker or antigen specific characteristic of the tissue sample, comprising:

1. A computing device having image processing software executed on or by said image processing software, said image processing software comprising a trained deep neural network executed using one or more processors of said computing device, said trained deep neural network being trained on a plurality of matched immunohistochemistry (IHC) stained training images or image patches and their corresponding autofluorescence training images or image patches of a same tissue sample, said image processing software being configured to receive one or more autofluorescence images of said unlabeled tissue sample using a fluorescence imaging device, and to output said digitally stained IHC microscopic image of said unlabeled tissue sample that reveals said features specific to said at least one target biomarker, said digitally stained IHC microscopic image further appearing to be substantially equivalent to a corresponding image of the same unlabeled tissue sample if the biomarker had been chemically IHC stained. Including, the system.

17. 20. The system of claim 16, wherein the trained deep neural network comprises a convolutional neural network.

18. 20. The system of claim 17, wherein the trained deep neural network is trained using a generative adversarial network (GAN) model.

19. The system of claim 16 , wherein the tissue comprises breast tissue.

20. 17. The system of claim 16, wherein the at least one target biomarker or antigen of the tissue is human epidermal growth factor receptor 2 (HER2).

21. 17. The system of claim 16, wherein the fluorescence imaging device comprises a fluorescence microscope configured to acquire the one or more autofluorescence images of the unlabeled tissue sample.

22. 22. The system of claim 21, further comprising a plurality of filters used to acquire a plurality of autofluorescence images of the label-free tissue sample captured at different excitation-emission wavelengths.

23. 20. The system of claim 17, wherein the trained deep neural network comprises an attention-gated neural network.

Images