Virtual birefringence imaging and histological staining of amyloid deposits in label-free tissue using autofluorescence microscopy and deep learning

Abstract

A system and method for performing virtual birefringence imaging and virtual staining (e.g., Congo red) of label-free human tissue is disclosed to show that a single trained neural network can rapidly transform autofluorescence images of label-free tissue into brightfield and polarized microscopy images, matching their histochemically-stained versions. Blind testing with quantitative metrics and pathologist evaluations on cardiac tissue showed that the virtually stained polarization and brightfield images highlight amyloid patterns in a consistent manner, mitigating challenges due to variations in chemical staining quality and manual imaging processes in the clinical workflow.

Description

VIRTUAL BIREFRINGENCE IMAGING AND HISTOLOGICAL STAINING OF AMYLOID DEPOSITS IN LABEL-FREE TISSUE USING AUTOFLUORESCENCE MICROSCOPY AND DEEP LEARNINGRelated Application

[0001] This Application claims priority' to U.S. Provisional Patent Application No. 63 / 564,944 filed on March 13, 2024, which is hereby incorporated by reference in its entirety'. Priority is claimed pursuant to 35 U.S.C. § 119 and any other applicable statute.Technical Field

[0002] The technical field generally relates to a deep learning-based method of generating virtual birefringence images and virtual Congo red staining images of label-free human tissue. A single trained neural network rapidly transforms autofluorescence images of label- free tissue sections into brightfield and polarized light microscopy equivalent images, matching the histochemically-stained versions of the same samplesStatement Regarding Federally Sponsored Research and Development

[0003] This invention was made with government support under 2141157 awarded by the National Science Foundation. The government has certain rights in the invention.Background

[0004] Systemic amyloidosis is a heterogeneous group of disorders characterized by the deposition of abnormally folded proteins in tissue. The clinical picture of systemic amyloidosis is not specific, with profound fatigue, weight loss, and edema being the common presenting symptoms. The real prevalence of systemic amyloidosis is not known. A retrospective evaluation of kidney biopsies suggests that amyloidosis is not as rare as it is thought to be, accounting for -43% of nephrotic proteinuria above age 60. In another study, 31% of the patients with multiple myeloma had confirmed evidence of systemic amyloidosis. Cardiac amyloid deposition, causing infiltrative / restrictive cardiomyopathy, is the leading cause of morbidity and mortality in systemic amyloidosis, regardless of the underlying pathogenesis of amyloid production. Similar to other organs involved with amyloidosis, cardiac amyloidosis remains substantially underdiagnosed, and it is advised to test for cardiacamyloidosis presence during the initial work-up of all patients >65 years old hospitalized with heart failure. Early diagnosis of systemic amyloidosis is essential to reducing morbidity and mortality of the disease. A prompt intervention following early-stage amyloid detection may save patients from extensive and irreversible tissue damage. In addition, a definitive diagnosis has become increasingly important since a number of impactful treatment options have developed.

[0005] Diagnosis of amyloidosis is usually based on the demonstration of amyloid deposits in a tissue biopsy. Cardiac biopsy provides the most definitive diagnostic evidence in amyloid cardiomyopathy, and endomyocardial biopsy w as shown to be a safe and relatively simple procedure. Congo red is considered the gold standard stain used in the vast majority of histopathology laboratories to identify amyloid in tissues. When tested under cross-polarized light microscopy, Congo red-stained amyloid areas demonstrate birefringence, which is considered a specific feature of amyloidosis. However, the traditional workflow (as depicted in FIG. 1A - dashed region) has several drawbacks. Congo red staining analysis exhibits inter-observer variability, partly attributed to the challenging nature of Congo red staining. In addition to a standard brightfield microscopy evaluation, visualization under polarized light microscopy is needed to examine the presence of birefringence. The quality of the examining clinical-grade microscope for highlighting amyloid birefringence can potentially limit the accuracy of pathologist evaluations, leading to an increase in both false-negative and falsepositive results. A false negative tissue pathology report can mislead diagnosticians and lead to the exclusion of the amyloidosis diagnosis, often without reconsideration among the differential diagnoses. Recent reports showed that the median time from the symptom onset to amyloidosis diagnosis w as ~2 years. Additionally, nearly a third of patients reported seeing at least five (5) physicians before receiving a diagnosis of amyloidosis.

[0006] These technical challenges of Congo red staining and diagnostic inspection under polarized light microscopy also pose a barrier to the broader adoption of digital pathology. There is currently no clinically approved digital pathology' slide scanner with the required polarization imaging components, and the existing scanning polarization imagers are limited to well-resourced settings, mostly for research use.Summary

[0007] In one embodiment, a deep learning-based virtual tissue staining method is disclosed that is designed to digitally label amyloid deposits in unstained, label-free tissuesections (as illustrated in FIGS. 1A-1C). This technique achieves autofluorescence to birefringence image transformations along with virtual Congo red staining of label-free tissue. It employs conditional generative adversarial networks (cGANs) to rapidly and digitally convert autofluorescence microscopy images of unstained tissue slides into virtually stained birefringence (polarization) and brightfield images, closely resembling the corresponding images of the histochemically stained samples, helping the identification of amyloid deposits in label-free tissue slides. The deep learning model was designed to simultaneously learn the cross-modality image transformations for the two output modalities within a single neural network, performing both autofluorescence-to-birefringence and autofluorescence-to-brightfield image transformations using a digital staining matrix (DSM) (see e.g., FIG. IB), which is an additional channel concatenated to the input autofluorescence microscopy images, indicating the desired output modality (birefringence vs. brightfield). An auxiliary registration module was also integrated into the cGAN in the training process to mitigate spatial misalignments in the training data by learning to register the output and the ground truth histochemically stained images (FIG. 5A).

[0008] After a one-time training phase, when given a new label-free test sample, the virtual staining network successfully generated microscopic images of the Congo red-stained tissue in both brightfield and birefringence channels. These digitally generated, virtually stained images were then stitched into whole slide images (WSI) and examined by pathologists using a customized multi -modality WSI viewer, offering the flexibility to swiftly toggle between the brightfield and polarization / birefringence views. The methodology's effectiveness was affirmed by three board-certified pathologists attesting to the non-inferior quality of the images generated by the network model compared to histochemically stained images, demonstrating a high degree of concordance with the ground truth. Validated by a group of pathologists, the approach effectively bypasses the limitations of traditional histological staining w orkflow- manually performed by histotechnologists and also eliminates the need for tedious polarization imaging with specialized optical components, facilitating a faster and more reliable diagnosis of amyloidosis.

[0009] In one embodiment, a method of generating a virtually stained brightfield and birefringence microscopic images of a label-free sample with a deep neural netw ork includes the following operations. A deep neural network is provided that is executed by software of a computing device, wherein the deep neural network is trained with a plurality of matched autofluorescence images or image patches and their respective chemically stained images orimage patches of the same sample(s) imaged with a brightfield microscope and a polarization microscope capable of birefringence imaging. One or more autofluorescence images of the label-free sample are obtained using a fluorescence microscope or fluorescence imaging device. The one or more autofluorescence images of the label-free sample are input to the deep neural network, wherein the one or more autofluorescence images are concatenated with a digital staining matrix that corresponds to either a virtually stained brightfield microscope image or a virtually stained birefringence microscopic image. The deep neural network outputs a virtually stained brightfield microscope image and a virtually stained birefringence microscopic image of the label-free sample that are substantially equivalent to corresponding brightfield and birefringence microscopic images of the same sample that has been chemically stained.

[0010] In another embodiment, a system for generating a virtually stained brightfield and birefringence microscopic images of a label -free sample with a deep neural network is disclosed. The system includes a computing device having software executed thereon or thereby, the software having a deep neural network that is executed by software of a computing device, wherein the deep neural network is trained with a lurality of matched autofluorescence images or image patches and their respective chemically stained images or image patches of the same sample(s) imaged with a brightfield microscope and a polarization microscope capable of birefringence imaging, the software configured to receive one or more autofluorescence images of the label-free sample and output the virtually stained brightfield microscope image and the virtually stained birefringence microscopic image of the label-free sample that are substantially equivalent to corresponding brightfield and birefringence microscopic images of the same sample that has been chemically stained.Brief Description of the Drawings

[0011] FIG. 1A schematically illustrates virtual tissue staining and birefringence imaging of label-free amyloid deposits. The virtual staining schematic pipeline and its comparison to clinical workflow is illustrated.

[0012] FIG. IB illustrates the virtual staining network and digital staining matrix framework to generate two output modalities (brightfield and birefringence channels).

[0013] FIG. 1C illustrates the system for generating a virtually stained brightfield and birefringence microscopic images of a label-free sample.

[0014] FIGS. 2A-2B: Blind testing results of virtual birefringence imaging and histological staining of amyloid deposits. Two whole slide samples along with zoomed-in regions are shown. The upward-pointing arrows in ROI3 highlight amyloid deposition between cardiac myocytes, while the right-pointing arrows highlight areas devoid of amyloid deposits. The upward-pointing arrows in ROI2 denote amyloid deposition within blood vessels. The images were generated from a single neural network inference, which is deterministic for a given input.

[0015] FIGS. 3A-3B: Pathologists’ blind evaluation of brightfield Congo-red stain image quality (virtually stained vs. histochemically stained). A total of 163 image patches are evaluated, each by three pathologists. The scores are given for four metrics where their distributions are shown in violin plots in 3A. The mean and standard deviation values (as error bars) are plotted for each individual pathologist in FIG. 3B.

[0016] FIGS. 4A-4B: Pathologists' blind evaluation of birefringence microscopic images of Congo red staining (virtually stained vs. histochemically stained). A total of 112 images (56 histochemical-virtual pairs) underwent blind evaluation by three pathologists. Image quality was assessed across four metrics. Violin plots in FIG. 4A illustrate the distribution of scores for each metric. FIG. 4B presents the mean and standard deviation values (as error bars) for each pathologist.

[0017] FIGS. 5A-5D: Network architecture for virtual birefringence imaging and virtual staining of amyloid deposits in label-free tissue using autofluorescence microscopy and deep learning. FIG. 5A is a schematic overview of the generator, discriminator and registration module. FIG. 5B shows a detailed architecture and building blocks of the generator. FIG. 5C illustrates the detailed architecture and building blocks of the discriminator. FIG. 5D illustrates the detailed architecture of the registration module.

[0018] FIGS. 6A-6B show data preparation and the workflow of pathologist evaluations.

[0019] FIG. 7: Examples of image patches and pathologist scores for bright-field Congo red stain image quality.

[0020] FIG. 8: Examples of larger image patches and pathologist scores for birefringence image quality.

[0021] FIG. 9: Confusion matrices for the birefringence image quality scores of pathologists (Pl, P2 and P3) blindly comparing virtually stained and histochemically stained images.

[0022] FIGS. 10A-10C illustrates the quantitative evaluation results for comparing the histochemically and virtually stained polarization Congo red images. FIG. 10A shows two examples of dow n-sampled intersection-over-union (D-IoU) measurements between the segmented apple-green birefringence masks from histochemically and virtually stained polarization Congo red images. FIG. 10B illustrates the D-IoU distributions across all 56 testing pairs (histochemically and virtually stained fields-of-view (FOVs)). For the box plot, the median represents the central value. The box marks the interquartile range, with the lower and upper quartiles defining its boundaries. Whiskers extend from the box to encompass data points within 1.5 times the interquartile range; values beyond this range are considered outliers and plotted individually. All data points are overlaid on the plot with x-axis jitter to enhance visualization. FIG. IOC shows two examples of color histograms in YCbCr color space for the whole FOV and the birefringence regions only. The darker curves represent the distributions of histochemically stained images, while the lighter ones present the virtually stained images.

[0023] FIGS. 11A-1 IB illustrate virtual birefringence imaging with a shifted angle. FIG. 11 A illustrates the virtual staining neural netw ork is trained to simultaneously generate brightfield, cross-polarized birefringence, and birefringence images with a shifted angle using three digital staining matrices having pixel values of 1, -1 and 2, respectively. FIG. 11B shows three separate fields-of-view are shown in three columns. The top row displays histochemical polarization Congo red images. The second row presents virtually stained cross-polarized birefringence images. The bottom row shows virtual polarization images with a shifted angle that features yellow -colored amyloid deposits.

[0024] FIGS. 12A-12B illustrate a paired comparison of brightfield image quality. Confusion matrices (FIG. 12A) for the bright-field image quality scores of pathologists (Pl, P2 and P3) blindly comparing virtually stained and histochemically stained images. FIG. 12B illustrates per sample comparison of histochemical and virtual scores.

[0025] FIG. 13 illustrates example fields-of-view with scores for M5-M7. All images are selected from the training dataset.

[0026] FIGS. 14A-14D illustrates examples of larger image patches and pathologist scores for birefringence image quality. FIG. 14A illustrates the histochemically stained of FOV1 and score table. FIG. 14B illustrates a virtually stained of FOV1 and score table. FIG. 14C illustrates the histochemically stained of FOV2 and score table. FIG. 14D illustrates a virtually stained of FOV2 and score table.

[0027] FIGS. 15A-15B: Workflow for calculating quantitative metrics used for the comparison of histochemically and virtually stained Congo-red images. FIG. 15 A shows the workflow of quantifying various metrics for histochemical and virtual brightfield Congo-red images, including MAE, MS-SSIM, PSNR, FID, the number of nuclei per FOV and the average area of nuclei. FIG. 15B shows the w orkflow of quantifying various metrics for histochemical and virtual polarization Congo-red images, including MAE, MS-SSIM, PSNR, FID, D-IoU between segmented apple-green birefringence masks (histochemical and virtual), color histograms in YCbCr color space for the whole FOV and the birefringence regions only.

[0028] FIG. 16 illustrates color histograms in YCbCr color space for the entire test dataset. The YCbCr color histograms were obtained from all the test image FOVs. The blue (B) curves represent the distributions of the histochemically stained images, while the red (R) ones present the virtually stained images.

[0029] FIGS. 17A-17B: Quantitative evaluation results for comparing histochemically and virtually stained brightfield Congo-red images. FIG. 17A illustrates the number of nuclei per FOV within histochemically stained brightfield Congo-red images vs. the number of nuclei per FOV within virtually stained brightfield Congo-red images. FIG. 17B shows the average area of nuclei within histochemically stained brightfield Congo-red images vs. the average area of nuclei within virtually stained brightfield Congo-red images.

[0030] FIG. 18: Transfer learning for a sensor with a higher noise level. Panels (a-d) are autofluorescence images as captured using different fluorescence channels. Panels (e-h) are autofluorescence images with added Gaussian noise (cr=0.025). (i-j) Virtual Congo red images from the model trained and tested with the original autofluorescence images. Panels (k-1) are virtual Congo red images from the model trained with original autofluorescence images and tested with autofluorescence images that include additional Gaussian noise. Panels (m-n) are virtual Congo red images from the model trained and tested with autofluorescence images containing added Gaussian noise. Panels (q-x) are zoomed-in regions highlighted in panels (a-h), showing the differences introduced by the added noise.

[0031] FIG. 19: Ablation study with different numbers of input channels. Virtual brightfield and polarization Congo red images inferred by the network that was trained and tested with 1 channel (Cy5), 2 channels (TxRed and Cy5), 3 channels (FITC, TxRed and Cy5), and 4 channels (DAPI, FITC, TxRed, and Cy5) of autofluorescence images. The histochemical staining is shown in the last column for comparison.Detailed Description of Illustrated Embodiments

[0032] With reference to FIGS. 1 A and 1C, a system 10 is provided that uses a single trained deep neural network 12 to generate or output virtual histologically stained (e.g., Congo Red) brightfield microscopic images 14 and virtual birefringence microscopic images 16 of label-free tissue 100. The tissue 100 is typically mammalian (e.g., human) tissue and may include a fixed tissue sample that is formalin-fixed and paraffin-embedded. The tissue demonstrated herein included human cardiac tissue although the invention is not so limited (e.g., other tissues may include kidney, liver, and spleen by way of example). The tissue 100 may also include fresh tissue. In other embodiments, the sample may include tissue that is imaged in vivo. A fluorescence microscope 18 or other fluorescence imaging device is used to obtain autofluorescence images 20 of the label-free tissue 100. The fluorescence imaging device may include whole slide image (WSI) scanners or the like. The one or more autofluorescence images 20 may be obtained at a plurality of different channels (e.g., DAPI, FITC, TxRed, and Cy5). These channels may be obtain using different filters / filter sets as part of the fluorescence microscope 18 or other fluorescence imaging device. The trained deep neural network 12 is executed using a computing device 22 executing software 24 that includes the trained deep neural network 12. The computing device 22 includes one or more processors 26 that execute the software 24.

[0033] The deep neural network 12 is first trained with a plurality of matched autofluorescence images or image patches and their respective chemically stained images or image patches of the same sample(s) imaged with a brightfield microscope and a polarization microscope capable of birefringence imaging (i.e., training sample(s)). After training, the deep neural network 12 is then ready to receive autofluorescence images 20 of a test sample 100. The software 24 executing the deep neural network 12 is configured to receive one or more autofluorescence images 20 of the sample 100 and generate or output a virtually stained brightfield microscope image 14 and a virtually stained birefringence microscopic image 16 of the sample 100 that are substantially equivalent to corresponding brightfield and birefringence microscopic images of the same sample 100 that has been chemically stained.

[0034] The trained deep neural network 12, in one embodiment, is a generative adversarial network (GAN) (or in another embodiment a conditional GAN or (cGAN)) to rapidly and digitally convert autofluorescence microscopy images 20 of unstained tissue slides containing the sample 100 which are input to the trained deep neural network 12 into virtually stained brightfield images 14 and virtually stained (e.g.. Congo Red) birefringence images 16, closelyresembling the corresponding microscopic images of the histochemically stained samples, helping the identification of amyloid deposits in label-free tissue 100 disposed on microscope slides. The virtual histologically stained brightfield images 14 and virtual birefringence images 16 that are generated or output by the trained deep neural network 12 may include image patches that are then digitally stitched together into whole slide images (WSIs). These WSIs can then be reviewed by healthcare professionals such as pathologists to rapidly and reliably diagnose amyloidosis. For example, the output images 14. 16 (either patches or WSIs) may be displayed to the user on a display 30 with a graphical user interface 32 associated with the computing device 22 or another computing device that receives the WI virtual histologically stained brightfield images 14 and the virtual birefringence images 16. The user or pathologist is able to quickly navigate back and forth between brightfield and polarization visualization states. This allows the user to focus on a specific area of tissue 100 and examine its morphology in both brightfield and birefringence.

[0035] In some embodiments, as illustrated in FIGS. 1C, 2A, 2B, the virtual histologically stained brightfield images 14 and the virtual birefringence images 16 generated or output by the trained deep neural network 12 may have certain regions 34 of the respective images 14. 16 pre-highlighted for the user (e g., amyloid deposits) to highlight the problematic or other areas of concern. This may include certain visual cues or indicia over the virtual images 14, 16 that identify certain regions of tissue images that may warrant further inspection. A heat map may also be provided on the display 30 to the user to aid in diagnosing amyloidosis.

[0036] Experimental

[0037] Results

[0038] Label-free virtual birefringence imaging and amyloid staining

[0039] The virtual birefringence imaging and virtual Congo red staining system 10 and method were demonstrated by training a deep-learning model or neural network 12 on a dataset comprising label-free tissue sections with a total of 386 and 65 training / validation and testing image patches, respectively, where each image patch had 2048x2048 pixels, all obtained from eight distinct patients. To obtain this dataset, unlabeled / label-free tissue sections were labeled with a 4-channel autofluorescence microscope 18 and then sent these label-free slides for standard histochemical Congo red staining (for ground truth generation) to be used for training. The histochemically stained tissue slides were then imaged using a standard brightfield microscope scanner as well as a polarization microscope (see the Methods section). Manual fine-tuning of the polarizer and analyzer settings was conducted bya board-certified pathologist to ensure the quality of the captured birefringence patterns that served as the ground truth for the polarization channel. Following the image data acquisition, an image registration process was performed in the training phase to spatially register the brightfield and the polarization images to label-free autofluorescence training images to mitigate potential image misalignments. The total data size for all the training, validation and testing images amounts to ~40 GB. For more details on image dataset acquisition and preprocessing of data, refer to the Method section.

[0040] With reference to FIG. 5 A, the deep learning model or network 12 was composed of three sub-modules: (1) a generator G designed to leam the two necessary cross-modality image transformations, i.e., autofluorescence-to-birefringence and autofluorescence-to- brightfield imaging, (2) a discriminator D that engages in adversarial learning to differentiate between the output and ground truth images, thereby aiding the generator G during the training process, and (3) an image registration module R tasked with aligning the output images with the ground truth, which helps to mitigate residual misalignments within the dataset. The image transformations from autofluorescence to brightfield and birefringence modalities were learned within a single neural network model or deep neural network 12. To determine which output (brightfield or birefringence) is desired, a digital staining matrix 46 was concatenated to the input autofluorescence images 20 as seen in FIG. IB. This digital staining matrix 46, matching the pixel count and shape of the input images 20, determines the output modality on a per-pixel basis with “1” indicating brightfield and “-1” indicating polarization / birefringence channel. During the training, the target images from both modalities were mixed and concatenated the corresponding digital staining matrix 46 to the autofluorescence input images 20.

[0041] After the model convergence, a blind evaluation of the model or deep neural network 12 was conducted using sixty-five (65) test images (each with 2048x2048 pixels) obtained from two previously unseen patients. During the model testing phase, brightfield and birefringence output images 14, 16 were naturally aligned with the corresponding input autofluorescence images 20. For both the brightfield and polarization channels, the predicted virtual images 14, 16 exhibited a high degree of agreement with the ground truth images, as demonstrated in FIGS. 2A, 2B, which showcases side-by-side visual comparisons between the virtually stained images 14, 16 produced by the deep learning model alongside the corresponding histochemically stained ground-truth images. FIGS. 2A and 2B include two representative slides from distinct patients. Specifically, in FIG. 2A, the right-side displayszoomed-in sections of the histochemically stained brightfield images, revealing regions of interest (ROIs) with a pinkish hue indicative of congophilic areas. In the corresponding polarization images, these regions exhibit an apple-green birefringence, characteristic of amyloid deposits. The virtually created images 14, 16 of these same regions also manifest the same morphology of amyloid presence, affirming that the inference model has effectively learned to transform autofluorescence label-free images 20 into both virtual brightfield images 14 and virtual birefringence images 16, matching the histochemically stained counterparts, presenting an accurate appearance of amyloid deposits. The upward-pointing arrows highlight amyloid deposition between cardiac myocytes, while the right-pointing, arrows highlight areas devoid of amyloid deposits. In FIG. 2B, a similar comparative analysis is shown for another patient. In this case, the histochemically stained images display a specific region, labeled as ROI3, which is an area without any amyloid deposits ('‘negative region"). No congophilic features can be identified in the brightfield image, and no applegreen birefringence is seen in the polarization channel. The upward-pointing arrows denote amyloid deposition within blood vessels. As indicated by the WSIs, zoomed-in regions and arrow-pointed specific areas, the virtual staining model or deep neural network 12 correctly replicated these morphological characteristics, aligning closely with the histochemically stained ground truth images, without introducing false positive staining.

[0042] Pathologist evaluations and performance quantification

[0043] To further assess the effectiveness of the virtual staining approach, three board- certified pathologists were engaged to blindly evaluate the quality of histochemically stained and virtually stained images 14, 16. The evaluation comprised two distinct parts: 1) assessing the image quality of brightfield Congo-red stain and 2) evaluating the appearance and quality of amyloid deposits using large, bundled patch images of the polarization and brightfield channels. The first part involved examining small patches, solely from brightfield modality, which were randomly cropped from histochemically stained and virtually stained images 14, 16 without overlapping field-of-views (FOVs) (see FIG. 6A). A total of 163 test image patches (each with 1024x1024 pixels) underwent random augmentations, were shuffled and then presented to the pathologists in a blinded manner. The expert evaluations concentrated on four primary metrics: stain quality of nuclei (Ml), cytoplasm (M2), and extracellular space (M3), as well as the staining contrast of congophilic areas (M4). The first three metrics (Ml -M3) are standard for evaluating stained tissue images, while the last one (M4) is unique to Congo red staining, clinically relevant for amyloidosis diagnosis. For all metrics, thegrading scale ranged from 1 to 4, where 4 represents “perfect'’, 3 represents “very' good’", 2 represents “acceptable”, and 1 represents “unacceptable” quality. FIG. 3A visualizes the results using violin plots to compare the scores given to histochemically stained and virtually stained images 14, 16. The distributions of these plots show no major differences for any of the evaluation metrics. The mean values for each pathologist (P1-P3) are plotted in FIG. 3B with an error bar representing the standard deviations (also see Table 1 below).Table 1Top Number = AverageBottom Number = Standard Deviation

[0044] For the first three metrics (M1-M2-M3), slightly higher scores are given to the histochemical images compared to the virtually stained images 14, 16, with a mean difference of 0.229 (5.72%), 0.334 (8.35%), and 0. 114 (2.85%), averaging across all pathologists and all patches - out of a scale of 4. However, for M4, which is the most relevant for Congo red staining, the difference in the mean pathologist scores falls to a negligible level of -0.067 (1.67%) out of 4. Overall, the stain quality scores corresponding to the virtual and histochemical staining are closely matched, each falling within their respective standard deviations, as shown in FIGS. 3A-3B. Additional image FOVs supporting this conclusion can be found in FIGS. 7-8. The brightfield images of paired histochemical and virtually stained FOVs were further scored after a two-month washout period. These results, summarized in FIGS. 12A-12B. further support that the virtual brightfield images 14, 16 generated by the neural network 12 or model remain satisfactory and consistent.

[0045] The second part of the expert evaluations focused on the analysis of birefringence appearance and image quality using larger FOVs and included bundled brightfield and polarization images 14, 16. The same FOVs from histochemically stained and virtually stained images w ere included with different image augmentations (FIG. 6B). This evaluationincorporated three distinct metrics (M5-M7): amyloid quantification (M5), the quality of birefringence appearance (M6), and the consistency between brightfield and polarization images (M7). The grading scale ranges from 1 to 3 and is different for each metric: percentage for amyloid quantification with one-third spacing (i.e., 1 = 100%); very good, moderate, and non-diagnostic for apple-green birefringence image quality: categories of low, medium, and high for inconsistency between brightfield and polarized Congo red images (lower scores indicate higher quality with less inconsistent features). A detailed description of M5-M7 scoring can be found in the Methods section with some visual examples reported in FIG. 13. The pathologists conducted their evaluations using a custom image viewer or GUI 32 that offered a user-friendly interface, enabling easy toggling between the brightfield and polarization images of the same FOV and detailed examination of tissue areas of interest (refer to the Methods). Prior to the evaluation, the pathologists were familiarized with the examination process through a brief tutorial that utilized scored examples from the training dataset. FIG. 4A presents a violin plot corresponding to the test image set, comparing the distributions of these scores (M5-M7) given to histochemical and virtually stained images 14, 16. For metrics M6 and M7. the expert score distributions for the virtually stained images 14, 1 surpassed those of the histochemically stained images, with mean improvements of 0.17 (5.67 %) and 0.24 (8.33%) for M6 and M7, respectively. For M5, however, the mean difference in performance dropped to an insignificant level of 0.0066 (0.22%) in favor of the histochemically stained images. FIG. 4B further depicts the mean and the standard deviation values across all the samples for each pathologist (also see Table I).

[0046] Note that because the histochemically stained and virtually stained images 14, 16 for the same tissue FOVs were both included in the expert evaluation process, one can also compare the pathologists’ scores for individual image FOVs. For this image pair-based analysis, the average scores of the 3 pathologists were first compared on the same image FOVs, generated for the histochemically stained and virtually stained images 14, 16. The results reveal that the virtually stained label-free images 14, 16 were blindly given higher or equal performance scores by the expert panel in 66%, 91%, and 89% of the tissue FOVs for metrics M5, M6 and M7, respectively. A more detailed comparison of each pathologist’s individual scores is also reported in FIG. 9. Additional image FOVs can be found in FIGS. 14A-14D. These quantitative comparisons further demonstrate the success of the virtual birefringence imaging and label-free tissue staining method using deep learning.

[0047] Virtual staining model quantitative evaluation

[0048] To further validate the results, several quantitative metrics were employed (see the Methods section and FIGS. 15A-15B) to assess the level of agreement between virtually stained images 14, 16 and histochemically stained images. Initially, birefringence images 16 were the focus due to their clinical significance in amyloidosis detection. The analysis comprised two aspects: the area of the apple-green regions and the color distribution of the images as seen in FIGS. 10A, IOC. Table 2 below presents the image comparison metrics that were used: mean absolute error (MAE), multiscale structural similarity index metric (MS- SSIM), peak signal-to-noise ratio (PSNR), and Frechet inception distance (FID).Table 2

[0049] The metric values (low MAE and FID; high MS-SSIM and PSNR) and low standard deviations indicate a strong agreement between virtually stained images 14. 16 and the corresponding histochemical ground truth. The apple-green birefringent regions were segmented and measured the accuracy of the system’s predictions using dow n-sampled intersection-over-union (D-IoU); see the Methods section for details. Two example FOVs with their segmented masks and the corresponding D-IoU values are shown in FIG. 10 A. FIG. 10B also displays the D-IoU distribution of all the samples, highlighting the high concordance between the output images 14, 16 and the ground truth, where the regions containing amyloid deposits in the output images overlap significantly with those in the target images.

[0050] Next, the color distribution of (1) the segmented apple-green areas and (2) the entire images were assessed. The virtual and histochemical images of two FOVs w ere converted into Y CbCr channels histograms of each channel w ere plotted where the luminance information is stored as a single component (Y), and the chrominance information is stored as two color-difference components (Cb and Or). The results, shown in FIG. 10C. demonstrate a very good agreement in the color distribution, validating the color accuracy of the inference of the deep neural network 12. To further confirm the color accuracy, the pixel value distribution across the entire testing dataset was calculated and plotted the results in FIG. 16. which revealed a strong concordance across all the channels, as desired.

[0051] For the brightfield Congo red images, similar quantitative metrics were applied (MAE, MS-SSIM, PSNR, and FID), as shown in Table 3 below.Table 3

[0052] Two additional metrics commonly used for brightfield images were added: the total count and the average size of nuclei in an image per FOV. The results are displayed in the scatter plots shown in FIGS. 17A and 17B. These results further affirm the concordance between the virtual staining results and the histochemically stained ground truth brightfield images, demonstrating the effectiveness of the method.

[0053] Transfer learning for noise resilience

[0054] To demonstrate the neural network ’s / model’s ability to handle variations in image signal-to-noise (SNR), transfer learning was used to bring resilience against noise. The results of this analysis are summarized in FIG. 18 where Gaussian noise was digitally added to simulate autofluorescence images captured with lower SNR image sensors. Direct inference using those noisy autofluorescence images as input to a model trained on original images (before the addition of Gaussian noise) generated hallucinated output images that are not acceptable. However, after applying transfer learning to the original model or deep neural network 12 using noise-added images, the model / neural network 12 quickly learned to adapt to the noisy inputs and inferred stained images 14, 16 that closely match the histochemical ground truth images (see FIG. 18). This showcases that the virtual staining neural network 12 can be rapidly fine-tuned through transfer learning to effectively adapt to new configurations and maintain high fidelity.

[0055] Discussion

[0056] A virtual birefringence imaging and virtual Congo red staining system 2 and method were created that allows for amyloidosis identification. The innovative approach enables label-free imaging and detection of amyloid deposits. Specifically, tissue autofluorescence texture was leveraged at the sub-micron scale to generate virtual images 14, 16 that mimic the Congo red-stained tissue brightfield and polarization images, facilitating accurate identification of amyloid deposits within label -free tissue 100.

[0057] In recent years, there have been accelerated efforts to overcome some of the challenges seen in traditional glass slide-based pathology. These have led to the developmentand adoption of digital imaging systems and WSI scanners that have helped with the transition of pathology into the digital era. While WSI scanners have demonstrated their capabilities to digitize various histochemical and immunohistochemical stained slides, automatically digitizing slides under polarized light remains technically challenging in histology labs due to delicate polarization components and varying illumination conditions. When visualizing a tissue slide under polarized light, constant adjustments of the tissue orientation and polarizer are required to ensure accurate and robust detection of birefringence patterns. In fact, none of the commercially available, clinically approved WSI scanners in the digital pathology field can automatically digitize birefringence images of tissue samples. As a result, pathologists continue to rely on manually operated light microscopes and standard polarizers for amyloid deposit inspection / detection, which is also at the heart of diagnostician-induced errors. Moreover, the quality of the light microscope and polarizers significantly influences the detection ability of amyloidosis by the practicing pathologist. In instances with low amounts of amyloid deposits within the tissue, the microscope’s ability to visualize the slide with high contrast and resolution can make the difference between a correct and false diagnosis. By training a virtual staining neural network 12 to transform label-free tissue autofluorescence images 20 into brightfield and birefringence microscopy images 14, 16 that are equivalent to the Congo red stained images of the same tissue 100, as it would normally appear after chemical staining, this offers a deep learning-based solution to both the challenging nature of the histochemical Congo red staining process as well as the digitization of the birefringence images of stained tissue sections, which currently does not exist in clinical WSI scanners. The virtual polarization imaging and tissue staining method ensures consistent and reproducible imaging of amyloid deposits within label-free tissue, eliminating the manual processes needed in both the chemical staining of tissue and the constant polarizer and tissue adjustments routinely performed by diagnosticians when using a polarized light microscope. Bypassing such manually operated polarization microscopes, the method can be readily adopted on standard pathology WSI scanners approved for digital pathology’ by simply using standard filter sets commercially available for capturing autofluorescence images 20 of tissue slices. This also eliminates the need for microscopyhardware changes or specialized optical parts in digital pathology scanners that are already deployed.

[0058] In addition to cross-polarized imaging, the system 2 can also enable pathologists to virtually rotate the polarization filters by a small angle. This introduces an additionalcapability to virtual Congo red staining and can be used to further validate the presence of amyloid deposits, which should change from an apple-green color to a yellow-like color with ~10 degrees rotation of the polarization filter. For its proof-of-concept, this capability was demonstrated by adding another channel in the DSM 46, where a pixel value of 2 in the staining matrix represents the angle-shifted birefringence channel, as shown in FIG. 11 A. This new neural network 12. with three (3) different virtually stained output images in its inference, was then trained using digitally emulated images where the color of the amyloid deposits changed from apple-green to yellow (see the Methods section), as expected from a slight shift of the polarizer angle. The blind testing results of this new virtual staining network, shown in FIG. 1 IB, indicate that it successfully generated birefringence output images where the amyloid deposits appear in yellow, in addition to generating the other two channels, i.e., the cross-polarized and brightfield images. These results serve as a proof-of- concept demonstration of the DSM's multiplexing capability and its potential for additional output channels to further assist diagnosis by virtually transforming the polarization filter into different states, as desired.

[0059] Quantitative and comparative analyses of three board-certified pathologists revealed that the virtually stained slides, with their birefringence and brightfield image channels 14, 16, were on par with chemically stained Congo red slides. Since the label-free approach is not critically dependent on manual labor in its staining and imaging processes, it is particularly beneficial in cases with low amounts of amyloid deposits, which can be easily missed when a diagnostician examines the slide without a high-quality polarizing microscope. In such cases, the method may increase the diagnostic yield and decrease false negative rates. An interesting future direction can be automated label-free detection of regions of amyloid deposits to highlight the problematic areas, which may further assist pathologists with attention heatmaps to accelerate diagnosis speed and reduce false negative rates.

[0060] An ablation study was also conducted (see FIG. 19) to show that with the decreasing number of input autofluorescence channels, the output image quality of the virtual staining network decreases. This indicated that four input autofluorescence channels (DAPI. FITC, TxRed and Cy5) are indeed essential to acquire the necessary information for successfully generating the virtually stained images 14, 16 of amyloid deposits.

[0061] The presented label-free approach has several inherent advantages compared to traditional histochemical Congo red staining. It minimizes staining artifacts compared tochemical staining techniques; it substantially reduces reliance on manual labor, and the virtual staining process diminishes the utilization of hazardous chemicals during slide preparation. It is important to emphasize that all the training processes described for the virtual staining algorithm, including the use of different modalities and processing pipelines, are a one-time effort. Once the model or deep neural network 12 is trained, the blind inference process (virtual staining with brightfield and birefringence images 14, 16 or channels) of a new, unknown sample will only require seconds per FOV (e.g., the inference time for a FOV size of 2048 x 2048 pixels is < 2 sec); therefore, it can transform a label-free whole slide within a few minutes using a state-of-the-art GPU processor 26.

[0062] In addition to the technical challenges in amyloidosis detection, the spotty nature of the disease and variations in the density of amyloid deposits increase the odds of false diagnoses. Thick tissue sections (e.g., 8-10 pm) are recommended for accurate Congo red birefringence visualization, as they provide more intense staining and allow for the identification of smaller amyloid deposits compared to the commonly used 4 pm sections in pathology. Nonetheless, thicker sections can negatively affect the amount of residual tissue and may have a detrimental impact on the depletion of small biopsy blocks, such as those used in cardiac biopsies. Insufficient tissue is often a limiting factor in performing additional stains or molecular studies. This method, relying on tissue autofluorescence rather than imaging under polarized light, would be less sensitive to tissue thickness and potentially can assist clinicians in reaching the same diagnostic conclusions while sparing tissue.

[0063] The virtual staining network's performance was superior in the polarization Congo red channel and comparable in the brightfield channel, when benchmarked against the histochemical ground truth. This change in behavior might be due to the trade-off introduced by the DSM 46: compared to training two distinct neural networks for brightfield Congo red and birefringence image generation, where an optimal model for each modality could be created, employing a DSM 46 within a single neural network 12 may result in a model that is optimal for one imaging modality7but slightly suboptimal for the other. On the other hand, DSM inference provides better structural alignment between the output channels (as quantified by M7). whereas two separate network models might produce divergent artifacts and inconsistencies, potentially confusing pathologists during diagnosis; that is why, the presented architecture of the DSM-based multi-modal inference w as utilized to bring consistency between virtually generated polarization and brightfield channels.

[0064] A limiting factor in evaluating the system 2 was the scarcity and limited size of cardiac biopsy samples that were available. Virtual Congo red staining was trained and tested on tissue slides obtained from a single pathology laboratory, which may not exhaustively represent all clinically significant features observable in cardiac amyloidosis samples. For example, other tissue components like collagen can exhibit some birefringence that differs in appearance from amyloid fibrils. It is important to note that Congo red staining highlights amyloid deposits, and stains collagen / elastic fibers with far less affinity. Therefore, the typical apple-green birefringence signatures distinguish amyloid deposits from other fibrils and from the white birefringence of fibrin or collagen. Moreover, since the virtual staining approach does not rely on birefringence images at its input channel and only uses autofluorescence images 20 of label-free tissue 100. this approach should, in principle, be immune to various forms of non-specific tissue birefringence as long as their microscopic spatial features at multiple autofluorescence image channels do not substantially overlap with the spectral and spatial features of label-free amyloid autofluorescence. Thus, the fact that the input images 20 utilize four different channels of autofluorescence (i.e. , DAPI, FITC, TxRed and Cy5) helps with the specificity of the virtual staining approach for amyloid deposits.

[0065]

[0066] In conclusion, virtual birefringence imaging and virtual tissue staining has been demonstrated that effectively transforms slides with label -free cardiac tissue 100 into virtual images 14. 16 that match their Congo red-stained histochemical counterparts, viewable in both brightfield and polarization channels. The method, as a fully digital process, can generate virtually stained WSIs in minutes with high repeatability, eliminating the variations and limitations associated with manual tissue staining methods and the use of polarization microscope components. By reducing the turnaround time, manual labor, and reliance on hazardous chemicals, this method can potentially transform the traditional workflow for the diagnosis of amyloidosis and set the stage for a large-scale, multi-center trial to further validate the clinical utility of the results.

[0067] Methods

[0068] Sample preparation and data acquisition

[0069] Unlabeled heart tissue sections (label-free tissue 100) were obtained from USC Keck School of Medicine under Institutional Review Board (IRB) #HS-20-00151 with ethical approval granted. Following the clinical standard for amyloid inspection, 8 pm sections were cut from archived cardiac tissue blocks that tested positive for Congo red. Samples with lessthan 5 mm2tissue area or less than 5% amyloid-involved tissue, determined by the original pathology report, were excluded. After autofluorescence scanning, the standard Congo red staining was performed.

[0070] Autofluorescence images 20 of the aforementioned label-free cardiac tissue samples 100 were taken using a conventional scanning fluorescence microscope 18 (IX-83, Olympus) equipped with a *40 / 0.95NA objective lens (UPLSAPO. Olympus). These images 20 were captured at four distinct excitation and emission wavelengths, each using a fluorescence filter set in a filter cube: DAPI (Semrock DAPI-5060C-OFX, EX 377 / 50 nm, EM 447 / 60 nm), FITC (Semrock FITC-2024B-OFX, EX 485 / 20 nm, EM 522 / 24 nm), TxRed (Semrock TXRED-4040C-OFX. EX 562 / 40 nm, EM 624 / 40 nm), and Cy5 (Semrock CY5- 4040C-OFX. EX 628 / 40 nm, EM 692 / 40 nm). Note that the selection of the filters was primarily empirical, based on the availability of commonly used fluorescence filter sets, which spanned the visible spectrum while having spectral selectivity to capture distinct autofluorescence bands effectively. The autofluorescence images 20 were recorded using a scientific complementary metal-oxide-semiconductor (sCMOS) image sensor (ORCA- flash4.0 V2, Hamamatsu Photonics) using exposure times of 150 ms. 500 ms, 500 ms. and 1000 ms for the DAPI, FITC, TxRed, and Cy5 filters, respectively. The exposure time for each channel was selected based on the intensity levels observed, which used a substantial portion of the dynamic range of the image sensor. pManager (version 1.4) software, designed for microscope management, was used for the automated image capture process. Autofocus is applied on the first autofluorescence channel (DAPI) for each FOV. Following the completion of the standard Congo red staining, high-resolution brightfield WSIs were obtained using a scanning microscope (AxioScan Zl, Zeiss) with a *20 / 0.8NA objective lens (Plan-Apo) at the Translational Pathology Core Laboratory (TPCL) at UCLA. Polarized images of Congo red-stained slides were captured using a modified conventional brightfield microscope (IX-83, Olympus) with a halogen lamp used as the illumination source without blue filters. The microscope was equipped with a linear polarizer (U-POT, Olympus) and an analyzer with an adjustable wave plate (U-GAN, Olympus) with a x20 / 0.75NA objective lens (UPlanSApo). The linear polarizer was placed on the condenser adapter (between the light source and the sample slide) and fixed in a customized 3D printed holder (Ultimaker S3, PETG black) to ensure polarization orientation during the scanning process. The analyzer was inserted into the slider-compatible revolving nosepiece, between the sample slide and theimage sensor. The orientations of all polarization components were adjusted by a board- certified pathologist for optimal image quality.

[0071] Image preprocessing and registration

[0072] The registration process is essential to successfully train an image-to-image translation network. An alternative is to apply CycleGAN-like architectures which do not require paired samples for training. However, such unpaired image-based approaches result in inferior image quality with potential hallucinations compared to training with precisely registered image pairs. Here, both the brightfield and birefringence images were registered individually to match the autofluorescence images 20 of the same samples using a two-step registration process. First, all the images were stitched into WSIs for all three imaging modalities and globally registered them by detecting and matching speeded-up robust features (SURF) on downsampled WSIs. Then, the spatial transformations (projective) were estimated using the detected features with an M-estimator sample consensus algorithm and accordingly warped the brightfield and birefringence WSIs. Following this, the coarsely matched autofluorescence, brightfield and birefringence WSIs were divided into bundles of image tiles, each consisting of 2048x2048 pixels. Using these image tiles, the accuracy of the image registration was further improved to address optical aberrations among different imaging systems and morphological changes that occurred in the histochemical staining process with a correlation-based elastic registration algorithm. During this elastic registration process, a registration neural network model was trained to align the style of the autofluorescence images with the styles of the brightfield and birefringence images. The registration model used for this purpose shared the same architecture and training strategy as the virtual staining network (detailed in the next section), however, without a registration submodule and was only used for the data preparation stage. After the image style transformation using the registration model, the pyramid elastic image registration algorithm was applied. This process involved hierarchically matching the local features of the subimage blocks of different resolutions and calculating transformation maps. These transformation maps were then used to correct the local distortions in the brightfield and birefringence images, resulting in a better match with their autofluorescence counterparts. This training and registration process was repeated for both brightfield and birefringence images until precise pixel-level registration was achieved. The registration steps were implemented using MATLAB (MathWorks), Fiji and PyTorch. To simulate the polarization filter rotation and generate images with yellow-colored amyloid deposits, the apple-green partof an image was first extracted using a segmentation algorithm (see the “Quantitative evaluation metrics for polarization Congo red virtual staining" section). Then, the hue channel was multiplied by a fixed parameter (~0.6) to transform the color of the amyloid deposits from apple-green to yellow. Meanwhile, the hue channel of the background regions was multiplied by -1.1 to simulate the background color change to dark blue. These parameters were tuned and approved by a pathologist.

[0073] Quantitative evaluation metrics for brightfield Congo red virtual staining

[0074] To quantitatively evaluate the performance of brightfield Congo red virtual staining, 56 FOVs of virtually generated brightfield Congo red images were organized together with their corresponding histochemically stained images for paired image comparison. Standard metrics of MAE, MS-SSIM, PSNR, and FID, as well as some other customized metrics were used for brightfield images, including the number of nuclei per FOV, and average area of nuclei per FOV, as shown in FIG. 15 A. MAE was defined as,

[0075] where A, B represent histochemically and virtually stained brightfield Congo red images, respectively, m and n are the pixel indices, and M x N denotes the total number of pixels in each image.

[0076] The MS-SSIM evaluated the SSIM between two images at different spatial levels, where the number of scales was set to 6. and the weights applied to each scale were set to [0.05, 0.05, 0.1, 0.15, 0.2, 0.45],

[0077] The PSNR was calculated using,

[0078] where A presents the ground truth image (histochemically stained), and MSE was defined as.

[0079] The FID was calculated using the default feature number.

[0080] As for the quantifications of nuclei properties, a stain deconvolution method was adopted to first separate the channel corresponding to nuclei staining. Then, Otsu’s thresholding (See Ostu, N. A threshold selection method from gray -level histograms. IEEE Trans SMC 9, 62 (1979), incorporated herein by reference) and a series of morphologicaloperations, such as image dilation and erosion, were applied to the nuclei channel to obtain the segmented binary nuclei mask. The number of nuclei per FOV was defined as the number of connected components in the binary nuclei mask, and the average area of nuclei per FOV corresponds to the average area of connected components across the whole binary nuclei mask with a unit of pixel.

[0081] Quantitative evaluation metrics for polarization Congo red virtual staining

[0082] To perform the quantitative evaluation of polarization Congo red virtual staining, a paired comparison was conducted on fifty-six (56) paired histochemically and virtually stained images 1 with the same FOVs used in the brightfield image evaluation. The metrics of MAE, MS-SSIM, PSNR, and FID were used as defined earlier. There were also some differences: (1) for MS-SSIM. the weights applied on each scale are [0.45, 0.2. 0. 15. 0. 1, 0.05, 0.05]; (2) for the calculation of FID, to adjust the image brightness, the histochemically and virtually stained polarization images 16 were transferred into YCbCr color space, multiplied the Y-channel values by 1.5, and then transferred the image back to the ordinary RGB color space. Additionally, a segmentation algorithm was used to isolate the apple-green birefringence regions in polarization Congo red images by empirically setting thresholds on the HSV channels. The Hue channel in the HSV space enabled the direct selection of the green color range for amyloid deposits. Thresholds were determined from sample FOVs and then validated and finetuned by a board-certified pathologist to ensure accurate amyloid segmentation. Subsequently, several morphological operations were applied, such as image opening and image closing. As shown in FIG. 15B, the D-IoU metric was used to measure the similarity betw een the segmented birefringence masks of the histochemically and virtually stained polarization Congo red images, defined as follow s,D_ . „ ,0Zm Zn[ min {(A32(m, n)) + (B32(m,n)), 1}]

[0083] where A32and B32correspond to the 32x down-sampled (bilinear) version of the segmented birefringence masks for the histochemically and virtually stained polarization Congo red images, respectively, and m and n denote the pixel indices. Before computing D- loU, 432and B32were binarized with a small threshold for logical operations. Compared to the traditional per-pixel definition of loU, this down-sampled D-IoU is more suitable to measure the similarity between binary masks and is resilient to small pixel misalignments. Finally, for the whole FOV and the apple-green birefringence regions only, the color distributions w ere drawn using probability7density’ functions fitted from histograms separatelycalculated in Y, Cb, and Cr channels to compare the color similarity between the histochemically and virtually stained polarization Congo red images.

[0084] Network architecture and training strategy

[0085] A conditioned GAN architecture for the deep neural network 12 was used to leam the transformation from the 4-channel label-free autofluorescence images (DAPI, FITC, TxRed. and Cy5) into the corresponding microscopic images of Congo red stained tissue under brightfield microscopy and polarized light microscopy. As shown in FIG. lb, this conditioned GAN employed a digital staining matrix 46, digitally concatenated with the autofluorescence input images 20, with all the elements of the matrix set to “1” corresponding to brightfield Congo red images, while “-1” refers to birefringence images. The conditioning of this digital staining matrix 46 can be represented as: c = [c or [c_ (5)

[0086] where c is concatenated to the input as an additional channel,denote matrices with all the elements equal to 1 and -1, respectively. Note that for the additional 3rdchannel required for yellow-colored amyloid deposits (see FIG. 11 A), Eq. 5 is expanded to c = [c_ or [c or [c2],

[0087] The GAN framework consists of two deep neural networks, a generator G and a discriminator D. During the training, the generator G is tasked with learning a statistical transformation to create virtually stained images. Concurrently, the discriminator D learns to distinguish the generated virtually stained images and their histochemically stained counterparts. This competitive training paradigm fosters the simultaneous enhancement of both neural networks. In the training, the generator (G) and the discriminator networks (D) were optimized to minimize the following loss functions:

[0088] where G(-) and D (■) refer to the outputs of the generator and discriminator networks, respectively; 7targetdenotes the ground truth image; and 7inputdenotes the input label-free autofluorescence images. The binary cross-entropy (BCE) loss is defined as:BCE(p(a), p(b)) = — [p(b)ln(p(a)) + (1 - p(b))ln(l - p(a))] (8)

[0089] where p(a) represents the discriminator prediction and p(h) represents the actual label (0 or 1). The total variation (TV) loss acted as a regularizer term and can be defined as:

[0090] where p and q are the pixel indices of the image / . The coefficients (a, (3, y) in Eq. 6 were empirically set as (10,1, 0.0001) based on the validation image set and were fixed for the finalized model evaluation on test samples. In Eq. 6, an additional registration module R (as shown in FIG. 5A and 5D) was also incorporated into virtual staining network 12 to remove the residual alignment errors between the generated images and their corresponding ground truth images. The registration module R was fed with a pair of images: a 'fixed1image, which was the ground truth, and a 'moving' image, the virtually generated one. This registration module outputs a Iransformalion / displacemenl matrix detailing pixel displacements needed to align the virtually stained image with the ground truth. Using the displacement matrix, a grid flow matrix containing new locations of each pixel was generated and applied to the virtually stained images via a resampling operation (denoted as °), which was implemented using PyTorch. During the training, the loss function of the registration module was defined as:TpSMTH (R(G( / input, c), / target)) (10)

[0091] where the smooth loss (SMTH) is defined as:

[0092] where T is the displacement matrix. X X Y denotes the total number of elements in the matrix; x and y are the element indices. The coefficients (A, p) in Eq. 6 were empirically set as (20,10).

[0093] Note that this registration module R is not the same as the registration models mentioned in the previous section: a registration model leams the style-transfer to help with the elastic registration and is solely used in the data preparation stage. Unlike other shift-invariant losses, the registration module A herein was trained simultaneously with the virtual staining model 12 to learn the misalignments between the target and output images 14. 16 from the virtual staining network 12. These misalignments include not only static optical aberrations but also complex tissue distortions introduced during the histochemical staining process, which can vary' from one tissue FOV to another. These distortions are difficult to characterize with a simple loss function or data augmentation techniques but can be effectively learned with an optimizable network module.

[0094] Both the discriminator D and registration modules R w ere only used in training; during the testing phase, only the generator netw ork G was used with a simple forw ard process to infer the virtually stained images.

[0095] To avoid possible exploding gradients, the smooth L loss represented as:

[0096] was used where A, B represent the images in the comparison, m and n are the pixel indices, and M x N denotes the total number of pixels in each image. <p was empirically set to 1.

[0097] FIG. 5B depicts the generator network G modeled on the attention U-Net architecture. This generator network G is composed of a series of four downsampling blocks 50 and four upsampling blocks 52. Each downsampling block 50 comprises a three- convolution-layer residual block 54, followed by a leaky' rectified linear unit (Leaky ReLU) with a slope of 0. 1. and a 2 x 2 max pooling layer with a stride size of 2, which downsample the feature maps and double the number of channels. The three-convolution-layer residual block 54 is formed by three consecutive convolution layers and a convolutional residual path building a bridge between the input and output tensors of the residual block.

[0098] The input for each upsampling block 52 is a fused tensor, concatenating the output from the preceding block with the corresponding feature maps at the matched level of the downsampling path passing through the attention gate AG connections. The attention gateAG passes a tensor through three convolution layers and a sigmoid operation, generating an activation weight map applied to the tensor to strengthen salient features. The upsampling blocks 52 bilinearly resize (2x) the concatenated tensors and then use the three-convolution- layer residual block to reduce the number of channels by a factor of four. The final upsampling block 52 was followed by a three-convolutional layer residual block together with another single convolution layer, reducing the number of channels to three (3), matching the ground truth images.

[0099] The discriminator network D, as depicted in FIG. 5C, processes the inputs that are either the virtually stained images generated by the network 12 or the actual ground truth images. It begins by transforming the input image into a 64-channel tensor via a single convolutional layer, followed by activation through a Leaky ReLU. Then, the resulting tensor passed through five successive two-convolutional-layer residual blocks 56, each of which set the stride size of the second convolutional layer as 2 for 2* downsampling and doubling the number of channels. These blocks were follow ed by a global pooling layer and two dense layers to obtain the output probability of the input being the ground truth (histochemically stained) image.

[0100] For the registration module R, as depicted in FIG. 5D, a U-net architecture was adopted akin to that of the generator, albeit with several key modifications. The network comprises seven pairs of downsampling and corresponding upsampling blocks, each integrated with a residual block 58. Following the final downsampling stage, a convolution layer is employed to double the feature channels, succeeded by a series of three consecutive residual blocks. Subsequently, another convolution layer halved the channels, transitioning the processed features to the upsampling block sequence. The output layer of the network utilizes a single convolution layer to condense the number of channels to two (2), corresponding to the two normal components (x and y) of the displacement matrix.

[0101] The training dataset contained 386 image patches, each with dimensions of 2048 x 2048 pixels, extracted from 8 distinct patients. During the training, the netw orks received image patches sized 256 x 256 pixels, randomly cropped from the larger 2048 x 2048 patches in the dataset. The generator G, discriminator D and registration modules R were all optimized using the Adam optimizers, starting with learning rates of 2 X 10-5, 2 X 10-6and 2 X 10“6, respectively. A batch size of 32 was maintained throughout the training phase. The generator / discriminator / registration module update frequency was set to 4: 1 : 1. The network converged after ~48 hours of training. The training and testing are done on a standardworkstation with GeForce RTX 3090 Ti graphics processing units (GPU) 26 in computing workstations 22 with 256GB of random-access memory (RAM) and Intel Core i9 central processing unit (CPU). The virtual staining network 12 was implemented using Python version 3. 12.0 and PyTorch version 1.9.0 with CUDA toolkit version 11.8.

[0102] Pathologists’ blind evaluations

[0103] Three board-certified pathologists were included to blindly evaluate the image quality of histochemically stained cardiac tissue sections. The blind evaluations were conducted in two ways: (1) Brightfield Congo red stain image quality on small image patches with M1-M4, which are standard quantification metrics for histology images; and (2) Large patch bundled images with both brightfield and birefringence images of the same FOV. In anatomic pathology, the evaluation of Congo red-stained slides under polarized light microscopy yields a binary result — positive or negative for amyloidosis. To enhance the quantification of the virtual polarization model, additional metrics were incorporated: amyloid quantification (M5), the quality of birefringence appearance (M6), and the consistency between brightfield and polarization images (M7). M7 indicates that amyloid deposits on Congo red-stained slides typically appear pink-salmon under a brightfield microscope, while the same areas exhibit apple-green birefringence under polarized light microscopy. Instances where pink-salmon areas do not show birefringence, or where applegreen birefringence occurs without the typical brightfield morphology, indicate inconsistency. Several representative FOVs were selected from the training dataset to establish baseline scores for evaluation metrics M5 to M7 (see FIG. 13). Note that these examples, along w ith their scores, w ere presented to the evaluating pathologists prior to their assessments to aid in their calibration and understanding of the scoring system.

[0104] For part 1 evaluation, small patches (1024 x 1024 pixels, each corresponding to ~166 X 166 pm2) are randomly cropped from histochemical and virtual images 14, 16 without any overlapping FOVs. Then, each image is randomly augmented in the following ways: original, left-right flip, top-bottom flip, and random rotation at 90, 180 and 270 degrees. A total of 163 images were randomly shuffled and sent to pathologists without labeling. Pathologists scored four metrics evaluating the brightfield Congo red image quality: stain quality of nuclei, stain quality of cytoplasm, stain quality of extracellular space and stain contrast of congophilic areas. For part 2, 56 large image patches (2048 x 2048 pixels, each corresponding to -332 x 332 pm2) were selected, and the brightfield and birefringence images of the same FOVs were collected. For each image bundle, both histochemically andvirtually stained images 14, 16 were collected, and randomly augmented the image bundles while ensuring a different augmentation method for each, resulting in a total of 112 image bundles to be scored. Pathologists scored three metrics evaluating the birefringence channel: amyloid quantification (% of positive areas / total tissue surface), birefringence image quality, and inconsistency between brightfield and polarization Congo red images. For part 2, the evaluation of the bundled images, the Napari (0.4. 18) viewer was customized as a graphical user interface (GUI) 32 for pathologists to conveniently evaluate the high- resolution images. The interface features keyboard bindings for switching between brightfield and birefringence views, selecting images, and controlling zoom functions for regions of interest using a mouse. All the participating pathologists were given a short tutorial on using the GUI 32, which included scored example FOVs selected from the training dataset. In both evaluations, any images that pathologists declined to give a score are excluded from the analysis.

[0105] While embodiments of the present invention have been shown and described, various modifications may be made without departing from the scope of the present invention. For example, while Congo red stain is emphasized in the results herein, it should be understood that other stains may also be used. The invention, therefore, should not be limited except to the following claims and their equivalents.

Claims

What is claimed is:

1. A method of generating a virtually stained brightfield and birefringence microscopic images of a label-free sample with a deep neural network comprising: providing the deep neural network that is executed by software of a computing device, wherein the deep neural network is trained with a plurality of matched autofluorescence images or image patches and their respective chemically stained images or image patches of the same sample(s) imaged with a brightfield microscope and a polarization microscope capable of birefringence imaging; obtaining one or more autofluorescence images of the label-free sample using a fluorescence microscope or fluorescence imaging device; inputting the one or more autofluorescence images of the label-free sample to the deep neural network, wherein the one or more autofluorescence images are concatenated with a digital staining matrix that corresponds to either a virtually stained brightfield microscope image or a virtually stained birefringence microscopic image; and the deep neural network outputting a virtually stained brightfield microscope image and a virtually stained birefringence microscopic image of the label-free sample that are substantially equivalent to corresponding brightfield and birefringence microscopic images of the same sample that has been chemically stained.

2. The method of claim 1 , wherein the stained brightfield microscope image and the virtually stained birefringence microscopic image of the label-free sample are substantially equivalent to a corresponding brightfield and birefringence microscopic images of the same sample that has been chemically stained with Congo red.

3. The method of claim 1, wherein the deep neural network is trained using a generative adversarial network (GAN) model.

4. The method of claim 1. wherein the deep neural network is trained using a generator network configured to learn statistical transformation between the matched autofluorescence images or image patches and their respective chemically stained images or image patches of the same sample(s) imaged with a brightfield microscope and a polarization microscope capable of birefringence imaging and a discriminator network configured todiscriminate between a ground truth chemically stained image of the sample(s) and the outputted virtually stained brightfield and birefringence microscopic images of the sample(s).

5. The method of claim 1, wherein the label-free sample comprises mammalian tissue.

6. The method of claim 1. wherein the label -free sample comprises a fixed or fresh tissue sample.

7. The method of claim 1. wherein the label -free sample comprises tissue imaged in vivo.

8. A system for generating a virtually stained brightfield and birefringence microscopic images of a label-free sample with a deep neural network, the system comprising: a computing device having software executed thereon or thereby, the software comprising a deep neural network that is executed by software of a computing device, wherein the deep neural netw ork is trained with a plurality of matched autofluorescence images or image patches and their respective chemically stained images or image patches of the same sample(s) imaged with a brightfield microscope and a polarization microscope capable of birefringence imaging, the software configured to receive one or more autofluorescence images of the label-free sample and output the virtually stained brightfield microscope image and the virtually stained birefringence microscopic image of the label-free sample that are substantially equivalent to corresponding brightfield and birefringence microscopic images of the same sample that has been chemically stained.

9. The system of claim 8, wherein the stained brightfield microscope image and the virtually stained birefringence microscopic image of the label-free sample are substantially equivalent to a corresponding brightfield image and birefringence images of the same sample that has been chemically stained with Congo red.

10. The system of claim 8, wherein the deep neural network is trained using a generative adversarial network (GAN) model.

11. The system of claim 8, wherein the deep neural network is trained using a generator network configured to learn statistical transformation between the matched autofluorescence images or image patches of the same label-free sample and their respective chemically stained images or image patches of the same sample(s) imaged with a brightfield microscope and a polarization microscope capable of birefringence imaging and a discriminator network configured to discriminate between a ground truth chemically stained image of the sample(s) and the outputted virtually stained brightfield and birefringence microscopic images of the sample(s).

12. The system of claim 8, further comprising a fluorescence microscope or fluorescence imaging device configured to obtain the one or more autofluorescence image(s) of the label-free sample.

13. The system of claim 8, wherein the one or more autofluorescence images are concatenated with a digital staining matrix that corresponds to either a virtually stained brightfield microscope image or a virtually stained birefringence microscopic image.

Images