Virtual histopathology staining from label-free autofluorescence lifetime images

Abstract

Methods of obtaining a stained image of a biological sample using a single channel label -free fluorescence lifetime image of the sample comprising a fluorescence intensity image and associated fluorescence lifetime image as described. The methods comprise obtaining a single channel composite autofluorescence intensity and lifetime image from the single channel label -free fluorescence lifetime image, and providing the single channel composite autofluorescence intensity and lifetime image as input to a deep learning model that has been trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image. Related methods, systems and products are also described.

Description

VIRTUAL HISTOPATHOLOGY STAINING FROM LABEL-FREE AUTOFLUORESCENCE LIFETIMEIMAGESFIELD OF THE DISCLOSURE

[0001] The present invention relates to methods of obtaining stained histopathology images from label- free autofluorescence lifetime images, and in particular chemically stained images such as Hematoxylin & Eosin (H&E) stained images and immunohistochemistry (IHC) stained images such as P40 and TTF1 stained images. Related methods, systems and products are also described.BACKGROUND

[0002] Fluorescence lifetime is characterised by a decay from the excited state to the ground state, which is independent of its intensity but extremely sensitive to the surrounding bio-environment. Fluorescence lifetime imaging microscopy (FLIM) has been shown to provide valuable information about the underlying metabolic state, pathological conditions, and constitution of biological samples by analysing endogenous fluorescence. By measuring the fluorescence lifetime of fluorescent molecules within a sample, FLIM can provide insights into various microenvironmental factors, including pH, ion concentration, and molecular interactions. Due to lifetime contrast, this has led to various applications in biology and medicine, such as cancer detection and diagnosis (Marcu, 2012; Wang, 2016).

[0003] Conventionally, quantitative analysis of lifetime contrast is prevailed by statistical methods, such as histogram of FLIM images or phasor characterisation (Datta, 2020; Alfonso-Garcia, 2016; Rahim, 2022), with the assistance of reference data, e.g., histological images. Due to the nature of statistical analysis, most current approaches can only reveal averaged lifetime of dominant components in FLIM images, rather than cellular-level information. Although a few papers (Unger, 2020; Wang, 2022) have addressed this by co-registering FLIM and histology images together, there are still outstanding challenges associating with this co-registration, such as underlying structure change caused by tissue preparation (Madabhushi, 2016). In addition, such reference data is not always available alongside FLIM images, and therefore, unable to provide instant interpretation immediately after FLIM imaging procedures.

[0004] Indeed, histopathology using stained tissue images remains the gold standard for tissue analysis, and crucial component of the clinical pathway for most cancer types, informing diagnosis, prognosis and treatment. For example, lung cancer is typically differentiated between non-small cell lung cancer (NSCLC) and small cell lung cancer (SCLC). Non-small cell lung cancer is the most common type of cancer (about 80-85% of lung cancers). There are different histological subtypes of NSCLC including adenocarcinoma, squamous cell carcinoma, large cell carcinoma, large cell neuroendocrine carcinoma, adenosquamous carcinoma and sarcomatoid carcinoma. Adenocarcinoma and squamous cell carcinoma are the most frequent types of NSCLC, together representing over 80% of cases. Adenocarcinoma starts in cells that secrete mucus whereas squamous cell carcinoma starts in squamous cells. A correct histologic diagnosis is becoming increasingly important because it has been found to predict responseand toxicity to therapies (Dietel et al. 2016). At present, histological subtyping is performed by expert pathologist analysis of stained tissue slices (e.g. H&E stained slices). The process is relatively slow and costly due to the manual labour involved in sample collection, processing and analysis. Even with automation, obtaining stained tissue images is associated with a significant cost and time delay.

[0005] The present invention has been devised in light of the above considerations.SUMMARY OF THE INVENTION

[0006] In this work, the present inventors investigated the capability of FLIM for generating virtual H&E stained digital images and immunohistochemistry stained images (in particular P40 and / or TTF1 stained images). The present inventors postulated that this may be possible using deep learning, and that if that success in this would represent an extremely valuable improvement to current clinical practice as it would avoid the need to analyse stained images. Indeed, FLIM images are label-free and an analysis of such images using deep learning could be done within minutes of collection at minimal cost, rather than days as is the case in current practice. Additionally, FLIM images can even be acquired in vivo using fluorescence lifetime endomicroscopy. They further postulated that the use of a combination of the autofluorescence intensity and lifetime information available in FLIM images, particularly when provided as a single image rather than an image stack (comprising an intensity image and a lifetime image) would enable computationally efficient and accurate prediction of stained images. They demonstrated that by combining a deep learning model with a contemporary image quality metric during training, they can generate clinical-grade virtual H&E-stained images and IHC images from label-free FLIM images acquired on unstained tissue samples. This was possible using single channel FLIM images, resulting in a fast, easy to implement (requiring no specialised preset filters, no multiplicity of filters and specialised microscopes, simply relying on a commercial FLIM microscope) and computationally efficient -but highly accurate- prediction (requiring processing of a single image). They further demonstrated that the inclusion of lifetime information, an extra dimension beyond intensity, results in more accurate reconstructions of virtual staining when compared to using intensity -only images.

[0007] Thus, according to a first aspect, there is provided: a method of obtaining a stained image of a biological sample using a single channel label-free fluorescence lifetime image of said biological sample, the label-free fluorescence lifetime image comprising a fluorescence intensity image and associated fluorescence lifetime image, the method comprising: obtaining, by a processor, a single channel composite autofluorescence intensity and lifetime image from the single channel label-free fluorescence lifetime image; and said processor providing the single channel composite autofluorescence intensity and lifetime image as input to a deep learning model that has been trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image. A composite image refers to an image where the value of each pixel depends on the value of the pixel in each of the autofluorescence intensity image and corresponding lifetime image.

[0008] Embodiments of the first aspect may have any one or more of the following optional features.

[0009] The deep learning model may have been trained using a loss function that includes an image quality metric. A loss function that includes an image quality metric may be a loss function that includesone or more terms that penalise a difference in structure and / or texture between a predicted stained image and a corresponding ground truth stained image. The deep learning model may have been trained using a loss function selected from DISTS loss, perceptual loss, and texture loss, optionally wherein the deep learning model has been trained using a DISTS loss. The deep learning model may be an image- to-image deep neural network, such as a generator of a generative adversarial network (e.g. a conditional GAN or denoising diffusion GAN), an autoencoder (such as a variational autoencoder), or a vision transformer (such as a residual vision transformer, ResViT).

[0010] A composite autofluorescence intensity and lifetime image may be an intensity -weighted lifetime image. An intensity weighted lifetime image may be a false-colour lifetime image with colour depending on lifetime and the corresponding intensity image as the alpha channel. An intensity weighted lifetime image may be an image obtained by multiplying pixel values in an autofluorescence intensity image by the corresponding pixel values in a corresponding fluorescence lifetime image. In embodiments, an intensity weighted lifetime image is a false-colour lifetime image with colour of each pixel depending on lifetime and the saturation of each pixel corresponding to the intensity of the corresponding pixel in the corresponding intensity image. In embodiments, the method comprises obtaining a false-colour lifetime image from the autofluorescence lifetime image by converting the (greyscale) autofluorescence lifetime image to a 3-channel RGB image using colour mapping. The conversion may use a fixed range of lifetime, such as e.g. [1.0ns, 5.0 ns]. In embodiments, an intensity weighted lifetime image is an image obtained by multiplying pixel values in an autofluorescence intensity image by the corresponding pixel values in a corresponding fluorescence lifetime image. Use of such composite images has been found by the present inventors to result in superior quality predicted stained images.

[0011] The single channel florescence lifetime image may be a fluorescence lifetime image that has been acquired using a excitation wavelength and range of emission wavelength identified using a A-to-A scan of one or more biological samples. The single channel florescence lifetime image may be a fluorescence lifetime image that has been acquired using a excitation wavelength and range of emission wavelength that is the same as that used to acquire a plurality of single channel fluorescence lifetime images used to train the deep learning model. The single channel fluorescence lifetime images used to train the deep learning model may have been images of a plurality of training biological samples of the same type as the biological sample for which a stained image is being predicted. Biological samples of the same type may be samples comprising the same types of cells or tissues. The single channel fluorescence lifetime images used to train the deep learning model may be images of a plurality of training biological samples comprising a first plurality of samples of a different type from the biological sample for which a stained image is being predicted and a second plurality of samples of the same type as the biological sample for which a stained image is being predicted. In embodiments, the deep learning model has been trained using images of the first plurality of samples then further trained using images of the second plurality of samples. The single channel florescence lifetime image may be a fluorescence lifetime image that has been acquired using a excitation wavelength and range of emission wavelength that is the same as that used to acquire a plurality of single channel fluorescence lifetime images used to trainthe deep learning model, wherein the excitation wavelength and range of emission wavelength have been identified using a A-to-A scan of one or more of the training biological samples.

[0012] The stain may be a chemical stain. The stain may be a H&E stain. The stain may be a immunohistochemistry stain. The stain may be a P40 stain or TTFI stain. The biological sample may be a tumour tissue sample. The methods of the present disclosure are applicable to any types of tissue samples, and in particular any types of tissue sample comprising tumour cells. The tumour may be a lung tumour, a colorectal tumor (e.g. coloreactal adenocarcinoma), or an endometrial tumour (e.g. endometrial adenocarcinoma). The biological sample may be a lung tissue sample, a colorectal adenocarcinoma sample, or an endometrial adenocarcinoma sample. The biological sample may be an ex vivo tumour tissue sample that has been previously obtained from a patient. The biological sample may be a fixed tissue sample, or a tumour microarray. A fixed tissue sample may be a biopsy. A fixed tissue sample may be a FFPE tissue sample. As the skilled person understands, any tumour tissue sample may comprise a mixture of tumour and normal (healthy) tissue). The fluorescence lifetime images may have been acquired using a fluorescence lifetime imaging microscope. The fluorescence lifetime images may instead be images of in vivo tissue, the images having been previously obtained from subject, for example using a fluorescence lifetime endomicroscope.

[0013] The method may further comprise obtaining a plurality of tiles of predetermined size from the composite fluorescence intensity and lifetime image, wherein providing the single channel composite autofluorescence intensity and lifetime image as input to a deep learning model comprises providing one or more of the plurality of tiles as input to the deep learning model. The method may further comprise excluding any tile that includes more than a predetermined threshold proportion of background pixels . The predetermined threshold may be 75%. The proportion of background pixels may be evaluated on the intensity image. Thus, the method may comprise receiving a previously acquired fluorescence lifetime image and obtaining a plurality of tiles (patches) from the image, wherein a tile / patch is a subset of an image of a predetermined size, and wherein the one or more images provided as input to the deep learning model are individual patches. Background pixels may be identified as pixels associated with an intensity value in the intensity image or the composite image below a first threshold or above a second threshold depending on whether background pixels are expected to have high or low intensity. For example, when the intensity image has been colour inverted or the composite image has been obtained using a colour inverted intensity image, pixels with an intensity above a second threshold (e.g. in the intensity image) may be identified as background pixels. The present inventors have empirically found that when the intensity image is an 8-bit greyscale image a second threshold of 250 was an advantageous value to use. The method may therefore comprise obtaining a predicted stained image corresponding to each of the one or more of the plurality of tiles, and optionally stitching said predicted stained images.

[0014] The method may further comprise pre-processing the single channel label-free fluorescence lifetime image using one or more of: thresholding of the intensity image, thresholding of the lifetime image, normalizing the intensity image, and colour inverting the intensity image. Thresholding of the intensity information may comprise setting all pixels in an intensity image that are below a predetermined minimum intensity to a value of 0. The predetermined minimum intensity may be selected such that all pixels th atare not associated with sample in the image are below the threshold. The predetermined minimum intensity may be 0. Thresholding of the intensity information may comprise setting all pixels in an intensity image that are above a predetermined maximum intensity to the predetermined maximum intensity. The predetermined maximum intensity may be 2000. Thresholding of the lifetime information may comprise setting all pixels in a lifetime image that are below a predetermined minimum lifetime to a value of 0. The predetermined minimum lifetime may be selected such that all pixels that are not associated with sample in the image are below the threshold. The predetermined minimum lifetime may be 0.0ns. Thresholding of the lifetime information may comprise setting all pixels in a lifetime image that are above a predetermined maximum lifetime to the predetermined maximum lifetime. The predetermined maximum intensity may be 5.0 ns. Colour inverting the intensity image of the single channel label-free fluorescence lifetime image may comprise converting the intensity image to an 8-bit greyscale image and performing bitwise inversion. Normalising the intensity image may comprise normalizing the pixel intensity values in the intensity image by dividing each pixel value by the maximum observed pixel value for the image. When the method comprising obtaining a plurality of tiles of predetermined size from the composite fluorescence intensity and lifetime image, the maximum value may be the maximum observed intensity value in the image prior to tiling. In other words, normalization may be performed at the whole slide level, not at the tile level. The deep learning model may have been trained using a training data set comprising, for each of a plurality of biological samples: (i) one or more training single channel composite autofluorescence intensity and lifetime images each obtained from a respective single channel label-free fluorescence lifetime image; and (ii) one or more corresponding training stained images. The training single channel composite autofluorescence intensity and lifetime images and the corresponding training stained images may have the same pixel size. Images with the same pixel size may have been obtained by downsampling one or more images to a predetermined pixel size, for example using bicubic interpolation. The method may further comprise downsampling the label-free fluorescence lifetime image to obtain an image at a predetermined pixel size. The predetermined pixel size may be determined based on the pixel size of the images used to train the deep learning model. The pixel size may be a pixel size within a predetermined range (e.g. within 10%) of the pixel size of the images used to train the deep learning model.

[0015] The deep learning model may have been trained using training composite autofluorescence intensity and lifetime images and corresponding stained images, wherein the corresponding stained images are images have been coregistered with the composite autofluorescence intensity and lifetime images using preprocessed corresponding stained images are images that have been preprocessed using contrast enhancing. Contrast enhancing may be performed using a histogram-based approach as known in the art, such as e.g. histogram equalisation, or by saturating any pixels with intensity in the top predetermined percentage of pixels in an image and in the bottom predetermined percentage of pixels in the image. For example, the top 1 % of pixels (by intensity) in an image may be set to the value of the highest observed intensity. Instead or in addition to this, the bottom 1% of pixels (by intensity) in an image may be set to the value of the lowest observed intensity (e.g. 0). The stained images may, instead or in addition to contrast enhancing, have been preprocessed prior to coregistration by converting to grayscale.For example, the stained images may have been converted to 8-bit grayscale images. Greyscale conversion may be performed prior to contrast enhancement. In embodiments, the stained images used for subsequent training of the deep learning model may be coloured images (i.e. stained images prior to preprocessing with contrast enhancement and / or grayscale).

[0016] The deep learning model may have been trained using training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images, wherein the training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images are images that have been coregistered using an affine transformation. Coregistration may use, for each pair of images to be coregistered, coregistration of an autofluorescence intensity image from which the composite autofluorescence intensity and lifetime image is obtained, and a corresponding ground truth stained image. The deep learning model may have been trained using training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images, wherein the training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images comprise images that have been obtained from coregistered autofluorescence intensity and lifetime images and corresponding ground truth stained images using image augmentation. Image augmentation may comprise creating a flipped version of one or more of the images and / or creating a randomly rotated version of one or more of the images.

[0017] The deep learning model may have been trained using a method comprising a pretraining step comprising training the deep learning model to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image thereby obtaining a pre-trained model, wherein the corresponding stained image is a chemically stained image, optionally a H&E stained image, using training data comprising, for each of a plurality of biological samples: (i) one or more training single channel composite autofluorescence intensity and lifetime images each obtained from a respective single channel label-free fluorescence lifetime image; and (ii) one or more corresponding training chemically stained images; and a transfer learning step in which the pretrained model is trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image, wherein the corresponding stained image is a an immunohistochemically stained image, optionally a P40 or TTF1 -stained image, using training data comprising, for each of a plurality of biological samples: (i) one or more training single channel composite autofluorescence intensity and lifetime images each obtained from a respective single channel label-free fluorescence lifetime image; and (ii) one or more corresponding immunohistochemically training stained images.

[0018] The method may further comprise one or more of: (i) identifying one or more regions of interest in the biological sample using the stained image output by the deep learning model, and determining a property of the single channel label-free fluorescence lifetime image associated with the regions of interest, optionally wherein the property is a fluorescence lifetime imaging signature; (ii) providing a diagnosis associated with the biological sample using the stained image optionally in combination with the single channel label-free fluorescence lifetime image, optionally wherein providing a diagnosis comprises identifying the presence of cancer or identifying a cancer subtype; (iii) providing a prognosisor treatment recommendation for a subject associated with the biological sample using the stained image optionally in combination with the single channel label-free fluorescence lifetime image; and (iv) selecting a subject associated with the biological sample based on analysis of the stained image optionally in combination with the single channel label-free fluorescence lifetime image, optionally wherein said analysis comprises using information in the stained image to identify a disease or disease subtype likely to be present in the subject.

[0019] The stain may be an H&E stain, and the method may further comprise providing the single channel composite autofluorescence intensity and lifetime image as input to one or more further deep learning models that have each been trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image, wherein the stain is a respective immunohistochemistry stain, optionally a P40 stain and / or TTF1 stain . In embodiments, the biological sample is from a subject diagnosed as having, being likely to have, or suspected of having non-small cell lung cancer, the respective immunohistochemistry stains include P40 stain and TTF1 stain, and the method further comprises diagnosing a NSCLC subtype using the immunohistochemically stained images output by the further deep learning models.

[0020] Also described according to a second aspect is a method of diagnosing a disease or disorder in a subject, the method comprising: receiving, by a processor, one or more single channel label-free fluorescence lifetime images of a biological sample from said subject; obtaining, by said processor, one or more stained images from each of said one or more single channel label-free fluorescence lifetime images using a method as described herein, such as e.g. the method of any embodiment of the first aspect; and providing, optionally by said processor, a diagnosis for said subject using said one or more stained images. The subject may be a subject diagnosed as having or being likely to have cancer, and providing a diagnosis comprises identifying a cancer subtype. The cancer may be lung cancer. Obtaining one or more stained images from each of said one or more single channel label-free fluorescence lifetime images may comprise obtaining a H&E stained image, a P40 stained image and a TTF1 stained image using respective deep learning models trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a respective corresponding stained image, and identifying a cancer subtype comprises determining whether the subject has one or more of large cell carcinoma, squamous cell carcinoma and adenocarcinoma.

[0021] Also described according to a third aspect is a method of providing a prognosis for a subject, selecting a treatment for a subject or selecting a subject for further diagnostic testing, the method comprising: diagnosing, by a processor, a disease or disorder in the subject using the method of any embodiment of the second aspect, and determining a prognosis for a subject, identifying a treatment for the subject or selecting the subject for further diagnostic testing using the results of the determining. Said determining a prognosis, identifying a treatment or selecting the subject for further diagnosis may be performed by the processor. The method may comprise: (i) identifying a lung cancer subtype in the subject using the one or more stained images; and determining a prognosis for the subject, wherein a subject identified as having adenocarcinoma has better prognosis than a subject identified as having squamous cell carcinoma; (ii) identifying a lung cancer subtype in the subject using the one or morestained images; and selecting a treatment for the subject based on the identified lung cancer subtype and optionally one or more predetermined clinical characteristics; or (iii) identifying a lung cancer subtype in the subject using the one or more stained images; and selecting the subject for further diagnostic test based on the identified lung cancer subtype and optionally one or more predetermined clinical characteristics. The further diagnostic test may comprise molecular testing.

[0022] Also described according to a fourth aspect is a method of obtaining a trained deep learning model for obtaining a stained image of a biological sample using a single channel label-free fluorescence lifetime image of said biological sample, the label-free fluorescence lifetime image comprising a fluorescence intensity image and associated fluorescence lifetime image, the method comprising: obtaining or receiving, by a processor, a training data set comprising, for each of a plurality of biological samples: (i) one or more training single channel composite autofluorescence intensity and lifetime images each obtained from a respective single channel label-free fluorescence lifetime image; and (ii) one or more corresponding training stained images; and said processor training a deep learning model to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image, using said training data set. The methods according to the present aspect can have any one or more of the features described in relation to the first aspect. In particular, any features of how the deep learning model has been trained described in relation to the first aspect apply as features of the method of training the deep learning model according to the present aspect.

[0023] Also described according to a further aspect is a system comprising: one or more processors and computer readable memory storing instructions that cause the processor to perform any method described herein, such as the method of any embodiment of the first, second, third or fourth aspects. The system may further comprise data acquisition means configured to obtain fluorescence lifetime images.

[0024] Also described according to a further aspect is a non-transitory computer readable storage medium containing machine executable instructions which, when executed on a processor, cause the processor to perform any method described herein, such as the method of any embodiment of the first, second, third or fourth aspects.

[0025] Also described according to a further aspect is computer program product comprising instructions which, when executed on a processor, cause the processor to perform any method described herein, such as the method of any embodiment of the first, second, third or fourth aspects.

[0026] The invention includes the combination of the aspects and preferred features described except where such a combination is clearly impermissible or expressly avoided.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0027] Embodiments and experiments illustrating the principles of the invention will now be discussed with reference to the accompanying figures in which:

[0028] FIG. 1 is a flow diagram illustrating a method for obtaining virtual stained images and applications of such methods in clinical contexts.

[0029] FIG. 2 shows a flow diagram of a method for providing a tool for obtaining virtual stained images.

[0030] FIG. 3 shows an embodiment of a system for implementing methods of the disclosure.

[0031] FIG. 4 shows results of virtual H&E staining from label-free FLIM images of non-small cell lung cancers, a and c are the false-colour FLIM images inputted into the deep learning model, where b and d are the corresponding patches within the rectangle indicated, respectively. Corresponding virtual H&E images are shown (e-h), along with true H&E images (i-l).

[0032] FIG. 5 shows results on the blind evaluation of virtual and real H&E stained lung cancer cores, a shows the results of a subjective quantification of virtual (dark) and real (light) H&E staining of six separate TMA cores by three pathologists, b shows an overall quantification of the results of a by the inter-observer scores of true and virtual histological images. Each score combines evaluation of nuclear and cytoplasmic detail, overall staining quality, and diagnostic confidence based on staining quality, quantified using a range of 1 (unacceptable) to 4 (perfect).

[0033] FIG. 6 shows lifetime signatures of seven cellular components in lung cancerous tissue, including tumours, lymphocytes, plasma cells, macrophages, neutrophils, fibroblasts, and red blood cells (RBCs). Deep learning-generated virtual H&E staining (b) has the capability to reconstruct those cell types from the FLIM image (a), enabling precise and timely identification of lifetime signatures of cellular components in the tissue. True H&E image is also presented for comparison, d illustrates the distribution of lifetime of the cells, demonstrating that seven cellular components annotated have distinct lifetime values.

[0034] FIG. 7 shows a comparison of virtual H&E staining using intensity, false-colour FLIM with the intensity image as the alpha channel (a-FLIM), and intensity-weighted false-colour FLIM (IW-FLIM). a is the true H&E staining, and e, f, g are the virtual H&E staining from intensity (b), a-FLIM (c), and IW-FLIM (d), respectively. The rectangle highlights variations in the synthetic results. Arrows marked 1 point out macrophages that are entirely absent in the intensity-based synthesis (e) but appear in the a-FLIM (f) and IW-FLIM-based (g) outcomes. Synthesis based on intensity can reconstruct macrophages when the signal is strong (bright dots pointed out by the arrows marked 2 and 3 in b), but it struggled to identify those with dimmer signals (the grey dot surrounded by the bright white dots, as indicated by the arrow marked 2 in b. The arrow marked 4 also indicates a few missing immune cells in intensity-based synthesis.

[0035] FIG. 8 shows a schematic diagram of the methods used in an example of the disclosure, a. High level overview of the method; b image post-processing to generate paired FLIM and H&E images in a. c. conditional GAN (pix2pix) and Deep Image Structure and Texture Similarity (DISTS) loss for the synthesis in a.

[0036] FIG. 9 shows, for each of four tumour microarray (TMA) cores: IW-FLIM images, corresponding virtual P40 stained images and real P40 stained images.

[0037] FIG. 10 shows, for each of the corresponding TMA cores on FIG. 9, enlarged areas (i.e. zoomed in versions) of IW-FLIM images, corresponding virtual P40 stained images and real P40 stained images.

[0038] FIG. 11 shows, for each of four tumour microarray (TMA) cores: IW-FLIM images, corresponding virtual TTF1 stained images and real TTF1 stained images.

[0039] FIG. 12 shows, for each of the corresponding TMA cores on FIG. 11 , enlarged areas (i.e. zoomed in versions) of IW-FLIM images, corresponding virtual TTF1 stained images and real TTF1 stained images.

[0040] FIG. 13 shows results of virtual H&E staining from label-free FLIM images of colorectal adenocarcinoma, a and d are the intensity-weighted FLIM images inputted into the deep learning model, where b and e are the virtually stained images, c and f are the true corresponding H&E stained images.

[0041] FIG. 14 shows results of virtual H&E staining from label-free FLIM images of endometrial adenocarcinoma, a and d are the intensity-weighted FLIM images inputted into the deep learning model, where b and e are the virtually stained images, c and f are the true corresponding H&E stained images.

[0042] FIG. 15 shows results of virtual H&E staining from label-free FLIM images of a Formalin-Fixed Paraffin-Embedded lung biopsy sample, a and b are the intensity -weighted FLIM images inputted into the deep learning model, c and d are the virtually stained images, e and f are the true corresponding H&E stained images.

[0043] FIG. 16 shows results of virtual H&E staining from label-free FLIM images of non-small cell lung cancers, a to d were conducted using the same cohort as depicted in Fig. 4. e to j were conducted using a second cohort of lung TME samples, and different scanning parameters.

[0044] FIG. 17 shows results of a comparison of virtual H&E staining with affine and elastic registration. Two FLIM images, after affine transformation (a and g), undergo further transformation using elastic registration (c and i). The corresponding displacement fields are presented in b and h, respectively. Notably, for optimal illustration, the vectors in b and h have been scaled up 10 times as the original vectors are too small. Virtual H&E images of a, c, g, and i are shown in d, f, j, and I, respectively.

[0045] FIG. 18 shows results of a comparison of virtual H&E staining by various DL models, a shows true H&E staining, b pix2pix without additional loss functions, c pix2pix with DISTS loss, d the original ResVit (Dalmaz et al., 2022) e the original DDGAN (Xiao et al., 2021).

[0046] FIG. 19 shows results of virtual H&E staining with random noise. Each column shows an input FLIM image with random noise added, sampled from a Gaussian distribution with the indicated parameters (top), and a corresponding virtual H&E staining image (bottom).

[0047] FIG. 20 shows results of virtual TTF-1 staining on two TMA cores. Figs. 20a-g show Virtual TTF-1 images from core 1 , where both intensity-based (b) and FLIM-derived virtual images (c, e) closely resemble real TTF-1 staining (a), enabling consistent lung AC diagnosis by pathologists (f, m). Figs. 20h- n show virtual TTF-1 images from core 2. Compared with real TTF-1 staining (h), both intensity- (i) and FLIM-derived (k) images produce virtual staining (j, I), which is suitable for lung AC diagnosis (m, n). However, red arrows show some mis-reconstructed cells, where intensity is inferior to FLIM in accurately reconstructing TTF-1 + cells.

[0048] FIG. 21 shows IHC, FLIM, Intensity and virtual IHC (based on FLIM or Intensity) for an example TMA core where intensity-based virtual TTF-1 image is ambiguous for pathologists to make confident decisions.

[0049] FIG. 22 shows results of virtual p40 staining on two TMA cores. Fig. 22a-g show virtual p40 images from core 1 , where both intensity-based (b) and FLIM-derived (c, e) reconstructions exhibit high fidelity to real p40 staining (a), enabling pathologists to reliably diagnose lung SqCC (f, m). Fig. 22h-n show virtual p40 images from core 2. Relative to real p40 staining (h), virtual staining (j, I) is achieved using both intensity-based (i) and FLIM-derived (k) images, providing a reliable basis for lung SqCC diagnosis (m, n). However, red arrows indicate regions of mis-reconstructed cells, where intensity-based imaging demonstrates inferior accuracy compared to FLIM in reconstructing p40+ cells.

[0050] FIG. 23 shows results for a case where both TTF-1 and p40 expression are observed. Fig. 23a shows the virtual staining results of p40 and TTF-1 on both intensity and lifetime from two sequential slices of one patient. Fig. 23b shows a histogram of normalised intensity on TTF-1 + and p40+ cells. Fig. 23c shows a lifetime histogram of TTF-1+ and p40+ cells.

[0051] FIG. 24 shows results for a pathologist evaluation of virtual TTF-1 and p40 staining compared to corresponding true IHC images. For each case, pathologists assessed positive and negative expression in tumour cells, normal pulmonary epithelial / basal cells as well as non-specific staining (including background and incorrect cell staining). Diagnostic confidence and decision-making were also evaluated to illustrate the overall quality of the methods for clinical decision-making. Fig. 24a shows results for intensity-based virtual TTF-1 staining. Fig. 24b shows results for FLIM-based virtual TTF-1 staining. Fig. 24c shows results for intensity-based virtual p40 staining. Fig. 24d shows results for FLIM- based virtual p40 staining.

[0052] FIG. 25 shows distributions of means and standard deviations for individual cores of four subtypes, within the test datasets. Fig. 25a shows normalised intensity value distributions. Fig. 25b shows lifetime values distributions. The distribution of intensity values exhibits higher homogeneity than lifetime values.DETAILED DESCRIPTIONAspects and embodiments of the present invention will now be discussed with reference to the accompanying figures. Further aspects and embodiments will be apparent to those skilled in the art. All documents mentioned in this text are incorporated herein by reference.

[0053] The methods described herein are computer implemented unless context indicates otherwise. Indeed, image analysis using deep learning models, and the process of training deep learning models is of a complexity, and in particular requires the analysis of large amounts of data through complex mathematics, that places the methods described herein far beyond the capability of mental investigation. Thus, any method described herein may be implemented in a computer system (in particular in computer hardware or in computer software) in addition to the structural components and user interactions described.

[0054] The term “computer system” includes the hardware, software and data storage devices for embodying a system or carrying out a method according to the above-described embodiments. For example, a computer system may comprise one or more processing units such as central processingunits (CPU) and / or graphics processing units (GPU), input means, output means and data storage. Preferably the computer system has a monitor to provide a visual output display. The data storage may comprise RAM, disk drives or other computer readable media. The computer system may include a plurality of computing devices connected by a network and able to communicate with each other over that network. It is explicitly envisaged that the computer system may consist of or comprise a cloud computer.

[0055] The methods described herein may be provided as computer programs or as computer program products or computer readable media carrying a computer program which is arranged, when run on a computer, to perform the method(s) described herein. The term “computer readable media” includes, without limitation, any non-transitory medium or media which can be read and accessed directly by a computer or computer system. The media can include, but are not limited to, magnetic storage media such as floppy discs, hard disc storage media and magnetic tape; optical storage media such as optical discs or CD-ROMs; electrical storage media such as memory, including RAM, ROM and flash memory; and hybrids and combinations of the above such as magnetic / optical storage media.

[0056] The present disclosure relates to the use of a deep learning model for obtaining stained images of tissue from label free autofluorescence lifetime (FLIM) images of the tissues.

[0057] As used herein, the term “deep learning model”, also referred to as “deep artificial neural network” or “deep neural network” refers to a machine learning model that has a neural network architecture with one or more hidden layers. Any DL model suitable for image-to-image translation may be used in the context of the present disclosure. A deep learning model as used herein can be an image- to-image deep neural network. For example, a deep learning model may be selected from: a generative adversarial network (GAN, see e.g. Goodfellow et al. 2014), an autoencoder (e.g. a variational autoencoder), and a vision transformer (e.g. ResiT, see Dalmaz et al. 2022) . A GAN is a neural networkbased machine learning model comprising a first neural network model called a generator, and a second neural network model called a discriminator. The GAN may be a conditional GAN or an unconditional GAN. The generator may be a convolutional neural network (e.g. a U-Net orV-Net, or an encoder-decoder without skip connections). The discriminator may be a convolutional neural network, such as e.g. a PatchGAN model. In the present context, the generator is configured to predict a stained image from an input FLIM image, and the discriminator is configured to classify input images between real stained images and virtual stained images. Both neural networks are trained together for their respective tasks. In embodiments, the machine learning model is conditional GAN. A conditional GAN is a GAN where the discriminator takes as input both the input of the generator and the output or ground truth image to be classified. In the present context, such a discriminator is trained to classify a stained image between a first class of real stained images (ground truth images) and a second class of virtual stained images (synthetic images output by the generator), conditional on a corresponding FLIM image. An unconditional GAN is a GAN where the discriminator takes as input the output of the generator or ground truth image to be classified (but noot the input of the generator). In embodiments, the DL model is selected from: a U-Net based GAN, FCNN-p2p, VGG-a2p, pix2pix, or conditional GAN. In embodiments, the DL model is a pix2pix model as described in Isola et al. (2017). In embodiments, the DL model is a variationalautoencoder. A variational autoencoder comprises an encoder model that learns a posterior distribution of a latent variable representation of input data, and a decoder model that learns to map the latent variable representation to output data. The decoder model may also be referred to as a generative model. In embodiments, the DL model is a residual vision transformer (ResViT).

[0058] The deep learning model used in embodiments of the disclosure is trained using a loss function that includes an image quality metric. An image quality metric refers to a metric that quantifies differences in one or both of texture and structure (or conversely rewards texture and / or structure similarity) between a ground truth image and a predicted (virtual image). Use of an image quality metric as part of the loss function used to train a machine learning model as described herein improves the quality of th e resulting virtual stained images. An image quality metric can be a texture-based metric, a structure-based metric, or a metric that integrates structure and texture features. A loss function that includes an image quality metric can include one or more texture-based metrics and / or one or more structure-based metrics. For example, a quality metric or loss function that includes an image quality metric can be selected from: Total Variation (TV, where the corresponding loss function can be any function that minimises the TV between ground truth and predicted images), Structural Similarity Index Measure (SSIM, where the corresponding loss function can be any function that minimises the SSIM between ground truth and predicted images), perceptual loss (a loss function that measures the difference between the high-level features of the ground truth and predicted images, where the high level features are typically extracted from a pre-trained CNN), texture loss (a loss function that measures differences in texture between two images, such as e,g, the style reconstruction loss described in Gatys 2016), and DISTS loss (a loss that combines a texture loss and a structure loss, as described below, see Equation (4) to Equation (6)). In the case of a GAN, the total loss used to train the model may comprise a generator loss, a discriminator loss, and a loss function that includes an image quality metric. For example, a loss function as provided in Equation (7) below may be used.

[0059] The explosive development of deep learning (DL) technologies is transforming conventional biomedical imaging analysis, leading to automated processing that could surpass human capabilities. One of the beneficial fields is virtual histological staining from microscopic images acquired by various imaging modalities using DL models, such as U-Net-based Generative Adversarial Networks (GANs). Rivenson et al. 2019a demonstrated the effectiveness of translating label-free autofluorescence intensity images to multiple histology staining for different organs using a supervised GAN with an extra constraint of total variation. Li et al. 2020 synthesised bright-field images on unstained carotid artery samples to Haematoxylin and Eosin (H&E), Picrosirius Red, and Orcein stain using the pix2pix GAN (Isola, 2017). Borhani et al. 2019 applied a custom DL model to generate virtual H&E stain from a combination of 16- channel two-photon excited fluorescence images with 1 -channel lifetime images collected by multi-modal microscopy on unstained tissue. WO 2021 / 133847A1 describes methods for digital staining of microscopy images using deep learning. In particular, a GAN architecture was used to learn the transformation from a label-free unstained auto-fluorescence input image (intensity only) to the corresponding brightfield image of the chemically stained sample, or from multiple channel autofluorescence lifetime images to the corresponding HER2, PR (progesterone receptor) or ER(estrogen receptor) immunohistochemistry (IHC) stained images. Two chemical stains were used: H&E and melanin. In both cases only the autofluorescence intensity was used, and the images were obtained using fixed predetermined filters, in particular a DAPI filter cube (for prediction of H&E images) and a DAPI filter cube and Cy5 filter cube (for prediction of melanin images). Separately, fluorescence lifetime images were used to predict HER2, PR and ER IHC stained images. In this case, a standard fluorescence lifetime microscope was used with two separate hybrid photodetectors receiving fluorescence signal in fixed predetermined channels (435-485 nm and 535-595 nm wavelength range, respectively). Thus, in the case of IHC two channels lifetime images are used. Further, although intensity is mentioned as potentially also usable, in such cases lifetime and intensity images are fed to the model as separate images. The methods described differ from those of the present disclosure in several important ways. First, none of the implementations uses a composite FLIM image which combines lifetime and intensity information. Indeed, lifetime alone is used for IHC (across multiple channels), and intensity alone is used for H&E. No single image combining both lifetime and intensity is used in any case. By contrast, the present inventors have shown that the use of such composite images led to an improved accuracy of predicted stained image. Further, none of the implementations use a deep learning model trained with a loss function that includes an image quality metric. Additionally, all implementations used predetermined fixed filters for autofluorescence data acquisition, and in the case of IHC and at least one of the chemical stains a plurality of channels is considered necessary to obtain a prediction. By contrast, the present inventors have shown that FLIM data acquired over a single channel can be used to obtain clinical grade H&E and IHC images, using the methods described herein, and in particular when using a channel identified using a A-to-A scan for the type of sample under analysis.

[0060] As used herein, the term “fluorescence lifetime imaging microscopy” (FLIM) refers to a technology that acquires time resolved fluorescence spectra. In the context of the present disclosure, the fluorescence spectra are acquired in a label-free manner, and therefore the fluorescence is autofluorescence of biological tissues, rather than fluorescence associated with fluorescent labels. FLIM enables the quantification of both fluorescence intensity and fluorescence lifetime, where the latter is obtained by estimating a fluorescence decay curve from time resolved fluorescence data by exponential curve fitting. In particular, a fluorescence decay curve is typically assumed to be an exponential function lo(t)=lo Exp(-t / T) where T is the fluorescence lifetime and Io is an initial maximal fluorescence. Estimation of fluorescence lifetime from time resolved fluorescence spectra is known in the art, and commercial devices and software exist to acquire time resolved fluorescence spectra and obtain lifetime from these. As used herein, a lifetime image may refer to a single component or multicomponent lifetime image. A single component lifetime image is an image obtained from FLIM data using single exponential fitting. A multi-component lifetime image is an image obtained from FLIM data using multi -exponential fitting. The present inventors have found the use of single component lifetime images to be advantageously simple and resulting in accurate prediction of stained images. A FLIM device as used herein may be a microscope, such as e.g. a confocal microscope (e.g. Leica TCS SP8 Confocal Microscope or Leica STELLARIS 8 FALCON FLIM microscope) or an endoscope, such as e.g. a fiber based FLIM system as described in Wang et al. 2022. FLIM images as described herein are acquired using a single excitationwavelength and a spectral band of emission. The excitation wavelength may be set to an excitation wavelength that has been identified as optimal for the particular tissue type and instrument, for example through a A-to-A scan. The methods described herein may predict stained images using single channel FLIM images. Single channel FLIM images are FLIM images acquired in a single acquisition step (i.e. the biological sample is scanned once using a selected excitation wavelength and range of emission wavelengths). This contrasts with methods that make use of a plurality of images acquired with respective filters, such as e.g. A DAPI filter, a cy5 filter and a cy3 filter. Indeed, the use of a single channel, preferably one that has been selected for the specific tissue being imaged through a A-to-A scan is both simpler to acquire, and more computationally efficient to process. However, this contains very different information from images acquired using a plurality of present filters, since the physical phenomena that manifest themselves in different fluorescence channels are different. The term “FLIM image” refers to an image that includes both intensity signal (also referred to herein as an intensity image) and lifetime signal (also referred to as a lifetime image). A lifetime image may be reconstructed from raw FLIM data using exponential fitting.

[0061] The methods described herein use FLIM images of tissues, such as cancer tissue. The images may be acquired ex vivo, such as e.g. using tissue from biopsies, resected tumours (such as e.g. tumour microarray samples). Such images may be acquired using a conventional FLIM imaging system. Alternatively, the images may be images that have been acquired in vivo (i.e. in situ), for example using a fiber based FLIM system.

[0062] The images used herein are images of tissue, the images and / or tissue having been previously obtained from a subject. The terms “subject and “patient” are used herein interchangeably. The subject is typically a human subject. The subject may be a subject who has been diagnosed as having or being likely to have cancer. The cancer may be lung cancer. The subject may be a subject who has not yet been treated for cancer (treatment naive). The subject may be a subject who has been diagnosed as having or being likely to have cancer, such as e.g. Lung cancer (e.g. using imaging technologies such as chest X-rays or molecular technologies) and who has been subject to a tissue biopsy (e.g. diagnostic biopsy) for further characterisation of the cancer (e.g. when the methods described herein are performed using images of ex vivo tissue samples). Alternatively, the subject may be a subject who has been diagnosed as having cancer and who has been subject to endomicroscopy using a fiber based FLIM system.

[0063] A stained image refers to an image of a tissue that has been stained using one or more stains. The stains may be selected from: chemical stains, and immunohistochemistry stains. Each stain can be referred to as a marker, and associates with a respective target or set of targets. A target may be a cellular structure or protein. A stained image as used herein typically refers to an image of a biological sample (e.g. a tissue) that has been stained with a single stain or a set of stains imaged together (e.g. a H&E stain). For example, a biological sample may be stained with a plurality of markers (e.g. a plurality of antibodies each associated with a respective label and recognising a respective target) that are each detected using a respective imaging channel (e.g. a different fluorescence channel associated with a respective fluorophore associated with each antibody). In such cases, a plurality of stained images maybe obtained, one for each channel, each image detecting the presence of a single marker. As another example, a biological sample may be stained with a set of stains (e.g. H&E) which are imaged together using the same imaging modality (e.g. brightfield microscopy). While practically speaking the stains in a set of stains may be associated with signals in respective ranges of wavelengths, these may still be referred to herein as a single channel image if the image is acquired as a single image using the same imaging modality. This is by contrast with multiplex immunofluorescence images in which each fluorophore is imaged in a separate acquisition step and therefore represents a separate channel. Machine learning models of the present disclosure may be trained to predict virtual stained images associated with a single target (which may have been imaged using any of a plurality of markers and respective channels) and / or with a single stain (e.g. H&E stain). Machine learning models of the present disclosure may be trained to predict virtual stained images associated with a single channel image. A chemical stain is a stain that uses one or more chemical dyes that associate with biological structures. A chemical stain can be a Hematoxylin & Eosin stain (H&E). The present inventors have found H&E staining to be particularly amenable to methods of the present disclosure as providing clinically relevant information that they have demonstrated could be accurately recapitulated using deep learning prediction from FLIM images. H&E staining uses a combination of two chemical dyes: hematoxylin and eosin. Hematoxylin stains cell nuclei (in purple-blue), and eosin stains the extracellular matrix and cytoplasm (in pink). Other cellular structures take different combinations of these colours. Thus, H&E staining enables the visualisation of cell and nuclei morphology, as well as tissue organisation. H&E staining is commonly used for cancer identification and typing. For example, in the context of lung cancer, H&E staining is typically sufficient to identify large cell carcinoma. However, this is the least common type of non-small cell lung cancer (NSCLC). Depending on the outcome of H&E analysis, further histopathology testing using immunohistochemistry is typically required to identify a NSCLC subtype (e.g. identify adenocarcinoma and squamous cell carcinoma).

[0064] Immunohistochemistry stains make use of labelled antibodies that target (i.e. selectively bind to) respective targets. The antibodies used are labelled with a detectable marker, such as a chromophore or fluorophore. An immunohistochemistry stain can be selected from: P40 and TTF1. The present inventors have found the prediction of P40 and / or TTF1 staining from FLIM images using the methods described herein to be particularly accurate and to have highly significant clinical meaning that is demonstrably reliably captured using the methods described herein. P40 and TTF1 stains are obtained using labelled antibodies that recognise (i.e. specifically bind to) the proteins ANp63 and TTF1 , respectively. The former is referred to as a P40 antibody. P40 and TTF1 are both markers that are commonly used for pathologic subtyping in lung cancer. Pathologic subtyping to distinguish between adenocarcinoma and squamous cell carcinoma is essential for the correct diagnosis, prognosis and treatment of lung cancer. The p63 gene contains two promoters that produce two isoforms: a first isoform contains the N-terminal transactivation domain (TAp63) and the other lacks this domain (ANp63). The P40 antibody specifically recognises the latter p63 isoform, which is expressed in squamous cell carcinoma (SCC) but not in adenocarcinoma (ADE). Therefore, P40 staining is commonly used to distinguish SCC from adenocarcinoma (ADE) and other non-small cell lung cancer (NSCLC)subtypes. P63 is normally expressed in the basal or progenitor cell layer of stratified epithelia (e.g. squamous, urothelial, bronchial), basal cells of some glandular epithelia (e.g. prostate), as well as myoepithelial cells of breast and salivary glands (Bishop, 2012). The p63 antibody routinely used in pathology laboratories recognises both the TAp63 and ANp63 isoforms, meaning poor discrimination between SCC and ADE. However, p40 recognises ANp63 exclusively, and is therefore able to accurately distinguish between SCC and ADE. Thyroid transcription factor 1 (TTF1), also known as NKX2-1 or thyroid specific enhancer binding protein, is a homeodomain containing transcription factor (Guazzi, 1990). TTF1 is preferentially expressed in thyroid, lung, and brain structures of diencephalic origin (Agoff, 2000; Lazzaro, 1991). TTF1 is expressed in a high percentage of lung ADE and is therefore used to differentiate between primary lung ADE (overexpressed in up to 95% of cases), and SCC (virtually all negative) (Kim, 2018). Thus, P40 and TTF1 can be used alone or in combination (for more confident and precise diagnosis) to perform lung cancer subtyping, where P40+ / TTF1 - tissue is indicative of SCC, P40- / TTF1+ tissue is indicative of ADE, and any other combination of markers is likely to be another (less common) NSCLC subtype.

[0065] FIG. 1 shows a flow diagram of a method of obtaining stained tissue images according to the disclosure. At optional step 104, a biological sample (such as e.g. a tissue sample) may be obtained, such as e.g. by performing a tissue biopsy on a subject or receiving a previously obtained tissue biopsy. This is optional as the methods described herein may start from previously obtained images and / or may use images obtained by endomicroscopy. The method comprises a step 106 of obtaining one or more previously acquired fluorescence lifetime tissue images, e.g. from a data store, user interface or computing device. Alternatively, step 106 can comprise obtaining one or more fluorescence lifetime images of the tissue in the sample obtained by imaging the sample obtained at step 104. A fluorescence lifetime image is acquired as a dataset that comprises two sets of information for each location (pixel) on the image: fluorescence intensity (also referred to as autofluorescence intensity image, fluorescence intensity image, intensity image or intensity information), and lifetime (also referred to as autofluorescence lifetime image, fluorescence lifetime image, lifetime image or lifetime information). Thus, step 106 typically comprises obtaining both of these sets of information.

[0066] At optional step 108, the images obtained may be preprocessed using one or more of upsampling, downsampling, stitching, grayscaling, thresholding, normalizing, colour inverting, tiling, filtering and contrast enhancing. In particular, the FLIM image may be upsampled or downsampled to obtain an image that has a predetermined pixel size. The predetermined pixel size may be between 100pm and 500pm, such as e.g. about 100pm, about 150pm, about 200pm, about 250pm, about 300pm, about 350pm, about 400pm, about 450pm, or about 500pm. The predetermined pixel size may have be chosen to match or be similar to a pixel size used in the training data set used to train the deep learning model. For example, a FLIM image that has a pixel size smaller than the predetermined pixel size may be downsampled to obtain an image that has the predetermined pixel size. Thresholding an image may comprise setting any pixel value outside of a predetermined range to the nearest boundary of the range. For example, raw intensity data from the FLIM images may be limited to a predetermined range, such as e.g. [0, 2000] and / or raw lifetime data from the FLIM images may be limited to a predetermined range,such as e.g. [0.0 ns, 5.0 ns]. Further, intensity images may be grayscaled, for example by converting each respective image to an 8-bit grayscale image. Further, the intensity image may be colour inverted. For example, an 8-bit grayscale image can be inverted via bitwise inversion (e.g. 0000 0000 (0 decimal) is converted to 1111 1111 (255 decimal); 0000 1111 (15 decimal) is converted to 1111 0000 (240 decimal), etc.). The intensity and / or lifetime image may be contrast enhanced. For example, a predetermined percentage of the pixel values at one or both low and high bounds of the pixel intensity range may be saturated (i.e. set to the upper boundary of the range). In embodiments, the intensity and lifetime images are not contrast enhanced. The present inventors have found that this was not necessary to obtain informative predictions. Stitching refers to the process of assembling patches / tiles of an image into a larger image. This may be advantageous for example when the data for a sample was acquired as a plurality of tiles. Thus, the plurality of tiles may be stitched into a whole slide image. The whole slide images may be used for resampling, i.e. for obtaining a plurality of tiles of a predetermined size that may not be the same as the original tile size. Normalising an image may comprise dividing the value of each pixel in the image by the maximum pixel value in the image. Normalising may be performed at the tile level or at the whole slide image level. Normalising is advantageously performed at the whole slide level, as this compensates for differences between samples in the training data and ensures that all tiles corresponding to the same sample have pixel values between 0 and 1 corresponding to the same range of original pixel values. For example, an intensity image may be normalised prior to being used to obtain a corresponding composite image. Normalisation may be performed after thresholding. Tiling refers to the process of obtaining a plurality of tiles of predetermined size. The predetermined size may be expressed in terms of numbers of pixels and may be selected based on the input data size expected by the deep learning model. Filtering refers to excluding one or more images or tiles of an image using one or more predetermined criteria. For example, tiles that have more than a predetermined percentage of pixels identified as background pixels in the composite FLIM image, the intensity image or the lifetime image may be removed from the plurality of tiles that are provided as input to the deep learning model. This advantageously saves using computing time to process images that are unlikely to be informative. In embodiments, tiles that have more than a predetermined percentage of pixels identified as background pixels in the intensity image may be removed from the plurality of tiles that are provided as input to the deep learning model.

[0067] At step 110, a composite image is obtained for each fluorescence lifetime image obtained, which is a composite of autofluorescence intensity and lifetime images obtained from the same fluorescence lifetime image (i.e. from the dataset comprising corresponding intensity and lifetime information). Such an image may be referred to herein as a composite FLIM image or simply composite image. In embodiments, the composite is an intensity -weighted lifetime image. An intensity weighted lifetime image is a lifetime image that is augmented with intensity information. An intensity weighted lifetime image is not a stack comprising separate lifetime and intensity images. An intensity weighted lifetime image can be referred to as alpha-FLIM image or IW-FLIM, depending on how the intensity information is used to weight the lifetime image. The intensity weighted lifetime image may be a false-colour lifetime image with colour depending on lifetime and the corresponding intensity image as the alpha channel. Such animage is also referred to herein as alpha-FLIM. Thus, an intensity weighted lifetime image may be a false- colour lifetime image with colour of each pixel depending on lifetime and the saturation of each pixel corresponding to the intensity of the corresponding pixel in the corresponding intensity image. Alternatively, the intensity weighted lifetime image may be an image obtained by multiplying pixel values in an autofluorescence intensity image by the corresponding pixel values in a corresponding fluoresce nee lifetime image. Such an image is referred to herein as IW-FLIM. When multicomponent lifetime images are used, the composite image may comprise a plurality of intensity weighted lifetime images, one for each component of the lifetime image.

[0068] At step 112, a composite image obtained at step 110 is processed using a deep learning model to obtain a corresponding stained image. The deep learning model has been trained using a plurality of training fluorescence lifetime images of biological samples (and in particular, single composite lifetime images obtained as described above) and, for each training fluorescence lifetime image, a corresponding stained image. A corresponding stained image is an image that shows the same tissue, after staining with a selected stain and imaging using an imaging modality appropriate for detecting the selected stain. Step 112 can comprise processing a composite image obtained at step 110 using a plurality of deep learning models each trained to generate a stained image associated with a different stain or set of stains. For example, a composite image may be processed using a first deep learning model trained to generate a corresponding H&E image from a composite FLIM image, and a further one or more deep learning models each trained to generate a corresponding IHC stained image. For example, the further one or more deep learning models can comprise a second deep learning model trained to generate a corresponding P40 stained image, and / or a third deep learning model trained to generate a corresponding TFF1 stained image. Each deep learning model has been trained using a training data set comprising composite FLIM images and corresponding stained images associated with the respective stain or set of stains that the model is trained to predict.

[0069] Steps 104 to 112 can be used as parts of a method of selecting a subject for further diagnostic testing. For example, the virtual stained image can be used to determine a likely cancer type or subtype present in the tissue, at optional step 114. At optional step 116, a further diagnostic test to be performed for the subject can be identified based on the result of the determination at step 114. Further, the one or more tests identified at step 116 may be performed at optional step 118. For example, the further diagnostic testing may be selected from mutation analysis and confirmatory histopathology using stained tissue slices.

[0070] Steps 104 to 112 may be used as parts of a method of providing a prognosis for a subject that has been diagnosed as having cancer. Such a method may comprise optional step 114 of determining a likely cancer type or subtype present in the tissue using one or more virtual stained images obtained at step 112, and optional step 123 of determining a prognosis for the subject, wherein the prognosis is associated with the identified cancer subtype. For example, in the context of lung cancer, a subject identified as having adenocarcinoma has better prognosis than a subject identified as having squamous cell carcinoma. Thus, step 123 may comprise identifying a first prognosis for a subject identified as having a first subtype of cancer (e.g. adenocarcinoma) and a second prognosis different form the first prognosisfor a subject identified as having a second subtype of cancer (e.g. squamous cell carcinoma). At optional step 125 the subject may be treated using a treatment that depends on the prognosis identified at step 123.

[0071] Steps 104 to 112 may be used as parts of a method of identifying a treatment for a subject that has been diagnosed as having cancer. Such a method may comprise optional step 114 of determining a likely cancer type or subtype present in the tissue using the virtual stained image, and optional step 120 of identifying a treatment for the subject based on the identified cancer subtype and optionally one or more predetermined clinical characteristics and / or results of further diagnostic tests, such as one or more tests identified at step 116 and performed at step 118. Selecting a treatment for the subject may comprise determining a tumour grade, wherein the determining uses a tumour grading scheme that is dependent on the identified cancer subtype, and selecting a treatment associated with the determined tumour grade. At optional step 122the subject may be treated using a treatment identified at step 120.

[0072] At optional step output results 124, the results of any one or more of the preceding steps may be output, such as e.g. provided to a user through a user interface, provided to a computing device or database.

[0073] Current clinical practice in the context of lung cancer includes a step of stratifying patients with non-small cell lung cancer (NSCLC) subtypes using histologically stained images. This allows patients to be selected for different downstream diagnostic steps and pathological analyses, all of which ultimately enable patients to be selected to receive specific therapies. Therefore, the cornerstone of treatment decision is based on the classification between lung cancer subtypes. For example, lung tissue biopsies are typically stained with a standard chemical stain such as H&E, to verify the presence of cancer tissue and identify cancer subtypes such as large cell carcinoma. When lung cancer is suspected and non-small cell carcinoma is suspected, further tissue slices are typically stained using an adenocarcinoma marker (e.g. TTF1) and a squamous cell carcinoma marker (e.g. P40). This enables detection of ADE and SCC. Then, once patients are diagnosed as having adenocarcinoma, the current clinical practice invo lves selecting these patients for mutation detection to determine whether they will benefit from targeted therapies. The methods of the present disclosure can be used to replace the multistep stained histopathology analysis with a single label-free image acquisition followed my one or more virtual staining analyses, leading to an accurate and crucially significantly faster and cheaper stratification step from which all subsequent clinical steps are derived.

[0074] The methods of the present disclosure find use in the treatment, prognosis and management of patients with cancer, including selection of patients for participating in a clinical trial, selection of an appropriate treatment for a subject, selection of appropriate further diagnostic testing, etc. Thus, also described herein is a method of selecting a subject that has been diagnosed as having cancer (e.g. lung cancer) for participation in a clinical trial, the method comprising: identifying a cancer subtype i n the subject by analysing one or more virtual stained images obtained using a method as described herein; and selecting or excluding the subject from participation in the clinical trial depending on whether the subject was classified as having a first cancer subtype or a second cancer subtype. Thus, also describedherein is a method of providing a prognosis for a patient that has been diagnosed as having cancer (e.g. lung cancer), the method comprising: identifying a cancer subtype in the subject by analysing one or more virtual stained images obtained using a method as described herein; and determining a prognosis based on the analysis. Identifying a treatment may comprise identifying a likely genetic alteration based at least in part on the analysis, and / or obtaining results of one or more genetic alteration tests based at least in part on the analysis, and identifying a treatment associated with a genetic alteration identified as likely to be present in the patient. Obtaining results of one or more genetic alteration tests may comprise receiving test results from a user or computing device. Alternatively, the method may further comprise analysing a sample obtained from the patient to determine the presence of one or more genetic alterations. The one or more genetic alterations may have been selected based on the classification. The treatment may be selected based on a combination of the cancer subtype identified and the identified genetic alterations. The method may further comprise administering the selected treatment.

[0075] FIG. 2 shows a flow diagram of a method for providing a tool (e.g. providing a trained deep learning model) for obtaining a stained image. In particular, the tool may refer to a computer implemented tool comprising a trained deep learning model for use as described herein such as e.g. by reference to Figure 1 .

[0076] The method comprises step 16 of obtaining a plurality of training fluorescence lifetime images of tissue, preferably from a plurality of samples or subjects. The method further comprises step 18 of obtaining, for each training FLIM image, a corresponding stained image of the same sample. A training fluorescence lifetime image can be a composite FLIM image as described herein. Each set comprising a FLIM image and a stained image of the same tissue may be referred to as a pair of training images. The stained image may also be referred to as ground truth image as it represents the ground truth that the machine learning will be trained to predict. The training images may be processed at step 20 using steps as described in FIG. 1 for the FLIM images, in particular by reference to step 108. The stained images may also be preprocessed using one or more of upsampling, downsampling, stitching, grayscaling, thresholding, normalizing, colour inverting and contrast enhancing. The FLIM images (i.e. either or both of the intensity and lifetime image) may be pre-processed using one or more of: upsampling, downsampling, stitching, grayscaling, thresholding, normalising and colour inverting. In particular, any FLIM and / or stained image may be upsampled or downsampled to obtain an image that has a predetermined pixel size. The predetermined pixel size may be chosen to be the largest pixel size available in the training data set. For example, all images that have a pixel size smaller than the predetermined pixel size may be downsampled to obtain a training dataset that has a common pixel size. Thresholding an image may comprise setting any pixel value outside of a predetermined range to the nearest boundary of the range. For example, raw intensity data from the FLIM images may be limited to a predetermined range, such as e.g. [0, 2000] and / or raw lifetime data from the FLIM images may be limited to a predetermined range, such as e.g. [0.0 ns, 5.0 ns]. Further, intensity images and / or stained images may be grayscaled, for example by converting each respective image to an 8 -bit grayscale image. Further, intensity images and / or stained images may be colour inverted. In embodiments, the intensity images are colour inverted. In embodiments, the stained images are not colour inverted. In embodiments,the intensity images and stained images are grayscaled for coregistration. In embodiments, the intensity images and stained images are not grayscaled for deep learning model training and providing as input to the deep learning model. For example, an 8-bit grayscale image can be inverted via bitwise inversion (e.g. 00000000 (0 decimal) is converted to 1111 1111 (255 decimal); 0000 1111 (15 decimal) is converted to 1111 0000 (240 decimal), etc.). Any of the training images may be contrast enhanced. For example, a predetermined percentage of the pixel values at one or both low and high bounds of the pixel intensity range may be saturated (i.e. set to the upper boundary of the range). This may be applied to the FLIM intensity images and / or the stained images. Stitching refers to the process of assembling patches / tiles of an image into a larger image. This may be advantageous for example when the data for a sample was acquired as a plurality of tiles. Thus, the plurality of tiles may be stitched into a whole slide image. The whole slide images may be used for co-registration of the FLIM and stained images as will be explained further below. The whole slide images may be used for resampling, i.e. for obtaining a plurality of tiles of a predetermined size that may not be the same as the original tile size. Normalising an image may comprise dividing the value of each pixel in the image by the maximum pixel value in the image. Normalising may be performed at the tile level or at the whole slide image level. Normalisi ng is advantageously performed at the whole slide level, as this compensates for differences between samples in the training data and ensures that all tiles corresponding to the same sample have pixel values between 0 and 1 corresponding to the same range of original pixel values. For example, intensity images in the training data may be normalised prior to being used to obtain composite images. Normalisation may be performed after thresholding.

[0077] The training images may comprise images from at least 50 samples. The training images may comprise images from a plurality of samples from the same tissue type, e.g. from lung tissue and / or lung cancer tissue, from colorectal cancer tissue, from endometrial cancer tissue, or any other type of tissue comprising tumour cells. The training images may comprise images from a plurality of samples from different tissue types. The different tissue types may all be tumour samples. The training images may comprise images from a plurality of samples of the same tissue type, showing different subtypes of cancer tissue. For example, the training images may show lung tissue, comprising images showing lung adenocarcinoma tissue and images showing squamous cell carcinoma tissue. The method may comprise step 22 of co-registering a training FLIM image and a corresponding training stained image. Coregistering a FLIM image and a stained image of the same sample may comprise assembling an image (e.g. a whole slide image / whole sample image) from a plurality of image patches, for the FLIM image, the stained image or both. Co-registering FLIM image and a stained image of the same sample may comprise applying any mapping method known in the art, such as an affine transformation and / or elastic registration. In embodiments, co-registration is performed using affine transformation (e.g. without an additional non-linear method). Affine transformation is a linear mapping method that preserve points, straight lines and planes. Co-registering a FLIM image and a corresponding stained image refers to the process of establishing a correspondence between a location on the FLIM image and a location on the stained image. Co-registration may be performed using an intensity image of the FLIM image and the corresponding stained image. In other words, only the intensity information of the FLIM image may beused for the co-registration process. This is particularly advantageous when the resolution of the two images is not the same, i.e. when a pixel in one image does not correspond to a physical area of the same size on the other image of the pair. Co-registration may not be necessary, although it may still be beneficial, when the images in a pair have the same pixel size. The training images that are used to train the deep learning model may be image patches. Thus, the method may comprise step 24 of obtaining patches as described by reference to FIG. 1 (and optionally filtering said training patches). For example, patches that have more than a predetermined percentage of pixels identified as background pixels in either the composite FLIM image (or a component thereof, e.g. the intensity image) or the corresponding stained image may be removed from the training data. Thus, the training images may comprise at least 5000 pairs of image patches, each pair comprising a FLIM image patch and a corresponding stained image patch. Obtaining the training images may comprise step 24 of obtaining a plurality of pairs of images that each comprise a pair of patches of a respective larger image (i.e. a patch of a FLIM image and a corresponding patch of the corresponding FLIM image). A patch may also be referred to herein as a tile. A patch or tile is a portion of a larger image. A patch or tile is associated with a predetermined size, typically expressed as two numbers of pixels (e.g. 256 x 256). Patch sizes are typically matched to the available input size for a chosen machine learning model. Obtaining training images can comprise downsampling or sub-sampling either or both of the FLIM and stained images, for example to obtain input images (whether whole images or patches) with a desired pixel size that represent a physica l area of a desired size, and / or to obtain FLIM and stained images of similar resolution (e.g. prior to co-registration), and / or to increase processing speed (e.g. where higher resolution images are unnecessary or where one of the two types of images in a pair has a lower resolution, making it unnecessary to maintain the higher resolution information in the other type of training images). At optional step 26, one or more additional pairs of images may be obtained from one or more of the pairs of images obtained at step 24, using image augmentation. For example, step 26 may comprise creating a flipped version of one or more of the image pairs and / or creating a randomly rotated version of one or more of the image pairs. For example, a set of images (e.g. one or more, or all of the original images) may be rotated by a randomly selected amount between predetermined boundaries, such as e.g. -15 to +15 degrees. As another example, a set of images (e.g. one or more, or all of the original images) may be horizontally flipped. The set of images that are flipped and / or rotated may be randomly drawn from an original set of tissue images included in the training images. Any augmentation applied to a training FLIM images is typically also applied to the corresponding stained image.

[0078] At step 28, the method comprises using the images obtained through steps 16 to 26 to train a deep learning model to predict a stained image from a composite fluorescence lifetime image, using the training fluorescence lifetime images and associated ground truth stained images. As described elsewhere, the deep learning model may be a deep artificial neural network, such as a generative adversarial network, trained with a loss function that uses an image quality metric. The model may be trained using a predetermined proportion of the training images, and tested using the remaining proportion of images. Thus, training the DL model may comprise evaluating the performance of the model using one or more test datasets. For example, training the DL model may comprise evaluating theperformance of the model using a test dataset that is a subset of the training dataset, such as where the training dataset is dividing into a training set comprising 80% (or 85%, 90%) of the training images which is used for training the model, and a test set comprising the remaining e.g. 20% of the training images which is used to test the model. The training set and test set may be formed from the training set such that all images from the same sample are in the same set. The training and test sets may be formed such that all patches from the same image are in the same set. In embodiments, the training images comprise a first plurality of images from a first tissue type and a second plurality of images from a second tissue type, and the model is trained using the first plurality of images, then further trained using the second plurality of images. Thus, the model may be trained using transfer learning, comprising a first training step in which the model is trained using a subset of the training set, then fu rther trained using another subset of the training set. This may be particularly useful for example when smaller amounts of training data are available for one tissue type (e.g. used for further training using transfer learning) than for another tissue type (e.g. used for the initial training). Alternatively, the model may be trained from the start using training images from a plurality of tissue types. This may be particularly useful when similar amounts of training data are available for each of the plurality of tissue types.

[0079] At optional step 30, the results of any one or more of the preceding steps may be provided to a user, computing device or memory. This may include the trained model, such as e.g. for use in a method as described herein such as by reference to FIG. 1.

[0080] FIG. 3 shows an embodiment of a system for analysing FLIM images (e.g. obtaining stained images, providing a diagnosis, prognosis or treatment recommendation using such images, etc.) and / or for providing a tool for analysing FLIM medical images, according to the present disclosure. The system comprises a computing device 318 which comprises a processor 306 and computer readable memory 308. In the embodiment shown, the computing device 318 also comprises a user interface 302, which is illustrated as a screen but may include any other means of conveying information to a user such as e.g. through audible or visual signals. The computing device 318 is communicably connected, such as e.g. through a network, to data acquisition means 314 (also referred to as “FLIM imaging means”), such as fluorescence lifetime imaging microscope or endomicroscope or computing device associated therewith, and / or to one or more databases 312 storing FLIM data. The one or more databases 312 may further store one or more of: one or more deep learning algorithms, training data, parameters (such as e.g. parameters of a deep learning model used to diagnose lung cancer subtypes), image data acquisition parameters, clinical and / or sample related information, etc. The computing device may be a smartphone, tablet, personal computer, server or other computing device. The computing device 318 is configured to implement a method for analysing FLIM images (e.g. by obtaining stained images), obtaining a tool for analysing FLIM images, and / or identifying a prognosis, treatment or selecting a subject for a clinical trial, as described herein. In alternative embodiments, the computing device 318 is configured to communicate with a remote computing device (not shown), which is itself configured to implement a method as described herein. In such cases, the remote computing device may also be configured to send the result of the method to the computing device. Communication between the computing device 318 and the remote computing device may be through a wired or wireless connection, and may occur over a local orpublic network 310 such as e.g. over the public internet. The data acquisition means 314 and / or the databases 312 may be in wired connection with the computing device 318, or may be able to communicate through a wireless connection over a network 310, such as e.g. through WiFi and / or over the public internet, as illustrated. The connection between the computing device 318 and the data acquisition means 314 and / or databases 312 may be direct or indirect (such as e.g. through a remote computer).

[0081] The following is presented by way of example and is not to be construed as a limitation to the scope of the claims.EXAMPLESEXAMPLE 1 - Deep learning-based virtual H&E staining from label-free autofluorescence lifetime images

[0082] Introduction

[0083] Label-free autofluorescence lifetime is a unique feature of the inherent fluorescence signals emitted by natural fluorophores in biological samples. Fluorescence lifetime imaging microscopy (FLIM) can capture these signals enabling comprehensive analyses of biological samples. Despite the fundamental importance and wide application of FLIM in biomedical and clinical sciences, existing methods for analysing FLIM images often struggle to provide rapid and precise interpretations without reliable references, such as histology images, which are usually unavailable alongside FLIM images. To address this issue, these examples propose a deep learning (DL)-based approach for generating virtual Hematoxylin and Eosin (H&E) staining.

[0084] In this example, the present inventors showed that by combining an advanced DL model with a contemporary image quality metric, it is possible to generate clinical-grade virtual H&E-stained images from label-free FLIM images acquired on unstained tissue samples. These experiments also show that the inclusion of lifetime information, an extra dimension beyond intensity, results in more accurate reconstructions of virtual staining when compared to using intensity-only images. This advancement allows for the instant and accurate interpretation of FLIM images at the cellular level without the complexities associated with co-registering FLIM and histology images. Using these results, the inventors show that they are able to identify distinct lifetime signatures of seven different cell types commonly found in the tumor microenvironment, opening up new opportunities towards biomarker-free tissue histology using FLIM.

[0085] Methods

[0086] Sample Preparation and Data Collection. Surgically resected, early stage non-small cell lung carcinomas were identified from the clinical archival and a TMA was constructed as previously described (Koppensteiner, 2023). FLIM images were collected using a confocal FLIM system (Leica STELLARIS 8 FALCON FLIM Microscope) with a 20x / 0.75NA objective. The excitation and emission wavelengths were set at 485 nm and [500 nm, 720 nm], which was determined by a A-to-A scan of the tissue. The image size was fixed at 512x512 pixels with the pixel size at -0.455 pm. Following FLIM acquisition the same sample underwent rehydration in a series of ethanol dilutions, staining with Haematoxylin and Eosin andthen dehydration and subsequent imaging on a bright-field slide scanner (ZEISS Axio Scan.ZI), with a 20x / 0.75NA objective. The digitalised H&E-stained images have a pixel size of -0.22 pm. In total, 84 lung cancer tissue samples (including 69 TMA cores scanned with the pixel size -0.455 pm, 5 biopsies scanned with the pixel size at -0.4 pm, and another 10 TMA cores from a separate cohort scanned with the pixel size at -0.25 pm) were used for this study, 10 cores were discarded due to unsatisfactory staining and / or imaging. 72 samples were used for model training and 15 more were used as the independent testing test (as explained further below). An additional 4 slides of colorectal cancer scanned with the pixel size -0.4 pm, and 4 sections of endometrial cancer scanned with the pixel size -0.4 pm were also used.

[0087] Data Post-Processing. FLIM images were reconstructed by an exponential fitting using the Leica LAX-X software and exported as intensity and lifetime images separately. Note that to ensure consistency when visualising FLIM images, raw intensity data was limited to [0, 2000] (intensity cannot be negative and the max count in this dataset was not over 2000 per the scanning configuration) and raw lifetime to [0.0 ns, 5.0 ns]. Values outside of each range were set to the boundary of the range. Afterwards, all tiles per core within the tumour microarray (TMA) were stitched using MIST Chalfoun, 2017. To achieve optimal registration, only intensity images were exploited. Intensity images were colour inverted for the white background. In particular, intensity images are converted to 8-bit grayscale images (0-255) and intensity values inverted via bitwise inversion (e.g. 0000 0000 (0 decimal) is converted to 1111 1111 (255 decimal); 0000 1111 (15 decimal) is converted to 1111 0000 (240 decimal); etc.). H&E images were converted to greyscale and contrast-enhanced by saturating 1 % of the values at both low and high bounds of the intensity range. The processed intensity and processed H&E images were used as the input for the co-registration. Since the tissue processing did not change the underlying structural and cellular information, Affine transformation in MATLAB was employed for the co-registration of FLIM and H&E images (see FIG. 8b). The co-registered RGB H&E images were subsequently used for DL model training. Unless indicated otherwise, the DL model takes as input an RGB image.

[0088] The co-registered images are then resampled to the patches of 256x256 pixels. Patches with over 75% background in the intensity images were disregarded. In particular, patches with over 75% background pixels, where a background pixel is defined as a pixel with 8-bit intensity value of >250 (an empirically set threshold, 0 being black pixels and 255 being white pixels) were disregarded (in both the training and test sets). For the input into the pix2pix network, three different formats were tested. The first was the greyscale intensity images, the second was the false-colour lifetime images with the corresponding normalised intensity images as the alpha channel, and the third was the false-colour lifetime images pixel-multiplied with the corresponding intensity images. Note that intensity images underwent global normalisation by division by the maximum intensity value per TMA core (i.e. dividing each pixel value by the maximum value of the whole TMA intensity image) before applying to the lifetime images.

[0089] GAN Architecture. In this study, the pix2pix network (Isola, 2017) was utilised. A conceptual diagram is depicted in FIG. 8c. Essentially, the pix2pix GAN is a conditional model, where the generator(G) has a U-Net-like architecture and the discriminator (D) is a multi-layer classifier using the input FLIM images as the condition. The objective of the discriminator is given by Equation (1): (G, D) = Ef J41og£> s|f)] + S / )fe[log(Equation (1): where f, s, and h are FLIM, synthetic H&E, and true H&E images, respectively, Ef,s[logD(s|f)] is the expectation (e.g. average) of the log output of the discriminator for channel-wise concatenated synthetic H&E and FLIM images (D(s|f)), and Ef,h[log(1 -D(h |f))] is the expectation (e.g. average) of log (1 - the output of the discriminator for channel-wise concatenated true H&E and FLIM images (D(h|f))) . The objective of the generator is provided by Equation (2):Equationwhere Li distance is applied, i.e. L(G) is the expectation of the L1 distance between true and synthetic H&E images (i.e. between h, a true H&E image and G(f,s), the corresponding generator output for the corresponding FLIM image f). Accordingly, the overall objective for the GAN is provided by Equation (3): = a (G, D) + 0£(G)Equation (3): where a and 0 are the regularisation parameter to adjust the weight of the discriminator and generator, respectively. Values of 0.1 and 1 were used for a and 0, respectively.

[0090] D / STS Loss. The regularisation terms in the original GAN networks are usually insufficient to synthesise acceptable virtual histological staining and various extra constraints can be applied to overcome the challenges, such as total variations (TV) (Rivenson, 2019a; Rivenson, 2019b; Zhang, 2020) and Structural Similarity Index Measure (SSIM) (Borhani, 2019; Liu, 2021). To achieve the optimal synthesis, the Deep Image Structure and Texture Similarity (DISTS) metric was employed in this study, which yields superiority to existing single metrics by combining structural and texture information together and has proven to be tolerant to mild local and global structural distortion (Ding et al, 2022).

[0091] Given r and s are true and synthetic images, respectively, and F is a function to retrieve feature maps of input images from a trained model, the extracted featurescan be defined as ^(s) = =E [0,5]: j E fl, n / ]; x E |r, s]J , where i is the i‘hconvolution layer of the trained model, andn' is feature maps in the ithconvolution layer. DISTS is defined as Equation (4):Equation (4):where q and 0 are learnable weights, satisfyingare quality measurement for the texture and structure, respectively, governed by Equation (5) and Equation (6), respectively:Equation (5): andEquation (6): whereare the global means and variances of the extracted feature1j*’ maps for r and s, respectively,3is the global covariance between the feature maps, and ci and C2 are parameters set to very small values (here T6), used only for numeric stability.

[0092] The final objective of the network combines the overall objective for the GAN loss (Equation (3)) and the DISTS loss (Equation (4)) as:Equation (where A is a regularisation parameter for DISTS loss.

[0093] Blinded evaluation of virtual and true H&E staining. A set of 12 lung TMA cores were used for the blinded evaluation, where all images were anonymised to allow three experienced pathologists to perform unbiased inspection. To holistically appraise the quality of the images, pathologists meticulously examined each image's four fundamental aspects: nuclei clarity via Haematoxylin staining, cytoplasm precision via Eosin staining, overall staining integrity, and suitability for diagnostic decision -making. For each parameter, the expert pathologists were asked to assign scores ranging from Unacceptable (1) to Acceptable (2), Excellent (3), and Exemplary (4).

[0094] Implementation Details. The model was implemented using PyTorch and adapted for three input formats, including greyscale intensity images, false-colour lifetime images with normalised intensity as the alpha channel, and false-colour lifetime images pixelwise weighted by normalised intensity. Adaptive Moment Estimation optimiser (Kingma, 2017) was deployed with 01 as 0.5 and 02 as 0.999.Learning rate was initialised at 0.0001 and decayed by 10 at very 60 epochs. Total epochs were set to 300. a, p, and A in the final objective equation (Equation (7)) were set to 0.1 , 1 , and 5, respectively.

[0095] 72 lung cancer samples (including 70 Lung TMA cores and 2 biopsies) were utilised for the training and 15 independent lung cancer sections (including 12 TMA cores and 3 biopsies) were split out as the independent testing set. For colorectal and endometrial cancers, both were trained via transfer learning on 2 sections and evaluated on 2 separate sections. The patches in the training stage were fixed at 256x256. Simple data augmentations, including horizontal flipping and 15-degree rotation, were applied during the training. Training was performed on NVIDIA V100 GPUs provided by the EPSRC Tier- 2 National HPC Services Cirrus hosted by EPCC (www.epcc.ed.ac.uk / hpc-services / cirrus) at The University of Edinburgh.

[0096] The TMA was approved by NHS Lothian REC and facilitated by NHS Lothian SAHSC Bioresource (REC No: 15 / ES / 0094) and approved by delegated authority granted to R&D by the NHS Lothian Caldicott Guardian (Application number CRD19031).

[0097] The pix2pix used in this study is publicly available at github.com / mrzhu-cool / pix2pix-pytorch. The DISTS loss is available at github.com / dingkeyan93 / DISTS. FLIM images were stitched using Fiji MIST stitching plugin (github.com / usnistgov / MIST). MATLAB was used for affine transformation to coregister FLIM and true histology images.

[0098] Results

[0099] The inventors used deep learning to perform virtual histological staining of label-free FLIM images. As explained above, 69 surgically resected, early-stage non-small cell lung carcinoma tissue microarrays (TMA) were imaged using a confocal FLIM system. Next, the same samples underwent staining with Haematoxylin and Eosin, followed by imaging on a bright-field scanner. After postprocessing as described above, the FLIM images and digitalised H&E images were co-registered using an affine transformation. 63 of the lung TMA cores were used to train a pix2pix GAN model with a DISTS loss function. The remaining 6 cores were used for testing of the model and blind evaluation of virtual and true staining by three experienced pathologists.

[0100] Virtual H&E staining from FLIM. FIG. 4 summarises examples of the virtual H&E staining from label-free FLIM images using an independent dataset collected on two TMA (tumour microarray) cores of lung cancer which were not utilised in training, where one contains relatively few distinct cell populations (FIG. 4a) while the other is a more complex mixture of cell types (FIG. 4c). Both are visualised as intensity-weighted false-colour lifetime images.

[0101] The results demonstrate that the method described herein is able to reconstruct various cellular components in the lung tissue samples from FLIM images to match the corresponding H&E-stained brightfield images. Specifically, the method proves its effectiveness in faithfully reproducing the morphological and textural attributes of a wide range of cell types present within the lung TME (Tumour Micro-Environment), including tumour cells, stromal components (such as fibroblasts) and inflammatory cells, as well as red blood cells. Each of these was confirmed with the True HE (FIG. 4i -I).

[0102] While the DL model was trained using input patches sized at 256 256, it is important to note that these models can accommodate input of various sizes during the inference stage. In addition, H&E- stained images were downsampled to match the pixel size of FLIM images, where the pixel size was increase from 0.22 pm to 0.455 pm using bicubic interpolation.

[0103] Blind evaluation of virtual staining quality. In pathology laboratories, the visual inspection of histological images by experienced pathologists is an integral step in current clinical practice for reaching diagnostic decisions. Similar to Rivenson, 2019a and Zhang, 2022, a blind evaluation was conducted by three seasoned pathologists to assess the subjective quality of virtual H&E staining. Twelve pairs of true and virtual H&E-stained images were prepared using lung cancer tissues, with all images anonymised. To comprehensively evaluate the quality of virtually stained images, the pathologists examined each image in four key aspects, including nuclei detail via Haematoxylin stain, cytoplasm detail via Eosin stain, overall staining quality, and quality for diagnostic decision-making. For each dimension, pathologists assigned scores based on predetermined criteria, ranging from Unacceptable (1) to Acceptable (2), Excellent (3), and Perfect (4). This meticulous evaluation process offers valuable insights into the efficacy and dependability of virtual H&E staining compared to conventional staining methodologies.

[0104] Table 1 outlines the scores on six lung tissue samples by three pathologists, and FIG. 5 depicts the mean scores provided by each pathologist on four image quality metrics. In general, the average scores shown in FIG. 5b demonstrate the outstanding quality of virtual histological images on all metrics, where all scores are over 3 (excellent), except the overall staining quality of virtually stained images (2.95). Although each pathologist rated the images differently (FIG. 5a), their outcomes are consistent on the differences between true and virtual images. For example, pathologist 1 ranked both stains are above excellent, while pathologist 3 considered they are between acceptable (2) and excellent (3). Table 1 presents the detailed ranking on each lung tissue sample. An interesting result is on sample 6 (lung sample 2 in FIG. 4), where all pathologists voted the virtual image over the true one on all four metrics. In summary, the subjective evaluation demonstrated a high degree of agreement between the tech niques, where the pathologists were able to examine histological features on virtual staining and reach the same level of diagnostic decision-making.Table 1: Blind evaluation of twelve lung cancer tissues by three pathologists. The evaluation includes nuclei detail, cytoplasm detail, overall stain quality, and diagnosis decisions, and each with a score of 1 (unacceptable), 2 (acceptable), 3 (excellent), and 4 (perfect).

[0105] Lifetime signatures of various cell types in lung tissue. By leveraging virtual H&E staining, the inventors directly mapped cellular components annotated by pathologists to FLIM images, enabling identification of lifetime signatures of diverse cell types, without the need for conventional staining and co-registration processing procedures. FIG. 6 depicts an example of label-free autofluorescence lifetime signatures of 7 different cell types, where the lifetime is determined by the peak values in the histogram of pixel lifetime values within the marked cells (cells manually annotated using the virtual H&E images).

[0106] FIG. 6a is a FLIM tile patched from FIG. 4c, FIG. 6b is the corresponding virtual H&E image, and FIG. 6c is the true H&E image as the reference. An experienced pathologist annotated 7 different cells commonly found in the TME on virtual histology image and confirmed on the true one, including tumour, fibroblasts, lymphocytes, plasma cells, neutrophils, macrophages, and red blood cells (RBCs). Afterwards, lifetime histograms were derived for all the cells annotated, and averaged lifetimes were extracted. FIG. 6d clearly demonstrates the lifetime differences of the cells. Interestingly, the averaged lifetimes for various cell types follow a progressive increase: RBCs have an average lifetime of 0.251 ns, fibroblasts 0.376 ns, tumours 0.424 ns, lymphocytes 0.471 ns, neutrophils 0.659 ns, plasma cells 0.69 ns, and macrophages have the longest average lifetime at 2.42 ns. RBCs show the shortest lifetime and appear too dark in the FLIM image (arrow 1 in FIG. 6a)) to be discerned. On the other hand, macrophages have the longest lifetime and are easily identifiable in the FLIM image (arrow 2 in FIG. 6a)). Plasma cells and neutrophils possess similar lifetime values, resulting in their visual similarity (highlighted by dots indicated by arrows 3 and 4 in FIG. 6a)). In comparison, lymphocytes (arrow 5) are less visually distinguishable than these two cell types. Tumor cells (arrow 6) and fibroblasts (arrow 7) exhibit shorter lifetime values, making them appear dimmer in FLIM images compared to other cells. While most tumor cells can be identified in FLIM with the aid of the H&E image as reference, pinpointing fibroblasts remains a challenging task on the FLIM image. Lifetime variability across different cell types or even within the same cell type is expected, and arises from the diverse composition of molecules within individual cells, each with its own characteristic lifetime. In the context of Fig. 6, a specific lifetime suggests the presence of a specific molecule shared among all 7 cell types, with the counts on the y-axis reflecting the relative abundance of this molecule within the cells. In practice, distinguishing between different cell types often involves leveraging both intensity and lifetime data. While intensity provides valuable morphological information, lifetime measurements offer insights into molecular composition. By analysing cellular-level histograms of lifetime, users can gain a deeper understanding of the functional characteristics of various cell types, complementing the morphological details provided by intensity data.

[0107] Comparison of different image formats. Various DL-based approaches have been proposed to leverage label-free autofluorescence images for the generation of virtual histological images, such as Rivenson, 2019a. Since FLIM images also contain autofluorescence intensity, the inventors compared synthetic histology images generated from three different formats. These formats included autofluorescence intensity images in greyscale, false-colour lifetime images with normalised intensity as the alpha channel, and intensity-weighted false-colour lifetime images. For intensity images, the inventors followed the post-processing described in Rivenson, 2019a. In the case of false-colour lifetime images, the process involved initially converting greyscale lifetime images into 3-channel RGB images through conventional colour mapping, with a fixed range of [1.0 ns, 5 ns]. Subsequently, the normalised intensity is appended as the alpha channel to create a-FLIM, resulting in four-channel images. For the generation of IW-FLIM images, the normalised intensity is employed as a soft weight, undergoing pixelwise multiplication with each channel in the false-colour lifetime images, resulting in a three-channel images where each channel is intensity weighted. Note that despite different processing methods, a-FLIM and IW-FLIM images are visually identical. These three formats were then input into the DL model with the same hyper-parameters for training on the same training dataset.

[0108] FIG. 7 depicts the synthesis of virtual H&E staining from the aforementioned three image formats. Visual inspection by three pathologists was carried out initially to assess the overall quality of the synthesis. This evaluation revealed that all three formats can generate satisfactory virtual H&E- stained images, without compromising the detection and diagnosis of lung cancer. However, some misreconstructed details are observed in intensity-based synthesis, which can impact the biological analysis of the tissue.

[0109] The inventors discovered that a noticeable limitation of intensity-based reconstruction is its inability to locate some macrophages. In FIG. 7, the arrows marked 1 point to clusters of macrophages that were misinterpreted as tumour cells in virtual H&E staining from intensity (FIG. 7e), whereas they were clearly visible in the virtual images generated from a-FLIM (FIG. 7f) and IW-FLIM (FIG. 7g). However, macrophages were accurately identified in intensity images when they exhibit strong signals, as indicated by the bright dots highlighted by the arrow marked 2 in the intensity images (FIG. 7b).

[0110] Additionally, a few immune cells were misclassified as other cell types in the intensity-based virtual image, as shown by the arrow marked 4 in FIG. 7e. In contrast, both a-FLIM (FIG. 7f) and IW-FLIM (FIG. 7g) have excellent capabilities for virtual H&E staining.

[0111] To quantify the comparison, the inventors applied four predominant metrics (Zhou, 2019) for measuring the similarity between true and virtual H&E-stained images, including the normalised root mean-squared error (NRMSE), the normalised mutual information (NMI), the peak signal to noise ratio (PSNR), and the mean structural similarity index metric (MSSIM). The results are listed in Table 2 and the best one is highlighted in bold for each metric. Overall, the quantitative results are consistent with the visual results shown in FIG. 7, where all values are comparable, albeit with some differences. Intensity images are less effective than false-colour FLIM images for synthesising virtual images, with IW-FLIMbeing the most suitable format for achieving optimal virtual H&E staining. a-FLIM also produces competitive results similar to IW-FLIM, with marginal differences compared to IW-FLIM.Table 2: Quantitative comparison of virtual H&E staining on two TMA sections of lung cancerous tissues from intensity, a-FLIM, and IW-FLIM using the normalised root mean-squared error (NRMSE), the normalised mutual information (NMI), the peak signal to noise ratio (PSNR), and the mean structural similarity index metric (MSSIM). Note that smaller NRMSE and NMI indicates more similar images, whereas larger PSNR and MSSIM represents better similarity between images. The best metric is highlighted in bold in each case.

[0112] Virtual H&E staining on various tissue samples. In addition to lung tissue samples, the inventors expanded their experiments to include primary colorectal and endometrial cancers. Four slides of colorectal cancer scanned with the pixel size ~0.4 pm, and four sections of endometrial cancer scanned with the pixel size -0.4 pm were used for this study. The samples were prepared, data collected and post-processed as described above for lung samples. The results are presented in Figures 13 and 14 for colorectal and endometrial adenocarcinomas, respectively. Similar to the results shown above, the virtual images (Figure 13b and 13e for colorectal cancer and 14b and 14e for endometrial cancer) exhibit significant consistency with the real ones (Figure 13c and 13f for colorectal cancer and 14c and 14f for endometrial cancer). As was seen above with lung cancer samples, most cellular details in the colorectal and endometrial adenocarcinoma images are accurately reconstructed. The reconstructed images are suitable for both visual and quantitative analysis. A notable phenomenon is observed in the FLIM image of endometrial cancer, where the fluorescence lifetime appears dramatically shorter compared to other samples, resulting in a visually dimmer appearance. However, it's important to note that this shorter lifetime does not affect the virtual result obtained through the inventors’ method.

[0113] The inventors also expanded their work to include Formalin-Fixed Paraffin-Embedded (FFPE) lung biopsies. 5 biopsies scanned with the pixel size at -0.4 pm were used for this study, images were processed as explained above. Fig. 15 presents the results on an FFPE lung biopsy. The virtually stained image (Fig. 15d) of the lung biopsy exhibits consistency with the real one (Fig. 15f), revealing cellulardetails within the area. Although the performance of the method on the biopsy is not as remarkable as demonstrated on TMA slides above, due to the degraded nature of the sample, the virtual image does not compromise clinical decision-making. This importantly demonstrates the power of the methods described herein, as FFPE biopsies are a widely available type of samples, and the ability to obtain predictions that are suitable for clinical decision making using the methods described herein is therefore very valuable.

[0114] Use case exploration. Compared to other imaging modalities used in the literature for label-free virtual H&E staining, FLIM microscope is more complicated and usually requires longer time to image tissue samples. For example, the images presented in Figs. 4, 14, 15 (see above), and Fig. 16a-d (showing virtual H&E staining from FLIM conducted using the same cohort as on Fig. 4) were acquired within 40 minutes per TMA core with a pixel size of 455 nm. However, following the optimisation of scanning parameters (in particular by testing various combinations of pixel size and scanning mode (line and / or frame repetition)), the scanning time reduced to less than 15 minutes while maintaining the same quality. This allows the scanning of a TMA core within 30 minutes at a much higher resolution (a pixel size of 200 nm), as shown in Fig. 16e-j (virtual H&E staining from FLIM conducted on a separate cohort of lung TME samples using different scanning parameters, as indicated). H&E staining is a routine test in biology and pathology laboratories, the process of which takes about 30 minutes. While it may therefore seem counterproductive based purely on time to use FLIM for virtual H&E staining, it is worthwhile when considering the quality of FLIM-based virtual H&E staining. Additionally, FLIM has demonstrated the capability for providing metabolic information, without requiring exogenous biomarkers (Marcu, 2012; Alfonso-Garcia, 2016; Walsh, 2021 ; Linders, 2023; Yan, 2023). Together, this shows that virtual IHC staining has great potential to address challenges in managing small biopsies and limited tissue samples in advanced-stage lung cancer patients (and other cancer types) undergoing only small biopsies rather than resection procedures. By leveraging FLIM, tissue utilization can be optimised by reducing the necessity for additional ancillary tests while achieving the same level of diagnostic and phenotypic characterisation.

[0115] Optimisation. In order to further optimise their method, the inventors tested affine registration with and without elastic registration, various deep learning models with and without an advanced loss function, and random noise sampling. First, the inventors performed affine transformation with and without further transformation using elastic registration. Two FLIM images, after affine transformation (Fig. 17a and 17g), undergo further transformation using elastic registration (Fig. 17c and 17i). The corresponding displacement fields are presented in Fig. 17b and 17h, respectively. The inventors determined that the synthetic results with additional elastic registration do not contribute to better virtual staining, at least with the quality of the sample preparation process used in these examples.

[0116] Second, the inventors tested various deep learning models for virtual H&E staining. The results are shown in Fig. 18. While using pix2pix with DISTS loss performs well (Fig. 18c), pix2pix without additional loss function (Fig. 18b), ResVit (Fig. 18d; Dalmaz, 2022) and DDGAN (Fig. 18e; Xiao, 2021) without any further optimization do not generate as high quality virtual H&E staining when compared to the true H&E staining (Fig. 18a).

[0117] Finally, the inventors tested virtual H&E staining using images to which random noise has been added, in order to test the robustness of the method. Results are shown in Fig. 19. The noise was sampled using Gaussian distributions with a mean of 0.01 and standard deviations ranging from 0.1 to 0.3. With a fixed mean value, it is evident that increasing the standard deviation reduces the quality of the reconstruction. However, the results show that the method still works with FLIM images of lower quality. This means that the method would be usable with further reduction of the scanning time, and increase the compatibility of the methods with other FLIM imaging modalities.

[0118] Discussion

[0119] In this example, the inventors explored the use of DL techniques to generate virtual H&E staining from label-free FLIM images. Specifically, they employed the supervised DL model, pix2pix GAN (Isola, 2017), along with the advanced image quality metric, Deep Image Structure and Texture Similarity (DISTS) (Ding, 2022), to synthesize H&E-stained histology images. By doing so, the inventors demonstrated the potential of this technique in enabling rapid and precise interpretation at the cellular level of FLIM images, which allows for the identification of label-free autofluorescence lifetime signatures for various types of tumour cells. Additionally, the inventors compared the effectiveness of different input data formats, such as intensity and weighted lifetime images, demonstrating the benefits of using lifetime information in addition to fluorescence intensity information for the synthesis of virtual H&E staining. The results demonstrate the ability of the proposed method to generate clinical-grade virtual H&E staining from label-free FLIM images acquired on unstained tissue samples.EXAMPLE 2 - Deep learning-based virtual immunohistochemistry staining from label-free autofluorescence lifetime images

[0120] Introduction

[0121] In lung cancer subtyping, Immunohistochemistry (IHC) staining is the common practice for precise detection and diagnosis. In particular, the markers thyroid transcription factor-1 (TTF1) and P40 are routinely used in the clinic for diagnosing lung adenocarcinoma and squamous cell carcinoma, respectively, which are two most common subtypes of non-small cell lung cancer (NSCLC). However, obtaining and analysing IHC stained images is time-consuming and costly, requiring sample processing, and special instrument and expertise.

[0122] In this example, the inventors developed a novel methodology using deep learning techniques to generate virtual TTF1 and P40 from label-free FLIM images in sub-seconds, without the need for conventional tissue processing procedures. This allows FLIM to perform rapid and precise subtyping of NSCLC.

[0123] Methods

[0124] FLIM images. FLIM images of unstained tumor microarray cores were acquired using a Leica STELLARIS 8 FALCON FLIM Microscope. The acquired images had a Pixel distance: ~0.2 micron. The excitation and emission wavelengths were set at 446nm and [460 nm, 640 nm], which was determined by a A-to-A scan of the tissue as was the case in Example 1 . As was the case in Example 1 , raw intensitydata was limited to [0, 2000] and raw lifetime to [0.0 ns, 5.0 ns]. Intensity images were colour inverted for the white background.

[0125] IHC images. After FLIM imaging, the TMA were TTF1 and P40 stained following a standard protocol by the Royal Infirmary of Edinburgh Pathology Laboratory, including deparaffinisation, conditioning, blocking, incubation with primary anti-P40 / anti-TTF1 antibody, and incubation with a secondary fluorescent antibody. The TMA slides were digitalised using a commercial slider scanner (Zeiss AxioScan) with a pixel distance of 0.22 pm.

[0126] Data. The data used consisted of images from 50 TMA cores from 25 patients, with for each core, paired intensity-weighted lifetime images (IW-FLIM) and IHC-stained images (one image for each stain). The pixel size for both the FLIM images and the IHC images was 0.2pm. The FLIM tiles were stitched to form per-core images and co-registered using affine transformation as described in Example 1. The co-registered images are then resampled to the patches of 256 256 pixels. Patches with over 75% background were disregarded, as explained in Example 1.

[0127] Deep learning. The same model described in Example 1 was used. The model was trained in the same way as described in Example 1 , using the weights from the H&E trained model as a warm start for the training (i.e. transfer learning). A separate model was trained for each stain. For each stain, 45 cores were used for training DL models, and 4 cores were left as an independent testing set.

[0128] Results

[0129] As described in Example 1 , the inventors determined that IW-FLIM is the most suitable format for achieving optimal virtual H&E staining, based on the metrics provided in Table 2. Therefore, the inventors extended the model to IHC staining using transfer learning and IW-FIM images of tumour microarray cores.

[0130] The results of this are shown on FIG.9 to FIG. 12. FIG.9 shows IW-FLIM images for four TMA cores, together with the corresponding virtual and real P40 staining. FIG. 10 shows enlarged areas of corresponding panels in FIG.9 (i.e. zoomed in view). These figures show a very good level of agreement between the real and virtual P40 stained images, down to the cellular level. This was true in all cases, even though the different cores showed significant high level morphological differences.

[0131] FIG. 11 shows IW-FLIM images for four TMA cores, together with corresponding virtual and real TTF1 stained images. FIG. 12 shows enlarged areas of corresponding panels in FIG. 11 (i.e. zoomed in views). These figures again show a very good level of agreement between the real and virtual P40 stained images, down to the cellular level, in all slides. This was true in all cases, even though the different cores showed significant high level morphological differences.

[0132] The data shows virtual and real staining in both cases show good agreement, as confirmed by experienced pathologists (data not shown). In other words, the data in this example demonstrates that the methods described herein are able to reliably label P40+ / TTF1+components from IW-FLIM images. Additional model architectures were also tried including ResVit (Dalmaz et al. 2022) and denoising diffusion GANs (Xiao et al. 2022), and these did not improve the results in terms of the number ofP40+ / TTF1+ cells. The use of multi-exponential fitting to extract multi-component lifetime images was also tested and led to similar results.EXAMPLE 3 - Virtual IHC staining for NSCLC subtyping

[0133] Introduction

[0134] Lung cancer remains the most commonly occurring and leading cause of cancer-associated mortality globally, accounting for 12.4% of all cancer diagnoses and 18.7% of cancer-related deaths. Pathological diagnosis, subtyping, and molecular phenotyping are central to the effective management of the disease as well as informing prognosis. Non-small cell lung cancer (NSCLC), accounts for approximately 80% of newly diagnosed lung cancers, with adenocarcinoma (AC) and squamous cell carcinomas (SqCC) comprising approximately 50% and 30% of NSCLC cases respectively. However, distinguishing between these and other subtypes based on morphological features alone can be challenging due to the loss of distinct histological differences in more poorly differentiated carcinomas. Immunohistochemical (IHC) staining is therefore frequently employed to aid in phenotypic classification, but this process requires additional time, labour, and cost, which can impact timely diagnosis and delay treatment decisions. Furthermore, additional tissue sections required for IHC risk exhausting the limited available cellular material which is also required for downstream DNA and RNA-based molecular profiling risking the need for repeat biopsy. Given advancements in computational power, data-driven algorithms, and efficient imaging modalities that explore cellular function and morphology, a fast and accurate computer-aided classification solution is highly desirable to improve diagnostic efficiency and enable rapid diagnosis of label-free, stain-free tissues.

[0135] Label-free autofluorescence imaging leverages the intrinsic fluorescence of biological tissues for cancer diagnosis by capturing metabolic and structural changes at the cellular level. One of the widely used endogenous signals is autofluorescence intensity, which has been utilised for the detection and diagnosis of various cancers. Another important feature of autofluorescence signals is lifetime, characterised by a decay of a fluorophore from the excited state to the ground state (Becker et al. 2012). Fluorescence lifetime imaging microscopy (FLIM) can capture this unique feature to investigate subtle changes in the bio-environment at a molecular level (Becker et al. 2012). Fluorescence lifetime has broad applications in cancer diagnosis, such as lung (Fernandes et al. 2024; Adams et al. 2024), prostate (Zang et al. 2022), breast (Sorrells et al. 2021), and skin cancer (Luo et al. 2017). Due to its capability at the molecular level, fluorescence lifetime can also be facilitated to differentiate cell types and phenotypes, such as T-cell activation (Walsh et al. 2021), cancer cell phenotypes (Hu et al. 2023), and macrophage subtypes (Alfonso-Garcia et al. 2016). Despite the use of label-free signals for lung cancer detection, the effectiveness of these features for lung cancer subtyping — or cancer subtyping in general — remains uncertain, particularly in addressing interpatient heterogeneity. The inventor’s recent research has demonstrated the feasibility of combining DL techniques with FLIM images for lung cancer diagnosis (Wang et al. 202; Wang et al. 2023; Wang et al. 2022) Furthermore, the inventors have managed to translate FLIM images into virtual H&E images across multiple tumour types (Wang et al. 2024). Both allow for timely and accurate lung cancer detection without requiring the conventional tissue processingand staining procedures. All these indicate the potential of label-free signals for advanced cancer characterisation, with the integration of DL for improved fidelity and reduced tissue consumption.

[0136] Generally, virtual staining techniques can be categorised into two groups: label-free virtual staining and stain-to-stain (S2S) transformation. In the label-free domain, autofluorescence images (Rivenson et al. 2019; Zhang et al. 202; Li et al. 2021 ; DoanNgan et al. 2022), bright-field images (Bai et al. 2023; Zhang et al. 2022), FLIM images (Wang et al. 2024; Bohrani et al. 2019), and photoacoustic images (Cao et al. 2023; Kang et al. 2022) have been used as inputs to synthesise H&E and IHC images (HER2- DoanNgan et al. 2022, SOX10 “ Levy et al. 2020, FAP-CK- Hong et al. 2021 , etc) for different organ types. To date, no virtual staining approach has targeted the proteins used to characterise AC and SqCC in routine clinical practice, namely, Thyroid Transcription Factor 1 (TTF-1 ; Moldvay et al. 2004) and p40 (Affandi et al. 2018) respectively In light of this, the present inventors proposed a virtual staining strategy to distinguish between major NSCLC subtypes. They use a generative adversarial network (GAN), employed for virtual H&E staining in Example 1 , to generate synthetic TTF-1 (Moldvay et al. 2004) and p40 images for AC and SqCC, respectively. To assess the quality of the generated virtual IHC images, we conducted a blind evaluation by three certified pathologists, complemented by quantitative analysis.

[0137] Methods

[0138] TMA construction. A first TMA (labelled SR1208) was constructed from consecutive patients undergoing curative resection surgery for NSCLC in a regional thoracic centre over a two-year period. In this cohort, no patient received adjuvant immunotherapy in line with the standard of care at the time. An experienced pathologist annotated each resection block, and cores were taken and embedded into the TMA. For each patient case one area of non-cancerous lung and three punches of tumour areas were taken and embedded into separate blocks. A second TMA (labelled SR1949) included selected cases to ensure a balance of adenocarcinoma (AC, 10 cases), squamous cell carcinoma (SqCC, 10 cases), other subtype (OS) to include adenosquamous, large cell carcinoma, neuroendocrine and carcinoid (5 cases) and non-cancerous lung (5 cases). For cancer regions TMA cores were taken in duplicate and single punches from non-cancerous lung were embedded into separate blocks. A third TMA (labelled SR2046) included an archival NSCLC cohort and included 100 lung cancer cases of varying subtype and mutational status, with triplicate TMA punches being taken separate blocks. For each TMA block, slides were prepared by cutting 4-micron tissue sections on glass slides. For fluorescence imaging samples were deparaffinised and mounted with a coverslip. Following imaging, coverslips were removed with xylene incubation and the same slides were transferred to NHS laboratories for subsequent staining.

[0139] TTF-1 and p40 Staining. TMA sections were stained with antibodies to TTF-1 (Agilent; Clone: 8G7G3 / 1 ; Dilution 1 :200) or p40 (Biocare Medical; Clone: BC28; Dilution 1 :100) using IHC protocol F on the Leica Bond III Platform. Digital whole slide images were captured using the Leica GT450 scanner at 40 x magnification. Bright-field TMA projects are imported into QuPath, where each core is individually identified and exported as an uncompressed histology image for co-registration with the corresponding FLIM image.

[0140] Data Acquisition and Processing. The virtual staining work described here is based on a large- scale dataset comprising samples from across multiple cohorts. Intensity and FLIM image acquisition share the same imaging export setup. Images were acquired using Leica STELLARIS 8 FALCON FLIM microscope with a 20*70.75 NA objective. The pixel size was configured to 0.3001 m. The excitation and emission wavelengths were set at 445 nm and [460, 640] nm, determined by a wavelength-by- wavelength scan of the tissue. After scanning, fluorescence lifetime images were reconstructed from the raw data under the hood by the multi-exponential fitting algorithm in Leica LAX-X software. The number of lifetime components in the fluorescence lifetime decay is determined by x2, with four-lifetime components being adopted for fitting as this configuration achieves the smallest x2for all cores. While exporting images, each core on a TMA was segmented by 512 x 512 pixels for each tile. Exported intensity images were applied with a threshold of 10 photon counts to filter out some background. Photon count range [0, 2000] and lifetime range [0, 5] ns were applied when exporting the images to achieve consistent visualisation. After exporting image tiles of intensity and FLIM images from the Leica LAX-X system, an ImageJ-embedded stitching method (Preibisch et al. 2009) was used to assemble the tiles into a complete image. Since some cancer tissues exhibit low photon emission, leading to dim intensity images, we enhanced the brightness of the normalised intensity images using histogram stretching within a constant range to improve feature extraction, as described in Wang et al. 2024. An image co-registration process was applied to intensity and histology images to align morphological structures. Due to differences in imaging modalities, pixel sizes between FLIM and brightfield images were standardised using bicubic interpolation in MATLAB. Additionally, an affine geometric transformation in MATLAB was applied to correct geometric distortions. Co-registration was performed as described in Wang et al. 2024. Co-registered FLIM images and real stained TTF-1 and p40 images were cropped into 256 x 256 patches and fed into the pix2pix model.

[0141] Deep Neural Network. Virtual IHC staining was performed as described in Examples 1 -2. Specifically, the inventors integrated the pix2pix GAN with additional loss functions, including the Structural Similarity Index Measurement (SSIM) (Wang et al. 2004) and Style Loss (Gatys et al. 2015). The training was carried out on the EPSRC Tier-2 National HPC Service, Cirrus, hosted by EPCC. For TTF-1 , 49 cores were used for the study, with 40 on training and 9 on testing. For p40, 50 cores were used for the study, with 40 on training and 10 on testing. The models were trained for 50 epochs using transfer learning to shorten convergence time, with a batch size of 16. The initial weight decay was set to 10-4 and reduced by a factor of 10 every 15 epochs for both the generator and discriminator. Further details about the model and training process can be found in Examples 1 and 2 above and in Wang et al. 2024.

[0142] Blind assessment of images. Virtual IHC images were blind evaluated by three thoracic pathologists with over 30 years of combined experience, where intensity and lifetime-derived images were anonymised and mixed. The evaluation was conducted in 5 aspects, including: (1) Total slide staining assessment: 1.1. Staining of tumour cells: Yes / No. 1.2. Staining of normal pulmonary epithelial / basal cells: Yes / No. (2) Tumour staining assessment: 2.1. Intensity: Strong / Weak / Negative. 2.2. Proportion: Diffuse / Focal / Negative. (3) Staining Quality assessment: 3.1. Background / non-specificstaining: Yes / No. 3.2. Expression in incorrect cell populations (e.g. lymphocytes): Yes / No. (4) Diagnostic confidence to use virtual IHC image compared to true IHC: Very / Moderate / Not confident. (5) Diagnostic decision based on autofluorescence image alone: 5.1. The virtual IHC image: Positive / Negative / Ambiguous. 5.2. Is the decision the same as on the real IHC image: Yes / No.For all questions in the questionnaire, pathologists selected one option based on the virtual IHC images and were blinded to each other's responses. The outcomes presented in the Section Results were statistically analysed according to the pathologists’ selections.

[0143] Results

[0144] The inventors evaluated virtual TTF-1 staining on an independent cohort containing 9 TMA cores, including 4 lung AC, 3 lung SqCC, and 2 other NSCLC subtypes. They also conducted a similar evaluation of virtual p40 staining using a separate cohort of 10 TMA cores for testing, which included 4 AC, 4 SqCC, and 2 OS. To comprehensively evaluate the quality of virtual staining and its clinical suitability, eight cases from each virtual staining method were scored by three experienced thoracic pathologists. The evaluation focused on overall staining quality and its utility in diagnosing AC and SqCC. For staining details, tumour cells were assessed for the presence and accuracy of staining, using corresponding H&E images from adjacent slices and real IHC images as references. Other cell types were also examined to identify incorrect staining, as well as non-specific staining in cellular components and background. For these assessments, pathologists recorded their evaluations as “Yes” or “No”. The inventors calculated the percentages for each category based on the answers to demonstrate the consistency of the assessment across the pathologists. Additionally, pathologists were asked to rate their confidence (Very, Moderate, and Not Confident) in using the virtual images for NSCLC subtyping. In this regard, the inventors calculated the overall percentage of pathologists’ confidence in using virtual staining for diagnosis. Their diagnostic decisions were then compared to those made using real IHC images to evaluate consistency and reliability.

[0145] Fig. 20 presents virtual TTF-1 staining on two TMA cores, where both are TTF-1 + cases, indicating lung AC. Fig. 20a-g are for core 1 and Fig.20h-n are for core 2, including both intensity and FLIM-based derivation. Within the figure, the presented FLIM images are the false-colour lifetime images with normalised intensity as the alpha channel. In general, both modalities produce satisfactory outcomes (Fig. 20c, e, j, I) that closely resemble real TTF-1 images (Fig. 20a, h). All pathologists are confident in making accurate lung AC diagnoses using the virtual images (Fig. 20f, g, m, n), underscoring their reliability for robust clinical decision-making. However, discrepancies exist, as highlighted by arrows in Fig. 20, which indicate instances where TTF-1 + cells are mis-reconstructed in intensity-based virtual images but accurately reconstructed in FLIM-based images. These underscore subtle differences in reconstruction accuracy between the two modalities, demonstrating that FLIM-based reconstruction generally outperforms intensity-based reconstruction. This becomes more obvious for TTF-1- cases. A particular case (Fig. 21) further demonstrates that intensity-based staining lacks clarity, making it challenging for pathologists to make confident diagnostic decisions. In contrast, FLIM-based reconstruction does not introduce the ambiguity.

[0146] Fig. 22 illustrates the results of virtual p40 staining using both intensity and FLIM. Like virtual TTF-1 staining, both intensity and FLIM-based approaches can generate high-quality virtual p40 images (Fig. 22c, e. j, I) for consistent clinical decision-making by pathologists (Fig. 22f, g, m, n). Arrows point to some differences in reconstruction accuracy between intensity- and FLIM-derived virtual p40 images. Intensity-based reconstruction tends to overestimate the presence of p40+ cells, whereas FLIM-derived reconstruction more accurately captures the true distribution of p40+ cells. These differences highlight the superior precision of FLIM-based imaging in faithfully identifying p40+ cells. Nevertheless, both approaches enable pathologists to conduct consistent assessments for lung SqCC diagnosis, ensuring reliable clinical evaluations despite differences in reconstruction accuracy.

[0147] An interesting outcome of virtual IHC staining is a TMA core containing both TTF-1 and p40 expressing cells. Fig. 23 illustrates such a case, with two sequential cores from one patient having both p40+ and TTF-1 + cells. By plotting the histograms of those cells in intensity and lifetime, those cells can be differentiated from each other by average intensity (Fig. 23b) and lifetime (Fig. 23c), highlighting the potential to utilise autofluorescence signals for cell differentiation.

[0148] Fig. 24 presents the evaluation outcomes from pathologists, categorised by markers (TTF-1 and p40), imaging modalities (intensity and FLIM), and marker expressions (positive and negative). Fig. 24a and b show intensity-and FLIM-based virtual TTF-1 staining on eight cases (3 lung AC, 3 lung SqCC, and 2 OS), respectively. Overall, both intensity- and FLIM-based approaches produce satisfactory virtual TTF-1 images. For positive expressions, the reconstructed virtual images exhibit flawless staining details, with consistent scores across all metrics, enabling pathologists to make confident diagnostic decisions. For TTF-1 -negative cases, while staining details are not perfectly reconstructed, particularly in intensitybased results, pathologists were still able to perform accurate diagnoses, except in one negative case where two out of three pathologists found it challenging to make confident decisions. The results in Fig. 24 clearly highlight the superiority of FLIM-based virtual TTF-1 staining over intensity-based methods, especially in negative cases, where consistent diagnosis was made on virtually stained images using FLIM. Fig. 24c and Fig. 24d illustrate the inspection outcomes on virtual p40 staining using eight NSCLC cases (4 SqCC, 3 AC, and 1 OS). In general, both modalities can generate virtual p40 images suitable for clinical diagnosis. However, there are two cases in which not all pathologists were able to make consistent decisions on both real and virtual p40 images. This may be caused by the ambiguity in the real p40 staining. Similar to virtual TTF-1 , all virtual p40+ images offer excellent quality for diagnosis. However, 2 virtual p40- images are unclear, making confident decisions challenging. As far as both modalities are concerned, FLIM images are superior to intensity images in synthesis. Only one FLIM- based virtual image was considered ambiguous for decision-making, whereas four intensity-based virtual p40 images were misinterpreted by the pathologists.

[0149] Table 3 presents quantitative comparisons of four widely used similarity metrics, including mean-squared error (MSE), the normalised mutual information (MNI), the peak signal-to-noise ratio (PSNR), and the structural similarity index metric (SSIM). The results clearly indicate that lifetime surpasses intensity for virtual TTF-1 and p40 for all metrics, which is consistent with the visual evaluation of the virtual images.Table 3. Quantitative comparison of virtual TTF-1 and p40 by intensity and lifetime images. Results, presented as mean and standard deviation across all testing data, indicate that FLIM consistently outperforms intensity images in virtual IHC staining accuracy on each of these metrics, including mean- squared error (MSE), the normalised mutual information (MNI), the peak signal-to-noise ratio (PSNR), and the structural similarity index metric (SSIM). Lower MSE values correspond to better results, whereas higher NMI, PSNR, and SSIM (close to 1) values indicate better performance.

[0150] Discussion

[0151] This example demonstrates the feasibility of label-free lung cancer subtyping and evaluated it using autofluorescence intensity and lifetime images acquired from unstained NSCLC samples . The virtual staining outcomes reported illustrate the capability of generating clinical-grade virtual p40 and TTF-1 images for lung cancer diagnosis, which is routinely used in clinical pathology practice. The approach enhances the efficiency of lung cancer diagnosis and supports clinical decision-making.

[0152] Existing studies have shown that endogenous autofluorescence could be utilised for lung cancer subtyping based on statistical methods. However, it may not be effective for all cases due to the interpatient heterogeneity (Zhang et al. 2022). To assess the interpatient heterogeneity in the data set used here, the intensity and lifetime value distributions of the samples are visualised in Fig. 25. Overall, the intensity values of AC, SqCC, and OS exhibit high similarity, with closely aligned mean and standard deviation values, making subtype differentiation challenging using statistical metrics due to the high homogeneity. In terms of lifetime distributions, while normal and OS tissues demonstrate distinct separation from AC and SqCC, the latter two present highly overlapping distributions. These data show that it is not immediately apparent that FLIM information can be used for NSCLC subtyping (let alone precise prediction of markers that are indicative of such subtyping), as traditional methods for analysing FLIM data indicate large interpatient variability. The present inventors have surprisingly discovered that this could be done with high accuracy using the methods described herein.

[0153] Additionally, the GAN-based virtual staining work described here is the first work to synthesise stained images for specific biomarkers for AC and SqCC, the most prevalent forms of NSCLC, using label-free autofluorescence images. The label-free virtual IHC staining on two markers for lung cancer subtyping demonstrates the potential of virtual histological staining to go beyond the current state-of-the- art in autofluorescence-based virtual H&E and other common histological staining techniques. In addition, the results also indicate that single-band autofluorescence images are effective for the purposes, rather than multi-channel images used in the existing method (Bai et al. 2022). The visual evaluation by experienced pathologists highlights the effectiveness of our methodology in converting autofluorescenceimages into virtual IHC images for diagnostic use. Since true IHC stains were generated in an accredited pathology laboratory, this indicates that the synthetic outcomes align with clinical standards. In combination with virtual H&E staining as described in Example 1 , the techniques described here can now generate virtual H&E, TTF-1 , and p40 images from a single autofluorescence image. By bypassing the traditional multi-step tissue processing procedures, these methods can provide these routine tests in minutes, without compromising the accuracy of clinical decision-making. The success of virtual IHC staining suggests that autofluorescence signals may vary across different tumour phenotypes, highlighting the efficiency of lung cancer subtyping using autofluorescence images. Furthermore, the advantage of lifetime over intensity for both subtyping and virtual staining demonstrates a higher contrast of lifetime in tumour phenotypes, providing clearer differentiation than intensity images alone. Our virtual IHC images also demonstrated the contrast in intensity and lifetime between different cell phenotypes (Fig. 23). The virtual staining reused the DL technique described in Example 1. This has several advantages. For example, the training does not require extensive experiments on determining hyperparameters, making the transfer learning straightforward without any modification. This will also help simplify the integration of all these techniques into a unified platform able to generate all these synthetic images in one run.CONCLUSIONS

[0154] The inventors explored the use of DL techniques to generate virtual H&E and IHC staining from label-free FLIM images. The results above show successful implementation of a DL model to virtually stain label-free histological images, with excellent agreement between real H&E and virtual H&E staining as validated by experienced pathologists. The DL model was then extended to be applied to IHC stains against P40 and TTF1 , demonstrating its utility in providing fast, reliable, interpretable images to pathologists for the diagnosis of non-small cell lung cancer subtypes.

[0155] Specifically, the inventors present a method to combine the pix2pix network (Isola, 2017) with the DISTS loss (Ding, 2022) to generate clinical-grade virtual H&E and P40+ / TTF1+staining from label- free FLIM images acquired on unstained lung tissue samples. Despite a variety of virtual histological staining methodologies having been proposed, this study differs from existing methods in several aspects. Firstly, the imaging modality is different from those in existing methods, resulting in different types of label-free images. The inventors’ FLIM images were obtained using single channel images, at excitation and emission wavelengths selected for the particular application. These were obtained using a confocal microscope at the optimal excitation and emission wavelengths determined by a A-to-A / spectral scan on the tissue provided by the Leica FLIM system, containing the intensity and the corresponding lifetime images. In contrast, the autofluorescence images used in existing methods are either collected using particular filters, such as those in Rivenson, 2019a, or a combination of multi-channel intensity images and lifetime images (Borhani, 2019). Additionally, the methods described in the examples are applicable to datasets where the pixel sizes of the input and reference images is not the same, wherein in the existing approaches the pixel sizes of the input and reference images are at matching levels. In particular, in Example 1 the pixel size of FLIM images is only half that of true H&E images. Thus, the H&E-stained images were downsampled to match the pixel size of FLIM images using bicubicinterpolation, then image tiles were stitched together, co-registered (using only the intensity channel from the FLIM images) using affine transformation, and tiles of size matching the input size of the DL model were obtained from both the FLIM and H&E images. Furthermore, virtual histological staining from FLIM images will not only bypass the conventional tissue processing procedures, but more importantly, it will overcome the current challenges for instant perception of FLIM images. The ability to instantly perceive FLIM images through virtual histological staining will significantly broaden the applications of FLIM in tissue analysis and cancer detection and diagnosis, enabling cellular-level analysis in more complex situations, such as cellular components in the tumour microenvironment (TME), which is still in its infancy.

[0156] The present method's superiority over existing approaches, such as autofluorescence intensity images, is primarily attributed to the additional information introduced by fluorescence lifetime, which is independent of intensity but highly sensitive to surrounding bio-environment. An initial scan of the unstained tissue using a commercial FLIM imaging system can identify the optimal excitation / emission wavelengths best suited for the samples, ensuring the acquisition of high-quality FLIM images. The inventors previously demonstrated that in the context of tumour detection using statistical analyses of FLIM images, combining both the intensity and lifetime information that is available in FLIM can achieve optimal outcomes (Wang, 2022). In the present context of virtual staining, the present inventors compared image formats that enable combining and analysing this information at the pixel level, revealing that intensity-weighted false-colour lifetime images are most suited for the task of virtual staining. Consequently, the present method applies intensity-weighted false-colour images as the input into a supervised GAN model for paired image-to-image translation, namely pix2pix, with an advanced image quality metric (DI STS).

[0157] In the process of sample preparation and data acquisition, a common practice for virtual histological staining from autofluorescence images involves first scanning unstained samples using autofluorescence imaging modalities and subsequently staining the same samples (Rivenson, 2019a; Zhang, 2020; Li, 2020). This approach offers a significant advantage in achieving optimal co -registration at the pixel level with minimal distortion on both images, which is crucial for the effective utilisation of supervised DL techniques. To achieve this, diverse techniques have been applied, such as multi-stage elastic-based registration (Rivenson, 2019a) or RANSAC-based warping (Borhani, 2019). Nonetheless, the inventors demonstrate that with the same sample preparation and data collection procedure, employing an affine transformation for the co-registration is adequate to achieve satisfactory virtual staining, thus, substantially reducing technical uncertainties for enhanced reproducibility.

[0158] It is important to note that although the inventors leveraged pix2pix and DISTS as their DL method, other advanced models could also be exploited for the purpose, such as UNet-based GAN (Rivenson, 2019a; Zhang, 2020; Li, 2021), FCNN-p2p and VGG-a2p (Borhani, 2019), or conditional GAN (Li, 2020). Generally, all DL models described in Pang, 2022 for image-to-image translation may be applicable to DL-based virtual histological staining. However, direct applications may not be feasible for this purpose. The inventors experimented this with three advanced DL models without any optimisation, including the original pix2pix GAN, ResVit (Dalmaz, 2022), and Denoising diffusion GAN (DDGAN, Xiao et al. 2021), and the results are depicted in Fig. 18. Despite the success of these models, they do notproduce virtual H&E staining images of the same quality as those using the combination of an image-to- image translation model and an advanced loss function . This shows that in addition to these promising GAN models, incorporating advanced loss functions to further refine the learning process is beneficial for achieving optimal results. For example, the perceptual loss (Johnson, 2016) can be employed for high- level feature reconstruction while preserving realistic textural information, and the texture loss (Gatys, 2016) can help retain fine textural details from the original images. The DISTS loss (Ding, 2022) harnessed in this study combines spatial information with textural features, making it a more favourable choice for enhancing the reconstruction.

[0159] Various virtual histological staining techniques have been proposed over the past few years (Bai, 2023), and the inventors’ approach perfectly complements these methods in a novel aspect. Existing methods for virtual H&E staining usually facilitate autofluorescence images acquired at specific excitation / emission wavelengths using specific filters, such as DAPI and Cy5 channels (Rivenson, 2019a). In contrast, the inventors’ FLIM image acquisition utilised excitation and emission wavelengths determined through a A-to-A scan of the lung tissue samples, resulting in optimal excitation (485 nm and 446nm, respectively in Examples 1 and 2) and emission ([500 nm, 720 nm] and [460nm, 640nm], respectively in Examples 1 and 2) wavelengths for each study. This represents a significant difference in the present approach compared to other proposals, which is independent of specific pre-set filters, allowing the collection of high-quality images at optimal excitation / emission wavelengths. This enhancement was confirmed through blinded evaluations by experienced pathologists and quantitative assessments using commonly used similarity metrics.

[0160] Despite the prevalence of FLIM in biomedical and clinical applications, most FLIM image analysis methods primarily rely on statistical techniques, such as histogramming, to summarise the data. These approaches are effective at identifying dominant components within FLIM images but may fall short when it comes to providing cellular-level information. This limitation becomes particularly apparent when attempting to discern the intricate details of cellular components within complex environments like the tumour microenvironment. This may be overcome by co-registering FLIM with histology images, which is still challenging, and sometimes may be infeasible. Virtual staining (such as H&E staining) can address the limitations by offering a rapid and precise reference for FLIM images at the cellular level. This reference enables the identification of distinct lifetime signatures associated with various cell types within tissue samples, as illustrated in FIG. 6. By providing this valuable reference, virtual H&E staining greatly expands the potential applications of FLIM beyond its current scope. This advancement has the great potential for label-free characterisation of both morphological and phenotypical features of cells, opening up new possibilities for research and clinical purposes.REFERENCESA number of publications are cited above in order to more fully describe and disclose the invention and the state of the art to which the invention pertains. Full citations for these references are provided below. The entirety of each of these references is incorporated herein.1 . Bishop, J. et al. p40 (ANp63) is superior to p63 for the diagnosis of pulmonary squamous cell carcinoma. Mod Pathol 25, 405-415 (2012).2. Guazzi, S. et al. “Thyroid nuclear factor 1 (TTF-1) contains a homeodomain and displays a novel DNA binding specificity.” The EMBO Journal 9, 3631-9. (1990).3. Agoff, S. N. et al. Thyroid transcription factor-1 is expressed in extrapulmonary small cell carcinomas but not in other extrapulmonary neuroendocrine tumors. Modern pathology: an official journal of the United States and Canadian Academy of Pathology, Inc, 13, 238-242 (2000).4. Lazzaro, D. et al. The transcription factor TTF-1 is expressed at the onset of thyroid and lung morphogenesis and in restricted regions of the foetal brain. Development (Cambridge, England) 113, 1093-104 (1991).5. Kim, J. H. et al. Prognostic Impact of TTF-1 Expression in Non-Squamous Non-Small-Cell Lung Cancer: A Meta-Analysis. Journal of Cancer 9, 4279-4286 (2018).6. Marcu, L. Fluorescence lifetime techniques in medical applications. Ann Biomed Eng 40, 304- 31 (2012).7. Datta, R., Heaster, T. M., Sharick, J. T., Gillette, A. A. & Skala, M. C. Fluorescence lifetime imaging microscopy: fundamentals and advances in instrumentation, analysis, and applications. J Biomed Opt25, 1-43 (2020).8. Wang, Y. L. et al. Rapid, label-free, and highly sensitive detection of cervical cancer with fluorescence lifetime imaging microscopy, leee Journal of Selected Topics in Quantum Electronics 22 (2016).9. Alfonso-Garcia, A. et al. Label-free identification of macrophage phenotype by fluorescence lifetime imaging microscopy. J Biomed Opt 21 , 46005 (2016).10. Rahim, M. K. et al. Phasor analysis of fluorescence lifetime enables quantitative multiplexed molecular imaging of three probes. Analytical Chemistry 94, 14185-14194 (2022). PMID: 36190014.11. Unger, J. et al. Real-time diagnosis and visualization of tumor margins in excised breast specimens using fluorescence lifetime imaging and machine learning. Biomed. Opt. Express 11 , 1216— 1230 (2020).12. Wang, Q. et al. Deep learning-assisted co-registration of full-spectral autofluorescence lifetime microscopic images with h&e-stained histology images. Communications Biology 5, 1119 (2022).13. Madabhushi, A. & Lee, G. Image analysis and machine learning in digital pathology: Challenges and opportunities. Medical image analysis 33, 170-175 (2016).14. Bai, B. et al. Deep learning-enabled virtual histological staining of biological samples. Light: Science & Applications 12, 57 (2023).15. Goodfellow, I. et al. Generative adversarial nets, (eds Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. & Weinberger, K.) Advances in Neural Information Processing Systems, Vol. 27 (Curran Associates, Inc., 2014).16. Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nature Biomedical Engineering 3, 466-477 (2019a).17. Li, D. et al. Deep learning for virtual histological staining of bright-field microscopic images of unlabeled carotid artery tissue. Molecular imaging and biology 22, 1301-1309 (2020).18. Isola, P., Zhu, J. Y., Zhou, T. H. & Efros, A. A. Image-to-image translation with conditional adversarial networks. 30th leee Conference on Computer Vision and Pattern Recognition (Cvpr 2017) 5967-5976 (2017).19. Borhani, N., Bower, A. J., Boppart, S. A. & Psaltis, D. Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy. Biomedical Optics Express 10, 1339- 1350 (2019).20. Ding, K. Y., Ma, K. D., Wang, S. Q. & Simoncelli, E. P. Image quality assessment: Unifying structure and texture similarity, leee Transactions on Pattern Analysis and Machine Intelligence 44, 2567-2581 (2022).21. Wang, Q. et al. A layer-level multi-scale architecture for lung cancer classification with fluorescence lifetime imaging endomicroscopy. Neural Computing and Applications 34, 18881-18894 (2022).22. Zhang, Y. J. et al. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light-Science & Applications 9 (2020).23. Li, J. X. et al. Biopsy-free in vivo virtual histology of skin using deep learning. Light-Science & Applications 10 (2021).24. Pang, Y., Lin, J., Qin, T. & Chen, Z. Image-to-image translation: Methods and applications. IEEE Transactions on Multimedia 24, 3859-3881 (2022).25. Johnson, J., Alahi, A. & Fei-Fei, L. Perceptual losses for real-time style transfer and superresolution. Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I1 14 694-711 (2016).26. Gatys, L. A., Ecker, A. S. & Bethge, M. Image style transfer using convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition 2414-2423 (2016).27. Koppensteiner, L. et al. Location of cd39+ t cell subpopulations within tumors predict differential outcomes in non-small cell lung cancer. Journal for Immunotherapy of Cancer 11 (2023).28. Chalfoun, J. et al. Mist: Accurate and scalable microscopy image stitching tool with stage modeling and error minimization. Scientific Reports 7 (2017).29. Rivenson, Y. et al. Phasestain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light-Science & Applications 8 (2019b).30. Liu, S. T. et al. Unpaired stain transfer using pathology-consistent constrained generative adversarial networks, leee Transactions on Medical Imaging 40, 1977-1989 (2021).31. Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv (2017).32. Zhang, L. et al. Wavelet Knowledge Distillation: Towards Efficient Image-to-lmage Translation. arXiv (2022).33. Zhang, Y. et al. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light Sci Appl 9, 78 (2020).34. Li, D. et al. Deep Learning for Virtual Histological Staining of Bright-Field Microscopic Images of Unlabeled Carotid Artery Tissue. Mol Imaging Biol. 22, 1301-1309 (2020).35. Xiao et al. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. arXiv:2112.07804v2. Mon, 4 Apr 202236. Dalmaz et al. ResViT: Residual vision transformers for multi-modal medical image synthesis. arXiv:2106.16031v3. Sun, 6 Mar 202237. Walsh, A. J. et al. Classification of t-cell activation via autofluorescence lifetime imaging. Nat Biomed Eng 5, 77-88 (2021).38. Linders, P. T., loannidis, M., Ter Beest, M. & van den Bogaart, G. Fluorescence lifetime imaging of pH along the secretory pathway. ACS chemical biology 17, 240-251 (2022).39. Yan, Y. et al. Fluorescence intensity and lifetime imaging of lipofuscin -like autofluorescence for label-free predicting clinical drug response in cancer. Redox Biol 59, 102578 (2023).40. Becker, W. Fluorescence lifetime imaging-techniques and applications. J. Microsc. 247, 119— 136 (2012).41. Fernandes, S. et al. Fibre-based fluorescence-lifetime imaging microscopy: a real-time biopsy guidance tool for suspected lung cancer. Transl. Lung Cancer Res. 13, 355 (2024).42. Adams, A. C. et al. Fibre-optic based exploration of lung cancer autofluorescence using spectral fluorescence lifetime. Biomed. Opt. Express 15, 1132-1147 (2024).43. Zang, Z. et al. Fast analysis of time-domain fluorescence lifetime imaging via extreme learning machine. Sensors 22, 3758 (2022).44. Sorrells, J. E. et al. Real-time pixelwise phasor analysis for video-rate two-photon fluorescence lifetime imaging microscopy. Biomed. Opt. Express 12, 4003-4019 (2021).45. Luo, T., Lu, Y., Liu, S., Lin, D. & Qu, J. Phasor-FLIM as a Screening tool for the differential diagnosis of actinic keratosis, Bowen’s disease, and basal cell carcinoma. Anal. Chem. 89, 8104-8111 (2017).46. Hu, L., Ter Hofstede, B., Sharma, D., Zhao, F. & Walsh, A. J. Comparison of phasor analysis and biexponential decay curve fitting of autofluorescence lifetime imaging data for machine learning prediction of cellular phenotypes. Front. Bioinforma. 3, 1210157 (2023).47. Wang, Q. et al. Deep learning in ex -vivo lung cancer discrimination using fluorescence lifetime endomicroscopic images, in 2020 42nd annual international conference of the IEEE engineering in medicine & biology society (EMBC) 1891-1894 (IEEE, 2020).48. Wang, Q., Vallejo, M. & Hopgood, J. Fluorescence lifetime endomicroscopic image-based ex- vivo human lung cancer differentiation using machine learning. Authorea Prepr. (2023).49. Wang, Q. et al. A layer-level multi-scale architecture for lung cancer classification with fluorescence lifetime imaging endomicroscopy. Neural Comput. Appl. 34, 18881 -18894 (2022).50. Wang, Q. et al. Deep learning-based virtual H& E staining from label-free autofluorescence lifetime images. Np] Imaging 2, 17 (2024).51. Li, X. et al. Unsupervised content-preserving transformation for optical microscopy. Light Sci. Appl. 10, 44 (2021).52. DoanNgan, B., Angus, D., Sung, L., & others. Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning. BME Front. (2022).53. Zhang, G. et al. Image-to-images translation for multiple virtual histological staining of unlabeled human carotid atherosclerotic tissue. Mol. Imaging Biol. 1 -11 (2022).54. Kang, L., Li, X., Zhang, Y. & Wong, T. T. Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining. Photoacoustics 25, 100308 (2022).55. Cao, R. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nat. Biomed. Eng. 7, 124-134 (2023).56. Levy, J. J., Jackson, C. R., Sriharan, A., Christensen, B. C. & Vaickus, L. J. Preliminary evaluation of the utility of deep generative histopathology image translation at a mid-sized NCI cancer center. BioRxiv 2020- 01 (2020).57. Hong, Y. et al. Deep learning-based virtual cytokeratin staining of gastric carcinomas to measure tumor-stroma ratio. Sci. Rep. 11 , 19255 (2021).58. Moldvay, J. et al. The role of TTF-1 in differentiating primary and metastatic lung adenocarcinomas. Pathol. Oncol. Res. 10, 85-88 (2004).59. Affandi, K. A., Tizen, N. M. S., Mustangin, M. & Zin, R. R. M. R. M. p40 immunohistochemistry is an excellent marker in primary lung squamous cell carcinoma. J. Pathol. Transl. Med. 52, 283-289 (2018).60. Zhang, M. et al. Decreased green autofuorescence of lung parenchyma is a biomarker for lung cancer tissues. J. Biophotonics 15, e202200072 (2022).61. Bai, B. et al. Label-free virtual HER2 immunohistochemical staining of breast tisse using deep learning. BME Front. 2022; 2022: 9786242.62. Preibisch, S., Saalfeld, S. & Tomancak, P. Globally optial stiching of tied 3D microscopic image acquisitions. Bioinformatcs 25, 1463-1465 (2009).63. Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600-612 (2004)For standard molecular biology techniques, see Sambrook, J., Russel, D.W. Molecular Cloning, A Laboratory Manual. 3 ed. 2001 , Cold Spring Harbor, New York: Cold Spring Harbor Laboratory Press.The features disclosed in the foregoing description, or in the following claims, or in the accompanying drawings, expressed in their specific forms or in terms of a means for performing the disclosed function, or a method or process for obtaining the disclosed results, as appropriate, may, separately, or in any combination of such features, be utilised for realising the invention in diverse forms thereof.While the invention has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the invention set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may bemade without departing from the spirit and scope of the invention. For the avoidance of any doubt, any theoretical explanations provided herein are provided for the purposes of improving the understanding of a reader. The inventors do not wish to be bound by any of these theoretical explanations. Any section headings used herein are for organizational purposes only and are not to be construed as limiting the subject matter described. Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps. It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%. The expression “and / or” where used herein is to be taken as specific disclosure of each of the two specified features or components with or without the other. For example “A and / or B” is to be taken as specific disclosure of each of (i) A, (ii) B and (iii) A and B, just as if each is set out individually herein.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method of obtaining a stained image of a biological sample using a single channel label-free fluorescence lifetime image of said biological sample, the label-free fluorescence lifetime image comprising a fluorescence intensity image and associated fluorescence lifetime image, the method comprising: obtaining a single channel composite autofluorescence intensity and lifetime image from the single channel label-free fluorescence lifetime image; and providing the single channel composite autofluorescence intensity and lifetime image as input to a deep learning model that has been trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image.

2. The computer-implemented method of claim 1 , wherein the deep learning model has been trained using a loss function that includes an image quality metric.

3. The computer-implemented method of claim 2, wherein a loss function that includes an image quality metric is a loss function that includes one or more terms that penalise a difference in structure and / or texture between a predicted stained image and a corresponding ground truth stained image.

4. The computer-implemented method of claim 2 or claim 3, wherein the deep learning model has been trained using a loss function selected from DISTS loss, perceptual loss, and texture loss, optionally wherein the deep learning model has been trained using a DISTS loss.

5. The computer-implemented method of any one of claims 1 to 4, wherein a composite autofluorescence intensity and lifetime image is an intensity -weighted lifetime image.

6. The computer-implemented method of claim 5, wherein an intensity weighted lifetime image is a false-colour lifetime image with colour depending on lifetime and the corresponding intensity image as the alpha channel, or an image obtained by multiplying pixel values in an autofluorescence intensity image by the corresponding pixel values in a corresponding fluorescence lifetime image.

7. The computer-implemented method of any one of claims 1 to 6, wherein the deep learning model is an image-to-image deep neural network, optionally a generator of a generative adversarial network, optionally a conditional GAN, or an autoencoder, optionally a variational autoencoder.

8. The computer-implemented method of any one of claims 1 to 7, wherein the single channel florescence lifetime image is a fluorescence lifetime image that has been acquired using an excitation wavelength and range of emission wavelength identified using a A-to-A scan of one or more biological samples.

9. The computer-implemented method of any one of claims 1 to 8, wherein the single channel florescence lifetime image is a fluorescence lifetime image that has been acquired using an excitation wavelength and range of emission wavelength that is the same as that used to acquire a plurality of single channel fluorescence lifetime images used to train the deep learning model.

10. The computer-implemented method of any one of claims 1 to 9, wherein the single channel fluorescence lifetime images used to train the deep learning model comprise or consist essentially of images of a plurality of training biological samples of the same type as the biological sample for which a stained image is being predicted, wherein biological samples of the same type are samples comprising the same types of cells or tissues, or wherein the single channel fluorescence lifetime images used to train the deep learning model are images of a plurality of training biological samples comprising a first plurality of samples of a different type from the biological sample for which a stained image is being predicted and a second plurality of samples of the same type as the biological sample for which a stained image is being predicted, optionally wherein the deep learning model has been trained using images of the first plurality of samples then further trained using images of the second plurality of samples.11 . The computer-implemented method of claim 9 or claim 10, wherein the single channel florescence lifetime image is a fluorescence lifetime image that has been acquired using a excitation wavelength and range of emission wavelength that is the same as that used to acquire a plurality of single channel fluorescence lifetime images used to train the deep learning model, wherein the excitation wavelength and range of emission wavelength have been identified using a A-to-A scan of one or more of the training biological samples.

12. The computer-implemented method of any one of claims 1 to 11 , wherein the stain is a chemical stain, optionally a H&E stain.

13. The computer-implemented method of any one of claims 1 to 11 , wherein the stain is immunohistochemistry stain, optionally a P40 stain or TTF1 stain.

14. The computer-implemented method of any one of claims 1 to 13, wherein the method further comprises pre-processing the single channel label-free fluorescence lifetime image using one or more of: thresholding of the intensity image, thresholding of the lifetime image, normalizing the intensity image, and colour inverting of the intensity image.

15. The computer-implemented method of any one of claims 1 to 14, wherein the method further comprises obtaining a plurality of tiles of predetermined size from the composite fluorescence intensity and lifetime image, wherein providing the single channel composite autofluorescence intensity and lifetime image as input to a deep learning model comprises providing one or more of the plurality of tiles as input to the deep learning model, and optionally excluding any tile that includes more than apredetermined threshold proportion of background pixels in the intensity image, optionally wherein the predetermined threshold is 75%.

16. The computer-implemented method of any one of claims 1 to 15, wherein the deep learning model has been trained using training composite autofluorescence intensity and lifetime images and corresponding stained images that have been coregistered, wherein the training composite autofluorescence intensity and lifetime images and corresponding stained images have been coregistered using corresponding stained images that have been preprocessed using contrast enhancing.

17. The computer-implemented method of any one of claims 1 to 16, wherein the deep learning model has been trained using training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images, wherein the training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images are images that have been coregistered using an affine transformation, optionally using, for each pair of images to be coregistered, coregistration of an autofluorescence intensity image from which the composite autofluorescence intensity and lifetime image is obtained, and a corresponding ground truth stained image .

18. The computer-implemented method of any one of claims 1 to 17, wherein the deep learning model has been trained using training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images, wherein the training composite autofluorescence intensity and lifetime images and corresponding ground truth stained images comprise images that have been obtained from coregistered autofluorescence intensity and lifetime images and corresponding ground truth stained images using image augmentation, optionally wherein image augmentation comprises creating a flipped version of one or more of the images and / or creating a randomly rotated version of one or more of the images.

19. The computer-implemented method of any one of claims 1 to 18, wherein the biological sample is a tumour sample, an ex vivo tumour tissue sample that has been previously obtained from a patient, a fixed tissue sample, or a tumour microarray, and / or wherein the biological sample is a lung tissue sample, a colorectal tissue sample, or an endometrial tissue sample, and / or wherein the fluorescence lifetime images have been acquired using a fluorescence lifetime imaging microscope.

20. The computer-implemented method of any one of claims 1 to 19, wherein the deep learning model has been trained using a method comprising: a pretraining step comprising training the deep learning model to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image thereby obtaining a pre-trained model, wherein the corresponding stained image is a chemically stained image, optionally a H&E stained image, using training data comprising, for each of a plurality of biological samples: (i) one or more training single channel composite autofluorescence intensity andlifetime images each obtained from a respective single channel label-free fluorescence lifetime image; and (ii) one or more corresponding training chemically stained images; a transfer learning step in which the pretrained model is trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image, wherein the corresponding stained image is a an immunohistochemically stained image, optionally a P40 or TTF1 -stained image, using training data comprising, for each of a plurality of biological samples: (i) one or more training single channel composite autofluorescence intensity and lifetime images each obtained from a respective single channel label-free fluorescence lifetime image; and (ii) one or more corresponding immunohistochemically training stained images.

21. The computer-implemented method of any one of claims 1 to 20, wherein the method further comprises one or more of:(i) identifying one or more regions of interest in the biological sample using the stained image output by the deep learning model, and determining a property of the single channel label-free fluorescence lifetime image associated with the regions of interest, optionally wherein the property is a fluorescence lifetime imaging signature;(ii) providing a diagnosis associated with the biological sample using the stained image optionally in combination with the single channel label-free fluorescence lifetime image, optionally wherein providing a diagnosis comprises identifying the presence of cancer or identifying a cancer subtype;(iii) providing a prognosis or treatment recommendation for a subject associated with the biological sample using the stained image optionally in combination with the single channel label-free fluorescence lifetime image; and(iv) selecting a subject associated with the biological sample based on analysis of the stained image optionally in combination with the single channel label-free fluorescence lifetime image, optionally wherein said analysis comprises using information in the stained image to identify a disease or disease subtype likely to be present in the subject.

22. The computer-implemented method of any one of claims 1 to 21 , wherein the stain is an H&E stain, and wherein the method further comprises providing the single channel composite autofluorescence intensity and lifetime image as input to one or more further deep learning models that have each been trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image, wherein the stain is a respective immunohistochemistry stain, optionally a P40 stain and / or TTF1 stain.

23. The computer-implemented method of claim 22, wherein the biological sample is from a subject diagnosed as having, being likely to have, or suspected of having non-small cell lung cancer, the respective immunohistochemistry stains include P40 stain and TTF1 stain, and the method further comprises diagnosing a NSCLC subtype using the immunohistochemically stained images output by the further deep learning models.

24. A method of diagnosing a disease or disorder in a subject, the method comprising: receiving one or more single channel label-free fluorescence lifetime images of a biological sample from said subject; obtaining one or more stained images from each of said one or more single channel label-free fluorescence lifetime images using the method of any one of claims 1 to 23; and providing a diagnosis for said subject using said one or more stained images, optionally wherein the subject is a subject diagnosed as having or being likely to have cancer, and providing a diagnosis comprises identifying a cancer subtype.

25. The method of claim 24, wherein the cancer is lung cancer, obtaining one or more stained images from each of said one or more single channel label-free fluorescence lifetime images using the method of any one of claims 1 to 19 comprises obtaining a H&E stained image, a P40 stai ned image and a TTF1 stained image using respective deep learning models trained to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a respective corresponding stained image, and identifying a cancer subtype comprises determining whether the subject has one or more of large cell carcinoma, squamous cell carcinoma and adenocarcinoma.

26. A method of providing a prognosis for a subject, selecting a treatment for a subject or selecting a subject for further diagnostic testing, the method comprising: diagnosing a disease or disorder in the subject using the method of any one of claims 24 to 25, and determining a prognosis for a subject, identifying a treatment for the subject or selecting the subject for further diagnostic testing using the results of the determining.

27. The method of claim 26, wherein the method comprises: (i) identifying a lung cancer subtype in the subject using the one or more stained images; and determining a prognosis for the subject, wherein a subject identified as having adenocarcinoma has better prognosis than a subject identified as having squamous cell carcinoma; (ii) identifying a lung cancer subtype in the subject using the one or more stained images; and selecting a treatment for the subject based on the identified lung cancer subtype and optionally one or more predetermined clinical characteristics; or (iii) identifying a lung cancer subtype in the subject using the one or more stained images; and selecting the subject for further diagnostic test, optionally molecular testing, based on the identified lung cancer subtype and optionally one or more predetermined clinical characteristics.

28. A computer-implemented method of obtaining a trained deep learning model for obtaining a stained image of a biological sample using a single channel label-free fluorescence lifetime image of said biological sample, the label-free fluorescence lifetime image comprising a fluorescence intensity image and associated fluorescence lifetime image, the method comprising: obtaining a training data set comprising, for each of a plurality of biological samples: (i) one or more training single channel composite autofluorescence intensity and lifetime images each obtainedfrom a respective single channel label-free fluorescence lifetime image; and (ii) one or more corresponding training stained images; and training a deep learning model to take as input a single channel composite autofluorescence intensity and lifetime image and provide as output a corresponding stained image, using said training data set.

29. A system comprising: one or more processors and computer readable memory storing instructions that cause the processor to perform the method of any one of claims 1 -28, optionally wherein the system further comprises data acquisition means configured to obtain fluorescence lifetime images.

Images