Virtual staining based on multiple imaging data sets with multiple phase contrasts

Abstract

Various examples of the present disclosure relate to techniques for virtually staining a tissue sample. One or more output images having a virtual stain are determined. These techniques are directed to a more robust process. To this end, a plurality of imaging data sets are used as inputs for a digital process, the plurality of imaging data sets having a plurality of different phase contrasts.

Description

Technical Field

[0001] Various embodiments relate to techniques for virtual staining by utilizing machine learning logic. Various examples specifically relate to processing multiple imaging datasets with different phase contrasts. Background Art

[0002] Histopathology is an important tool for diagnosing disease. Histopathology involves the optical examination of tissue samples. This facilitates diagnosis by analyzing the cells within a tissue sample.

[0003] Typically, a histopathological examination begins with surgery, biopsy, or autopsy to obtain the tissue to be examined. The tissue may be treated to remove moisture and prevent decay. The treated sample may then be embedded in a wax block. Thin slices can be cut from the wax block. These slices may be referred to as tissue samples hereinafter.

[0004] Tissue samples can be analyzed under a microscope by a histopathologist. Tissue samples can be stained with chemical stains using appropriate staining laboratory procedures to facilitate analysis. In particular, chemical stains can reveal cellular components that are difficult to observe in unstained tissue samples. Furthermore, chemical stains can provide contrast. Chemical stains can highlight one or more biomarkers or predefined structures in a tissue sample.

[0005] The most commonly used chemical stain in histopathology is the combination of hematoxylin and eosin (abbreviated as H&E). Hematoxylin stains cell nuclei blue, while eosin stains the cytoplasm and extracellular connective tissue matrix pink. Hundreds of other techniques have been used to selectively stain cells. More recently, antibodies have been used to stain specific proteins, lipids, and carbohydrates. This technique, called immunohistochemistry, has greatly improved the ability to unambiguously identify various cell types under the microscope. Staining with H&E stains is considered the common gold standard for histopathological diagnosis.

[0006] By staining tissue samples with chemical stains, the previously nearly transparent, indistinguishable structures of the tissue sample / tissue section become visible to the human eye. This allows pathologists and researchers to study tissue samples under a microscope or using a digital brightfield equivalent image and assess tissue morphology (structure) or explore the presence or prevalence of specific cell types, structures, and even microorganisms (such as bacteria).

[0007] Preferably, several chemical stains are used to fully assess a pathology case. Typically, only one chemical stain can be applied to a tissue sample. Therefore, if several chemical stains are needed for diagnosis, several tissue samples must be prepared. In addition, different chemical stains may require different staining protocols. Therefore, known chemical staining techniques are labor- and cost-intensive.

[0008] WO 2019 / 154987 A1 discloses a method that uses machine learning logic to provide a virtual stained image that looks like a typical image of a tissue sample that has been stained with a conventional chemical stain. The virtual staining technique bypasses the typically labor-intensive and expensive histological staining procedure and can be used as a blueprint for virtual staining of tissue images acquired with other label-free imaging modalities. The virtual staining method can be used to micro-guide molecular analysis at the level of unstained tissue by locally identifying regions of interest based on virtual staining and by using this information to guide subsequent analysis of the tissue, for example, micro-immunohistochemistry or sequencing. This type of virtual micro-guidance on unlabeled tissue samples can facilitate high-throughput identification of disease subtypes and the development of customized therapies for patients.

[0009] This related technology faces certain limitations. In particular, the accuracy of the virtual stain may be limited. The flexibility of selecting different virtual stains may also be limited.

[0010] To alleviate this limitation, WO 2021 / 198241 discloses a virtual stain determined based on multiple imaging datasets depicting a tissue sample and acquired using multiple imaging modalities. The multiple imaging modalities are selected from the group consisting of: hyperspectral microscopy; fluorescence imaging; autofluorescence imaging; light sheet imaging; digital phase contrast; and Raman spectroscopy.

[0011] While such techniques improve the accuracy of virtual stains, techniques such as hyperspectral microscopy, fluorescence imaging, light-sheet imaging, and Raman spectroscopy are relatively complex and time-consuming. Furthermore, the required light intensity and / or light dose can be significant, posing a risk of damaging the sample. Summary of the Invention

[0012] Therefore, there is a need for advanced techniques for virtually staining tissue samples. In particular, there is a need to determine output images that depict tissue samples with accurate virtual stains. Robust prediction of virtual stains is required.

[0013] This need is met by the features of the independent claim. Embodiments are defined by the features of the dependent claims.

[0014] A method for virtually staining a tissue sample is disclosed. The method includes obtaining a plurality of imaging datasets depicting the tissue sample. The plurality of imaging datasets have a plurality of different phase contrasts. The method also includes fusing and processing the plurality of imaging datasets in machine learning logic. The method further includes obtaining at least one output image from the machine learning logic, each of the at least one output image depicting the tissue sample including a corresponding virtual stain.

[0015] As disclosed, a computing device includes at least one processor and a memory. The at least one processor can load program code from the memory and execute the program code. Executing the program code causes the at least one processor to perform the method disclosed above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The workflow for virtual staining of tissue samples is schematically illustrated.

[0017] Figure 2 An image with a virtual stain according to various examples is schematically illustrated.

[0018] Figure 3 is a flow chart of a method according to various examples.

[0019] Figure 4 A system for determining imaging data with digital phase contrast is illustrated.

[0020] Figure 5 Differential digital phase contrast according to various examples is schematically illustrated.

[0021] Figure 6 Schematically illustrates optical intensity transmission digital phase contrast according to various examples.

[0022] Figure 7 Schematic illustration of the spatial frequency coverage of differential digital phase contrast and intensity transmission digital phase contrast.

[0023] Figure 8 A virtual stain according to an example is illustrated.

[0024] Figure 9 A virtual stain according to an example is illustrated. DETAILED DESCRIPTION

[0025] Some examples of the present disclosure generally provide multiple circuits or other electrical devices. All references to circuits and other electrical devices and the functions they each provide are not intended to be limited to the contents illustrated and described herein. Although specific labels may be given to the various circuits or other electrical devices disclosed, such labels are not intended to limit the operating scope of these circuits and other electrical devices. Such circuits and other electrical devices can be combined with each other and / or separated in any way based on the specific type of desired electrical implementation. It should be recognized that any circuit or other electrical device disclosed herein may include any number of microcontrollers, machine learning dedicated hardware (e.g., graphics processor units (GPUs) and / or tensor processing units (TPUs), integrated circuits, memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or other suitable variants thereof)) and software, which work together to perform the operations disclosed herein. In addition, any one or more electrical devices can be configured to execute a set of program codes embodied in a non-transient computer-readable medium that is programmed to perform any number of functions disclosed.

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the following description of the embodiments should not be construed as limiting. The scope of the present invention is not intended to be limited by the embodiments or drawings described below, which are intended to be illustrative only.

[0027] The accompanying drawings should be considered as schematic representations, and the elements illustrated in the drawings are not necessarily illustrated to scale. Instead, the various elements are represented so that their functions and general uses become apparent to those skilled in the art. Any connection or coupling between the functional blocks, devices, components or other physical or functional units shown in the accompanying drawings or described herein may also be implemented by indirect connection or coupling. Coupling between components may also be established by wireless connection. Functional blocks may be implemented with hardware, firmware, software or a combination thereof.

[0028] Below, techniques for imaging samples are disclosed. When light interacts with a specimen of interest (specimen), such as biological tissue, three primary contrast mechanisms can be used to form an image. First, the sample can attenuate the incident light due to absorption. Second, the sample can deform the incident optical wavefront, leaving behind phase contrast. Third, the light illuminating the sample can be scattered non-elastically, either via fluorescence from the sample itself (autofluorescence) or by means of chemical markers added to the sample.

[0029] For thin biological samples, such as tissue sections or adherent cell cultures, both absorption and autofluorescence effects are generally relatively weak. Therefore, biologists often resort to one of two light microscopy imaging modalities to determine structural information: (1) Phase contrast microscopy. This contrast modality allows the sample to be preserved in its native state. Hardware phase contrast and digital phase contrast are known. (2) Chromogenic immunohistochemical stains or fluorescent markers can be used to chemically alter the contrast induced by the sample to the incident light. This contrast modality changes the chemical composition of the sample.

[0030] Various techniques are based on the following observations: Of the two imaging modalities, phase contrast (particularly digital phase contrast) is relatively easy to obtain, but it has been found that inferring chemically specific information from it is relatively challenging. Fluorescent markers can be used to label specific functional groups. However, both dye-based and fluorescent labeling have the disadvantages of requiring time-consuming sample preparation protocols and, in some cases, irreversibly altering the native state of the sample.

[0031] Recently, various machine learning techniques (called virtual staining or in-silico staining) have been reported that allow digitally converting one image modality to another - an overview of the latest related techniques can be found in the following paper: Kreiss, Lucas et al. "Digital staining in optical microscopy using deep learning--a review." arXiv preprint arXiv:2303.08140 (2023).

[0032] The various techniques described herein generally relate to virtually staining tissue samples by utilizing trained machine learning logic (MLL). The MLL can be implemented, for example, by a support vector machine or a deep neural network comprising at least one encoder branch and at least one decoder branch. Examples include U-net, see Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015. MLL is trained using a cycle-based generative adversarial network, see, for example, Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision, 2017. This architecture includes a forward loop and a backward loop, each of which includes a generator MLL and a discriminator MLL. The generator MLLs of both the forward and backward loops are implemented using the MLLs, respectively.

[0033] More specifically, according to various examples, multiple imaging datasets can be fused and processed by MLL. This is referred to as a multi-input scenario. An output image is provided. The output image depicts a tissue sample including a virtual stain. That is, the output image can have a similar appearance to a corresponding image depicting a tissue sample including a corresponding chemical stain. Thus, the virtual stain can correspond to a chemical stain of a tissue sample stained using a staining laboratory process.

[0034] For example, MLL can generate a virtual H&E (hematoxylin and eosin) stained image of a tissue sample, and / or a virtual stained image of a tissue sample highlighting the HER2 (human epidermal growth factor receptor 2) protein and / or the ERBB2 (Erb-B2 receptor tyrosine kinase 2) gene.

[0035] Another example involves virtual fluorescent staining. For example, in life science applications, transmitted light microscopy is used to acquire images of cells, for example, arranged in a multiwell plate in vitro. Furthermore, reflected light microscopy can be used, for example, in endoscopes or as surgical microscopes. It is then possible to selectively stain certain organelles, such as the nucleus, ribosomes, endoplasmic reticulum, Golgi apparatus, chloroplasts, or mitochondria. A fluorophore (or fluorescent dye, similar to a chromophore) is a fluorescent compound that regenerates upon light excitation. Fluorophores can be used to provide fluorescent chemical stains. By using different fluorophores, different chemical stains can be achieved. For example, Hoechst stain is a fluorescent dye that can be used to stain DNA. Other fluorophores include 5-aminolevulinic acid (5-ALA), fluorescein, and indocyanine green (ICG), which can even be used in vivo. Fluorescence can be selectively excited using light of a corresponding wavelength; the fluorophore then emits light of another wavelength. Corresponding fluorescence microscopes use corresponding light sources. It has been observed that irradiation with light to excite fluorescence can damage the sample; this is avoided when virtual fluorescent staining is provided. Virtual fluorescent staining mimics fluorescent chemical staining without exposing the tissue to the corresponding excitation light.

[0036] According to an example, virtual staining is facilitated by multiple phase contrasts (e.g., hardware phase contrast and / or digital phase contrast). Various techniques disclosed herein are used to robustly convert multiple imaging datasets of tissue samples with multiple different phase contrasts to other modalities, including fluorescent stains and immunohistochemical stains.

[0037] The choice between different phase contrast methods often depends on the tissue sample being studied, and acquiring multiple imaging datasets using multiple phase contrast imaging modalities can be beneficial. Multiple phase contrast methods can be combined to enhance image conversion performance in virtual staining. When using multiple phase contrast methods, MLL can obtain more comprehensive information about the sample, enabling more accurate prediction of virtual stains. It also allows for more flexible prediction of different virtual stain types.

[0038] Figure 1 Various aspects of a workflow for generating an image depicting a tissue sample including a stain (eg, a chemical stain or a virtual stain) are illustrated. Figure 1 An example of a histopathology workflow is schematically illustrated. As mentioned above, virtual staining can also be applied to use cases beyond histopathology. Different workflows for generating images can then be applied. For example, for fluorescence imaging of cells, a tissue sample containing a cell sample can be acquired using other methods and imaged using a corresponding microscope. Furthermore, in vivo imaging using an endoscope is a possible use case for generating imaging data for tissue samples.

[0039] like Figure 1 As shown, for histopathology, tissue 2102 can be obtained from an organism 2101 through surgery, biopsy, or autopsy. After processing to remove moisture and prevent decay, the tissue 2102 can be embedded in a wax block 2103. From the block 2103, multiple slices 2104 can be obtained for further analysis. One slice in the multiple slices 2104 can also be referred to as a tissue sample 2105. Corresponding tissue samples 2105 can be obtained from adjacent slices.

[0040] As previously mentioned, tissue may also include cell samples or in vivo examination using, for example, a surgical microscope or endoscope.

[0041] Before analyzing the tissue sample 2105, a chemical stain can optionally be applied to the tissue sample 2105 using a staining laboratory process to obtain a chemically stained tissue sample 2106. In some examples, the tissue sample 2105 can also be directly analyzed ( Figure 1 ) tissue sample 2105. Chemically stained tissue sample 2106 can facilitate this analysis. In particular, chemical stains can reveal cellular components or typically well-defined cellular structures that are difficult to observe in unstained tissue sample 2105. In addition, chemical stains can provide increased contrast.

[0042] Application of chemical stains may include pre-transfection or direct application of a fluorophore such as 5-ALA.

[0043] Traditionally, tissue samples 2105 or 2106 are analyzed by experts using bright field microscopy 2107 .

[0044] At the same time, it is becoming more common to use an image acquisition system 2108 that is configured to use one or more imaging modalities to acquire digital image data of the tissue sample 2105 or chemically stained tissue sample 2106. The use of different imaging modalities can facilitate the acquisition of imaging data 2109 of the tissue sample 2105, for example, 1-D, 2-D, or 3-D imaging data.

[0045] The imaging data 2109 can be processed in a tissue analyzer 2110. The tissue analyzer 2110 can be implemented by a computer and / or through cloud processing at a server. The tissue analyzer 2110 can include a memory circuit system 2111 for storing the digital image data 2109 and / or program code, and can include circuitry 2112 for processing the digital image data 2109, for example, when loading the program code. The tissue analyzer 2110 can process the imaging data 2109 to provide one or more output images 2113, which can be displayed on a display 2114 for analysis by an examiner. For example, multiple output images 2113 can be provided depicting tissue samples 2105 and 2106 including different virtual stains. The tissue analyzer 2110 can include different types of trained or untrained machine learning logic (details regarding the machine learning logic are described below) for analyzing unstained tissue samples 2105 and / or chemically stained tissue samples 2106 (i.e., the circuitry 2112 can execute the machine learning logic). The output image 2113 may depict the tissue sample 2105 with one or more virtual stains. The image acquisition system 2108 may be used to provide training data and / or reference images as ground truth for training the machine learning logic.

[0046] More generally, the tissue analyzer 2110 includes circuitry 2112, which may include a CPU and / or a GPU and / or a TPU. The circuitry 2112 may load program code from the memory 2111. The circuitry 2112 may execute the program code. When executing the program code, the circuitry 2112 may perform one or more of the following logical operations as described throughout this disclosure: obtaining imaging data, for example, via an input / output (I / O) interface of the tissue analyzer 2204 or by loading the imaging data from the memory; preprocessing the imaging data, for example, to determine digital phase contrast; virtually staining the tissue sample depicted by the imaging data; executing machine learning logic to process the imaging data (inference); obtaining at least one output image from the machine learning logic / obtaining at least one output image when executing the machine learning logic, for example, to output at least one output image via the I / O interface; setting parameters or hyperparameters of the machine learning logic when training the machine learning logic; training the machine learning logic, etc.

[0047] Figure 2Schematically illustrated are images 801-803 depicting tissue samples. Image 801 / a plurality of imaging data sets 801 depicts a tissue sample that does not include any chemical stain or virtual stain. For example, image 801 may have digital phase contrast. Differently, image 802 depicts a tissue sample that includes a chemical stain or virtual stain. Furthermore, image 803 depicts a tissue sample that includes a chemical virtual stain, wherein the chemical virtual stain of the tissue sample depicted by image 803 is different from the chemical stain or virtual stain of the tissue sample depicted by image 802: different structures or (multiple) biomarkers ( Figure 2 (the completely black area in the image).

[0048] Figure 3 is a flow chart of method 3300 according to various examples. For example, according to Figure 3 Method 3300 can be executed by at least one circuit (e.g., a CPU and / or GPU and / or TPU) when loading program code from a non-volatile memory. Figure 3 The method can be performed by the tissue analyzer 2110. Figure 3 The method facilitates virtual staining of tissue samples.

[0049] At block 3301, a plurality of imaging data sets depicting a tissue sample are obtained (e.g., loaded from a memory or obtained from a data acquisition unit via an input interface), and the plurality of imaging data sets have a plurality of phase contrasts. Each imaging data set may include multiple instances of imaging data, e.g., multiple images taken at different locations of the sample (e.g., for stitching) and / or multiple images taken at different times (e.g., for stitching). Figure 2 : Image 801).

[0050] The multiple imaging data sets may have different phase contrasts because they were acquired using different phase contrast imaging modalities. For example, at least one of the multiple imaging data sets may have been acquired using a digital phase contrast imaging modality. Examples include differential digital phase contrast (DPC) and transfer intensity (TIE) phase contrast. Other digital phase contrast imaging modalities include on-axis holographic phase contrast, off-axis holographic phase contrast, or phase shifting interferometry phase contrast. At least one of the multiple imaging data sets may also have been acquired using a non-digital ("classical" hardware-based) phase contrast imaging modality. The multiple phase contrast imaging modalities may include at least one of: Zernike phase contrast, Jamin-Lebedeff interferometry phase contrast, and shearing interferometry phase contrast.

[0051] It is not required in all scenarios to acquire multiple imaging data sets with different phase contrasts using different phase contrast imaging modalities. For example, at least two imaging data sets in the multiple imaging data sets may be acquired using the same phase contrast imaging modality but under different imaging settings. Examples include a first imaging data set in the multiple imaging data sets being acquired using a given digital phase contrast imaging modality (such as DPC or TIE phase contrast) at a first wavelength of light (e.g., red or blue or yellow or infrared or ultraviolet); and a second imaging data set in the multiple imaging data sets being acquired using the given digital phase contrast imaging modality at a second wavelength of light that is different from the first wavelength. As an alternative or supplement to using multiple wavelengths, multiple polarizations may also be used, for example, left circularly polarized light and right circularly polarized light.

[0052] The tissue sample can be a cancer tissue sample taken from a patient, or a tissue sample from another animal or plant.

[0053] Figure 3 Method 3300 optionally includes: performing preprocessing after obtaining multiple imaging data sets, such as one or a combination of the following processing techniques: noise filtering; aligning between imaging data of different imaging data sets (e.g., any pair of imaging data sets); resizing the imaging data, etc.

[0054] Imaging data including one or more images with digital phase contrast is also obtained from the pre-processed raw images. For example, for DPC, multiple raw images of the tissue sample acquired under different oblique illumination settings are combined. Similarly, for TIE, multiple raw images of the tissue sample acquired under different defocus settings are combined. Figure 3 The method 3300 may include such pre-processing for obtaining one or more imaging data sets having a plurality of different phase contrasts.

[0055] The multiple imaging datasets are fused and processed by the MLL at block 3302. The MLL has been trained using supervised learning, semi-supervised learning, or unsupervised learning.

[0056] In general, various implementations of MLL are contemplated. In one example, a deep neural network can be used. For example, a U-net implementation is possible. See Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

[0057] More generally, a deep neural network may include multiple hidden layers. A deep neural network may include an input layer and an output layer. The hidden layer is arranged between the input layer and the output layer. There may be spatial contraction and spatial expansion implemented by one or more encoder branches and one or more decoder branches, respectively. That is, the xy-resolution of the corresponding representations of the imaging data and the output image may be reduced (increased) layer by layer along one or more encoder branches (decoder branches). At the same time, the feature channel may be increased and decreased respectively along one or more encoder branches and one or more decoder branches. The one or more encoder branches and the one or more decoder branches are connected via a bottleneck. At one or more output layers, a deep neural network may include a decoder head, which includes an activation function, for example, a linear or nonlinear activation function.

[0058] Thus, the MLL may include at least one encoder branch and at least one decoder branch. The at least one encoder branch provides spatial contraction of the respective representations of the plurality of imaging data sets, and the at least one decoder branch provides spatial expansion of the respective representations of the at least one output image. However, it is not required that the MLL implement both spatial concentration and expansion in all scenarios. In other examples, spatial resolution may not be affected (perhaps with the exception of edge cropping).

[0059] For example, the MLL may include multiple encoder branches, one for each of the multiple imaging datasets. Different encoder branches may be trained to handle different phase contrasts.

[0060] In general, the fusion of the multiple imaging data sets can be implemented by concatenating or stacking the corresponding representations of the multiple imaging data sets at at least one layer of the neural network. This can be the input layer (sometimes referred to as the scenario of early fusion or input fusion) or the hidden layer (sometimes referred to as the scenario of mid-term fusion or late fusion). For mid-term fusion, fusion can even be implemented at the bottleneck (sometimes referred to as bottleneck fusion). In the case of multiple encoder branches, the connection connecting the multiple encoder branches defines the layer where the fusion is implemented. In general, the fusion of different imaging data pairs can be implemented at different locations (e.g., different layers).

[0061] Details regarding processing multiple imaging datasets with different contrast ratios are known from WO 2021 / 198241, the disclosure of which is incorporated herein by reference. Similar techniques can be used in the present disclosure to process multiple imaging datasets with a variety of different phase contrast ratios. Next, details regarding digital phase contrast will be explained.

[0062] At block 3303, at least one output image is obtained from the MLL 3500, and each of the at least one output image depicts the tissue sample 3400 including a corresponding virtual stain. Some example virtual stains are: a virtual H&E (hematoxylin and eosin) stained image of the tissue sample, a virtual staining image of the tissue sample highlighting antibodies such as anti-panCK, anti-CK18, anti-CK7, anti-TTF-1, anti-CK20 / anti-CDX2, and anti-PSA / anti-PSMA or other biomarkers. Other examples are primary IHC markers such as HER2 (ERBB2), ER (estrogen receptor / ESR1), PR (progesterone receptor / PGR); and proliferation markers such as Ki-67 (MKI67).

[0063] Figure 4 Schematically illustrated is a system 70 for acquiring imaging data with digital phase contrast. The system 70 comprises a microscope 90 and a computer 80. The microscope 90 comprises an illumination module 91 , an optical system 92 and a detector module 93.

[0064] The system 70 can be a tabletop optical microscope. The system 70 can be relatively compact and lightweight. This is an advantage of using digital phase contrast (such as TIE phase contrast or DPC).

[0065] The detector module 93 includes one or more cameras for acquiring microscope images.

[0066] The lighting module 91 is configured to provide switchable / reconfigurable oblique illumination of an imaging plane defined along an optical path 94 of the system 70. This means that the illumination angle can be controlled. In addition to controlling the main illumination angle, the angular spectrum, e.g. the width or contribution etc., can optionally also be controlled. For example, multiple lighting configurations with angular spectrums of different widths can be activated. Sometimes only a single illumination direction can be activated (smallest width of the angular spectrum), sometimes multiple illumination directions can be superimposed (larger width of the angular spectrum). The illumination is partially coherent, e.g. using a point source, a collimated laser or a surface light source. In practice, an array of light emitting diodes (LEDs) is used, which exhibit a sufficiently high degree of spatial and temporal coherence.

[0067] The optical system 92 is configured to illuminate the imaging plane and further image the imaging plane onto at least one camera of the detector module 93 .

[0068] The microscope 90 also includes a control module 95 that is configured to control the various components of the microscope. For example, the control module 95 can be implemented using a CPU or a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The control module 95 may include a memory. The control module 95 can be configured to control the illumination module to activate a plurality of tilted illumination configurations. The control module 95 can also be configured to control the detector module 93, and specifically control at least one camera of the detector module 93 to acquire a plurality of images. Optionally, the control module 95 can be configured to control the optical system 92, for example, by moving a sample holder configured to hold a sample or by driving a movable nosepiece holding an objective lens or more generally implementing a plurality of defocus values.

[0069] Also shown is a computer 80, which includes an interface 84 configured to communicate with the microscope 90, specifically the control module 95. For example, a processor 81 (e.g., a CPU, FPGA, or ASIC) of the computer 80 can provide control data to the control module 95 to implement certain control functions, such as triggering image acquisition, triggering activation of certain oblique illumination configurations, triggering implementation of certain defocus values, etc. The processor 81 can also retrieve image data from the microscope 90 via the interface 84 and perform post-processing on the image data. The processor 81 can provide imaging data sets with a variety of different digital phase contrasts to the tissue analyzer 2110.

[0070] The processor 81 is configured to load program code from the memory 83 and execute the program code to perform this technique. In particular, digital post-processing for determining a phase contrast image based on a plurality of light intensity images acquired from the microscope 90 can be performed by the processor 81. The computer 80 can also include a user interface 82, such as a GUI, for outputting the phase contrast image thus determined.

[0071] Computer 80 may implement tissue analyzer 2110. Thus, edge reasoning of MLL may be used to provide a virtual stain.

[0072] Although Figure 4 , a scenario is illustrated in which the computer architecture is split between the control module 95 and the computer 80, but in some scenarios, digital post-processing of the image data may also be performed by the control module 95. Alternatively or additionally, component-level control of the various components of the microscope 90 may also be a task that is at least partially delegated to the computer 80.

[0073] Figure 5 and Figure 6 The experimental hardware used in acquiring differential phase contrast (DPC) phase contrast and TIE phase contrast are shown respectively.

[0074] DPC( Figure 5 ) uses a programmable illumination unit (PIU) 201. It needs to acquire at least three focal plane images under a changing illumination pattern 202 to produce a phase contrast image of the sample. In particular, these variable illumination patterns 202 correspond to different oblique illumination configurations for illuminating the imaging plane 203 at different angles while holding the sample in a fixed position. Each pattern 202 includes activated LEDs in a different asymmetric distribution relative to the optical axis 207 (dashed line and open circle). That is, the corresponding illumination configuration includes multiple illumination directions (defined by the activated LEDs). Using an objective lens 204 and a tubular lens 205 (part of the optical system, refer to Figure 1 : optical system 92) to image the imaging plane 203 onto a pixelated detector 206 (camera). Different LEDs are activated for different variable illumination patterns 202. A subsequent numerical deconvolution routine converts the recorded image into a phase contrast image of the sample, see the following reference: Tian, ​​Lei and Laura Waller. "Quantitative differential phase contrast imaging in an LED array microscope." Optics express 23.9 (2015): 11394-11403.

[0075] Now refer to Figure 6: TIE phase contrast is based on a diffusion equation that relates the axial light intensity derivative to the phase of the sample. The axial light intensity derivative can be approximated by recording at least two images, wherein the imaging plane and the sample are moved axially and close to the focal plane, thereby implementing multiple defocus values ​​311. The illumination unit is ideally fully coherent, such as an on-axis point source 301 or an on-axis collimated laser. After recording images under defocus variation (also called z-stack), sufficient information can be used to solve the basic diffusion equation to obtain the phase information of the sample, see the following document: Streibl, Norbert. "Phase imaging by the transport equation of intensity." Opticscommunications [Optics Communications] 49.1 (1984): 6-10. This includes a numerical deconvolution routine that converts the recorded image into a phase contrast image of the sample. TIE is a diffusion equation that relates the axial light intensity derivative to the phase of the sample. The axial light intensity derivative can be approximated by recording at least two images, wherein the sample is moved axially and close to the focal plane.

[0076] The various techniques are based on the discovery that specifically beneficial input data for MLL can be obtained by combining firstly DPC and secondly TIE phase contrast. This is because the spatial frequencies within the numerical aperture covered by DPC on the one hand and the spatial frequencies within the numerical aperture covered by TIE on the other hand are complementary. Therefore, by fusing in MLL a first imaging dataset with DPC contrast with a second imaging dataset with TIE phase contrast, more comprehensive information is available for the input to the MLL. Since both DPC and TIE phase contrast are digital phase contrast using similar acquisition hardware, as described above, the acquisition process for acquiring multiple imaging datasets can be implemented using the same hardware. More specifically, a desktop optical microscope (refer to Figure 4 ) can be sufficient to acquire high-quality imaging datasets with multiple phase contrasts. This has the advantage of enabling edge inference of MLL at the user's site, at the computational circuitry co-deployed with a desktop optical microscope. Sample exposure to light is also limited, for example, if compared to fluorescence or interferometry techniques.

[0077] Figure 7 The PTFs of different digital phase contrast techniques (DPC, TIE and their combination) and the associated achievable numerical aperture (NA) coverage are illustrated. Example phase images 501-503 are shown for each technique.

[0078] Figure 7 The process of customizing the spatial frequency coverage of d(k) is illustrated. In the following, the quantity It is called cumulative NA coverage.

[0079] Figure 7 The first row illustrates the DPC 551 (refer to Figure 5 Although most DPCs can reach the cumulative NA radius of NAi+NAd ( Figure 7 In the example of FIG501 , the center of the NA coverage exhibits a hole of radius approximately NAd − NAi, resulting in a loss of coverage at low spatial frequencies. This in turn causes the resulting phase reconstruction to exhibit low contrast (as is evident from the phase contrast image 501 ).

[0080] Figure 7 The TIE phase contrast 552 shown in the second row (refer to Figure 6 ) using axial defocus. When evaluating the corresponding phase transfer function ( Figure 7 When the NA coverage is calculated using the DPC (second row, left column), it can be seen that the NA coverage reaches lower values ​​in k-space compared to DPC (simply, the black circle around the center of k-space is smaller). This results in improved contrast in the final phase reconstruction compared to DPC (refer to phase contrast image 502). However, since only a single on-axis point source is used, the illumination NA of a standard TIE phase contrast system is vanishing. Therefore, the NA coverage only reaches the bandwidth of the NAd (simply, the white circle has a finite radius), which is less effective than the achievable lateral resolution (given by λ / [NA]) compared to DPC. i +NA d ] is given, where λ is the wavelength) is disadvantageous.

[0081] Next, Figure 7 The third row illustrates the spatial frequency coverage that can be used to virtually stain MLL when combining two imaging datasets with first DPC and second TIE phase contrast. By fusing the PTFs from both the variable illumination pattern and the defocus, a wider range of spatial frequencies can be covered (the rings in the middle column have a smaller inner hole and a wider radius). The resulting phase reconstruction (right column) is characterized by superior phase contrast compared to DPC while achieving higher resolution compared to TIE phase contrast (reference phase contrast image 503).

[0082] Figure 8Virtual staining results are shown, in which a U-net is trained to convert phase contrast images into H&E stained images. The first imaging dataset is based on TIE digital phase contrast, and the second imaging dataset is based on DPC phase contrast, both using only red LEDs. In addition, red, green, and blue brightfield images of the same sample are acquired. The procedure is repeated for various different regions of interest. The acquired data is then used to train the U-net. Panel a) shows a phase contrast image (TIE phase contrast) obtained from a defocused stack by solving the light intensity transfer equation. The image contains visible artifacts, as evidenced by local inhomogeneities in the displayed grayscale values. The reduction in quality of this TIE phase contrast image (compared to DPC in panel d) is due to a violation of the non-absorbing specimen assumption inherent in TIE. However, the H&E prediction from the TIE phase contrast image in panel c) has a high degree of similarity to the ground truth data shown in panel d), however, closer inspection reveals certain shortcomings. The black arrows point to areas where fine details cannot be reliably predicted by the TIE phase contrast image (only two of the three black dots are visible). In panel d), a DPC phase contrast image is shown. This DPC phase contrast image is used to predict an H&E image, as shown in panel f. Again, this image is very similar to the ground truth (reproduced in panel e) for ease of comparison). The black arrows show the same area as previously in panel c), but now the three black dots are correctly predicted, as judged by comparison with the ground truth. Thus, the black arrows highlight the areas where the DPC phase contrast image produces superior predictions compared to the TIE phase contrast image. Conversely, areas are also identified where the TIE phase contrast image produces superior H&E predictions compared to the DPC phase contrast image. This is the case with the red tissue areas indicated by the white arrows. Here, the ground truth and the predictions from the TIE phase contrast image show homogeneous red areas (compare white arrows in panels b and c), while the H&E predictions from DPC incorrectly predict inhomogeneous tissue areas (compare panels e and f).

[0083] Figure 8 The example shown in Figure 3 shows that DPC phase contrast images and TIE phase contrast images have different prediction performances for virtual staining applications. Next, MLL that fuses two digital phase contrast imaging modalities is evaluated.

[0084] Figure 9The results of training the virtual staining neural network with both DPC phase contrast source images and TIE phase contrast source images are shown. Again comparing the black arrows indicating the ability to predict detailed structure and the white arrows indicating the ability to predict uniform background, it can be seen that providing both DPC phase contrast images and TIE phase contrast images (panel d) as input to the virtual staining MLL is better than using only a single source image (TIE in panel b or DPC in panel c, but not both combined).

[0085] In summary, a multimodal phase contrast image is provided as input for virtual staining. This improves the robustness of the predicted image. The above mainly discloses the combination of a first imaging dataset with DPC and a second imaging dataset with TIE phase contrast; however, various other combinations of imaging datasets with phase contrast are possible. Next, a possible example is disclosed: multiple imaging datasets with classical phase contrast can be combined, which classical phase contrast includes, for example, Zernike phase contrast, differential interference contrast (DIC), and Jamin-Lebedeff interference contrast. Alternatively or additionally, in addition to DPC phase contrast and TIE phase contrast, alternative digital phase contrast methods can also be used. Examples include shearing interferometry, on-axis and off-axis holography, and phase shifting interferometry. As an alternative or supplement to using different phase contrast imaging modalities, a single phase contrast imaging modality can be used when other optical parameters are varied. The first example involves changes in color / wavelength. For example, multiple colors can be used in a single phase contrast experimental setup, for example, DPC uses an LED array with red, green and / or blue illumination channels (and similar for the aforementioned phase contrast methods). Similarly, multiple spectral bands in the infrared or UV can be used as input to enhance predictions via the MLL. A second example involves changes in polarization. This involves changing the principal axis or linear polarization and / or circular polarization of light upstream and / or downstream of the sample.

[0086] Although the invention has been shown and described with respect to certain preferred embodiments, equivalents and modifications will occur to others skilled in the art upon the reading and understanding of the specification. The present invention includes all such equivalents and modifications, and is limited only by the scope of the appended claims.

Claims

1. A method for virtually staining a tissue sample (2105), the method comprising: - obtaining (3301) a plurality of imaging data sets (801) depicting the tissue sample (2105), the plurality of imaging data sets (801) having a plurality of different phase contrasts, - fusing and processing (3302) the plurality of imaging data sets (801) in machine learning logic, and - obtaining (3303) at least one output image (802, 803) from the machine learning logic, each of the at least one output image (802, 803) depicting the tissue sample (2105) including a corresponding virtual stain.

2. The method according to claim 1, in, The plurality of different phase contrasts include at least two different phase contrasts acquired using a plurality of different phase contrast imaging modalities.

3. The method according to claim 2, in, The plurality of different phase contrast imaging modalities include a first digital phase contrast imaging modality and a second digital phase contrast imaging modality, The first digital phase contrast imaging modality is a light intensity transfer equation digital phase contrast imaging modality. The second digital phase contrast imaging modality is a differential digital phase contrast imaging modality.

4. The method according to claim 2 or 3, in, The plurality of different phase contrast imaging modalities comprises at least one of: Zernike phase contrast, Jamin-Lebedeff interferometric phase contrast, shearing interferometry phase contrast, on-axis holographic phase contrast, off-axis holographic phase contrast or phase-shifting interferometry phase contrast.

5. The method according to any one of the preceding claims, in, The plurality of different phase contrasts include at least two different phase contrasts acquired using the same phase contrast imaging modality under different imaging settings.

6. The method according to claim 5, in, The at least two different phase contrasts are acquired at a plurality of different wavelengths using a digital phase contrast imaging modality.

7. The method according to claim 5 or 6, in, The at least two different phase contrasts are acquired at a plurality of different polarizations using a digital phase contrast imaging modality.

8. A computing device (80, 2110), comprising at least one processor (81) and a memory (83), wherein: The at least one processor is configured to load a program code from the memory and execute the program code, wherein the at least one processor performs the following steps when executing the program code: - obtaining (3301) a plurality of imaging data sets (801) depicting the tissue sample (2105), the plurality of imaging data sets (801) having a plurality of different phase contrasts, - fusing and processing (3302) the plurality of imaging data sets (801) in machine learning logic, and - obtaining (3303) at least one output image (802, 803) from the machine learning logic, each of the at least one output image (802, 803) depicting the tissue sample (2105) including a corresponding virtual stain.

9. The computing device of claim 8, in, The computing device is coupled to an optical imaging system (70), the optical imaging system being configured to acquire the plurality of imaging data sets (801), The processor is configured to load the plurality of imaging data sets from an optical imaging device (801).

10. The computing device of claim 8 or 9, in, The at least one processor is configured to perform the method of any one of claims 1 to 7.

Images