Virtual staining based on multiple sets of imaging data with multiple phase contrasts

Abstract

Various examples of the disclosure relate to techniques for the virtual staining of a tissue sample. One or more output images exhibiting virtual staining are determined. The techniques aim to achieve a more robust process. To this end, multiple sets of imaging data serve as an input to the digital process, wherein the multiple sets of imaging data exhibit multiple different phase contrasts.

Description

TECHNICAL FIELD

[0001] Various embodiments relate to techniques for virtual coloring using machine learning logic. Various examples specifically relate to processing multiple sets of imaging data with different phase contrasts. BACKGROUND

[0002] Histopathology is an important tool in the diagnosis of disease. Histopathology refers to the optical examination of tissue samples. This facilitates the diagnosis of cells in the tissue sample.

[0003] Histopathological examination typically begins with surgery, biopsy, or autopsy to obtain the tissue to be examined. The tissue can be processed to remove water and prevent deterioration. The processed sample can then be embedded in a wax block. Thin sections can be cut from the wax block. These thin sections can then be referred to as tissue samples.

[0004] The tissue samples can be examined by a histopathologist under a microscope. The tissue samples can be stained with a chemical stain using an appropriate laboratory staining procedure to facilitate examination of the tissue sample. In particular, chemical stains can reveal cellular components that are very difficult to observe in the unstained tissue sample. In addition, chemical stains can provide contrast. The chemical stains can highlight one or more biomarkers or predefined structures in the tissue sample.

[0005] The most commonly used chemical stain in histopathology is a combination of hematoxylin and eosin (H&E). Hematoxylin is used to stain cell nuclei blue, while eosin stains cytoplasm and the extracellular connective tissue matrix pink. There are hundreds of other techniques that have been used to selectively stain cells. Recently, antibodies have been used to stain specific proteins, lipids, and carbohydrates. This technique, called immunohistochemistry, has greatly improved the ability to specifically identify cell categories under a microscope. Staining with an H&E stain can be considered the usual gold standard for histopathological diagnosis.

[0006] By staining tissue samples with chemical dyes, structures / tissue sections of the tissue sample that would otherwise be nearly transparent and indistinguishable become visible to the human eye. This allows pathologists and researchers to examine the tissue sample under a microscope or with a digital brightfield equivalent image and assess tissue morphology (structure) or examine for the presence or prevalence of specific cell types, structures, or even microorganisms, such as bacteria.

[0007] Preferably, multiple chemical stains are used to fully assess the pathological case. Typically, only one chemical stain can be applied to a tissue sample. If multiple chemical stains are required for diagnosis, multiple tissue samples must be prepared. Furthermore, different chemical stains may require different staining protocols. The known chemical staining techniques are therefore labor-intensive and cost-intensive.

[0008] WO 2019 / 154987 A1 discloses a method for providing a virtually stained image resembling a typical image of a tissue sample stained with a conventional chemical stain using machine learning logic. Virtual staining techniques bypass typically labor-intensive and expensive histological staining techniques and could be used as a blueprint for virtual staining of tissue images acquired with other label-free imaging modalities. Virtual staining pathways could be used for microguidance of molecular analysis at the unstained tissue level by locally identifying regions of interest based on virtual staining and using this information to guide subsequent analysis of the tissue, for example, microimmunohistochemistry or sequencing.This type of virtual microguidance on an unlabeled tissue sample could facilitate high-throughput identification of disease subtypes and the development of tailored therapies for patients.

[0009] Such state-of-the-art techniques suffer from certain limitations. In particular, the accuracy of the virtual coloring may be limited. The flexibility in selecting different virtual colorings may be limited.

[0010] To mitigate these limitations, WO 2021 / 198241 discloses a virtual staining determined based on multiple sets of imaging data depicting a tissue sample and acquired using multiple imaging modalities. The multiple imaging modalities are selected from the group comprising: hyperspectral microscopic imaging; fluorescence imaging; autofluorescence imaging; light-sheet microscopy; digital phase contrast; and Raman spectroscopy.

[0011] Although these techniques offer greater accuracy for virtual staining, techniques such as hyperspectral microscopy, fluorescence imaging, light-sheet microscopy, and Raman spectroscopy are relatively complex and time-consuming. Furthermore, the required light intensity and / or light dose can be significant, posing a risk of sample damage. SUMMARY

[0012] Accordingly, there is a need for advanced techniques for virtual staining of a tissue sample. In particular, there is a need to determine an output image that shows the tissue sample with precise virtual staining. Robust prediction of virtual staining is required.

[0013] This need is met by the features of the independent claims. The features of the dependent claims define embodiments.

[0014] A method for virtually staining a tissue sample is disclosed. The method comprises obtaining multiple sets of imaging data depicting a tissue sample. The multiple sets of imaging data have multiple different phase contrasts. The method also comprises fusing and processing the multiple sets of imaging data in machine learning logic. The method further comprises obtaining, from the machine learning logic, at least one output image, each of the at least one output image depicting the tissue sample including a respective virtual stain.

[0015] Disclosed is a computer device comprising at least one processor and a memory. The at least one processor can load program code from the memory and execute the program code. Execution of the program code causes the at least one processor to perform this method as disclosed above. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 schematically illustrates a workflow for the virtual staining of a tissue sample. Fig. Figure 2 schematically illustrates images with virtual coloring according to various examples. Fig. 3 is a flowchart of a method according to various examples. Fig. Figure 4 illustrates a system for determining imaging data using digital phase contrast. Fig. Figure 5 schematically illustrates a digital differential phase contrast according to various examples. Fig. Figure 6 schematically illustrates a digital intensity transport phase contrast according to various examples. Fig. Figure 7 schematically illustrates the spatial frequency coverage of the digital differential phase contrast and the digital intensity transport phase contrast. Fig. 8 illustrates virtual coloring according to examples. Fig. 9 illustrates virtual coloring according to examples. DETAILED DESCRIPTION

[0016] Some examples of the present disclosure generally provide for multiple circuits or other electrical devices. All references to the circuits and other electrical devices, and the functionality provided thereby, are not intended to be limited to including only what is illustrated and described herein. Although specific labels may be assigned to the various circuits or other electrical devices disclosed, such labels are not intended to limit the scope of operation for the circuits and other electrical devices. Such circuits and other electrical devices may be combined and / or separated from one another in any manner based on the particular type of electrical implementation contemplated.It should be understood that any circuit or other electrical device disclosed herein may include any number of microcontrollers, machine learning-specific hardware, e.g., a graphics processing unit (GPU) and / or a tensor processing unit (TPU), integrated circuits, memory devices (e.g., FLASH, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other suitable variations thereof), and software that cooperate with one another to perform any operation(s) disclosed herein. Furthermore, any one or more of the electrical devices may be configured to execute a set of program code embodied in a non-transitory computer-readable medium programmed to perform any number of the functions as disclosed.

[0017] Embodiments of the invention will be described in detail below with reference to the accompanying drawings. It should be understood that the following description of embodiments is not to be construed in a limiting sense. The scope of the invention is not intended to be limited by the embodiments described below or by the drawings, which are for illustrative purposes only.

[0018] The drawings are to be considered schematic representations, and elements illustrated in the drawings are not necessarily shown to scale. Rather, the various elements are represented in such a way that their function and general purpose will be apparent to one skilled in the art. Any connection or coupling between functional blocks, devices, components, or other physical or functional entities shown in the drawings or described herein may also be implemented through an indirect connection or coupling. Coupling between components may also be established via a wireless connection. Functional blocks may be implemented in hardware, firmware, software, or a combination thereof.

[0019] Hereinafter, imaging techniques for a sample are disclosed. When light interacts with a sample of interest (specimen), such as biological tissue, three primary contrast mechanisms can be used for image formation. First, the sample can attenuate the incoming light due to absorption. Second, the sample can deform an incoming optical wavefront, thereby imposing phase contrast. Third, the light illuminating the sample can be inelastically scattered by fluorescence from the sample itself (autofluorescence) or by chemical labels added to the sample.

[0020] For thin biological samples, such as tissue sections or adherent cell cultures, both absorption and autofluorescence effects are typically relatively weak. Therefore, biologists often resort to one of the following two light microscopy imaging modalities to examine structural information: (1) Phase-contrast microscopy. This contrast modality allows the sample to be preserved in its native state. Hardware and digital phase contrast techniques are well known. (2) Chromogenic immunohistochemical stains or fluorescent markers can be used to chemically alter the contrast induced by the sample in the incident light. This contrast modality alters the chemical composition of the sample.

[0021] Various techniques are based on the realization that phase contrast—especially digital phase contrast—is relatively easy to achieve among these two imaging modalities, but deriving chemically specific information from it has been found to be relatively challenging. Fluorescent labels can be used to mark specific functional groups. However, both staining and fluorescent labeling have the disadvantage of requiring time-consuming sample preparation protocols and, in some cases, irreversibly altering the native state of the sample.

[0022] Recently, machine learning techniques known as virtual staining or in silico staining have been reported, enabling the digital transfer of one imaging modality to another. For a recent review of the state of the art, see Kreiss, Lucas, et al., "Digital staining in optical microscopy using deep learning—a review." arXiv preprint arXiv:2303.08140 (2023).

[0023] Various techniques described herein generally relate to the virtual coloring of a tissue sample using trained machine learning logic (MLL). The MLL can be implemented, for example, by a support vector machine or a deep neural network that has at least one encoder branch and at least one decoder branch. Examples include a U-Net, see Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. “U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015. The MLL can be trained with a cyclic generating adversarial network, see, for example, Zhu, Jun-Yan, et al. “Unpaired image-to-image translation using cycle-consistent adversarial networks.” Proceedings of the IEEE international conference on computer vision. 2017.This architecture includes a forward cycle and a reverse cycle, with the forward cycle and the reverse cycle each having a generator MLL and a discriminator MLL. The generator MLLs of the forward cycle and the reverse cycle are each implemented using the MLL.

[0024] In particular, according to various examples, multiple sets of imaging data can be fused and processed using MLL. This is referred to as a multiple-input scenario. An output image is provided. The output image shows the tissue sample including a virtual stain, i.e., the output image may have a similar appearance to respective images showing the tissue sample including a corresponding chemical stain. Accordingly, the virtual stain may correspond to a chemical stain of a tissue sample stained using a staining method in the laboratory.

[0025] For example, MLL can generate virtual H&E (hematoxylin and eosin) stained images of the tissue sample and / or virtual stained images of the tissue sample highlighting HER2 (human epidermal growth factor receptor 2) proteins and / or ERBB2 (Erb-B2 receptor tyrosine kinase 2) genes.

[0026] Another example would involve virtual fluorescence staining. In life science applications, for example, images of cells—e.g., arranged ex vivo in a multiwell plate—are acquired using transmitted light microscopy. A reflected light microscope can also be used, e.g., in an endoscope or as a surgical microscope. It is then possible to selectively stain specific cell organelles, e.g., the nucleus, ribosomes, the endoplasmic reticulum, the Golgi apparatus, chloroplasts, or mitochondria. A fluorophore (or fluorochrome, similar to a chromophore) is a fluorescent chemical compound that can re-emit light when excited by light. Fluorophores can be used to provide a fluorescent chemical stain. By using different fluorophores, different chemical stains can be achieved.For example, a Hoechst stain is a fluorescent dye that can be used to stain DNA. Other fluorophores include 5-aminolevulinic acid (5-ALA), fluorescein, and indocyanine green (ICG), which can even be used in vivo. Fluorescence can be selectively excited using light at specific wavelengths; the fluorophores then emit light at a different wavelength. Fluorescence microscopes use specific light sources. It has been observed that illumination using light to excite fluorescence can damage the sample; this is avoided by providing virtual fluorescence staining. Virtual fluorescence staining mimics chemical fluorescence staining without exposing the tissue to specific excitation light.

[0027] According to the examples, virtual staining is facilitated by multiple phase contrasts, e.g., hardware and / or digital phase contrasts. Various techniques disclosed herein for robustly transferring multiple sets of imaging data from a tissue sample have multiple different phase contrasts to other modalities, including fluorescence and immunohistochemical staining.

[0028] The choice between different phase contrasts often depends on the tissue sample being examined, and it can be advantageous to acquire multiple sets of imaging data using multiple phase-contrast imaging modalities. Multiple phase contrasts are combined to make image transfer performance more robust during virtual staining. More comprehensive information regarding the sample may be available for MLL when using multiple phase contrasts. Virtual staining can be predicted more accurately. Greater flexibility may be available in predicting different types of virtual staining.

[0029] Fig. 1 illustrates aspects related to a workflow for generating images showing a tissue sample including a stain, e.g., a chemical stain or a virtual stain. Fig. Figure 1 schematically illustrates an example of a histopathology workflow. As explained above, virtual staining can be applied to other applications besides histopathology. Different workflows for generating images may then be applicable. For fluorescence imaging of cells, for example, tissue samples, including cell samples, can be acquired in other ways and imaged in a respective microscope. In vivo imaging using an endoscope would also be possible for generating imaging data from tissue samples.

[0030] As it is in Fig. As can be seen in Figure 1, tissue 2102 for histopathology can be removed from a living being 2101 by surgery, biopsy, or autopsy. After some processing steps to remove water and prevent deterioration, the tissue 2102 can be embedded in a wax block 2103. A plurality of sections 2104 can be obtained from the block 2103 for further examination. A section of this plurality of sections 2104 can also be referred to as a tissue sample 2105. Corresponding tissue samples 2105 can be obtained from adjacent sections.

[0031] As mentioned above, the tissue could also include cell samples or in vivo inspection, e.g., using a surgical microscope or an endoscope.

[0032] Prior to examining the tissue sample 2105, a chemical stain may optionally be applied to the tissue sample 2105 using a laboratory staining procedure to obtain a chemically stained tissue sample 2106. In some examples, the tissue sample 2105 may also be examined directly (dashed arrow in Fig. 1). A chemically stained tissue sample 2106 can facilitate analysis. In particular, chemical stains can reveal cellular components or generally well-defined cell structures that are difficult to observe in the unstained tissue sample 2105. Furthermore, chemical stains can provide increased contrast.

[0033] Application of chemical staining may involve a priori transfection or direct application of a fluorophore such as 5-ALA.

[0034] Traditionally, the tissue sample 2105 or 2106 is examined by a specialist using a bright-field microscope 2107.

[0035] It has become more common to use image acquisition systems 2108 designed to acquire digital image data of the tissue sample 2105 or the chemically stained tissue sample 2106 using one or more imaging modalities. Using different imaging modalities can facilitate the acquisition of imaging data 2109—e.g., 1D, 2D, or 3D imaging data—of the tissue sample 2105.

[0036] The imaging data 2109 can be processed in a tissue analyzer 2110. The tissue analyzer 2110 can be implemented by a computer and / or through cloud processing on a server. The tissue analyzer 2110 can include a memory circuit 2111 for storing the digital image data 2109 and / or program code, and it can include a circuit 2112 for processing the digital image data 2109—e.g., after loading the program code. The tissue analyzer 2110 can process the imaging data 2109 to provide one or more output images 2113 that can be displayed on a display 2114 for analysis by an examiner. For example, multiple output images 2113 can be provided showing the tissue sample 2105, 2106 including various virtual stainings.The tissue analyzer 2110 may include various types of trained or untrained machine learning logic (details regarding the machine learning logic are described below) for examining the unstained tissue sample 2105 and / or the chemically stained tissue sample 2106 (the circuitry 2112 may execute the machine learning logic). The output images 2113 may show the tissue sample 2105 with one or more virtual stains. The image acquisition system 2108 may be used to provide training data and / or reference images as ground truth for training the machine learning logic.

[0037] More generally, the tissue analyzer 2110 includes circuitry 2112, which may include a CPU and / or a GPU and / or a TPU. The circuitry 2112 may load program code from the memory 2111. The circuitry 2112 may execute the program code. Upon executing the program code, the circuitry 2112 may perform one or more of the following logical operations, as described throughout the disclosure: Obtaining imaging data, e.g., via an input / output (I / O) interface of the tissue analyzer 2110 or by loading the imaging data from memory; Preprocessing the imaging data, e.g., to determine digital phase contrast; Virtually staining the tissue sample revealed by the imaging data; Executing machine learning logic to process the imaging data (inference); Obtaining at least one output image from the machine learning logic / upon executing the machine learning logic, e.g.,for outputting the at least one output image via the I / O interface; setting parameters or hyperparameters of the machine learning logic when training the machine learning logic; training the machine learning logic; etc.

[0038] Fig. 2 schematically illustrates images 801-803 showing a tissue sample. Image 801 shows the tissue sample that has no chemical or virtual staining. For example, image 801 may have digital phase contrast. Alternatively, image 802 shows the tissue sample that has chemical or virtual staining. Image 803 also shows the tissue sample, including chemical virtual staining, wherein the chemical virtual staining of the tissue sample shown by image 803 differs from the chemical or virtual staining of the tissue sample shown by image 802; different structures or biomarkers are highlighted (solid black areas in Fig. 2).

[0039] Fig. 3 is a flowchart of a method 3300 according to various examples. For example, the method 3300 may be Fig. 3 by at least one circuit - e.g., a CPU and / or a GPU and / or a TPU - when program code is loaded from a non-volatile memory. The method of Fig. 3 can be performed by the tissue analyzer 2110. The method of Fig. 3 facilitates the virtual staining of a tissue sample.

[0040] At block 3301, multiple sets of imaging data of a tissue sample are obtained (e.g., loaded from a memory or obtained via an input interface from a data acquisition unit), and the multiple sets of imaging data include multiple phase contrasts. Each set of imaging data may include multiple instances of imaging data, e.g., multiple images (see Fig. 2: Image 801), which were taken at different positions of the sample (e.g. for stitching) and / or at different times.

[0041] The multiple sets of imaging data may have different phase contrasts because they were acquired using different phase-contrast imaging modalities. For example, at least one of the multiple sets of imaging data may have been acquired using a digital phase-contrast imaging modality. Examples include digital differential phase contrast (DPC) and transport intensity (TIE) phase contrast. Other digital phase-contrast imaging modalities include inline holographic phase contrast, off-axis holographic phase contrast, or phase-shift interferometry phase contrast. It would also be possible for at least one of the multiple sets of imaging data to have been acquired using a non-digital ("classical" hardware-based) phase-contrast imaging modality.The multiple phase contrast imaging modalities may include at least one of a Zernike phase contrast, a Jamin-Lebedeff interference phase contrast, and a shear interferometry phase contrast.

[0042] It is not necessary in all scenarios that the multiple sets of imaging data were acquired with different phase contrasts using different phase-contrast imaging modalities. For example, at least two of the multiple sets of imaging data may have been acquired using the same phase-contrast imaging modality but with different imaging settings. Examples would include a first set of the multiple sets of imaging data acquired with a particular digital phase-contrast imaging modality, such as DPC or TIE phase contrast, at a first wavelength (e.g., red, blue, yellow, infrared, or ultraviolet) of light; a second set of the multiple sets of imaging data acquired with that particular digital phase-contrast imaging modality at a second wavelength of light that is different from the first wavelength.Alternatively, or in addition to using multiple wavelengths, it would also be possible to use multiple polarizations, e.g., left circular and right circular polarized light.

[0043] The tissue sample can be a cancer tissue sample taken from a patient, a tissue sample from other animals or plants.

[0044] The procedure 3300 of Fig. 3 optionally includes preprocessing after obtaining the multiple sets of imaging data, such as one or a combination of the following processing techniques: noise filtering; registration between imaging data from different sets of imaging data, e.g., arbitrary pairs of the sets; resizing of the imaging data; etc.

[0045] Imaging data comprising one or more images with digital phase contrast are also obtained by preprocessing raw images. For DPC, for example, multiple raw images of the tissue sample acquired with different angular illumination settings are combined. Similarly, for TIE, multiple raw images of the tissue sample acquired with different defocus settings are combined. It would be possible for the method 3300 of Fig. 3 this preprocessing involves obtaining one or more sets of imaging data with several different phase contrasts.

[0046] At block 3302, the multiple sets of imaging data are fused and processed by an MLL. The MLL was trained using supervised learning, semi-supervised learning, or unsupervised learning.

[0047] In general, various implementations of MLL are conceivable. In one example, a deep neural network can be used. For example, a U-Net implementation is possible. See Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

[0048] More generally, the deep neural network may have multiple hidden layers. The deep neural network may have an input layer and an output layer. The hidden layers are arranged between the input layer and the output layer. There may be spatial contraction and spatial expansion, implemented by one or more encoder branches and one or more decoder branches, respectively. That is, the xy resolution of respective representations of the imaging data and the output images may be decreased (increased) from layer to layer along the one or more encoder branches (decoder branches). At the same time, feature channels may increase and decrease along the one or more encoder branches and one or more decoder branches, respectively. The one or more encoder branches and the one or more decoder branches are connected via a bottleneck.At the output layer or layers, the deep neural network may have decoder heads containing an activation function, e.g., a linear or non-linear activation function.

[0049] Thus, the MLL may include at least one encoder branch and at least one decoder branch. The at least one encoder branch provides spatial contraction of respective representations of the multiple sets of imaging data, and the at least one decoder branch provides spatial expansion of respective representations of the at least one output image. However, it is not necessary in all scenarios for the MLL to implement spatial contraction and expansion. In other examples, spatial resolution (possibly with the exception of edge clipping) may not be compromised.

[0050] For example, it would be possible for the MLL to have multiple encoder branches, one for each of the multiple sets of imaging data. Different encoder branches can be trained to process different phase contrasts.

[0051] In general, the fusion of the multiple sets of imaging data can be implemented by concatenating or stacking the respective representations of the multiple sets of imaging data at at least one layer of the neural network. This can be an input layer (a scenario sometimes referred to as early fusion or input fusion) or a hidden layer (a scenario sometimes referred to as middle fusion or late fusion). For middle fusion, it would even be possible for the fusion to be implemented at the bottleneck (sometimes referred to as bottleneck fusion). When multiple encoder branches are present, the connection connecting the multiple encoder branches defines the layer where the fusion is implemented. In general, the fusion of different pairs of imaging data can be implemented at different positions, e.g., different layers.

[0052] Details regarding the processing of multiple sets of imaging data with different contrasts are known from WO 2021 / 198241, the disclosure of which is incorporated herein by reference. Similar techniques can be used in the present disclosure to process the multiple sets of imaging data with multiple different phase contrasts. Next, details regarding digital phase contrasts will be explained.

[0053] At block 3303, at least one output image is obtained from the MLL 3500, and each of the at least one output image shows the tissue sample 3400 including a respective virtual stain. Some examples of virtual stains are: virtually stained images of the tissue sample with H&E (hematoxylin and eosin), virtually stained images of the tissue sample highlighting antibodies such as anti-panCK, anti-CK18, anti-CK7, anti-TTF-1, anti-CK20 / anti-CDX2, and anti-PSA / anti-PSMA or other biomarkers. Other examples include primary IHC markers, e.g., HER2 (ERBB2), ER (estrogen receptor / ESR1), PR (progesterone receptor / PGR); and proliferation markers, e.g., Ki-67 (MKI67).

[0054] Fig. Figure 4 schematically illustrates a system 70 for acquiring imaging data using digital phase contrast. The system 70 includes a microscope 90 and a computer 80. The microscope 90 has an illumination module 91, an optical system 92, and a detector module 93.

[0055] System 70 may be an optical tabletop microscope. System 70 may be relatively compact and lightweight. This is an advantage of using a digital phase contrast system such as a TIE phase contrast system or DPC.

[0056] The detector module 93 has one or more cameras for capturing microscopic images.

[0057] The illumination module 91 is designed to provide switchable / reconfigurable angular illumination of an imaging plane defined along the path of the light 94 of the system 70. This means that the illumination angle can be controlled. Beyond controlling the main illumination angle, it would optionally also be possible to control the angular spectrum, e.g., the width of the contributions, etc. For example, it would be possible to activate multiple illumination configurations having angular spectra with different widths. Sometimes only a single illumination direction can be activated (minimum width of the angular spectrum), sometimes multiple illumination directions can be superimposed (greater width of the angular spectrum). The illumination is partially coherent, e.g., using a point source, a collimated laser, or an area light source.In practice, an array of light-emitting diodes (LEDs) is used that exhibit a sufficiently high degree of spatial and temporal coherence.

[0058] The optical system 92 is designed to illuminate the imaging plane and further to image the imaging plane on the at least one camera of the detector module 93.

[0059] The microscope 90 also includes a control module 95 configured to control the various components of the microscope. For example, the control module 95 may be implemented using a CPU, a Field Programmable Gated Array (FPGA), or an Application Specific Integrated Circuit (ASIC). The control module 95 may include memory. The control module 95 may be configured to control the illumination module to activate the multiple angular illumination configurations. The control module 95 may also be configured to control the detector module 93, and in particular, the at least one camera of the detector module 93 to capture multiple images. Optionally, the control module 95 may be configured to control the optical system 92, e.g., by moving a sample holder configured to hold a sample, or by driving a movable nosepiece holding an objective, or more generally by implementing multiple defocus values.

[0060] Also shown is the computer 80, which has an interface 84 configured for communication with the microscope 90, in particular the control module 95. For example, a processor 81 of the computer 80—e.g., a CPU, an FPGA, or an ASIC—can provide control data to the control module 95 for implementing certain control functionalities, e.g., triggering image acquisition, triggering activation of certain angular illumination configurations, triggering implementation of certain defocus values, etc. The processor 81 can also retrieve image data from the microscope 90 via the interface 84 and subject the image data to post-processing. The processor 81 can provide sets of imaging data with multiple different digital phase contrasts for the tissue analyzer 2110.

[0061] The processor 81 is configured to load program code from the memory 83 and execute the program code to perform these techniques.

[0062] In particular, digital post-processing for determining a phase-contrast image based on multiple intensity images retrieved from microscope 90 may be performed by processor 81. Computer 80 may also include a user interface 82, e.g., a GPU, for outputting phase-contrast images thus determined.

[0063] The computer 80 may implement the tissue analyzer 2110. This enables edge inference of the MLL to provide virtual staining.

[0064] While in Fig. 4 shows a scenario in which the computer architecture is divided between the control module 95 and the computer 80. In some scenarios, it would also be possible for the control module 95 to perform digital post-processing of image data. Alternatively or additionally, control of the various components of the microscope 90 at the component level may also be a task that is at least partially delegated to the computer 80.

[0065] Fig. 5 and Fig. 6 show the experimental hardware used in the acquisition of a differential phase contrast (DPC) phase contrast and a TIE phase contrast, respectively.

[0066] DPC ( Fig. 5) uses a programmable illumination unit (PIU) 201. This requires the acquisition of at least three images in the focal plane under different illumination patterns 202 to generate a phase-contrast image of a sample. In particular, these variable illumination patterns 202 correspond to different angular illumination configurations for illuminating the imaging plane 203 at different angles while the sample is held in a fixed position. Each pattern 202 has activated LEDs in different asymmetric distributions with respect to the optical axis 207 (dashed line and open circle). This means that the respective illumination configuration has multiple illumination directions (defined by the activated LEDs). An objective lens 204 and a tube lens 205 (part of the optical system, see Fig. 1; optical system 92) are used to image the imaging plane 203 onto a pixelated detector 206 (camera). Different LEDs are activated for the different variable illumination patterns 202. A subsequent numerical deconvolution routine converts the recorded images into a phase-contrast image of the sample. See Tian, ​​Lei, and Laura Waller. "Quantitative differential phase contrast imaging in an LED array microscope." Optics express 23.9 (2015): 11394-11403.

[0067] Now referring to Fig. 6: TIE phase contrast is based on a diffusion equation that relates an axial intensity derivative to the phase of the sample. The axial intensity derivative can be approximated by recording at least two images, shifting the imaging plane and the sample axially and close to the focal plane, thus implementing multiple defocus values ​​311. The illumination unit is ideally fully coherent, such as an on-axis point source 301 or an on-axis collimated laser. After recording images under defocus variation (also known as Z-stack), sufficient information is available to solve the underlying diffusion equation to yield the phase information of the sample, see Streibl, Norbert. “Phase imaging by the transport equation of intensity.” Optics communications 49.1 (1984): 6-10.This includes a numerical deconvolution routine that converts the acquired images into a phase-contrast image of the sample. TIE is a diffusion equation that relates an axial intensity derivative to the phase of the sample. The axial intensity derivative can be approximated by acquiring at least two images, translating the sample axially and close to the focal plane.

[0068] Various techniques are based on the realization that certain advantageous input data for the MLL can be obtained by combining, first, DPC with, second, TIE phase contrast. This is because the spatial frequencies with the numerical aperture covered by DPC on the one hand and TIE on the other hand are complementary. By fusing in the MLL the first set of imaging data with the DPC contrast with the second set of imaging data with the TIE phase contrast, more comprehensive information is therefore available for input to the MLL. Since both DPC and TIE phase contrast are digital phase contrasts using similar acquisition hardware—as explained above—the acquisition method for acquiring the multiple sets of imaging data can be implemented with one and the same hardware. In particular, an optical tabletop microscope (see Fig. 4) be sufficient for acquiring high-quality sets of imaging data with multiple phase contrasts. This has the advantage of enabling edge inference of the MLL with a computer circuit used at a user location in conjunction with the optical tabletop microscope. The exposure of the sample to light is also limited, e.g., compared to fluorescence or interferometry techniques.

[0069] Fig. Figure 7 shows the PTFs for various digital phase contrast techniques—DPC, TIE, and a combination thereof—and the associated achievable numerical aperture (NA) coverage. An example of a phase image 501-503 is shown for each technique.

[0070] Fig. Figure 7 shows the procedure for adapting the spatial frequency coverage of d (k). The following describes the set d(k) referred to as cumulative NA coverage.

[0071] Fig. 7, first row, shows DPC 551 (see Fig. 5) under unmatched illumination condition (numerical aperture of the illumination part of the optical system Nai=0.3) and numerical aperture of the detector part of the optical system NAd=0.6). While DPC is basically capable of achieving a cumulative NA radius of NAi+NAd (radius of the dashed line in the middle of the first row of Fig. 7), the center of the NA coverage exhibits a hole with a radius of approximately NAd-NAi, resulting in a loss of the narrow spatial frequency coverage. This, in turn, causes the resulting phase reconstruction to exhibit low contrast (as seen in phase-contrast image 501).

[0072] TIE Phase Contrast 552 - shown in Fig. 7, second row (see Fig. 6) - uses axial defocusing. When evaluating the corresponding phase transfer function ( Fig. 7, second row, left column), it can be seen that the NA coverage achieves lower values ​​in k-space compared to DPC (simply put, the black circle around the center of k-space is smaller). This leads to improved contrast in the final phase reconstruction compared to DPC (see phase contrast image 502). However, since only a single on-axis point source is used, standard TIE phase contrast systems have a vanishing illumination NA. The NA coverage therefore only reaches a bandwidth of NAd (simply put, the white donut has a limited radius), which represents a disadvantage compared to DPC in terms of the achievable lateral resolution (which is determined by λ / [NA i + NA d ], where λ is the wavelength).

[0073] Next, Fig. Figure 7, third row, shows the spatial frequency coverage available for virtual staining MLL when combining two sets of imaging data, firstly, DPC and secondly, TIE phase contrast. By fusing the PTFs from both variable illumination patterns and defocus, a wider range of spatial frequencies can be covered (the donut in the middle column has a smaller inner hole and a wider radius). The resulting phase reconstruction (right column) shows superior phase contrast compared to DPC while achieving higher resolution compared to TIE phase contrast (see phase contrast image 503).

[0074] Fig. Figure 8 shows a virtual staining result, where a U-Net was trained to map phase-contrast images to H&E-stained images. A first imaging dataset is based on digital TIE phase contrast, and a second imaging set is based on DPC phase contrast, both using only a red LED. Additionally, red, green, and blue bright-field images of the same sample were acquired. This procedure was repeated for several different regions of interest. The acquired data was then used to train the U-Net. Panel a) shows a phase-contrast image obtained from a defocused stack by solving the intensity transport equation (TIE phase contrast). This image contains visible artifacts, as evidenced by the local inhomogeneities in the displayed gray values.The lower quality of this TIE phase-contrast image (compared to DPC in panel d) is caused by a violation of the non-absorbing sample assumption inherent in TIE. Nevertheless, the H&E prediction from the TIE phase-contrast image in panel c) shows high similarity to the ground-truth data shown in panel b), but closer inspection reveals certain deficiencies. The black arrow points to a region where fine details cannot be reliably predicted with the TIE phase-contrast image (only two of three black dots are visible). The DPC phase-contrast image is shown in panel d). This DPC phase-contrast image was used to predict the H&E image shown in panel f. This image, in turn, shows high similarity to the ground-truth (reproduced in panel e) for ease of comparison).The black arrow shows the same region as before in panel c), but now the three black dots were correctly predicted, as assessed by comparison with the ground truth. The black arrow thus highlights regions where the DPC phase-contrast image produced superior predictions compared to the TIE phase-contrast image. Conversely, regions were also identified where the TIE phase-contrast image produced superior H&E predictions compared to the DPC phase-contrast image. This applies to the red tissue region indicated by the white arrow. Here, the ground truth and the prediction from the TIE phase image show a homogeneous red region (compared to white arrows in panels b and c), whereas the H&E prediction from DPC incorrectly predicts an inhomogeneous tissue region (see panels e and f).

[0075] The Fig. The example shown in Figure 8 indicates that DPC and TIE phase-contrast images exhibit different predictive performance for virtual staining applications. Next, an MLL that fuses both digital phase-contrast imaging modalities was evaluated.

[0076] Fig. Figure 9 shows the results of training a virtual staining neural network with both DPC and TIE phase-contrast output images. Again, comparing the black arrows, indicating the ability to predict detailed structures, and the white arrows, indicating the ability to predict a homogeneous background, it can be seen that providing DPC and TIE phase-contrast images (panel d) as input to a virtual staining MLL performs better than using a single output image (either TIE in panel b or DPC in panel c, but not both).

[0077] In summary, multimodal phase-contrast images are provided as input for virtual staining. This improves the robustness of the predicted images. While the above primarily disclosed a combination of a first set of imaging data with DPC and a second set of imaging data with TIE phase contrast, various other combinations of sets of imaging data with phase contrast are also possible. Possible examples are disclosed next: Multiple sets of imaging data with classical phase contrast can be combined, e.g., Zernike phase contrast, differential interference contrast (DIC), and Jamin-Lebedeff interference contrast. Alternatively or additionally, alternative digital phase-contrast methods beyond DPC and TIE phase contrast can be used. Examples include shear interferometry, inline and off-axis holography, and phase-shift interferometry.Alternatively, or in addition to using different phase-contrast imaging modalities, a single phase-contrast imaging modality can be used with modification of other optical parameters. A first example concerns the variation of color / wavelength. For example, a variety of colors can be used in a single phase-contrast experimental setup, for example, DPC with an LED array with red, green, and / or blue illumination channels (and similarly for the phase-contrast methods mentioned above). Analogously, multiple spectral bands in the infrared or UV range can be used as input to make MLL predictions more robust. A second example concerns the variation of polarization. This involves varying the principal axis, or linear and / or circular polarization, of the light either upstream and / or downstream of the sample.

[0078] Although the invention has been shown and described with reference to certain preferred embodiments, equivalents and modifications will become apparent to others skilled in the art upon reading and understanding the specification. The present invention includes all such equivalents and modifications and is limited only by the scope of the appended claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] WO 2019 / 154987 A1

[0008] WO 2021 / 198241 [0010, 0052] Cited non-patent literature

[0000] Kreiss, Lucas, et al. „Digital staining in optical microscopy using deep learning--a review.“ arXiv preprint arXiv:2303.08140 (2023

[0022] Ronneberger, Olaf, Philipp Fischer und Thomas Brox. „U-net: Convolutional networks for biomedical image segmentation.“ International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015 [0023, 0047] Zhu, Jun-Yan, et al. „Unpaired image-to-image translation using cycle-consistent adversarial networks.“ Proceedings of the IEEE international conference on computer vision. 2017

[0023] Tian, Lei und Laura Waller. „Quantitative differential phase contrast imaging in an LED array microscope.“ Optics express 23.9 (2015): 11394-11403

[0066] Streibl, Norbert. „Phase imaging by the transport equation of intensity.“ Optics communications 49.1 (1984): 6-10

[0067]

Claims

[1] A method for virtually staining a tissue sample (2105), the method comprising: - Obtaining (3301) a plurality of sets of imaging data (801) showing a tissue sample (2015), wherein the plurality of sets of imaging data (801) have a plurality of different phase contrasts, - fusing and processing (3302) the multiple sets of imaging data (801) in a machine learning logic, and - Obtaining (3303), from the machine learning logic, at least one output image (802, 803), wherein each of the at least one output image (802, 803) shows the tissue sample (2105) comprising a respective virtual coloration. [2] The method of claim 1, wherein the plurality of different phase contrasts comprise at least two different phase contrasts acquired with a plurality of different phase contrast imaging modalities. [3] Method according to claim 2, wherein the plurality of different phase contrast imaging modalities comprise a first digital phase contrast imaging modality and a second digital phase contrast imaging modality, wherein the first digital phase contrast imaging modality is a digital phase contrast imaging modality with intensity transport equation, wherein the second digital phase contrast imaging modality is a digital differential phase contrast imaging modality. [4] The method of claim 2 or 3, wherein the plurality of different phase contrast imaging modalities comprise at least one of a Zernike phase contrast, a Jamin-Lebedeff interference phase contrast, a shear interferometry phase contrast, an in-line holography phase contrast, an off-axis holography phase contrast, or a phase-shift interferometry phase contrast. [5] A method according to any one of the preceding claims, wherein the plurality of different phase contrasts comprise at least two different phase contrasts acquired with the same phase contrast imaging modality at different imaging settings. [6] The method of claim 5, wherein the at least two different phase contrasts were acquired using a digital phase contrast imaging modality at a plurality of different wavelengths. [7] The method of claim 5 or 6, wherein the at least two different phase contrasts were acquired using a digital phase contrast imaging modality at a plurality of different polarizations. [8] A computer device (80, 2110) comprising at least one processor (81) and a memory (83), wherein the at least one processor is designed to load program code from the memory and to execute the program code, wherein the at least one processor performs the following steps after executing the program code: - Obtaining (3301) a plurality of sets of imaging data (801) showing a tissue sample (2015), wherein the plurality of sets of imaging data (801) have a plurality of different phase contrasts, - fusing and processing (3302) the multiple sets of imaging data (801) in a machine learning logic, and - Obtaining (3303), from the machine learning logic, at least one output image (802, 803), wherein each of the at least one output image (802, 803) shows the tissue sample (2105) comprising a respective virtual coloration. [9] Computer device according to claim 8, wherein the computer device is connected to an optical imaging system (70) configured to acquire the plurality of sets of imaging data (801), wherein the processor is configured to load the plurality of sets of imaging data (801) from the optical imaging device. [10] A computer device according to claim 8 or 9, wherein the at least one processor is configured to carry out the method according to any one of claims 1 to 7.

Images