Custom virtual staining

Abstract

A method of virtually staining a tissue sample includes obtaining imaging data (501-503) depicting the tissue sample. The method also includes processing the imaging data (501-503) in at least one machine learning logic (500) configured to provide a plurality of output images (521) all comprising a given virtual stain of the tissue sample, the plurality of output images (521) depicting the tissue sample comprising the given virtual stain with different colorations associated with different staining laboratory processes. The method further includes obtaining at least one output image (521) of the plurality of output images from the at least one machine learning logic (500).

Description

Technical Field

[0001] The various examples generally involve virtual staining of tissue samples, i.e., providing one or more output images depicting a tissue sample including a virtual staining agent. The various examples specifically involve customizing virtual staining to provide output images depicting tissue samples including virtual staining agents with different colors. Different staining can be associated with different staining laboratory procedures. Background Technology

[0002] Histopathological examination is an important tool for diagnosing diseases. Histopathology refers to the optical examination of tissue samples. It helps in the diagnosis of cells within tissue samples.

[0003] Typically, histopathological examination begins with surgery, biopsy, or autopsy to obtain the tissue to be examined. The tissue may be processed to remove moisture and prevent decay. The processed tissue may then be embedded in a paraffin block. Thin slices can be cut from the paraffin block. These slices may be referred to hereinafter as tissue samples.

[0004] Tissue samples can be analyzed under a microscope by a histopathologist. Tissue samples can be stained with chemical staining agents using appropriate laboratory staining procedures, thereby facilitating the analysis of tissue samples. In particular, chemical staining agents can reveal cellular components that are difficult to observe in unstained tissue samples. Furthermore, chemical staining agents can provide contrast. Chemical staining agents can highlight one or more biomarkers or predefined structures in a tissue sample.

[0005] The most commonly used chemical staining agents in histopathology are the combination of hematoxylin and eosin (abbreviated as H&E). Hematoxylin stains the cell nucleus blue, while eosin stains the cytoplasm and extracellular connective tissue matrix pink. Hundreds of other techniques have been used for selective staining of cells. Recently, antibodies have been used to stain specific proteins, lipids, and carbohydrates. This technique, called immunohistochemistry, has greatly improved the ability to clearly identify various cell types under a microscope. Staining with H&E staining agents can be considered the common gold standard for histopathological diagnosis.

[0006] By staining tissue samples with chemical dyes, previously almost transparent and indistinguishable structural / tissue sections become visible to the human eye. This allows pathologists and researchers to study tissue samples under a microscope or using digital brightfield equivalent images, and to assess tissue morphology (structure) or explore the presence or prevalence of specific cell types, structures, or even microorganisms (such as bacteria).

[0007] WO 2019 / 154987 A1 discloses a method that uses machine learning logic to provide virtual stained images that look like typical images of tissue samples that have been stained with conventional chemical staining agents.

[0008] It has been found that the ability to correctly interpret images depicting tissue samples containing chemical stains can depend on the specific staining laboratory procedures used to produce the chemical stains. Therefore, pathologists often need to perform specific staining laboratory procedures. This can limit the ability to obtain a second opinion based on existing images of tissue samples containing chemical stains prepared using a specific staining laboratory procedure. For example, in order to provide a second opinion, it may be necessary to restain the tissue sample using a further staining laboratory procedure.

[0009] For example, in a reference embodiment, the following workflow can be envisioned to obtain an accurate diagnosis. Pathologist A sends a tissue sample to laboratory A. Laboratory A prepares and stains the tissue sample and returns the stained probe to pathologist A. Laboratory A uses the appropriate laboratory staining procedure. Pathologist A can then analyze the tissue sample, including the chemical staining agent, for example, using a microscope. Sometimes, pathologist A may require a second opinion. The stained probe can then be sent to pathologist B. Pathologist B can examine the stained probe and provide feedback to pathologist A. In some cases, pathologist B may not be able to analyze the tissue sample stained using laboratory A's laboratory procedure. Another tissue sample—e.g., another part or section of the sample—can then be transferred to laboratory B to provide chemical staining using another staining laboratory procedure. Pathologist B can then examine this additional tissue sample. Opinions can be combined.

[0010] This method is both time-consuming and expensive. It is limited by the availability of sufficient tissue samples: for example, if restaining is required, different tissue samples may be considered, which could introduce potential variations in the diagnosis. Summary of the Invention

[0011] Therefore, there is a need for advanced technologies to depict images of tissue samples, including staining agents. There is also a need for technologies that facilitate customized virtual staining.

[0012] This requirement is satisfied by the features of the independent claims. The features of the dependent claims define the embodiments.

[0013] A method for virtually staining a tissue sample includes acquiring imaging data. The imaging data depicts the tissue sample. The method further includes processing the imaging data in at least one machine learning logic. The at least one machine learning logic is configured to provide multiple output images. All of the multiple output images depict the tissue sample including a given virtual stain. Different staining is associated with different staining laboratory procedures and / or the configuration of the imaging mode used to acquire the imaging data. Further, the method includes obtaining at least one output image from the multiple output images from the at least one machine learning logic.

[0014] Tissue samples can refer to thin slices of paraffin blocks comprising the processed samples embedded as described above. However, the term "tissue sample" can also refer to tissue that has been processed differently or has not been processed at all. For example, a tissue sample can refer to a portion of tissue observed in vivo and / or tissue excised from a human, animal, or plant, wherein the observed tissue sample has been further processed in vitro, for example, prepared using cryosectioning. Tissue samples can be any kind of biological sample. The term "tissue sample" can also refer to cells, which can be of prokaryotic or eukaryotic origin, multiple prokaryotic and / or eukaryotic cells (such as a single-cell array), multiple adjacent cells (such as a cell population or cell culture), or complex samples (such as a biofilm or microbiome containing a mixture of different prokaryotic and / or eukaryotic cell species and / or organoids).

[0015] A computer program product, computer program, computer-readable storage medium, or data signal includes program code. The program code can be loaded and executed by at least one processor. When the program code is executed, the at least one processor performs a method for virtually staining a tissue sample. The method includes acquiring imaging data. The imaging data depicts the tissue sample. The method further includes processing the imaging data in at least one machine learning logic. The at least one machine learning logic is configured to provide a plurality of output images. All of the plurality of output images depict a tissue sample including a given virtual stain. All of the plurality of output images depict a tissue sample including a given virtual stain with different stainings. The different stainings are associated with different staining procedures (such as staining laboratory procedures) and / or the configuration of an imaging mode used to acquire the imaging data. Further, the method includes obtaining at least one output image from the plurality of output images from the at least one machine learning logic.

[0016] An apparatus includes a processor. The processor is configured to acquire imaging data. The imaging data depicts a tissue sample. The processor is further configured to process the imaging data in at least one machine learning logic. The at least one machine learning logic is configured to provide a plurality of output images. All of the plurality of output images depict the tissue sample. The tissue sample includes a given dummy stain. The plurality of output images depict tissue samples including different tints of the given dummy stain. The different tints are associated with different staining procedures (such as staining laboratory procedures) and / or the configuration of the imaging mode used to acquire the imaging data. The processor is further configured to obtain at least one of the plurality of output images from the at least one machine learning logic.

[0017] A method for training at least one machine learning logic to virtually stain tissue samples includes obtaining training imaging data. The training imaging data depicts one or more tissue samples. The method also includes obtaining multiple reference output images. The multiple reference output images depict one or more tissue samples, or one or more further tissue samples, all including a given chemical stain. Different reference output images depict one or more tissue samples, or one or more further tissue samples, including different staining with a given chemical stain provided by different staining procedures (such as staining laboratory procedures) and / or configurations of imaging modes used to acquire the imaging data. The method also includes training the at least one machine learning logic based on the training imaging data and the multiple reference output images.

[0018] The term "chemical staining" can also include molecules that modify any of the aforementioned different types of tissue samples. This modification can produce fluorescence under certain irradiation (e.g., irradiation with ultraviolet (UV) light). For example, chemical staining can include modifications to the genetic material of a tissue sample. Chemically stained tissue samples can include transfected cells. Transfection can refer to the process of intentionally introducing naked or purified nucleic acids into eukaryotic cells. It can also refer to other methods and cell types. It can also refer to the transfer of nonviral DNA in bacteria and non-animal eukaryotic cells (including plant cells).

[0019] Modifying the genetic material of a tissue sample can make it observable using a specific imaging modality. For example, the genetic material can exhibit fluorescence. In some examples, modifying the genetic material of a tissue sample can induce the production of molecules observable using a specific imaging modality. For instance, modifying the genetic material of a tissue sample can induce the production of fluorescent proteins.

[0020] A computer program product, computer program, computer-readable storage medium, or data signal includes program code. The program code can be loaded and executed by at least one processor. When the program code is executed, the at least one processor performs a method for training at least one machine learning logic by virtually staining tissue samples. The method includes obtaining training imaging data. The training imaging data depicts one or more tissue samples. The method also includes obtaining a plurality of reference output images. The plurality of reference output images depict one or more tissue samples, or one or more further tissue samples, all including a given chemical stain. Different reference output images depict one or more tissue samples, or one or more further tissue samples, including different stainings of a given chemical stain provided by different staining procedures (such as staining laboratory procedures) and / or configurations of imaging modes used to acquire imaging data. The method also includes training the at least one machine learning logic based on the training imaging data and the plurality of reference output images.

[0021] An apparatus includes a processor. The processor is configured to acquire training imaging data. The training imaging data depicts one or more tissue samples. The processor is further configured to acquire a plurality of reference output images. The plurality of reference output images depict one or more tissue samples, or depict one or more further tissue samples, all of which include a given chemical stain. Different reference output images depict one or more other tissue samples including different stainings of the given chemical stain. The different stainings are provided by different staining procedures (e.g., staining laboratory procedures) and / or the configuration of the imaging mode used to acquire the imaging data. The processor is further configured to train at least one machine learning logic based on the training imaging data and the plurality of reference output images.

[0022] A method for virtually staining a tissue sample includes acquiring imaging data depicting a tissue sample comprising a chemical stain having a first color. The method further includes processing the imaging data in machine learning logic. The method further includes obtaining an output image from the machine learning logic depicting a tissue sample comprising a virtual stain. The virtual stain is associated with a chemical stain. The virtual stain includes a second color that is different from the first color.

[0023] The association between virtual staining agents and chemical staining agents can involve both virtual staining agents and chemical staining agents highlighting the same type of structure or (multiple) biomarkers.

[0024] A computer program product, computer program, computer-readable storage medium, or data signal includes program code. The program code can be loaded and executed by at least one processor. When the program code is executed, the at least one processor performs a method for virtually staining a tissue sample. The method includes acquiring imaging data. The imaging data depicts a tissue sample including a chemical stain having a first color. The method further includes processing the imaging data in machine learning logic. The method further includes obtaining an output image from the machine learning logic depicting a tissue sample including a virtual stain. The virtual stain is associated with a chemical stain. The virtual stain includes a second color that is different from the first color.

[0025] An apparatus includes a processor. The processor is configured to acquire imaging data depicting a tissue sample including a chemical stain having a first color. The processor is also configured to receive the imaging data from machine learning logic. The processor is further configured to obtain an output image from the machine learning logic depicting a tissue sample including a virtual stain. The virtual stain is associated with the chemical stain. The virtual stain includes a second color that is different from the first color.

[0026] It should be understood that, without departing from the scope of the invention, the features mentioned above and those to be explained below can be used not only in the indicated combinations, but also in other combinations or individually. Attached Figure Description

[0027] Figure 1 The machine learning logic based on each example is illustrated.

[0028] Figure 2 It is a flowchart based on the methods of each example.

[0029] Figure 3 It is a flowchart based on the methods of each example.

[0030] Figure 4 The apparatus is illustrated schematically according to various examples. Detailed Implementation

[0031] Some examples in this disclosure generally provide multiple circuits or other electrical devices. All references to circuits and other electrical devices and the functions they each provide are not intended to be limited to what is illustrated and described herein. While specific labels may be assigned to the various circuits or other electrical devices disclosed, such labels are not intended to limit the scope of operation of these circuits and other electrical devices. Such circuits and other electrical devices may be combined with each other and / or separated in any way based on a particular type of desired electrical implementation. It should be recognized that any circuit or other electrical device disclosed herein may include any number of microcontrollers, machine learning-specific hardware (e.g., graphics processing units (GPUs) and / or tensor processing units (TPUs), integrated circuits, memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or other suitable variations thereof), and software that work together to perform the operations disclosed herein. Furthermore, any one or more electrical devices may be configured to execute program code embodied in a non-transitory computer-readable medium that is programmed to perform any number of the functions disclosed.

[0032] In the following description, embodiments of the invention will be described in detail with reference to the accompanying drawings. It should be understood that the following description of the embodiments should not be construed as limiting. The scope of the invention is not intended to be limited by the embodiments or drawings described below, which are considered to be illustrative only.

[0033] The accompanying drawings should be considered schematic representations, and the elements shown in the drawings are not necessarily shown to scale. Rather, the various elements are shown in such a manner that their function and general purpose will be obvious to those skilled in the art. Any connection or coupling between functional blocks, devices, components, or other physical or functional units shown in the drawings or described herein may also be implemented via indirect connection or coupling. Coupling between components may also be established via wireless connection. Functional blocks may be implemented using hardware, firmware, software, or a combination thereof.

[0034] The various techniques described in this article generally involve machine learning. Machine learning, especially deep learning, provides a data-driven strategy for solving problems. Classical inference techniques are able to extract patterns from data to solve problems based on hand-designed features; an example technique is regression. However, such classical inference techniques depend heavily on the accurate selection of hand-designed features, which in turn depends on the designer's skill. One solution to this problem is to leverage machine learning to discover the mapping from features to output, as well as the features themselves. This is known as training the machine learning logic.

[0035] The various techniques described herein generally involve virtually staining tissue samples by utilizing at least one machine learning logic (MLL). At least one MLL can be implemented, for example, by a support vector machine or a deep neural network, and may include at least one encoder branch and at least one decoder branch.

[0036] The techniques described in this article facilitate the customization of virtual coloring. Customization can involve providing an output image with the desired coloring using virtual coloring. Therefore, different appearances of the same virtual coloring are possible.

[0037] Customization is possible for different staining procedures (such as laboratory staining procedures). For example, different staining procedures (such as laboratory staining procedures) may exhibit variations in chemical processing parameters (such as temperature, concentration, etc.). Different staining procedures (such as laboratory staining procedures) may be affected by different laboratory noise levels. For instance, it has been found that different staining procedures (such as laboratory staining procedures)—while nominally providing the same chemical stains—can provide tissue samples containing chemical stains with different staining properties. On the other hand, medical personnel interpreting images of tissue samples containing chemical stains often require specific staining to reliably interpret the images.

[0038] Virtual staining can be used in the histopathology of tissue samples.

[0039] For example, it has been found that different pathologists often send probes to different preferred chemistry labs to obtain tissue samples containing the appropriate chemical stains with the desired color. In such cases, it can be difficult to obtain a second opinion from a second pathologist, as the second pathologist may need to restain the tissue sample in another chemistry lab using a different staining lab procedure (albeit nominally using the same chemical stains). This is expensive, time-consuming, and sometimes even impossible, for example, if the amount of available tissue sample is limited.

[0040] Another example involves virtual fluorescence staining. For instance, in life science applications, transmitted light microscopy is used to obtain images of cells—e.g., cells arranged as living or fixed cells in a multi-well plate or other suitable container. Furthermore, reflected light microscopy can be used, for example, in endoscopes or as a surgical microscope. Certain organelles, such as the nucleus, ribosomes, endoplasmic reticulum, Golgi apparatus, chloroplasts, or mitochondria, can then be selectively stained. A fluorophore (or fluorescent dye, similar to a chromophore) is a fluorescent compound that re-emits light upon photoexcitation. Fluoresters can be used to provide fluorescent chemical staining agents. Different chemical staining agents can be achieved by using different fluorophores. For example, Hoechst stain is a fluorescent dye that can be used to stain DNA. Other fluorophores include 5-aminolevulinic acid (5-ALA), fluorszine, and indocyanine green (ICG), which can even be used in vivo. Fluorescence can be selectively excited by using light of a corresponding wavelength; the fluorophore then emits light of another wavelength. The corresponding fluorescence microscope uses the corresponding light source. Irradiation using light to excite fluorescence has been observed to damage samples; this can be avoided by providing a fluorescence-like image through virtual staining. Virtual fluorescent staining mimics fluorescent chemical staining without exposing the tissue to the corresponding excitation light. It has been observed that even when using the same fluorophore (i.e., nominally the same chemical staining agent), the laboratory procedure used to apply the fluorophore and / or the configuration of the optical microscope used to observe the tissue sample can cause changes in appearance.

[0041] For example, the optical devices used, the illumination, and the excitation source can all introduce different appearances. It has been observed that, although the same chemical staining agents are nominally used to highlight cellular parts, the interpretation of individual images can be difficult due to the variability introduced by the various available configurations of optical microscopes.

[0042] To mitigate these limitations and drawbacks, imaging data of tissue samples is obtained, according to various examples. The imaging data is then processed on at least one machine learning logic (MLL). At least one MLL is configured to provide multiple output images. All of the multiple output images depict tissue samples including the same virtual stain, but with different colorings. This means, for example, that one or more identical biomarkers or organelles can be highlighted in multiple output images because these output images depict the same tissue including the same virtual stain selectively applied to that one or more biomarkers or organelles. That is, the same structures can be highlighted in multiple images. The appearance of one or more biomarkers or organelles may vary between output images. The virtual stain can be associated with a chemical stain, for example, for histopathology: hematoxylin and eosin (H&E), Ki67, human epidermal growth factor receptor 2 (HER2), estrogen receptor (ER), progesterone receptor (PR)—or other chemical stains. The virtual stain can be associated with the same fluorophore (e.g., a given Hurst stain). It is then possible to obtain at least one output image from a plurality of output images, each depicting a tissue sample with a given virtual stain having the desired coloring, from at least one MLL. For example, each iteration of the execution of the MLL may yield a single output image depicting a tissue sample with a given virtual stain having the corresponding coloring associated with that iteration. Different iterations then executed may output different output images, all depicting the same tissue with the same virtual stain but with different coloring; the MLL may be configured accordingly to select a particular output image.

[0043] Generally, coloring can define the relative contribution of different colors to the appearance of a virtual stain, such as a histogram across the color spectrum. Different colorings can be associated with different user preferences. Different colorings can show dependence on structure: thus, while similar structures are highlighted, the color of such labels may depend on the specific structure type and vary with coloring. Different colorings may be associated with different staining processes (such as laboratory staining processes): that is, when using different staining processes (such as laboratory staining processes) to produce nominally identical chemical stains, the color appearance may still differ; this difference can be reflected by the coloring of the corresponding virtual staining agent. Alternatively or additionally, different colorings can be associated with different configurations of the corresponding imaging modality. Thus, although nominally mimicking the same chemical staining agent, the appearance may vary depending on the configuration.

[0044] Therefore, although different output images depict tissue samples including the same virtual stain, the graphic appearance of the tissue sample may vary between output images. Thus, by adapting the graphic appearance to the pathologist's requirements for receiving and interpreting the output images, accurate analysis of the informational content of the output images can be facilitated. In particular, multiple pathologists may be able to provide their own opinions and analyses, each using a customized virtual stain. Furthermore, all pathologists can analyze the same tissue sample, rather than different tissue samples stained in different ways: for example, identical tissue samples (especially identical sections of tissue) can be analyzed with different customized virtual stains.

[0045] As used herein, imaging data of a tissue sample refers to any type of data representing a tissue sample or a portion thereof, particularly digital imaging data. For example, depending on the imaging modality, the dimensionality of imaging data of a tissue sample may vary. Imaging data can be two-dimensional (2-D), one-dimensional (1-D), or even three-dimensional (3-D). If more than one imaging modality is used to obtain imaging data, some of the imaging data may be two-dimensional, while other portions may be one-dimensional or three-dimensional. For example, microscopic imaging can provide imaging data including images with spatial resolution (i.e., including multiple pixels). Scanning a tissue sample with a confocal microscope can provide imaging data including three-dimensional voxels. Spectroscopy of a tissue sample can produce imaging data that provides spectral information for the entire tissue sample or a large portion thereof without spatial resolution. In another embodiment, spectroscopy of a tissue sample can produce imaging data that provides spectral information for several locations of the tissue sample, resulting in imaging data that includes spatial resolution but is sparsely sampled.

[0046] As used herein, imaging modalities may include, for example, imaging of tissue samples in one or more specific spectral bands (particularly in the ultraviolet, visible, and / or infrared ranges (multispectral microscopy)). Imaging modalities may also include Raman analysis of tissue samples, particularly stimulated Raman scattering (SRS) analysis, coherent anti-Stokes Raman scattering (CARS) analysis, and surface-enhanced Raman scattering (SERS) analysis. Further, imaging modalities may include fluorescence analysis of tissue samples, particularly fluorescence lifetime imaging microscopy (FLIM) analysis. Imaging modalities may specify phase-sensitive acquisition of digital imaging data. Imaging modalities may also specify polarization-sensitive acquisition of digital imaging data. A further example is transmission light microscopy, for example, for observing cells. Generally, imaging modalities can be used for imaging in vivo or in vitro. Endoscopes may be used for image acquisition, such as confocal microscopy or endoscopic optical coherence tomography (e.g., scanning or full-field). Confocal fluorescence scanners may be used. Endoscopic two-photon microscopy is another imaging modality. A surgical microscope can be used; the surgical microscope itself can provide a variety of imaging modes, such as microscopic images or fluorescence images, for example in a specific spectral band or a combination of two or more wavelengths, or even hyperspectral images.

[0047] Generally, one or more MLLs can be used to provide multiple output images depicting tissue samples with virtual staining agents that include different tints of corresponding staining schemes. For example, a dedicated MLL can be used for each tint. That is, for each tint, there may be a corresponding MLL. In this case, each MLL can be trained separately based on corresponding training imaging data and ground truth output images, which depict tissue samples including tissue samples with corresponding chemical staining agents (corresponding to the desired virtual staining agents) using various tints (e.g., obtained from the corresponding staining process, such as a staining laboratory process).

[0048] In other examples, a single MLL including conditional inputs can be used. By setting the conditional inputs, selection can be made between staining processes associated with different staining procedures (such as laboratory staining procedures), thus obtaining the corresponding output image from the single MLL. Figure 1 This situation was demonstrated.

[0049] Figure 1This diagram illustrates various aspects of MLL 500. MLL 500 can be implemented, for example, using deep neural networks with a U-net architecture. See Ronneberger, Olaf, Philipp Fischer, and Thomas Brox, “U-net: Convolutional networks for biomedical image segmentation,” International Conference on Computational Medical Images and Computer-Aided Intervention, Springer, Cham, 2015.

[0050] More generally, a deep neural network may include multiple hidden layers. A deep neural network may include an input layer and an output layer. Hidden layers are arranged between the input and output layers. Spatial shrinkage and spatial expansion may occur, implemented by one or more encoder branches and one or more decoder branches respectively (but this is optional: in other cases, the spatial dimensions may remain unchanged, possibly excluding edge cropping). For spatial shrinkage (expansion), the xy-resolution of the corresponding representations of the imaging data and the output image may decrease (increase) layer by layer along one or more encoder branches (decoder branches); simultaneously, the feature channels may increase (decrease) along one or more encoder branches (one or more decoder branches). In one or more output layers, the deep neural network may include a decoder head, which may include activation functions, such as linear or nonlinear activation functions.

[0051] Each layer can provide one or more operations on the corresponding representation of the input imaging data or the output image, such as convolution, activation functions (e.g., ReLU, sigmoid function, tanh, Maxout, ELU, Scaled Exponential Linear Unit (SELU), Softmax, etc.), downsampling, upsampling, batch normalization, dropout, etc.

[0052] MLL 500 processes imaging data: In Figure 1 In this process, the MLL 500 handles multi-group imaging data 501-503, such as imaging data obtained using different imaging modes and / or with different spatial resolutions. Multi-group imaging data 501-503 can depict tissue samples with or without chemical staining agents. Different imaging data within imaging data 501-503 can have different staining characteristics, i.e., different chemical staining agents or with / without chemical staining agents (i.e., some groups of imaging data can depict tissue samples with chemical staining agents, while other imaging data depict tissue samples without chemical staining agents).

[0053] For example, imaging data 501-503 may include input images with spatial resolution. For example, a multispectral microscope may be used. For example, one or more of the following imaging modes may be used: hyperspectral microscopy; fluorescence imaging; autofluorescence imaging; light sheet imaging; Raman spectroscopy; etc.

[0054] MLL 500 also processes data 511, which is provided to the conditional input 531 of MLL 500. Data 511 can be tagged as metadata because it configures the operations of MLL 500. Data 511 can be provided as a scalar or a vector.

[0055] For example, data 511 may include an indicator indicating the selection of a colorimeter from a set of predefined candidate colorimeters associated with a predefined staining laboratory procedure. In other words, there may be a finite set of candidate colorimeters, and—for example, based on a corresponding codebook—an appropriate colorimeter may be selected from that set of predefined candidate colorimeters, for example, using one-hot encoding, etc. This is illustrated in conjunction with Table 1 below.

[0056] The value of data 511 Coloring explain 001 Staining Laboratory Procedure A H&E staining agent 010 Staining Laboratory Procedure B H&E staining agent 100 Staining Laboratory Procedure C H&E staining agent 011 Staining Laboratory Procedure D H&E staining agent 101 Staining Laboratory Process E H&E staining agent 111 Staining Laboratory Process F H&E staining agent

[0057] Table 1: Conditional Input Data for MLL, Example

[0058] In another example, the conditional input can be selected between shading according to the target color map provided to the conditional input.

[0059] The target color map parameterizes the coloring associated with the corresponding staining laboratory procedure. Different options are available for implementing the target color map. For example, the target color map can be a red-green-blue vector (here, for training, the corresponding training input in grayscale and the associated red-green-blue vector, along with the associated ground truth reference output image obtained from the corresponding laboratory staining procedure associated with the red-green-blue vector). Other color spaces (such as CYMK or HSV) are also possible. This corresponds to a color mapping. For example, the target color map can specify a color histogram, or more generally, the relative contribution of various colors to the virtual staining of the tissue sample in the corresponding output image, for example, depending on the biomarkers highlighted by the virtual stain. In general, the target color map can specify which colors to use to highlight which structural types of the tissue sample. Therefore, by changing the values ​​of the target color map, the appearance / coloring of the virtual stain can be changed. The target color map thus parameterizes the coloring.

[0060] Generally, there are various options available for implementing MLLs to process the data 511 at the conditional input 531. For example, different decoder heads of the deep neural network can be selected depending on the data 511 at the conditional input 531 (i.e., by setting the conditional input), thereby obtaining different output images among multiple output images. Different decoder heads can be selected depending on the data 511 at the conditional input 531. Another example implementation can rely on conditional neural networks, such as those described in the following literature: Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros, “Image-to-Image Translation with Conditional Adversarial Networks”, CVPR 2017, https: / / arxiv.org / pdf / 1611.07004 / / Pix2Pix. Another example implementation may rely on reversible neural networks that use conditional information, as described in the following literature: "Guided image generation with conditional invertible neural networks" by Ardizzone, Lynton et al., arXiv preprint arXiv:1907.02392 (2019).

[0061] Figure 2 It is a flowchart based on the methods of each example. For example, Figure 2 The method can be executed by at least one processor when program code is loaded from non-volatile memory. This method facilitates the virtual staining of tissue samples. That is, it provides one or more output images depicting a tissue sample with a given virtual staining agent. For example, a given virtual staining agent can be associated with a corresponding chemical staining agent that can be achieved through a corresponding staining laboratory procedure. Thus, example virtual staining agents include, for example, virtual H&E, virtual Ki67, virtual HER2, virtual ER, or virtual PR.

[0062] Initially, at box 5011, imaging data 501-503 depicting a tissue sample is obtained. Imaging data 501-503 can be obtained via a communication interface, for example, from a remote node. For example, multiple sets of imaging data 501-503 can be obtained. Different imaging modes can be used to acquire different sets. Imaging data 501-503 may include spatially resolved images. Different imaging data can depict tissue samples with different chemical stains. For example, imaging data 501-503 can depict tissue samples including or excluding a given chemical stain associated with a virtual stain. For example, imaging data 501-503 can—at least partially—depict a tissue sample including H&E stain. The chemical stain of the tissue sample depicted by imaging data 501-503 may differ from the given virtual stain; this corresponds to virtual restaining. The chemical stain of the tissue sample depicted by imaging data 501-503 may be the same as the given virtual stain; this corresponds to virtual recoloring; thus, based on the chemical staining provided by the first laboratory process, a corresponding virtual staining corresponding to the second laboratory process can be produced. Therefore, for example, one or more input images can depict a tissue sample including a first stain; and one or more output images can depict a tissue sample including at least one second stain different from the first stain. This facilitates coordination between different laboratory procedures.

[0063] At option 5012, data 511 for defining or selecting shading can be determined. For example, data 511 can be used for conditional input 531 in an MLL, or for selection among multiple MLLs. For example, indicator values ​​can be determined to select an appropriate shading from a set of predefined shadings, such as those shown in Table 1. A target color map for parameterizing the shading can also be determined. For example, the target color map can specify which color to use to highlight which type of structure or which geometry.

[0064] Generally, at box 5012, there are various options available to determine data 511, including the target color map.

[0065] For example, the target color map can be determined based on one or more predefined target color maps parameterized to the staining associated with one or more corresponding further staining laboratory procedures. In other words, this means that the target color map determined at box 5012 can be derived from one or more predefined target color maps. For example, the target color map can be obtained by incrementally changing the value of a given predefined target color map. In another example, the target color map can be determined using interpolation between two or more predefined target color maps. Thus, transitions from one or more further staining laboratory procedures can be implemented, for example, a transition between two further staining laboratory procedures. Trade-offs can be made between multiple further laboratory staining procedures. For example, a temporal evolution of the staining from a first further staining laboratory procedure to a mixture of staining from a second further staining laboratory procedure can be implemented. That is, over time, a corresponding transition from the staining associated with the first further staining laboratory procedure to the staining associated with the second further staining laboratory procedure can be implemented. In a specific example, it is possible that in week 1, an output image depicting a tissue sample is presented to the pathologist, which includes the staining associated with a given further staining laboratory procedure (procedure A) preferred by the corresponding pathologist. Then, over time, the coloring can be altered; for example, in week 2, the coloring can be determined to be 90% according to process A and 10% according to another further staining laboratory process (process B). Mixing can continue toward process B, for example, reaching 80%-20% in week 3, and so on. This technique helps to slowly but steadily obtain a single, consistent color that can be shared across multiple staining laboratories or staining laboratory processes. In summary, it is therefore possible that the determination of data 511, including the target color map, depends on the temporal evolution associated with the transition from one or more predefined target color maps to the corresponding target color map. The target color map can therefore include two or more target color maps, wherein the target color map is determined based on a weighted combination of at least two of the two or more predefined target color maps.

[0066] At option 5013, appropriate machine learning logic can then be selected, for example, based on the data in box 5012. For example, different neural networks can be used for different target color maps.

[0067] In other examples, the same machine learning logic can be configured to receive data 511 at conditional input 531. This has already been combined with the above. Figure 1 An explanation has been provided. Then, box 5013 is no longer needed.

[0068] Next, at box 5014, the imaging data of box 5011 is processed. This is done using, for example, the selected machine learning logic of box 5013; or using a single machine learning logic that receives the data 511 at its conditional input 531 in parallel.

[0069] Then, in box 5015, an output image depicting a tissue sample stained with a given virtual stain associated with a corresponding staining laboratory procedure for a relevant chemical staining agent can be received. For example, a single output image depicting a tissue sample including a virtual staining agent with the color defined by data 511 in box 5012 can be received. The imaging data of box 5011 can also be processed in multiple iterations of multiple data 511s to obtain multiple output images depicting tissue samples including virtual staining agents with corresponding colorings.

[0070] Figure 2 The method involves using trained machine learning logic 500 for inference. Next, combined with... Figure 3 Discuss the details of training Logical500 in machine learning.

[0071] Figure 3 It is a flowchart based on the methods of each example. For example, Figure 3 The method can be executed by at least one processor when program code is loaded from non-volatile memory.

[0072] Figure 3 The method can be used to train machine learning logic that is configured to provide virtual staining of tissue samples in one or more output images provided by the machine learning logic.

[0073] pass Figure 3 The method can train a single machine learning logic or multiple machine learning logics. This relates to whether a single machine learning logic includes conditional input 531 or multiple machine learning logics are used to provide output images depicting tissue samples including virtual stains with various colorings, as described above. Figure 2 The explanation given in box 5013.

[0074] At box 5101, training imaging data is obtained. This training imaging data depicts one or more tissue samples. The tissue samples may or may not include chemical stains. For example, if the training imaging data depicts one or more tissue samples including chemical stains, restaining or recoloring may be performed. One or more imaging modalities can be used to acquire the training imaging data. The training imaging data can be obtained from a database.

[0075] The training imaging data can correspond to the imaging data obtained at box 5011, that is, including similar chemical stains or no stains, having corresponding dimensions, and having been acquired with the same corresponding imaging mode, etc.

[0076] Next, at box 5102, multiple reference output images are obtained. All of the multiple reference output images depict one or more tissue samples including a given chemical stain; that is, one or more tissue samples all highlight the same biomarker / have the same chemical stain in the multiple reference output images.

[0077] Therefore, the reference output image is used as the ground truth for further training of one or more machine learning logics.

[0078] In some examples, training imaging data can be registered to multiple reference output images; more specifically, training images included in the training imaging data can be registered to multiple reference output images. This means that corresponding portions included in the training images and reference output images are linked to each other. Therefore, the value of the loss function considered in training can be determined by determining the differences between the training imaging data and the training output images generated by one or more machine learning logics based on the training imaging data, and between the training imaging data and the reference output images. For example, differences in contrast can be considered. Differences in structure or texture can be considered. More generally, differences in appearance can be considered. However, it is not necessary to register training imaging data to multiple reference output images in all cases. For example, in other cases, unregistered training imaging data and reference output images can be used. Here, for example, a recurrent generative adversarial network (GAN) can be used.

[0079] As used in this paper, the term "recurrent GAN" can encompass any generative adversarial network and can refer to any generative adversarial network that utilizes some form of recurrent consistency during training. In particular, the term "recurrent generative adversarial network" can include cycleGAN, DiscoGAN, StarGAN, DualGAN, CoGAN, and UNIT.

[0080] Examples of such architectures are described below: CycleGAN: see, for example, “Unpaired image-to-image translation using cycle-consistent adversarial networks” by Zhu, JY, Park, T., Isola, P., and Efros, AA (2017), in the proceedings of the IEEE International Conference on Computer Vision (CVPR). DiscoGAN: see, for example, “Learning to discover cross-domain relations with generative adversarial networks” by Kim, T., Cha, M., Kim, H., Lee, JK, and Kim, J. (August 2017), in the proceedings of the 34th International Conference on Machine Learning, Volume 70 (pp. 1857–1865), JMLR.org. StarGAN: See, for example, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation” by Choi, Y., Choi, M., Kim, M., Ha, JW, Kim, S., and Choo, J. (2018), in the proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). DualGAN: See, for example, “Dualgan: Unsupervised dual learning for image-to-image translation” by Yi, Z., Zhang, H., Tan, P., and Gong, M. (2017), in the proceedings of the IEEE International Conference on Computer Vision (ICCV). CoGAN. See, for example, M.-Y. Liu and O. Tuzel, “Coupledgenerative adversarial networks,” Advances in Neural Information Processing Systems (NIPS), 2016.UNIT: See, for example, “Unsupervised image-to-image translation networks” by Liu, Ming-Yu, Thomas Breuel, and Jan Kautz, Advances in Neural Information Processing Systems (NIPS), 2017.

[0081] Then, at box 5103, training of one or more MLLs is performed. This is based on the training imaging data obtained at box 5101 and the reference output image obtained at box 5102. For example, in the case of training multiple machine learning logics, this can be done individually for each of the multiple machine learning logics based on a loss function determined based on a corresponding subset of the reference output images depicting one or more tissue samples with corresponding colored chemical stains.

[0082] Training can be performed using a recurrent GAN approach: here, the generator logic can be implemented in each of the forward and backward loops; and another neural network can be used as a discriminator.

[0083] As explained above, training can consider spatial registration to determine the loss function based on contrast differences between different spatial regions. Pixel-level differences are possible. In some implementations, the structured similarity index (https: / / www.ncbi.nlm.nih.giv / pubmed / 28924574) can be used as the loss function. Multiple loss functions can be combined, for example, in a weighted manner.

[0084] Backpropagation can be used to adjust the weights of each layer in a neural network.

[0085] Figure 4 A device 701 according to various examples is schematically shown. Device 701 includes a processor 702 and a memory 704. Processor 702 can load program code from memory 704. Processor 702 can execute the program code. When executing the program code, processor 702 can perform one or more of the following logical operations as described throughout this disclosure: acquiring imaging data, e.g., via interface 703 of device 701 or by loading imaging data from memory 704; virtually staining a tissue sample depicted by the imaging data; executing at least one machine learning logic to process the imaging data (inference), e.g., in multiple iterations; obtaining at least one output image from / while executing the machine learning logic, e.g., outputting at least one output image via interface 703; setting parameters or hyperparameters of the machine learning logic while training the machine learning logic; training the machine learning logic; etc. For example, Figure 2 Methods Figure 3 The method can be executed by processor 702 when loading program code.

[0086] Although the invention has been shown and described with respect to specific preferred embodiments, equivalents and modifications will occur to those skilled in the art upon reading and understanding the specification. The invention includes all such equivalents and modifications and is limited only by the scope of the appended claims.

[0087] For illustration, various examples have been described in conjunction with a single MLL, which is configured to output multiple output images, for example, depending on the conditional input of the MLL. Similar techniques can be applied to cases using multiple MLLs, each configured to output a single output image. Then, an appropriate selection can be implemented among the multiple MLLs instead of using conditional input. Furthermore, it is conceivable that a (single) MLL is configured to provide only a single output image, for example, in the context of recoloring as described below.

[0088] To further illustrate, the scenario where restaining is possible has already been described. Generally, an MLL configured to provide an output image depicting a tissue sample including a dummy stain can be used; here, the input image provided to the MLL can depict a tissue sample including a chemical stain associated with the dummy stain (i.e., both are H&E type or Ki67 stain, etc.). The input and output images depict tissue samples including the corresponding stains with different staining.

[0089] To further illustrate, various cases of customized coloring of virtual stains for tissue samples in output images have been disclosed above, whereby the customized coloring is associated with a specific staining process using nominally identical base chemical stains. Specifically, staining laboratory processes can be used to produce chemical stains for histopathology. In a further example, images with customized coloring using virtual stains can be provided, whereby the customized coloring is associated with different configurations of the same imaging modality. For example, it has been observed that, depending on a specific configuration of the imaging modality, even nominally identical chemical stains can have different graphic appearances. By providing customized coloring, these different graphic appearances depending on the configuration of the imaging modality can be mimicked. For example, it has been observed that for fluorescence imaging of cell samples, the configuration of the corresponding transmitted light microscope can alter the appearance of the highlighted organelles. For example, the brightness level of the fluorescent light source used to excite fluorophores can alter the appearance of the fluorophores in the corresponding image.

Claims

1. A method for virtually staining tissue samples, the method comprising: - Obtain imaging data (501-503) depicting the tissue sample. - The imaging data (501-503) is processed in at least one machine learning logic (500), which is configured to provide multiple output images (521) all depicting tissue samples including a given virtual stain, the multiple output images (521) depicting tissue samples including different stainings of a given virtual stain associated with different staining laboratory procedures, and - Obtain at least one output image (521) from the plurality of output images from the at least one machine learning logic (500).

2. The method as described in claim 1, in, The at least one machine learning logic includes a single machine learning logic, which includes conditional input (531). The conditional input (531) is set to select among the staining processes associated with these different staining laboratory procedures, thereby obtaining the corresponding output image from the multiple output images from the single machine learning logic.

3. The method as described in claim 2, in, The condition input (531) selects between stained colors from a set of predefined candidate stained colors associated with a predefined staining laboratory procedure.

4. The method as described in claim 2, in, The condition input (531) selects between colorings provided to the condition input according to a target color map that parameterizes the colorings associated with the corresponding staining laboratory process.

5. The method of claim 4, further comprising: - The target color map is determined based on one or more predefined target color maps that are parameterized for staining associated with one or more corresponding further staining laboratory procedures.

6. The method as described in claim 5, in, The determination of the target color map depends on the time evolution associated with the transition from the one or more predefined target color maps to the target color map.

7. The method as described in claim 5 or 6, in, The one or more predefined target color maps include two or more predefined target color maps. The target color map is determined based on a weighted combination of at least two of the two or more predefined target color maps.

8. The method as described in any one of claims 2-6, in, The at least one machine learning logic includes a neural network, which includes a decoder branch and a plurality of decoder heads for the decoder branch, wherein different decoder heads among the plurality of decoder heads are used to obtain different output images among the plurality of output images.

9. The method as described in claim 8, in, The different decoder heads among these multiple decoder heads are selected by setting this conditional input.

10. The method according to any one of claims 1-6, in, The imaging data of the tissue sample includes one or more input images that depict a tissue sample excluding chemical stains associated with the virtual stain and / or including another chemical stain not associated with the virtual stain.

11. The method as described in any one of claims 1-6, in, The imaging data for the tissue sample includes one or more input images that depict the tissue sample, including a given virtual stain with a first coloring. At least one output image depicting the tissue sample includes a second coloring that differs from the first coloring.

12. A method for training at least one machine learning logic by virtually staining tissue samples, the method comprising: - Obtain training imaging data depicting one or more tissue samples. - Obtain multiple reference output images depicting one or more tissue samples or one or more further tissue samples, all including a given chemical staining agent. Different reference output images depict one or more tissue samples or one or more further tissue samples including a given chemical staining agent with different staining methods provided by different staining laboratory procedures. - Train at least one machine learning logic based on the training imaging data and the multiple reference output images.

13. The method as described in claim 1, in, The machine learning logic is trained using the method described in claim 12.

14. An apparatus (701) for virtually staining a tissue sample, comprising a processor (702) configured to: - Obtain imaging data (501-503) depicting tissue samples. The imaging data is processed in at least one machine learning logic (500), which is configured to provide multiple output images (521) all depicting tissue samples including a given virtual stain, the multiple output images depicting tissue samples including different stainings of a given virtual stain associated with different staining laboratory procedures, and - Obtain at least one output image from the plurality of output images from the at least one machine learning logic.

15. The apparatus as claimed in claim 14, in, The at least one machine learning logic includes a single machine learning logic, which includes conditional input (531). The conditional input (531) is set to select among the staining processes associated with these different staining laboratory procedures, thereby obtaining the corresponding output image from the multiple output images from the single machine learning logic.

16. The apparatus of claim 15, in, The condition input (531) selects between stained colors from a set of predefined candidate stained colors associated with a predefined staining laboratory procedure.

17. The apparatus as claimed in claim 15, in, The condition input (531) selects between colorings provided to the condition input according to a target color map that parameterizes the colorings associated with the corresponding staining laboratory process.

18. The apparatus of claim 17, wherein, The processor is further configured as follows: - The target color map is determined based on one or more predefined target color maps that are parameterized for staining associated with one or more corresponding further staining laboratory procedures.

19. The apparatus of claim 18, in, The determination of the target color map depends on the time evolution associated with the transition from the one or more predefined target color maps to the target color map.

20. The apparatus as claimed in claim 18 or 19, in, The one or more predefined target color maps include two or more predefined target color maps. The target color map is determined based on a weighted combination of at least two of the two or more predefined target color maps.

21. The apparatus as claimed in any one of claims 15-18, in, The at least one machine learning logic includes a neural network, which includes a decoder branch and a plurality of decoder heads for the decoder branch, wherein different decoder heads are used to obtain different output images among the plurality of output images.

22. The apparatus of claim 21, in, The different decoder heads among these multiple decoder heads are selected by setting this conditional input.

23. The apparatus as claimed in any one of claims 14-18, in, The imaging data of the tissue sample includes one or more input images that depict a tissue sample excluding chemical stains associated with the virtual stain and / or including another chemical stain not associated with the virtual stain.

24. The apparatus as claimed in any one of claims 14-18, in, The imaging data for the tissue sample includes one or more input images that depict the tissue sample, including a given virtual stain with a first coloring. At least one output image depicting the tissue sample includes a second coloring that differs from the first coloring.

25. An apparatus (701) for virtually staining tissue samples, comprising a processor (702) configured to: - Obtain training imaging data depicting one or more tissue samples. - Obtain multiple reference output images depicting one or more tissue samples or one or more further tissue samples, all including a given chemical staining agent. Different reference output images depict one or more tissue samples or one or more further tissue samples including a given chemical staining agent with different staining methods provided by different staining laboratory procedures. - Train at least one machine learning logic based on the training imaging data and the multiple reference output images.

26. The apparatus of claim 14, in, The machine learning logic is trained by the apparatus as described in claim 25.

Images