System and method for generating digitally colored histological images

Abstract

A computer-implemented method for processing an image, comprising: receiving at least one image of an unsectioned sample; and processing at least the image using a neural network arrangement configured to perform digital staining to obtain a digitally stained histological image of the unsectioned sample. The image of the unsectioned sample is obtained by an imaging system having an excitation light source that provides light to cause the emission of light from one or more substances, such as one or more photoluminescent substances, in the unsectioned sample to image the unsectioned sample.The digitally stained histological image generally corresponds to, or is substantially equivalent to, an image of the non-sectioned specimen that has been chemically stained using one or more chemical stains. The non-sectioned specimen may be an unstained biological specimen with endogenous fluorophores, and the image may be an autofluorescence image of the unstained biological specimen.

Description

TECHNICAL FIELD

[0001] The present invention relates to the generation of digitally stained histological images of samples. BACKGROUND

[0002] Conventional techniques for obtaining histological images of a specimen require relatively complicated and time-consuming preparation procedures for histological specimens to prepare the specimen for imaging. Histological specimen preparation procedures typically include fixing, embedding, and sectioning (cutting into a thin slice), as well as chemically staining the thin slice. The stained thin slice is then imaged using an imaging system, such as a microscope, to obtain a histological image of the specimen. The quality and, consequently, the usability of the resulting histological image would depend on the execution of the histological specimen preparation procedures, the quality of the imaging system used, etc. It is problematic that some histological specimen preparation procedures are time-consuming, labor-intensive, and / or prone to human error. SUMMARY

[0003] According to a first aspect, a computer-implemented method for processing an image is provided. The computer-implemented method comprises receiving at least one image of an unsectioned sample; and processing at least the image using a neural network arrangement configured to perform digital coloring to obtain a digitally colorized histological image of the unsectioned sample. The unsectioned sample is a sample that has not been sectioned into a thin slice having a thickness of about 2 micrometers to about 7 micrometers, as required in conventional histological sample preparation procedures, i.e., the unsectioned sample has a thickness greater than 7 micrometers. Preferably, the unsectioned sample is an optically thick sample having a thickness greater than the depth of field of the imaging system.In some cases, the imaging system has a depth of field of less than one hundred micrometers, so that the thickness of the unsectioned sample is at least on the order of several or several hundred micrometers (e.g., at least 100 micrometers, preferably at least 500 micrometers). The image is obtained by an imaging system having an excitation light source that provides light (i.e., optical radiation) to cause the emission of light from one or more substances (e.g., one or more photoluminescent substances) in the unsectioned sample to image the unsectioned sample. The digitally stained histological image generally corresponds to, or is substantially equivalent to, an image of the unsectioned sample that has been chemically stained using one or more chemical (e.g., histochemical) stains.By using a neural network arrangement to digitally stain the unsectioned sample, a digitally equivalent or comparable version of a chemically stained image of the unsectioned sample can be easily obtained without having to perform the conventional histological sample preparation procedures of sectioning (cutting into a thin slice) and chemically staining, which are often time-consuming and labor-intensive. The digitally stained histological image obtained using the computer-implemented method can be used for diagnosis or preliminary screening of diseases, such as cancer. According to some examples, the digitally stained histological image obtained using the computer-implemented method can be used for diagnosis during surgery.

[0004] The neural network arrangement is configured to take at least the image as input for processing. According to some examples, the neural network arrangement is configured to take a single image (i.e., the image) as input to provide the digitally stained histological image. According to some examples, the neural network arrangement is configured to take multiple images (i.e., the image and one or more other images) as input to provide the digitally stained histological image. According to these examples, the multiple images may include multiple images of the unsectioned sample obtained by the imaging system having multiple excitation light sources, each for providing respective light (i.e., optical radiation) to detect respective light emission from one or more substances (e.g.,one or more photoluminescent substances) in the non-sectioned sample.

[0005] The neural network array may include one or more neural networks, e.g., one or more deep learning-based generating models. According to some examples, the neural network array may be configured to perform only digital inking. According to some examples, the neural network array may be configured to perform one or more further image processing operations (in addition to digital inking).

[0006] According to some embodiments, the imaging system is for performing fluorescence imaging, such as wide-field fluorescence imaging, where the image may be a fluorescence image. According to some examples, the imaging system for performing fluorescence imaging may be used to perform only fluorescence imaging. According to some other examples, the imaging system for performing fluorescence imaging may be used to perform one or more other imaging methods.

[0007] According to some embodiments, the one or more substances comprise fluorophores. The fluorophores may include endogenous fluorophores (naturally derived from the unsectioned sample) and / or exogenous fluorophores (added to the unsectioned sample). According to embodiments in which the one or more substances consist only of endogenous fluorophores, the fluorescence image is an autofluorescence image.

[0008] According to some embodiments, the unsectioned sample is an in vivo sample and the imaging system is for performing in vivo imaging.

[0009] According to some embodiments, the imaging system is a portable (e.g., handheld) imaging system.

[0010] According to some embodiments, the imaging system is used to perform fluorescence microscopy, such as wide-field fluorescence microscopy. The imaging system may, for example, include a fluorescence microscope, such as a wide-field fluorescence microscope.

[0011] According to some embodiments, the imaging system is used to perform fluorescence endoscopy (endoscopic fluorescence imaging). The imaging system may, for example, include an endoscopic imaging system.

[0012] According to some embodiments, the excitation light source provides ultraviolet light (having one or more wavelengths between about 100 nm and about 400 nm) to cause the emission of light from one or more substances in the unsectioned sample. According to some embodiments, the ultraviolet light comprises ultraviolet light in the UVC range (e.g., having one or more wavelengths between about 100 nm and about 280 nm, or having one or more wavelengths between about 200 nm and about 280 nm, or having one or more wavelengths between about 200 nm and about 230 nm, or having a wavelength of about 265 nm).

[0013] According to some embodiments, the excitation light source comprises a light-emitting diode (LED). According to some embodiments, the excitation light source comprises a UV LED operable to provide the ultraviolet light. The UV LED is a relatively inexpensive means for providing ultraviolet light.

[0014] According to some embodiments, the imaging system may include one or more additional excitation light sources, each configured to provide respective light to cause the emission of light from the one or more substances in the unsectioned sample to obtain one or more additional images of the unsectioned sample. The one or more additional excitation light sources may, for example, be one or more UV LEDs, each configured to provide ultraviolet light (having one or more wavelengths between about 100 nm and about 400 nm). According to one example, the excitation light source and the one or more additional excitation light sources each include a UV LED. The excitation light source and the one or more additional excitation light sources may provide the same ultraviolet light (the same wavelength(s), e.g.,at 265 nm) or can provide different ultraviolet lights (one or more different wavelengths).

[0015] According to some embodiments, the neural network arrangement is configured to perform image enhancement and digital coloring to provide a digitally colorized histological image. Because the unsectioned sample is an optically thick sample, the light (provided by the imaging system) passing through the unsectioned sample may be scattered by various components at different depths of the unsectioned sample. As a result of such scattering and / or the configuration of the imaging system, the image of the unsectioned sample obtained using the imaging system may be of reduced quality. Performing image enhancement may compensate for the reduced quality to allow a lower quality image of the unsectioned sample to be processed for digital coloring. This may exceed the quality requirement of the imaging system (e.g.,any one or more of its parts such as the excitation light source, the camera / sensor, etc.) for imaging an unsectioned sample and / or may allow an optically thicker unsectioned sample to be processed for digital coloring using the computer-implemented method. The neural network arrangement may include one or more neural networks. The neural network arrangement may be configured to perform image enhancement and digital coloring sequentially (one after the other, in any order) or substantially simultaneously. According to some examples, the neural network arrangement may be configured to perform one or more further operations.

[0016] According to some embodiments, the neural network assembly has been trained using a training dataset. The training dataset may include multiple images (e.g., of multiple samples, such as unsectioned samples) obtained by one type of imaging system (e.g., with one type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) of the samples), and multiple corresponding chemically stained histological images (e.g., of multiple thin slices of the multiple samples that have been sectioned (if these samples are unsectioned samples or are otherwise needed) and stained using the same chemical (e.g., histochemical) stain).The type of imaging system for obtaining the multiple images may be the same type as the imaging system for obtaining the image to be processed by the neural network arrangement. According to one example, the imaging system for obtaining the multiple images may be the same type of imaging system for obtaining the image to be processed by the neural network arrangement.

[0017] According to some embodiments, the plurality of images of the training dataset may include labeled data (e.g., pixel-level aligned or registered images), wherein the array of a neural network may be trained using the labeled data according to supervised learning. According to some embodiments, the plurality of images of the training dataset may include unlabeled data (e.g., images that are not aligned or registered), wherein the array of a neural network may be trained using the unlabeled data according to unsupervised learning (e.g., self-supervised learning). According to some embodiments, the plurality of images of the training dataset may include partially labeled data (e.g., patch-level aligned or registered images), wherein the array of a neural network may be trained using the partially labeled data according to weakly supervised learning.

[0018] According to some embodiments, image enhancement includes increasing spatial resolution and / or contrast enhancement. As mentioned, the light (provided by the imaging system) passing through the unsectioned sample may be scattered by various components at different depths of the unsectioned sample. As a result of such scattering and / or configuration of the imaging system, in some cases, the image of the unsectioned sample obtained using the imaging system may have reduced spatial resolution and / or contrast. Performing an increase in spatial resolution and / or contrast enhancement may be useful to compensate for the reduced spatial resolution and / or contrast, improve the scattering problem, and / or address the limitations presented by the imaging system.According to some embodiments, the arrangement of a second neural network is configured to perform at least one of an increase in spatial resolution and a contrast enhancement.

[0019] According to some embodiments, increasing spatial resolution comprises increasing axial resolution and / or increasing lateral resolution. As mentioned, scattering and / or the configuration of the imaging system may result in reduced spatial resolution and / or reduced contrast. In some cases, a thicker sample with more components at greater depths of the unsectioned sample would result in degradation of image contrast. Consequently, in some cases, performing the increase in axial resolution may be useful to reduce imaging thickness and consequently improve image contrast. In some cases, an imaging system may obtain a lower lateral resolution image of a region of a sample at a faster speed and may obtain a higher lateral resolution image of the same region of the sample at a slower speed.Consequently, performing an increase in lateral resolution may be useful to compensate for the reduced lateral resolution or to enable imaging of the sample to be performed more quickly. According to some embodiments, the arrangement of a second neural network is configured to perform at least one of an increase in axial resolution and an increase in lateral resolution.

[0020] According to some embodiments, the neural network array comprises a first neural network array configured to perform digital colorization and a second neural network array configured to perform image enhancement. Using different neural network arrays to perform digital colorization and image enhancement may allow the digital colorization performance and the image enhancement performance to be optimized or tuned separately and / or independently. This, in turn, may improve the performance of the neural network array in performing one or both tasks. The first neural network array may include one or more neural networks. The second neural network array may include one or more neural networks.

[0021] The array of a second neural network may include a deep learning-based generating model. According to some embodiments, the array of a second neural network includes a generating adversarial network (GAN)-based neural network (i.e., a GAN-based generating model). According to some examples, the GAN-based neural network may include a generating adversarial network with super-resolution (SRGAN) or its improved version, such as an improved generating adversarial network with super-resolution (ESRGAN). According to some embodiments, the GAN-based neural network may include an attention mechanism (i.e., a GAN-based generating model with attention mechanism). According to some embodiments, the array of a second neural network includes a diffusion-based neural network (i.e., a diffusion-based model).

[0022] According to some embodiments, the array of a second neural network has been trained using a training dataset comprising multiple images containing lower-quality images and corresponding higher-quality images of multiple samples (e.g., unsectioned samples). The lower-quality image may have lower axial resolution, lower lateral resolution, and / or lower contrast than the corresponding higher-quality image.

[0023] According to some embodiments, the training dataset for training the array of a second neural network includes labeled data, wherein the array of a second neural network has been trained using the labeled data according to supervised learning. According to some examples, for the labeled data, each of the plurality of lower-quality images is registered or aligned at the pixel level to its corresponding high-quality image. The registration or alignment may be performed digitally.

[0024] According to some embodiments, the training dataset for training the array of a second neural network includes partially labeled data, wherein the array of a second neural network has been trained using the partially labeled data according to weakly supervised learning. According to some examples, for the partially labeled data, each of the plurality of lower-quality images is registered or aligned to its corresponding high-quality image at the patch level (each patch may contain multiple pixels). The registration or alignment may be performed digitally.

[0025] According to some embodiments, the training dataset for training the array of a second neural network includes unlabeled data, wherein the array of a second neural network has been trained using the unlabeled data according to unsupervised learning (e.g., self-supervised learning). According to some examples, for the unlabeled data, each of the plurality of lower-quality images is not registered or aligned to its corresponding high-quality image at the pixel and patch levels. In some cases, the registration or alignment has not been specifically performed on the images.

[0026] According to some embodiments, the plurality of images of the training dataset includes a plurality of image pairs, each image pair including a lower quality image and a corresponding higher quality image of a sample (e.g., an unsectioned sample). Each image pair may be for a respective sample. According to some embodiments, for each image pair, the lower quality image and the corresponding higher quality image are obtained by the same imaging system, which includes a first excitation light source providing light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample for imaging the sample to obtain the lower quality image, and a second excitation light source (different from the first excitation light source) providing light to cause the emission of light from one or more substances (e.g.,photoluminescent substances) in the sample to image the sample to obtain the corresponding higher-quality image. According to some examples, by using the same imaging system with different excitation light sources to obtain the corresponding lower- and higher-quality images, less effort may be required for registering or aligning the corresponding lower- and higher-quality images. According to some examples, the imaging system is for wide-field fluorescence microscopy. According to some examples, the first excitation light source comprises a UV LED and / or the second excitation light source comprises a UV laser.

[0027] According to some embodiments, the lower-quality images are obtained by a first type of imaging system having a first type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample, whereas the corresponding higher-quality images are obtained by a second type of imaging system having a second type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample. According to some examples, the first type of imaging system differs from the second type of imaging system. As such, different types of imaging systems may be used to obtain the corresponding lower- and higher-quality images.According to some examples, the first type of imaging system is an imaging system for wide-field fluorescence microscopy and / or the second type of imaging system is an imaging system for light-sheet fluorescence microscopy. According to some examples, the first type of excitation light source differs from the second type of excitation light source. According to some examples, the first type of excitation light source comprises a UV LED and / or the second type of excitation light source comprises a UV laser.

[0028] The array of a first neural network may include a deep learning-based generating model. According to some embodiments, the array of a first neural network includes a generating adversarial network (GAN)-based neural network (i.e., a GAN-based generating model). The GAN-based neural network may include a conditional generating adversarial network (cGAN). According to some embodiments, the GAN-based neural network may include an attention mechanism (i.e., a GAN-based generating model with attention mechanism). According to some embodiments, the array of a first neural network includes a diffusion-based neural network (i.e., a diffusion-based model).

[0029] According to some embodiments, the assembly of a first neural network has been trained using a training dataset comprising a plurality of images including a plurality of first images of a plurality of samples (e.g., unsectioned samples), each obtained by a type of imaging system having a type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample before the samples are chemically stained, and a plurality of corresponding second chemically stained histological images of the samples, each obtained after the respective sample has been sectioned into a thin slice (if necessary) and chemically stained (using the one or more chemical (e.g., histochemical) stains, the assembly of a neural network being configured to digitally mimic them).According to some of these embodiments, the type of imaging system is a wide-field fluorescence microscopy imaging system. According to some of these embodiments, the type of excitation light source is an excitation light source comprising a UV laser or a UV LED.

[0030] According to some embodiments, the training dataset for training the array of a first neural network includes labeled data, wherein the array of a first neural network has been trained using the labeled data according to supervised learning. According to some examples, for the labeled data, each of the plurality of first images is registered or aligned at the pixel level to its corresponding second image. The registration or alignment may be performed digitally.

[0031] According to some embodiments, the training dataset for training the array of a first neural network includes partially labeled data, wherein the array of a first neural network has been trained using the partially labeled data according to weakly supervised learning. According to some examples, for the partially labeled data, each of the plurality of first images is registered or aligned to its corresponding second image at the patch level (each patch may contain multiple pixels). The registration or alignment may be performed digitally.This can be useful for first images obtained from non-sectioned samples, since the first images and the second images of the sectioned and stained samples (obtained by conventional histological sample preparation procedures) would be unlikely to represent the exact same surface of the sample, and consequently would be difficult to register or align at the pixel level.

[0032] According to some embodiments, the training dataset for training the array of a first neural network includes unlabeled data, wherein the array of a first neural network has been trained using the unlabeled data according to unsupervised learning (e.g., self-supervised learning). According to some examples, for the unlabeled data, each of the plurality of lower-quality images is not registered or aligned to its corresponding high-quality image at the pixel and patch level. In some cases, the registration or alignment has not been specifically performed on the images.This can be useful for first images obtained from non-sectioned samples, since the first images and the second images of the sectioned and stained samples (obtained by conventional histological sample preparation procedures) would be unlikely to represent the exact same surface of the sample, and consequently would be difficult to register or align.

[0033] According to some embodiments, the output of the second neural network array is operatively connected to the input of the first neural network array. The output of the second neural network array may, for example, be directly connected to the input of the first neural network array. According to some embodiments, processing the image comprises processing the image using the second neural network array to obtain an enhanced image, and processing the enhanced image using the first neural network array to obtain the digitally colorized histological image. According to some embodiments, the target / output images of the training dataset for training the second neural network array may also be used as the input images of the training dataset for training the first neural network array.This may represent a more efficient use of the data and may better operationally couple the arrangements of a first and a second neural network.

[0034] According to some embodiments, the image of the non-sectioned sample is a grayscale image. According to some other embodiments, the image of the non-sectioned sample is a colorized image.

[0035] According to some embodiments, the digitally stained histological image of the unsectioned sample is a grayscale image. According to some other embodiments, the digitally stained histological image of the unsectioned sample is a colorized image.

[0036] According to some embodiments, the unsectioned sample comprises a biological sample, such as a tissue. According to some other embodiments, the unsectioned sample comprises a tissue-mimicking sample (e.g., a phantom). According to some embodiments, the biological sample is taken (e.g., excised) from a human, an animal, or a plant. According to some embodiments, the biological sample is an in vivo biological sample (e.g., not excised or otherwise taken from a living human, a living animal, or a living plant). According to some embodiments, the unsectioned sample may be an intact and minimally processed sample.

[0037] According to some embodiments, the unsectioned sample is not arranged on or in a slide and is not arranged between slides, and thus can be imaged based on a slide-free imaging technique. According to some embodiments in which the unsectioned sample comprises a tissue, the tissue can be a fixed tissue (e.g., a tissue fixed in formalin) or an unfixed tissue (e.g., a fresh tissue), wherein the tissue can be embedded in paraffin or not. The tissue has not been sectioned, as required in conventional preparation procedures for histological samples, i.e., the tissue has a thickness of more than 7 micrometers. The tissue is preferably an optically thick tissue, having a thickness greater than the depth of field of the imaging system. The thickness of the tissue is, for example,at least on the order of several or several hundred micrometers (e.g., at least 100 micrometers, preferably at least 500 micrometers). According to some examples, the unsectioned sample consists only of the tissue. According to some examples, the unsectioned sample comprises a formalin-fixed, paraffin-embedded tissue block containing the tissue.

[0038] According to some embodiments, the non-sectioned sample is an unstained sample for which the imaging system can perform label-free imaging. An unstained sample is a sample that has not been chemically stained (i.e., not labeled) and thus enables label-free imaging.

[0039] According to some embodiments, the digitally stained histological image generally corresponds to, or is substantially equivalent to, an image of the unsectioned sample that has been chemically stained using a hematoxylin and eosin (H and E) stain. According to some embodiments, the digitally stained histological image generally corresponds to, or is substantially equivalent to, an image of the unsectioned sample that has been chemically stained using one or more specific stains. According to some embodiments, the digitally stained histological image generally corresponds to, or is substantially equivalent to, an image of the unsectioned sample that has been chemically stained using one or more immunohistochemical stains.

[0040] According to some embodiments, the computer-implemented method comprises outputting the digitally colorized histological image. According to some embodiments, the computer-implemented method comprises displaying the digitally colorized histological image. According to some embodiments, the computer-implemented method comprises storing the digitally colorized histological image. According to some embodiments, the computer-implemented method comprises storing the image and the digitally colorized histological image for use as training data for training the neural network assembly.

[0041] According to a second aspect, an image processing system is provided, comprising one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs contain instructions for carrying out the computer-implemented method of the first aspect. The image processing system may be part of an imaging system, e.g., the imaging system that images the unsectioned sample. According to some examples, the image processing system may be incorporated into the imaging system. According to some examples, the image processing system may be remote from the imaging system and operatively connectable to the imaging system. According to some examples, the memory may store the layout of a neural network.

[0042] According to a third aspect, a carrier medium is provided carrying computer-readable instructions adapted to cause one or more processors to carry out the computer-implemented method of the first aspect.

[0043] According to a fourth aspect, a non-transitory computer-readable medium is provided storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing the computer-implemented method of the first aspect.

[0044] According to a fifth aspect, a method for digitally staining an unsectioned sample is provided. The method comprises imaging an unsectioned sample using an imaging system having an excitation light source that provides light to cause the emission of light from one or more substances in the unsectioned sample to obtain an image of the unsectioned sample. The one or more substances may include one or more photoluminescent substances, such as fluorophores. The method further comprises performing the computer-implemented method of the first aspect. According to this fifth aspect, the imaging system having the excitation light source may include one or more of the features of the imaging system recited in the computer-implemented method of the first aspect.

[0045] According to a sixth aspect, a system is provided comprising: an imaging system having an excitation light source that provides light to cause the emission of light from one or more substances in an unsectioned sample to image the unsectioned sample, and the image processing system of the second aspect. The one or more substances may include one or more photoluminescent substances, such as fluorophores. According to this sixth aspect, the imaging system having the excitation light source may include one or more of the imaging system features listed in the second aspect. The imaging system in the system of the sixth aspect may be a portable or handheld imaging system.

[0046] According to a seventh aspect, a method is provided for training a neural network array to perform digital colorization. The method comprises obtaining a training dataset comprising a plurality of images, and training the neural network array using the training dataset such that the neural network array is configured to perform digital colorization (and optionally one or more further image processing operations, such as image enhancement). The neural network array may include one or more neural networks, e.g., one or more deep learning-based generating models. The neural network array, after training, may be the neural network array in the computer-implemented method of the first aspect.

[0047] According to some embodiments, the training dataset may include a plurality of images (e.g., of a plurality of samples) obtained by an imaging system (e.g., having an excitation light source that provides light to cause the emission of light from one or more substances, such as one or more photoluminescent substances), and a plurality of corresponding chemically stained histological images (e.g., of the plurality of samples that have been sectioned (if necessary) and stained using the same one or more chemical (e.g., histochemical) stains).

[0048] According to some embodiments, the neural network array includes a first neural network array configured to perform digital colorization and a second neural network array configured to perform image enhancement, such as increasing spatial resolution (e.g., axial resolution and / or lateral resolution) and / or contrast enhancement. As such, training the neural network array may include training the first neural network array and training the second neural network array. In some cases, training may include training only the neural network array that needs to be trained. The first neural network array may include one or more neural networks. The second neural network array may include one or more neural networks.The training data set may include a training data set for training the arrangement of a first neural network and a training data set for training the arrangement of a second neural network.

[0049] According to some examples, the output of the arrangement of a second neural network is operatively connected directly or indirectly to the input of the arrangement of a first neural network.

[0050] According to some embodiments, the second neural network array comprises a deep learning-based generating model. According to some embodiments, the second neural network array comprises a generating adversarial network (GAN)-based neural network (i.e., a GAN-based generating model). According to some examples, the GAN-based neural network may include a generating adversarial network with super-resolution (SRGAN) or its improved version, such as an improved generating adversarial network with super-resolution (ESRGAN). According to some embodiments, the GAN-based neural network may include an attention mechanism (i.e., a GAN-based generating model with attention mechanism). According to some embodiments, the second neural network array comprises a diffusion-based neural network (i.e., a diffusion-based model).

[0051] According to some embodiments, the training dataset for training the assembly of a second neural network comprises a plurality of images containing lower-quality images and corresponding higher-quality images of a plurality of samples (e.g., unsectioned samples). The lower-quality image may have a lower axial resolution, a lower lateral resolution, and / or a lower contrast than the corresponding higher-quality image. The plurality of samples may include biological samples, such as tissue samples. The plurality of samples may be an intact and minimally processed sample. According to some embodiments, the method further comprises obtaining the training dataset.

[0052] According to some embodiments, the training dataset for training the array of a second neural network includes labeled data, wherein the array of a second neural network has been trained using the labeled data according to supervised learning. According to some examples, for the labeled data, each of the plurality of lower-quality images is registered or aligned at the pixel level to its corresponding high-quality image. The registration or alignment may be performed digitally. According to some embodiments, the method further comprises registering or aligning the corresponding lower- and higher-quality images at the pixel level to obtain the labeled data.

[0053] According to some embodiments, the training dataset for training the array of a second neural network includes unlabeled data, wherein the array of a second neural network has been trained using the unlabeled data according to unsupervised learning (e.g., self-supervised learning). According to some examples, for the unlabeled data, each of the plurality of lower-quality images is not registered or aligned to its corresponding high-quality image at the pixel level and at the patch level, respectively. For example, the registration or alignment has not been specifically performed on the images.

[0054] According to some embodiments, the training dataset for training the array of a second neural network includes partially labeled data, wherein the array of a second neural network has been trained using the partially labeled data according to weakly supervised learning. According to some examples, for the partially labeled data, each of the plurality of lower quality images is registered or aligned at the patch level (each patch may include multiple pixels) to its corresponding high quality image. The registration or alignment may be performed digitally. According to some embodiments, the method further comprises registering or aligning the corresponding lower and higher quality images at the patch level to obtain the partially labeled data.

[0055] According to some embodiments, the plurality of images of the training data set for training the assembly of a second neural network includes a plurality of image pairs, each image pair including a lower quality image and a corresponding higher quality image of a corresponding sample (e.g., an unsectioned sample).

[0056] According to some embodiments, for each respective image pair, the lower-quality image and the corresponding higher-quality image are obtained by the same imaging system, which includes a first excitation light source providing light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample to obtain the lower-quality image, and a second excitation light source (different from the first excitation light source) providing light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample to obtain the higher-quality image. According to some examples, the imaging system is for wide-field fluorescence microscopy. According to some examples, the first excitation light source comprises a UV LED and / or the second excitation light source comprises a UV laser.

[0057] According to some embodiments, the lower quality images are obtained by a first type of imaging system having a first type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample, whereas the corresponding higher quality images are obtained by a second type of imaging system having a second type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample. According to some examples, the first type of imaging system differs from the second type of imaging system. The first type of imaging system is, for example,an imaging system for wide-field fluorescence microscopy and / or the second type of imaging system is an imaging system for light-sheet fluorescence microscopy. According to some examples, the first type of excitation light source differs from the second type of excitation light source. According to some examples, the first type of excitation light source comprises a UV LED and / or the second type of excitation light source comprises a UV laser.

[0058] According to some embodiments, the first neural network array comprises a deep learning-based generating model. According to some embodiments, the first neural network array comprises a generating adversarial network (GAN)-based neural network (i.e., a GAN-based generating model). The GAN-based neural network may include a conditional generating adversarial network (cGAN). According to some embodiments, the GAN-based neural network may include an attention mechanism (i.e., a GAN-based generating model with attention mechanism). According to some embodiments, the first neural network array comprises a diffusion-based neural network (i.e., a diffusion-based model).

[0059] According to some embodiments, the training dataset for training the assembly of a first neural network comprises a plurality of images, each comprising a plurality of first images of a plurality of samples (e.g., unsectioned samples), each obtained by a type of imaging system having a type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample before the samples are chemically stained, and a plurality of corresponding second, chemically stained histological images of the samples, each obtained after the respective sample has been sectioned (if necessary) and chemically stained (using one or more chemical (e.g., histochemical) stains, such as the hematoxylin and eosin (Hu.E.) stain).According to some of these embodiments, the type of imaging system is a wide-field fluorescence microscopy imaging system. According to some of these embodiments, the type of excitation light source is an excitation light source comprising a UV laser or a UV LED.

[0060] According to some embodiments, the training dataset for training the array of a first neural network includes labeled data, wherein the array of a first neural network has been trained using the labeled data according to supervised learning. According to some examples, for the labeled data, each of the plurality of first images is registered or aligned at the pixel level to its corresponding second image. The registration or alignment may be performed digitally. According to some embodiments, the method further comprises registering or aligning the corresponding first and second images at the pixel level to obtain the labeled data.

[0061] According to some embodiments, the training dataset for training the array of a first neural network includes unlabeled data, wherein the array of a first neural network has been trained using the unlabeled data according to unsupervised learning (e.g., self-supervised learning). According to some examples, for the unlabeled data, each of the plurality of first images is not registered or aligned to its corresponding second image. In other words, the registration or alignment has not been specifically performed on the images.

[0062] According to some embodiments, the training dataset for training the array of a first neural network includes partially labeled data, wherein the array of a first neural network has been trained using the partially labeled data according to weakly supervised learning. According to some examples, for the partially labeled data, each of the plurality of first images is registered or aligned at the patch level (each patch may include multiple pixels) to its corresponding second image. The registration or alignment may be performed digitally. According to some embodiments, the method further comprises registering or aligning the corresponding first and second images at the patch level to obtain the partially labeled data.

[0063] According to an eighth aspect, a system is provided, comprising one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for performing the method of the seventh aspect. According to some embodiments, the system of the eighth aspect and the system of the second aspect may be the same system.

[0064] According to a ninth aspect, a carrier medium is provided carrying computer-readable instructions adapted to cause one or more processors to carry out the method of the seventh aspect.

[0065] According to a tenth aspect, there is provided a non-transitory computer-readable medium storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing the method of the seventh aspect.

[0066] According to an eleventh aspect, a method is provided for generating a digitally stained histological image from a sample. The method comprises receiving an autofluorescence image of an unsectioned and slide-free sample; and processing the autofluorescence image using a neural network arrangement to generate a digitally stained histological image of the unsectioned and slide-free sample. The autofluorescence image is obtained by or from an imaging system that uses an LED, such as a UV LED, as the excitation light source. The digitally stained histological image of the unsectioned and slide-free sample generally corresponds to, or is substantially equivalent to, an image of the unsectioned and slide-free sample that has been chemically stained.The array of a neural network can contain one or more trained deep neural networks.

[0067] According to some embodiments, the arrangement of a neural network may be the arrangement of a neural network in the computer-implemented method of the first aspect.

[0068] According to some embodiments, the arrangement of a neural network for generating digitally stained histological images has been trained using a training dataset comprising autofluorescence images (obtained by or from an imaging system that uses an LED, such as a UV LED, as an excitation light source) and corresponding chemically or historically stained images of one or more samples.

[0069] According to some embodiments, the neural network arrangement comprises a neural network arrangement for image enhancement and a neural network arrangement for digital colorization. According to some embodiments, the neural network arrangement for performing image enhancement has been trained using a training dataset comprising a first set of autofluorescence images (obtained by or from an imaging system using an LED, such as a UV LED, as an excitation light source) and a second set of corresponding autofluorescence images (obtained by or from a different imaging system and / or a different excitation light source) with higher image contrast and / or higher spatial resolution than those of the first set.

[0070] According to some embodiments, the output of the neural network array for image enhancement is input to the neural network array for digital colorization.

[0071] According to a twelfth aspect, an imaging system is provided with a UV LED as an excitation light source for exciting the autofluorescence of an unsectioned sample. According to some embodiments, the unsectioned sample is an unstained sample that may or may not be fixed in formalin and / or embedded in paraffin. The imaging system may be a standalone and / or portable imaging system.

[0072] Further features and aspects of the invention will become apparent upon consideration of the detailed description and the accompanying drawings. Any or all features described herein with respect to one aspect or embodiment may be combined with any or all other features described herein with respect to any other aspect or embodiment, where appropriate and applicable.

[0073] "Unsectioned," as used here, means not sectioned into a thin slice with a thickness of approximately 2 micrometers to approximately 7 micrometers, as required by conventional histological specimen preparation procedures. For example, an unsectioned specimen is one that has a thickness greater than 7 micrometers. An unsectioned tissue is one that has a thickness greater than 7 micrometers.

[0074] The terms of extent, such as "generally", "about", "substantially", or the like, as used herein are used to take into account a manufacturing tolerance, deterioration, trend, tendency, one or more imperfect practical conditions, etc., depending on the context.

[0075] Unless otherwise specified, the terms "connected", "coupled", "attached" and the like are intended to include both direct and indirect connection, coupling, attachment, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Some embodiments of the invention will now be described by way of example with reference to the accompanying drawings, in which: Fig. 1 is a schematic diagram illustrating an operation for generating a digitally stained histological image according to some embodiments of the invention; Fig. 2 is a schematic diagram of a neural network arrangement according to some embodiments of the invention; Fig. 3 is a schematic diagram of a system for digitally staining an unsectioned sample according to some embodiments of the invention; Fig. 4 is a schematic diagram of an imaging system according to some embodiments of the invention; Fig. 5 is a block diagram of an information processing system according to some embodiments of the invention; Fig. 6 is a schematic diagram of an operation for generating a digitally stained histological image according to some embodiments of the invention; Fig. 7 is a schematic diagram of an operation for generating a digitally stained histological image according to some embodiments of the invention; Fig. 8 is a schematic diagram illustrating an operation for training a neural network arrangement to generate a digitally stained histological image in accordance with some embodiments of the invention; Fig. 9a shows an enhanced autofluorescence image of a mouse brain according to an example; Fig. 9b a digitally stained histological image corresponding to the enhanced autofluorescence image of the mouse brain in Fig. 9a corresponds; Fig. 9c a wide-field LED image of the Fig. 9a region of the mouse brain marked by box 9a; Fig. 9d an improved image of the Fig. 9a region of the mouse brain marked by box 9a; Fig. 9e a high lateral resolution image of the Fig. 9a region of the mouse brain marked by box 9a; Fig. 9f a digitally colored histological image that corresponds to the wide-field LED image Fig. 9c corresponds; Fig. 9g a digitally colored histological image, which corresponds to the enhanced image Fig. 9d corresponds; Fig. 9h a digitally stained histological image corresponding to the digitally stained histological image Fig. 9e corresponds; Fig. 9i a chemically stained histological image of the Fig. 9a region of the mouse brain marked by box 9a; Fig. 10a shows an enhanced autofluorescence image of a mouse brain according to an example; Fig. 10b a digitally stained histological image corresponding to the enhanced autofluorescence image of the mouse brain in Fig. 10a corresponds; Fig. 10c a wide-field LED image of the Fig. 10a region of the mouse brain marked by box 10a; Fig. 10d a digitally enhanced image of the Fig. 10a region of the mouse brain marked by box 10a; Fig. 10e a high lateral resolution image of the Fig. 10a region of the mouse brain marked by box 10a; Fig. 10f a digitally colored histological image, which corresponds to the wide-field LED image Fig. 10c corresponds; Fig. 10g a digitally stained histological image that corresponds to the digitally enhanced image Fig. 10d corresponds; Fig. 10h a digitally stained histological image corresponding to the high lateral resolution image Fig. 10e corresponds; Fig. 10i a chemically stained histological image of the Fig. 10a region of the mouse brain marked by box 10a; Fig. 11a shows a wide-field LED image of a formalin-fixed thick lung cancer tissue according to an example; Fig. Figure 11b shows a digitally enhanced image of the formalin-fixed thick lung cancer tissue; Fig. 11c a high lateral resolution image of the formalin-fixed thick lung cancer tissue; Fig. 11d an enlarged image of the Fig. 11a area of ​​the formalin-fixed thick lung cancer tissue marked by a box 11a1; Fig. 11e an enlarged image of the Fig. 11a area of ​​the formalin-fixed thick lung cancer tissue marked by a box 11a2; Fig. 11f is an enlarged image of the area indicated by a box 11b1 in Fig. 11b marked area of ​​the formalin-fixed thick lung cancer tissue; Fig. 11g is an enlarged image of the box 11b2 in Fig. 11b marked area of ​​the formalin-fixed thick lung cancer tissue; Fig. 11h an enlarged image of the box 11c1 in Fig. 11c marked area of ​​formalin-fixed thick lung cancer tissue; Fig. 11i an enlarged image of the Fig. 11c area of ​​the formalin-fixed thick lung cancer tissue marked by a box 11c2; Fig. 12a shows a digitally enhanced image of human lung tissue according to an example; Fig. 12b a digitally colored histological image corresponding to the digitally enhanced image Fig. 12a corresponds; Fig. 12c a wide-field LED image of the Fig. 12a area of ​​human lung tissue marked by a box 12a; Fig. 12d a digitally enhanced image of the Fig. 12a area of ​​human lung tissue marked by box 12a; Fig. 12e a corresponding adjacent chemically colored image of the Fig. 12a area of ​​human lung tissue marked by box 12a; Fig. 12f a digitally colored histological image, which corresponds to the wide-field LED image Fig. 12c corresponds; Fig. 12g a digitally colored histological image corresponding to the digitally enhanced image Fig. 12d corresponds; Fig. 13a shows a wide-field LED image of a formalin-fixed thick lung cancer tissue according to an example; Fig. 13b a digitally enhanced image of the formalin-fixed thick lung cancer tissue; Fig. 13c a contrast-enhanced high axial resolution image of formalin-fixed thick lung cancer tissue; Fig. 13d an enlarged image of the Fig. 13a Area of ​​the formalin-fixed thick lung cancer tissue marked by a box 13a; Fig. 13e an enlarged image of the Fig. 13a the area of ​​the formalin-fixed thick lung cancer tissue marked by box 13a; and Fig. 13f an enlarged image of the Fig. 13a Area of ​​formalin-fixed thick lung cancer tissue marked by box 13a. DETAILED DESCRIPTION

[0077] Fig. 1 illustrates an operation for generating a digitally stained histological image according to some embodiments of the invention. Operation 100 is a computer-implemented operation and generally includes a neural network arrangement 102 receiving an image of an unsectioned sample, processing the image of the unsectioned sample, and outputting a digitally stained histological image of the unsectioned sample. The digitally stained histological image of the unsectioned sample generally corresponds to, or is substantially equivalent to, an image of the unsectioned sample that has been chemically stained using one or more chemical (e.g., histochemical) stains.The unsectioned sample is a sample that has not been sectioned into a thin slice of about 2 micrometers to about 7 micrometers, as required in a conventional histological sample processing operation. According to these embodiments, the unsectioned sample may include a biological sample, such as a tissue, with a thickness greater than 7 micrometers. The biological sample may be an in vitro biological sample, an ex vivo biological sample, or an in vivo biological sample. The tissue may be a human, animal, or plant tissue. The tissue may be fixed tissue (e.g., formalin-fixed tissue) or unfixed tissue (e.g., fresh tissue), and may or may not be paraffin-embedded. According to some examples, the sample may be an intact and minimally processed sample. According to some examples, the unsectioned sample is an unstained sample.The digitally stained histological image generally corresponds to, or is substantially equivalent to, an image of the unsectioned specimen that has been chemically stained using one or more types of chemical (e.g., histochemical) stains, such as a hematoxylin and eosin (H and E) stain, one or more specialty stains, or one or more immunohistochemical stains.

[0078] In operation 100, the image of the unsectioned sample, received and processed by the neural network arrangement 102, is obtained by an imaging system having an excitation light source that provides light (optical radiation) to cause the emission of light from one or more substances (e.g., photoluminescent substances, such as fluorescent and / or phosphorescent substances) in the unsectioned sample. The unsectioned sample is an optically thick sample that has a thickness greater than the depth of field of the imaging system used to obtain the image of the unsectioned sample. Since any existing imaging system typically has a depth of field less than one hundred micrometers, the thickness of the unsectioned sample can be at least on the order of several or several hundred micrometers (e.g.,at least 100 micrometers, preferably at least 500 micrometers). According to some embodiments, the imaging system is suitable for performing fluorescence imaging, such as wide-field fluorescence imaging. According to some embodiments, the imaging system is suitable for performing in vivo imaging. According to one example, the imaging system may be suitable for performing fluorescence microscopy, such as wide-field fluorescence microscopy. According to another example, the imaging system may be suitable for performing fluorescence endoscopy (endoscopic fluorescence imaging). The imaging system may be a portable (e.g., handheld) imaging system. According to those embodiments in which the imaging system is suitable for performing fluorescence imaging, the image may be a fluorescence image.Depending on the non-sectioned sample, one or more substances of the non-sectioned sample may contain fluorophores that may be endogenous or exogenous (or both). According to embodiments in which the substances consist of endogenous fluorophores, the image of the non-sectioned sample is an autofluorescence image of the non-sectioned sample.

[0079] According to some embodiments, the light provided by the excitation light source of the imaging system may include ultraviolet light, which may cause the emission of light from one or more substances (e.g., photoluminescent substances) in the unsectioned sample. The ultraviolet light may include ultraviolet light in the UVC range (i.e., deep ultraviolet light). The excitation light source may be an LED-based excitation light source, a laser-based excitation light source, etc., to provide the ultraviolet light.

[0080] As in Fig. 1, the neural network array 102 is configured to take the image as input for processing. The neural network array 102 may include one or more neural networks, such as one or more deep learning-based generating models. The neural network array 102 is configured to perform at least digital colorization. According to some embodiments, the neural network array 102 is configured to perform one or more further image processing operations. The order of performing the digital colorization and the one or more further image processing operations may be sequential (one after the other, in any order) or substantially concurrently.According to one example, the neural network arrangement may be configured to perform at least one of image enhancement and digital coloring to provide the digitally colorized histological image of the unsectioned sample (which has been at least enhanced and digitally colorized).

[0081] The neural network array 102 may be trained using a training dataset containing images of various samples and corresponding chemically stained histological images of the various samples, such as various unsectioned samples. The images of the various samples may be obtained by a type of imaging system having a type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the samples. The type of imaging system may be the same as the type of imaging system for obtaining the image to be processed by the neural network array 102. The type of excitation light source may be the same as the type of excitation light source for obtaining the image to be processed by the neural network array 102.

[0082] Fig. Figure 2 shows a neural network arrangement 200 according to some embodiments of the invention. According to some embodiments, the neural network arrangement 200 may be used like the neural network arrangement 102 in the operation 100 of Fig. 1 can be used.

[0083] As in Fig. As shown in Figure 2, neural network array 200 includes a neural network array 202A configured to perform image enhancement and a neural network array 202B configured to perform digital colorization. Neural network array 202A and neural network array 202B each include at least one neural network. According to some embodiments, the two neural network arrays 202A, 202B may be trained separately and / or independently.

[0084] According to some embodiments, the neural network arrangement 202A configured to perform image enhancement may be configured to perform at least one of spatial resolution (axial resolution and / or lateral resolution) enhancement and contrast enhancement.

[0085] The neural network array 202A configured to perform image enhancement may include a deep learning-based generating model. According to some embodiments, the neural network array 202A may include, for example, a generating adversarial network (GAN)-based neural network (i.e., a GAN-based generating model). The GAN-based neural network generally includes at least one generator network and at least one discriminator network. For example, different GAN-based neural networks may have different numbers of generator networks, different numbers of discriminator networks, different generator network designs (e.g., a different number of convolutional layers, different residual block / skip connections), or different discriminator network designs (e.g., a different number of convolutional layers, different residual block / skip connections).According to some examples, the GAN-based neural network may include a super-resolution generative adversarial network (SRGAN) or its enhanced version, such as an enhanced super-resolution generative adversarial network (ESRGAN). An example architecture of an ESRGAN can be found in Wang et al., “ESRGAN: Enhanced super-resolution generative adversarial networks” (2019), the entire contents of which are incorporated herein by reference. According to some examples, the GAN-based neural network may include a weakly supervised GAN-based model. An example architecture of a weakly supervised GAN-based model can be found in Dai et al., “A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transformation” (2022), the entire contents of which are incorporated herein by reference. According to some embodiments, the neural network array 202A may, for example, be a diffusion-based neural network (i.e.,, a diffusion-based model).

[0086] The neural network arrangement 202A configured to perform image enhancement has been trained using a training dataset comprising lower-quality images and corresponding higher-quality images of multiple samples (e.g., unsectioned samples). The lower-quality image may, for example, have lower axial resolution, lower lateral resolution, and / or lower contrast than the corresponding higher-quality image.

[0087] According to some embodiments, the training dataset includes labeled data, where the neural network array 202A has been trained according to supervised learning. According to one example, the labeled data may be provided by registering or aligning at the pixel level each of the lower-quality images to its corresponding high-quality image.

[0088] According to some embodiments, the training dataset includes partially labeled data, where the neural network array 202A has been trained according to weakly supervised learning. According to one example, the labeled data may be provided by registering or aligning at the patch level (e.g., multiple pixels, blocks, etc.) each of the lower-quality images to its corresponding high-quality image.

[0089] According to some embodiments, the training dataset includes unlabeled data, where the neural network array 202A has been trained according to unsupervised learning (e.g., self-supervised learning). According to one example, the unlabeled data may be provided by not registering or aligning each of the lower-quality images to its corresponding high-quality image at the pixel level and patch level.

[0090] According to some embodiments, the lower quality images and the corresponding higher quality images may form image pairs, each containing a lower quality image and a corresponding higher quality image of a sample (e.g., an unsectioned sample). For each respective image pair, the lower quality image and the corresponding higher quality image are obtained by the same imaging system, which has at least two excitation light sources: a first excitation light source providing light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample to obtain the lower quality image, and a second excitation light source (different from the first excitation light source) providing light to cause the emission of light from one or more substances (e.g.,photoluminescent substances) in the sample to obtain the corresponding higher-quality image. According to some examples, the imaging system is used for wide-field fluorescence microscopy, the first excitation light source includes a UV LED, and the second excitation light source includes a UV laser.

[0091] According to some embodiments, the lower quality images are obtained by a first type of imaging system having a first type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample, whereas the corresponding higher quality images are obtained by a second type of imaging system (e.g., different from the first type of imaging system) having a second type of excitation light source (e.g., different from the first type of excitation light source) that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample. The first type of imaging system is, for example,An imaging system for wide-field fluorescence microscopy and / or the second type of imaging system is an imaging system for light-sheet fluorescence microscopy. The first type of excitation light source contains, for example, a UV LED and / or the second type of excitation light source contains a UV laser.

[0092] The neural network array 202B configured to perform digital inking may include a deep learning-based generating model. According to some embodiments, the neural network array 202B may include, for example, a generating adversarial network (GAN)-based neural network (i.e., a GAN-based generating model). The GAN-based neural network generally includes at least one generator network and at least one discriminator network. Different GAN-based neural networks may, for example, have different numbers of generator networks, different numbers of discriminator networks, different generator network designs (e.g., a different number of convolutional layers, different residual block / skip connections), or different discriminator network designs (e.g., a different number of convolutional layers, different residual block / skip connections).The GAN-based neural network may include a conditional generating adversarial network. According to some examples, the GAN-based neural network may include a pix2pix model. An example architecture of the pix2pix can be found in Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks” (2017), the entire contents of which are incorporated herein by reference. According to some examples, the GAN-based neural network may include a one-sided model. An example architecture of a one-sided model can be found in Shi et al., “One-side Virtual Histological Staining Model for Complex Human Samples” (2022), the entire contents of which are incorporated herein by reference. According to some embodiments, the neural network array 202B may include, for example, a diffusion-based neural network (i.e., a diffusion-based model). An example architecture of a diffusion-based neural network is found in Saharia et al.a., “Palette: Image to-Image Diffusion Models” (2022), the entire contents of which are incorporated herein by reference.

[0093] According to some embodiments, the neural network array 202B has been trained using a training dataset comprising images of samples (e.g., unsectioned samples), each obtained by one type of imaging system with one type of excitation light source that provides light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in the sample before the samples are chemically stained, and corresponding chemically stained histological images of the samples, each obtained after the respective sample has been sectioned (if unsectioned samples are used) and chemically stained (using the one or more chemical (e.g., histochemical) stains, the neural network array being trained to digitally mimic this).According to some examples, the type of imaging system is a wide-field fluorescence microscopy imaging system. According to some examples, the type of excitation light source is an excitation light source that includes a UV laser.

[0094] According to some embodiments, the images and the corresponding chemically stained histological images may form image pairs, each with one of the images and a corresponding chemically stained histological image of a sample (e.g., an unsectioned sample).

[0095] According to some embodiments, the training dataset includes labeled data, wherein the neural network array 202B has been trained according to supervised learning. According to one example, the labeled data may be provided by registering or aligning at the pixel level each respective image of the sample to its corresponding chemically stained histological image.

[0096] According to some embodiments, the training dataset includes partially labeled data, where the neural network array 202B has been trained according to weakly supervised learning. According to one example, the labeled data may be provided by registering or aligning, at a patch level (e.g., multiple pixels, blocks, etc.), each respective image of the sample to its corresponding chemically stained histological image.

[0097] According to some embodiments, the training dataset includes unlabeled data, wherein the neural network array 202B has been trained according to unsupervised learning (e.g., self-supervised learning). According to one example, the unlabeled data may be provided by not registering or aligning each respective pixel-level and spot-level image of the sample to its corresponding chemically stained histological image.

[0098] As an example, for samples that are relatively thin, the thickness of the imaged sample would tend to match the thickness of the subsequently imaged chemically stained histological sample (the sample after histochemical processing) so that registration and / or alignment of the two images at the pixel or spot level can be performed more easily or accurately.

[0099] As an example, for samples that are relatively thick (e.g., more than 7 micrometers), since the thickness of the imaged sample tends to be completely different (incorrect) from the thickness of the subsequently imaged chemically stained histological sample (the sample after histochemical processing), the contents of the two images may be more different and the registration and / or alignment of the two images may be less easy or less accurate.

[0100] According to some embodiments, neural network array 202A and neural network array 202B may be connected in series. According to one example, the output of neural network array 202A may be directly connected to the input of neural network array 202B, such that the image of the unsectioned sample is first processed by neural network array 202A to obtain an enhanced image, and the enhanced image is then processed by neural network array 202B to obtain the digitally stained histological image.

[0101] Fig. 3 shows a system 300 for digitally staining a sample according to some embodiments of the invention. As in Fig. 3, the system 300 includes an imaging system 302 for obtaining an image of a sample and an image processing system 304 for processing the image obtained by the imaging system.

[0102] The imaging system 302 includes one or more excitation light sources, each arranged to provide respective light to cause the emission of light from one or more substances (e.g., photoluminescent substances) in a non-sectioned sample to obtain one or more images of the non-sectioned sample, such as the image of the non-sectioned sample in the operation 100 according to Fig. 1.

[0103] The imaging system 302 may be suitable for performing fluorescence imaging, such as wide-field fluorescence imaging. According to one example, the imaging system 302 may be a fluorescence microscope, such as a wide-field fluorescence microscope. According to another example, the imaging system 302 may be an in vivo imaging system, such as a fluorescence endoscope (endoscopic fluorescence imaging system).

[0104] The image processing system 304 may be configured to perform the operation 100 after Fig. 1 or any of the arrangements 102, 200 of a neural network according to the Fig. 1 and Fig. 2 to operate.

[0105] The image processing system 304 may be operatively connected to the imaging system 302. According to some embodiments, the image processing system 304 may be in data communication with the imaging system 302 via one or more communication links. According to some embodiments, at least a portion of the image processing system 304 may be part of the imaging system 302. According to some embodiments, the image processing system 304 and the imaging system 302 are separate systems.

[0106] Fig. 4 shows an imaging system 400 according to some embodiments of the invention. According to some embodiments, the imaging system 400 can be used as the imaging system for obtaining the image of the sample in the operation 100 according to Fig. 1 or the imaging system according to Fig. 3 can be used.

[0107] In Fig. 4, the imaging system 400 includes a carrier 402 for supporting an unsectioned biological tissue sample S to be imaged or being imaged, an excitation light source 404 configured to provide ultraviolet light (e.g., deep ultraviolet light) to cause the emission of light from one or more fluorophores in the unsectioned biological tissue sample S, an optical system 406 including one or more optical assemblies or arrays, and a camera / sensor 408 configured to capture a fluorescence image (or an autofluorescence image if the fluorophores are entirely endogenous) of the unsectioned biological tissue sample S.The unsectioned biological tissue sample S is optically thick (relative to the imaging system 400), so that the ultraviolet light provided by the excitation light source 404 is scattered by the biomolecules at multiple depths in the sample S, and consequently, the obtained image of the unsectioned biological tissue sample S may suffer from a loss of resolution and / or contrast. The imaging system 400 further includes a data processing system 410 operatively connected to the camera / sensor 408 for processing the data obtained by the camera / sensor 408. The imaging system 400 further includes a movement mechanism 412 for moving the carrier 402 (hence, the sample of the carrier 402) relative to the excitation light source 404, the optical system 406, and / or the camera / sensor 408.The movement mechanism 412 may be motorized and / or may relatively move (e.g., translate) the carrier 402 in at least one dimension, preferably in two dimensions, and more preferably in three dimensions.

[0108] According to some embodiments, the excitation light source 404 includes an LED, e.g., a UV LED. According to some embodiments, the excitation light source 404 additionally or alternatively includes a laser, e.g., a UV laser. According to embodiments in which the excitation light source 404 includes both the LED and the laser, the LED and the laser can be used selectively.

[0109] According to some embodiments, the data processing system 410 is configured to perform the operation 100 after Fig. 1. According to some embodiments, the data processing system 410 is configured to send the data received from the camera / sensor 408 to another system for further processing.

[0110] According to some embodiments, the unsectioned biological tissue sample S is not disposed on or in a slide and is not disposed between slides (thus enabling slide-free imaging). According to these embodiments, the imaging system 400 configured to image such unsectioned and slide-free tissues or samples may be referred to as a slide-free imaging system.

[0111] Fig. 5 shows an information processing system 500 according to some embodiments of the invention. According to some embodiments, the information processing system 500 can be used as one or more of the following: the image processing system 304 of Fig. 3 or the data processing system 410 in the imaging system 400 according to Fig. 4. According to some embodiments, the information processing system 500 may be used to perform the operation 100 according to Fig. 1 (to operate or facilitate the operation of the neural network arrangement 102) or to operate or facilitate the operation of the neural network arrangement 200.

[0112] The information processing system 500 generally includes suitable components necessary to receive, store, and execute suitable computer instructions, commands, and / or code. As described in Fig. 5, the information handling system 500 includes a processor 502 and a memory 504. The processor 502 may include one or more of the following: one or more CPUs, one or more MCUs, one or more GPUs, one or more logic circuits, one or more Raspberry Pi chips, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. The memory 504 may include one or more volatile memories (such as RAM, DRAM, SRAM, etc.), one or more non-volatile memories (such as ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.).) or any combination thereof. Suitable computer instructions, commands, code, information, and / or data (e.g., the one or more arrangements of a neural network) may be stored in memory 504. The computer instructions for carrying out or facilitating carrying out the method embodiments of the invention may be stored in memory 504. The processor 502 and memory 504 may be integrated or separate (and operatively connected).

[0113] Optionally, the information handling system 500 further includes one or more input devices 506. Examples of such input devices 506 include: a keyboard, a mouse, a stylus, an image scanner, a microphone, a tactile / touch input device (e.g., a touch-sensitive screen), an image / video input device (e.g., a camera), etc. Optionally, the information handling system 500 further includes one or more output devices 508. Examples of such output devices 508 include: a display (e.g., a monitor, a screen, a projector, etc.), a speaker, a headset, an earphone, a printer, an additive manufacturing machine (e.g., a 3D printer), etc. The display may include an LCD display, an LED / OLED display, or other suitable display that may be touch-sensitive.The information handling system 500 may further include one or more disk drives 512, which may include one or more of the following: a solid-state drive, a hard disk drive, an optical drive, a flash drive, a magnetic tape drive, etc. A suitable operating system may be installed in the information handling system 500, e.g., on the disk drive 512 or in the memory 504. The memory 504 and the disk drive 512 may be operated by the processor 502. Optionally, the information handling system 500 also includes a communication device 510 for establishing one or more communication connections with one or more other computing devices, such as servers, personal computers, terminals, tablets, telephones, watches, IoT devices, or other computing devices.The communication device 510 may include one or more of the following: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, a Bluetooth® transceiver, a radio frequency transceiver, a cellular transceiver (2G, 3G, 4G, 5G, beyond 5G, or the like), an optical port, an infrared port, a USB connection, or other wired or wireless communication interfaces. The transceiver may be implemented by one or more devices (one or more integrated transceivers, one or more separate transceivers, etc.). The one or more communication connections may be wired or wireless to transmit commands, instructions, information, and / or data.According to one example, the processor 502, the memory 504 (optionally the one or more input devices 506, the one or more output devices 508, the one or more communication devices 510, and the one or more disk drives 512, if present) are connected directly or indirectly via a bus, a peripheral component interconnection (PCI) bus such as PCI Express, a universal serial bus (USB), an optical bus, or other similar bus structure. According to one embodiment, at least some of these components may be wirelessly connected, for example, via a network such as the Internet or a cloud computing network. One of ordinary skill in the art would recognize that the method described in . Fig. 5 is merely an example and that in other embodiments, the information processing system 500 may have other configurations (e.g., include additional components, have fewer components, etc.).

[0114] The following disclosure provides some more specific embodiments of the invention.

[0115] The present inventors have discovered through their research, experiments, and trials that a slide-free imaging technique based on deep ultraviolet excitation light can be useful because various intrinsic biomolecules in biological samples (e.g., tissues) and fluorescent dyes can absorb deep ultraviolet excitation light to provide absorption contrast with or without fluorescence. Such absorption contrast can be equivalent to or similar to histological contrast. According to one example, ultraviolet-based photoacoustic microscopy can be applied for label-free histological imaging based on the absorption contrast provided by deep ultraviolet excitation light and a photoacoustic signal.However, according to this example, a relatively expensive high-repetition-rate pulsed UV laser may be required for high-throughput imaging due to its intrinsic point-scanning mechanism. According to an exemplary wide-field fluorescence imaging technique, the short penetration depth of the deep ultraviolet excitation light may help restrict the excitation of the fluorophores to only the surface of the biological sample (e.g., the unsectioned tissue), and consequently, it may help obtain high image contrast on the surface of the biological sample. According to one example, ultraviolet surface excitation microscopy may be used for histological imaging using a UV-excitable exogenous fluorescent dye. However, according to this example, the use of fluorescent labels (i.e.,, an exogenous fluorescent dye) may influence the subsequent analysis and may be difficult to integrate into clinical practice.

[0116] Zhang et al., “High-throughput, label-free and slide-free histological imaging by computational microscopy and unsupervised learning” (2022), the entire contents of which are incorporated herein by reference, disclosed a label-free imaging technique (method and system) called high-throughput computational autofluorescence microscopy with pattern illumination (CHAMP). CHAMP uses deep ultraviolet excitation light to excite the surface of biological tissue for imaging and can provide contrast-enhanced images based on various endogenous fluorophores, including cellular metabolites (e.g., reduced nicotinamide adenine dinucleotide, flavins), structural proteins (e.g., collagen, elastin), aromatic amino acids (e.g., tyrosine, tryptophan, phenylalanine), and others. The autofluorescence of these fluorophores naturally produces a negative nuclear contrast, which generally corresponds to the contrast media produced by hematoxylin and eosin (H and E).E) corresponds to or is substantially equivalent to colorized images. CHAMP uses pattern illumination to improve lateral resolution while maintaining a large field of view and depth of field using a low numerical aperture (NA) objective. This can improve image contrast of unsectioned tissue and provide a relatively large tolerance in tissue surface profiling. A potential disadvantage of the CHAMP design is that it requires the use of a UV laser as a coherent light source for interference-based speckle pattern generation, and the UV laser is expensive.While an incoherent light source with a spatial light modulator (SLM) or a digital micromirror device (DMD) can be used for sinusoidal structured illumination (SIM) microscopy, and commercially available DMDs have expanded their wavelength range from the visible to the near-infrared and UVA wavelength ranges, a DMD operable in the UVC region is uncommon due to the rapid degradation of DMD reflectivity in such a range. The present inventors recognized that, despite the rich molecular contrast provided by deep ultraviolet excitation light, there are limited, if any, cost-effective illumination tools for deep ultraviolet excitation light, and that this limits the availability of imaging techniques based on deep ultraviolet excitation light.

[0117] According to some embodiments, an imaging system modified based on the CHAMP imaging system is provided, wherein the expensive UV laser in the CHAMP has been replaced with a more cost-effective UV LED (M265L5 Thorlabs Inc.) as the excitation light source. Accordingly, according to some embodiments, a deep learning-based framework is provided for processing images obtained by the modified imaging system.

[0118] According to some embodiments, a trained generator network configured for digital coloring is used to process an autofluorescence image of an unsectioned tissue acquired using a wide-field microscope with an excitation light source including a UV LED providing deep ultraviolet light. According to the following disclosure, the image acquired using such a wide-field microscope is referred to as a "wide-field LED image." Fig. 6 illustrates an exemplary operation of these embodiments. As in Fig. As shown in Figure 6, a low-resolution autofluorescence image of an unsectioned tissue acquired using the wide-field microscope with the UV LED is processed by a trained generator network configured to perform digital staining to generate a digitally stained histological image.

[0119] According to some embodiments, multiple trained image enhancement and digital colorization generator models are used to process an autofluorescence image of an unsectioned tissue acquired using a wide-field microscope with an excitation light source including a UV LED providing deep ultraviolet light. Fig. Figure 7 illustrates an exemplary operation of these embodiments. As in Fig. As shown in Figure 7, a low-resolution autofluorescence image of an unsectioned tissue acquired using the wide-field microscope with the UV LED is processed by a trained generator network configured to perform image enhancement to generate an enhanced image with higher resolution and / or higher contrast, wherein the enhanced image is then processed by a trained generator network configured to perform digital coloring to generate a digitally colorized histological image. According to some examples, the trained generator network configured to perform image enhancement is trained based on a generating adversarial network (GAN)-based resolution augmentation and contrast enhancement algorithm.According to some examples, the trained generator network configured to perform image enhancement includes a GAN-based deep learning algorithm for image enhancement.

[0120] According to some embodiments, the trained generator network configured to perform image enhancement is configured to transform a low-resolution, wide-field LED image (fluorescence image) into a high-resolution image (fluorescence image) corresponding to an image obtained using the CHAMP imaging system (i.e., using a UV laser providing deep ultraviolet light as the excitation light source). According to the following disclosure, such a digitally generated high-resolution image corresponding to an image acquired using the CHAMP imaging system is referred to as an "LED-CHAMP image." To achieve this effect, the trained generator network configured to perform image enhancement may, for example,include a super-resolution algorithm trained using, as input, low-resolution wide-field LED images obtained using the modified imaging system with a UV LED as an excitation light source and, as a target, images obtained using the CHAMP imaging system with a UV laser as an excitation light source. According to the following disclosure, the image acquired using the CHAMP imaging system is referred to as a "laser CHAMP image." According to some embodiments, the generated LED CHAMP image is further processed by the trained generator network configured to perform digital coloring to obtain the digitally colorized histological image that generally corresponds to or is substantially equivalent to a digitally H & E stained histological image.

[0121] According to some embodiments, the trained generator network configured to perform image enhancement may be based on a generative adversarial network, such as a super-resolution generative adversarial network. An example of a generative adversarial network can be found in Goodfellow et al., "Generative Adversarial Networks" (2020), whereas an example of a super-resolution generative adversarial network can be found in Ledig et al., "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network" (2017). According to one embodiment, an enhanced SRGAN (ESRGAN) derived based on an SRGAN with improvements in the network architecture, adversarial loss, and perceptual loss is used. An example of an ESRGAN can be found in Wang et al., "ESRGAN: Enhanced super-resolution generative adversarial networks" (2019).According to other embodiments, other GAN-based networks may also be used for image enhancement.

[0122] Compared to the Laser-CHAMP method (i.e., the CHAMP method disclosed in Zhang et al.), according to some embodiments, the LED-CHAMP method (i.e., the modified CHAMP method according to the above embodiments) can improve image throughput. This is because the LED-CHAMP method only requires a single image to be acquired, whereas the Laser-CHAMP method requires a sequence of illuminated pattern images to be acquired. As a result, the LED-CHAMP method requires less time and effort for image reconstruction and can provide results more quickly.

[0123] The framework according to the above more specific embodiments is used for structured illumination with deep ultraviolet excitation light. However, it should be recognized that the framework can also be applied to other imaging techniques, such as other deep ultraviolet light-based imaging techniques.

[0124] According to some embodiments, the laser CHAMP images obtained using the CHAMP imaging system are used as a bridge to obtain the high-resolution training target for the LED CHAMP super-resolution algorithm.

[0125] According to one example, the wide-field LED images and the laser CHAMP images are acquired using the same inverted wide-field autofluorescence microscope, which uses a 4X objective (RMS4X, NA = 0.1, Thorlabs Inc.) and has two illumination paths (excitation light paths), one for obtaining the wide-field LED images and one for obtaining the laser CHAMP images. The wide-field LED images can be acquired using illumination from a 265 nm LED. The raw laser CHAMP image sequence can be acquired under speckle illumination generated by a 266 nm UV laser and a UV-fused quartz frosted glass diffuser (DGUV10-600, Thorlabs Inc.) as in the disclosed CHAMP system. According to one example, a laser CHAMP image sequence of 36 images is acquired with a sampling interval of 1 µm for each field of view.Then, high-resolution laser CHAMP images can be generated using the raw speckle-illuminated image sequences in CHAMP through an iterative image reconstruction framework. CHAMP's image acquisition and image reconstruction times generally increase with the number of fields of view required to scan the entire sample and can be very time-consuming.

[0126] According to some embodiments of the LED-CHAMP method, the generator network for image enhancement is an ESRGAN trained using the wide-field LED images with the laser CHAMP images. The wide-field LED images and the laser CHAMP images are paired for training. According to one example, the wide-field LED images and the laser CHAMP images are first globally registered for rotation, translation, and scaling using control point registration and then locally registered at the spot level using intensity-based registration with an affine transformation in MATLAB. The paired wide-field LED images and laser CHAMP images can then be used to train and evaluate the ESRGAN model.

[0127] ESRGAN can be trained using aligned data. According to some embodiments, the architecture of the ESRGAN includes a generator G and a discriminator D. The generator G can derive an LED-CHAMP image ŷ = G(x) from the wide-field LED image x with an upscaling factor (e.g., 2× or 3×) according to the upsampling factor used in the Laser-CHAMP reconstruction. According to one example, the discriminator D is based on the relativistic GAN, such as the one disclosed in Jolicoeur-Martineau, “The relativistic discriminator: A key element missing from standard GAN” (2019), where it is trained to classify the LED-CHAMP image ŷ and the original Laser-CHAMP image y. The adversarial loss of the generator G and the discriminator D can be LGRa and LDRa can be expressed with binary cross-entropy loss: LGRa=−Ey[log(1−σ(D(y)−Ey^[D(y^)]))]−Ey^[log(σ(D(y^)+−Ey[D(y)]))] LDRa=−Ey[log(σ(D(y)−Ey^[D(y^)]))]−Ey^[log(1−σ(D(y^)−Ey[D(y)]))] where σ denotes the sigmoid function for post-processing the discriminator output.

[0128] According to these embodiments, the LED-CHAMP image ŷ should share the same content as the original Laser-CHAMP image y. The L1 loss function is used to evaluate the distance between these two types of images: L1=Ex,y[‖y−G(x)‖1]

[0129] To measure the perceptual similarity between the LED-CHAMP image and the Laser CHAMP image, the perceptual loss is calculated using the representative features from the last convolutional layer of the pre-trained VGG19 network. Therefore, the total generator loss is: LG=Lvgg+λLGRa+L1

[0130] According to some embodiments, the generator architecture is based on SRResNet with 12 basic blocks using the residual in dense residual block (RRDB). Fig. Figure 8 illustrates the exemplary generator architecture and exemplary training operation according to these embodiments. According to some embodiments, the generator PixelShuffle method with 2× or 3× upscaling factors is used in the upsampling layer. According to some embodiments, the input wide-field LED images and the output LED-CHAMP images are both single-channel grayscale images. The discriminator architecture is based on PatchGAN to distinguish LED-CHAMP images from Laser-CHAMP images.

[0131] According to some embodiments, a weakly supervised method is used as an alternative to ESRGAN for training imperfectly aligned data. An example of such a weakly supervised method and the associated network structure can be found in Dai et al., "A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transformation." Such a network structure is essentially an extension of the basic GAN architecture with convolutional neural networks. According to these embodiments, the network contains two generators and two discriminators. Given a training sample (S i ,T i ) a generator S transforms i in T i , while the other generator T i into the itransformed. According to the embodiment of the training between the wide-field LED image and the light sheet image (the image obtained using a light sheet microscope with UV laser as the illumination / excitation light source, which will be further described below), S i the wide-field LED image and is T i the light sheet image. According to these embodiments, each generator includes an encoder that projects the input into a latent space shared between the generators, and a decoder that converts from the latent space to the original space.

[0132] According to these embodiments, a reconstruction loss is calculated by the smallest absolute deviations between the input and the decoded image by the same generator: Lrec=∑i|Si−US(ES(Si))|+∑i|Ti−UT(ET(Ti))| where E S and U Sare the encoder and decoder of the source image area and E T and U T the encoder or decoder of the target image area.

[0133] According to these embodiments, the discriminators not only check whether an image is real or fake, but they also check from which region it comes if it is real, which is expressed in the loss function as: CEd=−∑iYiln(DT(Ti))−∑iYiln(DS(Si))−Y0∑iln(DT(UT(ES(Si))))−Y0∑iln(DS(US(ET(Ti)))) may be included, whereby CE d is the cross-entropy loss for updating the discriminator in one iteration. D S and D T are the discriminators, Y i and Y0 are 1-out-of-n coded markers for region i and the fake image, respectively.

[0134] The cross-entropy loss for updating the generator in one iteration is defined as: CEg=−∑iYiln(DT(UT(ES(Si))))−∑iYiln(DS(US(ET(Ti))))

[0135] Apart from cross entropy, the network also contains a cycle consistency loss, which is defined as the sum of the forward consistency from the source to the target image and the backward consistency from the target to the source image: Lcycle=∑i|Si−US(ET(UT(ES(Si))))|+∑i|Ti−UT(ES(US(ET(Ti))))|

[0136] The generators and discriminators according to these embodiments may alternatively be optimized with other loss functions. According to some embodiments, the loss function for the generators may be: Lg=CEg+λ1Lcycle+λ2Lrec where λ1 and λ2 are the weights for the cycle consistency loss and the reconstruction loss, respectively. The loss function for the discriminators is defined as: Ld=CEd

[0137] According to some embodiments, the digital inking generator network may be based on a GAN. According to one example, pairwise data may be prepared by imaging the sample before and after inking. According to one example, for samples of thinner tissues, the digital inking generator network based on a supervised deep learning algorithm referred to as Pix2pix, as disclosed in Isola et al., "Image-to-Image Translation with Conditional Adversarial Networks" (2017), is used for digital inking.

[0138] According to one example, for thicker tissue samples, it is impossible to properly register or align the autofluorescence and histological image pairs because only the H and E layers adjacent to the imaged thick tissue surface can be obtained (due to loss and sample distortion during sample preparation). Consequently, it is difficult to perform training using a fully supervised algorithm; instead, a weakly supervised or unsupervised algorithm is preferred. According to one example, an unsupervised one-side algorithm, such as the one disclosed in Shi et al., "One-side Virtual Histological Staining Model for Complex Human Samples" (2022), can be used for digital staining.

[0139] To better demonstrate the effect of the above embodiments, a comparison between widefield LED images, LED-CHAMP images, and Laser CHAMP images of formalin-fixed paraffin-embedded (FFPE) thin mouse brain slices and formalin-fixed thick human lung cancer samples is shown. According to this example, the samples are fixed in a 10% formalin solution. To obtain a thin FFPE slice of mouse brain tissue, the fixed sample is embedded in paraffin and cut into 5 µm thin slices by a microtome. The tissue slices are placed on a quartz slide and deparaffinized before imaging. The same slice is stained with H and E after imaging according to the standard histology protocol. For the thick tissues, the samples are first imaged, then embedded in a paraffin block after imaging, followed by sectioning, deparaffinization, and H and E.E-staining to acquire the corresponding histological image. According to this example, all of the above H- and E-stained images are digitized into whole slide images using a digital slide scanner (20X, NA = 0.75) (NanoZoomer, Hamamatsu Photonics KK).

[0140] The Fig. 9a to 9i are image results related to the evaluation of the resolution improvement and the performance of downstream digital staining on an optically thin 5 µm FFPE mouse brain tissue according to an example (although not shown as such, the Fig. 9a, Fig. 9c, Fig. 9d, Fig. 9e grayscale images, whereas the Fig. 9b, Fig. 9f, Fig. 9g, Fig. 9am, Fig. 9i are color images). They illustrate a comparison of the wide-field LED image, the image obtained after deep learning-based image enhancement using the high lateral resolution image as the training target (referred to as an “enhanced image,” which is the “LED-CHAMP image”), the high lateral resolution image (as the target, which is the “Laser-CHAMP image”), and the corresponding H- and E-stained image of the thin 5-µm FFPE slices of mouse brain tissue. Fig. Figure 9a is more specifically an enhanced autofluorescence image of a mouse brain. Fig. 9b is the corresponding digitally colored image after Fig. 9a. The Fig. Figures 9c to 9e are enlarged images of the wide-field LED image, the enhanced image (“LED CHAMP image”) and the high lateral resolution image (“Laser CHAMP image”) of the area in box 9a in Fig. 9a. The Fig. 9f to 9h are the digitally colored images, each of which shows the Fig. 9c to 9e correspond. Fig. 9i is the corresponding chemically stained (H and E stained) histological image of the area in box 9a in Fig. 9a. From the Fig. 9f to 9i, it can be noted that while the digitally H- and E-colorized image obtained based on the wide-field LED image shows similar core features as the actual H- and E-colorized image, the digitally H- and E-colorized image obtained based on the LED-CHAMP image shows an improvement over the digitally H- and E-colorized image obtained based on the wide-field LED image.

[0141] By comparing the digitally H- and E-stained image obtained from the LED-CHAMP image with the digitally H- and E-stained image obtained from the wide-field LED image, the nuclei missing from the digitally H- and E-stained image obtained from the wide-field LED image (indicated by arrows) can still be visualized on the digitally H- and E-stained image obtained from the LED-CHAMP image. Using the actual H- and E-stained image as the ground truth, the structural similarity index measures are evaluated on the digitally H- and E-stained images obtained from the wide-field LED image, the LED-CHAMP image, and the Laser-CHAMP image, respectively. The obtained SSIMs are 0.399, 0.402, and 0.405, respectively. The SSIM values ​​show the improvement of digital colorization based on the LED CHAMP image compared to the wide-field LED image.

[0142] The Fig. 10a to 10i are image results related to the resolution improvement and the performance of the downstream digital staining on an optically thick 500 µm FFPE mouse brain tissue according to an example (although not shown as such, the Fig. 10a, Fig. 10c, Fig. 10d, Fig. 10e grayscale images, whereas the Fig. 10b, Fig. 10f, Fig. 10g, Fig. 10am, Fig. 10i color images). They illustrate a comparison of a wide-field LED image, an image obtained after deep learning-based image enhancement using the high lateral resolution image as the training target (referred to as the “enhanced image,” which is the “LED-CHAMP image”), and a high lateral resolution image (as the target, which is the “Laser-CHAMP image”) of thick fixed mouse brain tissue. More specifically, Fig. 10a an enhanced autofluorescence image of a mouse brain. Fig. 10b is the corresponding digitally colored image after Fig. 10a. The Fig. 10c to 10e are enlarged images of the wide-field LED image, the enhanced image (the “LED CHAMP image”) and the high lateral resolution image (the “Laser CHAMP image”) of the area in box 10a in Fig. 10a. The Fig. 10f to 10h are the digitally colored images, each of which shows the Fig. 10c to 10e correspond. Fig. 10i is the corresponding chemically stained (H and E stained) histological image of the area in box 10a in Fig. 10a. According to this example, the LED CHAMP image can be Fig. 10d the densely packed cell nuclei of the hippocampal area are better than the wide-field LED image after Fig. 10c. This result is further supported by a higher SSIM for the LED-CHAMP image (0.809 and 0.746).

[0143] According to this example, three digitally H and E colored images are generated based on the wide-field LED image, the LED CHAMP image, and the laser CHAMP image, respectively, as shown in the Fig. 10f to 10h. Because it is impossible to obtain an H- and E-stained slice that has exactly the same tissue layer as the tissue surface imaged by the dual-mode autofluorescence microscope (used to obtain the images), only an adjacent slice with H- and E-staining is obtained as a reference, as shown in Fig. 10i. While the hippocampal structure can be visualized on the digitally H- and E-stained image based on the wide-field LED image, the performance of digital staining is better in the LED-CHAMP image. In general, a similar nuclear packing density and morphology are observed between the digitally H- and E-stained image based on the LED-CHAMP image, the digitally H- and E-stained image based on the laser-CHAMP image, and the adjacent H- and E-stained image (the Fig. 10g to 10i), whereas in the digitally H- and E-stained image obtained based on the wide-field LED image, low nuclear density with false nuclear delimitation is shown.

[0144] The Fig. 11a to 11i are image results related to the resolution enhancement of an optically thick 500-µm human lung tissue according to an example. They illustrate a comparison of the wide-field LED image, the image obtained after deep learning-based image enhancement using the high-lateral-resolution image as a training target (referred to as an "enhanced image," which is the "LED-CHAMP image"), and the high-lateral-resolution image (as the target, which is the "Laser-CHAMP image") of the formalin-fixed, thick 500-µm lung cancer tissue (human lung adenocarcinoma tissue). Fig. 11a to 11c are more specifically the wide-field LED image, the enhanced image, and the high lateral resolution image of the tissue, respectively. Fig. 11d, Fig. 11f and Fig. 11h are enlarged images of the wide-field LED image, the enhanced image, and the high lateral resolution image, respectively, of the area in a box 11a in Fig. 11a. In the enhanced image, a sharper fiber structure can be observed as a result of the increased resolution. Fig. 11e, Fig. 11g and Fig. 11i are enlarged images of the wide-field LED image, the enhanced image and the high lateral resolution image, respectively, of a wider area in a box 11b in Fig. 11a. The alveolar macrophages are clearly visible in the enhanced image as a result of the improved resolution and contrast. In the LED-CHAMP image, which exhibits improved lateral resolution and contrast compared to the widefield LED image, sharper fiber features and individual cells from a group of alveolar macrophages are more clearly identified. Using the Laser CHAMP image as a reference, the LED-CHAMP image achieves a higher SSIM (0.796) than the widefield LED image (0.743).

[0145] The Fig. 12a to 12g are image results related to the improved performance of digital coloring by the embodiments of the LED-CHAMP method on an optically thick 500 µm human lung tissue according to an example (although not shown as such, the Fig. 12a, Fig. 12c, Fig. 12d grayscale images, whereas the Fig. 12b, Fig. 12th, Fig. 12f, Fig. 12g color images). They illustrate a comparison of the digitally colorized image generated from the wide-field LED image, the digitally colorized image generated from the enhanced image, and the actual chemically stained image of a thick human lung adenocarcinoma. Fig. 12a is more specifically an enhanced image of human lung tissue. Fig. 12b is a digitally colored image after Fig. 12a. The Fig. 12c and Fig. 12d are enlarged images of the wide-field LED image and the enhanced image of the area in a box 12a in Fig. 12a. Fig. 12e is the corresponding neighboring chemically stained image of the solid region. Fig. 12f and Fig. 12g are enlarged digitally enhanced images showing the Fig. 12c and 12d, respectively. The results can be used to validate the proposed workflow for digitally generating histology-like digitally stained images. According to this example, due to the sample complexity of human cancer tissue, the performance of digital staining has been optimized based on the aforementioned unsupervised single-site model. According to this example, the digitally H and E stained image obtained based on the LED-CHAMP image resembles the histology pattern of one of the subtypes of lung adenocarcinoma, acinar adenocarcinoma. However, the digitally H and E stained image generated from the wide-field LED image has a lower resolution and may not reveal the round glandular structure (with the central lumen space filled with carbon particles and blood-like artifacts, as in Fig. 12f). This illustrates that image enhancement can help improve the performance of digital colorization on images of relatively complex samples.

[0146] The deep learning-assisted framework according to the above embodiments may be transferable to other wide-field imaging techniques.

[0147] According to another example, a set of wide-field LED images (of samples) are acquired using a wide-field microscope with a UV LED as the excitation light source for illumination, and a set of image sequences are acquired using an open light sheet microscope with a UV laser as the excitation light source for illumination. According to this example, the two modalities are integrated for comparison so that they share the same detection configuration using a 5X UV objective (LMU-5X-NUV, NA = 0.12, Thorlabs Inc.). The image sequences are further processed to reconstruct a 3-dimensional (3D) volume. According to this example, a 2-dimensional slice of the 3D volume, representing the majority of the tissue surface features, is extracted from the 3D volume.In general, processed light-sheet images (generated and then processed by a light-sheet microscope) can exhibit high axial resolution and, consequently, better imaging contrast than wide-field LED images acquired using a simple wide-field microscope. According to this embodiment, the processed image (i.e., the processed light-sheet image) is used as the image target and, together with the wide-field LED image, is used in training the generator model to generate the enhanced-contrast image output (the generated enhanced-contrast output image is referred to as an "LED-LS" image). According to this example, due to the fundamental difference in imaging thickness between the wide-field LED image and the light-sheet image, a mask generated from the light-sheet image according to tissue content is used to extract the corresponding tissue content in the wide-field LED image.This can help reduce training errors.

[0148] According to this example, a weakly supervised GAN-based model, as disclosed in Dai et al., "A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transformation" (2022), is used for training. Such a weakly supervised model does not require perfect alignment between the wide-field LED image and the high-resolution image target for training.

[0149] The Fig. Figures 13a to 13f are image results related to image enhancement in another widefield imaging technique. They illustrate a comparison of the widefield LED image, the enhanced image (i.e., the LED-LS image), and the processed light sheet image of a formalin-fixed, optically thick 500-µm lung cancer tissue. More specifically, Fig. 13a a wide-field LED image of a formalin-fixed thick lung cancer tissue. Fig. Figure 13b is an enhanced image (i.e., the LED-LS image) of the formalin-fixed thick lung cancer tissue. Fig. Figure 13c is a contrast-enhanced high-axial resolution image (ie, a processed light sheet image) of the formalin-fixed thick lung cancer tissue. Fig. 13d to 13f are enlarged images of the wide-field LED image, the LED LS image and the processed light sheet image of the Fig. 13a shows the area of ​​the formalin-fixed thick lung cancer tissue marked by box 13a. It can be seen that, as a result of the improved axial resolution enabled by light-sheet microscopy, the LED-LS image according to this example can achieve a significant improvement in nuclear contrast and axial resolution compared to the wide-field LED image.

[0150] Some of the above more specific embodiments have provided a deep learning-assisted framework for image enhancement, particularly for resolution enhancement and contrast enhancement, and have demonstrated compatibility with the task of subsequent digital inking to generate histology images based on mouse brain and human lung tissue using a UV LED as the excitation light source for illumination. According to the above examples, only fixed tissues are imaged and tested, but the framework can be readily applied to fresh, unfixed tissue. According to the above examples, the neural network architecture or algorithm for image enhancement and digital inking is executed on a workstation with a Core i9-10980XE CPU @ 4.80 GHz, 8 × 32 GB RAM, and using one NVIDIA GeForce RTX 3090 GPU.Accordingly, according to one of the examples, a 2× upscaling factor is used for the human lung data, with the algorithm using approximately 12 s / 10 mm. 2 required for the calculation, which is much more computationally efficient than reconstruction in CHAMP. It should be noted that the algorithm can be operated even more computationally efficiently with further GPU acceleration, e.g., using a larger number of GPUs and / or one or more more powerful GPUs, making it suitable for use in fresh tissue operating rooms or in vivo imaging.

[0151] The deep learning-assisted image enhancement framework is particularly suitable for generating digitally colorized images for thick samples (e.g., optically thick samples).

[0152] The deep learning-assisted image enhancement framework could, according to some embodiments, be generalized to other wide-field deep-UV imaging techniques for image enhancement. The deep learning-assisted image enhancement framework can be used as a cost-effective alternative to approaches that require a costly UV laser. For example, the deep learning-assisted image enhancement framework can be used as an efficient alternative to time-consuming image enhancement procedures (e.g., extended depth of field and deconvolution for high-resolution scanning). For example, the deep learning-assisted image enhancement framework can be used to reduce UV exposure time by providing a larger field of view for fast and large-area scanning, e.g., in three-dimensional label-free histology in a paraffin block.According to some embodiments, the image enhancement method may be implemented as a computer-assisted function using software, thus reducing or avoiding the need for additional hardware, and may be provided as an add-on function in an imaging device, such as a microscope.

[0153] Some embodiments of the invention have provided an autofluorescence-based slide-free histological imaging technique (system and method) using a light-emitting diode (LED) as the excitation light source and a deep learning algorithm. Some embodiments of the invention can be used to generate a higher-resolution and / or contrast-enhanced image from a lower-resolution image, which can be further processed to generate digitally stained histological images for diagnosis.

[0154] According to some embodiments, a deep learning generating algorithm is used to improve the resolution and / or contrast of images ("widefield LED images") obtained using a widefield microscope with an LED as the excitation light source for illumination. The algorithm can be used to transform the low-resolution widefield LED image into a high-resolution and / or contrast-enhanced output image. According to some embodiments, another deep learning generating algorithm is used to transform the high-resolution and / or contrast-enhanced output image into a digitally colorized histological image.

[0155] According to some embodiments, the integration of a UV LED and the deep learning-assisted algorithm can lead to various improvements. For example, using a UV LED can be more cost-effective than using a UV laser as the excitation light source. Higher imaging speeds can be achieved because only a single-trigger image needs to be acquired for each field of view (instead of a multi-image sequence). For example, the deep learning-assisted algorithm can be computationally efficient because a high-resolution image can be generated in a short inference time. According to some embodiments, the deep learning-assisted framework can be applied to other wide-field deep-UV imaging techniques, such as those that require a UV laser or time-consuming image acquisition and processing.According to some embodiments, the deep learning-assisted framework may be used for other high-resolution learning / training targets, such as images acquired by light sheet microscopy or images acquired by combining deconvolution and extended depth of field.

[0156] According to some implementations, the invention can be used to generate digitally colorized images that can be used in cancer diagnosis. According to some implementations, the deep learning-assisted framework, according to some embodiments, can assist in translating the slide-free imaging technique into a diagnostic support tool. According to one example, the deep learning-assisted framework can be incorporated into an imaging device or a portable imaging device (such as a microscope based on the CHAMP discussed above) to provide additional image enhancement functionality (e.g., as an add-on).

[0157] One or more embodiments described with respect to the figures may be implemented as an application programming interface (API) or as a suite of libraries for use by a developer, or may be included within another software application, such as a terminal or computer operating system or an operating system for a portable computing device, although this is not required. Because program modules according to one or more embodiments include routines, programs, objects, components, and data files that assist in performing specific functions, those skilled in the art will recognize that the functionality of the software application may be distributed across a number of routines, objects, and / or components to achieve the same functionality desired herein.

[0158] It is further recognized that where the methods and systems of the invention are either fully implemented by a computing system or partially implemented by computing systems, then any suitable computing system architecture may be used. This includes standalone computers, network computers, and dedicated or non-dedicated hardware devices. Where the terms "computing system" and "computing device" are used, these terms are intended to include, for example, any suitable arrangement of computer or information processing hardware capable of implementing the described function.

[0159] One skilled in the art will recognize that variations and / or modifications may be made to the described and / or illustrated embodiments of the invention to provide other embodiments of the invention. The described and / or illustrated embodiments of the invention should therefore be considered in all respects as illustrative and not restrictive. Example optional features of some embodiments of the invention are provided in the summary and description. Some embodiments of the invention may include one or more of these example optional features. Some embodiments of the invention may be devoid of one or more of these example optional features.

[0160] For example, according to some embodiments, the neural network arrangement configured to perform at least one digital staining may process multiple images of the unsectioned sample to provide the digitally stained histological image of the unsectioned sample. These multiple images of the unsectioned sample may be obtained from the same imaging system with different excitation light sources, each exciting a corresponding optical response of the unsectioned sample. For example, according to some embodiments, the architecture of the neural network arrangement may be different from those of the illustrated embodiments. While the disclosure focuses on processing (e.g., digital staining) images of an unsectioned sample, the disclosed techniques may also be adapted for processing (e.g., digital staining) images of a sectioned sample. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Wang et al., “ESRGAN: Enhanced super-resolution generative adversarial networks” (2019 [0085, 0121] Dai et al., “A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transformation” (2022 [0085, 0148] Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks” (2017 [0092, 0137] Shi et al., “One-side Virtual Histological Staining Model for Complex Human Samples” (2022

[0092] Saharia u. a., „Palette: Image to-Image Diffusion Models“ (2022

[0092] Zhang u. a., „High-throughput, label-free and slide-free histological imaging by computational microscopy and unsupervised learning“ (2022

[0116] Goodfellow u. a., „Generative Adversarial Networks“ (2020

[0121] Ledig u. a., „Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network“ (2017

[0121] Jolicoeur-Martineau, „The relativistic discriminator: A key element missing from standard GAN“ (2019

[0127] Dai u. a., „A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transformation A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transformation

[0131] Shi u. a., „One side Virtual Histological Staining Model for Complex Human Samples“ (2022

[0138]

Claims

[1] A computer-implemented method for processing an image, the method comprising: Receiving at least one image of an unsectioned sample, the image being obtained by an imaging system having an excitation light source that provides light to cause the emission of light from one or more substances in the unsectioned sample to image the unsectioned sample; and Processing, using a neural network arrangement configured to perform digital coloring, at least the image to obtain a digitally colored histological image of the unsectioned sample, wherein the digitally stained histological image generally corresponds to or is substantially equivalent to an image of the unsectioned specimen that has been chemically stained using one or more chemical stains. [2] The computer-implemented method of claim 1, wherein the neural network arrangement is configured to perform image enhancement and digital coloring to provide the digitally colorized histological image. [3] The computer-implemented method of claim 2, wherein the image enhancement comprises an increase in spatial resolution and / or a contrast enhancement. [4] The computer-implemented method of claim 3, wherein increasing the spatial resolution comprises increasing the axial resolution and / or increasing the lateral resolution. [5] A computer-implemented method according to any one of claims 2 to 4, wherein the arrangement of a neural network comprises: an arrangement of a first neural network configured to perform the digital coloring; and an arrangement of a second neural network configured to perform the image enhancement. [6] Computer-implemented method according to claim 5, wherein the arrangement of a first neural network comprises a deep learning-based generating model; and / or wherein the arrangement of a second neural network comprises a deep learning-based generating model. [7] Computer-implemented method according to claim 5 or 6, wherein the arrangement of a second neural network has been trained using a training data set comprising a plurality of images containing lower quality images and corresponding higher quality images of a plurality of samples; and wherein the lower quality image has lower axial resolution, lower lateral resolution and / or lower contrast than the corresponding higher quality image. [8] Computer-implemented method according to claim 7, wherein the plurality of lower quality images are each registered or aligned at the pixel level with a corresponding one of the plurality of high quality images to provide tagged data; and wherein the array of a second neural network has been trained using the labeled data according to supervised learning. [9] Computer-implemented method according to claim 7, wherein the plurality of lower quality images are each not registered or aligned with a corresponding one of the plurality of high quality images to provide unlabeled data; and wherein the array of a second neural network has been trained using the unlabeled data according to unsupervised learning. [10] Computer-implemented method according to claim 7, wherein the plurality of lower quality images are each registered or aligned at the patch level with a corresponding one of the plurality of high quality images to provide partially labeled data; and wherein the array of a second neural network has been trained using the partially labeled data according to weakly supervised learning. [11] Computer-implemented method according to one of claims 7 to 10, wherein the plurality of images of the training data set includes a plurality of image pairs, each image pair including a lower quality image and a corresponding higher quality image of a corresponding sample; and wherein for each respective image pair, the lower quality image and the corresponding higher quality image are obtained by the same imaging system having a first excitation light source providing light to cause the emission of light from one or more substances in the sample to image the sample and obtain the lower quality image, and a second excitation light source providing light to cause the emission of light from one or more substances in the sample to image the sample and obtain the corresponding higher quality image. [12] A computer-implemented method according to any one of claims 5 to 11, wherein the arrangement of a first neural network has been trained using a training data set comprising a plurality of images containing: a plurality of first images of a plurality of samples, each obtained by an imaging system having an excitation light source that provides light to cause the emission of light from one or more substances in the respective sample before the respective sample is chemically dyed, and several corresponding second chemically stained histological images of the samples, each obtained after the respective sample has been chemically stained. [13] Computer-implemented method according to claim 12, wherein the plurality of first images are each registered or aligned at the pixel level with a corresponding one of the plurality of second images to provide tagged data; and wherein the array of a first neural network has been trained using the labeled data according to supervised learning. [14] Computer-implemented method according to claim 12, wherein the plurality of first images are each not registered or aligned with a corresponding one of the plurality of second images to provide unlabeled data; and wherein the array of a first neural network has been trained using the unlabeled data according to unsupervised learning. [15] Computer-implemented method according to claim 12, wherein the plurality of first images are each registered or aligned at the patch level with a corresponding one of the plurality of second images to provide partially labeled data; and wherein the array of a first neural network has been trained using the partially labeled data according to weakly supervised learning. [16] A computer-implemented method according to any one of claims 5 to 15, wherein the output of the arrangement of a second neural network is connected to the input of the arrangement of a first neural network. [17] Computer-implemented method according to one of claims 1 to 16, wherein the imaging system is an imaging system for performing fluorescence imaging, and wherein the one or more substances comprise one or more photoluminescent substances, in particular endogenous fluorophores. [18] The computer-implemented method of claim 17, wherein the imaging system is an imaging system for performing fluorescence microscopy. [19] Computer-implemented method according to one of claims 1 to 18, wherein the excitation light source provides ultraviolet light to cause the emission of light from the one or more substances in the sample; and where the ultraviolet light includes ultraviolet light in the UVC range. [20] The computer-implemented method of claim 19, wherein the excitation light source comprises a UV LED operable to provide the ultraviolet light. [21] A computer-implemented method according to any one of claims 1 to 20, wherein the unsectioned sample comprises a biological sample, such as a tissue. [22] Computer-implemented method according to claim 21, wherein the imaging system is for performing in vivo imaging; and / or wherein the imaging system is a handheld imaging system. [23] A computer-implemented method according to any one of claims 1 to 22, wherein the non-sectioned sample is an unstained sample. [24] Image processing system comprising: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for carrying out the computer-implemented method of any one of claims 1 to 23. [25] A carrier medium carrying computer-readable instructions adapted to cause one or more processors to carry out the computer-implemented method according to any one of claims 1 to 23.

Images