Systems and methods for generating digitally stained histological images

Abstract

A computer-implemented method for processing an image, comprising: receiving at least one image of an unsliced sample; and processing at least the image by a neural network device for performing digital staining to obtain a digitally stained histological image of the unsliced sample. The image of the unsliced sample is obtained by an imaging system having an excitation light source that provides light to cause one or more substances, such as one or more photoluminescent substances, in the unsliced sample to emit light in order to image the sample. The digitally stained histological image generally corresponds to or substantially corresponds to an image of the unsliced sample that has been chemically stained with one or more chemical stains. The unsliced sample may be an unstained biological sample with an endogenous fluorophore, and the image may be an autofluorescence image of the unstained biological sample.

Description

Technical Field

[0001] The present invention relates to the generation of digitally stained histological images of samples. Background Art

[0002] Traditional techniques for obtaining histological images of samples require a relatively complex and time-consuming histological sample preparation process to prepare samples for imaging. The histological sample preparation process generally includes sample fixation, embedding, sectioning (cutting into thin slices), and chemical staining of the slices. Subsequently, the stained slices are imaged using an imaging system such as a microscope to obtain a histological image of the sample. The quality and usability of the resulting histological image depend on the execution of the histological sample preparation process, the quality of the imaging system used, etc. However, some histological sample preparation processes are time-consuming, labor-intensive, and / or prone to human error. Summary of the Invention

[0003] In a first aspect, a computer-implemented method for processing images is provided. The computer-implemented method comprises: receiving at least one image of an unsectioned sample; and processing at least the image by a neural network device for performing digital staining to obtain a digitally stained histological image of the unsectioned sample. The unsectioned sample is a sample that has not been sliced into thin sections having a thickness of about 2 microns to about 7 microns according to conventional histological sample preparation processes; that is, the unsectioned sample has a thickness greater than 7 microns. The unsectioned sample is preferably an optically thick sample, having a thickness greater than the depth of field of the imaging system. In some cases, the depth of field of the imaging system is less than 100 microns, and thus the thickness of the unsectioned sample is at least about 100 or several hundred microns (e.g., at least 100 microns, preferably at least 500 microns). The image is obtained by an imaging system with an excitation light source that provides light (i.e., optical radiation) to cause one or more substances (e.g., one or more photoluminescent substances) in the unsectioned sample to emit light, thereby imaging the unsectioned sample. The digitally stained histological image generally corresponds to or is substantially equivalent to an image obtained by chemically staining the unsectioned sample with one or more chemical stains (e.g., a histochemical stain). By digitally staining an unsectioned sample using the neural network device, an image equivalent to or comparable to a chemically stained image of the unsectioned sample can be easily obtained, eliminating the need for traditional histological sample preparation processes such as sectioning (thinly slicing) and chemical staining, which are typically time-consuming and labor-intensive. The digitally stained histological images obtained by this computer-implemented method can be used for diagnosis or initial screening of diseases such as cancer. In some embodiments, the digitally stained histological images obtained by this computer-implemented method can be used for diagnosis during surgery.

[0004] The neural network device is configured to take at least the image as input for processing. In some embodiments, the neural network device is configured to take a single image (i.e., the image) as input to provide a digitally stained histological image. In some embodiments, the neural network device is configured to take multiple images (i.e., the image and one or more other images) as input to provide a digitally stained histological image. In such embodiments, the multiple images may include multiple images of an unsectioned sample obtained by an imaging system having multiple excitation light sources, each excitation light source being configured to provide corresponding light (i.e., optical radiation) to cause one or more substances (e.g., one or more photoluminescent substances) in the unsectioned sample to emit corresponding light.

[0005] The neural network device may include one or more neural networks, such as one or more deep learning-based generative models. In some embodiments, the neural network device may be used only to perform digital colorization. In some embodiments, the neural network device may be used to perform one or more other image processing operations (in addition to digital colorization).

[0006] In some embodiments, the imaging system is configured to perform fluorescence imaging, such as widefield fluorescence imaging, and the image may be a fluorescence image. In some embodiments, the imaging system configured to perform fluorescence imaging may be configured to perform only fluorescence imaging. In some other embodiments, the imaging system configured to perform fluorescence imaging may be configured to perform one or more other imaging methods.

[0007] In some embodiments, the one or more substances include fluorophores. Such fluorophores can include endogenous fluorophores (i.e., fluorophores naturally present in the unsectioned sample) and / or exogenous fluorophores (i.e., fluorophores added to the unsectioned sample). In embodiments where the one or more substances consist solely of endogenous fluorophores, the fluorescent image is an autofluorescence image.

[0008] In some embodiments, the unsectioned sample is an in vivo sample, and the imaging system is used to perform in vivo imaging.

[0009] In some embodiments, the imaging system is a portable imaging system (eg, a handheld imaging system).

[0010] In some embodiments, the imaging system is configured to perform fluorescence microscopy, such as wide-field fluorescence microscopy. For example, the imaging system can include a fluorescence microscope, such as a wide-field fluorescence microscope.

[0011] In some embodiments, the imaging system is used to perform fluorescence endoscopic imaging (endoscopic fluorescence imaging).For example, the imaging system may include an endoscopic imaging system.

[0012] In some embodiments, the excitation light source provides ultraviolet light (having one or more wavelengths between about 100 nm and about 400 nm) to cause one or more substances in the unsectioned sample to emit light. In some embodiments, the ultraviolet light comprises ultraviolet light in the UVC band (e.g., having one or more wavelengths between about 100 nm and about 280 nm, or having one or more wavelengths between about 200 nm and about 280 nm, or having one or more wavelengths between about 200 nm and about 230 nm, or having a wavelength of about 265 nm).

[0013] In some embodiments, the excitation light source comprises a light emitting diode (LED). In some embodiments, the excitation light source comprises an ultraviolet LED capable of providing the ultraviolet light. In terms of providing ultraviolet light, ultraviolet LEDs have a relatively high cost-effectiveness.

[0014] In some embodiments, the imaging system may have one or more other excitation light sources, each of which is used to provide corresponding light to cause one or more substances in the unsectioned sample to emit light, thereby obtaining one or more other images of the unsectioned sample. The one or more other excitation light sources may, for example, be one or more ultraviolet LEDs, each of which is used to provide ultraviolet light (having one or more wavelengths between about 100 nm and about 400 nm). In one embodiment, the excitation light source and each of the one or more other excitation light sources include an ultraviolet LED. The excitation light source and the one or more other excitation light sources may provide the same ultraviolet light (i.e., ultraviolet light with the same wavelength, in one embodiment, the wavelength is 265 nm), or may provide different ultraviolet light (i.e., ultraviolet light with different wavelengths).

[0015] In some embodiments, the neural network device is configured to perform image enhancement and digital staining to provide a digitally stained histological image. Because the unsectioned sample is optically thick, light (provided by the imaging system) passing through the unsectioned sample may be scattered by various components at different depths within the unsectioned sample. Due to this scattering and / or the configuration of the imaging system, the quality of the image of the unsectioned sample obtained by the imaging system may be degraded. By performing image enhancement, this degradation in image quality can be compensated, allowing low-quality images of the unsectioned sample to be digitally stained. This can reduce the requirements of the imaging system used to image the unsectioned sample (e.g., one or more components thereof, such as the excitation light source, camera / sensor, etc.), and / or enable the computer-implemented method to digitally stain optically thick unsectioned samples. The neural network device may include one or more neural networks. The neural network device may be configured to perform image enhancement and digital staining sequentially (in any order) or substantially simultaneously. In some embodiments, the neural network device may be configured to perform one or more other operations.

[0016] In some embodiments, the neural network device has been trained with a training dataset. The training dataset may include: a plurality of images (e.g., images of a plurality of samples, such as unsectioned samples) acquired by an imaging system (e.g., an imaging system having an excitation light source that provides light to cause one or more substances (e.g., photoluminescent substances) within the sample to emit light); and a plurality of corresponding chemically stained histological images (e.g., images of the plurality of samples obtained by cutting the plurality of samples into multiple thin slices (when the samples were originally unsectioned samples, or cut into multiple thin slices according to other requirements) and staining them with the same chemical stain (e.g., a histochemical stain). The type of imaging system used to acquire the plurality of images may be the same as the type of imaging system used to acquire the images to be processed by the neural network device. In one embodiment, the imaging system used to acquire the plurality of images may be the imaging system used to acquire the images to be processed by the neural network device.

[0017] In some embodiments, the multiple images in the training dataset may include labeled data (e.g., images aligned or registered at the pixel level), and the neural network device may be trained using the labeled data under supervised learning. In some embodiments, the multiple images in the training dataset may include unlabeled data (e.g., images that are not aligned or registered), and the neural network device may be trained using the unlabeled data under unsupervised learning (e.g., self-supervised learning). In some embodiments, the multiple images in the training dataset may include partially labeled data (e.g., images aligned or registered at the tile level), and the neural network device may be trained using the partially labeled data under weakly supervised learning.

[0018] In some embodiments, image enhancement includes spatial resolution enhancement and / or contrast enhancement. As described above, as light (provided by the imaging system) passes through the unsectioned sample, it may be scattered by various components at different depths of the unsectioned sample. Due to such scattering and / or the structure of the imaging system, the spatial resolution and / or contrast of the image of the unsectioned sample obtained by the imaging system may sometimes be reduced. By performing spatial resolution enhancement and / or contrast enhancement, it can help to compensate for the reduction in spatial resolution and / or contrast of the image, so as to alleviate the scattering problem and / or address the limitations of the imaging system. In some embodiments, the second neural network device described above is at least used to perform spatial resolution enhancement and / or contrast enhancement.

[0019] In some embodiments, spatial resolution enhancement includes axial resolution enhancement and / or lateral resolution enhancement. As described above, the scattering and / or imaging system configuration may result in a decrease in spatial resolution and / or contrast. In some cases, the thicker the unsectioned sample, the more components exist at a greater depth, which can result in a decrease in image contrast. Therefore, in some cases, by performing axial resolution enhancement, it can help to reduce the imaging thickness, thereby improving image contrast. In some cases, for the same sample area, the faster the image acquisition speed, the lower the lateral resolution; the slower the image acquisition speed, the higher the lateral resolution. Therefore, by performing lateral resolution enhancement, it can help to compensate for the decrease in lateral resolution or increase the sample imaging speed. In some embodiments, the second neural network device described above is at least used to perform axial resolution enhancement and / or lateral resolution enhancement.

[0020] In some embodiments, the neural network device includes a first neural network device for performing digital rendering and a second neural network device for performing image enhancement. By using different neural network devices to perform digital rendering and image enhancement, the digital rendering performance or image enhancement performance can be optimized or adjusted separately and / or independently, thereby improving the performance of the neural network device in performing one or more tasks. The first neural network device may include one or more neural networks, and the second neural network device may also include one or more neural networks.

[0021] The second neural network device may include a deep learning-based generative model. In some embodiments, the second neural network device includes a neural network based on a generative adversarial network (GAN) (i.e., a GAN-based generative model). In some embodiments, the GAN-based neural network may include a super-resolution generative adversarial network (SRGAN) or an enhanced version thereof, such as an enhanced super-resolution generative adversarial network (ESRGAN). In some embodiments, the GAN-based neural network may include an attention mechanism (i.e., a GAN-based generative model with an attention mechanism). In some embodiments, the second neural network device includes a diffusion-based neural network (i.e., a diffusion-based model).

[0022] In some embodiments, the second neural network device has been trained using a training dataset comprising a plurality of images, the plurality of images comprising low-quality images of a plurality of samples (e.g., unsectioned samples) and corresponding high-quality images. The low-quality images may have lower axial resolution, lower lateral resolution, and / or lower contrast than the corresponding high-quality images.

[0023] In some embodiments, the training dataset used to train the second neural network device includes labeled data, and the second neural network device has been trained using the labeled data under supervised learning. In some embodiments, each of the plurality of low-quality images is registered or aligned with its corresponding high-quality image at the pixel level for the labeled data. This registration or alignment can be performed digitally.

[0024] In some embodiments, the training dataset used to train the second neural network device includes partially labeled data, and the second neural network device has been trained using the partially labeled data under weakly supervised learning. In some embodiments, for the partially labeled data, each of the plurality of low-quality images is registered or aligned with its corresponding high-quality image at the tile level (each tile may include multiple pixels). This registration or alignment can be performed digitally.

[0025] In some embodiments, the training dataset used to train the second neural network device includes unlabeled data, and the second neural network device has been trained on the unlabeled data in an unsupervised manner (e.g., self-supervised learning). In some embodiments, for the unlabeled data, each of the plurality of low-quality images is not registered or aligned with its corresponding high-quality image at the pixel level and the tile level. In some cases, the images are not specifically registered or aligned.

[0026] In some embodiments, the multiple images in the training dataset include multiple pairs of images, each pair of images comprising a low-quality image and a corresponding high-quality image of a sample (e.g., an unsectioned sample). Each pair of images can be for a corresponding sample. In some embodiments, for each pair of images, the low-quality image and the corresponding high-quality image are acquired by the same imaging system having a first excitation light source and a second excitation light source, wherein the first excitation light source provides light that causes one or more substances (e.g., a photoluminescent substance) in the sample to emit light for imaging the sample, thereby acquiring the low-quality image; and the second excitation light source (different from the first excitation light source) provides light that causes one or more substances (e.g., a photoluminescent substance) in the sample to emit light for imaging the sample, thereby acquiring the corresponding high-quality image. In some embodiments, by acquiring each pair of low-quality and high-quality images using the same imaging system with different excitation light sources, registration or alignment of the low-quality and high-quality images can be more labor-efficient. In some embodiments, the imaging system is used for widefield fluorescence microscopy. In some embodiments, the first excitation light source comprises an ultraviolet LED, and / or the second excitation light source comprises an ultraviolet laser.

[0027] In some embodiments, the low-quality image is obtained by a first type of imaging system having a first type of excitation light source, which provides light to cause one or more substances in the sample (such as photoluminescent substances) to emit light. At the same time, the corresponding high-quality image is obtained by a second type of imaging system having a second type of excitation light source, which provides light to cause one or more substances in the sample (such as photoluminescent substances) to emit light. In some embodiments, the first type of imaging system is different from the second type of imaging system. In this way, each pair of low-quality and high-quality images can be obtained by different types of imaging systems. In some embodiments, the first type of imaging system is an imaging system for wide-field fluorescence microscopy, and / or the second type of imaging system is an imaging system for light-sheet fluorescence microscopy. In some embodiments, the first type of excitation light source is different from the second type of excitation light source. In some embodiments, the first type of excitation light source includes an ultraviolet LED, and / or the second type of excitation light source includes an ultraviolet laser.

[0028] The first neural network device may include a deep learning-based generative model. In some embodiments, the first neural network device includes a neural network based on a generative adversarial network (GAN) (i.e., a GAN-based generative model). The GAN-based neural network may include a conditional generative adversarial network (cGAN). In some embodiments, the GAN-based neural network may include an attention mechanism (i.e., a GAN-based generative model with an attention mechanism). In some embodiments, the first neural network device includes a diffusion-based neural network (i.e., a diffusion-based model).

[0029] In some embodiments, the first neural network device has been trained with a training dataset comprising a plurality of images, the plurality of images comprising: a plurality of first images of a plurality of samples (e.g., unsectioned samples), each first image acquired by an imaging system having an excitation light source prior to chemical staining of the sample, the excitation light source providing light to cause one or more substances (e.g., photoluminescent substances) in the sample to emit light; and a plurality of corresponding second chemically stained histological images of each sample, each second chemically stained histological image acquired after the corresponding sample was sliced (as needed) and chemically stained (e.g., with a histochemical stain, which the neural network device is configured to perform digital simulation). In some such embodiments, the imaging system is an imaging system for widefield fluorescence microscopy. In some such embodiments, the excitation light source is an excitation light source comprising an ultraviolet laser or an ultraviolet LED.

[0030] In some embodiments, the training dataset used to train the first neural network device includes labeled data, and the first neural network device has been trained using the labeled data under supervised learning. In some embodiments, each of the plurality of first images is registered or aligned with its corresponding second image at the pixel level for the labeled data. The registration or alignment can be performed digitally.

[0031] In some embodiments, the training dataset used to train the first neural network device includes partially labeled data, and the first neural network device has been trained using the partially labeled data under weakly supervised learning. In some embodiments, for the partially labeled data, each of the plurality of first images is registered or aligned with its corresponding second image at the tile level (each tile may include multiple pixels). This registration or alignment can be performed digitally. This can be beneficial for first images obtained from unsectioned samples. This is because when a sample is sectioned and stained (through traditional histological sample preparation processes), the first and second images obtained from such a sample may not accurately represent the same sample surface, making pixel-level registration or alignment difficult to achieve.

[0032] In some embodiments, the training dataset used to train the first neural network device includes unlabeled data, and the first neural network device has been trained on the unlabeled data under unsupervised learning (e.g., self-supervised learning). In some embodiments, for the unlabeled data, each of the plurality of low-quality images is not registered or aligned with its corresponding high-quality image at the pixel level and the tile level. In some cases, the images are not specifically registered or aligned. This can be beneficial for first images obtained from unsectioned samples. This is because when the sample is sectioned and stained (through traditional histological sample preparation processes), the first and second images obtained from such a sample may not accurately represent the same sample surface, making registration or alignment difficult.

[0033] In some embodiments, the output of the second neural network device is operably connected to the input of the first neural network device. For example, the output of the second neural network device can be directly connected to the input of the first neural network device. In some embodiments, processing the image includes: processing the image with the second neural network device to obtain an enhanced image; and processing the image with the first neural network device to obtain the digitally stained histological image. In some embodiments, the target / output image of the training dataset used to train the second neural network device can also be used as an input image of the training dataset used to train the first neural network device. This allows for more efficient data utilization and better operative connection between the first and second neural network devices.

[0034] In some embodiments, the image of the unsectioned sample is a grayscale image. In some other embodiments, the image of the unsectioned sample is a color image.

[0035] In some embodiments, the digitally stained histological image of the unsectioned sample is a grayscale image. In some other embodiments, the digitally stained histological image of the unsectioned sample is a color image.

[0036] In some embodiments, the unsectioned sample comprises a biological sample such as a tissue. In some other embodiments, the unsectioned sample comprises a simulated tissue sample (e.g., a phantom). In some embodiments, the biological sample is obtained (extracted) from a human, animal, or plant. In some embodiments, the biological sample is an in vivo biological sample (e.g., a sample that is not removed or otherwise removed from a living human, animal, or plant). In some embodiments, the unsectioned sample can be a raw sample that has been only minimally processed.

[0037] In some embodiments, the unsectioned sample is neither arranged on or within a glass slide nor between glass slides, thereby enabling imaging based on slide-free imaging technology. In some embodiments in which the unsectioned sample includes tissue, the tissue may be either fixed tissue (such as formalin-fixed tissue) or unfixed tissue (such as fresh tissue), and the tissue may or may not be embedded in paraffin. The tissue has not been sliced according to the requirements of the traditional histological sample preparation process, that is, the thickness of the tissue is greater than 7 microns. The tissue is preferably a tissue with a large optical thickness, and its thickness is greater than the depth of field of the imaging system. For example, the thickness of the tissue is at least one hundred or several hundred microns (such as at least 100 microns, preferably at least 500 microns). In some embodiments, the unsectioned sample consists only of the tissue. In some embodiments, the unsectioned sample includes a formalin-fixed paraffin-embedded tissue block containing the tissue.

[0038] In some embodiments, the unsectioned sample is an unstained sample that can be used for label-free imaging by the imaging system. An unstained sample is a sample that has not been chemically stained (ie, an unlabeled sample), and thus can be imaged without labeling.

[0039] In some embodiments, the digitally stained histology image generally corresponds to or is substantially equivalent to an image of an unsectioned sample chemically stained with hematoxylin / eosin (HE) stain. In some embodiments, the digitally stained histology image generally corresponds to or is substantially equivalent to an image of an unsectioned sample chemically stained with one or more special stains. In some embodiments, the digitally stained histology image generally corresponds to or is substantially equivalent to an image of an unsectioned sample stained with one or more immunohistochemical stains.

[0040] In some embodiments, the computer-implemented method includes outputting the digitally stained histological image. In some embodiments, the computer-implemented method includes displaying the digitally stained histological image. In some embodiments, the computer-implemented method includes storing the digitally stained histological image. In some embodiments, the computer-implemented method includes storing the image and the digitally stained histological image as training data for training the neural network device.

[0041] In a second aspect, an image processing system is provided, comprising: one or more processors; and a memory storing one or more programs to be executed by the one or more processors. The one or more programs include instructions for performing the computer-implemented method described in the first aspect. The image processing system may constitute part of an imaging system, such as the imaging system described above for imaging unsectioned samples. In some embodiments, the image processing system may be incorporated into the imaging system. In some embodiments, the image processing system may be remotely located relative to the imaging system and may be operably connected thereto. In some embodiments, the memory may store the neural network device.

[0042] In a third aspect, a carrier medium carrying computer-readable instructions is provided, wherein the computer-readable instructions are used to cause one or more processors to perform the computer-implemented method as described in the first aspect.

[0043] In a fourth aspect, a non-transitory computer-readable medium storing one or more programs is provided, wherein the one or more programs are for execution by one or more processors and include instructions for performing the computer-implemented method as described in the first aspect.

[0044] In a fifth aspect, a method for digitally staining an unsectioned sample is provided. The method includes imaging the unsectioned sample using an imaging system having an excitation light source, wherein the excitation light source provides light to cause one or more substances in the unsectioned sample to emit light, thereby obtaining an image of the unsectioned sample. The one or more substances may include one or more photoluminescent substances, such as fluorophores. The method also includes executing the computer-implemented method described in the first aspect. The imaging system having the excitation light source of the fifth aspect may include one or more technical features of the imaging system in the computer-implemented method described in the first aspect.

[0045] In a sixth aspect, a system is provided, comprising: an imaging system having an excitation light source, the excitation light source providing light to cause one or more substances in an unsectioned sample to emit light, thereby imaging the unsectioned sample; and an image processing system as described in the second aspect. The one or more substances may include one or more photoluminescent substances, such as fluorophores. The imaging system of the sixth aspect having the excitation light source may include one or more technical features of the imaging system described in the second aspect. The imaging system of the sixth aspect may be a portable or handheld imaging system.

[0046] In a seventh aspect, a method for training a neural network device for performing digital colorization is provided. The method comprises: obtaining a training dataset comprising a plurality of images; and training the neural network device with the training dataset, such that the neural network device is configured to perform digital colorization (and optionally one or more other image processing operations, such as image enhancement). The neural network device may comprise one or more neural networks, such as one or more deep learning-based generative models. After training, the neural network device may function as the neural network device described in the computer-implemented method of the first aspect.

[0047] In some embodiments, the training dataset may include: a plurality of images (e.g., a plurality of images of a plurality of samples) obtained by an imaging system (e.g., an imaging system having an excitation light source, the excitation light source providing light to cause one or more substances, such as one or more photoluminescent substances, to emit light); and a plurality of corresponding chemically stained histological images (e.g., a plurality of corresponding chemically stained histological images obtained after (as required) the plurality of samples are sliced and stained with the same chemical stain (e.g., a histochemical stain)).

[0048] In some embodiments, the neural network device includes a first neural network device for performing digital staining and a second neural network device for performing image enhancement such as spatial resolution enhancement (e.g., axial resolution and / or lateral resolution) and / or contrast enhancement. Accordingly, training the neural network device may include: training the first neural network device; and training the second neural network device. In some cases, the training may include: training only the neural network devices that require training. The first neural network device may include one or more neural networks. The second neural network device may include one or more neural networks. The training dataset may include: a training dataset for training the first neural network device; and a training dataset for training the second neural network device.

[0049] In some embodiments, an output terminal of the second neural network device is operatively connected directly or indirectly to an input terminal of the first neural network device.

[0050] In some embodiments, the second neural network device includes a deep learning-based generative model. In some embodiments, the second neural network device includes a neural network based on a generative adversarial network (GAN) (i.e., a GAN-based generative model). In some embodiments, the GAN-based neural network may include a super-resolution generative adversarial network (SRGAN) or an enhanced version thereof, such as an enhanced super-resolution generative adversarial network (ESRGAN). In some embodiments, the GAN-based neural network may include an attention mechanism (i.e., a GAN-based generative model with an attention mechanism). In some embodiments, the second neural network device includes a diffusion-based neural network (i.e., a diffusion-based model).

[0051] In some embodiments, a training dataset for training the second neural network device includes a plurality of images, the plurality of images comprising low-quality images of a plurality of samples (e.g., unsectioned samples) and corresponding high-quality images. The low-quality images may have lower axial resolution, lower lateral resolution, and / or lower contrast than the corresponding high-quality images. The plurality of samples may include biological samples such as tissue samples. The plurality of samples may be raw samples that have only been minimally processed. In some embodiments, the method further comprises: obtaining the training dataset.

[0052] In some embodiments, the training dataset used to train the second neural network device includes labeled data, and the second neural network device has been trained using the labeled data under supervised learning. In some embodiments, for the labeled data, each of the plurality of low-quality images is registered or aligned at the pixel level with its corresponding high-quality image. This registration or alignment can be performed digitally. In some embodiments, the method further comprises: registering or aligning the low-quality and corresponding high-quality images at the pixel level to obtain the labeled data.

[0053] In some embodiments, the training dataset used to train the second neural network device includes unlabeled data, and the second neural network device has been trained on the unlabeled data using unsupervised learning (e.g., self-supervised learning). In some embodiments, for the unlabeled data, each of the plurality of low-quality images is not registered or aligned with its corresponding high-quality image at the pixel level and the tile level. For example, the images are not specifically registered or aligned.

[0054] In some embodiments, the training dataset used to train the second neural network device includes partially labeled data, and the second neural network device has been trained on the partially labeled data under weakly supervised learning conditions. In some embodiments, for the partially labeled data, each of the plurality of low-quality images is registered or aligned with its corresponding high-quality image at the tile level (each tile may include multiple pixels). This registration or alignment can be performed digitally. In some embodiments, the method further includes: registering or aligning the low-quality and corresponding high-quality images at the tile level to obtain the partially labeled data.

[0055] In some embodiments, the plurality of images in the training dataset used to train the second neural network device includes a plurality of pairs of images, each pair of images including a low-quality image and a corresponding high-quality image of a sample (eg, an unsliced sample).

[0056] In some embodiments, for each pair of images, the low-quality image and the corresponding high-quality image are obtained by the same imaging system having a first excitation light source and a second excitation light source, wherein the first excitation light source provides light to cause one or more substances (e.g., photoluminescent substances) in the sample to emit light used to obtain the low-quality image, and the second excitation light source (different from the first excitation light source) provides light to cause one or more substances (e.g., photoluminescent substances) in the sample to emit light used to obtain the corresponding high-quality image. In some embodiments, the imaging system is used for widefield fluorescence microscopy. In some embodiments, the first excitation light source comprises an ultraviolet LED and / or the second excitation light source comprises an ultraviolet laser.

[0057] In some embodiments, the low-quality image is obtained by a first type of imaging system having a first type of excitation light source, which provides light to cause one or more substances (such as photoluminescent substances) in the sample to emit light. At the same time, the corresponding high-quality image is obtained by a second type of imaging system having a second type of excitation light source, which provides light to cause one or more substances (such as photoluminescent substances) in the sample to emit light. In some embodiments, the first type of imaging system is different from the second type of imaging system. For example, the first type of imaging system is an imaging system for wide-field fluorescence microscopy, and / or the second type of imaging system is an imaging system for light-sheet fluorescence microscopy. In some embodiments, the first type of excitation light source is different from the second type of excitation light source. For example, the first type of excitation light source includes an ultraviolet LED, and / or the second type of excitation light source includes an ultraviolet laser.

[0058] In some embodiments, the first neural network device comprises a deep learning-based generative model. In some embodiments, the first neural network device comprises a neural network based on a generative adversarial network (GAN) (i.e., a GAN-based generative model). The GAN-based neural network may comprise a conditional generative adversarial network (cGAN). In some embodiments, the GAN-based neural network may comprise an attention mechanism (i.e., a GAN-based generative model with an attention mechanism). In some embodiments, the first neural network device comprises a diffusion-based neural network (i.e., a diffusion-based model).

[0059] In some embodiments, the training dataset used to train the first neural network device includes a plurality of images, including: a plurality of first images of a plurality of samples (e.g., unsectioned samples), each first image being acquired by an imaging system having an excitation light source before chemical staining of the sample, the excitation light source providing light to cause one or more substances (e.g., photoluminescent substances) in the sample to emit light; and a corresponding plurality of second chemically stained histological images of the sample, each second chemically stained histological image being acquired after the corresponding sample was sectioned (as needed) and chemically stained (with one or more chemical stains (e.g., histochemical stains) such as hematoxylin / eosin (HE) stain). In some such embodiments, the imaging system is an imaging system for widefield fluorescence microscopy. In some such embodiments, the excitation light source is an excitation light source comprising an ultraviolet laser or an ultraviolet LED.

[0060] In some embodiments, the training dataset used to train the first neural network device includes labeled data, and the first neural network device has been trained using the labeled data under supervised learning. In some embodiments, for the labeled data, each of the plurality of first images is registered or aligned at the pixel level with its corresponding second image. This registration or alignment can be performed digitally. In some embodiments, the method further comprises: registering or aligning the first image with the corresponding second image at the pixel level to obtain the labeled data.

[0061] In some embodiments, the training dataset used to train the first neural network device includes unlabeled data, and the first neural network device has been trained on the unlabeled data using unsupervised learning (e.g., self-supervised learning). In some embodiments, for the unlabeled data, each of the plurality of first images is not registered or aligned with its corresponding second image at the pixel level and the tile level. In other words, the images are not specifically registered or aligned.

[0062] In some embodiments, the training dataset used to train the first neural network device includes partially labeled data, and the first neural network device has been trained using the partially labeled data under weakly supervised learning. In some embodiments, for the partially labeled data, each of the plurality of first images is registered or aligned with its corresponding second image at a tile level (each tile may include multiple pixels). This registration or alignment can be performed digitally. The method further includes registering or aligning the first image with the corresponding second image at a tile level to obtain the partially labeled data.

[0063] In an eighth aspect, a system is provided, comprising: one or more processors; and a memory storing one or more programs to be executed by the one or more processors. The one or more programs include instructions for executing the method described in the seventh aspect. In some embodiments, the system of the eighth aspect and the system of the second aspect can be the same system.

[0064] In a ninth aspect, a carrier medium carrying computer-readable instructions is provided, wherein the computer-readable instructions are used to cause one or more processors to execute the method as described in the seventh aspect.

[0065] In a tenth aspect, a non-transitory computer-readable medium storing one or more programs is provided, wherein the one or more programs are for execution by one or more processors and include instructions for executing the method described in the seventh aspect.

[0066] In an eleventh aspect, a method for generating a digitally stained histological image from a sample is provided. The method comprises: receiving an autofluorescence image of an unsectioned, slide-free sample; and processing the autofluorescence image with a neural network device to generate a digitally stained histological image of the unsectioned, slide-free sample. The autofluorescence image is obtained by, or from, an imaging system that uses an LED such as an ultraviolet LED as an excitation light source. The digitally stained histological image of the unsectioned, slide-free sample generally corresponds to or is substantially equivalent to an image of the unsectioned, slide-free sample after chemical staining. The neural network device may include one or more trained deep neural networks.

[0067] In some embodiments, the neural network device may be the neural network device in the computer-implemented method of the first aspect.

[0068] In some embodiments, the neural network device has been trained to generate digitally stained histological images using a training dataset, wherein the training dataset includes: autofluorescence images (obtained by an imaging system using an LED such as an ultraviolet LED as an excitation light source, or obtained from the imaging system); and corresponding chemical or histological staining images of one or more samples.

[0069] In some embodiments, the neural network device comprises an image enhancement neural network device and a digital staining neural network device. In some embodiments, the neural network device has been trained to perform image enhancement using a training dataset, wherein the training dataset comprises: a first set of autofluorescence images (obtained by or from an imaging system using an ultraviolet LED, such as an LED, as an excitation light source); and a corresponding second set of autofluorescence images having higher image contrast and / or spatial resolution than the first set of autofluorescence images (obtained by or from a different imaging system and / or a different excitation light source). In some embodiments, the output of the image enhancement neural network device is connected to the digital staining neural network device.

[0070] In a twelfth aspect, an imaging system is provided that includes a UV LED, which serves as an excitation light source to stimulate autofluorescence of an unsectioned sample. In some embodiments, the unsectioned sample is an unstained sample, which may or may not be fixed with formalin and / or embedded in paraffin. The imaging system can be standalone and / or portable.

[0071] By carefully considering the "Detailed Description" section and the accompanying drawings, other features and aspects of the present invention will be readily understood. As long as appropriate and feasible, any technical feature described herein with respect to a certain aspect or embodiment may be combined with any other technical feature described herein with respect to any other aspect or any other embodiment.

[0072] As used herein, "unsectioned" means not sliced into sections having a thickness of about 2 microns to about 7 microns according to conventional histological specimen preparation procedures. An example of an unsectioned specimen is a specimen having a thickness greater than 7 microns. An example of an unsectioned tissue is tissue having a thickness greater than 7 microns.

[0073] As used herein, terms of degree such as “generally,” “approximately,” and “substantially” are intended to take into account manufacturing tolerances, performance degradation, trends, tendencies, and actual non-ideal conditions, depending on the context.

[0074] Unless stated otherwise, the terms "connect," "connected," "mounted," etc. are intended to cover both direct and indirect connections, connections, mountings, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Hereinafter, some embodiments of the present invention will be described with reference to the accompanying drawings.

[0076] Figure 1 A schematic diagram of the operation of generating a digitally stained histological image in some embodiments of the present invention;

[0077] Figure 2 is a schematic diagram of a neural network device in some embodiments of the present invention;

[0078] Figure 3 Schematic diagram of a system for digitally staining unsectioned samples in some embodiments of the present invention;

[0079] Figure 4 is a schematic diagram of an imaging system in some embodiments of the present invention;

[0080] Figure 5 A block diagram of an information processing system in some embodiments of the present invention;

[0081] Figure 6 A schematic diagram of the operation of generating a digitally stained histological image in some embodiments of the present invention;

[0082] Figure 7 A schematic diagram of the operation of generating a digitally stained histological image in some embodiments of the present invention;

[0083] Figure 8 Schematic diagram of the operation of training a neural network device for generating digitally stained histological images in some embodiments of the present invention;

[0084] Figure 9a An enhanced autofluorescence image of a mouse brain in one embodiment;

[0085] Figure 9b for Figure 9a Corresponding digitally stained histological images of the enhanced autofluorescence images of the mouse brain shown;

[0086] Figure 9c for Figure 9a Wide-field LED image of the mouse brain region shown in box 9a;

[0087] Figure 9d for Figure 9a Enhanced image of the mouse brain region shown in box 9a;

[0088] Figure 9e for Figure 9a High-lateral-resolution image of the mouse brain region shown in Box 9a;

[0089] Figure 9f For Figure 9c Wide-field LED images corresponding to digitally stained histological images;

[0090] Figure 9g For Figure 9d The digitally stained histological image corresponding to the enhanced image;

[0091] Figure 9hFor Figure 9e The digitally stained histological images corresponding to the digitally stained histological images;

[0092] Figure 9i for Figure 9a Chemically stained histological images of the mouse brain region shown in Box 9a;

[0093] Figure 10a An enhanced autofluorescence image of a mouse brain in one embodiment;

[0094] Figure 10b For Figure 10a Digitally stained histological images corresponding to enhanced autofluorescence images of mouse brain;

[0095] Figure 10c for Figure 10a Wide-field LED image of the mouse brain region shown in box 10a;

[0096] Figure 10d for Figure 10a Digitally enhanced image of the mouse brain region shown in box 10a;

[0097] Figure 10e for Figure 10a High-lateral-resolution image of the mouse brain region shown in box 10a;

[0098] Figure 10f For Figure 10c Wide-field LED images corresponding to digitally stained histological images;

[0099] Figure 10g For Figure 10d The digitally enhanced images correspond to the digitally stained histological images;

[0100] Figure 10h For Figure 10e High lateral resolution images corresponding to digitally stained histological images;

[0101] Figure 10i for Figure 10a Chemically stained histological images of the mouse brain region shown in box 10a;

[0102] Figure 11a 1 is a wide-field LED image of a thick sample of formalin-fixed lung cancer tissue in one embodiment;

[0103] Figure 11b This is a digitally enhanced image of a thick sample of formalin-fixed lung cancer tissue;

[0104] Figure 11c High lateral resolution images of thick formalin-fixed lung cancer tissue samples;

[0105] Figure 11d for Figure 11a Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 11a;

[0106] Figure 11e for Figure 11a Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 11a2;

[0107] Figure 11f for Figure 11b Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 11b1;

[0108] Figure 11g for Figure 11b Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 11b2;

[0109] Figure 11h for Figure 11c Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 11c1;

[0110] Figure 11i for Figure 11c Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 11c2;

[0111] Figure 12a A digitally enhanced image of human lung tissue in one embodiment;

[0112] Figure 12b For Figure 12a Digitally enhanced images correspond to digitally stained histological images;

[0113] Figure 12c for Figure 12a Wide-field LED image of the human lung tissue region shown in box 12a;

[0114] Figure 12d for Figure 12a A digitally enhanced image of the human lung tissue region shown in box 12a;

[0115] Figure 12e for Figure 12a The corresponding adjacent chemical staining image of the human lung tissue region shown in box 12a;

[0116] Figure 12f For Figure 12c Wide-field LED images corresponding to digitally stained histological images;

[0117] Figure 12g For Figure 12d The digitally enhanced images correspond to the digitally stained histological images;

[0118] Figure 13a 1 is a wide-field LED image of a thick sample of formalin-fixed lung cancer tissue in one embodiment;

[0119] Figure 13b This is a digitally enhanced image of a thick sample of formalin-fixed lung cancer tissue;

[0120] Figure 13c High axial resolution and high contrast enhanced images of formalin-fixed thick lung cancer tissue;

[0121] Figure 13d for Figure 13a Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 13a;

[0122] Figure 13e for Figure 13a Magnified image of a thick sample area of formalin-fixed lung cancer tissue shown in box 13a;

[0123] Figure 13f for Figure 13a Magnified image of a thick section of formalin-fixed lung cancer tissue shown in box 13a. DETAILED DESCRIPTION

[0124] Figure 1 Illustrated are operations for generating a digitally stained histological image in accordance with some embodiments of the present invention. Operation 100 is a computer-implemented operation and generally includes: a neural network device 102 receiving an unsectioned sample image; processing the unsectioned sample image; and outputting a digitally stained histological image of the unsectioned sample. The digitally stained histological image of the unsectioned sample generally corresponds to or is substantially equivalent to an image of the unsectioned sample chemically stained with one or more chemical stains (e.g., a histochemical stain). An unsectioned sample is a sample that has not been sliced into thin sections of approximately 2 microns to approximately 7 microns according to conventional histological sample processing operations. In such embodiments, the unsectioned sample may include a biological sample, such as tissue having a thickness greater than 7 microns. The biological sample may be an in vitro biological sample, an ex vivo biological sample, or an in vivo biological sample. The tissue may be human tissue, animal tissue, or plant tissue. The tissue may be fixed tissue (e.g., formalin-fixed tissue) or unfixed tissue (e.g., fresh tissue), and may or may not be paraffin-embedded. In some embodiments, the sample may be an unprocessed, raw sample or a minimally processed sample. In some embodiments, the unsectioned sample is an unstained sample. A digitally stained histology image generally corresponds to or is substantially equivalent to an image of an unsectioned specimen chemically stained with one or more chemical stains (e.g., histochemical stains) such as hematoxylin / eosin stain (HE stain), one or more special stains, or one or more immunohistochemical stains.

[0125] In operation 100, neural network device 102 receives and processes an image of an unsectioned sample using an imaging system equipped with an excitation light source. The excitation light source provides light (optical radiation) to cause one or more substances (e.g., photoluminescent substances such as fluorescent and / or phosphorescent substances) in the unsectioned sample to emit light. The unsectioned sample is an optically thick sample, having a thickness greater than the depth of field of the imaging system used to obtain the image of the unsectioned sample. Because the depth of field of some existing imaging systems is typically less than one hundred microns, the thickness of the unsectioned sample can be at least one hundred or several hundred microns (e.g., at least 100 microns, preferably at least 500 microns). In some embodiments, the imaging system is adapted for fluorescence imaging, such as widefield fluorescence imaging. In some embodiments, the imaging system is adapted for in vivo imaging. In one embodiment, the imaging system can be adapted for fluorescence microscopy, such as widefield fluorescence microscopy. In another embodiment, the imaging system can be adapted for fluorescence endoscopic imaging (endoscopic fluorescence imaging). The imaging system can be portable (e.g., handheld). In embodiments where the imaging system is adapted for fluorescence imaging, the resulting image can be a fluorescence image. Depending on the unsectioned sample, one or more substances in the unsectioned sample may contain fluorophores, which may be endogenous fluorophores or exogenous fluorophores (or both). In embodiments where the substances are composed of endogenous fluorophores, the unsectioned sample image is an autofluorescence image of the unsectioned sample.

[0126] In some embodiments, the light provided by the excitation light source of the imaging system may include ultraviolet light capable of causing one or more substances (e.g., photoluminescent substances) in the unsectioned sample to emit light. The ultraviolet light may include ultraviolet light within the UVC band (i.e., deep ultraviolet light). The excitation light source may be an LED excitation light source, a laser excitation light source, or the like for providing the ultraviolet light.

[0127] like Figure 1 As shown, neural network device 102 is configured to take the above-mentioned image as input for processing. Neural network device 102 may include one or more neural networks, such as one or more generative models based on deep learning. Neural network device 102 is configured to at least perform digital staining. In some embodiments, neural network device 102 is configured to perform one or more other image processing operations. The digital staining and one or more other image processing operations may be performed sequentially (in any order) or substantially simultaneously. In one embodiment, the neural network device may be configured to at least perform image enhancement and digital staining to provide a digitally stained histological image of an unsectioned sample (that has been at least imaged and digitally stained).

[0128] Neural network device 102 can be trained using a training dataset comprising images of different samples, such as different unsectioned samples, and corresponding chemically stained histological images of such different samples. The images of the different samples can be obtained using an imaging system with an excitation light source that provides light to cause one or more substances (e.g., photoluminescent substances) in the sample to emit light. The type of imaging system can be the same as the type of imaging system used to obtain the images to be processed by neural network device 102. The type of excitation light source can be the same as the type of excitation light source used to obtain the images to be processed by neural network device 102.

[0129] Figure 2 A neural network device 200 is shown in some embodiments of the present invention. In some embodiments, the neural network device 200 can be used as Figure 1 Neural network device 102 is shown in operation 100 .

[0130] like Figure 2 As shown, neural network device 200 includes neural network device 202A for image enhancement and neural network device 202B for digital colorization. Neural network device 202A and neural network device 202B each include at least one neural network. In some embodiments, both neural network devices 202A and 202B can be trained separately and / or independently.

[0131] In some embodiments, neural network device 202A for image enhancement can be used to at least enhance spatial resolution (axial resolution and / or lateral resolution) and / or enhance contrast.

[0132] The neural network device 202A for image enhancement may include a generative model based on deep learning. For example, in some embodiments, the neural network device 202A may include a neural network based on a generative adversarial network (GAN) (i.e., a GAN-based generative model). A GAN-based neural network generally includes at least one generative network and at least one discriminative network. For example, different GAN-based neural networks may have different numbers of generative networks, different numbers of discriminative networks, different generative network designs (e.g., different numbers of convolutional layers, different residual blocks / skip connections), or different discriminative network designs (e.g., different numbers of convolutional layers, different residual blocks / skip connections). In some embodiments, the GAN-based neural network may include a super-resolution generative adversarial network (SRGAN), or an enhanced version thereof, such as an enhanced super-resolution generative adversarial network (ESRGAN). The architecture of ESRGAN can be found, for example, in Wang et al., "ESRGAN: Enhanced Super-Resolution Generative Adversarial Network" (2019), the entire contents of which are incorporated herein by reference. In some embodiments, the GAN-based neural network may include a weakly supervised GAN-based model. The architecture of a weakly supervised GAN-based model can be found, for example, in Dai et al., "A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transfer" (2022), which is incorporated herein by reference in its entirety. For example, in some embodiments, neural network device 202A may include a diffusion-based neural network (i.e., a diffusion-based model).

[0133] The neural network device 202A for image enhancement is trained using a training dataset containing low-quality images of a plurality of samples (e.g., unsliced samples) and corresponding high-quality images. For example, the low-quality images have lower axial resolution, lower lateral resolution, and / or lower contrast than the corresponding high-quality images.

[0134] In some embodiments, the training dataset includes labeled data, and the neural network device 202A has been trained under supervised learning conditions. In one embodiment, the labeled data can be provided by registering or aligning each low-quality image with its corresponding high-quality image at the pixel level.

[0135] In some embodiments, the training dataset includes partially labeled data, and neural network device 202A has been trained under weakly supervised learning conditions. In one embodiment, the labeled data can be provided by registering or aligning each low-quality image with its corresponding high-quality image at the image level (e.g., consisting of multiple pixels, multiple small patches, etc.).

[0136] In some embodiments, the training dataset includes unlabeled data, and the neural network device 202A is trained under unsupervised learning conditions (e.g., self-supervised learning). In one embodiment, the unlabeled data is provided in a manner such that each low-quality image is not registered or aligned with its corresponding high-quality image at the pixel level and the tile level.

[0137] In some embodiments, the low-quality image and the corresponding high-quality image can form an image pair, each image pair comprising one low-quality image and one corresponding high-quality image of a sample (e.g., an unsectioned sample). For each image pair, the low-quality image and the corresponding high-quality image are acquired by the same imaging system, which has at least two excitation light sources: a first excitation light source that provides light to cause one or more substances (e.g., a photoluminescent substance) in the sample to emit light, thereby acquiring the low-quality image; and a second excitation light source (different from the first excitation light source) that provides light to cause one or more substances (e.g., a photoluminescent substance) in the sample to emit light, thereby acquiring the corresponding high-quality image. In some embodiments, the imaging system is used for widefield fluorescence microscopy, and the first excitation light source comprises an ultraviolet LED, and the second excitation light source comprises an ultraviolet laser.

[0138] In some embodiments, a low-quality image is acquired by a first type of imaging system having a first type of excitation light source that provides light to cause one or more substances (e.g., photoluminescent substances) in a sample to emit light, while a corresponding high-quality image is acquired by a second type of imaging system (e.g., different from the first type of imaging system) having a second type of excitation light source (e.g., different from the first type of excitation light source) that provides light to cause one or more substances (e.g., photoluminescent substances) in the sample to emit light. For example, the first type of imaging system is an imaging system for widefield fluorescence microscopy, and / or the second type of imaging system is an imaging system for light-sheet fluorescence microscopy. For example, the first type of excitation light source comprises an ultraviolet LED, and / or the second type of excitation light source comprises an ultraviolet laser.

[0139] The digital coloring neural network device 202B may include a generative model based on deep learning. For example, in some embodiments, the neural network device 202B may include a neural network based on a generative adversarial network (GAN) (i.e., a GAN-based generative model). A GAN-based neural network generally includes at least one generative network and at least one discriminative network. For example, different GAN-based neural networks may have different numbers of generative networks, different numbers of discriminative networks, different generative network designs (e.g., different numbers of convolutional layers, different residual blocks / skip connections), or different discriminative network designs (e.g., different numbers of convolutional layers, different residual blocks / skip connections). The GAN-based neural network may include a conditional generative adversarial network. In some embodiments, the GAN-based neural network may include a pixel-to-pixel (Pix2Pix) model. A pixel-to-pixel architecture can be found, for example, in Isola et al., "Image-to-Image Translation with Conditional Adversarial Networks" (2017), the entire contents of which are incorporated herein by reference. In some embodiments, the GAN-based neural network may include a unilateral model. The architecture of a unilateral model can be found, for example, in Shi et al., "A Unilateral Virtual Histological Staining Model for Complex Human Samples" (2022), which is incorporated herein by reference in its entirety. For example, in some embodiments, neural network device 202B can include a diffusion-based neural network (i.e., a diffusion-based model). The architecture of a diffusion-based neural network can be found, for example, in Saharia et al., "Palette: An Image-to-Image Diffusion Model" (2022), which is incorporated herein by reference in its entirety.

[0140] In some embodiments, neural network device 202B has been trained using a training dataset comprising: images of samples (e.g., unsectioned samples), each image acquired by an imaging system having an excitation light source before chemical staining of the sample, the excitation light source providing light to cause one or more substances (e.g., photoluminescent substances) within the sample to emit light; and corresponding chemically stained histological images of the sample, each chemically stained histological image acquired after the corresponding sample was sectioned (in the case of unsectioned samples) and chemically stained (with one or more chemical stains (e.g., histochemical stains) that the neural network device is trained to digitally simulate). In some embodiments, the imaging system is an imaging system for widefield fluorescence microscopy. In some embodiments, the excitation light source is an excitation light source comprising an ultraviolet laser.

[0141] In some embodiments, the above-mentioned image and the corresponding chemically stained histological image form an image pair, and each image pair has one of the sample (eg, unsectioned sample) images and one of the corresponding chemically stained histological images.

[0142] In some embodiments, the training dataset contains labeled data and a neural network device 202B that has been trained under supervised learning conditions. In one embodiment, the labeled data can be provided by registering or aligning each sample image with its corresponding chemically stained histological image at the pixel level.

[0143] In some embodiments, the training dataset includes partially labeled data and a neural network device 202B trained under weakly supervised learning conditions. In one embodiment, the labeled data can be provided by registering or aligning each sample image with its corresponding chemically stained histological image at the tile level (e.g., consisting of multiple pixels, multiple small tiles, etc.).

[0144] In some embodiments, the training dataset includes unlabeled data and a neural network device 202B trained under unsupervised learning (e.g., self-supervised learning). In one embodiment, the unlabeled data may be provided by registering or aligning each sample image with its corresponding chemically stained histological image at both the pixel level and the tile level.

[0145] In one embodiment, for relatively thin samples, the thickness of the sample imaged first is often comparable to the thickness of the chemically stained histological sample (sample after histochemical treatment) imaged subsequently, thereby making the registration and / or alignment of the two images at the pixel level or tile level easier or more accurate.

[0146] In one embodiment, for relatively thick samples (e.g., thicker than 7 microns), since the thickness of the sample imaged first is often quite different (not equivalent) from the thickness of the chemically stained histological sample (the sample after histochemical treatment) imaged subsequently, the contents of the two images are more different, and the ease or accuracy of registration and / or alignment of the two images may be lower.

[0147] In some embodiments, neural network device 202A and neural network device 202B can be connected in series. In one example, the output of neural network device 202A can be directly connected to the input of neural network device 202B, such that an image of an unsectioned sample is first processed by neural network device 202A to obtain an enhanced image, and the enhanced image is then processed by neural network device 202B to obtain a digitally stained histological image.

[0148] Figure 3 Shown is a system 300 for digitally staining a sample in some embodiments of the present invention. Figure 3 As shown, the system 300 includes: an imaging system 302 for obtaining a sample image; and an image processing system 304 for processing the image obtained by the imaging system.

[0149] The imaging system 302 includes one or more excitation light sources, each of which is used to provide corresponding light to make one or more substances (such as photoluminescent substances) in the unsectioned sample emit light, thereby obtaining one or more images of the unsectioned sample, such as Figure 1 An image of an unsectioned sample from operation 100 is shown. Imaging system 302 may be adapted to perform fluorescence imaging, such as widefield fluorescence imaging. In one embodiment, imaging system 302 may be a fluorescence microscope, such as a widefield fluorescence microscope. In another embodiment, imaging system 302 may be an in vivo imaging system, such as a fluorescence endoscope (endofluorescence imaging system).

[0150] The image processing system 304 can be used to perform Figure 1 Operation 100 shown, or Figure 1 and Figure 2 Any of the neural network devices 102, 200 shown operates.

[0151] Image processing system 304 may be operatively connected to imaging system 302. In some embodiments, image processing system 304 may communicate data with imaging system 302 via one or more communication links. In some embodiments, at least a portion of image processing system 304 may constitute a portion of imaging system 302. In some embodiments, image processing system 304 and imaging system 302 are independent systems.

[0152] Figure 4 An imaging system 400 is shown in some embodiments of the present invention. In some embodiments, the imaging system 400 can be used to Figure 1 The imaging system for obtaining the sample image in operation 100 is used as Figure 3 The imaging system shown.

[0153] refer to Figure 4Imaging system 400 includes: a support 402 for supporting an unsectioned biological tissue sample S to be imaged or being imaged; an excitation light source 404 for providing ultraviolet light (e.g., deep ultraviolet light) to induce luminescence of one or more fluorophores in the unsectioned biological tissue sample S; an optical system 406 comprising one or more optical assemblies or devices; and a camera / sensor 408 for capturing a fluorescence image of the unsectioned biological tissue sample S (or, in the case of completely endogenous fluorophores, capturing an autofluorescence image thereof). Due to the large optical thickness of the unsectioned biological tissue sample S (relative to the imaging system 400), the ultraviolet light provided by excitation light source 404 is scattered by biomolecules at multiple depths within the sample S, resulting in a possible loss of resolution and / or contrast in the image of the unsectioned biological tissue sample S. Imaging system 400 also includes a data processing system 410 operatively connected to camera / sensor 408 for processing data acquired by camera / sensor 408. The imaging system 400 further includes a movement mechanism 412 for moving the support 402 (and thus the sample on the support 402) relative to the excitation light source 404, the optical system 406, and / or the camera / sensor 408. The movement mechanism 412 may be driven by a motor and / or may be configured to move the support 402 relative to the excitation light source 404 (e.g., translationally) in at least one dimension, preferably in two dimensions, and more preferably in three dimensions.

[0154] In some embodiments, the excitation light source 404 comprises an LED, such as a UV LED. Additionally or alternatively, in some embodiments, the excitation light source 404 comprises a laser, such as a UV laser. In embodiments where the excitation light source 404 comprises both an LED and a laser, the LED and the laser may be selectively used.

[0155] In some embodiments, the data processing system 410 is used to perform or facilitate Figure 1 Operation 100 is shown. In some embodiments, data processing system 410 is configured to send data received from camera / sensor 408 to another system for further processing.

[0156] In some embodiments, the unsectioned biological tissue sample S is neither placed on or within a glass slide nor between glass slides (thus achieving slide-free imaging). In such embodiments, the imaging system 400 for imaging such unsectioned, slide-free tissue or sample can be referred to as a slide-free imaging system.

[0157] Figure 5 The information processing system 500 of some embodiments of the present invention is shown. In some embodiments, the information processing system 500 can be used as one or more of the following: Figure 3 The image processing system 304 shown; or Figure 4 The data processing system 410 of the imaging system 400 is shown. In some embodiments, the information processing system 500 can be used to perform Figure 1 The illustrated operations 100 perform or facilitate performance of (perform or facilitate performance of) neural network device 102 , or perform or facilitate performance of neural network device 200 .

[0158] The information processing system 500 generally includes appropriate components required for receiving, storing, and executing corresponding computer instructions, commands, and / or codes. Figure 5 As shown, information processing system 500 includes a processor 502 and memory 504. Processor 502 may include one or more of the following: a CPU; an MCU; a GPU; a logic circuit; a Raspberry Pi chip; a digital signal processor (DSP); an application-specific integrated circuit (ASIC); a field-programmable gate array (FPGA); or any other one or more digital or analog circuit systems for interpreting and / or executing program instructions and / or for processing signals, information, and / or data. Memory 504 may include one or more volatile memories (e.g., RAM, DRAM, SRAM, etc.), one or more non-volatile memories (e.g., ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.), or any combination thereof. Memory 504 may store corresponding computer instructions, commands, code, information, and / or data (e.g., the neural network device described above). Memory 504 may store computer instructions for executing or facilitating the execution of methods according to embodiments of the present invention. The processor 502 and the memory 504 may be integrated together or separate from each other (and operatively connected).

[0159] Optionally, the information processing system 500 further includes one or more input devices 506. Examples of such input devices 506 include a keyboard, a mouse, a stylus, an image scanner, a microphone, a tactile / touch input device (e.g., a touch screen), an image / video input device (e.g., a camera), and the like. Optionally, the information processing system 500 further includes one or more output devices 508. Examples of such output devices 508 include a display (e.g., a monitor, a display screen, a projector), speakers, headphones, earphones, a printer, an additive manufacturing machine (e.g., a 3D printer), and the like. The display may include an LCD display, an LED / OLED display, or other suitable display, and may have touch functionality. The information processing system 500 may further include one or more disk drives 512. Such disk drives may include one or more of a solid-state drive, a hard drive, an optical drive, a flash drive, a tape drive, and the like. A suitable operating system may be installed in the information processing system 500 (e.g., the disk drive 512 or the memory 504). The memory 504 and the disk drive 512 may be operated by the processor 502. Optionally, the information processing system 500 also includes a communication device 510 for establishing one or more communication links with one or more other computing devices, such as a server, a personal computer, a terminal device, a tablet computer, a mobile phone, a watch, an Internet of Things device or other computing device. The communication device 510 may include one or more of a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a WiFi transceiver, a Bluetooth transceiver, a radio frequency transceiver, a cellular (2G, 3G, 4G, 5G, 5G or more) transceiver, an optical fiber port, an infrared port, a USB connection or other wired or wireless communication interface. The transceiver can be implemented by one or more devices such as a transmitter and a receiver integrated together, a transmitter and a receiver separated from each other. The communication link can be a wired or wireless link for transmitting commands, instructions, information and / or data. In one embodiment, the processor 502 and the memory 504 (when provided with an input device 506, an output device 508, a communication device 510, and a disk drive 512, such devices may also be optionally included) are directly or indirectly connected to each other via a bus, a peripheral component interconnect (PCI) bus (such as a PCIe bus), a universal serial bus (USB), a fiber optic bus, or other similar bus structures. In one embodiment, at least some of the above components may be wirelessly connected via a network such as the Internet or a cloud computing network. It should be understood by those skilled in the art that Figure 5 The information processing system 500 shown is only an example. In other embodiments, the information processing system 500 may have different structures (for example, with added or reduced components).

[0160] Some more specific embodiments of the present invention are given below.

[0161] Through research, experimentation, and testing, the inventors of the present invention have discovered that slide-free imaging technology based on deep UV excitation light has significant application value. This is because various intrinsic biomolecules and fluorescent dyes within biological samples (such as tissues) can absorb deep UV excitation light, achieving absorption contrast with or without fluorescence, and this absorption contrast can correspond to or be equivalent to histological contrast. In one embodiment, based on the absorption contrast and photoacoustic signal achieved by deep UV excitation light, UV-based photoacoustic microscopy can be used for label-free histological imaging. However, in this example, to achieve high-throughput imaging, a high-repetition-rate pulsed UV laser with a point scanning mechanism may be required, and the cost of such lasers is relatively high. In an exemplary widefield fluorescence imaging technique, the shorter penetration depth of deep UV excitation light may help limit the excitation of fluorophores to the surface of a biological sample (such as unsectioned tissue), thereby potentially facilitating high image contrast at the biological sample surface. In one embodiment, microscopy using UV surface excitation can be used for histological imaging using exogenous UV-excitable fluorescent dyes. However, in this case, the use of fluorescent labels (ie, exogenous fluorescent dyes) may affect subsequent analysis and may be difficult to integrate into clinical practice.

[0162] Zhang et al., in their article “High-throughput Label-free, Slide-free Histological Imaging by Computational Microscopy and Unsupervised Learning” (2022), the entire contents of which are incorporated herein by reference, disclose a label-free imaging technology (method and system) called the high-throughput computational autofluorescence microscopy (CHAMP) method with patterned illumination. The CHAMP method images the surface of biological tissues by stimulating them with deep ultraviolet excitation light, and can achieve image contrast through various endogenous fluorophores, including cellular metabolites (such as reduced nicotinamide adenine dinucleotide phosphate, flavins), structural proteins (such as collagen, elastin), and aromatic amino acids (such as tyrosine, tryptophan, and phenylalanine). The autofluorescence of such fluorophores naturally creates a negative nuclear contrast that generally corresponds to or is substantially equivalent to the contrast of hematoxylin / eosin (HE) staining images. In the CHAMP method, the patterned illumination function is used to achieve a large field of view and a large depth of field using a small numerical aperture (NA) objective lens while improving lateral resolution. In this way, the image contrast of unsectioned tissue can be improved, and a greater tolerance can be achieved for the topography of the tissue surface. A possible disadvantage of the CHAMP design is that the method requires an ultraviolet laser as a coherent light source and generates a speckle pattern through interference, and ultraviolet lasers are relatively expensive. Although incoherent light sources using spatial light modulators (SLMs) or digital micromirror devices (DMDs) can be used for sinusoidal structured illumination microscopy (SIM), and the wavelength range of commercially available DMDs has expanded from the visible light range to the near-infrared and UVA wavelength ranges, this usage is not common because DMDs that can operate in the UVC band experience a rapid drop in reflectivity performance in this band. The inventors of the present invention realized that although deep ultraviolet excitation light can achieve rich molecular contrast, the availability of imaging technologies based on deep ultraviolet excitation light is limited, or even non-existent, due to the limited number of cost-effective deep ultraviolet excitation light illumination tools.

[0163] In some embodiments, an improved imaging system based on the CHAMP imaging system is provided, wherein a low-cost UV LED (M265L5, Thorlabs Inc.) replaces the high-cost UV laser used in the CHAMP method as the excitation light source. Accordingly, in some embodiments, a deep learning-based image processing mechanism implemented by the improved imaging system is provided.

[0164] In some embodiments, a trained generative network for digital staining processes autofluorescence images of unsectioned tissue, where the autofluorescence images are captured using a wide-field microscope with an excitation light source comprising an ultraviolet LED providing deep ultraviolet light. Hereinafter, images captured using such a wide-field microscope are referred to as "wide-field LED images." Figure 6 Shown are exemplary operations in such an embodiment. Figure 6 As shown, low-resolution autofluorescence images of unsectioned tissue taken with a wide-field microscope equipped with a UV LED are processed by a trained generative network for digital staining to generate digitally stained histological images.

[0165] In some embodiments, multiple trained generative models for image enhancement and digital staining are used to process autofluorescence images of unsectioned tissue captured by a wide-field microscope with an excitation light source comprising an ultraviolet LED providing deep ultraviolet light. Figure 7 Shown are exemplary operations in such an embodiment. Figure 7 As shown, a low-resolution autofluorescence image of unsectioned tissue captured by a wide-field microscope with an ultraviolet LED is processed by a trained generative network for image enhancement to generate an enhanced image with higher resolution and / or contrast, which is then processed by a trained generative network for digital staining to generate a digitally stained histological image. In some embodiments, the trained generative network for image enhancement is trained based on an algorithm based on a generative adversarial network (GAN) for enhancing resolution and contrast. In some embodiments, the trained generative network for image enhancement includes a GAN-based deep learning algorithm for image enhancement.

[0166] In some embodiments, a trained generative network for image enhancement is used to convert low-resolution widefield LED images (fluorescence images) into high-resolution images (fluorescence images) corresponding to images acquired by a CHAMP imaging system (using an ultraviolet laser providing deep ultraviolet light as an excitation light source). Hereinafter, such digitally generated high-resolution images corresponding to images captured by the CHAMP imaging system are referred to as "LED-CHAMP images." To achieve this effect, the trained generative network for image enhancement may, for example, include a super-resolution algorithm trained on the following images: low-resolution widefield LED images acquired by an improved imaging system using an ultraviolet LED as an excitation light source; and images acquired by a CHAMP imaging system using an ultraviolet laser as an excitation light source as a target. Hereinafter, images captured by the CHAMP imaging system are referred to as "laser-CHAMP images." In some embodiments, the generated LED-CHAMP images are further processed by a trained generative network for digital staining to obtain digitally stained histological images that generally correspond to or are substantially equivalent to digital HE-stained histological images.

[0167] In some embodiments, the trained generative network for image enhancement can be based on a generative adversarial network, such as a super-resolution generative adversarial network. For example, a generative adversarial network can be found in Goodfellow et al., Generative Adversarial Networks (2020), and a super-resolution generative adversarial network can be found in Ledig et al., Photorealistic Single Image Super-Resolution with Generative Adversarial Networks (2017). In one embodiment, an enhanced SRGAN (ESRGAN) developed based on SRGAN and improved in network architecture, adversarial loss, and perceptual loss is used. For example, an ESRGAN can be found in Wang et al., ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (2019). In other embodiments, other GAN-based networks can also be used for image enhancement.

[0168] Compared to the laser-CHAMP method (the CHAMP method disclosed by Zhang et al.), in some embodiments, the LED-CHAMP method (i.e., the improved CHAMP method described in the above embodiments) can improve imaging throughput. This is because the LED-CHAMP method only requires the acquisition of a single image, while the laser-CHAMP method requires the acquisition of a series of patterned illumination images. Therefore, the LED-CHAMP method saves time and effort in image reconstruction and provides results more quickly.

[0169] Although the mechanism is described as being used for deep ultraviolet excitation light structured light illumination in the above specific embodiments, it should be understood that the mechanism can also be applied to other imaging technologies, such as other imaging technologies based on deep ultraviolet light.

[0170] In some embodiments, laser CHAMP images obtained by the CHAMP imaging system are used as a bridge to obtain high-resolution training targets for use with the LED-CHAMP super-resolution algorithm.

[0171] In one embodiment, widefield LED images and laser CHAMP images are captured using the same inverted widefield autofluorescence microscope, which utilizes a 4× objective (RMS4X, NA = 0.1, Thor Labs) and two illumination paths (excitation paths): one for widefield LED images and the other for laser CHAMP images. Widefield LED images can be captured using 265nm LED illumination. Consistent with the disclosed CHAMP system, raw laser CHAMP image sequences can be acquired under speckle illumination conditions created by a 266nm UV laser and a UV fused silica ground glass diffuser (DGUV10-600, Thor Labs). In one embodiment, a laser CHAMP image sequence consisting of 36 images is acquired for each field of view at a scan interval of 1μm. The raw speckle illumination image sequence obtained by the CHAMP method can then be used to generate high-resolution laser CHAMP images through an iterative image reconstruction mechanism. In general, the image acquisition time and image reconstruction time of the CHAMP method increase with the number of fields required to scan the entire sample and may become time-consuming.

[0172] In some LED-CHAMP method implementations, the generative network used for image enhancement is an ESRGAN trained on widefield LED images and laser CHAMP images. During training, the widefield LED images are paired with the laser CHAMP images. In one embodiment, the widefield LED and laser CHAMP images are first globally aligned for rotation, translation, and scaling using control point registration. Subsequently, MATLAB performs local registration at the tile level using intensity-based affine transformation registration. The paired widefield LED and laser CHAMP images are used for training and evaluation of the ESRGAN model.

[0173] ESRGAN can be trained with aligned data. In some embodiments, the ESRGAN architecture includes a generator G and a discriminator D. The generator G can infer an LED-CHAMP image with a magnification factor (e.g., 2× or 3×) from the wide-field LED image x according to the upsampling factor used in the laser-CHAMP reconstruction. In one embodiment, the discriminator D is based on a relative GAN, such as the one disclosed in Jolicoeur-Martineau, “Relative Discriminators: The Missing Key Element in Standard GANs” (2019), and is trained on LED-CHAMP images. and the original laser CHAMP image y. The adversarial loss between the generator G and the discriminator D can be expressed as a binary cross entropy loss: and

[0174]

[0175]

[0176] Here, σ represents the Sigmoid function used for post-processing the discriminator output.

[0177] In such embodiments, the LED-CHAMP image The content of should be the same as the original laser CHAMP image y. The L1 loss function is used to evaluate the distance between these two types of images:

[0178]

[0179] In order to measure the perceptual similarity between LED-CHAMP images and laser-CHAMP images, the perceptual loss is calculated using the representative features of the last convolutional layer of the pre-trained VGG19 network. Therefore, the total generator loss is:

[0180]

[0181] In some embodiments, the generator architecture is based on SRResNet with 12 basic blocks, where the basic blocks are residual dense residual blocks (RRDB). Figure 8 An example generator architecture and training operation in such an embodiment are shown. In some embodiments, the generator's upsampling layer uses a pixel shuffle method with an upscaling factor of 2× or 3×. In some embodiments, both the input wide-field LED image and the output LED-CHAMP image are single-channel grayscale images. The discriminator architecture uses PatchGAN to distinguish between LED-CHAMP images and laser-CHAMP images.

[0182] In some embodiments, as an alternative to ESRGAN, a weakly supervised method is used to train data with poor alignment. Such weakly supervised methods and related network structures can be found in the article "A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transfer" by Dai et al. Such a network structure is basically an extended version of the basic GAN architecture after extending it with a convolutional neural network. In such an embodiment, the network includes two generators and two discriminators. For a given training sample (S i , T i ), a generator will S i Convert to T i , another generator will T i Convert to S iIn an embodiment where training is performed between widefield LED images and light sheet images (as described below, images obtained using a light sheet microscope with a UV laser as the illumination / excitation light source), S i For wide-field LED images, T i In such embodiments, each generator includes: an encoder that projects the input into a latent space shared by all generators; and a decoder that performs a conversion from the latent space to the original space.

[0183] In such implementations, the reconstruction loss is calculated as the minimum absolute deviation between the input image and the decoded image of the same generator:

[0184]

[0185] Among them, E S and U S are the encoder and decoder of the source image domain, E T and U T are the encoder and decoder of the target image domain, respectively.

[0186] In this implementation, the discriminator not only checks whether the image is real or fake, but also checks which region the image is from if it is real. This functionality can be incorporated into the loss function as follows:

[0187]

[0188] Among them, CE d is the cross entropy loss used to update the discriminator in iterations, D S and D T For each discriminator, Y i and Y0 are the one-hot encoded labels of region i and pseudo image, respectively.

[0189] The cross entropy loss used to update the generator in an iteration is defined as:

[0190]

[0191] In addition to cross-entropy, the network also includes a cycle consistency loss, which is defined as the sum of the forward consistency loss from source to target image and the backward consistency loss from target image to source image:

[0192]

[0193] In such embodiments, the generator and discriminator can also be optimized with loss functions different from those described above. In some embodiments, the loss function of the generator can be defined as:

[0194]

[0195] Among them, λ1 and λ2 are the weights of cycle consistency loss and reconstruction loss respectively. The loss function of the discriminator is defined as:

[0196]

[0197] In some embodiments, the generative network for digital staining can be based on a GAN. In one embodiment, paired data can be obtained by imaging the sample before and after staining. In one embodiment, for thinner tissue samples, the generative network for digital staining is based on a supervised deep learning algorithm called "Pix2pix" published by Isola et al. in "Image-to-Image Translation with Conditional Adversarial Networks" (2017).

[0198] In one embodiment, for thicker tissue samples, ideal registration or alignment of paired fluorescence and histological images is not possible because only the HE layer near the surface of the thicker tissue being imaged is available (due to losses during sample preparation and sample deformation). For this reason, it is difficult to use a fully supervised algorithm for training, and weakly supervised or unsupervised algorithms are preferred. In one embodiment, an unsupervised single-sided algorithm can be used for digital staining, such as the unsupervised single-sided algorithm disclosed in "A Single-Sided Virtual Histological Staining Model for Complex Human Samples" by Shi et al. (2022).

[0199] In order to better demonstrate the effects of the above-mentioned embodiments, the following is a comparison between wide-field LED images, LED-CHAMP images, and laser CHAMP images of formalin-fixed paraffin-embedded (FFPE) mouse brain slices and formalin-fixed human lung cancer thick samples. In such embodiments, the samples are fixed in a 10% formalin solution. In order to obtain FFPE mouse brain tissue slices, the fixed samples are embedded in paraffin and cut into 5 μm slices with a microtome. The tissue sections are placed on quartz slides and imaged after dewaxing. After imaging, the sections are stained with HE according to standard histological methods. For thick tissue, the sample is first imaged. After imaging, the sample is embedded in a paraffin block, then sliced, dewaxed, and stained with HE to collect the corresponding histological images. In this example, all the HE-stained images obtained in the above manner were digitized into whole-slide images using a digital slide scanner (20×, NA=0.75, NanoZoomer, Hamamatsu Photonics KK).

[0200] Figures 9a to 9i Figure 1 is an example of an imaging result of resolution enhancement and downstream digital staining performance evaluation of 5 μm optically thin FFPE mouse brain tissue (although not shown in the figure, Figure 9a 、 Figure 9c 、 Figure 9d 、 Figure 9e is a grayscale image, and Figure 9b 、 Figure 9f 、 Figure 9g 、 Figure 9h 、 Figure 9i These results show a comparison of a wide-field LED image of a 5 μm FFPE mouse brain slice, an image obtained after deep learning-based image enhancement using a high lateral resolution image as a training target (referred to as an "enhanced image," i.e., "LED-CHAMP image"), a high lateral resolution image (training target image, i.e., "Laser-CHAMP image"), and the corresponding HE-stained image. More specifically, Figure 9a This is an enhanced autofluorescence image of the mouse brain; Figure 9b for Figure 9a The corresponding digital staining images; Figures 9c to 9e for Figure 9a A magnified image of the wide-field LED image, enhanced image (“LED-CHAMP image”), and high lateral resolution image (“Laser-CHAMP image”) of the area in box 9a; Figure 9f to Figure 9h For the respective Figures 9c to 9e corresponding digitally stained images; Figure 9i for Figure 9a The corresponding chemical staining (HE staining) histological image of the area in box 9a. Figures 9f to 9i It can be seen that although the digital HE staining images obtained based on the wide-field LED images show similar cell nucleus features to the actual HE staining images, the digital HE staining images obtained based on the LED-CHAMP images are better than those obtained based on the wide-field LED images.

[0201] Comparing the digital HE staining images obtained from LED-CHAMP images with those obtained from widefield LED images reveals that the missing cell nuclei (indicated by arrows) in the digital HE staining images obtained from widefield LED images are visible in the digital HE staining images obtained from LED-CHAMP images. Furthermore, using the actual HE staining images as the local ground truth, the structural similarity index (SSIM) of the digital HE staining images obtained from widefield LED images, LED-CHAMP images, and laser-CHAMP images was further evaluated. The resulting SSIM values were 0.399, 0.402, and 0.405, respectively. These SSIM values indicate that the digital staining based on LED-CHAMP images is superior to that based on widefield LED images.

[0202] Figures 10a to 10iThe imaging results of the resolution enhancement and downstream digital staining performance evaluation of 500 μm optically thick FFPE mouse brain tissue in one embodiment (although not shown in the figure, Figure 10a 、 Figure 10c 、 Figure 10d 、 Figure 10e is a grayscale image, and Figure 10b 、 Figure 10f 、 Figure 10g 、 Figure 10h 、 Figure 10i These results show a comparison of wide-field LED images of a thick fixed mouse brain tissue sample, images obtained after deep learning-based image enhancement using high lateral resolution images as training targets (referred to as "enhanced images," i.e., "LED-CHAMP images"), and high lateral resolution images (training target images, i.e., "Laser-CHAMP images"). More specifically, Figure 10a This is an enhanced autofluorescence image of the mouse brain; Figure 10b for Figure 10a The corresponding digital staining images; Figures 10c to 10e for Figure 10a A magnified image of the wide-field LED image, the enhanced image (“LED-CHAMP image”), and the high lateral resolution image (“Laser-CHAMP image”) of the area in box 10a; Figures 10f to 10h For the respective Figures 10c to 10e corresponding digitally stained images; Figure 10i for Figure 10a The corresponding chemical staining (HE staining) histological image of the area in the box 10a. Figure 10c Compared with the wide field LED image, Figure 10d The LED-CHAMP images of the 3D-UV imaging system can better show the densely packed nuclei in the hippocampus. In addition, the SSIM of the LED-CHAMP images is higher (0.809 and 0.746), which supports this conclusion.

[0203] In this example, three digital HE staining images are generated based on the wide-field LED image, LED-CHAMP image, and laser CHAMP image, as shown in Figure 2. Figures 10f to 10h As shown in Figure 2, HE-stained sections with exactly the same tissue layers as those imaged by the dual-mode autofluorescence microscope (used to obtain the above images) could not be obtained, Figure 10i As shown in the figure, only the adjacent layers stained with HE were obtained for reference. Although the digital HE staining images based on wide-field LED images can show the hippocampal structure, the digital staining performance of LED-CHAMP images is better. In general, the digital HE staining images based on LED-CHAMP images, the digital HE staining images based on laser CHAMP images, and the adjacent layer HE staining images ( Figure 10g to Figure 10i), while the digital HE staining images obtained based on wide-field LED images showed a lower nuclear density and incorrect nuclear boundaries.

[0204] Figures 11a to 11i The following are imaging results related to resolution enhancement of 500μm optically thick human lung tissue in one embodiment. These results show a comparison of a wide-field LED image of 500μm formalin-fixed thick lung cancer tissue (human lung adenocarcinoma tissue), an image obtained after deep learning-based image enhancement using a high lateral resolution image as a training target (referred to as an "enhanced image," i.e., a "LED-CHAMP image"), and a high lateral resolution image (training target image, i.e., a "laser-CHAMP image"). More specifically, Figures 11a to 11c They are wide-field LED image, enhanced image and high lateral resolution image of the tissue respectively. Figure 11d 、 Figure 11f and Figure 11h They are Figure 11a The wide-field LED image, enhanced image, and magnified image of the high-lateral-resolution image in the area of square frame 11a. With the resolution enhancement effect, the enhanced image shows a more distinct fiber structure. Figure 11e 、 Figure 11g and Figure 11i They are Figure 11a A magnified image of the widefield LED image, enhanced image, and high-lateral-resolution image of another region shown in box 11b. With improved resolution and contrast, the enhanced image clearly displays alveolar macrophages. In the LED-CHAMP image, which has higher lateral resolution and contrast than the widefield LED image, fibrous features within grouped alveolar macrophages and individual cells can be more clearly and distinctly identified. Using the laser-CHAMP image as a reference, the SSIM achieved by the LED-CHAMP image (0.796) is greater than that of the widefield LED image (0.743).

[0205] Figures 12a to 12g The following is an imaging result of improving the digital staining performance of 500 μm optically thick human lung tissue according to the LED-CHAMP implementation in an embodiment (although not shown in the figure, Figure 12a 、 Figure 12c 、 Figure 12d is a grayscale image, and Figure 12b 、 Figure 12e 、 Figure 12f 、 Figure 12g These results show the comparison of digital staining images generated from wide-field LED images, digital staining images generated from enhanced images, and actual chemical staining images of thick human lung adenocarcinoma tissue. More specifically, Figure 12a Enhanced images of human lung tissue; Figure 12b for Figure 12a Digital staining images of Figure 12c and Figure 12d for Figure 12a The wide-field LED image and the close-up image of the enhanced image in the area “12a” in the box; Figure 12e The corresponding adjacent chemical staining images are for the solid areas; Figure 12f and Figure 12g For the respective Figure 12c and Figure 12d The corresponding digital HE staining close-up image. These results can be used to verify the digital generation process of the histological digital staining image proposed in the present invention. In this example, in view of the complexity of human cancer tissue samples, the digital staining performance has been optimized based on the above-mentioned unsupervised unilateral model. In addition, in this example, the digital HE staining image obtained based on the LED-CHAMP image is similar to the histological pattern of acinar adenocarcinoma, a subtype of lung adenocarcinoma. In contrast, the resolution of the digital HE staining image generated based on the wide-field LED image is low and cannot display the circular glandular structure (such as Figure 12f As shown, the central cavity is filled with artifacts resembling carbon particles and blood. This result suggests that image enhancement can help improve the performance of digital staining for relatively complex sample images.

[0206] The deep learning-assisted mechanism described in the above embodiments can be transplanted to other wide-field imaging technologies.

[0207] In another example, a set of (sample) widefield LED images was acquired using a widefield microscope employing a UV LED as the excitation illumination source, and a set of image sequences was acquired using an open-top light-sheet microscope employing a UV laser as the excitation illumination source. In this example, for comparative purposes, the two modalities were integrated so that they shared the same detection configuration using a 5× UV objective lens (LMU-5X-NUV, NA = 0.12, Saul Laboratories). Each image sequence was further processed to reconstruct a three-dimensional (3D) volume. Furthermore, in this example, a two-dimensional layer representing most tissue surface features was extracted from the 3D volume. Overall, the processed light-sheet images (i.e., images generated and processed by the light-sheet microscope) can have higher axial resolution and, therefore, better imaging contrast than widefield LED images acquired using a simple widefield microscope. In this embodiment, the processed images (i.e., processed light-sheet images) were used as target images and, along with the widefield LED images, were used to train a generative model to generate output images with enhanced contrast (the generated output images with enhanced contrast are referred to as "LED-LS" images). Furthermore, in this example, given the essential difference in imaging thickness between widefield LED images and light-sheet images, a mask generated from the light-sheet image based on the tissue contents in the widefield LED image is also used to reduce errors in training. Furthermore, the training in this example uses a weakly supervised GAN-based model disclosed in Dai et al., "A Weakly Supervised Deep Generative Model for Complex Image Restoration and Style Transfer" (2022). This type of weakly supervised model does not require perfect alignment between the widefield LED image and the high-resolution training target image.

[0208] Figures 13a to 13f The following are the results of image enhancement related imaging of another widefield imaging technique. These results show the comparison of widefield LED image, enhanced image (i.e. LED-LS image) and processed light sheet image of formalin-fixed lung cancer tissue with a large optical thickness of 500μm. More specifically, Figure 13a This is a wide-field LED image of a thick sample of formalin-fixed lung cancer tissue; Figure 13b This is the enhanced image of a thick sample of formalin-fixed lung cancer tissue (i.e., LED-LS image); Figure 13c High axial resolution and high contrast enhanced images of formalin-fixed thick lung cancer tissue samples (i.e., processed light sheet images); Figures 13d to 13f They are Figure 13aBox 13a shows a widefield LED image, LED-LS image, and a close-up of the processed light-sheet image of a thick section of formalin-fixed lung cancer tissue. As can be seen, the LED-LS image in this example achieves significantly improved nuclear contrast and axial resolution compared to the widefield LED image, thanks to the enhanced axial resolution provided by light-sheet microscopy.

[0209] Some of the above-mentioned specific embodiments provide a deep learning-assisted mechanism for enhancing images, especially for enhancing resolution and contrast, and demonstrate its compatibility with subsequent digital staining tasks based on histological images of mouse brain and human lung tissue using ultraviolet LED as the excitation illumination light source. The above-mentioned embodiments only imaged and tested fixed tissues, but the present mechanism can also be well applied to fresh unfixed tissues. The neural network device or algorithm for image enhancement and digital staining in the above-mentioned embodiments is run by a workstation equipped with a Core I9-10980XE CPU @ 4.80Ghz and 8×32GB RAM, and uses 1 NVIDIAGeForce RTX3090 GPU. Accordingly, in the above-mentioned embodiment using a 2× magnification factor for human lung data, the calculation time of the algorithm is approximately 12s / 10mm. 2 , which is computationally much more efficient than the reconstruction process in the CHAMP method. It is important to note that by further accelerating the algorithm with more and / or more powerful GPUs, the computational efficiency of this algorithm can be further improved to the point where it is suitable for fresh tissue imaging in the operating room or for in vivo imaging.

[0210] This deep learning-assisted image enhancement mechanism is particularly suitable for generating digitally stained images of thick samples (such as optically thick samples).

[0211] The deep learning-assisted image enhancement mechanism in some embodiments can be extended to other deep ultraviolet wide-field imaging technologies for image enhancement. For example, for methods that must use high-cost ultraviolet lasers, the deep learning-assisted image enhancement mechanism can be used as a cost-effective alternative. For example, for more time-consuming image enhancement processing (such as depth of field extension and deconvolution processing during high-resolution scanning), the deep learning-assisted image enhancement mechanism can be used as a high-efficiency alternative. For example, for rapid large-area scanning in three-dimensional label-free histology, such as using paraffin blocks, the deep learning-assisted image enhancement mechanism can be used to shorten the ultraviolet exposure time by increasing the field of view. In some embodiments, the image enhancement method can be implemented as a computer-assisted function through software to reduce or avoid the need for additional hardware, and can also be implemented as an additional function of imaging equipment such as microscopes.

[0212] Some embodiments of the present invention provide a slide-free histological imaging technology (system and method) based on autofluorescence, using a light-emitting diode (LED) as an excitation light source and employing a deep learning algorithm. Some embodiments of the present invention can be used to generate high-resolution and / or high-contrast enhanced images from low-resolution images, which can be further processed to generate digitally stained histological images for diagnostic purposes.

[0213] In some embodiments, a generative deep learning algorithm is used to enhance the resolution and / or contrast of an image acquired using a widefield microscope using an LED as the excitation illumination source ("widefield LED image"). This algorithm can be used to convert a low-resolution widefield LED image into an enhanced output image with high resolution and / or high contrast. In some embodiments, another generative deep learning algorithm is used to convert the enhanced output image with high resolution and / or high contrast into a digitally stained histology image.

[0214] In some embodiments, by combining the use of UV LEDs with deep learning-assisted algorithms, various improvements can be achieved: for example, by using UV LEDs as excitation light sources, the cost-effectiveness can be improved compared to using UV lasers as excitation light sources; for example, because only a single image needs to be acquired for each field of view (rather than an image sequence consisting of multiple images), a higher imaging speed can be achieved; for example, because the computational time required to generate high-resolution images is short, the deep learning-assisted algorithm can achieve higher computational efficiency. In some embodiments, the deep learning-assisted mechanism can be applied to other deep UV wide-field imaging techniques, such as deep UV wide-field imaging techniques that require the use of UV lasers or that are more time-consuming to acquire and process images. In some embodiments, the deep learning-assisted mechanism can be used for other high-resolution learning / training targets, such as images acquired by light sheet microscopy, or images acquired by a combination of deconvolution and depth of field extension.

[0215] In certain embodiments, the present invention can be used to generate digitally stained images for cancer diagnosis. In certain embodiments, the deep learning-assisted mechanism in some embodiments can help transform slide-free imaging techniques into diagnostic aids. In one example, the deep learning-assisted mechanism can be incorporated into an imaging device or portable imaging device (such as a microscope based on the CHAMP method described above) to achieve further image enhancement capabilities (e.g., additional functions).

[0216] Although not necessarily required, one or more embodiments described in conjunction with the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for use by developers, or may be incorporated into other software applications, such as an operating system for a terminal device or computer, or an operating system for a portable computing device. In one or more embodiments, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, those skilled in the art will understand that the functionality of a software application may be distributed across multiple routines, objects, and / or components while still achieving the same functionality as described herein.

[0217] Furthermore, it should be understood that when the methods and systems of the present invention are implemented in whole or in part by a computing system, any suitable computing system architecture may be employed. Such architectures include stand-alone computers, networked computers, or dedicated or non-dedicated hardware devices. Where the terms "computing system" and "computing device" are used, such terms are intended to encompass, for example, any suitable computer or information processing hardware device capable of performing the functions described.

[0218] Those skilled in the art will appreciate that other embodiments of the present invention may be achieved by changing and / or modifying the embodiments of the present invention described in the specification and / or shown in the drawings. Therefore, the embodiments of the present invention described in the specification and / or shown in the drawings should be considered for illustrative purposes, not for restrictive purposes, from any perspective. The "Summary of the Invention" and "Detailed Description of the Invention" sections provide exemplary optional features of some embodiments of the present invention. Some embodiments of the present invention may include one or more of such exemplary optional features. Some embodiments of the present invention may not include one or more of such exemplary optional features.

[0219] For example, in some embodiments, at least a neural network device for performing digital staining can provide a digitally stained histological image of an unsectioned sample by processing multiple images of the unsectioned sample. The multiple images of the unsectioned sample can be obtained from the same imaging system using different excitation light sources, each of which is used to stimulate a corresponding optical response of the unsectioned sample. For example, in some embodiments, the architecture of the neural network device can differ from that of the illustrated embodiment. While the present disclosure focuses on image processing (e.g., digital staining) of unsectioned samples, the disclosed technology can also be used for image processing (e.g., digital staining) of sectioned samples.

Claims

1. A computer-implemented method for processing an image, characterized in that include: receiving at least one image of an unsectioned sample, the image being acquired by an imaging system having an excitation light source, wherein the excitation light source provides light to cause one or more substances in the unsectioned sample to emit light to image the unsectioned sample; and processing at least the image by a neural network device for performing digital staining to obtain a digitally stained histological image of the unsectioned sample, The digitally stained histological image generally corresponds to or is substantially equivalent to an image of the unsectioned sample after chemical staining with one or more chemical stains.

2. The computer-implemented method of claim 1 , wherein: The neural network device is used to perform image enhancement and digital staining to provide the digitally stained histological image.

3. The computer-implemented method of claim 2, wherein: The image enhancement includes spatial resolution enhancement and / or contrast enhancement.

4. The computer-implemented method of claim 3, wherein: The spatial resolution enhancement includes axial resolution enhancement and / or lateral resolution enhancement.

5. The computer-implemented method of any one of claims 2 to 4, wherein: The neural network device comprises: a first neural network device for performing said digital coloring; and A second neural network device is provided for performing said image enhancement.

6. The computer-implemented method of claim 5, wherein: The first neural network device includes a generative model based on deep learning; and / or The second neural network device includes a generative model based on deep learning.

7. The computer-implemented method of claim 5 or 6, wherein: The second neural network device has been trained with a training data set comprising a plurality of images, wherein the plurality of images comprises a plurality of sample low-quality images and corresponding high-quality images; and The low-quality image has lower axial resolution, lower lateral resolution and / or lower contrast than the corresponding high-quality image.

8. The computer-implemented method of claim 7, wherein: Each of the plurality of low-quality images is registered or aligned with a corresponding one of the plurality of high-quality images at a pixel level to provide annotation data; as well as The second neural network device has been trained with the labeled data under supervised learning.

9. The computer-implemented method of claim 7, wherein: Each of the plurality of low-quality images is not registered or aligned with a corresponding one of the plurality of high-quality images to provide unlabeled data; and The second neural network device has been trained with the unlabeled data under unsupervised learning.

10. The computer-implemented method of claim 7, wherein: Each of the plurality of low-quality images is registered or aligned with a corresponding one of the plurality of high-quality images at a tile level to provide partial annotation data; as well as The second neural network device has been trained with the partially labeled data under weakly supervised learning.

11. The computer-implemented method of any one of claims 7 to 10, wherein: The plurality of images of the training data set include a plurality of pairs of images, each pair of images including a low-quality image and a corresponding high-quality image of a corresponding sample; as well as Wherein, for each pair of images, the low-quality image and the corresponding high-quality image are obtained by the same imaging system having a first excitation light source and a second excitation light source, wherein the first excitation light source provides light so that one or more substances in the sample emit light so as to image the sample to obtain the low-quality image, and wherein the second excitation light source provides light so that one or more substances in the sample emit light so as to image the sample to obtain the corresponding high-quality image.

12. The computer-implemented method of any one of claims 5 to 11, wherein: The first neural network device has been trained with a training data set comprising a plurality of images, wherein the training data set comprises: a plurality of first images of the plurality of samples, each first image being acquired by an imaging system having an excitation light source before the corresponding sample is chemically stained, wherein the excitation light source provides light to cause one or more substances in the corresponding sample to emit light; and A plurality of corresponding second chemically stained histological images of the sample, wherein each second chemically stained histological image is obtained after the corresponding sample is chemically stained.

13. The computer-implemented method of claim 12, wherein: Each of the plurality of first images is registered or aligned with a corresponding one of the plurality of second images at a pixel level to provide annotation data; as well as The first neural network device has been trained with the labeled data under supervised learning.

14. The computer-implemented method of claim 12, wherein: Each of the plurality of first images is not registered or aligned with a corresponding one of the plurality of second images to provide unlabeled data; and The first neural network device has been trained with the unlabeled data under unsupervised learning.

15. The computer-implemented method of claim 12, wherein: Each of the plurality of first images is registered or aligned with a corresponding one of the plurality of second images at a tile level to provide partial annotation data; as well as The first neural network device has been trained with the partially labeled data under weakly supervised learning.

16. The computer-implemented method of any one of claims 5 to 15, wherein: An output terminal of the second neural network device is connected to an input terminal of the first neural network device.

17. The computer-implemented method of any one of claims 1 to 16, wherein: The imaging system is an imaging system for performing fluorescence imaging; and The one or more substances include one or more photoluminescent substances, in particular endogenous fluorophores.

18. The computer-implemented method of claim 17, wherein: The imaging system is an imaging system for performing fluorescence microscopy imaging.

19. The computer-implemented method of any one of claims 1 to 18, wherein: The excitation light source provides ultraviolet light to cause the one or more substances in the sample to emit light; and Wherein, the ultraviolet light includes ultraviolet light in the UVC band.

20. The computer-implemented method of claim 19, wherein: The excitation light source includes an ultraviolet LED capable of providing the ultraviolet light.

21. The computer-implemented method of any one of claims 1 to 20, wherein: The unsectioned sample includes a biological sample such as tissue.

22. The computer-implemented method of claim 21, wherein: The imaging system is for performing in vivo imaging; and / or The imaging system is a handheld imaging system.

23. The computer-implemented method of any one of claims 1 to 22, wherein: The unsectioned sample is an unstained sample.

24. An image processing system, characterized in that: include: one or more processors; as well as A memory storing one or more programs to be executed by the one or more processors, wherein the one or more programs include instructions for performing the computer-implemented method of any one of claims 1 to 23.

25. A carrier medium carrying computer-readable instructions, characterized in that: The computer-readable instructions are used to cause one or more processors to perform the computer-implemented method of any one of claims 1 to 23.

Images