Method for generating digitally stained images from unstained biological samples

Abstract

The present disclosure provides systems and methods for generating a virtually stained image of an unstained biological sample based on a captured image of the unstained biological sample, wherein the virtually stained image appears to have the appearance of the unstained biological sample as if it were chemically stained with a morphological stain.

Description

Technical Field

[0001] This disclosure relates to a microscopy method and system that utilizes deep neural network learning for virtual morphological staining of one or more images derived from unstained histological or cytological samples. The deep learning neural network is used to virtually morphologically stain one or more input images derived from unstained histological or cytological samples into one or more output images, which are equivalent (e.g., diagnostically equivalent) to bright-field images of the same morphologically stained sample. Background Technology

[0002] Microscopic imaging of tissue samples is a fundamental tool for diagnosing a wide range of diseases. Histopathologists use chemical staining techniques on tissue samples to highlight their microstructure and composition, thereby searching for abnormalities that indicate the presence, nature, and extent of disease. However, microscopic imaging of tissue samples is a time-consuming and expensive process. For example, the preparation of stained tissue samples involves fixing the tissue sample in formalin, embedding the formalin-fixed tissue sample to provide formalin-fixed paraffin-embedded (FFPE) tissue samples, sectioning the FFPE tissue samples into thin slides, staining the tissue slides, and mounting the stained slides onto glass slides for microscopic imaging.

[0003] Furthermore, tissue processing and staining are destructive. In fact, the aforementioned steps in preparing stained tissue samples use multiple reagents and introduce irreversible effects onto the tissue sample. Additionally, different assays often require multiple tissue sections, which can quickly deplete valuable biopsy samples and increase the likelihood of repeat biopsies from patients, further increasing costs, patient pain, and precious time. Moreover, each assay requires highly trained and scarce tissue technicians, generates chemical waste, and necessitates costly physical storage of the resulting glass slides for many years. Summary of the Invention

[0004] Systems and methods for generating virtual stained images of unstained biological samples (including histological and cytological samples) are desired. Systems and methods that reduce the amount of time required to generate useful stained images of biological samples are also desired. Systems and methods that facilitate the generation of multiple virtual stained images from a single acquired image derived from an unstained biological sample are also desired, thereby reducing the need for multiple tissue sections. To meet these needs, the applicant has developed systems and methods that facilitate the generation of one or more virtual stained images from an image derived from an unstained biological sample, wherein the virtual stained image is equivalent (e.g., diagnostically equivalent) to a corresponding bright-field image of the same biological sample that has been chemically stained. These and other embodiments are described herein.

[0005] A first aspect of this disclosure is a system for generating a virtual stained image of a test unstained biological sample disposed on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: (a) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; (b) supplying the obtained multispectral test transmission image of the test unstained biological sample to a virtual staining engine, wherein the virtual staining engine is trained to generate an image of the unstained biological sample stained with a staining agent, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples, (i) wherein a first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images; (ii) The second training image in each pair of co-registered training images comprises training bright-field image data of the same biological sample stained with a staining agent; and (c) a virtual staining image of a test unstained biological sample stained with a staining agent is generated using a trained virtual staining engine; wherein each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels. In some embodiments, a bright-field image acquisition device is used to acquire the training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire the training bright-field image data. In some embodiments, the virtual staining engine is not trained using fluorescence images.

[0006] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from two or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these two or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0007] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from three or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these three or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0008] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from four or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these four or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0009] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from twelve or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these twelve or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0010] In some embodiments, the first training image is derived from two or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the two or more training multispectral transmission image channels to derive the first training image.

[0011] In some embodiments, the first training image is derived from three or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the three or more training multispectral transmission image channels to derive the first training image.

[0012] In some embodiments, the first training image is derived from four or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the four or more training multispectral transmission image channels to derive the first training image.

[0013] In some embodiments, the first training image is derived from twelve or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the twelve or more training multispectral transmission image channels to derive the first training image.

[0014] In some embodiments, (i) at least three test multispectral transmission image channel images are acquired; and (ii) at least three training multispectral transmission image channel images are acquired; and wherein each of the at least three test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least three training multispectral transmission image channel images.

[0015] In some embodiments, (i) at least four test multispectral transmission image channels are acquired; and (ii) at least four training multispectral transmission image channels are acquired; wherein each of the at least four test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least four training multispectral transmission image channels. In some embodiments, the test multispectral transmission image is generated by performing dimensionality reduction (e.g., principal component analysis) on the at least four test multispectral transmission image channels. In some embodiments, a first training image in each pair of co-registered training images is generated by performing dimensionality reduction on the at least four training multispectral transmission image channels.

[0016] In some embodiments, (i) at least six test multispectral transmission image channels are acquired; and (ii) at least six training multispectral transmission image channels are acquired; wherein each of the at least six test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least six training multispectral transmission image channels. In some embodiments, the test multispectral transmission image is generated by dimensionality reduction of the at least six test multispectral transmission image channels. In some embodiments, a first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least six training multispectral transmission image channels.

[0017] In some embodiments, (i) at least twelve test multispectral transmission image channels are acquired; (ii) at least twelve training multispectral transmission image channels are acquired; and wherein each of the at least twelve test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least twelve training multispectral transmission image channels. In some embodiments, the test multispectral transmission image is generated by dimensionality reduction of the at least twelve test multispectral transmission image channels. In some embodiments, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least twelve training multispectral transmission image channels.

[0018] In some embodiments, the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm. In some embodiments, the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm, and / or about 850 + / - 10 nm. In some embodiments, the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm, and / or about 850 nm.

[0019] In some embodiments, the obtained virtual coloring engine includes a generative adversarial network (GAN) (e.g., a 3-channel GAN, a 12-channel GAN, etc.).

[0020] In some embodiments, the staining agent is a primary staining agent. In some embodiments, the staining agent is a special staining agent. In some embodiments, the staining agent includes hematoxylin. In some embodiments, the staining agent includes hematoxylin and eosin.

[0021] A second aspect of this disclosure is a method for generating a virtual stained image of a test unstained biological sample disposed on a substrate, the method comprising: (a) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; (b) supplying the obtained test multispectral transmission image of the test unstained biological sample to a virtual staining engine, wherein the virtual staining engine is trained to generate an image of the unstained biological sample stained with a staining agent, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples, (i) wherein a first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images; (ii) wherein a second training image in each pair of co-registered training images comprises training brightfield image data of the same biological sample stained with a morphological staining agent; and (c) A trained virtual staining engine generates virtual stained images of unstained test biological samples stained with morphological staining agents; wherein each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels. In some embodiments, a bright-field image acquisition device is used to acquire training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire training bright-field image data.

[0022] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from two or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these two or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0023] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from three or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these three or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0024] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from four or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these four or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0025] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from twelve or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these twelve or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0026] In some embodiments, the first training image is derived from two or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the two or more training multispectral transmission image channels to derive the first training image.

[0027] In some embodiments, the first training image is derived from three or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the three or more training multispectral transmission image channels to derive the first training image.

[0028] In some embodiments, the first training image is derived from four or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the four or more training multispectral transmission image channels to derive the first training image.

[0029] In some embodiments, the first training image is derived from twelve or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the twelve or more training multispectral transmission image channels to derive the first training image.

[0030] In some embodiments, (i) at least three test multispectral transmission image channel images are acquired; and (ii) at least three training multispectral transmission image channel images are acquired; wherein each of the at least three test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least three training multispectral transmission image channel images.

[0031] In some embodiments, (i) at least four test multispectral transmission image channels are acquired; (ii) at least four training multispectral transmission image channels are acquired; and wherein each of the at least four test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least four training multispectral transmission image channels. In some embodiments, the test multispectral transmission image is generated by dimensionality reduction of the at least four test multispectral transmission image channels. In some embodiments, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least four training multispectral transmission image channels.

[0032] In some embodiments, (i) at least six test multispectral transmission image channels are acquired; (ii) at least six training multispectral transmission image channels are acquired; and wherein each of the at least six test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least six training multispectral transmission image channels. In some embodiments, the test multispectral transmission image is generated by dimensionality reduction of the at least six test multispectral transmission image channels. In some embodiments, a first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least six training multispectral transmission image channels.

[0033] In some embodiments, (i) at least twelve test multispectral transmission image channels are acquired; (ii) at least twelve training multispectral transmission image channels are acquired; and wherein each of the at least twelve test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least twelve training multispectral transmission image channels. In some embodiments, the test multispectral transmission image is generated by dimensionality reduction of the at least twelve test multispectral transmission image channels. In some embodiments, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least twelve training multispectral transmission image channels.

[0034] In some embodiments, the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm. In some embodiments, the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm, and / or about 850 + / - 10 nm. In some embodiments, the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm, and / or about 850 nm.

[0035] In some embodiments, the obtained virtual coloring engine includes a generative adversarial network.

[0036] In some embodiments, the morphological staining agent is a primary staining agent. In some embodiments, the morphological staining agent is a special staining agent. In some embodiments, the morphological staining agent includes hematoxylin. In some embodiments, the morphological staining agent includes hematoxylin and eosin.

[0037] A third aspect of this disclosure is a system for generating a virtual stained image of a test unstained biological sample disposed on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: (a) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; (b) supplying the obtained multispectral test transmission image of the test unstained biological sample to a virtual staining engine, wherein the virtual staining engine is trained to generate an image of the unstained biological sample stained with a morphological staining agent; and (c) generating a virtual stained image of the test unstained biological sample stained with a morphological staining agent using the trained virtual staining engine.

[0038] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from two or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these two or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0039] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from three or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these three or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0040] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from four or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these four or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0041] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from twelve or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these twelve or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0042] In some embodiments, the virtual staining engine is trained using: (a) one or more training multispectral transmission image channels of unstained training biological samples; and (b) bright-field training image data of the same unstained training biological samples stained with a morphological stain. In some embodiments, each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels. In some embodiments, a bright-field image acquisition device is used to acquire the training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire the training bright-field image data.

[0043] In some embodiments, the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples. In some embodiments, a first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channels; and a second training image in each pair of co-registered training images contains training bright-field image data of the same biological sample stained with a morphological staining agent. In some embodiments, each test multispectral transmission image channel in the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each training multispectral transmission image channel in the one or more training multispectral transmission image channels.

[0044] In some embodiments, the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm. In some embodiments, the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm, and / or about 850 + / - 10 nm. In some embodiments, the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm, and / or about 850 nm.

[0045] A fourth aspect of this disclosure is a method for generating a virtual stained image of a test unstained biological sample disposed on a substrate, the method comprising (a) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; (b) supplying the obtained multispectral test transmission image of the test unstained biological sample to a virtual staining engine, wherein the virtual staining engine is trained to generate an image of the unstained biological sample stained with a morphological staining agent; and (c) generating a virtual stained image of the test unstained biological sample stained with a morphological staining agent using the trained virtual staining engine.

[0046] In some embodiments, the virtual staining engine is trained using: (a) one or more training multispectral transmission image channels of unstained training biological samples; and (b) bright-field training image data of the same unstained training biological samples stained with a morphological stain. In some embodiments, each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels. In some embodiments, a bright-field image acquisition device is used to acquire the training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire the training bright-field image data.

[0047] A fifth aspect of this disclosure is a system for generating two or more virtual staining images of a test unstained biological sample disposed on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: (a) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; (b) supplying the obtained test multispectral transmission image of the test unstained biological sample to two or more different virtual staining engines, wherein each of the two or more different virtual staining engines is trained to generate images of the unstained biological sample stained with different morphological staining agents; and (c) generating two or more virtual staining images of the test unstained biological sample using the trained virtual staining engines, wherein each of the generated two or more virtual staining images is stained with a different staining agent. In some embodiments, one of the virtual staining images is virtually stained with hematoxylin and / or eosin; while another virtual staining image in the virtual staining images is virtually stained with a special staining agent (e.g., Masson's trichrome stain).

[0048] In some embodiments, the two or more virtual staining engines are trained independently using different training datasets, such as each training dataset including images stained with a specific morphological staining agent. In some embodiments, the two or more virtual staining engines are trained independently using: (a) one or more training multispectral transmission image channels of unstained training biological samples; and (b) bright-field training image data of the same unstained training biological samples stained with a morphological staining agent. In some embodiments, each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels used to train each of the two or more virtual staining engines. In some embodiments, a bright-field image acquisition device is used to acquire the training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire the training bright-field image data.

[0049] In some embodiments, the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm. In some embodiments, the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm, and / or about 850 + / - 10 nm. In some embodiments, the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm, and / or about 850 nm.

[0050] In some embodiments, each of the two or more virtual staining engines is trained from multiple pairs of co-registered training images derived from one or more training biological samples. In some embodiments, a first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channels; and a second training image in each pair of co-registered training images contains training bright-field image data of the same biological sample stained with a morphological staining agent. In some embodiments, a bright-field image acquisition device is used to acquire the training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire the training bright-field image data.

[0051] A sixth aspect of this disclosure is a method for generating two or more virtual staining images of a test unstained biological sample disposed on a substrate, the method comprising (a) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; (b) supplying the obtained test multispectral transmission image of the test unstained biological sample to two or more different virtual staining engines, wherein each of the two or more different virtual staining engines is trained to generate an image of the unstained biological sample stained with a different morphological staining agent; and (c) generating two or more virtual staining images of the test unstained biological sample using the trained virtual staining engines, wherein each of the generated two or more virtual staining images is stained with a different morphological staining agent.

[0052] In some embodiments, each of the two or more virtual staining engines is trained from multiple pairs of co-registered training images derived from one or more training biological samples. In some embodiments, a first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channels; and a second training image in each pair of co-registered training images contains training bright-field image data of the same biological sample stained with a morphological staining agent. In some embodiments, a bright-field image acquisition device is used to acquire the training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire the training bright-field image data.

[0053] A seventh aspect of this disclosure is a system for generating virtual stained images of test unstained biological samples disposed on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: (a) obtaining a trained virtual staining engine trained to generate images of unstained biological samples stained with morphological staining agents, wherein the trained virtual staining engine is trained from multiple pairs of co-registered training images, (i) wherein a first training image in each pair of co-registered training images is derived from training multispectral transmission image data of unstained training biological samples acquired at three or more different wavelengths, and (ii) wherein a second training image in each pair of co-registered training images contains training bright-field image data of the same training biological sample stained with morphological staining agents; (b) (c) Obtain a test multispectral transmission image of a test unstained biological sample, wherein the obtained test multispectral transmission image is derived from test multispectral transmission image data of a test unstained biological sample illuminated by at least three different illumination sources; and (d) based on the obtained test multispectral transmission image, generate a virtual staining image of the test unstained biological sample stained with a morphological staining agent using the obtained trained virtual staining engine.

[0054] In some embodiments, a bright-field image acquisition device is used to acquire training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire training bright-field image data.

[0055] In some embodiments, the morphological staining agent is a primary staining agent. In some embodiments, the morphological staining agent is a special staining agent (e.g., Masson's trichrome stain or any of the other exemplary special staining agents described herein). In some embodiments, the morphological staining agent includes hematoxylin. In some embodiments, the morphological staining agent includes hematoxylin and eosin.

[0056] In some embodiments, training multispectral transmission image data of unstained training biological samples are acquired at four or more different wavelengths. In some embodiments, a first training image is generated without reducing the dimensionality of the training multispectral transmission image data at the four or more different wavelengths. In some embodiments, the first training image is generated by reducing the dimensionality of the training multispectral transmission image data at the four or more different wavelengths. In some embodiments, principal component analysis is used to reduce the dimensionality.

[0057] In some embodiments, training multispectral transmission image data of unstained training biological samples are acquired at six or more different wavelengths. In some embodiments, a first training image is generated without reducing the dimensionality of the training multispectral transmission image data acquired at the six or more different wavelengths. In some embodiments, the first training image is generated by reducing the dimensionality of the training multispectral transmission image data acquired at the six or more different wavelengths. In some embodiments, principal component analysis is used to reduce the dimensionality.

[0058] In some embodiments, training multispectral transmission image data of unstained training biological samples are acquired at twelve or more different wavelengths. In some embodiments, a first training image is generated without reducing the dimensionality of the training multispectral transmission image data acquired at the twelve or more different wavelengths. In some embodiments, the first training image is generated by reducing the dimensionality of the training multispectral transmission image data acquired at the twelve or more different wavelengths. In some embodiments, principal component analysis is used to reduce the dimensionality.

[0059] In some embodiments, the at least two wavelengths are selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm. In some embodiments, the at least two wavelengths are selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm, and / or about 850 + / - 10 nm. In some embodiments, the at least two wavelengths are selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm, and / or about 850 nm.

[0060] In some embodiments, a multispectral image acquisition device is used to acquire training multispectral transmission image data and test multispectral image data. In some embodiments, a bright-field image acquisition device is used to acquire training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire training bright-field image data, wherein the multispectral image acquisition device is configured to capture image data at 700 nm + / - 10 mm, approximately 550 nm + / - 10 mm, and approximately 470 nm + / - 10 mm.

[0061] In some embodiments, the obtained trained virtual coloring engine includes a generative adversarial network.

[0062] The eighth aspect of this disclosure is a method for generating a virtual staining image of a test unstained biological sample disposed on a substrate, the method comprising: (a) obtaining a trained virtual staining engine trained to generate an image of an unstained biological sample stained with a morphological staining agent, wherein the trained virtual staining engine is trained from multiple pairs of co-registered training images; (b) wherein a first training image in each pair of the multiple co-registered training images is derived from training multispectral transmission image data of an unstained training biological sample illuminated by at least two different illumination sources; (i) wherein a second training image in each pair of the multiple co-registered training images comprises training brightfield image data of the same training biological sample stained with a morphological staining agent; (ii) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the obtained test multispectral transmission image is derived from the test multispectral transmission image data of the test unstained biological sample illuminated by the at least two different illumination sources; and (c) Based on the obtained test multispectral transmission images, virtual staining images of unstained test biological samples stained with morphological staining agents are generated using the obtained trained virtual staining engine.

[0063] In some embodiments, the morphological staining agent is a primary staining agent. In some embodiments, the morphological staining agent is a special staining agent. In some embodiments, the morphological staining agent includes hematoxylin. In some embodiments, the morphological staining agent includes hematoxylin and eosin.

[0064] In some embodiments, the unstained training biological samples and the unstained test biological samples are illuminated with at least four different light sources. In some embodiments, the unstained training biological samples and the unstained test biological samples are illuminated with at least six different light sources. In some embodiments, the unstained training biological samples and the unstained test biological samples are illuminated with at least twelve different light sources.

[0065] In some embodiments, the at least two illumination sources are selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm. In some embodiments, the at least two illumination sources are selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm, and / or about 850 + / - 10 nm. In some embodiments, the at least two illumination sources are selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm, and / or about 850 nm.

[0066] In some embodiments, a multispectral image acquisition device is used to acquire training multispectral transmission image data and test multispectral image data. In some embodiments, a bright-field image acquisition device is used to acquire training bright-field image data. In some embodiments, a multispectral image acquisition device is used to acquire training bright-field image data, wherein the multispectral image acquisition device is configured to capture image data at 700 nm + / - 10 mm, approximately 550 nm + / - 10 mm, and approximately 470 nm + / - 10 mm. In some embodiments, the resulting trained virtual staining engine includes a generative adversarial network. In some embodiments, each pair of co-registered training images in the plurality of co-registered training images is derived from different training biological samples.

[0067] A ninth aspect of this disclosure is a non-transitory computer-readable medium storing instructions for generating a virtual stained image of a test unstained biological sample disposed on a substrate, the instructions comprising: (a) obtaining a test multispectral transmission image of the test unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; (b) supplying the obtained test multispectral transmission image of the test unstained biological sample to a virtual staining engine, wherein the virtual staining engine is trained to generate images of unstained biological samples stained with morphological staining agents, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples, (i) wherein a first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images; (ii) wherein a second training image in each pair of co-registered training images comprises training bright-field image data of the same biological sample stained with morphological staining agents; and (c) A trained virtual staining engine generates virtual stained images of unstained test biological samples stained with morphological staining agents; wherein each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels.

[0068] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from two or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these two or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0069] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from three or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these three or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0070] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from four or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these four or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0071] In some embodiments, the test multispectral transmission image of the unstained biological sample is derived from twelve or more test multispectral transmission image channels, and compression techniques or dimensionality reduction techniques (e.g., principal component analysis) are not applied to these twelve or more test multispectral transmission image channels to derive the test multispectral transmission image of the unstained biological sample.

[0072] In some embodiments, the first training image is derived from two or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the two or more training multispectral transmission image channels to derive the first training image.

[0073] In some embodiments, the first training image is derived from three or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the three or more training multispectral transmission image channels to derive the first training image.

[0074] In some embodiments, the first training image is derived from four or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the four or more training multispectral transmission image channels to derive the first training image.

[0075] In some embodiments, the first training image is derived from twelve or more training multispectral transmission image channels, and compression techniques or dimensionality reduction techniques are not applied to the twelve or more training multispectral transmission image channels to derive the first training image.

[0076] The tenth aspect of this disclosure is a method for virtual staining of images derived from unstained test biological samples, the method comprising: obtaining test multispectral image data from the unstained test biological sample, wherein the obtained test multispectral image data includes at least four multispectral transmission image channels acquired at different wavelengths; reducing the dimension of the obtained test multispectral image data to generate a multichannel multispectral test transmission image; and using a virtual staining engine to generate a virtual stained image from the multichannel multispectral test transmission image, the virtual staining engine being trained to generate an image of the unstained biological sample stained with a specific morphological staining agent. Attached Figure Description

[0077] For a general understanding of the features of this disclosure, please refer to the accompanying drawings. In the drawings, the same reference numerals are used throughout to identify the same elements.

[0078] Figure 1 compares a virtual stained image of a tissue sample with the same tissue sample that has been chemically stained (such as with a morphological staining agent).

[0079] Figure 2A provides an overview of a method for training a machine learning algorithm to generate morphologically stained images from test multispectral images derived from unstained biological samples, according to an embodiment of the present disclosure.

[0080] Figure 2B provides an overview of a method for generating a virtual stained image of a test unstained biological sample according to an embodiment of the present disclosure.

[0081] Figures 3A to 3C illustrate systems for acquiring image data and generating virtual staining for test biological samples, or for training virtual staining engines.

[0082] Figures 4A and 4B illustrate systems for acquiring image data and generating virtual staining for test biological samples, or for training virtual staining engines.

[0083] Figure 5 provides a block diagram of a multispectral image acquisition apparatus according to an embodiment of the present disclosure.

[0084] Figures 6A, 6B, and 6C illustrate methods for training a virtual coloring engine according to some embodiments of the present disclosure.

[0085] Figure 7 illustrates a method for training a virtual staining engine using different obtained training samples.

[0086] Figures 8A and 8B illustrate methods for generating training image data for training a virtual coloring engine.

[0087] Figure 9 illustrates a method for training a virtual staining engine using different obtained continuous tissue sections.

[0088] Figure 10 illustrates a method for generating multispectral training images according to an embodiment of the present disclosure.

[0089] Figure 11 illustrates a method for generating multispectral training images according to an embodiment of the present disclosure.

[0090] Figure 12A provides an example of a multi-channel multispectral training transmission image.

[0091] Figure 12B provides an example of training bright-field transmission.

[0092] Figure 13A illustrates a method for co-registering a multispectral training transmission image and a training brightfield transmission image according to an embodiment of the present disclosure to provide a pair of co-registered training images.

[0093] Figure 13B illustrates a method for co-registering a generated multi-channel multispectral training transmission image and a training bright-field transmission image according to an embodiment of the present disclosure.

[0094] Figures 14A and 14B show the placement of landmarks in both the generated multi-channel multispectral training transmission image and the generated training bright-field transmission image.

[0095] Figure 15 illustrates a method for virtually staining images of unstained biological samples using a trained virtual staining engine.

[0096] Figure 16 illustrates a method for generating multispectral test images according to an embodiment of the present disclosure.

[0097] Figure 17 shows three images of unstained tonsil tissue acquired at three different wavelengths using a multispectral imaging device. Figure 17 further illustrates the encoding of the unstained tonsil tissue images into RGB images. Finally, Figure 17 compares the virtual stained images generated using a trained virtual staining engine with images of chemically stained tissue samples.

[0098] Figures 18A and 18B provide pseudo-color images obtained from chemically stained H&E breast sections and scanned in a FLASH multispectral scanner (Figure 18A). Channels illuminated at wavelengths of 470 nm, 550 nm, and 635 nm were extracted and transformed to enhance chromaticity and white balance, and then encoded as RGB images to serve as ground truth. The virtual stained image obtained when the pix2pix algorithm was used to transform the spectral image, and the image on the left serving as ground truth, are shown (Figure 18B).

[0099] Figure 19 provides an image of a full slide of breast tissue with virtual H&E staining according to the method of this disclosure.

[0100] Figure 20 provides an image of a breast tissue sample stained with virtual H&E according to the method of this disclosure.

[0101] Figure 21 provides an image of a section of colorectal tissue virtually stained with Masson's trichrome stain according to the method of this disclosure.

[0102] Figure 22 illustrates a method for scanning an unstained glass slide according to the method of this disclosure.

[0103] Figure 23 provides a method for scanning stained slides according to the present disclosure, such as for acquiring images of samples stained with morphological staining agents or special staining agents for training machine learning algorithms.

[0104] Figure 24 illustrates a workflow for co-registering one or more images according to the method of this disclosure. Detailed Implementation

[0105] It should also be understood that, unless expressly indicated otherwise, in any method claimed herein that includes more than one step or action, the order of the steps or actions of the method is not necessarily limited to the order in which the steps or actions of the method are described.

[0106] References to "an embodiment," "embodiment," "illustrative embodiment," etc., in this specification indicate that the described embodiment may include a specific feature, structure, or characteristic; however, each embodiment may or may not necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, it should be noted that, whether explicitly stated or not, those skilled in the art will understand the effects of combining other embodiments to achieve the described feature, structure, or characteristic.

[0107] As used herein, unless the context clearly indicates otherwise, the singular forms “a / an” and “the / that” include plural referents. Similarly, unless the context clearly indicates otherwise, the word “or” is intended to include “and”. The term “including” is defined as inclusive, such as “including A or B” meaning including A, B, or A and B.

[0108] As used herein in the specification and claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” should be interpreted as inclusive, such as including several elements or at least one element in the list of elements, but also including more than one element, and optionally including additional unlisted items. Only terms indicating the opposite, such as “only one of them” or “exactly one of them,” or “consisting of…” as used in the claims, will refer to including several elements or exactly one element in the list of elements. In general, the term “or” as used herein should only be interpreted as indicating an exclusive alternative (e.g., “one or the other, but not two”) when preceded by exclusive terms such as “or,” “one of them,” “only one,” or “exactly one.” “Constitutes substantially of…” as used in the claims should have the ordinary meaning used in the field of patent law.

[0109] The terms “comprising,” “including,” and “having” are used interchangeably and have the same meaning. Similarly, each of these terms is defined in accordance with the common U.S. patent law definition of “comprising” and is therefore understood as an open-ended term meaning “at least the following” and is also understood not to exclude additional features, limitations, aspects, etc. Thus, for example, “an apparatus having components a, b, and c” means that the apparatus includes at least components a, b, and c. Similarly, the phrase “a method involving steps a, b, and c” means that the method includes at least steps a, b, and c. Furthermore, although steps and processes may be outlined herein in a specific order, those skilled in the art will recognize that the order of steps and processes may vary.

[0110] As used herein in the specification and claims, with respect to a list of one or more elements, the phrase "at least one" should be understood as at least one element selected from any one or more elements in the list, but does not necessarily include at least one of each element specifically listed in the list, nor exclude any combination of elements in the list. In addition to the elements specifically identified in the list of elements referred to by the phrase "at least one," this definition also allows for the optional presence of other elements, whether or not they are related to the specifically identified elements. Thus, as a non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B," or equivalently, "at least one of A and / or B") in one embodiment may refer to at least one, optionally including more than one A, and the absence of B (and optionally including elements other than B); in another embodiment, it refers to at least one, optionally including more than one B, and the absence of A (and optionally including elements other than A); in yet another embodiment, it refers to at least one, optionally including more than one A, and at least one, optionally including more than one B (and optionally including other elements); and so on.

[0111] As used herein, the term “about” means + / - 5%. In some embodiments, “about” means + / - 10%. In some embodiments, “substantially” means within about 15%. In some embodiments, “about” means + / - 20%.

[0112] As used herein, the terms “biological sample,” “sample,” or “tissue sample” refer to any sample obtained from any organism, including viruses, that includes biomolecules such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof. Other examples of organisms include mammals (such as humans; mammals such as cats, dogs, horses, cattle, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and tissue biopsies), cell samples (such as cytological smears, such as cervical smears or blood smears, or cell samples obtained through microdissection), or cell fractions, fragments, or organelles (such as those obtained by lysing cells and separating their components by centrifugation or other methods). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspiration, cerumen, breast milk, vaginal secretions, saliva, swabs (such as oral swabs), or any material containing biomolecules derived from the first biological sample. In some embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject (e.g., a homogenized or liquefied sample).

[0113] As used herein, the term "biomarker" or "marker" refers to a measurable indicator of a certain biological state or condition. Biomarkers can be proteins or peptides, such as surface proteins, which can be specifically stained and indicate biological characteristics of cells, such as cell type or physiological state. Immune cell markers are biomarkers that selectively indicate characteristics associated with the immune response in mammals. Biomarkers can be used to determine the extent of a body's response to treatment for a disease or condition or whether a subject is susceptible to a disease or condition. In the case of cancer, a biomarker refers to a biological substance that indicates the presence of cancer in the body. Biomarkers can be molecules secreted by a tumor or a specific response of the body to the presence of cancer. Genetic, epigenetic, proteomic, glycomic, and imaging biomarkers can be used for the diagnosis, prognosis, and epidemiology of cancer. Such biomarkers can be measured in non-invasively collected biological fluids, such as blood or serum. Several gene- and protein-based biomarkers have been used in patient care, including but not limited to AFP (liver cancer), BCR-ABL (chronic myeloid leukemia), BRCA1 / BRCA2 (breast cancer / ovarian cancer), BRAF V600E (melanoma / colorectal cancer), CA-125 (ovarian cancer), CA19.9 (pancreatic cancer), CEA (colorectal cancer), EGFR (non-small cell lung cancer), HER-2 (breast cancer), KIT (gastrointestinal stromal tumor), PSA (prostate-specific antigen), and S100 (melanoma). Biomarkers can be used for diagnosis (to identify early-stage cancer) and / or prognosis (to anticipate cancer invasiveness and / or predict the degree of response to specific treatments and / or the likelihood of cancer recurrence).

[0114] As used herein, "bright field" refers to data (e.g., image data) obtained from a biological sample via microscopy, where the sample is illuminated from below so that light waves pass through the transparent portions of the sample. Different brightness levels are then captured, such as in the form of an image.

[0115] As used herein, the term "cell" refers to a prokaryotic or eukaryotic cell. Cells can be adherent or non-adherent, such as adherent prokaryotic cells, adherent eukaryotic cells, non-adherent prokaryotic cells, or non-adherent eukaryotic cells. Cells can be yeast cells, bacterial cells, algal cells, fungal cells, or any combination thereof. Cells can be mammalian cells. Cells can be primary cells obtained from a subject. Cells can be cell lines or cells that proliferate indefinitely. Cells can be obtained from mammals, such as humans or rodents. Cells can be cancer or tumor cells. Cells can be epithelial cells. Cells can be red blood cells or white blood cells. Cells can be immune cells, such as T cells, B cells, natural killer (NK) cells, macrophages, dendritic cells, or other cells. Cells can be neuronal cells, glial cells, astrocytes, neuronal supporting cells, Schwann cells, or other cells. Cells can be endothelial cells. Cells can be fibroblasts or keratinocytes. Cells can be pericytes, hepatocytes, stem cells, progenitor cells, or other cells. The cells can be circulating cancer or tumor cells or metastatic cells. The cells can be label-specific cells, such as CD8+ T cells or CD4+ T cells. The cells can be neurons. Neurons can be central neurons, peripheral neurons, sensory neurons, interneurons, intraneurons, motor neurons, multipolar neurons, bipolar neurons, or pseudounipolar neurons. The cells can be neuronal supporting cells, such as Schwann cells. The cells can be one of the cells in the blood-brain barrier system. The cells can be cell lines, such as neuronal cell lines. The cells can be primary cells, such as cells obtained from the subject's brain. The cells can be a population of cells isolated from the subject, such as tissue biopsies, cytological samples, blood samples, fine-needle aspiration (FNA) samples, or any combination thereof. Cells can be obtained from bodily fluids such as urine, breast milk, sweat, lymph, blood, sputum, amniotic fluid, aqueous humor, vitreous humor, bile, cerebrospinal fluid, chyle, chyme, exudate, endolymph, perilymph, gastric acid, mucus, pericardial fluid, peritoneal fluid, pleural fluid, pus, rheumatism, saliva, sebum, serous fluid, sebaceous gland secretions, sputum, tears, vomit, or other bodily fluids. Cells can include cancer cells, non-cancer cells, tumor cells, non-tumor cells, healthy cells, or any combination thereof.

[0116] As used herein, the term "cytological sample" refers to a cell sample in which the cells have been partially or completely deaggregated, such that the sample no longer reflects the spatial relationships of the cells when they were present in the subject from whom the cell sample was obtained. Examples of cytological samples include tissue scrapings (such as cervical scrapings), fine-needle aspirations, and samples obtained by irrigating the subject.

[0117] As used herein, the term “fixation” refers to the process of preserving the molecular and / or morphological details of a cellular sample. There are generally three fixation processes: (1) heat fixation, (2) perfusion, and (3) immersion. In heat fixation, the sample is exposed to a heat source for a sufficient period of time to heat-kill the sample and adhere it to a slide. Perfusion involves distributing a chemical fixative throughout an organ or the entire organism using a vascular system. Immersion involves immersing the sample in a volume of chemical fixative and allowing the fixative to diffuse throughout the sample. Chemical fixation involves the diffusion or perfusion of chemicals throughout a cellular sample, where the fixative induces a reaction that preserves the structure (chemically and structurally) as closely as possible to that of a living cellular sample. Chemical fixatives can be broadly classified into two categories based on their mode of action: cross-linked fixatives and non-cross-linked fixatives. Cross-linked fixatives—often aldehydes—create covalent chemical bonds between endogenous biomolecules present in the tissue sample, such as proteins and nucleic acids. Formaldehyde is the most commonly used cross-linked fixative in histology. Formaldehyde can be used for fixation at various concentrations, but it is primarily used in 10% neutral buffered formalin (NBF), which is approximately 3.7% formaldehyde in a phosphate-buffered saline solution. Paraformaldehyde is a polymeric form of formaldehyde that depolymerizes upon heating to provide formalin. Glutaraldehyde works in a similar manner to formaldehyde, but is a larger molecule with a slower transmembrane diffusion rate. Glutaraldehyde fixation provides a more rigid or tightly bound fixative, causing rapid and irreversible changes. It fixes quickly and well at 4°C, providing good overall cytoplasmic and nuclear detail, but is not ideal for immunohistochemical staining. Some fixation protocols use a combination of formaldehyde and glutaraldehyde. Glyoxal and acrolein are less commonly used aldehydes. Denaturing fixatives—usually alcohols or acetone—work by displacing water in the cell sample (which destabilizes hydrophobic and hydrogen bonds within proteins). This also causes water-soluble proteins to become insoluble in water and precipitate, which is largely irreversible.

[0118] As used herein, the term "immunohistochemistry" refers to a method for determining the presence or distribution of an antigen in a sample by detecting the interaction between the antigen and a specific binding agent, such as an antibody. The sample is contacted with the antibody under conditions that allow antibody-antigen binding. Antibody-antigen binding can be detected using a detectable label conjugated to the antibody (direct detection) or a detectable label conjugated to a secondary antibody that specifically binds to a primary antibody (indirect detection). In some cases, indirect detection may include tertiary or higher-level antibodies to further enhance the detectability of the antigen. Examples of detectable labels include enzymes, fluorophores, and haptens; in the case of enzymes, they may be used in conjunction with chromogenic or fluorescent substrates.

[0119] As used in this article, the term “machine learning” refers to a class of learning in which machines (e.g., computer programs) can learn on their own without being programmed.

[0120] As used herein, the term "slide" refers to any substrate of suitable size on which a biological sample can be placed for analysis (e.g., a substrate made wholly or partially of glass, quartz, plastic, silicon, etc.), and more particularly to "microscope slides" such as standard 3 x 1-inch microscope slides or standard 75 mm x 25 mm microscope slides. Examples of biological samples that can be placed on a slide include, but are not limited to, cytological smears, thin tissue sections (such as those from biopsies), and arrays of biological samples (e.g., tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof). Thus, in one embodiment, tissue sections, DNA samples, RNA samples, and / or proteins are placed at specific locations on the slide. In some embodiments, the term "slide" may refer to SELDI and MALDI chips, as well as silicon wafers.

[0121] As used herein, the term "substantially" means a qualitative condition that exhibits all or nearly all of the range or degree of the target feature or characteristic. In some embodiments, "substantially" means within about 5%. In some embodiments, "substantially" means within about 10%. In some embodiments, "substantially" means within about 15%. In some embodiments, "substantially" means within about 20%.

[0122] As used herein, the term "virtual stained image" refers to an image of an unstained biological sample that simulates a chemically stained biological sample. In some embodiments, there is no discernible difference in diagnostic quality between the virtual stained image of the unstained biological sample and the corresponding image of the chemically stained biological sample, at least not to the extent that any difference would significantly alter the diagnostic outcome.

[0123] Overview

[0124] This disclosure provides systems and methods for generating virtual stained images of unstained biological samples from acquired images, wherein the virtual stained image shows the appearance of the unstained biological sample as if it had been chemically stained (e.g., stained with a morphological staining agent, such as a primary staining agent or a special staining agent). In some embodiments, a virtual stained image trained according to the methods described herein outputs a virtually generated stained image in response to a provided input image of the unstained biological sample, wherein the virtual stained image appears substantially equivalent to a corresponding bright-field image of the same biological sample that has been chemically stained (e.g., stained with a primary staining agent or a special staining agent) to a skilled observer (e.g., a trained histopathologist).

[0125] Those skilled in the art will understand that one or more differently trained virtual staining engines can be used to virtually stain a single input image of an unstained biological sample to provide one or more different virtual staining output images. For example, a single input image of an unstained biological sample can be virtually stained using: (i) a virtual staining engine trained for H&E morphological staining to provide a first virtual staining output image based on the unstained input image, wherein the first virtual staining output image is substantially equivalent to a corresponding bright-field image (at least for diagnostic purposes) of the same biological sample stained with H&E; and (ii) a virtual staining engine trained for resorcinol fuchsin morphological staining to provide a second virtual staining output image based on the unstained input image, wherein the second virtual staining output image is substantially equivalent to a corresponding bright-field image (at least for diagnostic purposes) of the same biological sample stained with resorcinol fuchsin.

[0126] As further described herein, the applicant has demonstrated that trained pathologists can identify histopathological features in both virtually generated images of unstained biological samples and images of chemically stained biological samples with high consistency (see Example 1 herein). For example, Figures 1 and 18 compare virtually stained images of unstained biological samples with those of the same biological samples that have been chemically stained, wherein both virtually stained images of unstained biological samples and corresponding images of chemically stained biological samples can be used equally for diagnostic purposes. In some embodiments, there is no discernible difference in diagnostic quality between virtually stained images of unstained biological samples and corresponding images of chemically stained biological samples, at least not to the extent that any difference would significantly alter the diagnostic outcome.

[0127] In view of the foregoing, this document describes systems and methods for generating virtual stained images (e.g., virtual morphological staining) from images (e.g., multispectral transmission images) of unstained biological samples. In some embodiments, machine learning algorithms (such as deep learning neural networks) are trained to generate virtual staining for specific morphological staining agents (such as primary staining agents (e.g., H&E) or special staining agents (Masson trichrome staining)). In this regard, this disclosure also provides systems and methods for training virtual staining engines. In some embodiments, this disclosure provides a system comprising a plurality of different trained virtual staining engines that can be used to virtually stain one or more unstained biological samples.

[0128] Figure 2A provides an overview of a method for training a machine learning algorithm to generate morphologically stained images from test multispectral images derived from unstained biological samples. As an initial step, training transmission image data, including training multispectral image data and training bright-field image data, is acquired from one or more training biological samples. In some embodiments, training multispectral transmission image data is acquired from unstained training biological samples (step 100), such as using a multispectral image acquisition device 12A. In other embodiments, training bright-field transmission image data is acquired from stained training biological samples, such as using a bright-field imaging acquisition device 12B or a multispectral imaging device 12A using RGB colors (e.g., acquiring image data from stained training biological samples using multispectral image acquisition device 12A at approximately wavelengths of 700 nm, 550 nm, and 470 nm) (step 100). The acquired training multispectral transmission image data and the acquired training bright-field transmission image data are then fed to a machine learning algorithm (e.g., a GAN algorithm) to train a model to predict or generate virtual stained images (step 102).

[0129] A method for generating virtual stained images using trained machine learning algorithms is also described (see, for example, Figure 2B The virtual staining process involves using virtual staining images with the same staining agent generated by a trained machine learning algorithm. Typically, a trained virtual staining engine specific to a particular morphological staining agent is obtained, whereby the obtained trained virtual staining engine is trained to generate images of unstained biological samples stained with the specific morphological staining agent it is trained to target (step 110). Test multispectral transmission image data is then obtained (step 111) and fed to the obtained trained virtual staining engine (step 112). Typically, the test multispectral transmission image data is acquired at approximately the same wavelengths used when training the virtual staining engine. For example, if training multispectral transmission images at wavelengths of 350 nm, 450 nm, and 500 nm are used to train a virtual staining engine specific to a particular morphological staining agent, then the test multispectral transmission images should be acquired at approximately 350 nm, approximately 450 nm, and approximately 550 nm. Then, the trained virtual staining engine generates a virtual staining image of the unstained test biological sample stained with a morphological staining agent, wherein the virtual staining image shows the appearance of the unstained test biological sample as if the unstained test biological sample were stained with a specific morphological staining agent generated by the trained virtual staining engine (step 113).

[0130] This disclosure also discloses a system suitable for using trained machine learning algorithms to acquire image data, process the acquired image data, train the machine learning algorithms, and generate virtual stained slides.

[0131] system

[0132] Figures 3A to 3C, 4A, and 4B illustrate a system 200 for acquiring image data and generating virtual staining for testing biological samples or for training a virtual staining engine. The system may include an image acquisition device 12 and a computer 14, wherein the image acquisition device 12 and the computer may be communicatively coupled together (e.g., directly or indirectly coupled via a network 20).

[0133] In some embodiments, the image acquisition device 12 is a multispectral image acquisition device 12A (see, for example, Figure 3B (See Figure 4A). In other embodiments, the image acquisition device is a bright-field image acquisition device. In other embodiments, system 200 includes both a multispectral image acquisition device 12A and a bright-field image acquisition device 12B (see, for example, Figures 3C and 4B). In some embodiments, images captured from image acquisition device 12 may be stored in binary form, such as locally or on a server. The captured digital image may also be divided into a pixel matrix. The pixels may include one or more bits of digital value defined by a bit depth.

[0134] Computer system 14 may include a desktop computer, laptop computer, tablet computer, etc., a digital electronic circuit system, firmware, hardware, memory 201, computer storage medium (240), computer program or instruction set (e.g., wherein the program is stored in memory or storage medium), one or more processors (209) (including programmable processors), and any other hardware, software, or firmware modules or combinations thereof (such as those further described herein). For example, system 14 shown in Figures 1A through 1C may include a computer having a display device 16 and a housing 18. The computer system may store acquired image data locally, such as in memory, on a server, or on another network-connected device.

[0135] Those skilled in the art will understand that other computer devices or systems can be used, and the computer systems described herein can be communicatively coupled to additional components, such as microscopes (in addition to the multispectral image acquisition device 12A or brightfield image acquisition device 12B described herein), automated slide preparation equipment, sample grinding / dissection devices, etc.

[0136] Multispectral image acquisition device / acquisition of multispectral image data

[0137] In some embodiments, the image acquisition device 12 is a multispectral image acquisition device 12A for acquiring transmission image data of biological samples at one or more wavelengths. Typically, the multispectral image acquisition device 12A is adapted to acquire multispectral transmission image data, such as multispectral transmission image channel images of biological samples disposed on a substrate (such as a microscope slide). As described herein, the multispectral image acquisition device 12A is adapted to illuminate a biological sample at a specific wavelength with an illumination source and acquire transmission image data of the biological sample illuminated at the specific wavelength (referred to herein as "acquiring image data at a specific wavelength").

[0138] A non-limiting example of a multispectral image acquisition device is disclosed in U.S. Patent No. 11,070,750, the disclosure of which is hereby incorporated herein by reference in its entirety. A block diagram of a suitable multispectral image acquisition device 12A is shown in Figure 5. In some embodiments, the multispectral image acquisition device 12A includes a CMOS sensor or a CCD sensor, such as a CMOS sensor or CCD sensor sensitive to all wavelengths of the light source utilized. In some embodiments, the sensor is focused to the intermediate wavelength used, or independently to each wavelength, or dynamically focused to each wavelength by moving the optics of a camera or moving the camera in a direction perpendicular to the sample. In some embodiments, a conventional white light source and color filter system may be used in the multispectral image acquisition device 12A. For example, the illuminator may include a white light source and a color filter to produce a set of color monochrome images. The colors of the monochrome images may be redefined and combined to produce an enhanced digital image. In some embodiments, an LED light source may be used in the detection step to generate a narrower illumination beam.

[0139] In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least two different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated with each of at least two different wavelengths. In some embodiments, the two or more wavelengths may be broadband wavelengths, such as wavelengths up to about 300 nanometers each, such as those used to capture large regions of spectral transmission. In some embodiments, two or more channels may be combined, for example, by averaging, to generate a single transmission channel representing a region of extended spectrum. In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least four different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated with at least four different wavelengths. In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample illuminated with at least six different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated with each of at least six different wavelengths. In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least eight different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated with each of at least eight different wavelengths.

[0140] In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least ten different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample at each of at least ten different wavelengths. In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least twelve different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated at each of at least twelve different wavelengths. In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least sixteen different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated at each of at least sixteen different wavelengths. In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least twenty different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated at each of at least twenty different wavelengths. In some embodiments, the multispectral image acquisition device 12A is configured to illuminate a biological sample with at least twenty-four different illumination sources and acquire multispectral transmission image data (e.g., multispectral transmission image channel images) of the biological sample illuminated with each of the at least twenty-four different wavelengths.

[0141] In some embodiments, the multispectral image acquisition device 12A includes one or more image capture devices; and one or more energy emitters, such as light sources, infrared light sources, ultraviolet light sources, etc., to illuminate biological samples at specific wavelengths. For example, the energy emitter may include, but is not limited to, one or more LEDs (e.g., edge-emitting LEDs, surface-emitting LEDs, superluminescent LEDs, etc.), laser diodes, electroluminescent light sources, incandescent light sources, cold cathode fluorescent light sources, organic polymer light sources, lamps, inorganic light sources, or other suitable light sources. In some embodiments, the light source may be a light-emitting diode (LED), which can be pulsed on and off to correspond to imaging frames, such that consecutive frames are recorded with different LED illumination.

[0142] In some embodiments, the energy emitter of the multispectral image acquisition device 12A is configured to generate energy emissions having different average wavelengths from each other. In some embodiments, the total number of different energy emissions (i.e., energy emissions with different average wavelengths) that can be generated by the multispectral image acquisition device 12A ranges, for example, from 3 to about 50, such as from 3 to 40, such as from 3 to 30, such as from 3 to 20, such as from 3 to 16, such as from 3 to 12. In some embodiments, the energy emitter may include, but is not limited to, light sources with two or more different average wavelengths, three or more different average wavelengths, four or more different average wavelengths, five or more different average wavelengths, six or more different average wavelengths, seven or more different average wavelengths, eight or more different average wavelengths, nine or more different average wavelengths, ten or more different average wavelengths, eleven or more different average wavelengths, twelve or more different average wavelengths, fifteen or more different average wavelengths, twenty or more different average wavelengths, etc.

[0143] For example, the multispectral image acquisition device 12A may include two or more different LEDs, each emitting a different average wavelength. For instance, the multispectral image acquisition device 12A, or an energy emitter included therein, may be configured to generate energy emission with average wavelengths of approximately 365 nm, approximately 400 nm, approximately 435 nm, approximately 470 nm, approximately 500 nm, approximately 550 nm, approximately 580 nm, approximately 635 nm, approximately 660 nm, approximately 690 nm, approximately 780 nm, and / or approximately 850 nm.

[0144] In some embodiments, the energy emitter of the multispectral image acquisition device 12A may be a blue LED with maximum intensity at wavelengths in the blue region of the spectrum. For example, the blue LED may have a peak wavelength and / or average wavelength in the range of about 430 nanometers to about 490 nanometers (nm). In some embodiments, the energy emitter may be a green LED with maximum intensity at wavelengths in the green region of the spectrum. For example, the green LED may have a peak wavelength and / or average wavelength in the range of about 490 nm to 560 nm. In some embodiments, the energy emitter may be an amber LED with maximum intensity at wavelengths in the amber region of the spectrum. For example, amber light may have a peak wavelength and / or average wavelength in the range of about 570 nm to 610 nm. In some embodiments, the energy emitter may be a red LED with maximum intensity at wavelengths in the red region of the spectrum. For example, red light may have a peak wavelength and / or average wavelength in the range of about 620 nm to 800 nm.

[0145] In some embodiments, two or more of the LED light sources can be combined to provide processing flexibility. Different light source arrangements can be selected to achieve the desired illumination field. For example, multiple LEDs of specific wavelengths can be combined so that the acquired multispectral image is similar to a multispectral image of an RGB brightfield image (allowing the use of such a configured multispectral image acquisition device instead of a brightfield image acquisition device). In some embodiments, the LED light source can be part of or form part of a light-emitting panel. In some embodiments, the number, color, and position of the LEDs can be selected to achieve the desired illumination.

[0146] In other embodiments, the multispectral image acquisition device 12A may include one or more lasers, halogen light sources, incandescent light sources, and / or other devices capable of emitting light. In some embodiments, each source may include a light emitter that outputs white light (e.g., a halogen lamp, incandescent light source, etc.) and a color filter that transmits certain wavelengths or bands of white light.

[0147] Bright-field image acquisition device / acquisition of bright-field image data

[0148] In some embodiments, the image acquisition device 12 is a bright-field image acquisition device 12B for acquiring bright-field transmission image data of biological samples. The bright-field image acquisition device 12B may include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), optics (e.g., one or more lenses, a sensor focusing lens group, a microscope objective, etc.), an imaging sensor (e.g., a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.), and photographic film. In some embodiments, the bright-field image acquisition device includes multiple lenses that can cooperate to achieve instant focusing. An image sensor, such as a CCD sensor, can capture digital images of the sample. In some embodiments, the digitized tissue data can be generated, for example, by an image scanning system (such as the Ventana DP 200® slide scanner from Ventana Medical Systems, Inc. (Tucson, Arizona) or other suitable imaging devices). Such scanners can be used to acquire images of test images and / or training images. In some embodiments, the scanner is used to acquire images of unstained samples (see, for example, Figure 22). In other embodiments, the scanner is used for acquisition. Additional imaging devices and systems are further described herein.

[0149] Those skilled in the art will understand that digital color images acquired by a bright-field image acquisition device can typically be composed of basic color pixels. Each color pixel can be encoded in three digital components, each containing the same number of bits, and each component corresponds to a primary color, typically red, green, or blue, also referred to as "RGB" components. Those skilled in the art will understand that the main difference between the bright-field image acquisition device 12B and the multispectral image acquisition device 12A is that the multispectral image acquisition device 12A uses a single white light source and the sensor has a color filter (i.e., a Bayern filter) to produce a three-channel (i.e., RGB) color image; while the bright-field image acquisition device 12B uses multiple colors in a sequential cycle and acquires images in a monochrome camera.

[0150] In some embodiments, the multispectral image acquisition device 12A can also be used to acquire “bright-field transmission image data” of biological samples using the multispectral imaging device 12A employing RGB colors, as the term is used herein (e.g., by acquiring image data at approximately wavelengths of 700 nm, 550 nm, and 470 nm). Therefore, a “bright-field transmission image” or “training bright-field transmission image” can include transmission image data acquired from the multispectral image acquisition device 12A or the bright-field image acquisition device 12B.

[0151] Module

[0152] Figures 4A and 4B provide an overview of the system 200 of this disclosure and the various modules utilized within the system. In some embodiments, the system 200 employs a computer device or computer-implemented method having one or more processors 209 and one or more memories 201, the one or more memories 201 storing non-transitory computer-readable instructions for execution by the one or more processors to cause the one or more processors to execute specific instructions as described herein.

[0153] Image acquisition module

[0154] In some embodiments, the image acquisition module 202 commands the multispectral imaging device 12A and / or the bright-field imaging device 12B to acquire multispectral and / or bright-field transmission image data of a biological sample (or a portion thereof) disposed on a substrate. In some embodiments, the image acquisition module 202 acquires training image data from one or more training biological samples to train the virtual staining engine 210. In other embodiments, the image acquisition module 202 acquires test image data from one or more test biological samples, enabling the use of the trained virtual staining engine 210 to generate virtual stains. The acquired test and / or training transmission image data of the test and / or training biological samples may be stored, respectively, in one or more memories 201 or one or more storage modules 240 communicatively coupled to the system 200 for downstream processing.

[0155] When acquiring image data from a biological sample using a multispectral image acquisition device 12A, the image acquisition module 202 can instruct the multispectral image acquisition device 12A to acquire one or more transmission image channel images of the biological sample, wherein each transmission image channel image is acquired at a different wavelength (i.e., the multispectral image acquisition device 12A illuminates the biological sample with an illumination source having a specific wavelength, and then the transmission image data of the biological sample is acquired at that specific wavelength). For example, the image acquisition module 202 can instruct the multispectral image acquisition device 12A to acquire transmission image data from the biological sample at 12 different wavelengths, thereby generating 12 multispectral transmission image channel images (collectively referred to as the acquired multispectral transmission image data). According to this example, the plurality of acquired multispectral transmission image channel images (i.e., the acquired multispectral transmission image data) are then stored in one or more memories 201 or one or more storage modules 240 for downstream processing.

[0156] In some embodiments, the image acquisition module 202 may command the image acquisition device 12 to acquire transmission image data for the entire biological sample. For example, transmission image data may be acquired for the entire biological sample set on a substrate (e.g., a microscope slide). In other embodiments, the image acquisition module 202 may command the image acquisition device 12 to acquire transmission image data from a portion of the biological sample. This is useful when only a specific target region of the biological sample is relevant to the analysis. For example, certain target regions may include a specific type of tissue or a relatively high number of specific types of cells compared to another target region. For instance, target regions may be selected in the biological sample that include target tissue but exclude non-target tissue (e.g., tumor tissue versus non-tumor tissue). In these embodiments, the image acquisition module 202 may be programmed to acquire transmission image data from one or more predefined portions of the sample; or one or more transmission images may be acquired by random sampling or by sampling at regular intervals across a grid covering the entire sample.

[0157] Image processing module

[0158] In some embodiments, system 200 further includes an image processing module 212 adapted to process acquired image data. In some embodiments, the image processing module is capable of converting or otherwise transforming acquired transmission image data, including multispectral transmission image data and / or bright-field transmission image data. For example, image processing module 212 can combine two or more image channels into a multichannel image (e.g., without using dimensionality reduction or compression techniques). As another example, image processing module 212 can reduce multispectral image data acquired at four or more different wavelengths to a multichannel RGB image, such as by using dimensionality reduction methods. In other embodiments, image processing module 212 can co-register one image with another. For example, image processing module 212 can register an image derived from acquired multispectral transmission image data with an acquired bright-field transmission image to provide a pair of co-registered images for training the virtual staining engine 210. In some embodiments, the image processing module 212 can convert multispectral image data acquired from stained biological samples using a multispectral image acquisition device into RGB images, and can use them as brightfield images, as described herein.

[0159] In some embodiments, the image processing module 212 may be further configured to preprocess image data, identify regions of the image corresponding to a substrate (e.g., a microscope slide) on which the sample is placed, identify regions of different tissue types (e.g., connective tissue), or interpret one or more annotations. In some embodiments, the image processing module 212 may further include one or more sub-modules, such as a tissue classification module, a glass identification module, etc. These one or more sub-modules may implement support vector machines and / or neural networks. Examples of overlay generation modules, tissue classification modules, and glass / slide identification modules are described in U.S. Publications 2020 / 0105413, 2021 / 0027462, 2021 / 0216746, 2021 / 0285056 and U.S. Patents 11,010,892 and 10,628,658, the disclosures of which are hereby incorporated herein by reference in their entirety.

[0160] Training module / virtual coloring engine

[0161] In some embodiments, system 200 further includes a training module 211 adapted to receive multiple pairs of co-registered training images and to use the received multiple pairs of co-registered training images to train a virtual staining engine 210. In some embodiments, the co-registered training images are used to train one or more machine learning algorithms, such as deep neural networks. In some embodiments, the deep neural network is based on an implicit generative model, such as a generative adversarial network (“GAN”). In the deep neural network 10 trained by the GAN, two models are used for training. A generative model that can capture the data distribution is used, while a second model estimates the probability that a sample comes from the training data rather than from the generative model. Details about GANs can be found in Goodfellow et al., Generative Adversarial Nets., Advances in Neural Information Processing Systems, 27, pp. 2672-2680 (2014), which is incorporated herein by reference. Network training of the deep neural network 10 (e.g., GAN) can be performed on the same or different computing devices. For example, in one embodiment, a personal computer can be used to train the GAN, although such training may require a considerable amount of time. To accelerate the training process, one or more dedicated GPUs can be used for training.

[0162] In some embodiments, system 200 further includes a virtual staining engine 210. Typically, the virtual staining engine 210 is trained to generate images of biological samples stained with morphological staining agents from unstained images of biological samples. In some embodiments, once the deep neural network has been trained, the deep neural network can be used or executed on different computing devices, which may include computing devices with fewer computing resources for the training process (although GPUs can also be integrated into the execution of the trained deep neural network).

[0163] This document discloses methods for training virtual staining engines and using such trained virtual staining engines to generate virtual stained slides. In some embodiments, system 200 includes a plurality of virtual staining engines 210, wherein each virtual staining engine is trained to generate different virtual stains, wherein each different virtual stain corresponds to a different morphological staining agent.

[0164] For example, a GAN model is used to train a trained deep neural network10.

[0165] Those skilled in the art will also understand that one or more additional modules may be incorporated into the workflow or into system 200. In some embodiments, one or more automated algorithms may be run, enabling the detection, classification, and / or scoring of cells (see, for example, U.S. Patent Publication No. 2017 / 0372117, the disclosure of which is hereby incorporated herein by reference in its entirety).

[0166] method

[0167] This disclosure provides methods for training a virtual staining engine 210, such as training the virtual staining engine to generate virtual morphological staining images derived from acquired images of unstained biological samples. Those skilled in the art will understand that multiple different machine learning algorithms can be trained to generate different virtual stains (e.g., to generate H&E virtual stain, to generate basic fuchsin virtual stain, to generate Masson's trichrome stain, etc.). This disclosure also provides methods for using the trained virtual staining engine 210 to generate virtual staining images of test unstained biological samples from images of test unstained biological samples, wherein the virtual staining images exhibit the appearance of the test unstained biological samples as if the test unstained biological samples had been stained with morphological stains. Each of these aspects will be described herein.

[0168] Training the virtual coloring engine

[0169] Figure 6A provides an overview of training the virtual staining engine 210. As a first step, one or more unstained training biological samples are obtained (step 610). Training multispectral transmission image data is then acquired from each of these one or more unstained training biological samples (step 611). In some embodiments, at least two training multispectral transmission image channel images are acquired from each of the unstained training biological samples at at least two different wavelengths (e.g., at different wavelengths). In some embodiments, a multichannel multispectral training transmission image is then generated based on the acquired training multispectral transmission image data for each of the unstained training biological samples (step 612). For example, the multichannel multispectral training transmission image may include three different wavelengths. Alternatively, for example, the multichannel multispectral training transmission image may include twelve different wavelengths. For example, and in some embodiments, the twelve different multispectral transmission image channel images (each acquired at a different wavelength) can be reduced to a multichannel multispectral training transmission image using a dimensionality reduction method (such as principal component analysis). As a further example, and in other embodiments, the twelve different multispectral transmission image channels (each acquired at a wavelength) are not compressed, such as without using any dimensionality reduction techniques (e.g., PCA) for compression.

[0170] The unstained training biological sample is then stained, such as with morphological staining, such as with a primary staining agent (e.g., H&E) or with a special staining agent, to provide a stained training biological sample (e.g., basic fuchsin staining, Masson's trichrome staining, etc.) (step 613). Non-limiting examples of suitable special staining agents are further described herein. Bright-field training transmission images of the stained training biological sample are then acquired (step 614). In some embodiments, bright-field training transmission images of the stained training biological sample are acquired using bright-field image acquisition device 12B. In other embodiments, bright-field training transmission images of the stained training biological sample are acquired using a multispectral image acquisition device 12A using RGB channels as further described herein.

[0171] Then, the multispectral training image and the brightfield training image are co-registered to provide at least one pair of co-registered training images (step 615). The first training image in the pair of co-registered training images is a multispectral transmission training image (as described above, derived from training multispectral transmission image data of unstained training biological samples). The second member of the pair of co-registered training images is a brightfield transmission training image (as described above, derived from training brightfield transmission image data of the same training biological samples used to generate multispectral transmission image data but stained with morphological staining agents). The training module 211 uses these multiple pairs of co-registered training images to train the virtual staining engine 210.

[0172] For example, twelve different image channels of a first training tissue sample can be acquired at twelve different wavelengths, and those training multispectral image channels can be used to generate a pair of co-registered training images. Twelve different image channels are acquired while the first training tissue sample is unstained. Then, the same first training tissue sample is morphologically stained according to methods known in the art. After morphological staining, bright-field transmission training images can be acquired from the stained first training tissue sample.

[0173] In some embodiments, the multispectral training image and the bright-field training image are optionally segmented to provide multiple segmented co-registered training images (e.g., segmented into 64 x 64 pixel image patches; 128 x 128 pixel image patches; 256 x 256 pixel image patches; 512 x 512 pixel image patches, etc.). The at least one pair of co-registered training images is then provided to a machine learning algorithm (such as a GAN algorithm) for training (step 616).

[0174] Multiple different virtual staining engines 210 can be trained, each trained to generate different virtual morphological stains for unstained biological samples (e.g., different virtual staining engines can be trained to generate virtual H&E stains or virtual special stains, such as virtual Masson trichrome stains). For example, virtual staining engine 210 can be trained using training biological samples that have already been stained with H&E. In this specific example, virtual staining engine 210 is trained to generate virtual staining images of test unstained biological samples, where the virtual staining images show the appearance of the test unstained biological samples as if they had been stained with H&E. As another example, virtual staining engine 210 can be trained using training biological samples that have been stained with the special stain, basic fuchsin. According to this second example, virtual staining engine 210 is trained to generate virtual staining images of test unstained biological samples, where the virtual staining images show the appearance of the test unstained biological samples as if they had been stained with basic fuchsin. For example, the virtual staining engine 210 can be trained using training biological samples that have been stained with Masson's trichrome stain. According to this third example, the virtual staining engine 210 is trained to generate virtual staining images of test unstained biological samples, where the virtual staining images show the appearance of the test unstained biological samples as if they had been stained with Masson's trichrome stain (see, for example, Figure 21). Furthermore, according to these examples, each differently trained virtual staining engine can then be used to generate one or more virtual staining output images based on a single acquired multispectral transmission image of an unstained biological sample.

[0175] An exemplary method for training a virtual staining engine is illustrated in Figure 6B. Specifically, Figure 6B shows a general workflow of a non-limiting training process according to this disclosure. In some embodiments, tissue samples are processed (e.g., such as biopsy samples, excised samples, FFPE samples, etc., obtained through one or more pre-analytical steps) (step 650), and sections are cut (step 651), such as in a microtome at a thickness of about 3 µm to about 5 µm. In some embodiments, the slides are then baked for 5 minutes and then dewaxed using heat to melt the wax and rinsed with EZ formulation to remove paraffin (see the paraffin removal procedure in HE600) (step 652). In some embodiments, after the slides have been dewaxed, they are placed on a multispectral scanner platform, such as a multispectral scanner described herein. After the target area is selected / detected by the user or by any of the automated devices, the sample is illuminated with a first wavelength (of multiple wavelengths) at a predefined illumination power (controlled by pulses with modulation circuitry to reduce / increase the duty cycle on the LED light source) (step 653). In some embodiments, the scanner objective is focused by moving the sample or objective, and a first field of view (FOV) is digitized with a camera having a predefined exposure time and gain. In some embodiments, the sample is illuminated with a second wavelength (step 653). In some embodiments, a closed-loop focusing algorithm is used to refocus the objective to the second wavelength, iterating until the correct focus is found (e.g., wavelength-controlled autofocus). In other embodiments, the focus of the center wavelength is calibrated only by autofocus, and the Z distances of other wavelengths are calculated (e.g., by using a predefined offset for each wavelength). In other embodiments, only one center wavelength is focused (using a feedback control algorithm), and all other wavelengths are digitized without moving the Z distance within the same FOV. In some embodiments, the process is repeated until all FOVs are digitized with all different wavelengths (step 653). Next, the platform is moved to different locations. In some embodiments, the next position is near the previous position, with some overlapping areas. In other embodiments, the next position is at a different location to collect non-neighboring areas, thereby accelerating digitization and increasing the diversity of the dataset. In other embodiments, each FOV is stored independently in memory. In other embodiments, FOVs are stitched together to form a full-slide hypercube.

[0176] In some embodiments, once all AOIs have been completed by multispectral scanning (step 653), the next step is to stain the sample according to a typical protocol using the desired staining method (i.e., H&E, with a special staining agent, such as Masson's tricolor stain or any other staining agent described herein); and then cover the slide with a coverslip (step 654). In some embodiments, once the slide has been stained and covered with a coverslip, the slide is introduced into the platform of the same multispectral scanner. In some embodiments, the imaging process described above is repeated, but only in a subset of wavelengths, such as three wavelengths closely matching the wavelengths of red (620 mm to 750 mm), green (495 nm to 570 nm), and blue (400 nm to 480 nm) (step 655). Other wavelengths may be used in other embodiments. In some embodiments, the FOV of the stained image location is matched to the FOV of the unstained sample by using reference points located in the sample. In some embodiments, these reference points (edges, labels, etc.) are present in the slide. In other embodiments, these reference points are engraved in the slide using a mechanical or laser device (i.e., a circular laser engraving in the slide). In some embodiments, once the multispectral digital microscope has completed scanning all AOIs, a full slide image (WSI) of the raw data is stored on a disk.

[0177] In some embodiments, the WSI of the stained sample is then processed to apply a previously calibrated color correction matrix to correct the colors and to perform white balance, in a manner that makes the image very similar to what the user would see using a bright-field microscope. It is believed that the advantage of this method is that the resolution of the digital files matches perfectly or substantially perfectly when the images are scanned in the same scanner. It is also believed that any optical artifacts will be similar. In some embodiments, co-registration of the spectral images of the unstained sample is anticipated. Furthermore, the FOVs of both the unstained and stained samples will be very closely registered to each other, thus simplifying the next step of registration.

[0178] In some embodiments, dimensionality reduction is performed. In some embodiments, principal component analysis (PCA) is performed on the spectral hypercube image to reduce the number of channels from n (in an exemplary 12 channels) to 3 (step 656) (see Figure 6B). In other embodiments, dimensionality reduction (e.g., PCA) is not performed, but a machine learning algorithm (such as GAN as described herein) is utilized, which can accept data from more than three channels (e.g., multispectral data) (see Figure 6C).

[0179] In some embodiments, the principal component values ​​are normalized to values ​​ranging from 0 to 255 and encoded in the channels of an RGB image, where the first channel (i.e., red) is the first principal component, the second channel (i.e., green) is the second principal component, and the third channel (i.e., blue) is the third principal component (step 657). This method is believed to reduce image complexity. In some embodiments, data from stained slides is encoded into an RGB image by performing white balance color correction to adjust the balance of each channel to achieve a white background as typically viewed with an optical microscope or from a calibrated digital scanner (steps 658A and 658B). In some embodiments, the hypercube image is used in its original form without further processing.

[0180] In some embodiments, once the dataset is complete (both unstained and stained), the next step is to co-register the images to the pixel level (steps 659A and 659B). To this end, in some embodiments, an automated algorithm transforms two images (e.g., a PCA image and a brightfield or baseline truth image) by applying color filters (i.e., Laplacian, Sobel, etc.) to areas where the general morphology of the tissue is enhanced, thereby causing the two color-filtered images to reveal similar structures (so-called descriptors). In some embodiments, a Fourier-based correlation algorithm is applied to translate and rotate the color-filtered image of the baseline truth to find locations and rotations that will produce higher correlation. In some embodiments, the desired transformations (translation and rotation) are applied to the baseline truth, resulting in two coarsely registered images (PCA and transformed brightfield).

[0181] In some embodiments, if WSI is being processed, the next step is to find the FOV of the stitched images. For this purpose, in some embodiments, a Laplacian color filter is applied to “find” discontinuities, thereby revealing stitching lines. In some embodiments, the two images are then split into separate field-of-view (FOV) images, and the above process is repeated, color-filtering the paired images to reveal descriptors, and finding optimal relevance by translating and rotating the color-filtered version of the FOV containing the reference truth. Once the optimal position is found, a transformation is applied to the FOV without color filtering, and the images are cropped to discard black areas created by rotating or shifting the reference truth, resulting in two co-registered images (PCA and reference truth). Finally, a final step is completed, further dividing each registered FOV into tiles, such as 270 x 270 pixel tiles. In some embodiments, this process is repeated at the tile level to create pixel-level registration tiles for a 256 x 256 pixel image (steps 660A and 660B). In some embodiments, the image is then saved to one or more memories (such as to a disk) communicatively coupled to the system of this disclosure to train the GAN algorithm (step 661).

[0182] In some embodiments, the previously stained tissue sections in H&E are then processed to remove the coverslips and remove the stain (e.g., by chemically removing the stain), and then re-stained with a different staining method (i.e., Masson's trichrome staining), and the coverslips are placed again. The re-stained slides are then scanned again in a multispectral microscope following the same procedure described above (RGB with three matching colors), and co-registered to an unstained hypercube to create a different dataset, which can be input to train the GAN algorithm.

[0183] Training biological samples

[0184] In some embodiments, multispectral and bright-field image data are acquired from one or more training biological samples. The obtained training biological samples can be obtained from any source. For example, the obtained training biological samples can be obtained from tumors, including, for example, tumor biopsy samples, excised samples, cell smears, fine-needle aspiration (FNA), liquid-based cytology samples, etc. In some embodiments, the obtained training biological samples are histological samples. In other embodiments, the obtained training biological samples are cytological samples.

[0185] As described herein, training multispectral transmission image data and training bright-field transmission image data are acquired from the same biological sample, i.e., training multispectral transmission image data is acquired from the biological sample when it is unstained, and training bright-field transmission image data is acquired from the same biological sample after it has been stained, such as with a primary staining agent (e.g., H&E), a special staining agent, or any other staining agent known in the art or as described herein.

[0186] In some embodiments, multiple training biological samples are obtained, and each of these multiple obtained training biological samples is used to train a different virtual staining engine (see, for example, Figure 7). For example, training multispectral transmission image data can be acquired from a first training biological sample in an unstained state; and then training bright-field transmission image data can be acquired from the first training biological sample after staining with H&E. The training multispectral transmission image data and the training bright-field transmission image data can then be used to train the H&E virtual staining engine. As another example, training multispectral transmission image data can be acquired from a second training biological sample in an unstained state; and then training bright-field transmission image data can be acquired from the second training biological sample after staining with a first special staining agent. The training multispectral transmission image data and the training bright-field transmission image data can then be used to train a first special staining agent virtual staining engine.

[0187] In other embodiments, multiple training biological samples are obtained, and each of these multiple obtained training biological samples is used to train the same virtual staining engine. In some embodiments, the multiple training biological samples obtained for training the same virtual staining engine originate from the same source (e.g., the same patient but different tissue blocks; the same tissue block but different sequential sections of different thicknesses, fixed time, etc.) (see, for example, Figure 8A). In other embodiments, the multiple training biological samples obtained for training the same virtual staining engine originate from different sources (e.g., different patients). For example, training multispectral transmission image data can be acquired from a first training biological sample in an unstained state; and then training bright-field transmission image data can be acquired from the first training biological sample after H&E staining. Training multispectral transmission image data can be acquired from a second training biological sample in an unstained state; and then training bright-field transmission image data can be acquired from the second training biological sample after H&E staining. The training multispectral transmission image data and training bright-field transmission image data from the first and second training biological samples can then be used to train the H&E virtual staining engine. It is believed that training the same virtual staining engine from training biological samples obtained from different sources allows for consideration of any lab-to-lab, tissue-to-tissue, fixed-level / quality variations, etc., during the training of the virtual staining engine.

[0188] In some embodiments, a plurality of training biological samples are obtained, wherein each of the plurality of training biological samples has the same tissue type (e.g., tonsil tissue), and wherein the plurality of training biological samples of the same tissue type are used to train the same virtual staining engine. In other embodiments, a plurality of training biological samples are obtained, wherein each of the plurality of training biological samples has a different tissue type, and wherein the plurality of training biological samples of different tissue types are used to train the same virtual staining engine.

[0189] In some embodiments, multiple training biological samples of the same type are obtained, and each of the multiple obtained training biological samples of the same type is used to train a different virtual staining engine. In some embodiments, multiple training biological samples of the same type are obtained from different sources. In other embodiments, the multiple training biological samples of the same type are different consecutive slices from the same tissue block, and each different training consecutive slice can be used to train a different virtual staining engine (see, for example, Figure 9).

[0190] In other embodiments, multiple training biological samples of the same type are obtained, but each of the multiple training biological samples of the same type obtained has a different thickness, a different type of glass substrate, and / or a different fixation state, etc.; and each of the multiple training biological samples of the same type obtained is used to train the same virtual staining engine.

[0191] Training multispectral transmission image data derived from unstained biological samples / training multispectral transmission images Channel Image

[0192] Training multispectral transmission image data is acquired from each of a plurality of training unstained biological samples. In some embodiments, two or more training multispectral transmission image channels are acquired for each of the plurality of training unstained biological samples, wherein each of the two or more training multispectral transmission image channels is acquired at a specific wavelength. In some embodiments, the two or more training multispectral transmission image channels may be processed and / or combined in one or more downstream operations.

[0193] In some embodiments, at least two training multispectral transmission image channels are acquired for each unstained training biological sample, wherein a multispectral image acquisition device 12A is used to acquire each of the at least two training multispectral transmission image channels, the multispectral image acquisition device being configured to illuminate each unstained training biological sample with at least two different illumination sources; and further configured to acquire transmission image data (e.g., at least two multispectral image channels) of the biological samples illuminated by the at least two different illumination sources. In other words, at least two training multispectral transmission image channels are acquired for each unstained training biological sample, wherein each of the at least two training multispectral transmission image channels is acquired at a different wavelength. In some embodiments, four training multispectral transmission image channels are obtained from at least four different illumination sources. In some embodiments, five training multispectral transmission image channels are obtained from at least five different illumination sources. In some embodiments, training multispectral transmission image channels are obtained from at least six different illumination sources. In some embodiments, training multispectral transmission image channels are obtained from at least seven different illumination sources. In some embodiments, training multispectral transmission image channels are obtained from at least eight different illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from at least nine different illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from at least ten different illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from at least eleven different illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from at least twelve different illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from twelve or more illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from sixteen or more different illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from twenty or more different illumination sources. In some embodiments, training multispectral transmission image channel images are obtained from twenty-four or more different illumination sources.

[0194] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources (see, for example, FIG. 5), wherein the at least two different illumination sources are from the ultraviolet (UV) spectrum, from the visible spectrum, and / or from the infrared (IR) spectrum. In some embodiments, the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum.

[0195] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with six or more different illumination sources, wherein the at least six different illumination sources are from the ultraviolet (UV) spectrum, from the visible spectrum, and / or from the infrared (IR) spectrum. In some embodiments, the at least six different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum.

[0196] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with twelve or more different illumination sources, wherein the at least twelve different illumination sources are from the ultraviolet (UV) spectrum, from the visible spectrum, and / or from the infrared (IR) spectrum. In some embodiments, the at least twelve different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum.

[0197] In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate the training biological sample with two or more different illumination sources, wherein at least two of the illumination sources are from at least two different wavelengths within the UV spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate the training biological sample with two or more different illumination sources, wherein at least two of the illumination sources are from at least two different wavelengths within the visible spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate the training biological sample with two or more different illumination sources, wherein at least two of the illumination sources are from at least two different wavelengths within the IR spectrum.

[0198] In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with four or more different illumination sources (such as narrow illumination sources), wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with: (i) at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least two different illumination sources in the IR spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with six or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with: (i) at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least two different illumination sources in the IR spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with eight or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with: (i) at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least two different illumination sources in the IR spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with ten or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with: (i) at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least two different illumination sources in the IR spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with twelve or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with: (i) at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least two different illumination sources in the IR spectrum.In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with sixteen or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least two different illumination sources in the IR spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with twenty or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least two different illumination sources in the IR spectrum.

[0199] In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with nine or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with: (i) at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least three different illumination sources in the IR spectrum. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with twelve or more different illumination sources, wherein the training multispectral transmission image channel images are acquired after illuminating the biological samples with: (i) at least two different illumination sources in the UV spectrum; (ii) at least two different illumination sources in the visible spectrum; and (iii) at least three different illumination sources in the IR spectrum.

[0200] In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate the training biological sample with two or more different illumination sources, wherein the two or more different illumination sources differ by at least 120 nm. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate the training biological sample with two or more different illumination sources, wherein the two or more different illumination sources differ by at least 100 nm. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate the training biological sample with two or more different illumination sources, wherein the two or more different illumination sources differ by at least 80 nm. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate the training biological sample with two or more different illumination sources, wherein the two or more different illumination sources differ by at least 60 nm. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the two or more different illumination sources differ by at least 40 nm.

[0201] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least three different illumination sources have wavelengths of approximately 365 + / - 300 nm, approximately 400 + / - 300 nm, approximately 435 + / - 300 nm, approximately 470 + / - 300 nm, approximately 500 + / - 300 nm, approximately 550 + / - 300 nm, approximately 580 + / - 300 nm, approximately 635 + / - 300 nm, approximately 660 + / - 300 nm, approximately 690 + / - 300 nm, and approximately 780 + / - 300 nm. And / or an average wavelength of approximately 850 + / - 300 nm.

[0202] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least three different illumination sources have wavelengths of approximately 365 + / - 200 nm, approximately 400 + / - 200 nm, approximately 435 + / - 200 nm, approximately 470 + / - 200 nm, approximately 500 + / - 200 nm, approximately 550 + / - 200 nm, approximately 580 + / - 200 nm, approximately 635 + / - 200 nm, approximately 660 + / - 200 nm, approximately 690 + / - 200 nm, and approximately 780 + / - 200 nm. And / or an average wavelength of approximately 850 + / - 200 nm.

[0203] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least three different illumination sources have wavelengths of approximately 365 + / - 100 nm, approximately 400 + / - 100 nm, approximately 435 + / - 100 nm, approximately 470 + / - 100 nm, approximately 500 + / - 100 nm, approximately 550 + / - 100 nm, approximately 580 + / - 100 nm, approximately 635 + / - 100 nm, approximately 660 + / - 100 nm, approximately 690 + / - 100 nm, and approximately 780 + / - 100 nm. And / or an average wavelength of approximately 850 + / - 100 nm.

[0204] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least three different illumination sources have wavelengths of about 365 + / - 50 nm, about 400 + / - 50 nm, about 435 + / - 50 nm, about 470 + / - 50 nm, about 500 + / - 50 nm, about 550 + / - 50 nm, about 580 + / - 50 nm, about 635 + / - 50 nm, about 660 + / - 50 nm, about 690 + / - 50 nm, about 780 + / - 50 nm, and / or about 850 + / - 50 nm. The average wavelength.

[0205] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least three different illumination sources have wavelengths of about 365 + / - 30 nm, about 400 + / - 30 nm, about 435 + / - 30 nm, about 470 + / - 30 nm, about 500 + / - 30 nm, about 550 + / - 30 nm, about 580 + / - 30 nm, about 635 + / - 30 nm, about 660 + / - 30 nm, about 690 + / - 30 nm, about 780 + / - 30 nm, and / or about 850 + / - 30 nm. The average wavelength.

[0206] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least three different illumination sources have wavelengths of about 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm, and / or about 850 + / - 20 nm. The average wavelength.

[0207] In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with two or more different illumination sources, wherein the at least two different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least one illumination source in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least three different illumination sources have average wavelengths of about 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0208] In some embodiments, training multispectral transmission image channels are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with six or more different illumination sources, wherein the at least six different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least two illumination sources in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least six different illumination sources have wavelengths of about 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm, and / or about 850 + / - 20 nm. The average wavelength. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with six or more different illumination sources, wherein the at least six different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least two illumination sources in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least six different illumination sources have wavelengths of about 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm, and / or about 850 + / - 10 nm. The average wavelength.In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with six or more different illumination sources, wherein the at least six different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least two illumination sources in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least six different illumination sources have an average wavelength of about 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0209] In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with twelve or more different illumination sources, wherein the at least twelve different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least two illumination sources in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least twelve different illumination sources have wavelengths of about 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm. The average wavelength is 20 nm. In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with twelve or more different illumination sources, wherein the at least twelve different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least two illumination sources in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least twelve different illumination sources have wavelengths of about 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm. The average wavelength is 10 nm.In some embodiments, training multispectral transmission image channel images are acquired from a multispectral image acquisition device 12A, which is configured to illuminate training biological samples with twelve or more different illumination sources, wherein the at least twelve different illumination sources include (i) at least one illumination source in the ultraviolet (UV) spectrum; (ii) at least two illumination sources in the visible spectrum; and (iii) at least one illumination source in the infrared spectrum; and wherein the at least twelve different illumination sources have an average wavelength of about 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0210] In some embodiments, the selection and / or number of different illumination sources in any of the UV, visible, or IR spectra are chosen based on a specific tissue type; or based on organelles or cellular structures within the biological sample relevant to downstream virtual staining generation and / or analysis. In these embodiments, transmission image data is acquired from the training biological sample using one or more illumination sources that match one or more average or peak absorbances of one or more cellular structures and / or one or more target organelles.

[0211] For example, nucleic acids (e.g., DNA) present within cells or the cell nucleus may be relevant to virtual staining generation and / or downstream analysis. Nucleic acids are known to absorb energy in the UV spectrum (e.g., below about 300 nm) and the visible spectrum (e.g., between 460 nm and about 490 nm). Training multispectral transmission image data from different illumination sources, emitting energy at approximately the peak absorbance wavelengths of nucleic acid molecules (and / or their associated structures, such as histones), can aid in the training of a virtual staining engine 210, which can then be used to generate virtual stains that mimic conventional nuclear stains (i.e., hematoxylin).

[0212] In some embodiments, the selection and / or number of different illumination sources in any of the UV, visible, or IR spectra are chosen based on the wavelength characteristics of a particular type of morphological staining agent. For example, H&E staining is used to provide a morphological map of the tissue sample based on the contrast of negatively or positively charged tissue components. This morphological map can be used to identify the presence, absence, arrangement, and appearance of tissue structures for diagnostic and prognostic pathology. It is possible to map the same structures using endogenous contrast of tissue at different wavelengths, for example, using 250 nm to highlight the cell nucleus and using 420 nm to highlight endogenous cytochromes corresponding to the cytoplasm.

[0213] Generation of multi-channel multispectral training transmission images

[0214] Typically, and referring to FIG10, one or more, such as two or more, training multispectral transmission image channel images are obtained (step 120). In some embodiments, the one or more, such as two or more, training multispectral transmission image channel images are simply combined into a multichannel image (i.e., no compression or dimensionality reduction is applied). In other embodiments, the obtained two or more training multispectral transmission images are reduced or compressed into a multichannel multispectral training transmission image (step 121).

[0215] In some embodiments, dimensionality reduction methods are used to compress the obtained two or more training multispectral transmission image channels to generate a multispectral training image. Examples of suitable dimensionality reduction methods include principal component analysis (PCA) (such as PCA plus discriminant analysis), projection to latent structure regression and t-distributed random neighbor embedding (t-SNE), and uniform manifold approximation and projection (UMAP). See Omucheni, Dickson L. et al., “Application of principal component analysis to multispectral-multimodal optical image analysis for malaria diagnostics.” Malaria journal 13 (2014): 1-11; and Farrugia, Jessica et al., “Principal component analysis of hyperspectral data for early detection of mould in cheeselets.” Current Research in Food Science 4 (2021): 18-27, the contents of which are hereby incorporated herein by reference in their entirety.

[0216] In some embodiments, dimension reduction techniques are used to compress four or more training multispectral transmission image channels into a multi-channel multispectral training transmission image. In some embodiments, dimension reduction techniques are used to compress five or more training multispectral transmission image channels into a multi-channel multispectral training transmission image. In some embodiments, dimension reduction techniques are used to compress six or more training multispectral transmission image channels into a multi-channel multispectral training transmission image. In some embodiments, dimension reduction techniques are used to compress seven or more training multispectral transmission image channels into a multi-channel multispectral training transmission image. In some embodiments, dimension reduction techniques are used to compress eight or more training multispectral transmission image channels into a multi-channel multispectral training transmission image. In some embodiments, dimension reduction techniques are used to compress nine or more training multispectral transmission image channels into a multi-channel multispectral training transmission image. In some embodiments, dimension reduction techniques are used to compress ten or more training multispectral transmission image channels into a multi-channel multispectral training transmission image. In some embodiments, dimensionality reduction techniques are used to compress 11 or more training multispectral transmission image channels into a multichannel multispectral training transmission image. In some embodiments, dimensionality reduction techniques are used to compress 12 or more training multispectral transmission image channels into a multichannel multispectral training transmission image. In some embodiments, dimensionality reduction techniques are used to compress 16 or more training multispectral transmission image channels into a multichannel multispectral training transmission image. In some embodiments, dimensionality reduction techniques are used to compress 20 or more training multispectral transmission image channels into a multichannel multispectral training transmission image. In some embodiments, dimensionality reduction techniques are used to compress 24 or more training multispectral transmission image channels into a multichannel multispectral training transmission image. Similarly, two or more training multispectral transmission image channels can be fed to a machine learning algorithm without any compression steps, such as any dimensionality reduction steps (e.g., principal component analysis (PCA)).

[0217] In some embodiments, PCA is used to reduce the dimensionality of a dataset comprising four or more multispectral transmission training image channels. Typically, PCA is used to reduce the dimensionality of a dataset consisting of many interrelated variables to the greatest extent possible while preserving the variability present in the dataset. This is accomplished by transforming the variables into a new set of variables called principal components (or simply PCs) and orthogonally ordered such that the preservation of variability present in the original variables decreases as the variable order shifts down. In this way, the first principal component preserves the variability present in the original components. Principal components are eigenvectors of the covariance matrix, and therefore they are orthogonal. Principal component analysis and methods employing it are described in U.S. Patent Publication No. 2005 / 0123202 and U.S. Patent Nos. 6,894,639 and 8,565,488, the disclosures of which are hereby incorporated herein by reference in their entirety. Khan et al.'s "Principal Component Analysis - Linear Discriminant Analysis Feature Extractor for Pattern Recognition," published in the IJCSI International Journal of Computer Sciences Issues, Volume 8, Issue 6, No. 2, November 2011, further describes PCA and linear discriminant analysis, the contents of which are hereby incorporated herein by reference in their entirety.

[0218] The t-SNE algorithm is a nonlinear dimensionality reduction technique well-suited for embedding high-dimensional data into low-dimensional spaces (two- or three-dimensional) for visualization. Specifically, it models each high-dimensional object using two- or three-dimensional points, such that similar objects are modeled by nearby points, while dissimilar objects are modeled with a high probability by distant points. The t-SNE algorithm comprises two main stages. First, t-SNE constructs a probability distribution over pairs of high-dimensional objects, such that similar objects have a high probability of being picked, while dissimilar points have a very low probability of being picked. Second, t-SNE defines a similarity probability distribution over points in a low-dimensional graph, and it minimizes the Kullback-Leibler divergence between the two distributions regarding the location of points in the graph. The t-SNE algorithm is further described in U.S. Patent Publications 2018 / 0046755, 2014 / 0336942, and 2018 / 0166077, the disclosures of which are hereby incorporated herein by reference in their entirety.

[0219] In embodiments utilizing image compression or dimensionality reduction, Figure 11 illustrates a method for compressing four or more training multispectral transmission image channel images into a single multispectral training image. First, four or more, five or more, six or more, seven or more, eight or more, nine or more, ten or more, eleven or more, twelve or more, sixteen or more, twenty or more, twenty or fourteen or more, etc., training multispectral transmission image channel images are obtained (step 130). Then, each of the obtained four or more training multispectral transmission image channel images is transformed into a data matrix (step 131). For example, for 12 acquired training multispectral image channel images, the stacked image is transformed into a 3D matrix with 2 spatial dimensions and 12 channels. Next, the data matrix is ​​transformed into a linear vector of pixels, where each pixel includes each of the four or more channels (step 132). Principal component analysis transformation is then applied to the linear vector to reduce the four or more channels to three (step 133). For example, for the 12 acquired training multispectral image channels, PCA reduction is applied to the matrix, reducing the number of channels from 12 to 3. The principal component channels are then normalized from 0 to 255 (step 134). Subsequently, the linear vector is returned to the three-dimensional matrix (step 135). The three-dimensional matrix is ​​then transformed into an image using the first of the three channels as the red channel, the second of the three channels as the green channel, and the third of the three channels as the blue channel (step 136). The resulting RGB multispectral training image is then stored in one or more memories 201 or one or more storage units 240 (step 137).

[0220] Training bright-field transmission images

[0221] After the acquisition of training multispectral transmission image data / training multispectral transmission image channel images, the training biological samples are stained with a morphological staining agent. Subsequently, one or more training bright-field transmission images can be acquired from the morphologically stained training biological samples, such as using bright-field image acquisition device 12B. Alternatively, training bright-field transmission images can be obtained using multispectral image acquisition device 12A, such as by acquiring image data at three or predetermined wavelengths, such as at approximately 700 nm + / - 10 mm, approximately 550 nm + / - 10 mm, and approximately 470 nm + / - 10 mm. An example of a training bright-field transmission image is shown in Figure 12B. Those skilled in the art will understand that the acquired training bright-field transmission images are used as a baseline truth when training a virtual staining engine, as described herein.

[0222] In some embodiments, the morphological staining agent is hematoxylin, which stains the cell nucleus blue. In other embodiments, the morphological staining agent is eosin, which stains the cytoplasm pink. In other embodiments, the obtained biological samples are stained with both hematoxylin and / or eosin (H&E). In some embodiments, an H&E staining protocol may be performed, which includes applying a hematoxylin staining agent or mordant mixed with a metal salt to a tissue section. The tissue section may then be rinsed in a weakly acidic solution to remove excess staining (differentiation) and then turned blue in weakly alkaline water. In some embodiments, the tissue may be counterstained with eosin after the application of hematoxylin. It should be understood that other H&E staining techniques may be performed.

[0223] In other embodiments, the morphological staining agent is a "specific staining agent". A "specific staining agent" means any chemical-based staining agent that is useful for histological analysis, and is not an immunohistochemical staining agent, in situ hybridization staining agent, or H&E. In some embodiments, the specific staining agent includes one or more reagents selected from the following: Acid Fuchsin (CI42685; maximum absorbance 546 nm), Alcian Blue 8 GX (CI74240; maximum absorbance 615 nm), Alizarin Red S (CI58005; maximum absorbance 556 and 596 nm), Auramine O (CI41000; maximum absorbance 370 and 432 nm), Azo Carmine B (CI50090; maximum absorbance 516 nm), Azo Carmine G (CI50085; maximum absorbance 511 nm), Azure A (CI52005; absorbance similar to Azure B), Azure B (CI52010; maximum absorbance 639 nm), Basic Fuchsin (CI42510; maximum absorbance 547 to 552 nm), Bismarck Brown Y (CI21000; maximum absorbance 643 nm). Brilliant cresol blue (CI51010; maximum absorbance 622 nm), carmine (CI75470; protonated maximum absorbance 490 to 495, increasing in alkali and when combined with metal salts), chlorazole black E (CI30235; maximum absorbance 500 to 504 nm and 574 to 602 nm), Congo red (CI22120; maximum absorbance 497 nm), cresol violet (maximum absorbance 596 to 601 nm), crystal violet (CI42555; maximum absorbance 590 nm), daro red (maximum absorbance 502 nm), ethyl green (CI42590; maximum absorbance 635 nm, 420 nm), fast green FCF (CI42053; maximum absorbance 624 nm, depending on pH), Giemsa staining agent (impure azure B, methylene blue, and eosin Y). A mixture of these, indigo carmine (CI73015; maximum absorbance 608 nm), Jenners green B (CI11050; maximum absorbance 630 nm), Janus 1899, light green SF (CI42095; maximum absorbance 422 and 630 nm), malachite green (CI42000; maximum absorbance 614 and 425 nm), Matthew yellow (CI10315; maximum absorbance 420 to 432 nm), methyl orange (CI13025; maximum absorbance 507 nm), methyl violet 2B (CI...42535; maximum absorbance 583 to 587 nm), Methylene Blue (CI52015; maximum absorbance 656 to 661 nm), Methylene Violet (Bernthsen), (CI52041; maximum absorbance 580 to 601 nm), Neutral Red (CI50040; maximum absorbance 454, 529, 541 nm, depending on pH and solvent), Aniline Black (CI50420; maximum absorbance 570 to 580 nm), Nile Blue A (CI51180; maximum absorbance 633 to 660 nm), Solid Red (CI60760; maximum absorbance 535 and 505 nm), Oil Red O (CI26125; maximum absorbance 518 and 359 nm), Orange G (CI16230; maximum absorbance 475 nm). (nm), Orange II (CI15510; maximum absorbance 483 nm), Lichen Red (maximum absorbance 575 to 590, depending on pH), Paracinnamon (CI42500; maximum absorbance 545 nm), Root Bark Red B (CI45410; maximum absorbance 548 and 510 nm), Pyrrolidine B (CI45010; closely related to Pyrrolidine Y), Pyrrolidine Y (CI45005; maximum absorbance 546 to 549 nm), Resazil (maximum absorbance 598 nm in water, 478 nm in methanol), Rose Red (CI45435; maximum absorbance 546 nm), Safranin O (CI50240; maximum absorbance 530 nm), Sudan Black B (CI26150; maximum absorbance 598 and 415 nm), Sudan III (CI26100; maximum absorbance 503 to 507 nm). And 503 nm), Sudan IV (CI26105; maximum absorbance 520 nm), four-color staining agent (MacNeal), thionine (CI52000; maximum absorbance 598 to 602 nm), toluidine blue (CI52040; maximum absorbance 626 to 630 nm), Weigert resorcinol fuchsin (maximum absorbance 508 nm), Wright's staining agent, and any combination thereof. In each of these examples, "CI"This refers to Color Index™. Color Index™ describes commercial products by their recognized use categories, their hues, and serial numbers (which simply reflect the chronological order in which the relevant colorant type has been registered in the Color Index). This definition allows a particular product to be classified together with other products that have the same chemical composition as their base colorant and where that base colorant comes from a single chemical reaction or a series of reactions. Other special stains include, but are not limited to, PAS staining kits, GMS II special staining kits, Reticulum II staining kits, iron staining kits, GIEMSA staining kits, trichrome staining kits, amylase kits, Benchmark AFB special staining kits, ALCIAN BLUE FOR PAS, LIGHT GREENFOR PAS, STEINER II staining kits, Congo Red staining kits, Van Gieson CS special staining kits, Elastic core staining kits, Jones staining kits, ALC BLUE staining kits PH2.5, mucin carmine staining kits, GRAM staining kits, GREEN FOR TRICHROME, Jones Light green staining kits, each of the above items, are available from Roche Diagnostics. In some embodiments, a special staining agent is used for the Grocott methenamine silver assay. In other embodiments, a special staining agent is used for the Masson trichrome staining assay.

[0224] In some embodiments, the samples are morphologically stained according to the procedures described in PCT application numbers PCT / EP2021 / 073738 or PCT / EP20-21 / 073733m, the disclosure of which is hereby incorporated herein by reference in its entirety.

[0225] In some embodiments, the acquired training multispectral transmission image data / training multispectral transmission image channel images and the acquired bright-field transmission images are each associated with one or more identifiers. For example, the identifiers may include sample number, sample type, staining agent type, fixation characteristics, etc. In some embodiments, the acquired training multispectral transmission image channel images and the acquired bright-field transmission images are each associated with a unique identifier, such that the training multispectral transmission image channel images (or any multispectral training images derived therefrom) and the training bright-field transmission images can be associated with each other and retrieved from the one or more memories 201 and / or one or more storage units 240 to facilitate the preparation of multiple pairs of co-registered training images for training the virtual staining engine 210.

[0226] Preparation of training images for multiple co-registration pairs

[0227] Those skilled in the art will understand that chemical treatment of unstained tissue with morphological staining agents can cause the tissue to stretch, contract, and / or move. Therefore, the training bright-field transmission image (derived from stained training biological samples) can have tissue or cellular structures that are stretched, contracted, and / or moved relative to the generated multi-channel multispectral training transmission image (derived from unstained training biological samples). These differences in tissue location, etc., are explained by aligning or co-registering the features present in each of the generated multi-channel multispectral training transmission image and the training bright-field transmission image. In view of the foregoing, after generating the multi-channel multispectral training image (which can be derived from one or more uncompressed image channels or from one or more compressed image channels) and acquiring the training bright-field transmission image from the same training biological sample, the image processing module 212 is used to generate a pair of co-registered training images from the generated multi-channel multispectral training transmission image and the training bright-field transmission image, such that the features in each image are aligned and / or co-registered (see steps 140 and 141 of Figure 13A). Then, the co-registered training images are used to train machine learning algorithms, that is, to train the virtual staining engine.

[0228] One method for co-registering images is illustrated in Figure 24. Other methods for co-registering images relative to each other are known and described in the literature, see, for example, D. Mueller et al., Real-time deformable registration of multi-modal whole slides for digital pathology. Computerized Medical Imaging and Graphics, Vol. 35, pp. 542-556 (2011); F. El-Gamal et al., Current trends in medical image registration and fusion, Egyptian Informatics Journal, Vol. 17, pp. 99-124 (2016); J. Singla et al., A systematic way of affine transformation using image registration, International Journal of Information Technology and Knowledge Management, July-December 2012, Vol. 5, No. 2, pp. 239-243; Z. Hossein-Nejad et al., An adaptive image registration method based on SIFT features and RANSAC transform, Computers and electrical engineering, Vol. 62, p. U.S. Patent Nos. 5240537 (August 2017); 8,605,972 and 9,785,818, the disclosures of which are incorporated herein by reference in their entirety. In some embodiments, the generated multi-channel multispectral training transmission image and training bright-field transmission image may be aligned or co-registered with RANSAC (Random Sample Consensus, a known algorithm for image alignment).

[0229] Figure 13B illustrates a method for co-registering a generated multi-channel multispectral training transmission image and a training bright-field transmission image. As an initial step, both the generated multi-channel multispectral training transmission image and the training bright-field transmission image are obtained according to the method described herein (step 150). In some embodiments, the obtained image is segmented, such as into 64 pixel x 64 pixel segments, 128 pixel x 128 pixel segments, 256 pixel x 256 pixel segments, 512 pixel x 512 pixel segments, etc. Next, one or more target regions (ROIs) are identified in each of the obtained generated multi-channel multispectral training transmission image and the training bright-field transmission image (step 151). Then, features are identified in each of the identified one or more ROIs (step 152). Non-limiting examples of features include epithelial cell nuclei, adipocytes, inflammatory cells, etc. In some embodiments, lumens, glands, and / or adipocytes are used as first landmarks to locate the region. After "magnifying" / increasing the magnification, epithelial cells, erythrocytes, and inflammatory cells can be located.

[0230] Landmarks are then placed in each of the generated multi-channel multispectral training transmission image and the generated training bright-field transmission image (step 153). The placement of landmarks in both the generated multi-channel multispectral training transmission image and the generated training bright-field transmission image is shown in Figures 14A and 14B, respectively. Transformations are then applied to the obtained training bright-field transmission image by translation, rotation, shearing, shrinking, and / or stretching to match the landmarks placed in the generated multi-channel multispectral training transmission image (step 154). In some embodiments, the transformation is an affine transformation, such as a transformation that allows shearing and scaling in addition to rotation and translation.

[0231] The machine learning algorithm is trained using multi-pair co-registration training images.

[0232] After co-registration of each multi-channel multispectral training transmission image and the training brightfield transmission image in the generated multi-channel multispectral training transmission images, multiple pairs of co-registered images (or segments of these paired images) are fed to a machine learning algorithm to train / teach the machine learning algorithm (virtual staining engine) to predict virtual staining images. Pixel-to-pixel, cell-to-cell, and / or tile-to-tile mappings are performed using multiple pairs of co-registered training image pairs.

[0233] Examples of machine learning algorithms include:

[0234] 1) Self-supervised learning neural networks.

[0235] 2) Convolutional Neural Networks. As used herein, the term "neural network" refers to one or more computer-implemented networks that can be trained to achieve a goal. Unless otherwise stated, references to neural networks herein include a single neural network or multiple interconnected neural networks trained together. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), fully connected neural networks, encoder neural networks (e.g., "encoders"), decoder neural networks (e.g., "decoders"), densely connected neural networks, and other types of neural networks. In some embodiments, neural networks may be implemented using dedicated hardware (e.g., GPUs, Tensor Processing Units (TPUs), systolic arrays, single instruction multiple data (SIMD) processors, etc.), software code, and a combination of general-purpose processors or dedicated hardware and software code.

[0236] In some embodiments, the neural network is configured as a deep learning network. Generally, "deep learning" is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data. Deep learning is part of a broader family of machine learning methods that learn representations based on data. Observations can be represented in many ways, such as a vector of intensity values ​​for each pixel, or in a more abstract way as a set of edges, regions of a specific shape, etc. Some representations are superior to others in simplifying the learning task. One of the prospects of deep learning is to replace handcrafted features with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction.

[0237] In some embodiments, the neural network can be a deep neural network with a set of weights that model the world based on data that has been fed to train it. Neural networks typically consist of multiple layers, with signal paths running from front to back between these layers. Any neural network can be implemented for this purpose. Suitable neural networks include LeNet, AlexNet, ZFnet, GoogLeNet, VGGNet, VGG16, DenseNet (also known as a dense convolutional network or DenseNet-121), MiniNet, and ResNet. In some embodiments, fully convolutional neural networks are utilized, such as those described in "Fully Convolutional Networks for Semantic Segmentation," ComputerVision and Pattern Recognition (CVPR), 2015 IEEE Conference, June 2015 (INSPEC Accession Number: 15524435), the disclosure of which is hereby incorporated herein by reference. Other suitable neural networks include convolutional neural networks (CNNs), recurrent neural networks, long short-term memory neural networks (LSTMs), composite scaling efficient neural networks (EfficientNet), non-normalized neural networks (NFNet), densely connected convolutional neural networks (DenseNet), aggregated residual transform neural networks (ResNeXT), channel-enhanced convolutional neural networks (CB-CNNs), wide residual networks (WRNs), or residual neural networks (RNNs). In some embodiments, the neural network is a neural network that operates on cross-entropy, for example, a neural network that can identify the cross-entropy of each pixel.

[0238] 3) Implicit generative models, such as Generative Adversarial Networks (GANs). This neural network approach is described in K. Bousmalis et al., Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks, https: / / arxiv.org / pdf / 1612.05424.pdf (August 2017), the entire contents of which are hereby incorporated by reference. In some embodiments, the neural network is a generative network. Further details on GANs can be found in Goodfellow et al., Generative Adversarial Nets., Advances in Neural Information Processing Systems, 27, pp. 2672-2680 (2014), which is hereby incorporated by reference in its entirety.

[0239] A “generative” network can generally be defined as a model that is inherently probabilistic. In other words, a “generative” network is not a network that performs forward simulation or a rule-based approach. Instead, a generative network can be learned based on a suitable training dataset (e.g., multiple training image datasets) because its parameters can be learned. In some embodiments, the neural network is configured as a deep generative network. For example, the network can be configured with a deep learning architecture, where the network can include multiple layers that perform a number of algorithms or transformations. As used herein, the term “layer” or “network layer” refers to the analysis phase in a neural network. Layers perform different types of analysis depending on the type of neural network. For example, layers in an encoder can perform different types of analysis on the input image to encode the input image. In some cases, a particular layer provides features based on a specific analysis performed by that layer. In some cases, a particular layer downsamples the received image. Additional layers perform additional downsampling. In some cases, each round of downsampling reduces the visual quality of the output image but provides features based on the relevant analysis performed by that layer.

[0240] Introduced in 2014 by I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative Adversarial Nets," in Advances in Neural Information Processing Systems, pp. 2672–2680, 2014, Generative Adversarial Nets (GANs) are used to model the distribution of training image data and then use it to generate new image samples from the same distribution. They consist of two networks: a generator (G) and a discriminator. The generative model G learns mappings from the training data to generate new samples from some prior distribution (a vector of random noise) by mimicking the distribution of the real data. On the other hand, the discriminator D attempts to classify the images generated by G, regardless of whether they come from the real training data (the real distribution) or fake ones. The two networks are trained and updated simultaneously, as if they were playing a game. That is, the generator G attempts to deceive the discriminator D, and the discriminator D then adjusts its parameters to make a better estimate to detect the fake image generated by G.

[0241] In some embodiments, the GAN is a recurrent GAN. Instances of suitable recurrent GAN network architectures are described by Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros in “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks” (November 24, 2017).

[0242] In some embodiments, the GAN is a three-channel GAN, which utilizes data from three channels, such as the three channels in a compressed (e.g., dimensionally reduced) input image. In other embodiments, the GAN is a multi-channel GAN, which utilizes data from multiple channels (e.g., two or more channels, three or more channels, four or more channels, five or more channels, six or more channels, seven or more channels, eight or more channels, nine or more channels, ten or more channels, eleven or more channels, twelve or more channels, sixteen or more channels, twenty or more channels, twenty-four or more channels, etc.).

[0243] In some embodiments, the GAN training procedure involves training two distinct networks: (i) a generator network, which in this case is designed to learn the statistical transformation between an unstained multichannel multispectral training transmission image and its corresponding bright-field image after the training biological sample has been stained with a morphological stain; and (ii) a discriminator network, which learns how to distinguish between the real bright-field image of the stained training sample and the output image of the generator network. Ultimately, the desired outcome of this training process is a trained deep neural network that transforms the input image of the unstained training sample into a virtual stained image of the unstained biological sample, which is diagnostically equivalent to a chemically stained bright-field image of the same biological sample. In some embodiments, there is no discernible difference in diagnostic quality between the resulting virtual stained image of the unstained biological sample and its corresponding image of the chemically stained biological sample, at least not to the extent that any difference would significantly alter the diagnostic outcome.

[0244] In some embodiments, the GAN is used to identify differences between multiple pairs of co-registered training images, wherein a bright-field training transmission image in each pair of co-registered training images serves as the baseline truth. In some embodiments, the process of identifying differences between a pair of co-registered training images can be performed within a "discriminator network". In some embodiments, the difference between the bright-field training transmission image and the multi-channel multispectral training transmission image (in the pair of co-registered training images) can then be formulated as a loss function. The loss function can be passed to the discriminator network and then to the generator network for backpropagation. In some embodiments, the input from the loss function enables the discriminator network to learn how to distinguish between actual or real images of stained training biological samples and the virtual histological images output by the generator network.

[0245] In some embodiments, a generator network computes virtual stained images, and a discriminator network evaluates the similarity of each virtual stained image to an actual histological image (i.e., a bright-field training transmission image included within a pair of co-registered training images). The generator network can then utilize a loss function to generate a corrected virtual image, which is then fed to the discriminator network. This process can be iterative until the discriminator network can no longer distinguish the generated virtual stained images from the bright-field training transmission images (images containing actual morphological stains).

[0246] In some embodiments, through backpropagation, the classification by the discriminator network helps the generator network update its weights, thereby fine-tuning the virtual histological images being generated. Eventually, after several iterations, the generator network begins to output higher-quality virtual stained histological images, and the discriminator network becomes better at distinguishing virtual stained histological images from actual histological images. Once trained, the network can generate virtual stained images representing morphological staining agents (e.g., H&E, basic fuchsin, Masson's trichrome, etc.).

[0247] Use a trained virtual staining engine to generate virtual stained slides.

[0248] As described herein, this disclosure provides a method for generating a virtual stained image of a test unstained biological sample based on an acquired input image of the sample, wherein the virtual stained image displays the appearance of the test unstained biological sample as if it were stained with a morphological stain. Once the virtual staining engine is trained, an unstained input image (e.g., a multispectral test transmission image) is fed to the trained virtual staining engine, and the trained virtual staining engine generates a predicted or virtual stained output image corresponding to the unstained input image. For example, if the selected virtual staining engine is trained to predict H&E staining from an unstained input image, the trained virtual staining engine generates a predicted or virtual stained H&E output image for the unstained input image. The predicted or virtual stained output image can then be stored in one or more memories or storage systems for later retrieval and downstream processing (e.g., analysis performed by a pathologist or other automated image analysis workflow).

[0249] Referring to Figure 15, in some embodiments, the method for generating a virtual stained slide includes (i) acquiring test multispectral transmission image data from an unstained test biological sample (step 161); (ii) supplying the test multispectral transmission image data to a trained virtual staining engine 210, which is trained to generate an image of an unstained biological sample stained with a morphological staining agent (step 162); and (iii) using the trained virtual staining engine to generate a virtual stained image of the unstained test biological sample stained with a morphological staining agent (step 163).

[0250] In some embodiments, test multispectral image data are acquired from one or more unstained biological samples. The obtained test biological samples can be obtained from any source. For example, the obtained test biological samples can be obtained from tumors, including, for example, tumor biopsy samples, excised samples, cell smears, fine needle aspiration (FNA), liquid-based cytology samples, etc. In some embodiments, the obtained test biological samples are derived from previously stained samples; and wherein the previous staining agent has been substantially removed.

[0251] Similar to the acquisition of training multispectral transmission image channel images described herein, one or more (such as two or more) test multispectral transmission image channel images are acquired for each test unstained biological sample, wherein each of the one or more (such as two or more) test multispectral transmission image channel images is acquired at a specific wavelength.

[0252] In some embodiments, at least one (such as at least two) test multispectral transmission image channels are acquired for each unstained test biological sample, wherein a multispectral image acquisition device 12A is used to acquire each of the at least one (such as at least two) test multispectral transmission image channels, the multispectral image acquisition device being configured to illuminate each unstained test biological sample with at least one (such as at least two) different illumination sources; and being further configured to acquire transmission image data (e.g., at least two multispectral image channels) of the biological samples illuminated by the at least one (such as at least two) different illumination sources. In other words, at least one (such as at least two) test multispectral transmission image channels are acquired for each unstained test biological sample, wherein each of the at least one (such as at least two) test multispectral transmission image channels is acquired at a different wavelength. In some embodiments, four test multispectral transmission image channels are obtained from at least four different illumination sources. In some embodiments, five test multispectral transmission image channels are obtained from at least five different illumination sources. In some embodiments, test multispectral transmission image channels are obtained from at least six different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from at least seven different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from at least eight different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from at least nine different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from at least ten different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from at least eleven different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from at least twelve different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from twelve or more illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from sixteen or more different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from twenty or more different illumination sources. In some embodiments, test multispectral transmission image channel images are obtained from twenty-four or more different illumination sources.

[0253] In some embodiments, the acquired test multispectral image data is acquired based on: a trained virtual staining engine selected for generating the virtual staining slide, and specifically, the number of image channels used during training and the specific wavelength at which the image data is acquired. Therefore, the number of different test multispectral image channels to be acquired and the wavelengths of the different illumination sources that should illuminate the unstained biological sample to be tested depend on (i) the number of different training multispectral image channels used during the training of the specific virtual staining engine; and (ii) the wavelengths of the different illumination sources used during the training of the specific virtual staining engine (such as within + / - 15% nm, + / - 12% nm, + / - 10% nm, + / - 9% nm, + / - 8% nm, + / - 7% nm, + / - 6% nm, + / - 5% nm, + / - 4% nm, + / - 3% nm, + / - 2% nm, or + / - 1 nm).

[0254] For example, if a selected trained virtual staining engine for a specific morphological stain is trained using training multispectral image data from five different illumination sources, test multispectral image data is acquired at approximately the same five different narrowband illumination sources (such as within + / - 15% nm, + / - 12% nm, + / - 10% nm, + / - 9% nm, + / - 8% nm, + / - 7% nm, + / - 6% nm, + / - 5% nm, + / - 4% nm, + / - 3% nm, + / - 2% nm, or + / - 1 nm of the same five different illumination sources).

[0255] For another example, if a trained H&E virtual staining engine is selected (which is trained using 12 training multispectral image channels under 12 different illumination sources at 350 nm, 375 nm, 400 nm, 425 nm, 450 nm, 475 nm, 500 nm, 525 nm, 550 nm, 575 nm, 600 nm, and 625 nm), then the following settings should be applied: 350 nm + / - 10% nm, 375 nm + / - 10% nm, 400 nm + / - 10% nm, 425 nm + / - 10% nm, 450 nm + / - 10% nm, 475 nm + / - 10% nm, 500 nm + / - 10% nm, 525 nm + / - 10% nm, 550 nm + / - 10% nm, 575 nm + / - 10% nm, 600 nm + / - 10% nm. Multispectral images were acquired at 10% nm and 625 nm + / - 10% nm.

[0256] Referring to Figure 16, two or more test multispectral transmission image channels are obtained from any unstained biological sample being tested (step 170). Then, each of the two or more obtained test multispectral transmission image channels is combined to form a multi-channel multispectral training image (step 171). In some embodiments, the two or more test multispectral transmission image channels are combined into a multi-channel multispectral training image without image compression, e.g., without any dimensionality reduction techniques (e.g., PCA). In other embodiments, the two or more test multispectral transmission image channels are combined into a multi-channel multispectral training image by compressing the image using a dimensionality reduction technique (e.g., PCA).

[0257] In embodiments where four or more test multispectral transmission image channels are obtained, in some embodiments, the obtained four or more test multispectral transmission images are reduced or compressed into a multichannel multispectral test transmission image. In some embodiments, a dimension reduction method is used to compress the obtained four or more test multispectral transmission image channels to generate a multispectral training image. Examples of suitable dimension reduction methods include principal component analysis (PCA) (such as principal component analysis plus discriminant analysis), projection to latent structure regression and t-distributed random neighbor embedding (t-SNE), and uniform manifold approximation and projection (UMAP).

[0258] In some embodiments, the steps outlined in Figure 11 herein (which describes a method for compressing four or more training multispectral transmission image channels into a single multispectral training image) can be similarly applied to generate a 3-channel multispectral test transmission image. In other embodiments, the resulting four or more test multispectral transmission images are combined into a multichannel multispectral test transmission image without any compression techniques or any dimensionality reduction techniques (e.g., PCA).

[0259] The test unstained multispectral image data (e.g., multichannel multispectral test transmission images) is then fed to a selected trained virtual staining engine to provide a virtual stained image based on the input image of the test unstained biological sample. An example of a virtual stained image of an unstained biological sample is illustrated in Figure 1.

[0260] In some embodiments, the same multichannel multispectral test transmission image is supplied to multiple different trained virtual staining engines to provide multiple different virtual staining images of the same unstained biological sample.

[0261] Other system components

[0262] The system 200 disclosed herein can be coupled to a sample processing device capable of performing one or more preparation processes on the tissue sample. The preparation processes may include, but are not limited to, sample deparaffinization, sample conditioning (e.g., cell conditioning), sample staining, antigen retrieval, immunohistochemical staining (including labeling) or other reactions, and / or in situ hybridization (e.g., SISH, FISH, etc.) staining (including labeling) or other reactions, as well as other processes for preparing samples for microscopic examination, microanalysis, mass spectrometry, or other analytical methods.

[0263] Processing equipment can apply fixatives to samples. Fixatives can include cross-linking agents (e.g., aldehydes such as formaldehyde, polyoxymethylene, and glutaraldehyde, as well as non-aldehyde cross-linking agents), oxidizing agents (e.g., metal ions and complexes such as osmium tetroxide and chromic acid), protein denaturing agents (e.g., acetic acid, methanol, and ethanol), fixatives with unknown mechanisms (e.g., mercuric chloride, acetone, and picric acid), combination reagents (e.g., Carnoy fixative, Methacarn, Bouin solution, B5 fixative, Rossman solution, and Gendre solution), microwave fixatives, and other fixatives (e.g., exclusion volume fixation and vapor fixation).

[0264] If the sample is embedded in paraffin, it can be deparaffinized using one or more suitable deparaffinizing fluids. After deparaffin removal, any number of substances can be continuously applied to the sample. These substances can be used for pretreatment (e.g., reversing protein crosslinks, exposing cells to acids, etc.), denaturation, hybridization, washing (e.g., rigorous washing), detection (e.g., linking display or marker molecules to probes), amplification (e.g., amplifying proteins, genes, etc.), counterstaining, coverslips, etc.

[0265] Specimen processing equipment can apply a variety of different substances to samples. These substances include, but are not limited to, staining agents, probes, reagents, washing agents, and / or conditioning agents. These substances can be fluids (such as gases, liquids, or gas / liquid mixtures) or similar substances. The fluids can be solvents (such as polar solvents, nonpolar solvents, etc.), solutions (such as aqueous solutions or other types of solutions), or similar substances. Reagents can include, but are not limited to, staining agents, wetting agents, antibodies (such as monoclonal antibodies, polyclonal antibodies, etc.), antigen recovery solutions (such as water-based or non-water-based antigen retrieval solutions, antigen recovery buffers, etc.), or similar substances. Probes can be isolated cellular acids or isolated synthetic oligonucleotides attached to detectable tags or reporter molecules. Labels can include radioactive isotopes, enzyme substrates, cofactors, ligands, chemiluminescent or fluorescent agents, nanoparticles, haptens, and enzymes. As used herein, the term "fluid" means any liquid or liquid composition, including water, solvents, buffers, solutions (e.g., polar solvents, nonpolar solvents), and / or mixtures. Fluids can be aqueous or non-aqueous. Non-limiting examples of fluids include washing solutions, rinsing solutions, acidic solutions, alkaline solutions, transfer solutions, and hydrocarbons (e.g., alkanes, isoalkanes, and aromatic compounds such as xylene). In some embodiments, the washing solution contains a surfactant to facilitate the diffusion of the washing liquid on the sample-bearing surface of the slide. In some embodiments, the acidic solution includes deionized water, an acid (e.g., acetic acid), and a solvent. In some embodiments, the alkaline solution includes deionized water, an alkali, and a solvent. In some embodiments, the transfer solution comprises one or more glycol ethers, such as one or more propylene glycol ethers (e.g., propylene glycol ether, di(propylene glycol) ether and tri(propylene glycol) ether, ethylene glycol glycol ether (e.g., ethylene glycol ether, di(ethylene glycol) ether and tri(ethylene glycol) ether) and their functional analogs. Non-limiting examples of buffer solutions include citric acid, potassium dihydrogen phosphate, boric acid, diethylbarbituric acid, piperazine-N,N'-bis(2-ethanesulfonic acid), dimethylarsine, 2-(N-morpholino)ethanesulfonic acid, tris(hydroxymethyl)methylamine (TRIS), 2-(N-morpholino)ethanesulfonic acid (TAPS), N,N-bis(2-hydroxyethyl)glycine (Bicine), N-tris(hydroxymethyl)methylglycine (Tricine), 4-2-hydroxyethyl-1-piperazine ethanesulfonic acid (HEPES), and 2-{[tris(hydroxymethyl)methyl]amino}ethanesulfonic acid (TES). and combinations thereof. In some embodiments, the demasking agent is water. In other embodiments, the buffer may be composed of tris(hydroxymethyl)methylamine (TRIS), 2-(N-morpholino)ethanesulfonic acid (TAPS), N,N-bis(2-hydroxyethyl)glycine (Bicine), N-tris(hydroxymethyl)methylglycine (Tricine), 4-2-hydroxyethyl-1-piperazine ethanesulfonic acid (HEPES), 2-{[tris(hydroxymethyl)methyl]amino}ethanesulfonic acid (TES), or combinations thereof.Further washing solutions, transfer solutions, acid solutions, and alkaline solutions are described in U.S. Patent Application Publication No. 2016 / 0282374, the disclosure of which is incorporated herein by reference in its entirety.

[0266] Staining can be performed using a histochemical staining module or a separate platform, such as an automated IHC / ISH slide stainer. An automated IHC / ISH slide stainer typically includes at least: reservoirs for the various reagents used in the staining protocol; a reagent dispensing unit fluidly connected to the reservoirs for dispensing reagents onto the slides; a waste removal system for removing used reagents and other waste from the slides; and a control system for coordinating the actions of the reagent dispensing unit and the waste removal system. In addition to performing the staining steps, many automated slide stainers can also perform staining-assisted steps (or are compatible with separate systems that perform such steps), including: slide baking (for adhering the sample to the slide), dewaxing (also known as deparaffin removal), antigen retrieval, counterstaining, dehydration and removal, and coverslipping. The full text of this paper is incorporated by reference in Prichard, Overview of Automated Immunohistochemistry, Arch Pathol LabMed., Vol. 138, pp. 1578–1582 (2014), which describes several specific examples and various features of automated IHC / ISH slide stainers, including those from intelliPATH (Biocare Medical), WAVE (Celerus Diagnostics), DAKO OMNIS and DAKO AUTOSTAINER LINK 48 (Agilent Technologies), BENCHMARK (Ventana Medical Systems, Inc.), Leica BOND, and Lab VisionAutostainer (Thermo Scientific). In addition, Ventana Medical Systems, Inc. is the assignee of several U.S. patents disclosing systems and methods for performing automated analyses, including U.S. Patent Nos. 5,650,327, 5,654,200, 6,296,809, 6,352,861, 6,827,901, and 6,943,029, and U.S. Patent Application Publications Nos. 20030211630 and 20040052685, each of which is incorporated herein by reference in its entirety. As used herein, the term “reagent” means a solution or suspension comprising one or more reagents capable of covalently or non-covalently reacting, coupling, interacting, or hybridizing with another entity.Non-limiting examples of such reagents include solutions or suspensions of specific binding entities, antibodies (primary antibodies, secondary antibodies, or antibody conjugates), nucleic acid probes, oligonucleotide sequences, detection probes, chemical moieties with reactive or protected functional groups, enzymes, dyes, or staining agents.

[0267] Commercially available staining devices typically operate on one of the following principles: (1) open single-slide staining, in which the slide is positioned horizontally and reagents are dispensed as pools onto the surface of the slide containing the tissue sample (such as those implemented on the DAKO Automated Stainer Link 48 (Agilent Technologies) and the intelliPATH (Biocare Medical) stainer); (2) liquid overlay techniques, in which reagents are covered by or dispensed through an inert fluid layer deposited on the sample (such as those implemented on the Ventana BenchMark and DISCOVERY stainers (Roche)); (3) capillary staining, in which a slide surface is positioned adjacent to another surface (which may be another slide or cover plate) to form a narrow gap through which capillary forces draw liquid reagents and bring the liquid reagents into contact with the sample (such as the staining principles used in the DAKO TECHMATE, Leica BOND, and DAKO OMNIS stainers). Some iterations of capillary gap staining do not mix the fluid in the gap (e.g., on DAKO TECHMATE and LeicaBOND). In a variant of capillary gap staining called dynamic gap staining, the sample is applied to a slide using capillary forces, and then parallel surfaces are translated relative to each other to agitate the reagents during incubation to achieve reagent mixing (e.g., the staining principle implemented on the DAKO OMNIS slide stainer (Agilent)). In translational gap staining, a translational head is positioned on the slide. The lower surface of this head is spaced from the slide by a first gap small enough to allow the liquid on the slide to form a liquid meniscus during slide translation. A mixing extension, with a lateral dimension smaller than the width of the slide, extends from the lower surface of the translational head to define a second gap smaller than the first gap between the mixing extension and the slide. During head translation, the lateral dimension of the mixing extension is sufficient to generate lateral movement in the liquid on the slide in a direction substantially extending from the second gap to the first gap. See WO 2011-139978 A1. Recently, there have been proposals to use inkjet technology to deposit reagents on glass slides. See WO 2016-170008 A1. This list of staining techniques is not intended to be exhaustive, and any fully or semi-automated system used to perform biomarker staining can be integrated into a histochemical staining platform.

[0268] For cases requiring morphological staining, automated H&E staining platforms can be used. Automated systems for staining typically operate on one of two staining principles: batch staining (also known as "immersion") or individual slide staining. Batch stainers typically use a tank or trough in which many slides are simultaneously immersed. Individual slide stainers, on the other hand, apply reagent directly to each slide, and no two slides share the same amount of reagent. Examples of commercially available stainers include: the VENTANA SYMPHONY (individual slide stainer) and VENTANAHE 600 (individual slide stainer) series H&E stainers from Roche; the DakoCoverStainer (batch stainer) from Agilent Technologies; and the Leica ST4020 small linear stainer (batch stainer), Leica ST5020 multi-stage stainer (batch stainer), and Leica ST5010 automated stainer XL series (batch stainer) H&E stainers from Leica Biosystems Nussloch GmbH.

[0269] After the sample is stained, the stained sample can be manually analyzed on a microscope, and / or digital images of the stained sample can be acquired for archiving and / or digital analysis (e.g., with image acquisition device 12B). Digital images can be captured via a scanning platform (such as a slide scanner) capable of scanning stained slides at 20x, 40x, or other magnifications to produce high-resolution digital images of the entire slide. At a basic level, a typical slide scanner includes at least: (1) a microscope with lens objectives, (2) a light source (such as halogen, light-emitting diode, white light, and / or multispectral light sources, depending on the dye), (3) an automated device that can move the glass slide around or move optics around the slide, or both, (4) one or more digital cameras for image capture, and (5) a computer and associated software that controls the automated device and manipulates, manages, and views the digital slides. Digital data at multiple different XY locations (and in some cases, multiple Z planes) on the slide are captured by the camera's charge-coupled device (CCD), and the images are combined to form a composite image of the entire scanned surface. Common methods for achieving this include:

[0270] (1) A tiling-based scanning method, in which the slide stage or optics moves in very small increments to capture square image frames that slightly overlap with adjacent squares. The captured squares are then automatically matched together to construct a composite image; and

[0271] (2) Line-based scanning, in which the slide stage moves along a single axis during acquisition to capture multiple composite image “strips”. The image strips can then be matched together to form a larger composite image.

[0272] A detailed overview of the various scanners (both fluorescence and bright field) can be found in Farahani et al., Whole slideimaging in pathology: advantages, limitations, and emerging perspectives, Pathology and Laboratory Medicine Int'l, Vol. 7, pp. 23–33 (June 2015), the contents of which are incorporated herein by reference in their entirety. Examples of commercially available slide scanners include: 3D Histech PANNORAMIC SCAN II; DigiPath PATHSCOPE; Hamamatsu NANOZOOMER RS, HT, and XR; Huron TISSUESCOPE 4000, 4000XT, and HS; Leica SCANSCOPE AT, AT2, CS, FL, and SCN400; Mikroscan D2; Olympus VS120-SL; Omnyx VL4 and VL120; PerkinElmer LAMINA; Philips ULTRA-FAST SCANNER; Sakura Finetek VISIONTEK; Unic PRECICE 500 and PRECICE 600x; and Zeiss AXIO SCAN.Z1. In some embodiments, the scanning device is a digital pathology device, such as that disclosed in any of U.S. Patent No. 9,575,301; U.S. Patent Application Publication No. 2014 / 0178169; U.S. Patent No. 9,575,301; U.S. Patent Application Publication No. 2014 / 0178169; U.S. Patent Publication No. 2021 / 0092308; and / or U.S. Patent Application Publication No. 2021 / 0088769, the contents of each of which are incorporated herein by reference in their entirety.

[0273] Exemplary commercial image analysis software packages include: the VENTANA VIRTUOSO software suite (Ventana Medical Systems, Inc.); the TISSUE STUDIO, DEVELOPER XD, and IMAGE MINER software suite (Definiens); the BIOTOPIX, ONCOTOPIX, and STEREOTOPIX software suite (Visiopharm); and the HALO platform (Indica Labs, Inc.).

[0274] In some embodiments, any imaging may be performed using any system disclosed in U.S. Patent Nos. 10,317,666 and 10,313,606, the disclosure of which is incorporated herein by reference in its entirety. The imaging device may be a brightfield imager, such as the iScan Coreo™ brightfield scanner or the DP200 scanner sold by Ventana Medical Systems, Inc.

[0275] In some cases, images can be analyzed on an image analysis system. An image analysis system may include one or more computing devices, such as a desktop computer, laptop computer (notebook computer), tablet computer, smartphone, server, dedicated computing device, or any other electronic device of one or more other types capable of performing the techniques and operations described herein. In some embodiments, the image analysis system may be implemented as a single device. In other embodiments, the image analysis system may be implemented as a combination of two or more devices that together perform the various functions discussed herein. For example, an image analysis system may include one or more server computers and one or more client computers communicatively coupled to each other via one or more local area networks and / or wide area networks (such as the Internet). An image analysis system typically includes at least memory, a processor, and a display. Memory may include any combination of any type of volatile or non-volatile memory, such as random access memory (RAM), read-only memory (such as electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk drive, solid-state drive, optical disk, etc.). It is understood that memory may be contained in a single device or may be distributed across two or more devices. The processor may include one or more processors of any type, such as a central processing unit (CPU), graphics processing unit (GPU), dedicated signal or image processor, field-programmable gate array (FPGA), tensor processor (TPU), etc. It is understood that the processor may be contained in a single device or distributed across two or more devices. The display may be implemented using any suitable technology, such as LCD, LED, OLED, TFT, plasma, etc. In some implementations, the display may be a touch-sensitive display (touchscreen). The image analysis system also typically includes a software system stored in memory, comprising a set of instructions that can be implemented on the processor, covering various image analysis tasks such as object recognition, staining intensity quantization, etc. Indicative commercially available software packages that can be used to implement modules as disclosed herein include: VENTANA VIRTUOSO; Definiens TISSUE STUDIO, DEVELOPER XD, and IMAGEMINER; and Visiopharm BIOTOPIX, ONCOTOPIX, and STEREOTOPIX software packages.

[0276] After sample processing, the user can transport the specimen slide to an imaging device. In some embodiments, the imaging device is a brightfield imager slide scanner. One brightfield imager is the iScan Coreo brightfield scanner sold by Ventana Medical Systems, Inc. In automated embodiments, the imaging device is a digital pathology device disclosed in International Patent Application No. PCT / US2010 / 002772 entitled “IMAGING SYSTEM AND TECHNIQUES” (Patent Publication No.: WO / 2011 / 049608) or U.S. Patent Application No. 61 / 533,114 entitled “IMAGING SYSTEMS, CASSETTES, AND METHODS OF USING THE SAME”, filed September 9, 2011. The disclosures of International Patent Application No. PCT / US2010 / 002772 and U.S. Patent Application No. 61 / 533,114 are incorporated herein by reference in their entirety.

[0277] The embodiments of the subject matter and operation described herein can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and similar structures, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, for example as one or more computer program instruction modules that can be encoded on a computer storage medium for execution by or control of the operation of a data processing device. Any module described herein can include logic executed by the processor. As used herein, "logic" refers to information having any form of instruction signals and / or data that can affect the operation of the processor. Software is an example of logic.

[0278] Computer storage media may be or may be contained in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Furthermore, although a computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in an artificially generated propagating signal. The computer storage medium may also be or may be contained in one or more separate physical components or media (such as multiple CDs, disks, or other storage devices). The operations described herein can be implemented by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0279] The term "programmable processor" encompasses a wide range of devices, apparatuses, and machines that process data, including, for example, programmable microprocessors, computers, systems-on-a-chip, or a combination thereof. The devices may include special-purpose logic circuitry such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the devices may also include code that creates an execution environment for the relevant computer program, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or one or more combinations thereof. The devices and execution environments can implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0280] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs may, but do not necessarily, correspond to files in a file system. Programs can be stored in portions of files that hold other programs or data (such as one or more scripts stored in a markup language file), in a single file dedicated to the program, or in multiple coordinating files (such as files storing one or more modules, subroutines, or code portions). Computer programs can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0281] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform operations by processing input data and producing output results. The processes and logic flows can also be executed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as a special-purpose logic circuit.

[0282] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in a digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor that executes operations according to instructions and one or more storage devices that store instructions and data. Generally, a computer will also include or be operatively coupled to one or more mass storage devices (disk, magneto-optical, or optical disk) for storing data to receive data from or transfer data to, or both. However, a computer does not require such a device. Furthermore, a computer may embed another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive), etc. Suitable devices for storing computer program instructions and data include various forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by or incorporated into special-purpose logic circuitry.

[0283] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as an LCD (liquid crystal display), an LED (light-emitting diode) display, or an OLED (organic light-emitting diode) display, as well as a keyboard and pointing devices, such as a mouse or trackball, through which the user can input information into the computer. In some embodiments, a touchscreen can be used to display information and receive user input. Other types of devices can also be used to facilitate user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and user input can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving files from the device used by the user; for example, by sending a webpage to the web browser in response to a request received from the web browser on the user's client device.

[0284] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface or a web browser through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), interconnected networks (such as the Internet) and peer-to-peer networks (such as dedicated peer-to-peer networks).

[0285] The computing system may include any number of clients and servers. Typically, clients and servers are remotely configured and generally interact via a communication network. Relationships between clients and servers arise from computer programs running on their respective computers and the client-server relationships between them. In some embodiments, the server transmits data (such as HTML pages) to a client device (e.g., for displaying data to a user interacting with the client device and receiving the user's input). Data generated on the client device (such as the result of user interaction) can be received from the client device on the server.

[0286] Example

[0287] Example 1 – Using multispectral microscopy and deep learning for digital contrast on unstained tissue slides

[0288] Foreword

[0289] Histopathologists use chemical staining techniques on tissue samples to highlight microstructure and composition, thereby searching for abnormalities that indicate the presence, nature, and extent of pathology. Tissue processing steps are time-consuming and expensive. Each assay requires highly trained and scarce tissue technicians and generates chemical waste. Furthermore, tissue processing and staining are destructive, and cutting through multiple tissue sections can deplete valuable biopsy samples.

[0290] Emerging technologies for label-free visualization of tissue samples hold promise for improving histological practice in the near future. Virtual staining utilizes one or more computer algorithms to produce artificial staining effects without physical alteration of the slides. It uses optical means to scan tissue sections and deep learning techniques to transform digitized signals into rendered images that pathologists can use as diagnostic tools. Some advantages include reduced variability, process robustness, increased speed, biomarker reuse, reduced processing expertise, and the potential for novel medical biomarkers.

[0291] This project utilized a multispectral scanner 12A (see also Figure 5, which illustrates non-limiting components that can be included in any multispectral scanner). Following a dewaxing protocol, unstained multi-tissue array (MTA) sections were scanned to obtain endogenous contrast at 12 different wavelengths (365 nm, 400 nm, 435 nm, 470 nm, 500 nm, 550 nm, 580 nm, 635 nm, 660 nm, 690 nm, 780 nm, and 850 nm). Deep learning techniques were used to process and transform the multispectral images. It was possible to obtain digital H&E renderings assessed as diagnostically high quality by two certified pathologists.

[0292] method

[0293] This experiment was designed as a feasibility test using a multispectral microscope scanner originally designed for multiple assays. The aim was to test the possibility of converting the endogenous contrast of unstained tissue scanned at multiple wavelengths into a H&E-like assay that could be used for primary diagnosis.

[0294] Step 1 – Sample Preparation

[0295] Multi-tissue arrays (MTAs) containing liver, kidney, skin, colon, and tonsils were cut and mounted on glass slides. A modified HE600 protocol was used to dewax the slides to remove paraffin from the samples.

[0296] Step 2 – Multispectral Scan

[0297] Unstained and dewaxed samples were scanned using a FLASH multispectral microscope 12A and digitized using 12 different wavelengths. Sections were scanned without a coverslip mounter to reveal endogenous contrasts at different wavelengths.

[0298] Step 3 – Staining and Digitization

[0299] Following multispectral scanning, the slides were stained in HE600 and digitized using DP200 (both available from Ventana Medical Systems, Tucson, AZ). The resulting digital images were used as baseline truth in algorithm training.

[0300] Step 4 – Algorithm Training

[0301] A dimensionality reduction algorithm (principal component analysis) was used to compress the 12 channels into 3 channels, which were then encoded into an RGB image. The PCA-reduced and DP200 images were co-registered, segmented, and paired. The pix2pix algorithm was trained using Roche's high-performance computing cluster at Santa Clara, CA.

[0302] Step 5 – Model Testing

[0303] The model was tested for each tissue type, and the virtual renderings were stitched back together. The images were shown to two certified pathologists in the clinical development lab. Individual features were evaluated, and the feasibility of using images of the same quality for diagnostic purposes was investigated.

[0304] result

[0305] The results of the experiment are shown in Figures 17 and 18.

[0306] Proper registration of multispectral images with benchmark truth is one of the most important and challenging steps in training the algorithm. Despite this, good results were achieved on both the overfitted training and test sets. Better results were obtained for tonsils, liver, and kidneys; while the worst array quality was observed for intestines and skin.

[0307] discuss

[0308] Co-registering the PCA-reduced multispectral digitized RGB-encoded image with the bright-field image is a crucial step, particularly using the pix2pix algorithm. This algorithm works better when the entire image is segmented into 256 x 256 pixel pieces, where the spectral image and the "benchmark truth" are co-registered as closely as possible. Nevertheless, chemically stained tissue samples will not perfectly match unstained tissue samples due to stretching and other transformations during staining. Furthermore, the DP200 and FLASH have slightly different resolutions, complicating the co-registration of the two images. To reduce the difficulty of co-registration, the benchmark truth (chemically stained slide) is also scanned in the multispectral scanner, encoding wavelengths closer to red, green, and blue into RGB images, which are then converted to images similar to bright-field H&E, as described below.

[0309] in conclusion

[0310] Feasibility testing was conducted using a single MTA, which was scanned at 12 wavelengths (FLASH), then chemically stained, and scanned under a bright-field microscope (DP200). This experiment used an overfitted model (the same dataset was used for both training and testing). Testing on a larger dataset (using independent data for both training and testing) is needed to validate the results. Significantly better results were obtained with the liver, tonsils, and kidneys than with the intestines and skin, likely due to the presence of a certain amount of fat and heterogeneous morphology in the latter two tissue types. Tests using three arbitrarily chosen wavelengths instead of PCA reduction rendered images similar to those obtained with H&E staining, but digital illusions were quickly identified by pathologists during evaluation.

[0311] Example 2

[0312] The method begins with sample preparation, consisting of FFPE sections mounted on a standard glass microscope slide, as is typically performed in a normal histopathology workflow. The slide is then dewaxed using a standard procedure, either chemically (by bathing in xylene and other chemicals) or by applying heat to melt the paraffin. The next step is imaging the slide (without coverslip) on a device capable of mapping the intrinsic absorption spectrum of the tissue sample at a series of discrete wavelengths to obtain co-registered images potentially at multiple wavelengths in the UV to IR range. This device must have an array of sensors (i.e., CCD, CMOS) capable of acquiring images at different wavelengths, a device for focusing the image at each wavelength, and a device for illuminating the sample at specific spectral wavelengths and / or combinations of wavelengths.

[0313] The obtained image is then compressed into 3 channels (using dimensionality reduction methods such as PCA, tSNE, and UMAP) and encoded as an RGB image. Weights for each channel are controlled to enhance the target features in each channel. The image is then segmented into 256 x 256 pixels and transformed using a previously trained GAN algorithm to convert it into H&E-like digital colorants. After recoloring, the images are stitched together and optionally smoothed to mitigate tiling artifacts.

[0314] For training purposes, the same slides used to scan the multispectral images were stained using the selected staining scheme (i.e., H&E in HE600). The slides were covered with coverslips and then scanned using a brightfield scanner. The TrakEM2 ImageJ plugin was then used to co-register the digital images obtained from the scans with the multispectral images. Landmarks were manually specified between the target and the images to be registered using the ImageJ plugin. Affine transformations were then used to co-register the images based on these landmarks. Once co-registered, the two images were segmented into 256 x 256 pixel images and paired (multispectral and brightfield scans of H&E-stained tissue samples) and used to train the GAN algorithm in a supervised manner. This process was repeated for a series of slides until the algorithm learned most of the variations in the tissue samples. The trained algorithm could then be used to transform future tissue slice slides without chemical staining agents.

[0315] Example 3

[0316] Tissue samples were selected to demonstrate the diagnostic capability of the described method in generating H&E virtual staining for the diagnosis of cancerous lesions. FFPE blocks containing cancerous mastectomy samples were selected, sectioned using a microtome (3 μm to 5 μm thickness), and mounted on positively charged glass slides. The samples were baked in an HE600 oven for 5 minutes according to standard procedures on the HE600, and heat was used to remove paraffin to melt the wax and EZ preparation for rinsing.

[0317] Next, the sample is placed on a multispectral scanner platform. After the target area is selected / detected by the user, the sample is illuminated with a first wavelength (of multiple wavelengths) at a predefined illumination power (controlled by pulses with modulation circuitry to decrease / increase the duty cycle on the LED light source). The scanner objective is focused by moving the sample or the objective, and the first field of view (FOV) is digitized using a camera with a predefined exposure time and gain. The sample is then illuminated with a second wavelength. The following wavelengths were used in this experiment (365 nm, 400 nm, 435 nm, 470 nm, 500 nm, 550 nm, 580 nm, 635 nm, 660 nm, 690 nm, 780 nm, and 850 nm). However, due to scanning defects, two wavelengths (470 nm and 780 nm) were discarded (because they contained optical artifacts), resulting in only 10 wavelengths.

[0318] Only one center wavelength (550 nm) is focused (using a feedback-controlled autofocus algorithm), and all other wavelengths are digitized without moving the Z distance within the same FOV. This process is repeated until all FOVs are digitized with all different wavelengths. Next, the platform is moved to different locations near the previous position, with some overlapping areas. The FOVs are then stitched together to form a full-slide hypercube.

[0319] Once all AOIs have been scanned using the multispectral scanner, the next step is to stain the samples in the H&E according to the standard HE600 protocol. Once the slide is stained and covered with a coverslip, it is introduced into the platform of the same multispectral scanner. This process is repeated, but only at three wavelengths: red at 660 nm, green at 550 nm, and blue at 435 nm. Once the multispectral digital microscope has completed scanning all AOIs, the WSI of the raw data is stored on disk.

[0320] The WSI of the stained sample was then processed to create an RGB image by placing the image illuminated at 660 nm in the red channel, the image illuminated at 550 nm in the green channel, and the image illuminated at 435 nm in the blue channel.

[0321] The WSI of the stained sample is then processed to apply the previously calibrated color correction matrix to correct the color and to perform white balance in a manner that makes the image very similar to what the user will see using a bright-field microscope.

[0322] Next, principal component analysis was performed on the spectral hypercube image to reduce the number of channels from 10 to 3. The principal component values ​​were normalized to values ​​from 0 to 255 and encoded in the channels of the RGB image, with the first channel (i.e., red) being the first principal component, the second channel (i.e., green) being the second principal component, and the third channel (i.e., blue) being the third principal component.

[0323] Once the datasets are complete (unstained and stained), the next step is to co-register the images to the pixel level. For this, an automated algorithm is applied to transform the two images (unstained PCA and stained RGB, or baseline truth) by applying color filters (i.e., Laplacian, Sobel, etc.) to areas where the general morphology of the tissue is enhanced, thus causing the two color-filtered images to reveal similar structures (so-called descriptors). Next, a Fourier-based correlation algorithm is applied to translate and rotate the color-filtered image of the baseline truth to find locations and rotations that will produce higher correlation. Finally, the desired transformations (translation and rotation) are applied to the baseline truth, resulting in two coarsely registered images (unstained PCA and baseline truth).

[0324] If processing WSI, the next step is to find the FOV of the stitched images. To do this, a Laplacian color filter is applied to find discontinuities, thus revealing the stitching lines. The two images are then split into separate FOV images, and the above process is repeated: color filtering the paired images to reveal descriptors, and finding the optimal relevance by translating and rotating the color-filtered version of the FOV containing the baseline truth. Once the optimal position is found, a transformation is applied to the FOV without color filtering, and the image is cropped to discard the black areas created by rotating or shifting the baseline truth, resulting in two co-registered images (PCA and baseline truth). Finally, a final step is performed to further divide each registered FOV into 270 x 270 pixel tiles. This process is repeated at the tile level to create pixel-level registration tiles for 256 x 256 pixel images. All images are then saved to disk to train the GAN algorithm. The co-registration method also employs strict quality control to discard tiles that do not exceed the Fourier correlation threshold, as well as background tiles (white tiles that do not contain tissue information).

[0325] To test the algorithm, the hyperspectral cube of the same slice of WSI was divided into 256 x 256 pixel patches with 100-pixel overlap in both the x and y directions. PCA reduction was then performed using the same method described above to create an RGB image from the PCA components. In the test dataset, 40% of the patches were used for training, and 60% came from regions not used in the training dataset. The trained algorithm was used to convert the PCA-reduced spectral image into a virtual H&E with a bright field similar to the stained sample. The patches were then stitched together and blended using a linear blending algorithm to reduce stitching artifacts and produce a bright field image of the WSI. Figure 19 shows the resulting virtual stained whole slide image.

[0326] Example 4

[0327] Tissue samples were selected to demonstrate the diagnostic capability of the described method in generating Masson's trichrome virtual staining. FFPE blocks containing samples from colorectal resection (CRC) were selected, sectioned using a microtome (3 μm to 5 μm thickness), and mounted on TOMO glass slides. The samples were brought to a Benchmark special stainer and deparaffinized using the deparaffinization step selected in the standard trichrome staining protocol. This involved using Benchmark special staining deparaffin solution (catalog number 860-036 / 06523102001) and Benchmark special staining cleaner II (catalog number 860-041 / 08309817001). Next, the samples were placed on a multispectral scanner platform. After the target area was selected / detected by the user, the samples were illuminated with a predefined illumination power (controlled by pulses with modulation circuitry to decrease / increase the duty cycle on the LED light source) at the first wavelength (of multiple wavelengths). The scanner's objective lens is focused by moving the sample or the objective lens, and the first field of view (FOV) is digitized using a camera with a predefined exposure time and gain. The sample is then illuminated with a second wavelength. The following wavelengths were used in this experiment (365 nm, 400 nm, 435 nm, 470 nm, 500 nm, 550 nm, 580 nm, 635 nm, 660 nm, 690 nm, 780 nm, and 850 nm).

[0328] Only one center wavelength (635 nm) is focused (using a feedback-controlled autofocus algorithm), and all other wavelengths are digitized without moving the Z distance within the same FOV. This process is repeated until all FOVs are digitized with all different wavelengths. Next, the platform is moved to different locations near the previous position, some of which overlap. The FOVs are then stitched together to form a full-slide hypercube.

[0329] After completing all AOIs in the multispectral scan, the next step is to perform Masson trichrome staining on the sample using a BenchMark special stainer according to the following protocol: Apply special staining agent LCS and 300 μL of trichrome staining Bouins A and incubate for 32 min; wash the slide with special staining agent cleaner II; add special staining agent LCS and deposit 150 μL each of hematoxylin A and B and incubate for 12 min; wash the slide with special staining agent cleaner II; add special staining agent LCS and 300 μL of trichrome staining red agent and incubate for 8 min; rinse the slide with special staining agent cleaner II; add special staining agent LCS and deposit 150 μL of trichrome staining Mordant and incubate for 12 min; wash the slide with special staining agent cleaner II and apply another 100 μL of trichrome staining Mordant and incubate for 4 min. Add the special staining agent LCS, then add 300 μL of tricolor blue staining agent and incubate for 24 min. Wash the slide with special staining agent cleaner II, add the special staining agent LCS and 100 μL of tricolor clearing agent and incubate for 4 min. Rinse the slide one last time with special staining agent cleaner II.

[0330] Once the slide is stained and covered with a coverslip, it is introduced into the platform of the same multispectral scanner. The above process is repeated, but only at three wavelengths: red at 660 nm, green at 550 nm, and blue at 435 nm. In some embodiments, the FOV of the stained image location is closely matched to the FOV of the unstained sample using reference points located in the sample (i.e., edge tags, etc.), and this is stored on a disk that creates the same focus map used in the unstained sample. Once the multispectral digital microscope has completed scanning all target areas (AOIs), the WSI of the raw data is stored on the disk.

[0331] The WSI of the stained sample was then processed to create an RGB image by placing the image illuminated at 660 nm in the red channel, the image illuminated at 550 nm in the green channel, and the image illuminated at 435 nm in the blue channel.

[0332] In this example, PCA reduction was not used. Instead, the RGB image was registered to 12 wavelengths without further processing.

[0333] Once the datasets are complete (unstained and stained), the next step is to co-register the images to the pixel level. For this, an automatic algorithm transforms the two images (stained RGB or baseline truth) by applying color filters (i.e., Laplacian, Sobel, etc.) to areas where the general morphology of the tissue is enhanced, thus making the two filtered images reveal similar structures (so-called descriptors). Next, a Fourier-based correlation algorithm is applied to translate and rotate the baseline truth's filtered image to find locations and rotations that will produce higher correlation. Finally, the desired transformations (translation and rotation) are applied to the baseline truth, resulting in two coarsely registered images (unstained hypercube and stained RGB).

[0334] Next, the two WSIs are split into separate FOV images, and the above process is repeated: color-filtering the paired images to reveal descriptors, and finding the optimal relevance by translating and rotating the color-filtered version of the FOV containing the baseline truth. Once the optimal position is found, a transformation is applied to the FOV without color filtering, and the image is cropped to discard the black areas created by rotating or shifting the baseline truth, resulting in two co-registered images (PCA and baseline truth). Finally, a final step is completed to further divide each registered FOV into 270 x 270 pixel tiles. This process is repeated at the tile level to create pixel-level registration tiles for 256 x 256 pixel images. All images are then saved to disk to train the GAN algorithm.

[0335] To test the algorithm, hyperspectral cubes of WSI from different slices not used for training were divided into 256x256 pixel patches and fed into the GAN algorithm to convert the hyperspectral cubes into RGB cubes. The slides were then stitched together to form an image (see Figure 21).

[0336] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications mentioned in and / or listed in the application data sheets are incorporated herein by reference in their entirety. Modifications may be made to various aspects of the embodiments as necessary to provide further embodiments employing the concepts of various patents, applications, and publications.

[0337] Although this disclosure has been described with reference to some illustrative embodiments, it should be understood that those skilled in the art can devise many other modifications and embodiments within the spirit and scope of the principles of this disclosure. More specifically, reasonable variations and modifications can be made to the components and / or arrangements of the subject matter combination within the scope of the foregoing disclosure, the drawings, and the appended claims without departing from the spirit of this disclosure. In addition to variations and modifications in the components and / or arrangements, alternative uses will also be apparent to those skilled in the art.

[0338] Additional Examples

[0339] Additional Example 1. A system for generating a virtual stained image of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including:

[0340] a. Obtain test multispectral transmission images of unstained biological samples, wherein the test multispectral transmission images are derived from one or more test multispectral transmission image channel images;

[0341] b. The obtained test multispectral transmission image of the unstained biological sample is fed to a virtual staining engine, wherein the virtual staining engine is trained to generate images of the unstained biological sample stained with a morphological staining agent, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples.

[0342] i. The first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images;

[0343] ii. wherein the second training image in each pair of co-registered training images comprises training bright-field image data of the same biological sample stained with morphological staining agent; and

[0344] c. Use a trained virtual staining engine to generate virtual stained images of unstained test biological samples stained with morphological staining agents;

[0345] Each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels.

[0346] Additional Example 2. According to the system of Additional Embodiment 1, (i) at least two test multispectral transmission image channel images are acquired; and (ii) at least three training multispectral transmission image channel images are acquired; wherein each of the at least three test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least three training multispectral transmission image channel images.

[0347] Additional Example 3. According to the system of Additional Embodiment 1, (i) at least four test multispectral transmission image channels are acquired; and (ii) at least four training multispectral transmission image channels are acquired; wherein each of the at least four test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least four training multispectral transmission image channels.

[0348] Additional Example 4. According to the system of Additional Embodiment 3, a test multispectral transmission image is generated by dimensionality reduction of the at least four test multispectral transmission image channels.

[0349] Additional Example 5. According to the system of Additional Embodiment 3, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least four training multispectral transmission image channel images.

[0350] Additional Example 6. According to the system of Additional Embodiment 1, (i) at least six test multispectral transmission image channels are acquired; and (ii) at least six training multispectral transmission image channels are acquired; wherein each of the at least six test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least six training multispectral transmission image channels.

[0351] Additional Example 7. According to the system of Additional Embodiment 6, a test multispectral transmission image is generated by dimensionality reduction of the at least six test multispectral transmission image channels.

[0352] Additional Example 8. According to the system of Additional Embodiment 6, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least six training multispectral transmission image channel images.

[0353] Additional Example 9. According to the system of Additional Embodiment 1, (i) at least twelve test multispectral transmission image channels are acquired; and (ii) at least twelve training multispectral transmission image channels are acquired; wherein each of the at least twelve test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least twelve training multispectral transmission image channels.

[0354] Additional Example 10. According to the system of Additional Embodiment 9, a test multispectral transmission image is generated by dimensionality reduction of the at least twelve test multispectral transmission image channels.

[0355] Additional Example 11. According to the system of Additional Embodiment 9, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least twelve training multispectral transmission image channel images.

[0356] Additional Example 12. The system according to any one of the additional embodiments 1 to 11, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

[0357] Additional Example 13. According to any one of the additional embodiments 1 to 11, the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

[0358] Additional Example 14.The system according to any one of the additional embodiments 1 to 11, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0359] Additional Example 15. The system according to any one of Additional Embodiments 1 to 11, wherein the obtained virtual coloring engine includes a generative adversarial network.

[0360] Additional Example 16. The system according to any one of Additional Examples 1 to 11, wherein the morphological staining agent is a primary staining agent.

[0361] Additional Example 17. The system according to any one of Additional Examples 1 to 11, wherein the morphological staining agent is a special staining agent.

[0362] Additional Example 18. The system according to any one of Additional Examples 1 to 11, wherein the morphological staining agent comprises hematoxylin.

[0363] Additional Example 19. The system according to any one of Additional Examples 1 to 11, wherein the morphological staining agent comprises hematoxylin and eosin.

[0364] Additional Example 20. According to any one of the additional embodiments 1 to 19, the method uses a multispectral image acquisition device to acquire training brightfield data.

[0365] Additional Example 21. A method for generating a virtual stained image of a test unstained biological sample set on a substrate, the method comprising:

[0366] a. Obtain test multispectral transmission images of unstained biological samples, wherein the test multispectral transmission images are derived from one or more test multispectral transmission image channel images;

[0367] b. The obtained test multispectral transmission image of the unstained biological sample is fed to a virtual staining engine, wherein the virtual staining engine is trained to generate images of the unstained biological sample stained with a morphological staining agent, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples.

[0368] i. The first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images;

[0369] ii. wherein the second training image in each pair of co-registered training images comprises training bright-field image data of the same biological sample stained with morphological staining agent; and

[0370] c. Use a trained virtual staining engine to generate virtual stained images of unstained test biological samples stained with morphological staining agents;

[0371] Each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels.

[0372] Additional Example 22. According to the method of Additional Embodiment 21, (i) at least three test multispectral transmission image channel images are acquired; and (ii) at least three training multispectral transmission image channel images are acquired; wherein each of the at least three test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least three training multispectral transmission image channel images.

[0373] Additional Example 23. According to the method of Additional Embodiment 21, (i) at least four test multispectral transmission image channel images are acquired; and (ii) at least four training multispectral transmission image channel images are acquired; wherein each of the at least four test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least four training multispectral transmission image channel images.

[0374] Additional Example 24. According to the method of Additional Embodiment 23, the test multispectral transmission image is generated by dimensionality reduction of the at least four test multispectral transmission image channels.

[0375] Additional Example 25. According to the method of Additional Embodiment 24, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least four training multispectral transmission image channel images.

[0376] Additional Example 26.According to the method of Additional Embodiment 21, (i) at least six test multispectral transmission image channel images are acquired; and (ii) at least six training multispectral transmission image channel images are acquired; wherein each of the at least six test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least six training multispectral transmission image channel images.

[0377] Additional Example 27. According to the method of Additional Embodiment 26, the test multispectral transmission image is generated by dimensionality reduction of the at least six test multispectral transmission image channels.

[0378] Additional Example 28. According to the method of Additional Embodiment 26, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least six training multispectral transmission image channel images.

[0379] Additional Example 29. According to the method of Additional Embodiment 21, (i) at least twelve test multispectral transmission image channel images are acquired; and (ii) at least twelve training multispectral transmission image channel images are acquired; wherein each of the at least twelve test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least twelve training multispectral transmission image channel images.

[0380] Additional Example 30. According to the method of Additional Embodiment 29, the test multispectral transmission image is generated by dimensionality reduction of the at least twelve test multispectral transmission image channels.

[0381] Additional Example 31. According to the method of Additional Embodiment 29, the first training image in each pair of co-registered training images is generated by dimensionality reduction of the at least twelve training multispectral transmission image channel images.

[0382] Additional Example 32.The method according to any one of Additional Examples 21 to 31, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

[0383] Additional Example 33. The method according to any one of Additional Examples 21 to 31, wherein the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

[0384] Additional Example 34. The method according to any one of Additional Examples 21 to 31, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0385] Additional Example 35. The method according to any one of Additional Embodiments 21 to 30, wherein the obtained virtual coloring engine includes a generative adversarial network.

[0386] Additional Example 36. The method according to any one of Additional Examples 21 to 31, wherein the morphological staining agent is a primary staining agent.

[0387] Additional Example 37. The method according to any one of Additional Examples 21 to 31, wherein the morphological staining agent is a special staining agent.

[0388] Additional Example 38. The system according to any one of Additional Examples 21 to 31, wherein the morphological staining agent comprises hematoxylin.

[0389] Additional Example 39. The method according to any one of Additional Examples 21 to 31, wherein the morphological staining agent comprises hematoxylin and eosin.

[0390] Additional Example 40. The method according to any one of Additional Embodiments 21 to 39, wherein a multispectral image acquisition device is used to acquire training brightfield data.

[0391] Additional Example 41. According to the method of Additional Embodiment 40, multispectral image data is acquired at wavelengths of approximately 700 nm, 550 nm, and 470 nm; and the acquired multispectral image data is converted into RGB images.

[0392] Additional Example 42. A system for generating a virtual stained image of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including:

[0393] a. Obtain test multispectral transmission images of unstained biological samples, wherein the test multispectral transmission images are derived from one or more test multispectral transmission image channel images;

[0394] b. The obtained test multispectral transmission images of the unstained biological samples are fed to a virtual staining engine, which is trained to generate images of the unstained biological samples stained with morphological staining agents; and

[0395] c. Use a trained virtual staining engine to generate virtual stained images of unstained test biological samples stained with morphological staining agents.

[0396] Additional Example 43. According to the system of Additional Embodiment 42, the virtual staining engine is trained using: (a) one or more training multispectral transmission image channels of unstained training biological samples; and (b) training bright-field image data of the same unstained training biological samples stained with morphological staining agents.

[0397] Additional Example 44. According to the system described in Additional Embodiment 43, a multispectral image acquisition device is used to acquire training brightfield data.

[0398] Additional Example 45.According to the system of Additional Embodiment 42, each of the one or more test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channel images.

[0399] Additional Example 46. The system according to any one of Additional Embodiments 42 to 43, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

[0400] Additional Example 47. The system according to any one of Additional Embodiments 42 to 43, wherein the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

[0401] Additional Example 48. The system according to any one of Additional Embodiments 42 to 43, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0402] Additional Example 49. According to the system of Additional Embodiment 42, the virtual staining engine is trained from multiple co-registered training images derived from one or more training biological samples.

[0403] Additional Example 50.According to the system of Additional Embodiment 49, the first training image in each pair of co-registered training images of the plurality of co-registered training images is derived from one or more training multispectral transmission image channel images; and the second training image in each pair of co-registered training images of the plurality of co-registered training images contains training bright-field image data of the same biological sample stained with morphological staining agent.

[0404] Additional Example 51. According to the system of Additional Embodiment 50, each of the one or more test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channel images.

[0405] Additional Example 52. A system for generating two or more virtual stained images of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including:

[0406] a. Obtain test multispectral transmission images of unstained biological samples, wherein the test multispectral transmission images are derived from one or more test multispectral transmission image channel images;

[0407] b. The obtained test multispectral transmission images of the unstained biological samples are fed to two or more different virtual staining engines, each of which is trained to generate images of unstained biological samples stained with different morphological staining agents;

[0408] c. Generate two or more virtual staining images of unstained biological samples for testing using a trained virtual staining engine, wherein each of the generated virtual staining images is stained with a different morphological staining agent.

[0409] Additional Example 53. According to the system of Additional Embodiment 52, the two or more virtual staining engines are trained independently using: (a) one or more training multispectral transmission image channels of unstained training biological samples; and (b) bright-field training image data of the same unstained training biological samples stained with morphological staining agents.

[0410] Additional Example 54.According to the system of Additional Embodiment 52, each of the one or more test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channel images used to train each of the two or more virtual staining engines.

[0411] Additional Example 55. The system according to any one of Additional Embodiments 52 to 54, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

[0412] Additional Example 56. The system according to any one of Additional Embodiments 52 to 54, wherein the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

[0413] Additional Example 57. The system according to any one of Additional Embodiments 52 to 54, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0414] Additional Example 58. According to the system of Additional Embodiment 52, each of the two or more virtual staining engines is trained from multiple co-registered training images derived from one or more training biological samples.

[0415] Additional Example 59.According to the system of Additional Embodiment 58, the first training image in each pair of co-registered training images of the plurality of co-registered training images is derived from one or more training multispectral transmission image channel images; and the second training image in each pair of co-registered training images of the plurality of co-registered training images contains training bright-field image data of the same biological sample stained with morphological staining agent.

[0416] Additional Example 60. A system for generating a virtual stained image of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including:

[0417] a. Obtain a trained virtual staining engine, which is trained to generate images of unstained biological samples stained with morphological staining agents, wherein the trained virtual staining engine is trained from multiple pairs of co-registration training images.

[0418] i. In each pair of co-registered training images, the first training image is derived from training multispectral transmission image data of unstained training biological samples acquired at three or more different wavelengths.

[0419] ii. wherein the second training image in each pair of co-registered training images contains training bright-field image data of the same training biological sample stained with morphological staining agent;

[0420] b. Obtaining a test multispectral transmission image of the unstained biological sample, wherein the obtained test multispectral transmission image is derived from test multispectral transmission image data of the unstained biological sample illuminated by at least three different illumination sources; and

[0421] c. Based on the obtained test multispectral transmission images, generate virtual stained images of the unstained test biological samples stained with morphological staining agents using the obtained trained virtual staining engine.

[0422] Additional Example 61. The system according to Additional Example 60, wherein the morphological staining agent is a primary staining agent.

[0423] Additional Example 62. The system according to Additional Example 60, wherein the morphological staining agent is a special staining agent.

[0424] Additional Example 63.The system according to Additional Example 60, wherein the morphological staining agent comprises hematoxylin.

[0425] Additional Example 64. The system according to Additional Example 60, wherein the morphological staining agents include hematoxylin and eosin.

[0426] Additional Example 65. According to any one of the additional embodiments 60 to 64, the training multispectral transmission image data of unstained training biological samples are acquired at four or more different wavelengths.

[0427] Additional Example 66. According to the system of Additional Embodiment 60, the first training image is generated by reducing the dimensionality of the training multispectral transmission image data at the four or more different wavelengths.

[0428] Additional Example 67. The system according to Additional Example 66 uses principal component analysis to reduce dimensionality.

[0429] Additional Example 68. According to any one of the additional embodiments 60 to 64, the training multispectral transmission image data of unstained training biological samples are acquired at six or more different wavelengths.

[0430] Additional Example 69. According to the system of Additional Embodiment 68, the first training image is generated by subtracting the dimension of the training multispectral transmission image data at the six or more different wavelengths.

[0431] Additional Example 70. The system according to Additional Example 69 uses principal component analysis to reduce dimensionality.

[0432] Additional Example 71. According to any one of the additional embodiments 60 to 64, the training multispectral transmission image data of unstained training biological samples are acquired at twelve or more different wavelengths.

[0433] Additional Example 72. According to the system of Additional Embodiment 71, the first training image is generated by approximately reducing the dimension of the training multispectral transmission image data at the twelve or more different wavelengths.

[0434] Additional Example 73. The system according to Additional Example 72 uses principal component analysis to reduce dimensionality.

[0435] Additional Example 74.The system according to any one of the additional embodiments 60 to 73, wherein the at least three wavelengths are selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

[0436] Additional Example 75. The system according to any one of the additional embodiments 60 to 73, wherein the at least three wavelengths are selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

[0437] Additional Example 76. The system according to any one of the additional embodiments 60 to 73, wherein the at least three wavelengths are selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0438] Additional Example 77. The system according to any one of the additional embodiments 60 to 76, wherein a multispectral image acquisition device is used to acquire training multispectral transmission image data and test multispectral image data.

[0439] Additional Example 78. The system according to any one of the additional embodiments 60 to 76, wherein a bright-field image acquisition device is used to acquire training bright-field image data.

[0440] Additional Example 79. The system according to any one of the additional embodiments 60 to 76, wherein a multispectral image acquisition device is used to acquire training bright-field image data, wherein the training bright-field image data is an RGB image.

[0441] Additional Example 80. The system according to any one of the additional embodiments 60 to 76, wherein a multispectral image acquisition device is used to acquire training brightfield image data, wherein the multispectral image acquisition device is configured to capture image data at 700 nm + / - 10 mm, about 550 nm + / - 10 mm, and about 470 nm + / - 10 mm.

[0442] Additional Example 81. The system according to any one of the additional embodiments 60 to 80, wherein the obtained trained virtual coloring engine includes a generative adversarial network.

[0443] Additional Example 82. A method for generating a virtual stained image of a test unstained biological sample set on a substrate, the method comprising:

[0444] a. Obtain a trained virtual staining engine, which is trained to generate images of unstained biological samples stained with morphological staining agents, wherein the trained virtual staining engine is trained from multiple pairs of co-registration training images;

[0445] i. The first training image in each pair of co-registered training images is derived from training multispectral transmission image data of unstained training biological samples illuminated by at least three different illumination sources;

[0446] ii. wherein the second training image in each pair of co-registered training images contains training bright-field image data of the same training biological sample stained with morphological staining agent;

[0447] b. Obtaining a test multispectral transmission image of the unstained biological sample, wherein the obtained test multispectral transmission image is derived from test multispectral transmission image data of the unstained biological sample illuminated by at least three different illumination sources; and

[0448] c. Based on the obtained test multispectral transmission images, generate virtual stained images of the unstained test biological samples stained with morphological staining agents using the obtained trained virtual staining engine.

[0449] Additional Example 83. The method according to Additional Example 82, wherein the morphological staining agent is a primary staining agent.

[0450] Additional Example 84. The method according to Additional Example 82, wherein the morphological staining agent is a special staining agent.

[0451] Additional Example 85.The method according to Additional Example 82, wherein the morphological staining agent comprises hematoxylin.

[0452] Additional Example 86. The method according to Additional Example 82, wherein the morphological staining agent comprises hematoxylin and eosin.

[0453] Additional Example 87. According to the method of Additional Example 82, the unstained training biological samples and the unstained test biological samples are illuminated with at least four different lighting sources.

[0454] Additional Example 88. According to the method of Additional Example 82, the unstained training biological samples and the unstained test biological samples are illuminated with at least six different lighting sources.

[0455] Additional Example 89. According to the method of Additional Example 82, the unstained training biological samples and the unstained test biological samples are illuminated by at least twelve different light sources.

[0456] Additional Example 90. The method according to any one of Additional Examples 82 to 89, wherein at least three illumination sources are selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

[0457] Additional Example 91. According to any one of the additional embodiments 82 to 89, the method wherein at least three illumination sources are selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

[0458] Additional Example 92.The method according to any one of Additional Examples 82 to 89, wherein at least three illumination sources are selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

[0459] Additional Example 93. The method according to any one of Additional Embodiments 82 to 92, wherein a multispectral image acquisition device is used to acquire training multispectral transmission image data and test multispectral image data.

[0460] Additional Example 94. The method according to any one of Additional Embodiments 82 to 92, wherein a bright-field image acquisition device is used to acquire training bright-field image data.

[0461] Additional Example 95. The method according to any one of Additional Embodiments 82 to 92, wherein a multispectral image acquisition device is used to acquire training brightfield image data.

[0462] Additional Example 96. The method according to any one of Additional Embodiments 82 to 92, wherein a multispectral image acquisition device is used to acquire training brightfield image data, wherein the multispectral image acquisition device is configured to capture image data at 700 nm + / - 10 mm, about 550 nm + / - 10 mm, and about 470 nm + / - 10 mm.

[0463] Additional Example 97. The method according to any one of Additional Embodiments 82 to 96, wherein the obtained trained virtual coloring engine includes a generative adversarial network.

[0464] Additional Example 98. According to the method of Additional Embodiment 82, each pair of co-registered training images in the plurality of co-registered training images is derived from different training biological samples.

[0465] Additional Example 99. A non-transitory computer-readable medium storing instructions for generating a virtual stained image of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including:

[0466] a. Obtain test multispectral transmission images of unstained biological samples, wherein the test multispectral transmission images are derived from one or more test multispectral transmission image channel images;

[0467] b. The obtained test multispectral transmission image of the unstained biological sample is fed to a virtual staining engine, wherein the virtual staining engine is trained to generate images of the unstained biological sample stained with a morphological staining agent, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples.

[0468] i. The first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images;

[0469] ii. wherein the second training image in each pair of co-registered training images comprises training bright-field image data of the same biological sample stained with morphological staining agent; and

[0470] c. Use a trained virtual staining engine to generate virtual stained images of unstained test biological samples stained with morphological staining agents;

[0471] Each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels.

Claims

1. A system for generating virtual stained images of unstained biological test samples set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: a. Obtain a test multispectral transmission image of the unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; b. The obtained test multispectral transmission image of the test unstained biological sample is fed to a virtual staining engine, wherein the virtual staining engine is trained to generate images of unstained biological samples stained with morphological staining agents, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples. i. The first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images; ii. wherein the second training image in each pair of co-registered training images of the plurality of co-registered training images contains training bright-field image data of the same biological sample stained with the morphological staining agent; as well as c. Generate virtual staining images of the unstained test biological samples stained with the morphological staining agent using a trained virtual staining engine; Each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels.

2. The system according to claim 1, wherein at least two test multispectral transmission image channels are acquired.

3. The system according to claim 1, wherein three or more test multispectral transmission image channels are acquired.

4. The system of claim 3, wherein the test multispectral transmission image is generated by dimensionality reduction of the three or more test multispectral transmission image channels.

5. The system of claim 3, wherein the test multispectral transmission image is generated without compressing any of the three or more test multispectral transmission image channels or performing dimensionality reduction on any of the three or more test multispectral transmission image channels.

6. The system of claim 1, wherein (i) at least four test multispectral transmission image channels are acquired; and (ii) at least four training multispectral transmission image channels are acquired; wherein each of the at least six test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least six training multispectral transmission image channels.

7. The system of claim 6, wherein the test multispectral transmission image is generated without performing dimensionality reduction on the at least six test multispectral transmission image channels.

8. The system of claim 6, wherein the first training image in each pair of co-registered training images is generated without performing dimensionality reduction on the at least six training multispectral transmission image channels.

9. The system of claim 1, wherein (i) at least twelve test multispectral transmission image channels are acquired; and (ii) at least twelve training multispectral transmission image channels are acquired; wherein each of the at least twelve test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least twelve training multispectral transmission image channels.

10. The system of claim 9, wherein the test multispectral transmission image is generated without performing dimensionality reduction on the at least twelve test multispectral transmission image channels.

11. The system of claim 9, wherein the first training image in each pair of co-registered training images is generated without performing dimensionality reduction on the at least twelve training multispectral transmission image channels.

12. The system according to any one of claims 1 to 11, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

13. The system according to any one of claims 1 to 11, wherein the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

14. The system according to any one of claims 1 to 11, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

15. The system according to any one of claims 1 to 11, wherein the obtained virtual coloring engine comprises a generative adversarial network.

16. The system according to any one of claims 1 to 11, wherein the morphological staining agent is a primary staining agent.

17. The system according to any one of claims 1 to 11, wherein the morphological staining agent is a special staining agent.

18. The system according to any one of claims 1 to 11, wherein the morphological staining agent comprises hematoxylin.

19. The system according to any one of claims 1 to 11, wherein the morphological staining agent comprises hematoxylin and eosin.

20. The method according to any one of claims 1 to 19, wherein a multispectral image acquisition device is used to acquire training brightfield data.

21. A method for generating a virtual stained image of a test unstained biological sample set on a substrate, the method comprising: a. Obtain a test multispectral transmission image of the unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; b. The obtained test multispectral transmission image of the test unstained biological sample is fed to a virtual staining engine, wherein the virtual staining engine is trained to generate images of unstained biological samples stained with morphological staining agents, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples. i. The first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images; ii. wherein the second training image in each pair of co-registered training images of the plurality of co-registered training images contains training bright-field image data of the same biological sample stained with the morphological staining agent; as well as c. Generate virtual staining images of the unstained test biological samples stained with the morphological staining agent using a trained virtual staining engine; Each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels.

22. The method of claim 21, wherein at least two test multispectral transmission image channels are acquired.

23. The method of claim 21, wherein at least three test multispectral transmission image channels are acquired.

24. The method of claim 23, wherein the test multispectral transmission image is generated by dimensionality reduction of the at least three test multispectral transmission image channels.

25. The method of claim 24, wherein the test multispectral transmission image is generated without compressing any of the at least three test multispectral transmission image channels or performing dimensionality reduction on any of the at least three test multispectral transmission image channels.

26. The method of claim 21, wherein (i) at least six test multispectral transmission image channel images are acquired; and (ii) at least six training multispectral transmission image channel images are acquired; wherein each of the at least six test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the at least six training multispectral transmission image channel images.

27. The method of claim 26, wherein the test multispectral transmission image is generated without performing dimensionality reduction on the at least six test multispectral transmission image channels.

28. The method of claim 26, wherein the first training image in each pair of co-registered training images is generated without performing dimensionality reduction on the at least six training multispectral transmission image channels.

29. The method of claim 21, wherein (i) at least twelve test multispectral transmission image channels are acquired; and (ii) at least twelve training multispectral transmission image channels are acquired; wherein each of the at least twelve test multispectral transmission image channels is acquired at approximately the same wavelength as each of the at least twelve training multispectral transmission image channels.

30. The method of claim 29, wherein the test multispectral transmission image is generated without performing dimensionality reduction on the at least twelve test multispectral transmission image channels.

31. The method of claim 29, wherein the first training image in each pair of co-registered training images is generated without performing dimensionality reduction on the at least twelve training multispectral transmission image channels.

32. The method according to any one of claims 21 to 31, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

33. The method according to any one of claims 21 to 31, wherein the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

34. The method according to any one of claims 21 to 31, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

35. The method of any one of claims 21 to 30, wherein the obtained virtual coloring engine comprises a generative adversarial network.

36. The method according to any one of claims 21 to 31, wherein the morphological staining agent is a primary staining agent.

37. The method according to any one of claims 21 to 31, wherein the morphological staining agent is a special staining agent.

38. The system according to any one of claims 21 to 31, wherein the morphological staining agent comprises hematoxylin.

39. The method according to any one of claims 21 to 31, wherein the morphological staining agent comprises hematoxylin and eosin.

40. The method according to any one of claims 21 to 39, wherein a multispectral image acquisition device is used to acquire training brightfield data.

41. The method of claim 40, wherein the multispectral image data is acquired at wavelengths of approximately 700 nm, 550 nm, and 470 nm; and wherein the acquired multispectral image data is converted into an RGB image.

42. A system for generating a virtual stained image of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: a. Obtain a test multispectral transmission image of the unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; b. The obtained test multispectral transmission image of the unstained biological sample is fed to a virtual staining engine, wherein the virtual staining engine is trained to generate images of the unstained biological sample stained with morphological staining agents; as well as c. Generate virtual staining images of the unstained test biological samples stained with the morphological staining agent using a trained virtual staining engine.

43. The system of claim 42, wherein the virtual staining engine is trained using: (a) one or more training multispectral transmission image channels of unstained training biological samples; and (b) training bright-field image data of the same unstained training biological samples stained with morphological staining agents.

44. The system according to claim 43, wherein a multispectral image acquisition device is used to acquire training brightfield data.

45. The system of claim 42, wherein each of the one or more test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channel images.

46. ​​The system according to any one of claims 42 to 43, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

47. The system according to any one of claims 42 to 43, wherein the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

48. The system according to any one of claims 42 to 43, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

49. The system of claim 42, wherein the virtual staining engine is trained from multiple co-registered training images derived from one or more training biological samples.

50. The system of claim 49, wherein the first training image in each pair of co-registered training images of the plurality of co-registered training images originates from one or more training multispectral transmission image channel images; and wherein the second training image in each pair of co-registered training images of the plurality of co-registered training images comprises training bright-field image data of the same biological sample stained with the morphological staining agent.

51. The system of claim 50, wherein each of the one or more test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channel images.

52. A system for generating two or more virtual stained images of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: a. Obtain a test multispectral transmission image of the unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; b. The obtained test multispectral transmission image of the test unstained biological sample is supplied to two or more different virtual staining engines, each of which is trained to generate images of unstained biological samples stained with different staining agents; c. Generate two or more virtual staining images of the test unstained biological sample using a trained virtual staining engine, wherein each of the two or more virtual staining images is stained with a different staining agent.

53. The system of claim 52, wherein the two or more virtual staining engines are each trained independently using: (a) one or more training multispectral transmission image channels of unstained training biological samples; and (b) bright-field training image data of the same unstained training biological samples stained with a staining agent.

54. The system of claim 52, wherein each of the one or more test multispectral transmission image channel images is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channel images used to train each of the two or more virtual staining engines.

55. The system according to any one of claims 52 to 54, wherein the wavelength is selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

56. The system according to any one of claims 52 to 54, wherein the wavelength is selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

57. The system according to any one of claims 52 to 54, wherein the wavelength is selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

58. The system of claim 52, wherein each of the two or more virtual staining engines is trained from multiple co-registered training images derived from one or more training biological samples.

59. The system of claim 58, wherein the first training image in each pair of co-registered training images of the plurality of co-registered training images originates from one or more training multispectral transmission image channel images; and wherein the second training image in each pair of co-registered training images of the plurality of co-registered training images comprises training bright-field image data of the same biological sample stained with the staining agent.

60. A system for generating a virtual stained image of a test unstained biological sample set on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: a. Obtain a trained virtual staining engine, said trained virtual staining engine being trained to generate images of unstained biological samples stained with morphological staining agents, wherein said trained virtual staining engine is trained from multiple pairs of co-registered training images. i. In each pair of co-registered training images, the first training image is derived from training multispectral transmission image data of unstained training biological samples acquired at three or more different wavelengths. ii. The second training image in each pair of co-registered training images of the plurality of co-registered training images contains training bright-field image data of the same training biological sample stained with morphological staining agent; b. Obtain a test multispectral transmission image of the unstained biological sample, wherein the obtained test multispectral transmission image is derived from test multispectral transmission image data of the unstained biological sample illuminated by at least three different illumination sources; and c. Based on the obtained test multispectral transmission image, generate the virtual staining image of the test unstained biological sample stained with the morphological staining agent using the obtained trained virtual staining engine.

61. The system of claim 60, wherein the morphological staining agent is a primary staining agent.

62. The system of claim 60, wherein the morphological staining agent is a special staining agent.

63. The system of claim 60, wherein the morphological staining agent comprises hematoxylin.

64. The system of claim 60, wherein the morphological staining agent comprises hematoxylin and eosin.

65. The system according to any one of claims 60 to 64, wherein the training multispectral transmission image data of the unstained training biological sample is acquired at four or more different wavelengths.

66. The system of claim 60, wherein the first training image is generated by reducing the dimension of the training multispectral transmission image data at the three or more different wavelengths.

67. The system of claim 60, wherein the first training image is generated without compressing or reducing the dimensionality of the training multispectral transmission image data at the three or more different wavelengths.

68. The system according to any one of claims 60 to 64, wherein the training multispectral transmission image data of the unstained training biological sample is acquired at six or more different wavelengths.

69. The system of claim 68, wherein the first training image is generated by subtracting the dimension of the training multispectral transmission image data at the six or more different wavelengths.

70. The system of claim 68, wherein the first training image is generated without compressing or reducing the dimensionality of the training multispectral transmission image data at the six or more different wavelengths.

71. The system according to any one of claims 60 to 64, wherein the training multispectral transmission image data of the unstained training biological sample is acquired at twelve or more different wavelengths.

72. The system of claim 71, wherein the first training image is generated by subtracting the dimension of the training multispectral transmission image data at the twelve or more different wavelengths.

73. The system of claim 71, wherein the first training image is generated without compressing or reducing the dimensionality of the training multispectral transmission image data at the twelve or more different wavelengths.

74. The system according to any one of claims 60 to 73, wherein at least three wavelengths are selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

75. The system according to any one of claims 60 to 73, wherein at least three wavelengths are selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

76. The system according to any one of claims 60 to 73, wherein at least three wavelengths are selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

77. The system according to any one of claims 60 to 76, wherein a multispectral image acquisition device is used to acquire the training multispectral transmission image data and the test multispectral image data.

78. The system according to any one of claims 60 to 76, wherein a bright-field image acquisition device is used to acquire the training bright-field image data.

79. The system according to any one of claims 60 to 76, wherein a multispectral image acquisition device is used to acquire the training bright-field image data, wherein the training bright-field image data is an RGB image.

80. The system according to any one of claims 60 to 76, wherein a multispectral image acquisition device is used to acquire the training bright-field image data, wherein the multispectral image acquisition device is configured to capture image data at 700 nm + / - 10 mm, about 550 nm + / - 10 mm, and about 470 nm + / - 10 mm.

81. The system according to any one of claims 60 to 80, wherein the obtained trained virtual coloring engine comprises a generative adversarial network.

82. A method for generating a virtual stained image of a test unstained biological sample set on a substrate, the method comprising: a. Obtain a trained virtual staining engine, the trained virtual staining engine being trained to generate images of unstained biological samples stained with morphological staining agents, wherein the trained virtual staining engine is trained from multiple pairs of co-registered training images. i. The first training image in each pair of co-registered training images is derived from training multispectral transmission image data of unstained training biological samples illuminated by at least three different illumination sources; ii. The second training image in each pair of co-registered training images of the plurality of co-registered training images contains training bright-field image data of the same training biological sample stained with morphological staining agent; b. Obtain a test multispectral transmission image of the unstained biological sample, wherein the obtained test multispectral transmission image is derived from test multispectral transmission image data of the unstained biological sample illuminated by at least three different illumination sources; and c. Based on the obtained test multispectral transmission image, generate the virtual staining image of the test unstained biological sample stained with the morphological staining agent using the obtained trained virtual staining engine.

83. The method according to claim 82, wherein the morphological staining agent is a primary staining agent.

84. The method according to claim 82, wherein the morphological staining agent is a special staining agent.

85. The method of claim 82, wherein the morphological staining agent comprises hematoxylin.

86. The method of claim 82, wherein the morphological staining agent comprises hematoxylin and eosin.

87. The method of claim 82, wherein the unstained training biological sample and the unstained test biological sample are illuminated by at least four different light sources.

88. The method of claim 82, wherein the unstained training biological sample and the unstained test biological sample are illuminated by at least six different light sources.

89. The method of claim 82, wherein the unstained training biological sample and the unstained test biological sample are illuminated by at least twelve different light sources.

90. The method according to any one of claims 82 to 89, wherein at least three illumination sources are selected from 365 + / - 20 nm, about 400 + / - 20 nm, about 435 + / - 20 nm, about 470 + / - 20 nm, about 500 + / - 20 nm, about 550 + / - 20 nm, about 580 + / - 20 nm, about 635 + / - 20 nm, about 660 + / - 20 nm, about 690 + / - 20 nm, about 780 + / - 20 nm and / or about 850 + / - 20 nm.

91. The method according to any one of claims 82 to 89, wherein at least three illumination sources are selected from 365 + / - 10 nm, about 400 + / - 10 nm, about 435 + / - 10 nm, about 470 + / - 10 nm, about 500 + / - 10 nm, about 550 + / - 10 nm, about 580 + / - 10 nm, about 635 + / - 10 nm, about 660 + / - 10 nm, about 690 + / - 10 nm, about 780 + / - 10 nm and / or about 850 + / - 10 nm.

92. The method according to any one of claims 82 to 89, wherein at least three illumination sources are selected from 365 nm, about 400 nm, about 435 nm, about 470 nm, about 500 nm, about 550 nm, about 580 nm, about 635 nm, about 660 nm, about 690 nm, about 780 nm and / or about 850 nm.

93. The method according to any one of claims 82 to 92, wherein a multispectral image acquisition device is used to acquire the training multispectral transmission image data and the test multispectral image data.

94. The method according to any one of claims 82 to 92, wherein a bright-field image acquisition device is used to acquire the training bright-field image data.

95. The method according to any one of claims 82 to 92, wherein a multispectral image acquisition device is used to acquire the training bright-field image data.

96. The method according to any one of claims 82 to 92, wherein a multispectral image acquisition device is used to acquire the training bright-field image data, wherein the multispectral image acquisition device is configured to capture image data at 700 nm + / - 10 mm, about 550 nm + / - 10 mm, and about 470 nm + / - 10 mm.

97. The method according to any one of claims 82 to 96, wherein the obtained trained virtual coloring engine comprises a generative adversarial network.

98. The method of claim 82, wherein each pair of co-registered training images in the plurality of co-registered training images is derived from different training biological samples.

99. A non-transitory computer-readable medium storing instructions for generating a virtual stained image of a test unstained biological sample disposed on a substrate, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: a. Obtain a test multispectral transmission image of the unstained biological sample, wherein the test multispectral transmission image is derived from one or more test multispectral transmission image channel images; b. The obtained test multispectral transmission image of the test unstained biological sample is fed to a virtual staining engine, wherein the virtual staining engine is trained to generate images of unstained biological samples stained with morphological staining agents, and wherein the virtual staining engine is trained from multiple pairs of co-registered training images derived from one or more training biological samples. i. The first training image in each pair of co-registered training images is derived from one or more training multispectral transmission image channel images; ii. wherein the second training image in each pair of co-registered training images of the plurality of co-registered training images contains training bright-field image data of the same biological sample stained with the morphological staining agent; as well as c. Generate virtual staining images of the unstained test biological samples stained with the morphological staining agent using a trained virtual staining engine; Each of the one or more test multispectral transmission image channels is acquired at approximately the same wavelength as each of the one or more training multispectral transmission image channels.

Images