Multi-Spectral Imaging Method and System Executed by a Computer

Abstract

Disclosed is a computer-executed multi-spectral imaging method for use in the analysis of a sample comprising a plurality of types of fluorescent labels, each of the plurality of types of fluorescent labels having its own emission spectrum. The method includes receiving multi-channel image data, where each channel in the multi-channel image data is derived from an unfiltered image of the sample and includes image data having respective spectral components; for each channel, i) forming a vector of photon counts measured from the image data for the channel, the vector having an entry for each pixel in the image; ii) repeatedly generating a vector of possible values having an entry for the contribution made by each of the plurality of types of fluorescent labels to the unfiltered image for each pixel in the image, and for each iteration, calculating a vector of expected photon counts having an entry for each pixel in the image by multiplying a mixing matrix that defines the relationship between the unfiltered image and the multi-channel image data by the vector of possible values for each pixel; iii) selecting a vector of possible values for which the negative log-likelihood function describes the probability of generating the measured vector of photon counts assuming that the corresponding vector of expected photon counts is minimized; and constructing a corresponding data structure containing the image data for each of the plurality of types of fluorescent labels in the sample, where in the image data, for each pixel, the data structure includes an entry for the contribution made by that type of fluorescent label from the vector of possible values for the pixel, whereby each data structure can be used to reconstruct an image of the sample using the spectral components corresponding to the respective emission spectra of the type of fluorescent label for which the data structure was constructed.

Description

Technical Field

[0001] The present invention relates to a multi - spectral imaging method executed by a computer for use in the analysis of samples containing multiple types of fluorescent labels. The present invention also relates to corresponding systems and computer - readable media.

Background Art

[0002] Fluorescence microscopy is a fundamental tool for studying biological processes. Biological structures are labeled with either a genetically - engineered fluorescent protein or a fluorescent dye or probe. These types of fluorescent labels are commonly referred to as "fluorophores". These biological structures can be imaged using a fluorescence microscope. Each fluorophore has a characteristic excitation spectrum and emission spectrum. The emission spectrum defines the wavelengths of light emitted by the fluorophore, while the excitation spectrum is the range of wavelengths that excite the fluorophore and cause it to emit light over its emission spectrum.

[0003] In a fixed sample, biological targets are labeled with fluorophores to reveal the structural and spatial organization within the sample that can be subsequently observed on a particular fluorescence microscope suitable for the observer's needs. Multiple different biological targets can be labeled with different fluorophores to reveal the spatial distribution and relationships between multiple biological structures.

[0004] For each of the selected phosphors, emission filters are used within the microscope to ensure that the signal observed at any given time originates only from the phosphor that matches the currently selected emission filter. The number of emission filters used during the experiment is equal to the number of phosphors being observed. One phosphor is excited and the emitted light travels through the corresponding emission filter to the detector. Then, the emission filter is selected, a second phosphor is excited, and the emitted light from the phosphor travels through the newly selected emission filter to the detector. This process continues for each of the phosphors present in the sample. However, the use of this sequential process means that the capture speed is a function of the number of phosphors being observed. Moreover, the requirements for using mechanical filters to switch for selecting each filter have a significant impact on the capture speed.

[0005] Furthermore, when there are four or more phosphors in a single sample, it is not possible to use this sequential process technique. This is because the excitation of one phosphor can potentially cause non-specific excitation of another phosphor, and because some of the light emitted by one phosphor can potentially pass through the emission filter corresponding to another phosphor, a situation known as "spectral bleed-through".

[0006] To image four or more phosphors, a technique known as multispectral imaging, which relies on discretizing the emission spectra of the phosphors into multiple spectral wavelength bands, must be employed. In this way, more information is collected about the emitted photons, and a mathematical process can be inductively applied to reveal the underlying labeled structures within the sample, a process called "spectral unmixing".

[0007] There are commercial technologies for multispectral imaging in microscopy. Perhaps the most widely used one is the technology of the Zeiss Quasar detector. By employing a linear array of detection channels, multiple emission bands are imaged in parallel, thereby enabling the selected spectral regions to be acquired over the sample in a single scan. In this implementation, a diffraction grating disperses the emitted fluorescence, which is then guided onto an array of precisely defined bandwidth channels (either 9.7 nanometers or 10.7 nanometers channels in a 32-channel detector) in a special multi-anode photomultiplier tube to generate separate images for each channel. This technology is only compatible with conventional point-scanning confocal microscopy, which has limited applications for some biological studies, such as live imaging of samples.

[0008] Another example of a commercial application of multispectral imaging is the application of the Leica SP8 FALCON, which uses a prism to spectrally separate the emitted light onto various detectors. The excitation of the phosphors used within that system can also be modulated via a white light excitation source with an adjustable excitation wavelength that uses an acoustooptic tuneable filter. Additionally, the fluorescence signal can be time-gated by measuring the time it takes for the emitted photons to reach the detector, or the "fluorescence lifetime" of the labels used in the experiment. However, again, this method is only compatible with slow point-scanning confocal microscopy, which has limited applications for live cell imaging as described above.

[0009] There is a commercial gap for multi - spectral imaging of biological samples that is compatible with optimized live - imaging microscopy methods (such as light - sheet microscopy). Multi - spectral imaging of biological samples has been demonstrated in a study set by Valm et al., Nature, June 1, 2017, 546(7656): 162 - 167. They were able to spectrally decompose the light emitted from six fluorophores within a biological sample using lattice light - sheet microscopy combined with an excitation - based multi - spectral imaging approach. A single image was taken at multiple excitation wavelengths, and this data was subsequently inductively spectrally separated. This was a basic application example of multi - spectral imaging. However, since the spectral data was captured sequentially, it was quite slow, limiting its application examples, for instance, for observing fast biological dynamics within cells.

[0010] Live imaging using fluorescence microscopy enables the exploration of the spatio - temporal characteristics of various important biological processes since the sample is kept alive while being observed under the microscope. An important advancement in live imaging uses light - sheet microscopy rather than slow point - scanning confocal microscopy. Using this technique, the entire illuminated area can be immediately imaged, much more rapidly and gently on the sample. When attempting to observe multiple fluorophores within a sample, the process described above can be used, and the fluorophores can be imaged sequentially and independently of the same constraints regarding the number of observable fluorophores within the time.

[0011] Multi - spectral imaging in biological samples can be applied to light - sheet microscopy but is more difficult for three reasons. First, most multi - spectral imaging methods are very slow because they sequentially collect each spectral wavelength band, significantly lengthening the capture time. This directly affects sample viability and leads to phototoxicity and death.

[0012] Second, existing multispectral imaging techniques are designed for point-scanning microscopes such as confocal microscopes, meaning that the advantages of light-sheet microscopy for live imaging cannot be realized.

[0013] Third, spectral separation using conventional algorithms is highly dependent on Poisson noise, which is an inherent aspect of fluorescence imaging. This is rarely a problem for fixed samples where viability is not a consideration and laser power and / or exposure time can be increased to improve the signal-to-noise ratio. However, in live imaging, it is important to reduce the sample's exposure to light (in terms of intensity and time), meaning that the resulting images can have higher noise components. There is a trade-off between keeping the sample alive and collecting sufficient signal.

Prior Art Documents

Non-Patent Documents

[0014]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0015] An example of a spectral separation technique has been investigated by Neher et al. in "Blind Source Separation Techniques for the Decomposition of Multiply Labeled Fluorescence Images", Biophysical Journal, Vol. 96, May 2009, 3791 - 3800. However, the techniques described in this literature have poor computational efficiency and do not enable the high - speed processing required for live imaging.

[0016] Another example is provided by T. Zimmerman in "Spectral Imaging and Linear Unmixing in Light Microscopy", Advances in Biochemical Engineering, Vol. 95. However, this is affected by the above - mentioned problem of not considering the influence of Poisson noise in spectral separation.

Means for Solving the Problem

[0017] According to a first aspect of the present invention, there is provided a multi - spectral imaging method executed by a computer for use in analyzing a sample containing a plurality of types of fluorescent labels, each of the plurality of types of fluorescent labels having a respective emission spectrum, the method comprising: Receiving multi - channel image data, wherein each channel in the multi - channel image data is derived from an unfiltered image of the sample and contains image data having respective spectral content; For each channel, i) Forming a vector of the number of quantum particles measured from the image data for the channel, the vector having entries for each pixel in the image; ii) For each pixel in the image, repeatedly generate a vector of possible values having entries for the contribution made by each of a plurality of types of fluorescent labels to the unfiltered image, and for each iteration, multiply a mixing matrix that defines the relationship between the unfiltered image and the multi-channel image data by the vector of possible values for each pixel to calculate a vector of expected quantum particle numbers having entries for each pixel in the image; iii) Selecting a vector of possible values, for which the negative log-likelihood function describes the probability that the measured vector of quantum particle numbers is generated on the premise that the corresponding vector of expected quantum particle numbers is minimized; Constructing a corresponding data structure containing the image data for each of a plurality of types of fluorescent labels in the sample, wherein in the image data, for each pixel, the data structure includes an entry for the contribution made by that type of fluorescent label from the vector of possible values for the pixel, whereby each data structure is usable to reconstruct an image of the sample using spectral components corresponding to the emission spectrum of each of the types of fluorescent labels for which the data structure was constructed; including.

[0018] The present invention solves the above problems. The present invention simultaneously collects light emitted by many fluorescent labels, resulting in a short capture time. The capture time is no longer a function of the number of phosphors labeled in the sample, meaning that more information can be collected and a more excellent insight into biological processes can be revealed without extending the capture time.

[0019] The present invention is compatible with any form of microscopy, such as light sheet microscopy. Moreover, by selecting vectors of possible values, for which the negative log-likelihood function gives that the corresponding vectors of the expected number of quantum particles are minimal, the present invention can address spectral separation within samples where there is a large contribution of Poisson noise (i.e., a low signal-to-noise ratio), by selecting vectors of possible values that describe the probability that a vector of measured numbers of quantum particles is generated.

[0020] The system can process light simultaneously collected from many fluorescent labels, but it should also be noted that the system is beneficial even when used with only one fluorescent label. Since no emission filter is used, a higher percentage of the emission spectrum of the fluorescent label is collected and the signal-to-noise ratio is improved. This has important implications for live imaging as the laser power and exposure time can be reduced and the survival rate of the sample can be improved.

[0021] In a preferred embodiment, the quantum particles in the measured number of quantum particles and the expected number of quantum particles are photons.

[0022] In other embodiments, the quantum particles can be electrons. This can occur, for example, when the present invention is used in the context of electron microscopy. In this case, it should be understood that the reference to spectral components is a reference to ranges of electron energy.

[0023] The number of channels in the multi-channel image data may be greater than 3. The typical number of channels in the multi-channel image data is 8, but larger numbers such as 16, 32 or 64 are possible. The number of channels may be equal to the number of types of fluorescent labels within the plurality of fluorescent labels. However, in other circumstances, the number of channels may be less than or greater than the number of types of fluorescent labels within the plurality of fluorescent labels.

[0024] The method may further include reconstructing an image of the sample using one or more of the data structures containing the image data.

[0025] The iterative generation of step (ii) and the selection of step (iii) may be performed using an optimization algorithm such as limited memory Broyden-Fletcher-Goldfarb-Shanno, conjugate gradient descent, Adam, or Richardson-Lucy.

[0026] In a preferred embodiment, the optimization algorithm is

number

[0027] The predetermined number of successive iterations can be any number from 2 onwards, with common numbers being 10, 20, 50 or 100. The threshold amount can be a fixed amount or it can be determined by the k+1 It may be a ratio of the value of

[0028] In addition to the relationship between the mixed matrix and the unfiltered image and the multi-channel image data, the mixed matrix can define a function for removing blurring caused by diffraction or other optical effects. Essentially, the mixed matrix required to reverse the effects of spectral mixing can be modified to generate a composite matrix that reverses both spectral mixing and blurring or any other undesirable linear image formation process.

[0029] The method may further include the step of generating a mixed matrix.

[0030] The method is a step of generating multi-channel image data by splitting the light received from an image source that creates an unfiltered image into a plurality of optical paths, where the number of optical paths is equal to the number of channels in the multi-channel image data, and each optical path has the same spectral components as the corresponding one of the channels, a step of forming an image from the light in each optical path, and a step of generating image data for the channel corresponding to the optical path from the formed image.

[0031] The light received from the image source is generally split into optical paths using an array of dichroic mirrors, and the image in each optical path is formed in each camera that generates the image data.

[0032] In another aspect of the present invention, a multispectral imaging system for use in analyzing a sample containing a plurality of types of fluorescent labels is provided, each of the plurality of types of fluorescent labels having a respective emission spectrum, the system comprising at least one processor coupled to at least one memory device, the memory device storing instructions that, when executed, cause the processor to execute the method of the first aspect of the present invention.

[0033] The system is an array of dichroic mirrors for splitting light received from an image source that creates an unfiltered image into a plurality of optical paths, the number of optical paths being equal to the number of channels in the multi-channel image data, each optical path having the same spectral components as a corresponding one of the channels, and the array and a camera in each optical path formed to generate image data for the channel corresponding to the optical path.

[0034] In yet another aspect of the present invention, there is provided a computer-readable medium storing instructions executed by a processor to form part of a multi-spectral imaging system for use in analyzing a sample containing a plurality of types of fluorescent labels, each of the plurality of types of fluorescent labels having a respective emission spectrum, the instructions causing the processor to execute the method of the first aspect of the present invention when executed in the processor.

[0035] Next, embodiments of the present invention will be described with reference to the accompanying drawings.

Brief Description of the Drawings

[0036]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6a

Figure 6b

Figure 6c

Figure 7

Figure 8a

Figure 8b

Figure 9a

Figure 9b

Figure 9c

Figure 9d

Figure 9e

Figure 9f

Modes for Carrying Out the Invention

[0037] Figure 1 shows a block diagram of a system suitable for use in analyzing a sample containing multiple types of fluorescent labels. In this system, a microscope 1 is optically coupled to an optical splitter 3 by an optical link 2. The microscope 1 can be any microscope having an appropriate optical output. For example, a microscope having an optical output for coupling to a camera for capturing an image from the microscope is generally appropriate. The nature of the optical link 2 depends on the nature of the optical output of the microscope, but in many cases, a simple condenser lens can be used to collimate the optical output from the microscope 1 for further optical processing by the optical splitter 3.

[0038] The structural arrangement of the optical splitter 3 will be described later with reference to FIG. 3. For the moment, the functional nature of the optical splitter 3 is described. The optical splitter 3 divides the light received from the optical link 2 into eight separate channels, each having its own bandwidth. For example, if the total emission spectrum received from the microscope via the optical link 2 is in the range of 450 nm to 650 nm, each channel can cover a respective 25 nm portion of this 200 nm range.

[0039] Figure 2 shows the spectral emissions of each of a set of eight different fluorophores known as EGYP, mCherry, mNeptune, mTFP1, EYFP, mKate2, mOrange, and tdTomato. As shown, despite the fact that the emission spectra of each fluorophore are relatively narrow, they overlap considerably. Therefore, the function of the optical splitter 3 is not sufficient to separate the light emitted from each fluorophore into one of each of the channels. Instead, each fluorophore provides a contribution to each of the channels. This contribution can be easily measured or predicted if knowledge is given about the bandwidth of the channel and the optical output of each fluorophore within this bandwidth.

[0040] Each channel focuses on the respective spectral part of light in the cameras shown as 4a - 4h in FIG. 1. Cameras 4a - 4h are coupled to a data capture interface 5. The data capture interface 5 is provided (e.g., in the form of a PCI card) within a computer 6 that executes software to process the captured data, as described below. In other embodiments, the data capture interface 5 can be external to the computer 6. The computer 6 can be any general - purpose personal computer. The data capture interface 5 includes a set of analog - to - digital converters that receive image data from the cameras 4a - 4h and convert it into a digital representation of the images captured by each camera 4a - 4h. The digital representation gives a measurement for each pixel that is proportional to the number of photons that have fallen on that pixel. Then, the numbers of photons are combined into vectors, and each channel from the splitter 3 can have a respective vector of the number of photons at each pixel. Thus, a vector of the measured number of photons for each pixel within each channel can be formed from the images captured by the cameras 4a - 4h.

[0041] The structure of the splitter 3 is shown in FIG. 3. The above - mentioned condenser lens 10 receives light having a full spectral bandwidth of 450 nm to 650 nm from the output of the microscope 1 and collimates the light. A set of seven long - pass dichroic mirrors D1 - D7 is responsible for splitting the light into eight channels by wavelength. Each of the mirrors D1 - D7 has a different cut - off wavelength and splits the light received by the mirror into eight separate channels in a wavelength - dependent manner, with each channel occupying a respective 25 - nm portion of the full spectral bandwidth. The light within each channel passes through three of the mirrors D1 - D7 before being focused onto one of the cameras 4a - 4h by the respective tube lenses 11a - 11h. This enables the simultaneous capture of eight different channels, which is eight times faster than using sequential capture techniques.

[0042] There is a trade-off between the number of channels and the channel bandwidth. If there are too many channels, the channel bandwidth decreases, and fewer photons are collected within each channel, so the signal-to-noise ratio increases. If there are too few channels, the spectral separation is not sufficient to enable effective spectral unmixing. The use of eight channels represents a good practical compromise in this trade-off.

[0043] The installation of dichroic mirrors D1 to D7 affects the spatial resolution of the splitter 3, similar to affecting the distance between the object plane and the image planes generated by each of the lenses 11a to 11h. However, it is easy to optimize the installation of these components in order to obtain appropriate image quality by modeling the point spread function using a suitable software tool such as Zemax OpticStudio. The lenses 11a to 11h visualize all of the visible spectrum and therefore there is no need to consider chromatic aberration.

[0044] As previously explained, the multi-channel image data provided by the optical splitter 3 in association with the data capture interface 5 is used to form a vector of the measured number of photons for each channel, and each of the vectors has an entry for each pixel in the image represented by the image data.

[0045] Subsequently, the processing of the data continues as shown in FIGS. 4 and 5. These will be described below. However, first, it is beneficial to explain some of the theoretical background for the processing techniques shown in FIGS. 4 and 5.

[0046] The expected light intensity y at a particular pixel j within a particular channel i ij is the sum of the contributions from different fluorescent labels k. These labels have their concentrations x at pixel jkj contributes to the light intensity in proportion to and radiation a that falls within the spectral range of channel i ik contributes to the proportion of.

Number

[0047] However, the expected value of the light intensity is hardly actually measured. This is because the actual number of detected photons is distributed according to a Poisson distribution parameterized by this expected value (due to the shot noise inherent in the quantized emission). Therefore, the probability of measuring N photons is

Number

[0048] As described above, conventional spectral separation algorithms ignore this shot noise behavior and instead assume that the detected signals are equal to the expected values (i.e., they function at the limit of a high signal-to-noise ratio where the contribution from shot noise can be ignored). In this approach, the intrinsic phosphor concentration is

Number

Number

[0049] In the present invention, the spectral separation scenario is instead formulated as an optimization of a negative log-likelihood function constructed using the knowledge that the data is affected by Poisson noise. In particular, given K pixels, as well as a vector E of the expected number of photons per pixel and a vector m of the measured number of photons per pixel, the likelihood of the expected number of photons generating the measured number of photons is

Equation

[0050] Taking the logarithm and negating this gives the negative log-likelihood function

Equation

[0051] Given some inherent assumption about the contribution made by each of the multiple types of fluorescent labels to the light emitted from the sample, if we have a method for calculating the vector of expected number of photons, we can find the most likely assumption that created the measured number of photons by minimizing this function. This minimization process can be easily performed by generating random values for the contribution made by each of the multiple types of fluorescent labels and repeating the calculation. This is shown in Figure 4.

[0052] This process starts in step 20 by generating a random estimate of the contributed emission in pixel units by each of the multiple types of fluorescent labels for the light emitted from the sample being analyzed. This is done by forming a vector for each pixel having an entry for the random estimate of the contribution provided by each of the multiple types of fluorescent labels.

[0053] In step 21, the expected number of photons for each pixel, which is generated when given the random estimate generated in step 20, can be calculated. This can be done by matrix multiplication of the random estimate generated in step 20 by the mixing matrix M. The mixing matrix can be easily determined when given knowledge about the arrangement of the optical splitter 3 and the operation of the data capture interface 5.

[0054] The negative log-likelihood function is calculated in step 22 using the above formula with the expected number of photons calculated in step 21 and the measured number of photons received as a parameter from the data capture interface 5.

[0055] In step 23, an evaluation is made as to whether the negative log-likelihood calculated in step 22 is lower than the current minimum value (any high seed value can be set prior to the first iteration of the process shown in FIG. 4). If it is lower than the current minimum value, the current minimum value is replaced with the value just calculated. This is done because the negative log-likelihood calculated in step 22 is currently the minimum, and thus the random estimate of the phosphor emission generated in step 20 of the current iteration over the loop shown in FIG. 4 is now most likely to have resulted in the number of photons measured from the data capture interface 5.

[0056] The process of steps 20 - 23 is repeated until a decision to interrupt the process is made in step 24. This decision can be based on several things. For example, the iterative process of steps 20 - 23 can be executed until a predetermined number of iterations are performed, in which case step 24 counts the iterations and interrupts the process when the predetermined number is reached. Alternatively, step 24 can interrupt the process when there is no new minimum value for the negative log-likelihood calculated in step 23 for a given number of iterations of the process of steps 20 - 23.

[0057] In step 25, the value of the random estimate recorded in step 23 corresponding to the minimum negative log-likelihood is selected. Steps 20 to 25 are repeated for each channel of the optical splitter 3.

[0058] The value selected in step 26 is used to construct a data structure for each of the multiple types of phosphors within the sample. Each of these data structures includes, for each pixel, image data containing an entry for the contribution made by that type of fluorescent label from the vector of random estimate values for the pixel calculated in step 20. Each data structure can then be used to reconstruct an image of the sample using spectral components corresponding to the emission spectrum of each of the fluorescent labels for which the data structure was constructed.

[0059] The process of FIG. 4 works, but it is not particularly computationally efficient. A more efficient process is shown in FIG. 5. In this process, the negative log-likelihood function is minimized by use of an optimization algorithm. Several different algorithms can be used. These include Limited Memory Broyden-Fletcher-Goldfarb-Shanno, conjugate gradient descent, and Adam. However, the process of FIG. 5 uses Richardson-Lucy to facilitate implementation, computational efficiency, and scalability.

[0060] Generally, the Richardson-Lucy algorithm forms a new estimate u k from

Equation

[0061] In the case of the spectral separation scenario, H is the mixing matrix M, and d represents the measured number of photons captured by the optical splitter 3 and the data capture interface 5. u k and u k+1 respectively represent the current and new estimates for the emissions contributed by each of the plurality of types of fluorescent labels to the light emitted from the sample under analysis.

[0062] The process begins at step 30 where an initial estimate is generated for the emissions contributed by each of the plurality of types of fluorescent labels to the light emitted from the sample under analysis. This is done by forming, for each pixel, a vector having entries for the estimates of the contributions provided by each of the plurality of types of fluorescent labels. At this stage, the estimates can be nothing more than randomly generated values.

[0063] In step 31, the expected number of photons can be calculated from the estimated values for the contributions provided by each of the plurality of types of fluorescent labels. In the first iteration over the loop of steps 31 - 33, the estimated values are the values calculated in step 30. In subsequent iterations, the estimated values are the new estimates calculated in step 33 of the preceding iteration. The expected number of photons is the term H in the above equation uk is.

[0064] In step 32, the ratio of the measured number of photons provided by the data capture interface 5 based on the measurement results given by the cameras 4a - 4h on the optical splitter 3 to the expected number of photons calculated in step 31 is calculated. Then, this ratio is multiplied by the transpose of the mixing matrix to form a correction factor. Then, in step 33, the estimates of the contributions provided by each of the plurality of types of fluorescent labels are multiplied by the correction factor. This results in new estimated values.

[0065] The loop of steps 31 to 33 is sequentially repeated until a decision to interrupt the process is made at step 34. This decision is made when the estimated value u k+1 calculated in step 33 does not change by more than a threshold amount over a predetermined number of consecutive iterations of steps 31 to 33. For example, the predetermined number of consecutive iterations can be 10, 20, 50, or 100, and the threshold amount can be any suitable fixed number or a percentage of the estimated value u k+1 .

[0066] At step 35, the estimated value u k+1 calculated in the last iteration is selected. Then, the process of steps 30 to 35 is repeated for each channel of the optical splitter 3.

[0067] The value selected at step 35 is used to construct a data structure for each of the multiple types of phosphors within the sample. Each of these data structures includes, for each pixel, image data that includes an entry for the contribution made by that type of fluorescent label from the vector of estimated values for the pixel calculated in step 33 of the last iteration. Each data structure can then be used to reconstruct an image of the sample using spectral components corresponding to the emission spectrum of each of the fluorescent labels for which the data structure was constructed.

[0068] An example of the operation of the system can be shown by the following description. This represents only a simplified version of the actual system for the sake of ease of understanding of the actual system. In the simplified description, the mixing matrix is the following

Equation

Equation

Equation

[0069] Conventional techniques of the prior art that use matrix inversion described previously

Number

[0070] This has the problem that one of the estimated values is negative.

[0071] As shown in FIG. 5, after iterating the Richardson-Lucy algorithm described above 138 times, the following estimates for the contribution of the underlying phosphor are given.

Number

[0072] Another advantage of using an optimization algorithm such as Richardson-Lucy to minimize the negative log-likelihood function is that it is relatively easy to include other effects in the overall measurement operator H. For example, we can include the effect of diffraction blur by generating a synthetic operator H that describes how the light from each phosphor is blurred and then mixed. The synthetic operator is obtained by the composition of the mixing operator M and another operator that defines the blurring process (the so-called "point spread function"). In this way, the present invention inverts both spectral mixing and diffraction blur.

[0073] Examples of the application of the present invention are shown below by way of several examples. Spectrum separation software called PRISM Unmixing was developed to carry out the present invention. This software was first tested on simulated multispectral data and compared with the conventional Linear Unmixing algorithm. The results are shown in FIGS. 6a and 6b. In FIG. 6a, eight ground truth targets were simulated to represent eight different biological targets tagged with eight phosphors (each simulated target is one of the characters of the word "SPECTRUM" shown in FIG. 6a). These targets are shown in the top row of FIG. 6a. They were spectrally mixed and Poisson noise was added to simulate fluorescence microscopy data with a low signal-to-noise ratio (SNR). This result is seen in the second row of FIG. 6a.

[0074] Next, we carried out the present invention using not only the "gold standard" linear separation but also the prism separation software. The results of one iteration and 100 iterations of the prism separation software are seen in the third and fourth rows of FIG. 6a. The result of the linear separation is seen in the fifth row of FIG. 6a. It is clear that the prism separation software has excellent performance. In the original image data, the red pixels indicate negative pixel values of errors in the dataset. These can only be reproduced in grayscale in this specification, but the result is the background artefact behind the desired output seen in the row of linear separation shown in FIG. 6a. The same artefact is not clearly seen, especially in the prism separation after 100 iterations. Prism separation succeeds in reconstructing the ground truth data, while linear separation does not.

[0075] Furthermore, the error worsens when separating more blurred samples. This is shown in Figure 6b, where the ground truth dataset is now an array of different brightnesses. This is shown in the top row of Figure 6b, and data spectrally mixed with Poisson noise is added in the next row.

[0076] The results of prism separation and linear separation are seen in the third row of Figure 6b. Prism separation succeeds in reconstructing the ground truth data even for the most blurred samples, while linear separation generates errors throughout, and the errors worsen for more blurred input data. Again, this demonstrates the superiority of prism separation over the linear separation technique of the "absolute criterion".

[0077] The prism separation software was continuously tested on actual microscopic data from a conventional multi-spectral imaging method (32-channel Zeiss Zesca) using fixed U2OS cells labeled with six different fluorescent proteins tagged to six biological structures: the cell nucleus, Golgi, endoplasmic reticulum, plasma membrane, mitochondria, and peroxisomes. Figure 6c shows the results of prism separation (left column), linear separation (middle column), and non-negative matrix factorization (NMF), a spectral separation technique that is an improvement over linear separation (right column). After repeating prism separation 1000 times, the six biological targets inherent in the cell were successfully separated and identified. Again, the results of linear separation were affected by significant errors with many negative pixel values. These negative pixel values were highlighted in red in the original image data, which can only be reproduced in grayscale herein. It results in a “blur” of the extreme background that obscures the desired image data. The separated data had many incorrect results, and the biological targets were incorrectly assigned to the wrong channels, resulting in incorrect identification of the six biological targets. Thus, the results by NMF were also inferior compared to prism separation. For example, the nuclear signal could be seen in the plasma membrane image, and the endoplasmic reticulum could be seen in the Golgi image. These errors do not occur in the results of prism separation. This data demonstrates that prism separation enhances the ability to spectrally separate microscopic data with a low signal-to-noise ratio compared to other existing techniques.

[0078] To take advantage of the benefits of our prism software, we constructed, for the first time, a novel eight-direction camera module for simultaneously imaging eight spectral channels. This was constructed according to the schematic shown in Figure 3. The completed module can be seen in Figure 7. This module was designed to function with any camera-based fluorescence microscope.

[0079] To demonstrate the capabilities of our technology, we attached the camera module to a spinning disk confocal microscope as shown in FIG. 8a and to an inclined light sheet microscope as shown in FIG. 8b. In FIG. 8a, the microscope stand can be seen on the left side of the photograph, the spinning disk confocal microscope in the center, and the camera module on the right side. In FIG. 8b, the camera module can be seen in the lower left corner of the photograph.

[0080] Using the configurations of FIGS. 8a and 8b, the inventors were able to demonstrate for the first time the simultaneous imaging of one to eight fluorescent probes among multiple samples of both live and fixed samples, followed by spectral separation of the data using prism separation software. Some of the results for datasets labeled with five to eight fluorescent probes are shown in FIGS. 9a - 9f. The fluorescent probes tested on the samples included both fluorescent proteins and fluorescent dyes. The fluorescent proteins tested were TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, mScarlett, and m0range2. The fluorescent dyes tested were DAPI, SPY 555 Tubulin, FASTACT 555 Actin, LysoTracker Yellow HCK-123, OP AL 520, OPAL 570, OPAL 650, and OPAL 690.

[0081] Figure 9a shows a 6-color image of live U2OS cells captured using the rotating disk confocal microscope of Figure 8a. The cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, and iRFP670 to highlight the following biological structures: the cell nucleus, plasma membrane, mitochondria, Golgi, endoplasmic reticulum, and peroxisomes. Separate, isolated data can be seen showing not only the individual biological structures but also composite images constructed from all of these. In the original data, the cell nucleus is shown in violet, mitochondria in red, Golgi in green, plasma membrane in blue, endoplasmic reticulum in orange, and peroxisomes in white, although the colors can only be reproduced in grayscale herein.

[0082] Figure 9b shows a 7-color image of live U2OS cells captured using the rotating disk confocal microscope of Figure 8a. The cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, and LysoTracker Yellow HCK-123. A composite image is shown. In the original data, the cell nucleus is shown in violet, mitochondria in red, Golgi in green, plasma membrane in blue, endoplasmic reticulum in orange, peroxisomes in cyan, and lysosomes in magenta, although the colors can only be reproduced in grayscale herein.

[0083] Figure 9c shows a 7-color image of live U2OS cells captured using the rotating disk confocal microscope of Figure 8a. The cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, and FASTACT 555 Actin. A composite image is shown. In the original data, the cell nucleus is shown in violet, mitochondria in red, Golgi in green, plasma membrane in gray, endoplasmic reticulum in orange, peroxisomes in white, and actin in magenta, although the colors can only be reproduced in grayscale herein.

[0084] Figure 9d shows a 7-color image of live U2OS cells captured using the spinning disk confocal microscope of Figure 8a. The cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, and SPY 555 Tubulin. A composite image is shown. In the original data, the cell nucleus is violet, mitochondria are red, Golgi is green, plasma membrane is purple, endoplasmic reticulum is orange, peroxisomes are magenta, and microtubules are cyan, but the colors can only be reproduced in grayscale herein.

[0085] Figure 9e shows an 8-color image of live U2OS cells captured using the oblique plane light sheet microscope of Figure 8b. The cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, SPY 555 Tubulin, and LysoTracker Yellow HCK-123. Separated images are shown. In these component images, the cell nucleus is violet, mitochondria are red, Golgi is green, plasma membrane is gray, endoplasmic reticulum is orange, peroxisomes are white, lysosomes are magenta, and microtubules are cyan, but the colors can only be reproduced in grayscale herein.

[0086] Figures 9a - 9e are single frames taken from a microtime-lapse movie showing the dynamics of biological targets over time.

[0087] Figure 9f shows a 5-color image of a fixed and stained mouse brain spatial transcriptomics sample captured using the spinning disk confocal microscope of Figure 8a. The sample was labeled with DAPI, OPAL 520, OPAL 570, OPAL 650, and OPAL 690. A composite image in which the colors of the labeled biological structures are clearly visible is shown, but the colors can only be reproduced in grayscale herein.

[0088] These examples demonstrate the ability of the present invention for a wide variety of samples, phosphor combinations, and sizes, ranging from intracellular dynamics to 3D mouse brain sections. Simultaneous 8-color dynamic imaging was possible.

[0089] The eight cameras of the camera module in FIG. 7 are monochromatic. The arrangement of the dichroic mirrors and tube lenses directs each respective range of wavelengths to each of the cameras in the manner described above with respect to the splitter shown in FIG. 3. Thus, each camera receives a different range of wavelengths, but the cameras themselves do not record the associated colors. Instead, they simply create monochromatic images. Therefore, after separation, there are eight monochromatic images, each of which is a respective biological structure. It is common practice in microscopy to add "pseudo-colors" to such images to enhance the visual representation. In the descriptions of FIGS. 9a-9f above, the colors mentioned are pseudo-colors selected for colorization to facilitate visualization. They are not the actual colors of the phosphors or biological structures. This explains why peroxisomes are colored white in FIGS. 9a and 9c, but magenta in FIG. 9d. The colors of white and magenta were selected as those that seemed to look best in each situation. Similarly, it explains why the plasma membrane is gray in the example of FIG. 9c, blue in the example of FIG. 9a, and purple in the example of FIG. 9d.

Explanation of Signs

[0090] 1 Microscope 2 Optical link 3 Optical splitter 4a Camera 4b Camera 4c Camera 4d Camera 4e Camera 4f Camera 4g Camera 4h Camera 5 Data capture interface 6 Computer 10 Condensing lens 11a Lens 11b lens 11c lens 11d lens 11e lens 11f lens 11g lens 11h lens D1 - D7 Long - pass dichroic mirror

Claims

1. A multi-spectral imaging method executed by a computer for use in analyzing a sample containing multiple types of fluorescent labels, wherein each of the multiple types of fluorescent labels has a respective emission spectrum, and the multi-spectral imaging method comprises: Receiving multi-channel image data, wherein each channel in the multi-channel image data is derived from an unfiltered image of the sample and contains image data having respective spectral components; For each channel, i) Forming a vector of the number of quantum particles measured from the image data for the channel, the vector having an entry for each pixel in the image; ii) Repeatedly generating a vector of possible values having an entry for the contribution made by each of the multiple types of fluorescent labels to the unfiltered image for each pixel in the image, and for each iteration, calculating a vector of expected number of quantum particles having an entry for each pixel in the image by multiplying a mixing matrix defining the relationship between the unfiltered image and the multi-channel image data by the vector of possible values for each pixel; iii) Selecting the vector of possible values for which the negative log-likelihood function describes the probability of generating the vector of measured number of quantum particles on the premise that the corresponding vector of expected number of quantum particles is minimized; Constructing a corresponding data structure containing image data for each of the multiple types of fluorescent labels in the sample, wherein in the image data, for each pixel, the data structure contains the entry for the contribution made by the type of fluorescent label from the vector of possible values for the pixel, whereby each data structure is usable to reconstruct an image of the sample using spectral components corresponding to the respective emission spectra of the type of fluorescent label for which the data structure is constructed; A method comprising the above steps.

2. The method according to claim 1, wherein the quantum particles in the measured number of quantum particles and the expected number of quantum particles are photons.

3. The method according to claim 1 or 2, wherein the number of channels in the multi-channel image data is greater than 3, for example 8, 16, 32 or 64.

4. The repeatedly generating in step (ii) and the selecting in step (iii) are performed using an optimization algorithm such as limited memory Broyden-Fletcher-Goldfarb-Shanno, conjugate gradient descent method, Adam or Richardson-Lucy, according to any one of claims 1 to 3.

5. The optimization algorithm is Richardson-Lucy that performs the repeatedly generating in step (ii) according to the following formula 【Number 1】 where d is the vector of the measured number of quantum particles. However, u k is the current iteration of the vector of possible values, and u k+1 is the next iteration of the vector of possible values, and H is the mixing matrix, H T is its transpose, d is the vector of the measured number of quantum particles. The vector of possible values selected in step (iii) is the final value of u generated before interrupting the repeated generation, the interruption being triggered when a predetermined number of consecutive repetitions is exceeded, and the values of u differing by an amount less than a threshold amount, the method of claim 4. k+1 and the interruption is triggered when a predetermined number of consecutive repetitions is exceeded, and the values of u k+1 differ by an amount less than a threshold amount, the method of claim 4.

6. The method according to any one of claims 1 to 5, wherein the mixing matrix defines a function for blurring removal caused by diffraction or other optical effects in addition to the relationship between the unfiltered image and the multi-channel image data.

7. The method according to any one of claims 1 to 6, further comprising the step of generating the mixing matrix.

8. A step of generating the multi-channel image data by splitting the light received from an image source for creating the unfiltered image into a plurality of optical paths, wherein the number of optical paths is equal to the number of channels in the multi-channel image data, and each optical path has the same spectral component as a corresponding one of the channels, further comprising the step of forming an image from the light in each optical path and the step of generating image data for the channel corresponding to the optical path from the formed image, according to any one of claims 1 to 7.

9. The method according to claim 8, wherein the light received from the image source is split into the optical paths using an array of dichroic mirrors, and the images in each optical path are formed in respective cameras for generating the image data.

10. A multi-spectral imaging system for use in analyzing a sample containing multiple types of fluorescent labels, each of the multiple types of fluorescent labels having a respective emission spectrum, the multi-spectral imaging system comprising at least one processor coupled to at least one memory device, the memory device storing instructions which, when executed, cause the processor to execute the method according to any one of claims 1 to 7.

11. An array of dichroic mirrors for splitting light received from an image source that creates the unfiltered image into a plurality of optical paths, the number of optical paths being equal to the number of channels in the multi-channel image data, each optical path having the same spectral components as a corresponding one of the channels, and a camera within each optical path configured to generate the image data for the channel corresponding to the optical path. The system according to claim 10, further comprising.

12. A computer-readable medium storing instructions executed by a processor forming part of a multi-spectral imaging system for use in analyzing a sample containing multiple types of fluorescent labels, each of the multiple types of fluorescent labels having a respective emission spectrum, the instructions causing the processor to execute the method according to any one of claims 1 to 7 when executed in the processor.

Images