Computer-implemented multispectral imaging method and system

Abstract

A computer-implemented multispectral imaging method is disclosed for analyzing a sample comprising a plurality of types of fluorescent labels, each of the plurality of types of fluorescent labels having a respective emission spectrum. The method comprises: receiving multi-channel image data, each channel in the multi-channel image data comprising image data derived from an unfiltered image of a sample and having a respective spectral content; for each channel: i) forming a vector for measuring quantum particle counts from the image data for the channel, the vector having an entry for each pixel in the image; ii) iteratively generating, for each pixel in the image, a vector of possible values having entries contributing to the unfiltered image by each of the plurality of types of fluorescent labels, and for each iteration, generating a vector of possible values for each pixel by multiplying the vector of possible values for each pixel by a hybrid matrix, the hybrid matrix defines a relationship between the unfiltered image and the multi-channel image data, calculates a vector of expected quantum particle counts, the vector having entries for each pixel in the image; and iii) selecting a vector of possible values for which, given a corresponding vector of expected quantum particle counts, a negative log-likelihood function describing the probability of the generated vector of measuring quantum particle counts is minimum; and for each of a plurality of types of fluorescent labels in the sample, constructing a corresponding data structure comprising image data, where, for each pixel, the data structure comprises an entry contributed by the fluorescent label type for a vector according to possible values for the pixel, such that each data structure is available to reconstruct an image of the sample, the spectral content thereof corresponds to a corresponding emission spectrum of the fluorescent label type for which the data structure is constructed.

Description

Technical Field

[0001] The present invention relates to a computer-implemented multispectral imaging method for analyzing a sample comprising multiple types of fluorescent markers. The present invention also relates to a corresponding system and a computer-readable medium. Background Art

[0002] Fluorescence microscopy is an essential tool for studying biological processes. Biological structures are labeled using genetically encoded fluorescent proteins or fluorescent dyes or probes. These types of fluorescent labels are often referred to as "fluorophores." These biological structures can be imaged using fluorescence microscopy. Each fluorophore has a characteristic excitation and emission spectrum. The emission spectrum defines the wavelengths of light emitted by the fluorophore, while the excitation spectrum is the range of wavelengths that excite the fluorophore and cause it to emit light across its own emission spectrum.

[0003] In fixed samples, biological targets are labeled with fluorophores to reveal the structure and spatial organization within the sample, which can then be observed on a specific fluorescence microscope that suits the needs of the observer. Multiple different biological targets can be labeled with different fluorophores to reveal the spatial distribution and relationship between multiple biological structures.

[0004] An emission filter for each selected fluorophore is used within the microscope to ensure that the signal being observed at any time originates only from the fluorophore that matches the currently selected emission filter. The number of emission filters used in the experiment is equal to the number of fluorophores to be observed. One fluorophore will be excited and the emitted light will pass through the corresponding emission filter to the detector. Subsequently, an emission filter will be selected and a second fluorophore will be excited, with the light emitted from the fluorophore passing through the newly selected emission filter to the detector. This process continues for each fluorophore present in the sample. However, the use of this sequential process means that acquisition speed is a function of the number of fluorophores to be observed. In addition, the requirement to use a mechanical filter switch to select each filter has a significant impact on acquisition speed.

[0005] Furthermore, this sequential processing technique cannot be used in cases where there are more than three fluorophores in the sample. This is because excitation of one fluorophore may result in nonspecific excitation of another fluorophore and because part of the light emitted by one fluorophore may pass through the emission filter corresponding to another fluorophore, a condition known as "spectral bleed-through."

[0006] In order to image more than three fluorophores, a technique known as multispectral imaging must be employed, which relies on discretizing the emission spectra of the fluorophores into multiple spectral bands. In this way, more information is gathered about the emitted photons, and mathematical processes can be applied a posteriori to reveal the underlying labeled structures in the specimen, a process known as "spectral unmixing."

[0007] There are commercial technologies for multispectral imaging in microscopes. Perhaps the most widely used is the Zeiss Quasar Detector. By using a linear array of detection channels, multiple emission bands are imaged in parallel, making it possible to obtain a selected spectral region in a single scan across the specimen. In this embodiment, a diffraction grating disperses the emitted fluorescence and then directs it to an array of precisely defined bandwidth channels (9.7 nanometer channels or 10.7 nanometer channels in a 32-channel detector) in a dedicated multi-anode photomultiplier tube to generate a separate image for each channel. This technology is only compatible with conventional point scanning confocal microscopes, which limits applications for certain biological studies, such as live imaging of specimens.

[0008] Another example of a commercial application of multispectral imaging is the Leica SP8 FALCON, which utilizes a prism to spectrally split the emitted light onto a variety of detectors. The excitation of the fluorophores used in the system can also be modulated via a white light excitation source that can have a tunable excitation wavelength using an acousto-optic tunable filter. Further, the fluorescence signal can be split based on time by measuring the time it takes for the emitted photons to reach the detector, or the “fluorescence lifetime” of the markers used in the experiment. However, again, this approach is only compatible with slow scanning confocal microscopes, which limits its application to live cell imaging as described above.

[0009] There is a commercial gap for multispectral imaging of living samples that is compatible with optimized live imaging microscopes such as light sheet microscopes. Valm et al., 2017, Nature. 2017 Jun 1;546(7656):162-167 Multispectral imaging of living samples has been demonstrated in a research setting. Using a lattice light sheet microscope coupled to an excitation-based multispectral imaging approach, they were able to spectrally resolve the light emitted from six fluorophores in a living sample. A single image was taken at multiple excitation wavelengths, and the data was subsequently spectrally unmixed a posteriori. This is a fundamental application of multispectral imaging. However, since the spectral data is collected sequentially, it is quite slow, which limits its application to observing, for example, fast biological dynamics within cells.

[0010] Intravital imaging using fluorescence microscopy enables the study of the spatiotemporal nature of a variety of key biological processes, as the specimen remains alive while being viewed under the microscope. A significant advance in intravital imaging has been the use of light sheet microscopy, rather than the slower scanning confocal microscope. Using this technique, the entire illuminated plane can be imaged at once, making it faster and gentler on the specimen. When attempting to observe multiple fluorophores within a specimen, the above process can be used, and the fluorophores can be imaged sequentially and independently, with the same limitations on the number of fluorophores that can be observed over time.

[0011] Multispectral imaging in living samples can be applied to light-sheet microscopy, but is more difficult for three reasons. First, most multispectral imaging methods are very slow because they collect each spectral band sequentially, which significantly increases the acquisition time. This directly affects the viability of the sample, causing phototoxicity and death.

[0012] Second, existing multispectral imaging methods have been designed for point-scanning microscopes such as confocal microscopy, which means that the advantages of light-sheet microscopy for in vivo imaging cannot be realized.

[0013] Third, spectral unmixing using conventional algorithms is highly dependent on Poisson noise, an inherent aspect of fluorescence imaging. This is not a problem for fixed samples where viability is not a concern, and laser power and / or exposure time can be increased to improve the signal-to-noise ratio. However, in live imaging, it is important to reduce the exposure of the sample to light (in terms of intensity and time), which means that the resulting image may have a higher noise component. There is a trade-off between maintaining sample viability and collecting sufficient signal.

[0014] One example of a spectral unmixing technique is explored in Neher et al., "Blind Source Separation Techniques for the Decomposition of Multiply Labeled Fluorescence Images", Biophysical Journal, Vol. 96, May 2009, 3791-3800. However, the techniques discussed in this document are computationally inefficient and do not allow for the fast processing required for in vivo imaging.

[0015] Another example is provided in T. Zimmerman, "Spectral Imaging and Linear Unmixing in Light Microscopy", Advances in Biochemical Engineering, Vol. 95. However, this suffers from the above-mentioned problem that it does not take into account the effect of Poisson noise in the spectral unmixing. Summary of the invention

[0016] According to a first aspect of the present invention, there is provided a computer-implemented multispectral imaging method for analyzing a sample comprising a plurality of types of fluorescent markers, each of the plurality of types of fluorescent markers having a corresponding emission spectrum, the method comprising:

[0017] receiving multi-channel image data, each channel of the multi-channel image data comprising image data derived from an unfiltered image of the sample and having corresponding spectral content;

[0018] For each channel:

[0019] i) forming a vector of measured quantum particle counts from the image data for the channel, the vector having an entry for each pixel in the image;

[0020] ii) for each pixel in the image, iteratively generating a vector of possible values ​​having entries for contributions to the unfiltered image by each of a plurality of types of fluorescent labels, and for each iteration, calculating a vector of expected quantum particle counts having entries for each pixel in the image by multiplying the vector of possible values ​​for each pixel by a mixing matrix that defines a relationship between the unfiltered image and the multi-channel image data; and

[0021] iii) selecting a vector of possible values ​​for which the negative log-likelihood function describing the probability of the vector of measured quantum particle counts being generated given the corresponding vector of expected quantum particle counts is a minimum; and

[0022] For each of a plurality of types of fluorescent markers in the sample, a corresponding data structure comprising image data is constructed, wherein, for each pixel, the data structure comprises an entry for the contribution made by the fluorescent marker type according to a vector of possible values ​​for the pixel, such that each data structure can be used to reconstruct an image of the sample whose spectral content corresponds to the corresponding emission spectrum of the fluorescent marker type for which the data structure is constructed.

[0023] The present invention solves the above problems. It collects light emitted by many fluorescent labels simultaneously, resulting in a shortened acquisition time. The acquisition time is no longer a function of the number of fluorophores labeled within the sample, which means that more information can be collected without increasing the acquisition time, revealing deeper insights into biological processes.

[0024] It is also compatible with any form of microscopy, such as light sheet microscopy. Furthermore, by selecting vectors of possible values ​​for which the negative log-likelihood function describing the probability of a vector of measured quantum particle counts being generated given a corresponding vector of expected quantum particle counts is a minimum, the present invention can handle spectral unmixing in samples where the Poisson noise contribution is large (i.e., low signal-to-noise ratio).

[0025] It should also be noted that although the system can process light collected from many fluorescent markers simultaneously, it is beneficial even when used with just one fluorescent marker. Since emission filters are not used, a greater proportion of the emission spectrum of the fluorescent marker is collected, which improves the signal-to-noise ratio. This has a profound impact on in vivo imaging, as laser power and exposure time can be reduced, improving sample viability.

[0026] In a preferred embodiment, the quantum particles in the measured quantum particle counts and the expected quantum particle counts are photons.

[0027] In other embodiments, the quantum particle may be an electron. This is the case, for example, when the present invention is used in the context of an electron microscope. In this case, reference to spectral content should be understood to refer to the range of electron energies.

[0028] The number of channels in the multi-channel image data may be greater than 3. A typical number of channels in the multi-channel image data is 8, although higher numbers, such as 16, 32, or 64, are possible. The number of channels may be equal to the number of types of fluorescent markers in the plurality of fluorescent markers. However, in other cases, the number of channels may be less than or greater than the number of types of fluorescent markers in the plurality of fluorescent markers.

[0029] The method may further comprise reconstructing an image of the sample using the one or more data structures comprising the image data.

[0030] The iterative generation of step (ii) and the selection of step (iii) may be implemented using an optimization algorithm such as limited memory Broyden Fletcher-Goldfarb-Shanno, conjugate gradient descent, Adam, or Richardson-Lucy.

[0031] In a preferred embodiment, the optimization algorithm is Richardson-Lucy, which performs the iterative generation of step (ii) according to the following equation:

[0032]

[0033] Where: u k is the current iteration of the vector of possible values;

[0034] uk+1 is the next iteration of the vector of possible values;

[0035] H is the mixing matrix, and H T is its transpose; and

[0036] d is the vector of measured quantum particle counts;

[0037] and where the vector of possible values ​​selected in step (iii) is the vector of u generated before the interruption of the iterative generation k+1 The final value of u k+1 If the value of is less than the threshold value, an interrupt is triggered.

[0038] The predefined number of consecutive iterations may be any number starting from 2, with typical numbers being 10, 20, 50 or 100. The threshold amount may be a fixed amount, or it may be u k+1 The ratio of the values.

[0039] In addition to the relationship between the unfiltered image and the multi-channel image data, the mixing matrix can also define a function for removing blur caused by diffraction or other optical effects. Essentially, the mixing matrix required to remove spectral mixing effects can be modified to create a composite matrix that will remove both spectral mixing and blur or any other undesirable linear image formation process.

[0040] The method may further include generating a mixing matrix.

[0041] The method may further include generating multi-channel image data by splitting light received from an image source that produces an unfiltered image into a number of optical paths, the number of optical paths being equal to the number of channels in the multi-channel image and each optical path having the same spectral content as one of the corresponding channels; forming an image based on the light in each optical path; and generating image data for the channel corresponding to the optical path based on the formed image.

[0042] Light received from an image source is typically split into optical paths using an array of dichroic mirrors, and an image in each optical path is formed on a corresponding camera that generates image data.

[0043] In another aspect of the present invention, a multispectral imaging system is provided for analyzing a sample comprising multiple types of fluorescent markers, each of the multiple types of fluorescent markers having a corresponding emission spectrum, the system comprising at least one processor coupled to at least one memory device storing instructions which, when executed, cause the processor to implement the method of the first aspect of the present invention.

[0044] The system may further include an array of dichroic mirrors to split light received from an image source that produces an unfiltered image into a number of optical paths, the number of optical paths being equal to the number of channels in the multi-channel image data and each optical path having the same spectral content as one of the corresponding channels; and a camera in each optical path on which an image is formed to generate image data for the channel corresponding to the optical path.

[0045] In still another aspect of the present invention, a computer-readable medium storing instructions is provided, wherein the instructions are executed by a processor forming part of a multispectral imaging system for analyzing a sample comprising multiple types of fluorescent markers, each of the multiple types of fluorescent markers having a corresponding emission spectrum, wherein when the instructions are executed on the processor, the instructions cause the processor to implement the method of the first aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0047] Figure 1 A block diagram of a system according to the present invention is shown.

[0048] Figure 2 Shown are the emission spectra of a panel of different fluorophores.

[0049] Figure 3 A schematic diagram of a beam splitter used in the present invention is shown.

[0050] Figure 4 A flow chart of an embodiment of a method according to the present invention is shown.

[0051] Figure 5 A flow chart showing another embodiment of a method using the Richardson-Lucy algorithm according to the present invention is shown.

[0052] Figures 6a to 6c Results of applying the method according to the invention to simulated multispectral data and different biological targets are shown.

[0053] Figure 7 The invention is shown in accordance with the example application of the present invention. Figure 3 Photograph of a schematic diagram of the beam splitter.

[0054] Figure 8a and Figure 8b Attached to a spinning disk confocal microscope and an oblique plane light sheet microscope, respectively. Figure 7 The camera tree.

[0055] Figures 9a to 9f The method according to the present invention is applied to Figure 8a and Figure 8b Results of biological targets imaged by the microscope. DETAILED DESCRIPTION

[0056] Figure 1 A block diagram of a suitable system for analyzing samples including multiple types of fluorescent labels is shown. In this system, a microscope 1 is optically coupled to a beam splitter 3 by means of an optical link 2. The microscope 1 may be any microscope having a suitable optical output. For example, a microscope having an optical output for coupling to a camera to capture images from the microscope is generally suitable. The nature of the optical link 2 depends on the nature of the optical output of the microscope, but in many cases a simple condenser lens may be used to collimate the light output from the microscope 1 for further optical processing by the beam splitter 3.

[0057] Will refer to later Figure 3 The structural arrangement of the beam splitter 3 is explained. Now, the functional properties of the beam splitter 3 will be discussed. The beam splitter 3 splits the light received from the optical link 2 into eight separate channels, each channel having a corresponding bandwidth. For example, when the entire emission spectrum received from the microscope via the optical link 2 is in the range of 450nm to 650nm, then each channel can cover a respective 25nm portion of this 200nm range.

[0058] Figure 2 The spectral emission of each of a set of eight different fluorophores, referred to as EGYP, mCherry, mNeptune, mTFP1, EYFP, mKate2, mOrange and tdTomato, is shown. As can be seen, despite the fact that the emission spectra of each fluorophore are relatively narrow, they do overlap considerably. Therefore, the function of the beam splitter 3 is not sufficient to isolate the light emitted by each fluorophore into one of the corresponding channels. Instead, each fluorophore will provide a contribution to each channel. Given the bandwidth of the channel and knowledge of the light output of each fluorophore within that bandwidth, this contribution can be easily measured or predicted.

[0059] Each channel focuses its corresponding spectral portion of light onto the camera, e.g. Figure 14a to 4h. Cameras 4a to 4h are coupled to a data acquisition interface 5. The data acquisition interface 5 is provided within a computer 6 (e.g., in the form of a PCI card) that executes software for processing the acquired data, as will be described below. In other embodiments, the data acquisition interface 5 may be external to the computer 6. The computer 6 may be any general-purpose personal computer. The data acquisition interface 5 includes a set of analog-to-digital converters that receive image data from cameras 4a to 4h and convert it into a digital representation of the image acquired by each camera 4a to 4g. The digital representation gives a reading for each pixel that is proportional to the number of photons that have landed on that pixel. The photon counts can then be combined into vectors, with each channel from the splitter 3 having a corresponding photon count vector at each pixel. Thus, a vector of measured photon counts for each pixel in each channel can be formed based on the images captured by the cameras 4a to 4h.

[0060] The structure of splitter 3 is Figure 3 . The aforementioned condenser lens 10 receives and collimates light from the output of the microscope 1 having a full spectral bandwidth of 450nm to 650nm. A set of seven long-pass dichroic mirrors D1 to D7 is responsible for splitting the light into eight channels by wavelength. Each of the mirrors D1 to D7 has a different cut-off wavelength and divides the light it receives into eight separate channels in a manner that depends on its wavelength, each channel occupying its own 25nm portion of the full spectral bandwidth. The light in each channel will pass through three of the mirrors D1 to D7 before being focused by the corresponding tube lens 11a to 11h onto a corresponding one of the cameras 4a to 4h. This enables the simultaneous acquisition of eight different channels, eight times faster than using sequential acquisition techniques.

[0061] There is a trade-off between the number of channels and channel bandwidth. If there are too many channels, the channel bandwidth decreases and the signal-to-noise ratio increases because fewer photons are collected in each channel. If there are too few channels, the spectral separation is not sufficient to enable efficient spectral unmixing. In this trade-off, using eight channels represents a good practical compromise.

[0062] The placement of the dichroic mirrors D1 to D7 affects the spatial resolution of the splitter 3, as does the distance between the object and image planes created by each lens 11a to 11h. However, by modeling the point spread function using a suitable software tool such as Zemax OpticStudio, it is straightforward to optimize the placement of these components to obtain suitable image quality. Lenses 11a to 11h visualize the entire visible spectrum, so chromatic aberration does not need to be considered.

[0063] As explained earlier, the multi-channel image data provided by the beamsplitter 3 in combination with the data acquisition interface 5 is used to form vectors of measured photon counts for each channel, each vector having an entry for each pixel in the image represented by the image data.

[0064] Then as Figure 4 and Figure 5 The processing of the data shown in continues. These will be explained below. However, first, we discuss Figure 4 and Figure 5 Some theoretical background to the processing techniques shown is helpful.

[0065] The expected light intensity y at a specific pixel j in a specific channel i ij is the sum of the contributions of different fluorescent labels k. The contribution of these labels to the light intensity is proportional to their concentration x at pixel j. kj is proportional to the emission a that falls within the spectral range of channel i ik Proportional:

[0066]

[0067] where the sum is run over all different fluorescent labels (k=1,…,F). As a matrix equation, this can be written as:

[0068] Y=MX

[0069] where M is a mixing matrix defining the behavior of the beam splitter 3 and the data acquisition interface 5, in other words, defining the relationship between the unfiltered image and the multi-channel image data.

[0070] However, in practice the expected value of light intensity is rarely measured, since the actual number of detected photons is distributed according to a Poisson distribution parameterized by this expected value (due to the shot noise inherent in quantized radiation). The probability of measuring N photons is therefore given by:

[0071]

[0072] As mentioned before, conventional spectral unmixing algorithms ignore this shot noise behavior and instead assume that the detected signal is equal to the expected value (i.e. they operate in the high signal-to-noise limit where the contribution from shot noise can be neglected). In this approach, the potential fluorophore concentration is estimated by:

[0073]

[0074] Among them, M -1 Denote the matrix inverse (or, in the case of non-square matrices, the Moore-Penrose pseudoinverse) of M. This may typically result in This is obviously meaningless because the concentration provided by the fluorophore cannot be negative.

[0075] In the present invention, the spectral unmixing scenario is instead formulated as an optimization of a negative log-likelihood function constructed using the knowledge that the data will be affected by Poisson noise. In particular, for K pixels, and given a vector E of expected photon counts per pixel, and a vector m of measured photon counts per pixel, the likelihood that the expected photon counts generate the measured photon counts is given by:

[0076]

[0077] Taking the logarithm and negating it gives the negative log-likelihood function:

[0078]

[0079] Given this, if there is a way to compute a vector of expected photon counts given some potential guesses for the contribution made by each of the multiple types of fluorescent labels to the light emitted from the sample, then the guess that is most likely to produce the measured photon counts can be found by minimizing this function. This minimization process can be implemented simply by creating random values ​​for the contribution made by each of the multiple types of fluorescent labels and iterating the calculations. This is done in Figure 4 Shown in.

[0080] The process begins at step 20 by generating, on a pixel basis, a random estimate of the emission contributed by each of the plurality of types of fluorescent labels to the light emitted from the sample being analyzed. This is done by forming a vector for each pixel having an entry for a random estimate of the contribution provided by each of the plurality of types of fluorescent labels.

[0081] In step 21, the expected photon counts for each pixel that will be generated may be calculated given the random estimates generated in step 20. This may be done by matrix multiplying the random estimates generated in step 20 with a mixing matrix M. Given knowledge of the arrangement of the beam splitter 3 and the operation of the data acquisition interface 5, the mixing matrix may be determined directly.

[0082] In step 22, a negative log-likelihood function is calculated using the above equation, using the expected photon count calculated in step 21 and the measured photon count received from the data acquisition interface 5 as parameters.

[0083] In step 23, it is determined whether the negative log-likelihood calculated in step 22 is lower than the current minimum value (which can be Figure 4If yes, then the current minimum value is replaced by the value just calculated. This is done because the negative log-likelihood calculated in step 22 is now the minimum value, and therefore around Figure 4 The random estimate of fluorophore emission generated in step 20 of the current iteration of the loop shown in is now most likely to result in the measured photon count from the data acquisition interface 5 .

[0084] The process of steps 20 to 23 is iteratively implemented until a decision is made to interrupt the process in step 24. This decision may be based on a number of factors. For example, the iterative process of steps 20 to 23 may be run until a predetermined number of iterations have been run, in which case step 24 will count the iterations and interrupt the process when the predetermined number has been reached. Alternatively, step 24 may interrupt the process when there is no new minimum value for the negative log-likelihood calculated in step 23 for a given number of iterations of the process of steps 20 to 23.

[0085] In step 25 , the random estimate corresponding to the minimum negative log-likelihood recorded in step 23 is selected. Steps 20 to 25 are repeated for each channel of the beam splitter 3 .

[0086] In step 26, the selected values ​​are used to construct data structures for each of a plurality of types of fluorophores in the sample. Each of these data structures includes image data including, for each pixel, an entry for the contribution made by the fluorescent marker type according to the vector of random estimate values ​​for the pixel calculated in step 20. Each data structure can then be used to reconstruct an image of the sample whose spectral content corresponds to the corresponding emission spectrum of the fluorescent marker type for which the data structure was constructed.

[0087] Although Figure 4 The process works, but it is not particularly computationally efficient. Figure 5 A more efficient procedure is shown in . In this procedure, the negative log-likelihood function is minimized by using an optimization algorithm. Several different algorithms can be used. These algorithms include limited memory Broyden Fletcher Goldfarb-Shanno, conjugate gradient descent, and Adam. However, Figure 5 The Richardson-Lucy algorithm was used in the process because of its ease of implementation, computational efficiency, and scalability.

[0088] In general, the Richardson-Lucy algorithm uses the following formula based on the current estimate u k Form a new estimate u k+1 :

[0089]

[0090] Among them, . indicates point-by-point multiplication, H T is the transpose (or dual) of the measurement operator H, and d is the measurement data.

[0091] In the case of spectral unmixing scenario, H is the mixing matrix M, and d represents the measured photon counts collected by the spectrometer 3 and the data acquisition interface 5. k and u k+1 ] represent respectively the current and new estimates for the emission contributed by each of the multiple types of fluorescent labels to the light emitted from the sample being analyzed.

[0092] The process begins in step 30 where an initial estimate is generated for the emission contribution by each of the plurality of types of fluorescent labels to the light emitted from the sample being analyzed. This is done by forming a vector for each pixel having an entry for an estimate of the contribution provided by each of the plurality of types of fluorescent labels. At this stage the estimate may simply be a randomly generated value.

[0093] In step 31, an expected photon count may be calculated based on an estimate of the contribution provided by each of the plurality of fluorescent labels. In the first iteration of the loop around steps 31 to 33, the estimate will be the value calculated in step 30. In subsequent iterations, the estimate will be the new estimate calculated in step 33 of the previous iteration. The expected photon count is Hu in the above equation k item.

[0094] In step 32, the ratio of the measured photon counts provided by the data acquisition interface 5 to the expected photon counts calculated in step 31 is calculated based on the measurements provided by the cameras 4a to 4h on the beam splitter 3. This ratio is then multiplied by the transpose of the mixing matrix to form a correction factor. In step 33, the estimate of the contribution provided by each of the multiple types of fluorescent labels is multiplied by the correction factor. This produces a new estimate.

[0095] The loop of steps 31 to 33 is repeated iteratively until a decision is made to interrupt the process in step 34. k+1 This decision is reached when the change in the predefined number of consecutive iterations from step 31 to step 33 does not exceed a threshold amount. For example, the predefined number of consecutive iterations may be 10, 20, 50 or 100, and the threshold amount may be any suitable fixed number or estimated value u k+1 proportion.

[0096] In step 35, the estimate u calculated in the final iteration is selectedk+1 Then, the process from step 30 to step 35 is repeated for each channel of the optical splitter 3 .

[0097] In step 35, the selected values ​​are used to construct data structures for each of the multiple types of fluorophores in the sample. Each of these data structures includes image data including, for each pixel, an entry for the contribution made by the fluorescent marker type according to the vector of estimated values ​​for the pixel calculated in step 33 of the final iteration. Each data structure can then be used to reconstruct an image of the sample whose spectral content corresponds to the corresponding emission spectrum of the fluorescent marker type for which the data structure was constructed.

[0098] An example of the operation of the system can be shown by the following description. For ease of understanding, this only represents a simplified version of the real system. In the simplified description, the mixing matrix is ​​as follows:

[0099]

[0100] The contribution provided by each of the three fluorophores to a single pixel is:

[0101]

[0102] And the measured photon counts in each of the three channels are:

[0103]

[0104] The conventional method using the prior art of matrix inversion discussed above gives:

[0105]

[0106] The problem with this is that one of the estimates is negative.

[0107] like Figure 5 As illustrated in and described above, after 138 iterations of the Richardson-Lucy algorithm, the following estimate of potential fluorophore contributions is provided:

[0108]

[0109] This does not suffer from the problem of negative estimates for the fluorophore contribution. Furthermore, the estimate for the first value is much closer to the true value, and the estimates for the second and third values ​​are also close to acceptable. After these 138 iterations, the change in the estimate from one iteration to the next is small enough that the algorithm can be considered to have converged to a final solution.

[0110] Another advantage of using an optimization algorithm (such as Richardson-Lucy) to minimize the negative log-likelihood function is that it is relatively straightforward to include other effects in the overall measurement operator H. For example, the effect of diffraction blur can be included by creating a composite H operator that describes how the light from each fluorophore is blurred and then mixed. The composite operator is obtained by compounding the mixing operator M that defines the blurring process and another operator, the so-called "point spread function". In this way, the present invention will reverse both spectral mixing and diffraction blur.

[0111] The following illustrates the application of the present invention by way of some examples. A spectral unmixing software named PRISM Unmixing was developed to implement the present invention. The software was first tested on simulated multispectral data and compared with a conventional linear unmixing algorithm. The results are shown in Figure 6a and Figure 6b In Figure 6a In this paper, eight groundtruth targets were simulated to represent eight different biological targets labeled with eight fluorophores (each simulated target is Figure 6a These targets are Figure 6a They are spectrally mixed and Poisson noise is added to simulate low signal-to-noise ratio (SNR) fluorescence microscopy data. This result is shown in Figure 6a The second line in .

[0112] Then, the present invention was implemented using PRISM unmixing software and "gold standard" linear unmixing. The results of 1 and 100 iterations of PRISM unmixing software are shown in Figure 6a The results of linear unmixing are shown in the third and fourth rows of Figure 6a The fifth row shows the superior performance of the PRISM unmixing software. In the original image data, the red pixels indicate erroneous negative pixel values ​​in the data set. These are only reproducible in grayscale here, but the results are Figure 6a Background artifacts behind the desired output seen in the linear unmixing row shown in . With PRISM unmixing, the same artifacts are significantly less visible, especially after 100 iterations. PRISM unmixing successfully reconstructs the ground truth data, while linear unmixing does not.

[0113] Furthermore, the error gets worse when unmixing darker samples. Figure 6b , where the ground truth dataset is now an array of different brightness. Figure 6b , where spectral mixing data with Poisson noise is added in the bottom row.

[0114] The results of PRISM unmixing and linear unmixing are Figure 6b This is shown in the third row of . PRISM unmixing successfully reconstructs the ground truth even for the darkest samples, while linear unmixing makes errors throughout the process, and the errors are worse for darker input data. This once again demonstrates the superiority of PRISM unmixing over the "gold standard" linear unmixing technique.

[0115] The PRISM unmixing software was then tested on real microscopy data from a conventional multispectral imaging modality (32-channel Zeiss QUASAR), using fixed U2OS cells labeled with six different fluorescent proteins to label six biological structures: nucleus; Golgi apparatus; endoplasmic reticulum; plasma membrane; mitochondria; and peroxisomes. Figure 6c Results for PRISM unmixing (left column), linear unmixing (middle column), and non-negative matrix factorization (NMF) (right column), which is a more advanced spectral unmixing technique than linear unmixing, are shown. After 1000 iterations of PRISM unmixing, six potential biological targets in the cell were successfully unmixed and identified. Similarly, the linear unmixing results suffer from severe errors with many negative pixel values. These negative pixel values ​​are highlighted in red in the original image data, but can only be reproduced in grayscale here. This can result in severe background "haze", which obscures the desired image data. The results from NMF are also poor compared to PRISM unmixing, because the unmixed data has many erroneous results, in which biological targets are incorrectly assigned to the wrong channel, resulting in incorrect identification of six biological targets. For example, nuclear signals can be seen in the plasma membrane image, and the endoplasmic reticulum can be seen in the Golgi image. These errors did not occur in the PRISM unmixing results. The data show that compared with other existing technologies, PRISM unmixing has enhanced capabilities in spectral unmixing of low signal-to-noise ratio microscopy data.

[0116] To take advantage of the PRISM software, a novel eight-way camera module was built to image eight spectral channels simultaneously for the first time. Figure 3 The completed module is constructed from the schematic diagram shown in Figure 7 The module has been designed to be used with any camera-based fluorescence microscope.

[0117] To demonstrate the capabilities of the technique, the camera module was attached to a spinning disk confocal microscope ( Figure 8a ) and attached to an oblique light sheet microscope (as shown in Figure 8b ). Figure 8a In the image, the microscope stand is visible on the left side of the photo, the spinning disk confocal microscope is in the middle, and the camera module is on the right. Figure 8b In the photo, the camera module is visible in the lower left corner.

[0118] use Figure 8a and Figure 8b Using a device developed by the inventors, we have successfully demonstrated for the first time the simultaneous imaging of one to eight fluorescent probes in multiple samples (both living and fixed) followed by spectral unmixing of the data using PRISM unmixing software. Figures 9a to 9f Some results for data sets labeled with five to eight fluorescent probes are shown. The fluorescent probes tested on the samples include both fluorescent proteins and dyes. The fluorescent proteins tested are: TagBFP; Cerulean; mAzami Green; Citrine; mCherry; iRFP670; mScarlett; and m0range2. The fluorescent dyes tested are: DAPI; SPY 555 tubulin; FASTACT 555 actin; LysoTracker Yellow HCK-123; OP AL 520; OPAL 570; OPAL 650; and OPAL 690.

[0119] Figure 9a Shown using Figure 8a Six-color image of a live U2OS cell captured with a spinning disk confocal microscope. The cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, and iRFP670 to highlight the following biological structures: nucleus; plasma membrane; mitochondria; Golgi apparatus; endoplasmic reticulum; and peroxisomes. You can see the separated, unmixed data showing the individual biological structures, as well as the merged image constructed from all of them. In the raw data, the nucleus appears in blue-violet, mitochondria in red, Golgi apparatus in green, plasma membrane in blue, endoplasmic reticulum in orange, and peroxisomes in white, although these colors can only be reproduced in grayscale here.

[0120] Figure 9b Shown using Figure 8a Seven-color images of live U2OS cells captured with a spinning disk confocal microscope. Cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, and LysoTracker Yellow HCK-123. A merged image is shown. In the raw data, the nucleus appears in violet, mitochondria in red, Golgi apparatus in green, plasma membrane in blue, endoplasmic reticulum in orange, peroxisomes in cyan, and lysosomes in magenta, although these colors can only be reproduced in grayscale here.

[0121] Fig.9c Shown using Figure 8a Seven-color images of live U2OS cells captured with a spinning disk confocal microscope. Cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, and FASTACT 555 actin. Merged images are shown. In the raw data, nuclei appear in blue-violet, mitochondria in red, Golgi apparatus in green, plasma membrane in gray, endoplasmic reticulum in orange, peroxisomes in white, and actin in magenta, although these colors can only be reproduced in grayscale here.

[0122] Figure 9d Shown using Figure 8a Seven-color images of live U2OS cells captured with a spinning disk confocal microscope. Cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, and SPY 555 tubulin. A merged image is shown. In the raw data, the nucleus appears in blue-violet, mitochondria in red, Golgi apparatus in green, plasma membrane in purple, endoplasmic reticulum in orange, peroxisomes in magenta, and microtubules in cyan, although these colors can only be reproduced in grayscale here.

[0123] Fig.9e Shown using Figure 8b Eight images of live U2OS cells captured with an oblique light sheet microscope. Cells were labeled with TagBFP, Cerulean, mAzami Green, Citrine, mCherry, iRFP670, SPY 555 tubulin, and LysoTracker Yellow HCK-123. Unmixed images are shown. In these component images, the nucleus appears in blue-violet, mitochondria in red, Golgi apparatus in green, plasma membrane in gray, endoplasmic reticulum in orange, peroxisomes in white, lysosomes in magenta, and microtubules in cyan, although these colors can only be reproduced in grayscale here.

[0124] Figures 9a to 9e is a single frame taken from a time-lapse movie showing the dynamic changes of a biological target over time.

[0125] Figure 9f Shown using Figure 8aFive-color images of a fixed and stained mouse brain spatial transcriptomics sample captured by a spinning disk confocal microscope. The sample was labeled with DAPI, OPAL 520, OPAL 570, OPAL 650, and OPAL 690. A merged image is shown, where the colors of the labeled biological structures are clearly visible, although these colors can only be reproduced in grayscale here.

[0126] These examples demonstrate the capabilities of the invention across a range of samples, fluorophore combinations, and size ranges, from subcellular dynamics to 3D mouse brain sections. Simultaneous eight-color dynamic imaging is possible.

[0127] Figure 7 The eight cameras of the camera module are monochrome. The arrangement of the dichroic mirrors and tube lenses is as described above. Figure 3 The splitters shown in the figure direct the corresponding wavelength range to each camera. Therefore, although each camera receives a different wavelength range, the cameras themselves do not record the associated colors. Instead, they simply produce a monochrome image. Therefore, after unmixing, there are eight monochrome images, each with a corresponding biological structure. In microscopy, it is conventional to add "false colors" to such images to enhance the visual representation. In the above Figures 9a to 9f In the description of , the colors referred to are pseudo colors chosen for coloring to enhance visualization. They are not the actual colors of the fluorophores or biological structures. This explains why in Figure 9a and Fig.9c In the , peroxisomes are white, and in Figure 9d In , they are magenta. White and magenta were chosen to look best in each case. Similarly, it explains why in Fig.9c In the example, the plasma membrane is gray, while in Figure 9a In the example it is blue and Figure 9d In the example, it is purple.

Claims

1. A computer-implemented multispectral imaging method for analyzing a sample comprising multiple types of fluorescent markers, each of the multiple types of fluorescent markers having a corresponding emission spectrum, the method comprising: receiving multi-channel image data, each channel of the multi-channel image data comprising image data derived from an unfiltered image of the sample and having corresponding spectral content; For each channel: i) forming a vector of measured quantum particle counts from the image data for the channel, the vector having an entry for each pixel in the image; ii) for each pixel in the image, iteratively generating a vector of possible values, the vector having an entry for the contribution to the unfiltered image by each of the plurality of types of fluorescent labels, and for each iteration, calculating a vector of expected quantum particle counts, the vector having an entry for each pixel in the image, by multiplying the vector of possible values ​​for each pixel by a mixing matrix, the mixing matrix defining a relationship between the unfiltered image and the multi-channel image data; and iii) selecting said vector of possible values ​​for which a negative log-likelihood function describing the probability of a vector of measured quantum particle counts being generated given a corresponding vector of expected quantum particle counts is a minimum; as well as For each of the multiple types of fluorescent markers in the sample, a corresponding data structure comprising image data is constructed, wherein, for each pixel, the data structure includes an entry for the contribution made by the fluorescent marker type according to the vector of possible values ​​for the pixel, so that each data structure can be used to reconstruct an image of the sample whose spectral content corresponds to the corresponding emission spectrum of the fluorescent marker type for which the data structure is constructed.

2. The method of claim 1, wherein the quantum particles in the measured quantum particle count and the expected quantum particle count are photons.

3. A method according to any one of the preceding claims, wherein the number of channels in the multi-channel image data is greater than 3, such as 8, 16, 32 or 64.

4. A method according to any one of the preceding claims, wherein the iterative generation of step (ii) and the selection of step (iii) are implemented using an optimization algorithm, such as limited memory Broyden Fletcher-Goldfarb-Shanno, conjugate gradient descent, Adam or Richardson-Lucy.

5. The method of claim 4, wherein the optimization algorithm is Richardson-Lucy, which performs the iterative generation of step (ii) according to the following equation: in: u k is the current iteration of said vector of possible values; u k+1 is the next iteration of said vector of possible values; H is the mixing matrix, and H T is its transpose; as well as d is the vector measuring the quantum particle count; and wherein the vector of possible values ​​selected in step (iii) is u generated before the interruption of the iterative generation k+1 The final value of u k+1 If the value of differs by less than the threshold amount, the interrupt is triggered.

6. A method according to any one of the preceding claims, wherein in addition to the relationship between the unfiltered image and the multi-channel image data, the mixing matrix also defines a function for removing blurring caused by diffraction or other optical effects.

7. The method according to any one of the preceding claims, further comprising generating the mixing matrix.

8. The method of any of the preceding claims, further comprising generating the multi-channel image data by splitting light received from an image source producing the unfiltered image into a number of optical paths, the number of optical paths being equal to the number of channels in the multi-channel image and each optical path having the same spectral content as one of the corresponding channels; forming an image based on the light in each optical path; and generating image data for the channel corresponding to the optical path based on the formed image.

9. The method of claim 8, wherein the light received from the image source is split into the optical paths using a dichroic mirror array, and the image in each optical path is formed on a corresponding camera that generates the image data.

10. A multispectral imaging system for analyzing a sample comprising multiple types of fluorescent markers, each of the multiple types of fluorescent markers having a corresponding emission spectrum, the system comprising at least one processor coupled to at least one memory device, the memory device storing instructions that, when executed, cause the processor to implement the method of any one of claims 1 to 7.

11. The system of claim 10, further comprising a dichroic mirror array to split light received from an image source that produces the unfiltered image into a number of optical paths, the number of optical paths being equal to the number of channels in the multi-channel image data, and each optical path having the same spectral content as one of the corresponding channels; and a camera in each optical path, on which an image is formed to generate the image data for the channel corresponding to the optical path.

12. A computer-readable medium storing instructions, the instructions being executed by a processor forming part of a multispectral imaging system for analyzing a sample comprising multiple types of fluorescent markers, each of the multiple types of fluorescent markers having a corresponding emission spectrum, wherein when the instructions are executed on the processor, the instructions cause the processor to implement the method of any one of claims 1 to claim 7.

Images