Method for generating a super-resolved reconstructed image of a fluorescent sample and structured illumination microscopy method

Abstract

A digital imaging process that can be used in structured illumination microscopy to obtain super-resolved images of samples having a distribution of fluorophores that emit light when illuminated by excitation light beams uses an innovative algorithm in which a super-resolved image, in a secant plane of the sample in which the excitation light beams are focused, is reconstructed also taking into account the light contributions coming from fluorophores located in out-of-focus planes. The digital image processing methods of this disclosure can be implemented through a program run by a computer and can be used in a related structured illumination microscopy method (SIM).

Description

[0001] METHOD FOR GENERATING A SUPER-RESOLVED RECONSTRUCTED IMAGE OF A FLUORESCENT SAMPLE AND STRUCTURED ILLUMINATION MICROSCOPY METHOD

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to digital imaging methods and more particularly to a computer-implemented method for generating a super-resolved reconstructed image of a fluorescent sample as well as a structured illumination microscopy method.

[0004] BACKGROUND

[0005] Two factors that deteriorate image resolution in fluorescence microscopy are noise and diffraction. The blurring of an object caused by the Abbe diffraction limit is mathematically described as a convolution of this object with the Point Spread Function (PSF) of the microscope, which is limited in bandwidth to spatial frequencies less than 2 NA / λ (where λ is the fluorescence wavelength and NA is the numerical aperture of the microscope objective).

[0006] To date, there are many optical microscopy techniques that allow the resolution limit imposed by the diffraction of light to be overcome; some of these super-resolution methods use a multi-spot "structured" light, for illuminating a sample, which exploits the physical phenomenon of beating to acquire information otherwise impossible to obtain in a normal microscopy apparatus. Many of these methods require ad hoc algorithmic processing of the data collected to obtain the final super-resolved image, that is, one with double the resolution compared to images obtained with traditional microscopy systems. Structured illumination microscopy (SIM) relies on excitation of a sample with a known spatial pattern of lights. Several images are acquired and a super resolution image is obtained.

[0007] In modern microscopy, structured illumination microscopy (SIM) can be used to examine single cells using spatially distributed light to excite fluorescence from the sample which is subsequently detected. One or more images are processed to produce a super-resolution image with twice the resolution of a conventional wide-field microscope image.

[0008] Among the various ways of structuring light to illuminate a distribution of fluorophores on a sample to be observed, there is that of using light focused through a grating of microlenses, for example as illustrated in figure 1, to acquire a number N of images to be reconstructed appropriately through a processing algorithm. The N images differ by a translation of the mask which generates structured illumination, such that the entire field of view of the microscope is illuminated in a substantially uniform manner. In such microscopes, the light source illuminates the microlens grating, focusing the pattern of dots on the biological sample; the light emitted by the fluorophores present in the sample is then collected by a detector placed in a plane conjugated to that of the sample; this operation is replicated N times, shifting the pattern of illumination points in such a way as to obtain a series of dotted images.

[0009] The typical data reconstruction method for multi-spot structured light is based on a two- dimensional Bayesian deconvolution

[0001] [2][3], in which an image formation model is used by an algorithm. A rigorous treatment from a mathematical point of view of the general ideas that lead to the reconstruction of two-dimensional super-resolution images with Bayesian deconvolution techniques is set out in [4], If the N two-dimensional images acquired by the microscope were not corrupted by noise of any kind, then the generic two-dimensional image μj(x,y) obtained by illuminating the distribution of fluorophores ρf(x,y) of the sample with the j -th distribution of light pj(x,y) focused in a secant plane of the sample and generated by the j-th source of structured illumination used to observe the sample is given by the following formula: wherein the symbol represents the convolution product in two-dimensional space and s a transmission function in two dimensions of the system illuminated with monochromatic light of wavelenght λ (or emission Point Spread Function). An example of a two-dimensional Point Spread Function (PSF) obtained with a microlens that focuses a light beam in a plane is represented in figure 2.

[0010] The generic j-th light distribution pj(x,y) must be known a priori, through an appropriate calibration measurement, and can be schematized mathematically with the following formula wherein mj(x,y) it is a generic j-th two-dimensional illumination mask placed in the plane conjugated to that of the camera and is a two-dimensional excitation Point Spread Function (PSF).

[0011] As explained in [4], what is detected by the microscope is not the image μj(x,y) but a version of it corrupted by noise:

[0012] Once the N images dj(x,y) have been captured with the microscope, as explained in

[0013] [4], an iterative procedure based on the maximum likelihood (MV) method is applied to estimate iteratively the distribution Pf according to the following formula: by imposing the condition wherein is the operator transposed with respect to the emission PSF

[0014] If we assume (see [4]) that the noise has a mixed probability distribution function between the Poisson probability distribution, which models an impulsive noise (shot noise), and the Gaussian probability distribution with variance σ2, which models a noise Gaussian generated by the microscope that captures the light, according to the following formula: thus it is possible to use the following formula:

[0015] In general, an iterative formula of this type is used: to calculate a new approximate distribution of the distribution Pf starting from the approximate distribution calculated in the previous step, wherein is a two-dimensional update operator at the k-th step and is given by a summation of products between the generic distribution of light pjfocused in the sample plane and a respective two-dimensional convolution product involving the transposed operator of the emission PSF

[0016] A drawback of the method described above consists in the fact that it does not respect the energy conservation condition, i.e. not all the light collected by the microscope is considered in the algorithm. In fact, the generic captured image dj(x,y) is ofthe type illustrated in figure 3, but in the algorithm only the part contained in the dotted circle is used, which has a diameter known a priori that expert technicians establish according to the size of the microlenses used and the microscope magnification. Since the reconstruction is carried out plane by plane and the operators used are all two- dimensional, the total energy of the detected photons can vary from one plane to another, which makes the images obtained difficult to interpret and compare. On the contrary, energy conservation is an advantage of the deconvolution algorithms commonly used in the processing of images obtained with various fluorescence microscopy techniques (wddefield, confocal).

[0017] In general, a digital imaging process that can be used in microscopy to improve the quality of images produced of specimens having a distribution of fluorophores that emit light when illuminated by excitation light beams would be desirable.

[0018] SUMMARY

[0019] The aforementioned limitations have been resolved, at least in part, with a computer- implemented method for generating a super-resolved reconstructed image as defined in claim 1. This excellent result has been achieved using an algorithm in which a superresolved image, in a plane that passes through the sample on which the excitation light beams are focused, is reconstructed also taking into account the light contributions that come from fluorophores that are found in the out-of-focus planes, which are instead neglected in the algorithms described in [4],

[0020] The methods of this disclosure can be implemented through a program executed by a computer and can be used in a related structured illumination microscopy (SIM) method.

[0021] Further embodiments are defined in the attached claims.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 shows a microlens array for generating focused light in a structured illumination microscopy system.

[0024] Figure 2 shows an example of a two-dimensional Point Spread Function (PSF) obtained with a microlens that focuses a beam of light in a plane.

[0025] Figure 3 illustrates an image of a light spot in which the pixels intended to be processed in a digital pinholing algorithm are included in a dashed circle.

[0026] Figure 4 shows an example of a three-dimensional Point Spread Function.

[0027] Figures 5a and 5b schematically illustrate the same illumination function for different values of the coordinate z0of the secant plane in focus of the sample.

[0028] Figure 6 illustrates a sample illuminated by excitation light beams focused in a secant plane passing through the sample, in which it is noted that the out-of-focus planes at a distance greater than are practically homogeneously illuminated.

[0029] Figures 7a, 7b and 7c show respectively a captured "widefield" image (figure 7a), the component in the in-focus plane (figure 7b) of the captured image and the component due to light coming from out-of-focus planes (figure 7c) of the captured image.

[0030] Figures 8a, 8b and 8c show three images obtainable through Singular Value Decomposition of N captured images.

[0031] Figures 9a, 9b and 9c represent super-resolved images, obtained after ten iterations, of the same fluorescent sample with RFP fluorophores respectively according to the method of the present disclosure (figure 9a), according to a known method with digital pinholing of 6 pixels (figure 9b) and according to a known method with digital pinholing of 10 pixels (figure 9c), in which the lateral dimension of the pixel is 32.5 nm and the images are obtained with a magnification of 100x and a wavelength of 561 nm.

[0032] Figures 10a, 10b and 10c show a progressive reduction in the likelihood value as the number of iterations increases to obtain the images of figures 9a, 9b and 9c respectively. Figures 11 a and 11 b show pixel light intensity diagrams of the elements highlighted with a dashed oval in Figures 9a and 9b, respectively.

[0033] DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0034] The processing carried out plan by plan using only two-dimensional operators involves only the information coming from the single focused plane, neglecting all that information corning from out-of-focus planes and which could potentially lead to a higher quality of the processed image in terms of contrast or resolution. Considering the excitation / emission PSFs as purely two-dimensional simplifies the numerical processing, but these PSFs are three-dimensional, as shown for example in figure 4 in the case corresponding to figure 2 where

[0035] By constructing the three-dimensional PSF of figure 4 with the focus in z = 0, the two- dimensional PSF of figure 2 is obtained, i.e. The two- dimensional model therefore does not satisfactorily represent what happens in reality as there are values relating to out-of-focus planes that are not considered in the calculation and this limits the accuracy of the super-resolved image that can be reconstructed with the algorithm iterative.

[0036] A difficulty in defining an iterative algorithm on three-dimensional space consists in the fact that, even if the illumination is focused in a generic z0plane, it still has a distribution in three-dimensional space and can be described by the formula:

[0037] Wherein x’, y’, z' are the silent variables to carry out the convolution product is the generic two-dimensional mask placed in the plane conjugated to that of the camera and the excitation PSFs are constructed with the focus in z = 0, i.e. so as

[0038] Figure 5a illustrates the illumination function of a sample to be analyzed (represented as a cloud) when the in-focus plane z0is halfway with respect to the chamber plane and the conjugate plane, while Figure 5b illustrates the same function when the plane a focus z0is closest to the plane conjugated to that of the chamber. Clearly, the values in the three-dimensional space of the function change as the z0coordinate of the plane in focus varies and the image obtained also changes, since as the z0coordinate varies, the way in which the sample is illuminated changes. According to this disclosure, the quality of the super-resolved image reconstructed starting from an image captured by an acquisition device is improved, using an iterative algorithm formulated starting from a three-dimensional model of the type: which can allow to resolve, at least partially, some of the above-mentioned drawbacks of the traditional reconstruction algorithms for the focused light SIM. The simple modification of the image model from two to three dimensions allows the contributions of the out-of-focus planes to be included exactly and therefore allows the counts of the out-of-focus contributions to be reallocated to the planes from which they really come.

[0039] What is detected by the microscope is not the image but a version of it dj(x,y, z0) corrupted by noise, so the minimization rule necessary to obtain the final super-resolved image, through a three-dimensional iterative algorithm of the type illustrated in [4], should be rewritten as follows: wherein

[0040] An advantage of these formulas consists in the fact that the digital pinholing operation is no longer necessary when dealing with experimental data , but the function does not allow the expression to be rewritten in the form of a convolution product, making its numerical implementation particularly onerous for relatively low cost hardware devices.

[0041] Assuming that each image captured by the microscope is corrupted both by "shot noise" (Poisson probability distribution) and by Gaussian noise of the camera, the generic captured image is given by:

[0042] Wherein n is a Gaussian noise having a standard deviation is an implementation of a Poisson process having an average . The likelihood function to be minimized is given by the following formula and its iterative minimization allows to reconstruct the super-resolved fluorophore distribution:

[0043] To calculate the distribution that minimizes the likelihood function, the following formula can be used:

[0044] According to this disclosure, the three-dimensional distribution of fluorophores of a super-resolved image is calculated with an iterative algorithm in which a three- dimensional distribution of fluorophores at the (k+1)-th step is obtained by multiplying the three-dimensional distribution of fluorophores at the k-th step for a k-th three- dimensional operator given by the following formula:

[0045] Wherein is the three-dimensional image corresponding to the k-th distribution of fluoroph ores and is given by

[0046] An advantage of the iterative algorithm described above consists in the fact that, in the case in which the Gaussian noise of the microscope is negligible (σ = 0), the number of total counts is constant for each iteration and is equal to:

[0047] This means that the number of photons emitted by the illuminated sample remains constant at each iteration. Furthermore, with this algorithm the image obtained relating to each z plane does not only depend on the PSF values relating to that same plane, but also depends on the illumination of other planes.

[0048] The three-dimensional algorithm of this disclosure requires a higher computational burden due to the function which is the structured and focused lighting in the plane conjugated to z0. However, it can be simplified to perform only two-dimensional convolution products, which are easier to perform with low-cost hardware devices.

[0049] According to one aspect, the k-th three-dimensional update operator can be approximated by linearizing piecewise in amplitude the convolution integral along the z. axis in a finite interval centered around the focus plane z0, as the sum of a first term in the plane in focus z0and a second term relating to planes out of focus: wherein

[0050]

[0051] The first term f the k-th update operator is two-dimensional and is referred to the in-focus plane z0, while the second two-dimensional term contains the contributions of the out-of-focus planes zn.

[0052] According to one aspect, one can restrict the convolution integral along the z axis to a width Lzcentered on the focus plane z0of the transposed operator of the three- dimensional emission PSF, by piecewise linearizing the integral along the z axis in 2M pieces of width wherein In this case the computational burden is further reduced, but with this approximation the total number of photons counted for each plane will only be approximately the same for all planes.

[0053] According to one aspect, as an alternative, or even in addition, to the simplification of the update operator at the k-th step, the formula, with which the three-dimensional image corresponding to the k-th distribution of fluorophores is calculated, can be simplified

[0054] With this approximation, the functional form of the minimization algorithm returns to that of the two-dimensional case, with the difference that now the out-of-focus contribution is explicitly added to the image model

[0055] According to one aspect, the calculation of the out-of-focus contribution can be further simplified as the greatest contribution to the total of the summation comes from those planes for which the lighting is no longer considered focused, but can be considered practically homogeneous and therefore independent of the index j :

[0056] Referring to figure 6, it can be seen that, indicating with h the distance between the foci of the light sources and with NA the numerical aperture of the focusing objective, the out-of-focus planes which are separated from the in-focus plane by a distance approximately equal to are practically illuminated in a homogeneous manner. With this approximation, the contribution of the out-of-focus planes to determine the k-th image does not have to be recalculated for all j sources and this further reduces the computational burden.

[0057] According to one aspect, the calculation of the out-of-focus contribution can be further simplified by assuming that the link between the acquired image and the out-of-focus contribution is of the type: wherein: and an example of such decomposition is shown in figures 7a, 7b and 7c which respectively show a captured widefield image (figure 7a), the in-focus component (figure 7b) and the out-of-focus component (figure 7c) of the captured image.

[0058] According to one aspect, the contribution of the out-of-focus light can be calculated from the acquired images through an SAT) (“Singular Value Decomposition”) in order to separate the part common to all N acquired images and define this common part as out-of-focus light.

[0059] To do this, each image is placed in a row of a matrix .4, wherein / essentially becomes the row index: which is then decomposed as:

[0060] Where U is a unit matrix of size m × m, Σ is a rectangular diagonal matrix of size m ×N , V* is the conjugate transpose of a unit matrix F of size N ×N, wherein m is the image size along the axis y. This decomposition allows to isolate in the image set a common "static" background image and all the N acquired images.

[0061] The left eigenvector corresponding to the values σ0. is the desired background image (see Fig. 8a) up to a constant. In figures 8b and 8c the images relating to the values σ1and σ2are represented for completeness.

[0062] The performance of the method according to the present disclosure has been compared to that achievable through known 6-pixel and 10-pixel digital pinholing methods, and the results are illustrated in Figures 9a through l ib. Figures 9a, 9b and 9c respectively illustrate super-resolved images, obtained after ten iterations, of the same fluorescent sample with RFP fluorophores respectively according to the method of the present disclosure (figure 9a), according to a known method with digital pinholing of 6 pixels (figure 9b) and according to a known method with digital pinholing of 10 pixels. In figure 9a the elements of the fluorescent sample are clearly distinguishable in the focused plane compared to the dark background. Differently, in the images of figures 9b and 9c of the same fluorescent sample, the elements in the in-focus plane are mixed with a distributed background brightness which does not allow their dimensions to be accurately determined. Figures 10a to 10c show the diagrams of the likelihood function as the number of iterations increases and demonstrate that the method of the present disclosure is generally characterized by levels of the likelihood function largely lower than those achievable with the known methods of digital pinholing of 6 pixels and 10 pixels. This excellent result is due to the fact that in the method of the present disclosure the intensity values of the in-focus plane are calculated also taking into account the contributions corning from out-of-focus fluorophores, so an image is obtained with a significantly better definition which allows to determine precisely the size of the depicted elements.

[0063] Figures 11 a and 11b respectively illustrate the light intensity distributions of the element highlighted with a dashed oval in figures 9a and 9b. One immediately notices that the light intensities of the pixels of the image in figure 9a are distributed in such a way as to allow the highlighted element to be clearly distinguished from the dark background, whereas (figure 11b) the same element is represented in figure 9b on a brightness distributed at the bottom.

[0064] Any variations or additions can be made by experts in the technical field to the embodiments described and illustrated here, while remaining within the scope of the following claims. In particular, further embodiments may include the technical characteristics of one of the following claims with the addition of one or more technical characteristics described in the text or illustrated in the figures, taken individually or in any reciprocal combination.

[0065] REFERENCES

[0066] [1] York A. G. et al - “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy”, Nature methods, Vol. 9, No. 7, p. 749- 754 (2012). [2] Ströhl F. et al - “A joint Richardson — Lucy deconvolution algorithm for the reconstruction of multifocal structured illumination microscopy data”, Methods Appl. Fluoresc. 3 (2015) 014002.

[0067] [3] Ingaramo, M. et al. - “Richardson-Lucy deconvolution as a general tool for combining images with complementary strengths”, ChemPhysChem, Vol. 15, No. 4, p. 794-800, (2014).

[0068] [4] Chakrova N. et al. - “Deconvolution methods for structured illumination microscopy”, Journal of the Optical Society of America A, Vol. 33, No. 7, p. B12-B20 (2016).

Claims

CLAIMS1. A computer-implemented method of generating a super-resolution reconstructed image of a fluorescent sample comprising fluorophores that emit light when illuminated with an excitation light beam, wherein said sample is illuminated with a plurality of excitation light beams focused in a secant plane traversing said sample to excite at least part of said sample fluorophores, said method comprising the following steps: providing a plurality of N two-dimensional images in a secant planetraversing said sample wherein each two-dimensional image in said secant plane ofsaid plurality of images was captured by illuminating said fluorescent sample with a corresponding j -th beam of excitation light focused in said secant plane, being known a generic distribution of light intensity irradiatedin three-dimensional space r by each excitation light beam focused in a generic secant plane of coordinate z' and being known a three-dimensional emission Point SpreadFunction and a corresponding three-dimensional transposed Point SpreadFunction of the excited fluorophores of the sample, iteratively computing anapproximate three-dimensional distribution of fluorophoresiteration step as the product of a k-th three-dimensional update operatorand an approximate three-dimensional distribution of fluorophores at a (k)-th iterationstep:wherein σ2is a variance of a Gaussian noise and where a reconstructed k-th two- dimensional image n a secant plane of coordinate z0, which ideally wouldbe obtained from the approximate three-dimensional distribution of the fluorophores determined at a generic k-th iteration step by focusing the j-th beam ofexcitation light in said secant plane of coordinate z0which passes through said sample, is given by:Wherein denotes a convolution product in three-dimensional space;determining a final three-dimensional distribution of the fluorophores as anapproximate three-dimensional distribution of the fluorophores at a generic k-thiteration step minimizing a maximum likelihood function calculated as:generating with said computer a file of a super-resolution reconstructed image of said sample as a summation of reconstructed images for each j-th beam offocused excitation light which theoretically would be obtained from the final three- dimensional distribution of fluorophores in which each reconstructed image isobtained as:

2. The method according to claim 1, comprising the operation of: approximating said k-th three-dimensional update operatoras the sum of a first two-dimensional term relating to a secant plane in focus z0and a secondtwo-dimensional termrelating to secant planes out of focus zn,and wherein is a piecewise linearization step.

3. The method according to claim 2, wherein said piecewise linearization step is determined as the ratio between a width Lzof said transposed three-dimensionalPoint Spread Function centered on the secant plane in focus z0, and a number2M of linearization pieces:

4. The method according to one of the preceding claims, comprising the operation of: approximating said k-th two-dimensional reconstructed imagethe sum of a first term, due to the fluorophores lying in said in-focus secant plane having coordinate z0, and of a second term due to the fluorophores lyingin out-of-focus secant planes having coordinate zn:wherein denotes a convolution product in a two-dimensional image plane and wherein is a two-dimensional Point Spread Function obtained from said three- dimensional Point Spread Function as:

5. The method according to claim 4, comprising the operation of: approximating said second term that isthe same for all excitation light beams:

6. The method according to claim 5, comprising the operation of: determining said third term constant at each k-th iteration step as:wherein N is a number of said excitation light beams, wherein pj is the luminous intensity distribution of the j -thexcitation light beam in the focus plane with coordinate z0, wherein is the two-dimensional image provided in the plane of focus withcoordinate z0.

7. The method according to claim 5, comprising the operation of: through a Singular Value Decomposition operation performed on the N supplied imagesdetermining said third term constant at each k-th iteration step as a common component to all the N supplied images8. A computer program, comprising software code configured so that, when executed by a computer, causes the computer to execute the method according to one of the preceding claims.

9. A structured illumination microscopy method for viewing a super-resolution reconstructed image of a fluorescent sample comprising fluorophores that emit light when illuminated with an excitation light beam, comprising: illuminating said sample with a plurality of excitation light beams focused in a secant plane traversing said fluorescent sample to excite at least a portion of said fluorophores in the sample; for each j-th beam of excitation light focused in said secant plane, capturing a related two-dimensional imagein the secant plane of the distribution of excited fluorophores of the sample; generating with a computer a file of a super-resolution reconstructed image of said sample by the method of one of claims 1 to 7; displaying on the screen said super-resolution reconstructed image.

Images