Method for ultra-high-resolution evaluation of microscope images illuminated using a structured illumination method, and structured illumination microscope.

Abstract

To provide a method for super-resolution evaluation of microscope images illuminated in a structured manner and a microscope having structured illumination capable of solving the problem that the known methods only achieve a small amount of increase in resolution which can be improved in-plane by a factor of two using conventional linear structured illumination (SIM), although when a non-linear method is used for the deconvolution, the achievable resolution can be improved beyond the theoretical limit.SOLUTION: The novel method is intended to make improved resolution or improved contrast possible. If a PSF / OTF that is manipulated (individually for each order) in the same (or in a corresponding) way as a relevant order spatial frequency spectrum is used during re-weighting in a spatial frequency domain (for the deconvolution), an actually achievable resolution can be nearly doubled in comparison with conventional SIMs, both in one-stage and in two-stage variants. Also, a microscopy with a structured illumination (SIM) is provided.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a method for super-resolution evaluation of a microscope image of a sample, comprising the steps of: providing a plurality of digital original images of a sample sequentially recorded using a microscope by illuminating the sample in different phases with periodically structured illumination light; providing an optical transfer function representing the microscope imaging; verifying a plurality of order spatial frequency spectra based on the raw images; reconstructing an intermediate result image spatial frequency spectrum; weighting the optical transfer function for each verified order spatial frequency spectrum; and re-weighting the intermediate result image spatial frequency spectrum based on the weighted optical transfer function of each verified order spatial frequency spectrum. The present invention further relates to a microscope having structured illumination designed accordingly. [Background technology]

[0002] The known resolution of a microscope depends on the aperture and wavelength of light, due to the diffraction of light received by the sample within the microscope's objective lens. Since the available wavelength range of visible light is finite, the resolution of a microscope is, in principle, limited (Abbe, 1873). Regarding the spatial frequency of the sample to be imaged, this means that the optical transfer function (OTF) of the microscope in the spatial frequency domain can only be supported in a finite region around the coordinate origin. As a result, the microscope can only image spatial frequencies within an interval where this support does not disappear. The OTF is the point spread function (PSF) of the microscope converted to the spatial frequency domain. The PSF shows how a point light source is imaged through the microscope.

[0003] Conventional structured illumination of a sample from multiple different angles, followed by calculation of the original images recorded phase by phase (known as "structured illumination microscopy (SIM)"), allows for up to a twofold improvement in in-plane resolution if the excitation intensity of the illumination and the emission intensity of the sample are linearly related. SIM is disclosed, for example, in Patent Document 1 and Non-Patent Document 1. This is based on generating a spatial periodic optical structure for the sample under inspection, for example, by sinusoidal interference of illumination light downstream of an optical grating. By convolution of the sample response in the spatial domain using the microscope's PSF, the spatial frequency domain of the sample structure that exceeds the support range of the OTF in the spatial frequency domain is shifted to the central support interval, where these are superimposed on the original spatial frequency intensity there, thereby making a moiré effect visible in the original image. Here, the Fourier transform of each original image contains multiple frequency-shifted "copies" of the spatial frequency spectrum of the sample in each frequency band. Each of these copies is called a separate "order". That number is determined by the number of mutually interfering beams that generate periodic optical structures.

[0004] According to the SIM method, from a set of such original images containing the superposition of shifted and original spatial frequencies, it is possible to reconstruct a resulting image that includes both the original spatial frequencies of the support intervals and higher original spatial frequencies temporarily shifted to the support intervals by structured illumination. As a result, the resulting image has a higher in-plane resolution than a conventional single recording with uniform illumination. This resolution is called superresolution when it is finer than the diffraction-limited resolution of the microscope.

[0005] SIM can also achieve axial super-resolution (referred to as 3D-SIM). For this, the illumination in the sample must be periodically structured axially as well, and for each of the N sample planes, a separate set of original images from the relevant planes is recorded. From the system of equations that describes the interactions within all N sample planes, a z-stack can be calculated from the N resulting images that have super-resolution axially and especially in the in-plane direction. 3D-SIM is described in Non-Patent Document 2.

[0006] Conventionally, the following five steps are performed for the image evaluation of SIM original images: 1. Check the spatial frequency spectrum in the frequency domain (i.e., extract and separate the frequency bands that carry high-resolution information in particular). For this purpose, the moiré patterns contained in the original image are decoded by solving a system of linear equations in the spatial frequency domain, and thus the respective spatial frequency spectra for each order are checked. In the 3D case, the system of equations describes multiple sample planes, and the spatial frequency spectrum in the frequency domain is three-dimensional.

[0007] 2. Identify the phase angle of the illumination pattern and the actual repetition frequency (e.g., the grating frequency) from the original image. Based on these parameters, preferably, artifacts can be minimized in subsequent steps. This step is not absolutely necessary, but in practice, it can compensate for the inaccuracies of the experiment.

[0008] 3. Perform frequency filtering on the spatial frequency spectrum in the frequency domain to suppress the repetition frequency, especially the low-frequency background (which will shift to higher frequencies by subsequent shifting and lead to artifacts in the final image).

[0009] 4. Shift the frequency bands extracted in the first step to the corresponding positions in the spatial frequency domain (to enable the compilation of high-resolution images in the spatial frequency domain in subsequent steps).

[0010] 5. Combine the shifted frequency bands to construct the super-resolution resulting image spatial frequency spectrum in a composite step including: a. A weighted summation of the shifted frequency domain, i) weights based on the illumination components due to differences in illumination intensity at individual frequencies, and ii) weights obtained from the optical transfer function. b. Smoothing of a configuration using a general-purpose Wiener filter, particularly at the inner edge of the frequency band. c. Suppression of ringing artifacts generated by the Wiener filter by using an apodization function.

[0011] Finally, the resulting super-resolution image is transformed from the spatial frequency domain back to the spatial domain using an inverse Fourier transform. As an alternative to this conventional procedure, a two-stage evaluation method is known, in which, in the fifth step, instead of the substep described above, the intermediate image spatial frequency spectrum is first examined by weighting based solely on the illumination components (not on PSF or OTF) and without using a Wiener filter. In an additional (sixth) step, the super-resolution resulting image spatial frequency spectrum is examined by inverse superposition of the intermediate image spatial frequency spectrum based on the overall PSF (in the case of SIM) synthesized from all orders, or based on the corresponding synthesized overall OTF. When a nonlinear method is used for inverse superposition, the achievable resolution can be improved beyond the theoretical limits of one-stage SIM (and two-state SIM by linear successive approximation), because frequencies beyond the supported range of the PSF can also be reconstructed due to a priori known constraints such as the non-negative constraint (Non-Patent Literature 3). This is particularly true for what is known as the Richardson-Lucy method (Non-Patent Literature 4).

[0012] The SIM method for such two-dimensional cases is described by Perez et al. in Non-Patent Document 5. In this case, the original image is inversely superimposed based on a microscopic PSF or microscopic OTF, and then further processed to suppress artifacts. The three-dimensional case is described by Zhang et al. in Non-Patent Document 6. In both cases, the inverse superimposition is performed by the Richardson-Lucy iterative approximation method, as indicated in the title. However, the resolution achieved is only 13% above the theoretical maximum of the conventional one-stage evaluation using a Wiener filter in Zhang et al.'s case, and only 20% above in Perez et al.'s case. [Prior art documents] [Patent Documents]

[0013] [Patent Document 1] German Patent Application Publication No. 19908883A1 [Non-patent literature]

[0014] [Non-Patent Document 1] (Article by M. Gustafsson, "Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy," Journal of Microscopy, Vol. 198, 2000, p. 82) [Non-Patent Document 2] Gustafsson et al., "Three-Dimensional Resolution Doubling in Wide-Field Fluorescence Microscopy by Structured Illumination," Journal of Biophysics, Vol. 94, 2008, p. 4957. [Non-Patent Document 3] Verveer et al., "A comparison of image restoration approaches applied to three-dimensional confocal and wide-field fluorescence microscopy," Journal of Microscopy (J. Micr.), Vol. 193, 1999, p. 50. [Non-Patent Document 4] Biggs, "3D Deconvolution Microscopy," Current Protocols in Cytometry, 12.19.1-12.19.20, April 2010. [Non-Patent Document 5] "Optimal 2D-SIM reconstruction by two filtering steps with Richardson-Lucy deconvolution," Scientific Reports, 6:371149 [Non-Patent Document 6] "Super-resolution algorithm based on Richardson-Lucy deconvolution for three-dimensional structured illumination microscopy," Journal of the Optical Society of America A, Vol. 36, No. 2. [Overview of the project] [Problems that the invention aims to solve]

[0015] The present invention is based on the objective of improving the methods and microscopes of the type described in the introduction to obtain improved resolution or improved contrast. [Means for solving the problem]

[0016] This objective is achieved by a method having the features specified in claim 1 and by a microscope having the features specified in claim 14. Advantageous improvements of the present invention are specified in the dependent claims.

[0017] This invention acknowledges that two-stage reconstruction certainly offers significant advantages over one-stage reconstruction, but the resolution improvement achievable through final reweighting in the spatial frequency domain for inverse superposition in currently known procedures is not the greatest possible, and this is because, to date, PSFs or OTFs that are not adapted for evaluation have been used. Specifically, the reason is that in known methods, contributions from different orders are manipulated during the evaluation process, while the PSF or OTF remains unchanged (except for the necessary shifts in the spatial frequency domain). However, by using PSFs / OTFs that are manipulated (individually for each order) in the same (and corresponding) way as the spatial frequency domains of the orders involved by this invention during reweighting (for inverse superposition), the resolution that can actually be achieved can be nearly double that of conventional SIMs, in both one-stage and two-stage configurations. This invention is not limited to iterative inverse superposition schemes and can be implemented using non-iterative methods. However, what can be achieved by such methods is not a significant improvement in resolution compared to conventional SIMs, but rather an improvement in contrast.

[0018] Accordingly, the present invention also provides, in correspondence with at least one operation of the order spatial frequency spectrum before or during reconstruction, an operation of the optical transfer function for the relevant order spatial frequency spectrum before or during reweighting (i.e., in the same calculation steps), so that the reweighting is performed based on the operated OTF. The operation in this case is not a shift to the original frequency band in the (known, required) spatial frequency domain, but rather, for example, relative reweighting between different spatial frequency components of the OTF. In particular, the relative reweighting during the operation can be performed such that the sum of all intensities in the discrete frequency domain is included in the change of the OTF.

[0019] By using an optical transfer function that is manipulated in the same way as the order-space frequency spectrum during the inverse superposition of the intermediate-result image-space spectrum, the resolution of the inversely superimposed image is substantially increased. The corresponding operation of the OTF can be performed, for example, using the same algorithm or mathematical operations as those used in the order-space frequency spectrum.

[0020] Just as the provided OTF (PSF) reveals the characteristics of the original images (raw images) produced by the microscope optical unit, the scanned OTF (PSF) also reveals the characteristics brought about by the digital processing of the original images. This allows for a significant improvement in the iterative method, by enabling a sufficiently large number of iterations. In this way, a resolution of up to approximately 60 nm can be achieved in the resulting image.

[0021] To achieve this, for example, a number of copies of the OTF corresponding to the number of order spatial frequency spectra to be examined are provided. Each OTF copy is uniquely assigned to its respective order spatial frequency spectrum, so that exactly one OTF copy belongs to each order spatial frequency spectrum. In each copy, the OTF is shifted to a position in the spatial frequency domain corresponding to the relevant order. When one operation is performed on an order spatial frequency spectrum, the associated OTF is also operated on in a corresponding manner. If multiple operations are performed on a single order spatial frequency spectrum, all of them, or only an actual subset of them, can be performed on the OTF accordingly. Here, it is irrelevant whether the operations are performed in the spatial domain or the spatial frequency domain.

[0022] In the simplest case, the provided OTF (from which, for example, a frequency-shifted copy as described above) can be based on a Gaussian PSF. Significantly better results can be achieved with a typical microscope OTF (wide-field) known in the prior art and used, for example, by Perez et al. and Jean et al. (Richards and Wolf: "Electromagnetic diffraction in optical systems, II. Structure of the image field in an aplanatic system," Proc. R. Soc. London Ser. A, Vol. 253, p. 358, 3D-PSF). Alternatively, the OTF can also be provided based on other PSF modes.

[0023] Alternatively, the provided OTF can be confirmed based on the measured OTF or PSF. In particular, such measurements can be performed with at least one (quasi) point fluorescence light source (e.g., one or more beads) as a calibration sample for the associated microscope used to record the original image of the actual sample. For this purpose, a simple wide-field recording (diffraction-limited) can be performed, which directly includes the PSF. Alternatively, structured illumination can be performed with multiple illumination phases, with one separate image recorded in each phase each time. Multiple orders are separated in SIM by demodulation of moiré patterns from the individual images of the calibration sample, shifted to their original positions in the spatial frequency domain based on one or more actual repetition frequencies confirmed, for example, preferably from the original image of the actual sample, and combined based on their respective illumination intensities. The result is provided as the OTF.

[0024] Alternatively, the provided OTF can be verified by simulating the imaging behavior of one or more point light sources through the relevant microscope. In this case, the simulation of structured illumination can be performed in multiple illumination phases, the illumination parameters of which are determined based on the original image. For example, one or more actual repetition frequencies of the optical structure and the actual position and / or orientation of each of the original images from which it was recorded can be determined from the original image. Then, the structured illumination of each image of the simulated SIM is performed at the actual position and / or orientation belonging to the relevant original image, based on one or more actual repetition frequencies. Multiple orders are separated from these individual images by demodulation of the moiré pattern, shifted to their original positions in the spatial frequency domain, for example, based on one or more actual repetition frequencies, and combined based on their respective illumination intensities. The result is provided as the OTF.

[0025] In the sense of the present invention, the illumination light is periodically structured one-dimensionally (line pattern) or two-dimensionally (cross grid pattern) across the optical axis of the detection direction used to record the original image, and in particular has at least one corresponding repetition frequency. When the illumination light is emitted through a detection objective lens, it is typically periodically structured at least across its propagation direction. This can be structured one-dimensionally, two-dimensionally, or three-dimensionally, typically as a sine wave, in which case the repetition frequency can differ depending on the spatial direction, although this is not essential. For example, when emitted by a second objective lens in the form of an optical sheet, the illumination light can be structured longitudinally with respect to its propagation direction (optical axis of illumination) (in which case it is only one-dimensional, i.e., does not cross its propagation direction). The order spatial frequency spectrum, in particular, in each case, includes contributions of spatial frequencies from different spatial frequency bands ("order") of the illumination light, thereby enabling the reconstruction of an intermediate resulting image with improved resolution compared to the original image. For this purpose, the original image can be transformed (optionally, after preprocessing) from the spatial domain to the spatial frequency domain, particularly by integral transforms or discrete transforms, such as Laplace transforms, Fourier transforms, wavelet transforms, or Z transforms, or by other transforms suitable for spectral analysis. The order is defined by the structure, particularly by the interference structure of the original image. Reconstruction of the intermediate image spectrum, in which the (inverse) transformation to the spatial domain (i.e., the intermediate image) has a higher resolution than the original image, is performed particularly based on the order spatial frequency spectrum. The optical transfer function can preferably represent the imaging of the microscope during the recording of the original image. The microscope used to record the original image is, in this case, a virtual microscope, and the original image is recorded from a simulation. Reweighting can be performed in particular so that the reweighting corresponds to the inverse superposition of the spatial domain based on the point image distribution function. After reweighting, the intermediate image spatial frequency spectrum is preferably transformed inversely from the spatial frequency domain to the spatial domain (typically by the inverse of the integral or discrete transform described above) to confirm the super-resolution image (and thereby inversely superimpose the intermediate image of the PSF).However, such inverse transformations are not always necessary. For example, if only the spatial frequency spectrum of the resulting image is needed for subsequent processing, the inverse transformation can be omitted.

[0026] An advantage is that each of the order spatial frequency spectra is manipulated in at least one image processing step, thereby allowing the resulting intermediate image to be reconstructed based on the manipulated order spatial frequency spectra, with increased resolution compared to the original image. The resulting intermediate image, having a higher resolution than the original image, is preferably reconstructed based on the manipulated order spatial frequency spectra.

[0027] In this case, steps 1 to 5 of the conventional SIM method can be performed here by the present invention. Alternatively, steps 1 to 6 of the known two-step method can be performed. According to the present invention, whenever an OTF (or PSF) is used in these methods, the corresponding operated OTF (or PSF) is used.

[0028] Embodiments in which the manipulation includes, or is performed, spatial-frequency-dependent operations, particularly spatial-frequency-dependent weighting, particularly spatial-frequency-dependent filtering (suppression), and more particularly, spatial-frequency-dependent notch filtering (suppression within a notch), especially around each coordinate origin (i.e., around zero-frequency locations), are particularly advantageous. Such operations, especially in the form of weighting or filtering, have a significant impact on the quality of the PSF / OTF. In this way, a significant improvement in resolution beyond the known SIM resolution can be achieved. By filtering the zeroth order of the illumination structure, out-of-focus light in the order spatial frequency spectrum can be suppressed particularly effectively. The corresponding filtering of the OTF enables particularly high resolution as part of the inverse superposition. Without it, the out-of-focus background would outweigh the structures contained within the image plane. Alternatively or additionally, a low-pass filter can be used to suppress high-frequency noise or per-camera frequencies (e.g., strips of sCMOS detectors).

[0029] Here, it is advantageous if the illumination light in the sample has at least one repetition frequency, and the spatial frequency-dependent filtering includes notch filtering that also depends on at least one repetition frequency, particularly notch filtering for suppressing the illumination structuring frequency (in other words, at least one repetition frequency). This repetition frequency-dependent filtering can filter (to suppress) higher non-zero orders, for example, when using a notch filter, in place of or in addition to the aforementioned zero-order (repetition frequency-dependent) filtering, especially near the center of that order. This can reduce artifacts.

[0030] Particularly preferred is an embodiment in which the illumination light is periodically structured along the optical axis of the image recording, and different original images are emitted from different planes of the sample, in which case the confirmation of the order spatial frequency spectrum based on the original images from different planes of the sample is performed so as to confirm the three-dimensional spatial frequency spectrum, the optical transfer function is provided three-dimensionally, and the inverse superposition is performed three-dimensionally. Such an embodiment makes it possible to greatly increase the resolution even in 3D-SIM.

[0031] In the reconstruction of the intermediate spatial frequency spectrum, it is preferable that the order spatial frequency spectrum is not weighted based on the optical transfer function, although it is possible to weight the order spatial frequency spectrum based on the (each) optical transfer function. An advantage is that the weighting by the illumination intensity of each relevant order can be done alternatively or additionally. In this way, the weighting of the contribution due to the effect of the OTF (which is manipulated like each order spatial frequency spectrum) can be taken into account, thereby enabling higher image resolution.

[0032] In the sense of the present invention, "manipulation" typically compensates for, corrects, or suppresses any kind of error or artifact, such as optical aberrations or out-of-focus light, by altering the mathematical entity being manipulated. This type of manipulation differs from integral and discrete transforms, particularly in that it alters the optical transfer function for the relevant order spatial frequency spectrum, and this alteration is especially significant. Integral and discrete transforms are not considered manipulations in the sense of the present invention, as they merely transform the processed mathematical entity into another representation without altering it.

[0033] Particularly advantageous are embodiments in which deconvolution is performed iteratively, especially nonlinearly, and more specifically by the Richardson-Lucie method or maximum likelihood estimation. These methods can achieve particularly high improvements in the resolution of the resulting image, because they are particularly sensitive to the quality of the PSF used in deconvolution. If a PSF that is not treated correspondingly, as in order of order, is used before deconvolution in conventional methods, the iterative algorithm will not converge. Non-iterative deconvolution is also possible. In this way, higher contrast can be achieved compared to conventional methods.

[0034] The inverse superposition is preferably repeated far more than 5 times, and especially more than 40 times. As a result, particularly high resolution may be achieved. It may be advantageous to perform operations on the original image, particularly spatial frequency-dependent operations, more particularly spatial frequency-dependent frequency reweighting, and more particularly, filtering and / or inverse superposition, on an unoperated optical transfer function before the order spatial frequency spectrum is confirmed, and then perform the confirmation of the order spatial frequency spectrum based on the operated original image. For example, preprocessing as described by Perez et al. can be performed.

[0035] A particularly high level of accuracy in the evaluation can be achieved by verifying at least one actual parameter of the structured illumination, in particular the repetition frequency and / or orientation and / or position, based on at least one original image, or based on a previously manipulated original image. This makes it possible, for example, to perform the entire OTF of a microscope with a high level of accuracy.

[0036] Preferably, the manipulation of the order spatial frequency spectrum and / or OTF includes, or is, the respective shifts and / or weights and / or weighted summations based on the actual parameters confirmed.

[0037] OTF can be performed based on at least one verified real parameter and a SIM evaluation of the individual simulated images from which the OTF is confirmed, particularly by verifying the individual images simulated under a structured illumination simulation using at least one verified real parameter. Therefore, particularly accurate PSF / OTF can be obtained, which in turn enables the reconstruction of the resulting image, particularly at high resolution (or high contrast in the case of nonlinear or linear iterative inverse superposition).

[0038] Conveniently, for each order spatial frequency spectrum, a copy of the OTF shifted to correspond to that order spatial frequency spectrum is provided, and in all subsequent method steps, one of each of these copies can be used as the OTF.

[0039] Preferably, as part of the SIM evaluation, different order spatial frequency spectra contained in the original image or a previously manipulated original image are demodulated and separated during the verification of the order spatial frequency spectra, shifted in the spatial frequency domain before or during the reconstruction of the intermediate result image, and the separated and shifted order spatial frequency spectra are combined during the reconstruction of the intermediate result image.

[0040] In one conceivable embodiment, multiple weighted optical transfer functions are combined to form an overall transfer function (OTF), and inverse superposition is performed based on the OTF, with particular manipulation of the optical transfer functions with respect to the relevant order spatial frequency spectrum occurring before or during the combination. This procedure enables simpler calculation steps.

[0041] The overall OTF is preferably, formula

[0042]

number

[0043] This was confirmed using the formula, where the sum (subscript j) spans all orders (order spatial frequency spectrum). Multiple original images are conveniently recorded by sequentially recording the sample light emitted and / or scattered by the sample at each phase using a detector, when the sample is illuminated at different phases using a microscope with structured illumination light (29). From these original images, it is possible to reconstruct intermediate result images with improved resolution compared to the original images. In all steps, the result of the reconstruction in this case does not absolutely have to be an image in the spatial domain (referred to as the "intermediate result image"), and alternatively or additionally, a spatial frequency spectrum can also be reconstructed (referred to as the "intermediate result image spatial frequency spectrum").

[0044] Preferably, the original image is recorded using a two-dimensional spatially resolved detector, and in particular, there is no aperture in front of the detector that optically sections the sample. In this way, shorter exposure times are possible compared to scanning image recording, thus reducing stress on the sample. Therefore, it is possible to generate illumination light structures by interference and / or modulation of (point-shaped or line-shaped) illumination light for scanning.

[0045] The present invention also includes a microscope, which has a control unit configured to perform the aforementioned method, and also has a light source, a two-dimensional spatially resolved detector for recording an original image of a sample, and means for generating illumination light periodically structured in different phases into the sample, and in particular, no aperture for optically cutting the sample is positioned in front of the detector. The control unit may have an input interface through which the original image and OTF can be provided. The control unit may have an output interface through which a resulting image can be output.

[0046] The present invention also includes a computer program configured to perform the method described above. Within the meaning of this invention, in all steps, the image and spatial frequency spectrum are merely different representations of the same mathematical entity and are always interchangeable (by conversion to or from the spatial frequency domain), and this also does not deviate from the scope of this invention. The same applies to PSF and OTF.

[0047] The original image of a fluorescent sample is typically recorded by the excitation of the fluorescent substance. The present invention may also include providing a sample labeled with at least one fluorescent dye. The present invention will be described in more detail below based on exemplary embodiments. [Brief explanation of the drawing]

[0048] [Figure 1] This shows a multimodal microscope. [Figure 2] This paper demonstrates known principles of structured illumination in multiple phases. [Figure 3] This demonstrates the generation of illumination patterns with three interference orders within the pupil. [Figure 4] This demonstrates the generation of illumination patterns with three interference orders within the pupil. [Figure 5] This shows the known 1-stage SIM evaluation. [Figure 6] This document shows a comparison between the two-stage 3D-SIM evaluation improved by the present invention and the two-stage 2D-SIM evaluation improved by the present invention. [Figure 7] This document compares the improved two-stage SIM evaluation method according to the present invention with a known two-stage SIM evaluation method. [Figure 8] This shows a comparison of images of the sample obtained using different methods. [Modes for carrying out the invention]

[0049] In all drawings, corresponding parts are given the same reference numeral. Figure 1 shows a microscope 1 having different operating modes. This microscope can perform both conventional microscopy, i.e., microscopy where the resolution is limited by diffraction, and high-resolution microscopy, i.e., microscopy where the resolution exceeds the diffraction limit. This is an inverted microscope. Alternatively, it may be embodied in the form of an upright microscope.

[0050] Microscope 1 images sample 2. For this purpose, it includes an objective lens 3 through which the radiation from all microscopy methods described later passes. The objective lens 3, along with the tube lens 5, images the sample onto the CCD detector 6 via the beam splitter 4, which in this example is a two-dimensional spatially resolved region detector. Up to this point, the microscope 1 has a conventional optical microscope module 7, and the beam path from the sample 2 through the objective lens 3 and tube lens 5 to the CCD detector 6 corresponds to a conventional widefield detection beam path 8. As shown by the double arrows in Figure 1, the beam splitter 4 is interchangeable and can be switched between beam splitters with different dichroic characteristics or a colorless beam splitter according to U.S. Patent Application No. 2008 / 0088920.

[0051] Furthermore, a laser scanning module 9 is connected to the beam path of the objective lens 3, and its LSM illumination and detection beam paths are connected to the beam path of the objective lens 3 via a switching mirror 11, which also functions as a beam splitter. The beam path from the switching mirror 11 through the beam splitter 4 to the objective lens 3 is therefore a beam path in which the illumination beam path and the detection beam path are combined. This applies to both the laser scanning module 9 and the wide-field detection beam path 8, because, as will be described later, the illumination radiation that realizes the microscopy method together with the wide-field detection beam path 8, i.e., the CCD detector 6, is also combined with the switching mirror 11.

[0052] The switching mirror 11 and the beam splitter 4 are combined to form a beam splitter module 12, and as a result, the switching mirror 11 and the beam splitter 4 can be interchanged depending on the application. This is also indicated by a double arrow. Furthermore, the beam splitter module 12 is also provided with an emission filter 13, which is located within the wide-field detection beam path 8 and appropriately filters out spectral components that can propagate through the wide-field detection beam path 8. The emission filter 13 in the beam splitter module 12 is also interchangeable.

[0053] The laser scanning module 9 receives the laser radiation necessary for operation from the laser module 15 via the optical fiber 14. In the configuration shown in Figure 1, a collective illumination beam path 16 through which illumination radiation for various microscopy techniques passes is connected in a beam splitter module 12, more specifically in a switching mirror 11. Different illumination beam paths of individual illumination modules are coupled to this collective illumination beam path 16. For example, a wide-field illumination module 17 couples its wide-field illumination radiation to the collective illumination beam path 16 via a switching mirror 18, thereby illuminating the sample 2 in a wide field through a tube lens 27 and an objective lens 3. The wide-field illumination module 17 may have, for example, an HBO lamp. Another illumination module is also provided, a TIRF illumination module 19, which achieves TIRF illumination by appropriately positioning the switching mirror 18. For this purpose, the TIRF illumination module 19 receives radiation from a laser module 15 via an optical fiber 20. The TIRF illumination module 19 has a mirror 21 that is displaceable in the longitudinal direction. Due to the longitudinal displacement, the illumination beam emitted by the TIRF illumination module 19 is displaced perpendicular to the main propagation direction of the emitted illumination beam, and as a result, the TIRF illumination is incident on the objective lens 3 at an adjustable angle with respect to the optical axis of the objective lens. In this way, the angle required for total internal reflection in the coverslip can be easily secured. Other means are, of course, also suitable for this angle adjustment.

[0054] Furthermore, the illumination beam path of the manipulator module 22 is connected to the collective illumination beam path, which similarly receives radiation from the laser module 15 via optical fibers (which are not further numbered here) and guides a point-like or line-like beam distribution in a scanning manner on the sample 2. The manipulator module 22 thus substantially corresponds to the illumination module of the laser scanning microscope, and as a result, the manipulator module 22 can also operate in combination with the detector of the laser scanning module 9 or the wide-field detection of the CCD detector 6.

[0055] A grating 23 is further provided in the collective illumination beam path 16, which has a grating constant lower than the cutoff frequency that can be transmitted from the microscope 1 to the sample 2. The grating 23 can be positioned, for example, in the plane of the illumination beam path 16 that is imaged into the sample (the intermediate image of the sample). The grating 23 is displaceable across the optical axis of the collective illumination beam path 16, preferably in two dimensions across this optical axis. For this purpose, a corresponding displacement drive unit 24 is provided.

[0056] An image field rotator 25, rotated by a rotor drive unit 26, is further positioned downstream of the grid in the illumination direction within the collective illumination beam path 16. The image field rotator can be, for example, an Abbe-Koenig type prism. If the grid 23 has a two-dimensional structure, the image field rotator 25 is not necessary, because rotation is not required for the resulting illumination structure. Instead, it can be displaced, for example, in two dimensions.

[0057] Microscope 1 includes a control unit 28, for example, a computer with a Von-Neumann architecture, which in particular has a processor as a computing and control unit, random access memory as working memory, and a magnetic hard disk as a mass storage means.

[0058] The modules, drive unit, and detector of microscope 1 are all connected to the control unit 28 via wiring (which will not be further numbered here). The connection can be achieved, for example, via a data and control bus. The control unit 28 controls microscope 1 in different operating modes. The control device 28 therefore enables microscope 1 to perform the functions of conventional microscopes, namely widefield microscopy (WF), especially structured illumination (SIM), laser scanning microscopy (LSM), and fluorescence microscopy utilizing total internal reflection (TIRF).

[0059] The microscope in Figure 1 has substantially two modules suitable for laser scanner illumination, specifically a laser scanning module 9 and a manipulator module 22. Other combinations are, of course, possible. The modules are coupled to the objective lens 3 on the sample 2 via a tubular lens. The manipulator module 22 simply includes the excitation part of the laser scanning module and does not perform detection. As a result, the sample can be illuminated in a point-like manner, and the illumination spot can be scanned on the sample 2.

[0060] Preferably, a switching unit, such as a switching lens or cylindrical lens, is used to switch between point and linear illumination, and this is also located within the manipulator module 22. Linear illumination is particularly advantageous when the grid 23 rotates along the line of linear illumination and is positioned perpendicular to the line of linear illumination. Alternatively, linear illumination can be used for the dynamic (sequential) generation of structured illumination within the sample 2.

[0061] A variably adjustable stripe modulator, DMD, or SLM can also be used instead of the grid 23 to generate structured illumination of the sample 2. In this case, the function of moving the displacement drive unit 24 and the grid 23 in and out by rotation is, of course, unnecessary.

[0062] The image field rotator 25 allows the structured illumination generated by the grating 23 (or an alternative element) to be rotated around the optical axis of the collective illumination beam path 16, thereby extending the structured illumination at different angles within the sample 2.

[0063] To switch between individual operation types, the switching mirrors 18 and 11 and the beam splitter 4 are appropriately adjusted. In practice, a folding mirror or a pivot mirror can be used for this purpose, thereby allowing sequential switching of operation types. Alternatively, a dichroic mirror that enables simultaneous operation of various modules is also possible.

[0064] The beam splitter 4 is preferably implemented in the form of a dichroic beam splitter, and its spectral characteristics are adjustable so that the spectral components of fluorescence emitted from the labeled molecule, which will be detected using the CCD detector 6, pass into the wide-field detection beam path 8, while the remaining spectral components are transmitted if possible. To increase flexibility in using labeled molecules with different emission characteristics, multiple different beam splitters 4 and absorption filters 13 are arranged within the beam splitter module 12 in a manner that allows them to be interchanged, for example, on a filter wheel.

[0065] The aforementioned microscope can play a role in generating super-resolution result images. For this purpose, the control device 28 has an appropriate configuration, which can be achieved, for example, by appropriate programming.

[0066] Figure 2 schematically illustrates the concept of generating super-resolved images within the plane of individual samples using the SIM method. The sample examined by microscope 1 in Figure 1 is imaged repeatedly in a wide field of view, with different illumination conditions set during the imaging process.

[0067] Figure 2 shows a set of raw images 40 from a single sample plane, all representing the same sample region but differing in the optical structure 29, which is transmitted into the sample 2 by structured illumination via the illumination beam path 16 during the recording of the raw images 40. As can be seen from the figure, for example, the transverse periodic optical structure 29 generated by the grating 23 has different orientations and positions in different raw images 40, but the repetition frequency in all raw images 40 is the same (depending on the grating frequency of the grating 23). Overall, this example includes nine raw images 40, which consist of up to three different orientations of the structure 29 and three different displacement positions of the structure 29. The various orientations and displacement positions are classified as phases. Different and more than a number of different phases are also possible, of course, as is known from the previously cited literature regarding the principle of SIM.

[0068] However, structure 29 should be understood as merely an example. In particular, it does not need to be a line structure. It is also possible to further structure a roughly drawn line along those lines. Instead of using the line structure used in the SIM literature initially cited, it is equally possible to use illumination by scanned confocal or confocal detection, as known from the literature by C. Mueller and J. Enderlein, "Image scanning microscopy," Physical Review Letters, 104, 198101 (2010). This principle is called ISM. In that case, of course, there are no other orientations for structured illumination than the nine, and there are several suitable original images obtained from scanning sample 2. Then, each original image 40 corresponds to a specific scanning position, i.e., a specific scanning state during the scanning of the image.

[0069] The control unit 28 calculates a super-resolution result image 50 from a plurality of recorded original images 40. The control unit is here configured, for example, to be switchable, thereby performing both known conventional SIM evaluation or the improved method according to the present invention.

[0070] Figure 3 schematically illustrates an example of structured illumination realized by the grating 23. For example, by diffracting coherent light from the manipulator module 22 through the grating 23, beams corresponding to different diffraction orders are generated. Except for the zeroth-order diffraction and the + / -1st-order diffraction, the remaining diffraction orders are blocked, resulting in three beams interfering with each other within the sample 2, which is also called the back focal plane. The interference of these three beams in the objective lens pupil creates a striped SIM illumination pattern 29 within the sample, and this pattern is also structured axially, i.e., longitudinally, with respect to the optical axis of the objective lens 3 passing through the sample 2, due to the Talbot effect.

[0071] Interferential beams can also be generated by other means, for example, by optical fibers terminated in (or near) the pupil, or by appropriately placed tilt-mirror matrices or lightwave-guiding chips such as those described in U.S. Patent Application Publication 2020 / 0064609A1. Furthermore, the number of beams is not limited to three. The generation of the illumination pattern 29 shown in Figure 3 is an example, but it represents a typical approach, and the various discrete frequency bands present within the illumination pattern 29 are referred to as orders.

[0072] The illumination pattern obtained from the three interference beams can be explained by the following equation.

[0073]

number

[0074] In the equation, a0 is the intensity of the central beam (zero-order diffraction of grating 23), and a is the intensity of the adjacent beams on both sides (1st and -1st-order diffraction of grating 23). Parameters

[0075]

number

[0076] These are the x-, y-, and z- components of the wave vector of the transverse beam,

[0077]

number

[0078] This is the z component of the wave vector of the central beam,

[0079]

number

[0080] This describes the depth dependence of the illumination pattern 29, which is formed by a cosine centered in the central plane, typically near the focal plane of illumination. Such a depth structure is also called a Talbot pattern. φ0 is the phase angle of the cosine profile with respect to the z coordinate. φ is the phase angle of the illumination pattern 15 in the xy plane, i.e., it is a parameter that characterizes individual SIM illumination patterns. These differ from each other only in the value of φ. For example, for the phases of five individual SIM illumination patterns, there are five possible values ​​for φ. The phase angles are preferably evenly distributed over a 180° region within a certain limit. As readily shown by the Fourier transform, the illumination pattern 29 is characterized by signal components having discrete spatial frequency bands corresponding to their order.

[0081] Generally, the illumination pattern I(x,y,z) is multiplied by its optical properties S(x,y,z), taking into account its interaction with sample 2, and then convolved with the detected PSF H(x,y,z) to obtain the measured light distribution I. emWe obtain (x, y, z, z0, φ).

[0082]

number

[0083] In the equation, z0 represents the distance of the sample plane under consideration from the focal plane, and z represents the position of the focal plane. By setting z=0, the sample plane at the focal plane of the illumination is considered, and as a result, the following equation is obtained.

[0084]

number

[0085] The optical properties S(x,y,z) of sample 2 are determined, for example, by the concentration distribution of the luminescent fluorescent dye in sample 2 or by the reflectance of sample 2. The intermediate step of evaluation is preferably performed in the spatial frequency domain. The following equation is obtained by the Fourier transform with respect to x, y, and z.

[0086]

number

[0087] The system of equations is based on the specific lighting pattern I(x,y,z) used, and more specifically, its Fourier transform I f (k x ,k y ,k z ), based on the detection of the PSF H(x,y,z) by the microscope, more specifically the Fourier transform H f (x,y,z) (i.e., OTF) and the intensity I recorded in the original image 40 em More specifically, the Fourier transform of the recorded original image 40 based on (x,y,z).

[0088]

number

[0089] It can be established based on For the three beams interfering in Sample 2, the result of the first step of substituting Equation (1) into Equation (4) is

[0090]

Number

[0091] as follows. Here, let j = 0, 1, 2. A j represents the intensity of the order of the illumination pattern resulting from the prefactor of Equation (1). Therefore, the following applies.

[0092] A0 = a0 2 / 2 + a 2 ; A±1 = a 2 and A±2 = a0a Furthermore, k = (k x , k y , k z ) is the wavevector indicating the order. Equation (5) can be expressed in terms of the order terms.

[0093]

Number

[0094] In the formula, the order D n (k) is defined as follows:

[0095]

Number

[0096] The optical transfer function of the order is For j = -2, 0, 2, OTF j (k x , k y , k z ) = H f ​​​​​​​​​

[0097]

number

[0098] This is the result. To obtain the five orders individually, we use different phase angles φ=φ for the illumination pattern 29. m Therefore, n=5 measurements are required at m=1,...,n. From the n different phase positions of the illumination pattern 29, n equations (6) can be obtained, namely the phase φ m There is one for each of these. Solving this system of equations separates the orders, resulting in n order spatial frequency spectra 41, each of which is of order D j Includes one of (k).

[0099] These are their original positions in the spatial frequency spectrum of the intermediate result image in each case.

[0100]

number

[0101] It is shifted by only a small amount and coupled to the spatial frequency spectrum of the intermediate result image.

[0102]

number

[0103] In the formula, φ(k x ,k y ,k z ) is the apodization function, and ω j (k x ,k y ,k z ) is the weight of the jth order (SR stands for super resolution). The order spatial frequency spectrum 41 or D is determined by the apodization function and weights. jThe operation (k) becomes possible, which helps to improve the results.

[0104] There is another possible approach that replaces apodization functions and weights: φ(k x ,k y ,k z ) For example, super-resolution OTF

[0105]

number

[0106] It can be defined as a Euclidean distance transform of a binary mask over a supported range. In the conventional single-stage SIM method (where the intermediate result image is already the final image), the following weights can be obtained by using a general-purpose Wiener filter (as in the case of Gustafsson et al.).

[0107]

number

[0108] In the equation, the Wiener parameter w and the order of intensity A j This is used. In addition, for example, a notch filter g can be used at the center of each order, which is described by Bozinovic et al. in "Fluorescence endomicroscopy with structured illumination," Optics Express, 2008, Vol. 16, p. 8016.

[0109]

number

[0110] In the formula, the notch filter g(x,y) is, for example, Gaussian-shaped.

[0111]

number

[0112] It can have. The variable numbers a and σ indicate the intensity and width of the suppression.

[0113]

number

[0114] And by calculating the inverse Fourier transform ("iFT"), the resulting image S SR (x, y, z) can be directly checked. In a two-stage approach, for example, the following weights can be used:

[0115]

number

[0116] In the formula, B j These are order-dependent pre-factors. However, other weights can be similarly assigned. Then, the intermediate result image spatial frequency spectrum 44 is shifted to order 41 (D j ) and weight ω j This is a combination of an object manipulated by (for example, a centrally notch-filtered object).

[0117]

number

[0118] As mentioned above, φ(k x ,k y ,kz ) is an apodization function, and g is, for example, a Gaussian notch filter. Artifacts can be reduced by applying this exemplary notch filter g to a higher order (non-zero). In contrast, at the zeroth order, the notch filter acts as a high-pass filter, blocking out-of-focus light.

[0119] Substituting equation (7) into equation (14) yields the intermediate spatial frequency spectrum 44:

[0120]

number

[0121] In other expressions

[0122]

number

[0123] And so the OTF of each degree (weight ω j Degree D j The following overall OTF is obtained, consisting of operations such as (for example, notch filtering at the center).

[0124]

number

[0125] Equation (16) is a convolution expressed in the spatial frequency domain, which is a combination of the shifted and manipulated orders calculated by equation (14).

[0126]

number

[0127] Based on and calculated by formula (17), the total OTF SRIt can be rewritten based on this and solved in various ways using known inverse superposition algorithms, preferably with constraints (e.g., non-negative constraints and smoothness) and possibly regularization, for example, using nonlinear iterative algorithms such as Richardson-Lucie. Other inverse superposition algorithms are also possible. In particular, non-iterative methods can be used.

[0128] For example, prefactor B i Regarding this, there are, for example, two simple possibilities: 1.B i ≡1, and the strength of the degree A i This determines the overall OTF, or 2.B i = 1 / A i The degree strength is A. i Instead, the order spatial frequency spectrum 41(D i Determine the combination of ).

[0129] Other prefactors B i It is also possible. There is no need to calculate the overall OTF as a numerical intermediate result. Rather, for example, the individual sums of the degrees can be used directly in the reweighting in equation (16).

[0130] To fully acquire sample information, the illumination structure 29, which is one-dimensional within the plane of the sample under consideration in this example, must be evaluated in multiple (typically 3 or 5) orientations. The above equation should be extended according to multiple rotational phase angles.

[0131] Figure 4 schematically illustrates another example of structured illumination. For example, there are five beams in the pupil of the objective lens 3, which interfere with each other within the sample 2 to form a three-dimensional periodic optical structure 29. For example, these five beams are generated using a chip (not shown) placed in the pupil. The resulting illumination pattern 29 can be described by the following equation.

[0132]

number

[0133] This illumination pattern has a two-dimensional periodicity (grid pattern) within the plane of the sample under consideration. As a result, there are n=13 different order D values. j This yields the order spatial frequency spectrum 41, which must be extracted and separated within the spatial frequency domain. Therefore, at least 13 original images 40 of each sample plane under consideration are required. For the remaining portion, the evaluation corresponds to the procedure described above for the three interference beams using corresponding equations for the 13 orders.

[0134] The intermediate result image spatial frequency spectra are then shifted to their original positions, resulting in 13 manipulated order spatial frequency spectra 41 (i.e., D j It consists of ).

[0135]

number

[0136] At that time, the overall OTF

[0137]

number

[0138] and

[0139]

number

[0140] And, If j=0,1,3,4,6, then OTF j (k x ,k y ,k z )=H f (k x ,k y ,k z ), For j=2 and j=5,

[0141]

number

[0142] This is the result. Prefactor B j Regarding this, for example, the possibility shown in Figure 3 exists, but other possibilities also exist.

[0143] In all cases (as is the case in Figure 3), the system of equations to be solved (and therefore, the other evaluation steps as well) can be extended to multiple sample planes (and therefore, three spatial dimensions) by introducing, for example, a Dirac comb function that links equations relating to different sample planes to equation (9). Then, both the order spatial frequency spectra 40 and OTF30 become three-dimensional.

[0144] Figure 5 provides a schematic diagram illustrating the procedure of a conventional one-stage SIM evaluation method. As a starting point, step S0 provides 13 original images 40 and OTF 30 from microscope 1. The original images were recorded, for example, at 13 different illumination phase angles in microscope 1.

[0145] In step S1, as described above, a system of linear equations is established and solved based on the original image 40 and the OTF 30. The result is 13 order spatial frequency spectra 41. The zero-order spectrum is conspicuously shown by a thick boundary line. In addition, in step S2, the actual phase angle (position) of each illumination structure 20 is determined from the original image, and the actual grid frequency (generally its repetition frequency) of the illumination structure is determined from the order spatial frequency spectra 41. These are used in subsequent evaluations.

[0146] In step S3, the order spatial frequency spectrum 41 is manipulated to obtain, for example, a filtered order spatial frequency spectrum 42. For example, a notch filter is applied to the center of each order 41 (indicated by the central circle of each spectrum 41). This is used to suppress out-of-focus light in the zeroth order 41 and to suppress grid frequencies in the remaining orders 41. Multiple filters can be applied sequentially or in combination.

[0147] In step S4, the filtered (or otherwise manipulated) order spatial frequency spectra 42 are shifted to their respective original positions in the spatial frequency domain. In step S5, these shifted spectra are combined by a general-purpose Wiener filter and apodization function, weighted based on their illumination intensity and based on the OTF 30, to form the super-resolution resulting image spatial frequency spectrum 44. The weighting based on the OTF 30 is indicated by the differences in the degree of interruption within the gray levels and lines.

[0148] The resulting spatial frequency spectrum 44 is finally transformed from the spatial frequency domain to the spatial domain using an inverse Fourier transform to obtain the super-resolution resulting image 50. The arrow labeled "A" in Figure 5 is used to indicate that data from the original image 40 is transmitted from one evaluation step to the next for further processing. The arrow labeled "B" indicates that the OTF 30 is used only in step S5 with the general-purpose Wiener filter and is not used during the manipulation of the order spatial frequency spectrum 41 in step S3.

[0149] Figure 6 schematically illustrates the procedure of the two-stage SIM evaluation method improved by the present invention, in comparison with the conventional two-stage SIM method. As a starting point, thirteen original images 40 are provided in step S0, which are transformed into the spatial frequency domain, for example, by a Fourier transform. In addition, the actual phase angles (position and orientation) of each illumination structure 29 are determined from the original images, and the actual repetition frequency (grid frequency) of the illumination structure is determined from the local frequency spectrum of the original images. The actual repetition frequency is used as the actual parameter of the illumination structure throughout the subsequent method.

[0150] The original image is recorded, for example, using microscope 1, at 13 different illumination phase angles, and is loaded, for example, from a large-capacity storage means to make it available. In step S1, the PSF of microscope 1 is verified by loading it, for example, from the large-capacity storage means, and the OTF 30 is calculated from it by Fourier transform and copied, resulting in 13 copies. The pre-stored PSF was verified, for example, by measuring a calibration sample with structured illumination.

[0151] In step S2, as described above, a system of linear equations is established and solved based on the original image 40. The result is 13 order spatial frequency spectra 41. The zero-order spectrum is conspicuously shown by a thick boundary line. Exactly one of the OTF copies 30 is assigned to each of the order spatial frequency spectra 41.

[0152] In step S3, the order spatial frequency spectrum 41(D j ) is manipulated, for example, the filtered order spatial frequency spectrum 42(ω i D j) is obtained. For example, a notch filter (g) is applied to the center of each order 41 (indicated by the central circle of each spectrum 41). Depending on the order particularly relevant, it is possible to apply multiple different filters sequentially or in combination. Depending on the type of operation (for example, depending on the filter), only one, several, or even all of the order spatial frequency spectra 41 are operated for each filter. In the improved method of the present invention, in the extended step S3, each OTF copy 30 is operated accordingly as the relevant order spatial frequency spectrum 41, for example, using the same parameters, particularly filter parameters.

[0153] In the conventional step S4, the filtered (or otherwise manipulated) order spatial frequency spectra 42 are shifted to their respective original positions in the spatial frequency domain, and in the conventional step S5, these are weighted based on their illumination intensity (not based on the OTF 30 and without using a Wiener filter). In both the conventional step S5 and the improved S5', the overall OTF is confirmed in that the OTF copy 30 for each order spatial frequency spectrum 42 is shifted, weighted, and added together as the order spatial frequency spectrum 42 associated with each, thereby forming the overall OTF (not shown). Step S5' of the improved method according to the present invention differs from the conventional step 5 in that it uses the OTF copy 30 manipulated in step S3, rather than the unmanipulated start OTF used in the conventional step S5.

[0154] In the conventional step S6, the intermediate image spatial frequency spectrum 44 is reweighted based on the (unmodified) OTF 30 and inversely superimposed by, for example, the Richardson-Lucy iterative method, resulting in the obtained result image spatial frequency spectrum (not shown). The resulting image 50 can then be verified using the inverse Fourier transform.

[0155] Step S6' of the improved method according to the present invention differs from the conventional step S6 in that the reweighting (and therefore inverse superposition) is performed based on the entire OTF, which consists of an OTF copy 30 manipulated as the order spatial frequency spectrum 42. Alternatively, the manipulated OTF copy 30 can be used directly without being pre-combined during the reweighting (inverse superposition).

[0156] The arrow labeled "B" indicates that the OTF copy is manipulated in accordance with the order spatial frequency spectrum 41 (unlike conventional methods). Figure 7 illustrates the improved method according to the present invention for three spatial dimensions in comparison to the aforementioned two-dimensional method. The OTF 30 provided for the three-dimensional case is three-dimensional from the outset. The system of equations for verifying the three-dimensional order spatial frequency spectrum 41 is also established and solved in three dimensions. The manipulation (filtering) of the order spatial frequency spectrum 41 and the OTF copy 30 is also performed in three dimensions, as is the weighting and coupling to form an intermediate result spatial frequency spectrum, as well as the reweighting (ultimately inverse superposition) based on the manipulated OTF 30 by forming, for example, an overall OTF (not shown).

[0157] Figure 8 shows, in succession, the diffraction-limited image (A), the super-resolution image obtained using a conventional one-stage SIM (B), and the super-resolution image obtained using a two-stage SIM according to the present invention (C). All images show the same region of sample 2, allowing for direct comparison. The lower section shows magnified portions of the conventional result image (D) and the result image obtained according to the present invention (E). It is clear that the resolution of images D and E is far better than that of images B and C.

[0158] The imaged sample 2 contained a 60 nm GATTAquant (https: / / www.gattaquant.com) DNA origami using the fluorescent dye Alexa488. Image A and the SIM original image (not shown) were recorded using a 63 × 1.4 plan apochromatic oil immersion objective. The illumination pattern was generated as a 5-beam interference as shown in Figure 4 and projected onto sample 2 in 13 different phases. A 3D stack of 7 planes was recorded, and 13 original images 40 were recorded in each plane. Only one plane is shown.

[0159] Based on the individually identifiable edges of the origami, the improved SIM evaluation according to the present invention achieves a resolution of 60nm, unlike conventional SIMs. [Explanation of Symbols]

[0160] Reference Code List 1. Microscope 2 samples 3. Objective lens 4-beam splitter 5 Tube Lenses 6 CCD detector 7. Optical Microscope Module 8. Wide-field detection beam path 9. Laser scanning module 11 Switchable Mirror 12 Beam Splitter Modules 13 Absorption filter 14 Optical Fiber 15 Laser Modules 16 Collective illumination beam paths 17. Wide-field illumination module 18 Switchable Mirrors 19 TIRF lighting modules 20 Fiber Optics 21 Mirror 22 Manipulator Modules 23 Lattice 24 Displacement drive unit 25 Image field rotator 26 Rotor drive unit 27 Tube Lenses 28 Control device 29 Lighting structure 30 PSF / OTF 40 Original image 41st-order spatial frequency spectrum 42 Filtered order spatial frequencies 43 Weighted and filtered order spatial frequency spectrum 44 Intermediate result image spatial frequency spectrum 50 Result Images A. Sample measurement data (original image) B PSF / OTF data

Claims

1. A method for super-resolution evaluation of multiple microscope images (40) of a sample (2), The steps include providing a plurality of digital original images (40) of the sample (2) that are sequentially recorded using a microscope (1) by illuminating the sample (2) with periodically structured illumination light (29) at different phases, The steps include providing an optical transfer function (30) that represents the imaging of the microscope (1), The steps include confirming multiple order spatial frequency spectra (41) based on the multiple digital original images (40), The steps include: reconstructing a spatial frequency spectrum (44) that can be reconstructed from an intermediate result image based on the plurality of order spatial frequency spectra (41); The steps include weighting the optical transfer function (30) for each confirmed spatial frequency spectrum (41), The process comprises the step of reweighting the intermediate result image into a reconstructible spatial frequency spectrum (44) based on the weighted optical transfer function (30) for each confirmed order spatial frequency spectrum (41), Before or during the reconstruction step, at least one of the plurality of order spatial frequency spectra (41) is manipulated. A method characterized by manipulating the optical transfer function (30) with an algorithm or mathematical operation corresponding to the operation performed on the manipulated order spatial frequency spectrum (41) before or during the reweighting step.

2. The method according to claim 1, wherein the operation includes or is a spatial frequency-dependent operation.

3. The method according to claim 2, wherein the spatial frequency-dependent calculation is spatial frequency-dependent weighting, spatial frequency-dependent filtering, spatial frequency-dependent notch filtering, or spatial frequency-dependent notch filtering centered on the respective coordinate origin.

4. The method according to claim 3, wherein the illumination light (29) in the sample has at least one repetition frequency, and the spatial frequency-dependent filtering includes notch filtering dependent on the at least one repetition frequency, or notch filtering for suppressing at least the at least one repetition frequency.

5. The method according to any one of claims 1 to 4, wherein the illumination light (29) is periodically structured along the optical axis of the recording microscope (1), different digital original images (40) are generated from different planes of the sample, the verification of the order spatial frequency spectrum (41) based on the digital original images (40) from different planes of the sample (2) is performed so that a three-dimensional order spatial frequency spectrum (41) is verified, the optical transfer function (30) is provided in three dimensions, and the reweighting is performed in three dimensions.

6. The method according to any one of claims 1 to 5, wherein the weighting of the order spatial frequency spectrum (41) based on the optical transfer function (30) is not performed during the reconstruction.

7. The method according to claim 6, wherein the weighting during the reconstruction is performed according to the respective illumination intensity.

8. The method according to any one of claims 1 to 7, wherein the operation differs from integral transforms and discrete transforms in a way that the optical transfer function (30) with respect to the operated order spatial frequency spectrum (41) is modified.

9. The method according to any one of claims 1 to 8, wherein the reweighting is performed for iterative inverse superposition or iterative and nonlinear inverse superposition.

10. The method according to any one of claims 1 to 9, wherein the manipulation of the plurality of digital original images (40) is performed before the confirmation of the order spatial frequency spectrum (41), and the confirmation of the order spatial frequency spectrum (41) is performed based on the manipulated plurality of digital original images (40).

11. The operation of the plurality of digital original images (40) is as follows: spatial frequency dependent operations, Spatial frequency-dependent frequency weighting, Filtering, Inverse superposition, or inverse superposition using at least one of the provided unmodified optical transfer functions (30) The method according to claim 10, wherein at least one of the following is the method according to claim 10.

12. The method according to any one of claims 1 to 11, wherein at least one actual parameter of the structured illumination light (29) is determined based on at least one of the plurality of digital original images (40), or based on the plurality of digital original images manipulated as described in claim 10 or 11.

13. The manipulation of the manipulated order spatial frequency spectrum (41) and the manipulation of the optical transfer function (30) for the manipulated order spatial frequency spectrum (41) are as follows: This includes at least one of the respective shifts and their respective weights and weighted sums based on the at least one confirmed actual parameter, or The method according to claim 12, wherein each shift and each weight and weighted sum are based on the at least one confirmed actual parameter.

14. The optical transfer function (30) is, Provided based on at least one verified actual parameter, or The method according to claim 12 or 13, provided by verification of individual images simulated under a simulation of structured illumination using the at least one verified actual parameter, and by SIM evaluation of the individual simulated images from which the optical transfer function (30) is verified.

15. The method according to any one of claims 1 to 14, wherein for each order spatial frequency spectrum (41), a copy of the optical transfer function (30) shifted in accordance with the order spatial frequency spectrum (41) is provided, and in all subsequent method steps, one of each of these copies is used as the optical transfer function (30).

16. The method according to any one of claims 1 to 15, wherein different order spatial frequency spectra contained in the plurality of digital original images (40) or the plurality of digital original images manipulated as described in claim 10 or 11 are demodulated and separated during the confirmation of the order spatial frequency spectrum (41) and shifted in the spatial frequency domain before or during the reconstruction of the spatial frequency spectrum (44) which is capable of reconstructing the intermediate result image, and the separated and shifted order spatial frequency spectrum (41) are combined during the reconstruction of the spatial frequency spectrum (44) which is capable of reconstructing the intermediate result image.

17. The method according to any one of claims 1 to 16, wherein the weighted plurality of optical transfer functions (30) are combined to form an overall transfer function, the reweighting is performed based on the overall transfer function, and in particular the operation of the optical transfer functions (30) with respect to the operated order spatial frequency spectrum (41) is performed before or during the combination.

18. The method according to claim 1, wherein, in the step of providing a plurality of digital original images (40) of the sample (2), when the sample (2) is illuminated in different phases with the structured illumination light (29) using the microscope (1), at least one of the sample light emitted by the sample (2) and the sample light scattered by the sample (2) is sequentially recorded in each digital original image (40) by the detector (6) for each phase, thereby enabling the reconstruction of an intermediate result image (44) having increased resolution compared to the digital original image (40) from the digital original image (40).

19. The method according to claim 18, wherein a two-dimensional spatially resolved detector (6) is used to record the plurality of digital original images (40), and in particular, the aperture for optically cutting the sample is not located in front of the two-dimensional spatially resolved detector (6).

20. The method according to any one of claims 1 to 19, wherein the optical transfer function (30) is operated by the same algorithm or mathematical operation as the operation performed on the operated order spatial frequency spectrum (41).

21. A microscope (1) having a control unit (28) configured to perform the method according to any one of claims 1 to 20, and further having a light source (15), a two-dimensional spatially resolved detector (6) for recording a plurality of digital original images (40) of the sample (2), and means for generating illumination light (29) periodically structured in different phases within the sample (2), wherein the aperture for optically cutting the sample (2) is not positioned in front of the two-dimensional spatially resolved detector (6).

22. A computer program configured to perform the method described in any one of claims 1 to 20 of the Method.

Images