Motion detection in high resolution microscopy with structured illumination mechanism

Abstract

A method for motion detection of a sample in high resolution SIM microscopy is introduced, the method comprising: illuminating the sample with a periodic illumination pattern consisting of a square or hexagonal grating structure of bright dots, generating n raw images of the sample, assigning m diffraction orders to the illumination pattern, m being a positive odd number; using for each of the n raw images a specific position of the illumination pattern in the sample plane, the specific positions being distributed over a phase interval of 0 to 2π; n = m + z, z being a positive odd number, preferably z = 1, such that an even number of raw images is generated; the raw images being numbered in ascending order according to their respective specific position in the phase interval of 0 to 2π; the method further comprising: combining all odd numbered raw images into a first comparative wide field image of the sample; combining all even numbered raw images into a second comparative wide field image of the sample, comparing the first comparative wide field image with the second comparative wide field image for motion detection.

Description

Technical Field

[0001] The present invention relates to motion detection in high-resolution microscopy with a structured illumination mechanism based on so-called SIM technology. Background Technology

[0002] A method for motion detection of a sample using high-resolution SIM microscopy includes generating n raw images of the sample by illuminating it with a periodic illumination pattern, the illumination pattern being designed as a periodic grating structure composed of bright spots and assigned m diffraction orders, wherein, for each of the n raw images, a specific location of the illumination pattern in the sample plane is used, these specific locations being distributed within a phase interval of 0 to 2π.

[0003] The corresponding microscope for performing high-resolution SIM microscopy on a sample includes: an illumination optical path for illuminating the sample, wherein the illumination optical path projects a periodic illumination pattern, designed as a periodic grating structure composed of bright spots, onto the sample and assigns m diffraction orders to the illumination pattern; a detection optical path for wide-field imaging of the illuminated sample onto a detector; and a control device that manipulates the illumination optical path to move the illumination pattern within the sample and read the detector, and is designed to generate n raw images of the sample by illuminating the sample, wherein for each of the n raw images, the illumination pattern has a specific position in the sample plane, and these specific positions are distributed within a phase interval of 0 to 2π.

[0004] The method and the microscope employ the lattice SIM technique, which is described in detail below.

[0005] The so-called "Structured Illumination Microscopy" (SIM) technique has become a mature microscopic imaging method. This microscopic imaging method is mainly based on R. Heintzmann's paper "Saturated patterned excitation microscopy with two-dimensional excitation patterns" (Micron 34 (6-7), 283-291 (2003)) and M. Gustafsson et al.'s paper "Three-Dimensional Resolution Doubling in Wide-Field Fluorescence Microscopy by Structured Illumination" (Biophysical Journal 94, 4957–4970 (2008)), and uses SIM illumination patterns to illuminate the sample. The sample is repeatedly imaged, with the illumination pattern being positioned at different distances from the sample, and a so-called raw image is taken for each such illumination condition. These raw images are then combined in image processing to obtain an improved image of the sample with a resolution higher than that of a standard wide-field image taken using the same microscope. This improvement is called "high resolution" because it surpasses the resolution achievable by conventional wide-field microscopy according to the Abbe criterion.

[0006] The illumination patterns in SIM (Signal Imaging System) are regular and periodic. Each illumination pattern at a specific location is identical in structure, differing only in its position relative to the sample. Initially, periodic stripe patterns were used for illumination, which were then translated and rotated. This technique is now known as 2D SIM. Further improvements were made to increase depth resolution, hence the term 3D SIM. For this, a 3D illumination pattern in the form of a Talbot pattern was projected onto the sample.

[0007] By using this multidimensional, periodic, multi-point SIM illumination pattern composed of bright spots, the acquisition speed of the original image can be significantly accelerated. This pattern can be generated by multiple interfering partial beams arranged in a two-dimensional point-symmetric manner within the illumination pupil in the illumination optical path. This technique is called Lattice SIM. The SIM illumination pattern here is generated by interference patterns produced by, for example, five beams. The SIM illumination pattern contains multiple discrete illumination pattern frequencies, which originate from the interference probabilities of the beams. These discrete spatial illumination pattern frequencies correspond to diffraction orders. Lattice SIM is achieved using the methods described above and microscopy, as detailed in the article "Introduction to Lattice SIM for ZEISS Elyra 7 - Structured Illumination Microscopy with a 3D Lattice for Live Cell Imaging" by J. Siebenmorgen et al., published in the December 2018 ZEISS Technology Note, available at https: / / asset-downloads.zeiss.com / catalogs / download / mic / b05157e2-bcd6-4c82-b9f9-7639b446a74a / EN_WP_Introducing_Lattice_SIM_for_Elyra-7.pdf (Introducing Lattice SIM for ZEISS Elyra 7 - Structured Illumination Microscopy with a 3D Lattice for Live Cell Imaging, ZEISS Technology note, December 2018). (https: / / asset-downloads.zeiss.com / catalogs / download / mic / b05157e2-bcd6-4c82-b9f9-7639b446a74a / EN_WP_Introducing_Lattice_SIM_for_Elyra-7.pdf).

[0008] In a lattice SIM, a single output beam can generate multiple beams. These output beams are diffracted into multiple diffraction orders by a two-dimensional grating located in the intermediate image plane of the illumination optical path. These diffraction orders then appear as interfering beams within the illumination pupil of the illumination beam. The term "diffraction order" refers to the process of generating an illumination pattern through the interference of portions of the beam within the pupil. This term is independent of diffraction at the grating that can be used to provide a portion of the beam. Therefore, the diffraction order corresponds to the illumination pattern itself, rather than to the grating that may be structurally illuminated.

[0009] In a SIM illumination pattern with specific positioning, multiple raw images are captured corresponding to specific variations in beam intensity. These raw images contain moiré fringes with high-frequency components that provide information about the sample. For SIM microscopy, specialized image analysis is required to obtain high-resolution images of the sample from the moiré fringes. Such analysis is described in detail in US11867894B1, which is incorporated herein by reference in its entirety regarding image processing.

[0010] For further discussion of SIM technology, please see: Neil MAA et al., "Method of obtaining optical sectioning by using structured light in a conventional microscope." OpticsLetters 22, 1905–1907 (1997); Heintzmann R. and Huuser T., "Super-Resolution Structured Illumination Microscopy." Chemical Reviews 117 (23), 13890–13908 (2017); Ströhl F. and Kaminski C., "Frontiers in structured illumination microscopy." Optica 3, 667–677 (2016). (2016)); Schermelleh et al.: Super-resolution microscopy demystified, Nature Cell Biology 21, 72-84 (2019); Schropp M. and Uhl R.: Two-dimensional structured illumination microscopy, Journal of Microscopy 256, 23-36 (2014); Ingerman EAGuo, Y, et al.: Signal, noise and resolution in linear and nonlinear structured‐illuminationmicroscopy, Journal of Microscopy 273, 3-25 (2019); Guo, Y, et al.: Visualizing Intracellular Organelle and Cytoskeletal Interactions at Nanoscale Resolution on Millisecond Timescales, Cell 175, 1430-1442 (2018); Frohn JT, et al.: True optical resolution beyond the Rayleigh limit achieved bystanding wave illumination, Proceedings of the National Academy of Sciences 97, 7232-7236 (2000). of Sciences of the United States of America 97, 7232–7236 (2000)); Beck M. et al.: Sub-100-nanometre resolution in total internal reflection fluorescence microscopy, Journal of Microscopy 232, 99–105 (2008); Gliko O.Fiolka R. et al.: Fast two-dimensional standing-wave total-internalreflection fluorescence microscopy using acousto-optic deflectors, Optics Letters 34, 836–838 (2009); Fiolka R.: Clearer view for TIRF and obliqueillumination microscopy, Optics Express 24, 29556–29567 (2016); Young LJ et al.: A Guide to Structured Illumination TIRF Microscopy at High Speed ​​with Multiple Colors, Journal of Visualized Experiments 111 (2016) (2016); Kner P. et al., “Super-Resolution Video Microscopy of Live Cells by Structured Illumination”, Nature Methods, Vol. 6, S.339 (2009) and Guo M. et al., “Single-shot super-resolution total internal reflection fluorescence microscopy”, Nature Methods 15, 425–428 (2018).

[0011] The number of raw images limits the shooting rate. More raw images require more time. In addition, the number of raw images reduces the number of high-resolution images of biological samples, thereby reducing the number of repetitions of the microscopy process, as each exposure causes optical damage until the sample is eventually damaged to an undesirable degree.

[0012] Limited image capture rates also present problems when attempting to detect motion of an object. In existing techniques, this can be achieved, for example, by capturing two wide-field images at different times and calculating the difference image. In regions where the sample has not changed, the difference is zero. In image regions where the sample has changed (i.e., moved, whether by the entire sample or by movement within the sample), a non-zero difference is produced. Because the illumination patterns of specific raw images differ in SIM microscopy, a single raw image is unsuitable for motion detection. Even without motion, the different illumination states on which the raw images are based can cause image content to persist when comparing raw images.

[0013] For SIM microscopy, there are other methods for motion detection. R. Förster's doctoral dissertation, "Detection and Classification of Motion Artifacts in Microscopy with Structured Illumination" (Friedrich-Schiller University Jena, 2018), achieved motion detection in two-dimensional SIM. The paper "Motion artifact detection in structured illumination microscopy for live cell imaging" by R. Förster et al. (Optics Express, Vol. 24, No. 19, 19 September 2016, pp. 22121-22133) calculates two independent wide-field images from different focal planes of a 3D image. Therefore, this algorithm is not suitable for motion detection in a single focal plane. For this reason, lattice SIM still relies on the reconstructed high-resolution image for motion detection. Consequently, the photon count is doubled, or the image acquisition rate is halved compared to image generation without motion detection. Summary of the Invention

[0014] The purpose of this invention is to provide a method for motion detection using microscopy with lattice SIM technology, which does not require doubling the number of photons or the image capture time, and can be operated on a single focal plane.

[0015] This invention is defined in the independent claims. The dependent claims relate to preferred designs.

[0016] To perform motion detection using high-resolution SIM microscopy, n raw images of the sample are required. For this purpose, the sample is illuminated using a periodic illumination pattern, and imaging is performed from a single sample plane. The illumination pattern is designed as a two-dimensional periodic grating structure composed of bright spots in at least one cross-sectional plane. The bright spots are arranged periodically in two orthogonal transverse spatial directions. In many cases, the bright spots also exhibit periodicity in the depth direction; often, illumination patterns in the form of a three-dimensional (3D) grating structure composed of bright spots exist. Diffraction orders correspond to this illumination pattern. These diffraction orders are the discrete spatial frequency components of the illumination pattern; in English literature, they are referred to as "spatial frequency components of illumination intensity," for example, as used by Gustafsson in his 2008 work. This can be understood as the bright spots in the three-dimensional grating structure being physically interpreted as diffraction orders of interfering beams arranged in illumination pupils with corresponding illumination paths, as described above in the overview of SIM techniques. Here, the terms "diffraction order" and "discrete spatial frequency components of illumination pattern" are used interchangeably.

[0017] The depth direction is along the incident direction of the radiation. The lateral direction is perpendicular to this direction. Such a SIM illumination pattern corresponds to m illumination orders (it has m discrete spatial frequency components). m is a positive integer. If the illumination pattern contains zero-order diffraction, an odd number of m diffraction orders will be produced, because zero-order diffraction, as well as pairs of higher-order diffractions (±1, ±2, ...), will occur. However, there are also cases where m is even, i.e., when zero-order diffraction is not produced or is suppressed.

[0018] The sample is imaged along the optical axis onto a detector that receives radiation from the sample plane. The sample is repeatedly illuminated and imaged in such a manner that a raw image of the sample is captured during each illumination and imaging process. A total of n raw images are captured in this manner. These raw images differ in the specific position of the illumination pattern in the sample plane perpendicular to the optical axis. For each raw image, a specific position of the illumination pattern is defined, which is known in SIM technology. A high-resolution wide-field image of the sample is generated from the n raw images. For each of the n raw images, a specific lateral position of the illumination pattern is used (i.e., specifically moved in the sample plane, so that the plane position is the same for all raw images; thus, there is a fixed focal plane for capturing all raw images). These specific positions have a phase shift relative to a reference position, generally covering a phase range of 0 to 2π.

[0019] Then, an even number of original images are captured; this can be achieved in two ways: one is to neither generate nor suppress zero-order diffraction (where m and n are a priori even numbers), and the other is to increase the number of original images to an even number (where m is odd and the number of original images increases to an even number n). In the latter case, the number of original images n is chosen such that it is both greater than the (odd) diffraction order m and even. Therefore, n = m + z original images are captured, where z is a positive odd number. Particularly preferred is z = 1.

[0020] The even-numbered original images are numbered. This is done by arranging them in ascending order based on their position within the phase interval (0 to 2π). The original image closest to the lower limit of the phase interval (0) is assigned the number "1". The next closest is assigned the number "2", and so on, until the last number "n" is assigned to the original image closest to the upper limit of the phase interval (2π). These original images, numbered "1", "2", "...", "n" in this manner, are then combined, with all odd-numbered original images (hereinafter referred to as "odd-numbered original images") combined to form the first comparison wide-field image. All even-numbered original images (hereinafter referred to as "even-numbered original images") are combined to generate the second comparison wide-field image. These comparison wide-field images do not necessarily have high resolution, but because they all originate from similar lighting conditions (one is an odd-numbered original image, and the other is an even-numbered original image), they are very convenient for motion detection. Motion detection can be performed by comparing these two comparison wide-field images.

[0021] For microscopes used for high-resolution SIM microscopy imaging of samples, an illumination optical path is similarly configured to project a periodic illumination pattern consisting of bright spots onto the sample. The microscope's detection optical path images the illuminated sample from the sample plane onto a detector for wide-field imaging. Specifically, a control device, including a processor and electronic memory, manipulates the illumination optical path to set specific positions of the illumination pattern and reads the detector accordingly. The control device is designed to generate the raw image described in the method and perform motion detection.

[0022] The method features described herein refer to the control device designed to perform these features in the implementation. The method may include a processor appropriately programmed for this purpose. Similar method features are disclosed when describing the operation of the control device.

[0023] Therefore, ideally, this method / microscope requires only one additional raw image to obtain an even number of raw images. Then, by combining the even and odd numbers of raw images, two relatively wide-field images are generated for motion detection. For example, for m=13 diffraction orders, 13 raw images would be required without motion detection. Now, in the preferred embodiment, only one additional raw image is needed (of course, 3, 5, 7, etc., additional raw images can also be generated) to perform motion detection on an even number of raw images. Compared to conventional methods, this method significantly reduces the increase in photon exposure and image acquisition time. Conventional methods require two high-resolution wide-field images for lattice simulation techniques, i.e., 2n raw images. For the 13th diffraction order, existing techniques require 26 raw images, while the scheme described herein requires only 14. Therefore, the photon exposure time and image acquisition time are only about 54% of those of existing techniques.

[0024] The key to this is that the number of original images required for reconstruction increases from an odd number to an even number. This allows the original images to be alternately combined to form first and second comparative wide-field images, resulting in two comparative wide-field images with the same illumination conditions, which can then be used for motion detection, for example, by calculating the difference.

[0025] During the generation of the original images, all original images have the same number of diffraction orders (discrete spatial frequency components of the illumination pattern). Since the original images that combine to form the first and second wide-field images are identical in terms of the phase they cover at various locations in the illumination pattern, the sum of all odd and even original images is equivalent to the unmodulated wide-field image. Therefore, the comparative wide-field image used for motion detection is actually a single wide-field image.

[0026] The generation of a high-resolution wide-field image (using lattice SIM technology according to an established image analysis method) can depend on the presence of any interfering motion. This allows us to verify whether a predetermined maximum motion has been exceeded. If motion detection indicates that the predetermined maximum motion has been exceeded, a high-resolution wide-field image can be generated additionally or alternatively, and a motion indicator can be added. This motion indicator can specifically indicate the image region where motion was detected, since a predetermined difference threshold between the first and second comparative wide-field images has been exceeded within that region. In this embodiment, a high-resolution wide-field image is always generated, for example, when motion is detected, the image region where motion was detected is marked. This is particularly advantageous when dealing with motion within a sample, such as when imaging biological structures. In this way, image regions unaffected by the motion can be displayed at high resolution, while the user can understand which image regions may be affected by motion interference.

[0027] By first capturing the original images with even-numbered positions and then capturing the original images with odd-numbered positions, the time interval between the two relatively wide-field images can be maximized, thereby improving the sensitivity of motion detection. This can be achieved by first generating the original images with odd (or even) numbered positions and increasing the interval between each position, instead of scanning the specific positions within the 0 to 2π phase interval sequentially from one interval boundary to another, and then generating the original images with even (or odd) numbered positions.

[0028] To ensure that the illumination conditions of two comparative wide-field images are as consistent as possible, it is preferable to directly space the positions in the consecutive original image numbering with a phase step of 2π / n. This allows each position to equally cover the phase range of 0 to 2π. The comparative wide-field images also exhibit particularly high sensitivity to motion. However, deviations from this equally spaced distribution are permissible, with a maximum of 2%, preferably a maximum of 1%, particularly preferably a maximum of 0.5%, and especially preferably a maximum of 0.1%. This reduces the accuracy requirements of the illumination optical path. Combinations are possible; that is, some original image positions can be equally spaced at 2π / n intervals, while other adjacent original image positions can have deviations within the aforementioned limits.

[0029] Wide-field images are preferably generated from all the original images captured; in any case, they should include both even-numbered and odd-numbered images. In the prior art evaluation methods described above, it is preferable to use all generated original images. However, the computational cost can also be reduced using a method known in US110867894B1, which requires fewer original images.

[0030] In the simplest case, the first and second comparison wide-field images are generated by merging the corresponding original images. This method is particularly desirable when using the aforementioned equally spaced phase interval illumination patterns. Especially preferred is that, in the case of a non-perfectly uniform arrangement, the original images are weighted using a weighting function before summing. This weighting function preferably depends on the deviation of a particular original image from the ideal uniform phase distribution of the corresponding illumination pattern. This weighting function can provide a multiplier, for example, which can be derived from the Penrose pseudo-inverse of the mixing matrix.

[0031] Of course, the features described above, as well as those explained below, can be used not only in the specified combinations, but also in other combinations or individually, without departing from the scope of the invention. Attached Figure Description

[0032] The present invention will now be described in more detail with reference to the embodiments and accompanying drawings, which also disclose the basic features of the invention. These embodiments are for illustrative purposes only and should not be construed as limiting the invention. For example, the description of embodiments comprising multiple elements or components should not be construed as implying that all such elements or components are necessary for implementation. Furthermore, other embodiments may include alternative elements and components, fewer elements or components, or additional elements or components. Unless otherwise specified, elements or components of different embodiments may be combined with each other. Modifications and variations described with reference to the embodiments are also applicable to other embodiments. To avoid repetition, the same or corresponding elements in different figures are referred to by the same reference numerals and are described only once. In the drawings:

[0033] Figure 1 A block diagram illustrating an implementation of the lattice SIM microscopy method;

[0034] Figure 2 The microscope shown is used to generate raw images, which are obtained using... Figure 1 The method is used for evaluation;

[0035] Figure 3 Show Figure 2 A schematic diagram of the illumination device for a medium-sized microscope;

[0036] Figure 4 a) to Figure 4 f) shows a schematic diagram of the diffraction order (discrete spatial frequency components of the illumination pattern) used to generate the original image. Detailed Implementation

[0037] Figure 1 A block diagram is shown for a method of motion detection based on lattice SIM microscopy. The method mainly includes steps S0 to S5, wherein step S3 is the step of performing motion detection. Figure 2A microscope 2 suitable for method execution is schematically shown, operating according to lattice SIM technology. Microscope 2 images a sample 4 located on a sample carrier 6. Imaging is achieved via an objective lens 8, which projects the object field in the sample 4 onto a detector 10 through a tube lens (not shown in detail). An illumination device 14 illuminates the sample 4 through the objective lens 8 via a beam splitter 12, thereby achieving structured illumination adjustable with respect to position within the sample 4, as is known with respect to lattice SIM technology. Microscope 2 is controlled by a control device 16, primarily comprising a processor 18.

[0038] The method begins with an initial step S0. In step S1, sample 4 is imaged multiple times, where the sample is illuminated using a multi-point SIM illumination pattern, as is known for lattice SIM technology. As is known for lattice SIM technology, the original images differ in terms of the illumination pattern. For each location of the illumination pattern, an original image is generated from the sample plane by imaging sample 4 onto detector 10. Therefore, at the end of step S1, n original images are obtained. All original images are images of the same sample in the same sample plane, where the sample is illuminated with the illumination pattern. The difference between the original images is that the illumination pattern is specifically located. The specific location of the illumination pattern is specified by an illumination pattern function, which defines or represents the position of the illumination pattern. The value n refers to the number of locations, which, due to lattice SIM technology, are typically the displacement positions of the illumination pattern within sample 4. All original images are from the same sample plane, i.e., imaged with a fixed focal plane.

[0039] The illumination pattern has illumination pattern frequencies, which are generated, for example, by interference. As mentioned above and further elaborated below, the illumination pattern frequencies represent m diffraction orders, i.e., the discrete spatial frequency components of the illumination pattern. This interpretation is generally applicable and reasonable, even if the generation methods differ. Considering that diffraction orders are complex conjugates in frequency space, the discrete frequencies of these m diffraction orders correspond to the spatial illumination pattern frequencies. Therefore, for each pattern frequency defined in position space, there always exists a pair of diffraction orders (above the zeroth order) with the same value in frequency space. This is known in the prior art, for example, in the publications of Gustafsson (2008). Due to the use of lattice SIM technology, m is always an odd number (and, of course, an integer), because there are always pairs of higher diffraction orders (±1, ±2, ±3, etc.) besides the zeroth order diffraction. A common case is m=13.

[0040] Each of the n original images is assigned one of n illumination pattern functions, which represents the position (typically an offset) relative to the reference position of the illumination pattern, expressed in phase. Using the n original images, n illumination pattern functions, and PSF (point spread function) generated in step S1, deconstruction is performed in step S3 using a Wiener filter or an iterative method, as explained in more detail in, for example, US11867894B2.

[0041] The total number of original images n is equal to the total number of illumination pattern functions n, which is the number of the specific positions (usually shifted positions) contained in all the original images in the sample plane, because each illumination pattern function precisely describes one of the specific positions used, in terms of the phase of the illumination pattern position.

[0042] In the prior art, the number n of original images is chosen to correspond to the diffraction order m. If the zeroth-order diffraction and higher diffraction orders are expressed as even numbers, then m in the prior art is also odd, because the zeroth-order diffraction is supplemented by at least + / -1 and + / -2-order diffractions. If the illumination pattern does not contain zeroth-order diffraction, then m is even.

[0043] To simplify motion detection as much as possible, in all implementations, an even number of original images are generated in step S1. If m is odd, then n = m + z, where m is not only odd but also an integer, and z is also an even number of original images. In the simplest and preferred case, z = 1.

[0044] The fact that n is an even number is crucial for motion detection. Based on this even number of original images, motion detection is performed in step S2. For this purpose, the original images are numbered according to the specific positions of the illumination patterns used for the corresponding original images. As previously mentioned, this sequence is arranged with a phase interval of 0 to 2π. The original images numbered in this manner are then combined, wherein the odd-numbered original images constitute the first comparison wide-field image, and the even-numbered original images constitute the second comparison wide-field image. For the commonly used lattice SIM technique, it is preferable to capture 14 original images, and combine the original images numbered "1", "3", "5", "7", "9", "11", and "13" into the first comparison wide-field image, and combine the original images numbered "2", "4", "6", "8", "10", "12", and "14" into the second comparison wide-field image. Thus, in the corresponding comparison wide-field images, all SIM sorting is eliminated, resulting in two independent wide-field images that can be used for motion detection because they have the same illumination conditions. This process requires an even number of original images. As described below, the synthesis is preferably achieved by merging the corresponding original images.

[0045] In the simplest case, wide-field images can be compared by calculating the difference. After calculating the difference, the image region containing the image content will be affected by motion. This information can be used to skip step S3 and proceed directly to step S5, thus avoiding the calculation of a high-resolution wide-field image and saving complex signal analysis that would otherwise result in an image with motion artifacts; alternatively, a wide-field image can be calculated, but the detected motion can be labeled. For example, this label can mark image regions where motion was detected, for example, because the difference image contains the image content there.

[0046] When calculating the high-resolution wide-field image, the result of step S3 in step S4 is the high-resolution image of sample 4. This image is based on the original image. However, unlike existing techniques, this method uses an even number of original images, and the high-resolution wide-field image is either unaffected by motion artifacts (no motion detected) or provides a hint indicating the presence of motion artifacts, preferably indicated by marking the affected image regions. Of course, this process can be repeated to generate another high-resolution image, check for motion artifacts, and selectively mark them as hints.

[0047] exist Figure 3 An example of illumination implementation is schematically illustrated. Illumination device 20 provides an initial beam 22, for example, from a laser (not shown in detail). The initial beam 22 is split into five beams 24a to 24e by means of a two-dimensional crosshair grating 24 arranged in the intermediate image plane. These beams are optically shaped by suitable lenses or other optical elements (not shown in detail) to form five light spots 30a to 30e in the entrance pupil 26 (also referred to as the "back focal plane" in the English literature) of objective lens 8. The center point 30a corresponds to the central beam, and four adjacent points 30b to 30e are arranged in a square around the center point, corresponding to the side beams. These five points in the objective pupil 26 interfere with each other to form a SIM illumination pattern 32 on sample 4 by objective lens 8. Here, Figure 3 The SIM illumination pattern 32 is shown in the form of a Fourier image. In spatial representation, the SIM illumination pattern is a two-dimensional pattern of bright spots arranged in a square grating. Figure 3 The Fourier image of the SIM illumination pattern 32 shows that the radiative interference from the illuminated points 30a to 30e produces 13 diffraction orders. This will be explained later. This pattern of light points in the illumination pupil 9 produces the pattern of illumination intensity in the sample plane as shown in the Fourier representation. Each point in the Fourier representation of illumination pattern 32 corresponds to a diffraction order. The farther the point is from the center, the higher its diffraction order. Points 30a to 30e can also be generated in other ways, such as by an optical fiber terminating in the pupil or a suitably arranged array of tilting mirrors.

[0048] The specific illumination pattern of the original image is generated by moving the grating 24 perpendicular to the optical axis. This grating is located in the intermediate image plane of the illumination optical path, thereby shifting it relative to the vertical axis of the square grating. This shift occurs in… Figure 3 Arrow 34 indicates the shift perpendicular to the principal axis of the square crosshair grating. This shift occurs in 14 steps within the phase range of 0 to 2π, resulting in 14 specific positions and consequently 14 different modulations of the intensity of bright spots 30a to 30e in the objective pupil 26. This is further referenced below. Figure 4 a) to 4f) will be explained in more detail.

[0049] The implementation described herein uses a square grating only as an example. Hexagonal grating structures are also applicable.

[0050] A specific original image is generated by illuminating sample 4 with a specific illumination pattern 32 and recording the light reflected by sample 4; or, under fluorescence excitation, recording the fluorescence emitted by sample 4.

[0051] Figure 3 The illumination pattern 32 shown is generated by interference from partial beams located within the pupil and focused at points 30a to 30e; this is merely an example, but it has become the standard method for historical reasons. In this method, the different discrete frequencies present in the illumination pattern 32 are called diffraction orders. In signal engineering, these frequencies represent carrier frequencies. Diffraction orders above the fundamental / zero-order diffraction frequency correspond to the frequencies of the spatial illumination pattern, but they are complex conjugates; therefore, above the fundamental frequency, there are always two diffraction orders with the same value. The fundamental frequency corresponds to the zero-order diffraction. In addition, there are 12 additional diffraction orders.

[0052] In lattice SIM, 13 raw images are required for image processing at the highest resolution (fewer raw images can be used under some acceptable constraints; see US11867894B1). However, in these embodiments, 14 raw images are generated instead of 13, i.e., 14 shift positions of grating 24 are used. Therefore, these embodiments require more raw images compared to the prior art. This is not a necessary condition for achieving high resolution; the 14th raw image is captured for motion detection (n=14).

[0053] Figure 4 This illustrates the meaning of diffraction orders when generating images using lattice SIM technology. Figure 4a) shows five diffraction orders 30a to 30e. For ease of illustration, each order can have five different amplitudes. The figure is presented in Fourier plot form. Amplitudes are represented by lowercase Greek letters. Due to the motion of grating 24, the possible amplitudes are simplified to two independent base values ​​α and β, which can be positive or negative. For ease of discussion, the amplitude of the zeroth-order diffraction is normalized to a unit value. The motion of grating 24 along the two principal axes (e.g., Figure 4 (b) The intensity of the outer points 30b to 30e was modulated. Diffraction produced 13 orders of the SIM illumination pattern 32 (see b). Figure 3 These levels are displayed in the sub-levels. Figure 4 c) and also shows according to the sub Figure 4 b) The corresponding amplitude of the bright spot interference (corresponding to) Figure 3 (Points in the central pupil 26). Up to this point, the amplitude of each point can be freely adjusted by moving the grating 24 accordingly. Of course, the same applies if fiber bundles (for example) are used instead of grating 24 to generate partial beams 28a to 28e, as described above. By choosing n as the number of original images, the number of adjustment possibilities is very limited, and when using a fixed step size based on the fundamental quantity p, the adjustment possibilities are reduced to, for example... Figure 4 The case shown in d) is an example. Positive or negative integer coefficients represent the diffraction order m. For clarity, in... Figure 4 Marked separately in e). Figure 4 As shown in f), the result is a diffraction order pattern in the Fourier representation, which indicates the characteristics by the step size between the diffraction order m, the number of original images n, and the successive positions of the illumination pattern 32 (or, if generated by a two-dimensional grating, also a two-dimensional grating 24).

[0054] In one implementation, it can be ensured that zero-order diffraction 30a is absent. In this case, the required number of original images is automatically even. For this purpose, a phase grating 40 without zero-order diffraction can be used (see related literature). If the spatial light modulator (SLM) is in an off-axis configuration, a phase pattern corresponding to the grating without zero-order diffraction can be displayed. Diffraction order 30a corresponds to zero-order diffraction. Alternatively, an aperture 42 can be introduced in the pupil plane of the illumination beam, which (additionally) blocks the beam 28a of zero-order diffraction 30a. It is preferable to move the aperture 42 mechanically so that zero-order diffraction 30a is blocked only when necessary. Instead of the aperture 42, a pinhole mirror can also be used. For this purpose, as Figure 2 As shown, the mirror 12 can be placed in the illumination pupil in a particularly simple manner, and the center hole 44 is provided so that the portion of the beam 28a that causes zero-order diffraction 30a is not reflected onto the objective lens 8.

[0055] Regarding the applicable theoretical background:

[0056] All original images have the same SIM order m. The wide field order is m=0.

[0057] Each SIM level m depends only on the "phase" it is modulated in each original image n. And change.

[0058] A uniform phase step size means that the phase of the SIM-order image always has the same value between the two original images. change.

[0059] therefore, ,in, It is the offset of level m.

[0060] To ensure that all sample regions in the comparison field image are illuminated by the same total intensity, the SIM order in each comparison wide-field image must completely vanish: .

[0061] Then, the sum of all the original images contains only wide-field orders, and is therefore equivalent to a wide-field image.

[0062] The lighting patterns can be evenly distributed within a phase range of 0 to 2π. Therefore, the applicable conditions are:

[0063] (in, ),therefore

[0064] The original images are divided into two groups according to image number n.

[0065] n is even → n even ={0, 2, 4, ..., 12}

[0066] n is odd → n odd ={1, 3, 5, ..., 13}

[0067]

[0068]

[0069] Generate two independent wide-field comparison images.

[0070] If phase For an exact match, the two sets of original images (even / odd) are merged. Otherwise, errors may occur, but these errors are tolerable if the required motion sensitivity allows.

[0071] As an alternative, the various lighting patterns are not uniformly distributed within the phase range of 0 to 2π. Therefore, the applicable approach is:

[0072] Phase in each original image Uneven distribution.

[0073] The original images are divided into two groups according to image number n.

[0074] n is even → n even ={0, 2, 4, ..., 12}

[0075] n is odd → n odd ={1, 3, 5, ..., 13}

[0076] To ensure that the sum of all original images in the two subgroups contains only wide-field orders, the corresponding final phases (12 and 13) in the two groups must be selected as follows:

[0077]

[0078]

[0079] This will result in two separate wide-field images.

[0080] Because of the deviation from the phase-equal distribution, summing the original image cannot completely eliminate the SIM order.

[0081] The greater the deviation, the more SIM information is retained in the wide-field image.

[0082] Summation can be extended using the weighting factor of the Penrose pseudoinverse of the mixing matrix. This significantly improves the separation between the SIM level and the wide-field level.

[0083] This method works well when the error is as low as 1%, and works exceptionally well when the error is as low as 0.1%.

[0084] Phase map generation:

[0085] All five orders of the grating amplitude spectrum in a lattice SIM can be obtained by moving the grating in the intermediate image plane. Figure 4 a) and take any value.

[0086] Moving one grating period along this plane corresponds to a phase of 2π.

[0087] Central level The phase can be normalized to 0.

[0088] If only the two principal directions of the lattice are allowed ( Figure 4 b) As for the direction of motion, there are only two independent phases, α and β.

[0089] The choices for α and β are as follows: ,

[0090] The phase of the SIM order (autocorrelation of the amplitude order) shows 13 orders, each with a different phase. Figure 4 c).

[0091] This ensures the required phase. ( Figure 4 c to Figure 4 f).

[0092] Therefore, the grating must be moved along its two lattice directions respectively by the lattice period. or .

[0093] The order of the 14 grating positions is irrelevant. By capturing the original images with even-numbered positions first, followed by those with odd-numbered positions, the time interval between the two wide-field images of the comparison samples can be maximized, thereby maximizing the sensitivity of motion detection.

[0094] By merging the two sets of original images, a wider field image can be obtained, as shown in the following calculations:

[0095] prove

[0096] = = = = =

[0097] = = = = 0

[0098] Where: q = 7; = ; n = 2k; p =

[0099] and: =

[0100] prove

[0101] = = = = =

[0102] = = = = 0

[0103] Where: q = 7; = ; r = n = 2k; p =

[0104] Another result: = .

Claims

1. A method for motion detection of a sample (4) in high-resolution SIM microscopy, wherein, The method includes: n original images of the sample (4) are generated by illuminating the sample (4) with a two-dimensional periodic illumination pattern (32) consisting of bright spots in at least one cutting plane, wherein, for each of the n original images, the illumination pattern (32) is used at a specific location in the sample plane, and these specific locations are distributed in a phase interval of 0 to 2π. Its features are: Generate an even number of original images. The original image is assigned an ascending number based on its specific position within the phase interval from 0 to 2π. The method further includes: combining all odd-numbered original images into a first comparison wide-field image of sample (4), combining all even-numbered original images into a second comparison wide-field image of sample (4), and comparing the first comparison wide-field image with the second comparison wide-field image to perform motion detection.

2. The method according to claim 1, characterized in that, Assign m diffraction orders to the lighting pattern (32), where m is a positive odd number, and choose n = m + z, where z is a positive odd number, preferably z = 1.

3. The method according to claim 2, characterized in that, When motion detection indicates that the motion has not exceeded a predetermined maximum, a high-resolution wide-field image is generated from the original image; and / or when motion detection indicates that the motion has exceeded a predetermined maximum, a high-resolution wide-field image is generated from the original image and a motion indicator is set for the high-resolution wide-field image, wherein the motion indicator preferably represents an image region in which a predetermined difference threshold between a first comparison wide-field image and a second comparison wide-field image is exceeded.

4. The method according to any one of the preceding claims, characterized in that, The specific positions of the lighting pattern (32) are set sequentially in such a way that the original images of all even-numbered numbers are captured first, and then the original images of all odd-numbered numbers are captured.

5. The method according to any one of the preceding claims, characterized in that, For all the original images that are numbered sequentially, the specific positions of the illumination pattern (32) are spaced at a phase step of 2π / n, or at a phase step of at most 2%, preferably at most 1%, and particularly preferably at most 0.5% of the deviation from 2π / n.

6. The method according to any one of the preceding claims, characterized in that, In the step of generating a high-resolution wide-field image, the high-resolution wide-field image is generated from the original images with even-numbered and odd-numbered identifiers.

7. The method according to any one of the preceding claims, characterized in that, The combination of the odd-numbered original images and the even-numbered original images is achieved through merging, and optionally, during the merging process, the odd-numbered original images and the even-numbered original images are each multiplied by a specific weighting factor.

8. A microscope for high-resolution SIM microscopy imaging of a sample (4), the microscope comprising: The illumination path for illuminating the sample (4) is a periodic illumination pattern (32) consisting of bright spots illuminating the sample (4) at least in the cutting plane. The detection optical path used to image the irradiated sample (4) wide-field onto the detector; and A control device, which manipulates the illumination path to move the illumination pattern (32) in the sample (4) and read the detector, is designed to: n original images of sample (4) are generated by illuminating sample (4), wherein the illumination pattern (32) has a specific position in the sample plane for each of the n original images (30), and these specific positions are distributed in the phase interval from 0 to 2π. Its features are: The control device is also designed to: Take an even number of raw images; Based on the specific positions of the illumination pattern (32) within the phase interval from 0 to 2π, the original image is numbered in ascending order, and All odd-numbered original images are combined into a first comparison wide-field image of sample (4), all even-numbered original images are combined into a second comparison wide-field image of sample (4), and the first comparison wide-field image and the second comparison wide-field image are compared for motion detection.

9. The microscope according to claim 8, characterized in that, The illumination pattern (32) is assigned m diffraction orders, where m is a positive odd number, and the control device selects n=m+z, where z is a positive odd number, preferably z=1.

10. The microscope according to claim 8 or 9, characterized in that, The control device is designed to: generate a high-resolution wide-field image from even-numbered and odd-numbered original images when motion detection indicates that the motion has not exceeded a predetermined maximum; and / or generate a high-resolution wide-field image from even-numbered and odd-numbered original images and set a motion indicator for the high-resolution wide-field image when motion detection indicates that the motion has exceeded a predetermined maximum; wherein the motion indicator preferably represents an image region in which a predetermined difference threshold between a first comparison wide-field image and a second comparison wide-field image is exceeded.

11. The microscope according to any one of claims 8 to 10, characterized in that, The control device is designed to set the specific positions of the lighting pattern (32) sequentially in such a way that the original images of all even-numbered numbers are captured first, and then the original images of all odd-numbered numbers are captured.

12. The microscope according to any one of claims 8 to 11, characterized in that, The control device is designed to set the position of the illumination pattern (32) sequentially in such a way that the positions of the illumination patterns (32) of all the original images that are numbered sequentially are spaced apart by a phase step of 2π / 14, or by a phase step of at most 2%, preferably at most 1%, and particularly preferably at most 0.5% of the deviation from 2π / 14.

13. The microscope according to any one of claims 8 to 12, characterized in that, The control device is designed to combine seven odd-numbered original images and seven even-numbered original images by merging them, and optionally multiply the seven odd-numbered original images and seven even-numbered original images by a weighting factor.

Images