Method for obtaining optical slice images of a sample and device suitable for such a method

Abstract

A method for obtaining an optical section image of a sample is presented. The method comprises providing an illumination beam through an imaging lens such that the illumination beam is focused at a focal plane of the imaging lens; obtaining a plurality of images of the sample. The obtaining comprises providing the illumination beam at a plurality of lateral positions on the focal plane and obtaining each image at each lateral position of the illumination beam such that an intensity of the illumination beam on a portion of the sample at the focal plane varies for each of the plurality of lateral positions. The method further comprises detecting a signal collected via the imaging lens using a detector; and constructing an optical section image based on the plurality of images. The constructing comprises obtaining a plurality of signal values from the portion of the sample from the plurality of images; evaluating a threshold value for the portion; and evaluating a pixel value by integrating a portion of the plurality of signal values based on the threshold value.

Description

Technical Field

[0001] This manual relates to optical imaging. Background Technology

[0002] Optical microscopy of biological samples is limited by defocused light, which degrades image quality and optical contrast. Confocal microscopy addresses this problem by using a pinhole in the optical path to reject light from defocused areas of the sample, thus providing optical sectioning. In confocal microscopy, the choice of pinhole size is crucial for achieving good optical sectioning without unduly reducing the amount of light collected from the focusing plane. For conventional confocal microscopy, the fixed pinhole size results in a fixed level of optical sectioning and limits the versatility of these microscopes. For example, the optimal pinhole size depends on various aspects of the biological sample. Therefore, fixed pinhole sizes and the corresponding fixed level of optical sectioning have always been one of the limitations of confocal microscopy.

[0003] To overcome the limitations of physical pinholes, Neil et al. (Optics Letters, 1997, Vol. 22, No. 24, pp. 1905–1907) proposed an alternative slicing method based on a combination of structured illumination and post-processing of recorded images. In this method, a single spatial frequency grid pattern is used to illuminate the sample. Three images are captured, with the grid shifted at a predetermined distance between the images, determined by the spatial frequency of the grid pattern. These three images are then used to calculate an optical slice image. However, in this case, the optical slice is inevitably limited by the grid properties; the spatial frequency determines not only how imaging is performed but also the level of optical slice achieved. Therefore, just as the slicing in confocal microscopy is determined by the choice of pinhole, the slicing according to Neil et al.'s method is determined by the choice of grid pattern.

[0004] Given the above, there is still a need for improved methods to generate optical slice images. Summary of the Invention

[0005] According to one aspect of the present invention, a method for obtaining an optical slice image of a sample is provided. The method includes: providing an illumination beam through an imaging lens such that the illumination beam is focused at a focal plane of the imaging lens; and obtaining multiple images of the sample. The obtaining includes: providing the illumination beam at multiple lateral positions on the focal plane, and obtaining each image at each lateral position of the illumination beam such that the intensity of the illumination beam on a portion of the sample at the focal plane varies with respect to each of the multiple lateral positions. The method further includes: detecting a signal collected via the imaging lens using a detector; and constructing an optical slice image based on the multiple images. The construction includes: obtaining multiple signal values ​​from the portion of the sample from the multiple images; evaluating a threshold for the portion; and evaluating pixel values ​​by integrating a portion of the multiple signal values ​​based on the threshold.

[0006] Advantageously, unlike the method taught by Neil et al. (where slicing is determined by the spatial characteristics of structured lighting), this method allows for the application of variable-level slicing to the same set of images in post-processing. In other words, starting from the same multiple images, the degree of slicing can be varied by changing the threshold. Essentially, by changing the threshold, the amount of focus signal contributing to the final sliced ​​image can be tuned.

[0007] Furthermore, compared to the method of Neil et al. (in which the imaging analysis method is again determined by the spatial characteristics of the structured illumination), the method of this invention is less demanding on the imaging steps. For example, compared to the method of Neil et al., this method is relatively less sensitive to the nature of the illumination and is less demanding on the ability of the imaging system to precisely change the relative positions of the illumination beam and the sample between images.

[0008] In some embodiments, the configuration further includes providing a slicing factor for determining the slicing degree of the optical slice image. This portion is evaluated by subtracting a threshold value from each of a plurality of signal values ​​or by dividing each of a plurality of signal values ​​by a threshold value.

[0009] In some implementations, the value of the threshold is a function of the threshold and the slice factor.

[0010] In some implementations, the threshold value is the product of the threshold and the slice factor.

[0011] In some embodiments, the illumination beam is focused at a focal plane, and the illumination beam includes a periodic pattern having a spatial period defined in at least one direction within the focal plane.

[0012] In some implementations, the illumination beam comprises a periodic array of linear focal points focused at a focal plane.

[0013] In some implementations, the illumination beam comprises an array of focused light spots focused on a focal plane.

[0014] In some implementations, the illumination beam comprises a combination of a linear focal point focused at the focal plane and a focused spot.

[0015] In some implementations, evaluating the threshold includes: providing a statistical function; and evaluating the threshold by performing the statistical function on multiple signal values. The statistical function receives multiple data values ​​and outputs a value based on the skewness of the multiple data values.

[0016] In some implementations, the statistical function outputs the median of the received data values, such that the threshold is the median of the multiple signal values.

[0017] According to another aspect of the invention, an apparatus for obtaining an optical slice image of a sample is provided, the apparatus comprising: an imaging lens; an illumination source configured to provide an illumination beam through the imaging lens such that the illumination beam is focused at a focal plane of the imaging lens; a detector configured to detect a signal collected from the sample via the imaging lens; and a control unit configured to: obtain multiple images of the sample by providing illumination beams at multiple lateral positions on the focal plane, such that the intensity of the illumination beam on a portion of the sample at the focal plane varies with respect to each of the multiple lateral positions; construct an optical slice image based on the multiple images by the following steps: obtaining multiple signal values ​​from the portion of the sample from the multiple images; evaluating a threshold of the portion; and evaluating a pixel value by integrating a portion of the multiple signal values ​​based on the threshold.

[0018] In some implementations, the control unit is also configured to: receive a slice factor for determining the slice extent of an optical slice image; and evaluate a portion of a plurality of signal values ​​by subtracting a threshold value from each of the plurality of signal values ​​or by dividing each of the plurality of signal values ​​by a threshold value.

[0019] In some implementations, the value of the threshold is a function of the threshold and the slice factor.

[0020] In some implementations, the threshold value is the product of the threshold and the slice factor.

[0021] In some embodiments, the device further includes a spatial modulator configured to provide an illumination beam focused at a focal plane. The illumination beam includes a periodic pattern having a spatial period defined in at least one direction within the focal plane.

[0022] In some implementations, the illumination beam comprises a periodic array of linear focal points focused at a focal plane.

[0023] In some implementations, the illumination beam comprises an array of focused light spots focused on a focal plane.

[0024] In some implementations, the illumination beam comprises a combination of a linear focal point focused at the focal plane and a focused spot.

[0025] In some implementations, the control unit is also configured to evaluate a threshold by performing a statistical function on multiple signal values. The statistical function receives multiple data values ​​and outputs a value based on the skewness of the multiple data values.

[0026] In some implementations, the statistical function outputs the median of the received data values, such that the threshold is the median of the multiple signal values.

[0027] According to another aspect of the invention, a method is provided for obtaining an optical slice image of a sample based on multiple images of the sample, the method comprising: obtaining multiple signal values ​​from a portion of the sample from the multiple images; evaluating a threshold of the portion; and evaluating a pixel value by integrating a portion of the multiple signal values ​​based on the threshold. The multiple images include optical images of the sample obtained by illuminating the sample with a focused illumination beam at multiple lateral positions at a focal plane, each optical image being obtained at each lateral position such that the intensity of the illumination beam on the portion of the sample at the focal plane varies for each of the multiple lateral positions.

[0028] In some implementations, the method further includes providing a slicing factor for determining the slicing degree of the optically sliced ​​image. A portion of the multiple signal values ​​is evaluated by subtracting a threshold value from each of the multiple signal values ​​or by dividing each of the multiple signal values ​​by the threshold value.

[0029] In some implementations, the value of the threshold is a function of the threshold and the slice factor.

[0030] In some implementations, the threshold value is the product of the threshold and the slice factor.

[0031] In some implementations, evaluating the threshold includes: providing a statistical function; and evaluating the threshold by performing the statistical function on multiple signal values. The statistical function receives multiple data values ​​and outputs a value based on the skewness of the multiple data values.

[0032] In some implementations, the statistical function outputs the median of the received data values, such that the threshold is the median of the multiple signal values. Attached Figure Description

[0033] Some embodiments of the invention will now be described by way of example with reference to the accompanying drawings, wherein:

[0034] Figure 1This is a schematic diagram illustrating an exemplary embodiment of an optical imaging system.

[0035] Figure 2 This is a flowchart of a method for obtaining optical slice images of a sample.

[0036] Figure 3a This is a schematic diagram illustrating the construction of an optical slice image.

[0037] Figure 3b A diagram illustrating the construction of an optical slice image is shown.

[0038] Figure 3c This is a schematic diagram illustrating the construction of an image stack from two sets of images.

[0039] Figure 4 shows an optical slice image with varying degrees of optical slice.

[0040] Figure 5a and Figure 5b This is a schematic diagram illustrating an exemplary embodiment of an optical imaging system that combines structured illumination and wide-field illumination. Detailed Implementation

[0041] To facilitate variable optical slices, it is necessary to physically alter the mechanical pinhole and rescan the sample.

[0042] This limits the speed of imaging and can cause sample degradation through, for example, photobleaching or phototoxicity. Furthermore, there is a physical limitation on the number of pinholes that can be added to the device.

[0043] To overcome the limitations of physical pinholes, alternative methods based on a combination of structured illumination and post-processing of recorded images have been realized. One such method, as described by Neil et al. (Optics Letters, 1997, Vol. 22, No. 24, pp. 1905–1907), involves projecting a line pattern onto the sample and then moving the phase of the illumination. This method computationally rejects defocused light. In this configuration, different levels of optical slicing are achieved by varying the illumination pattern and repeating scans of the sample.

[0044] This specification provides a method and an apparatus that enable continuously adjustable optical slicing while requiring single-group scanning implemented using fixed hardware, without prior knowledge of illumination patterns or hardware calibration.

[0045] Figure 1 This is a schematic diagram illustrating an exemplary embodiment of an optical imaging system.

[0046] The optical imaging system 100 is configured to perform optical detection, imaging, and study of a sample or specimen 10.

[0047] The optical imaging system 100 includes an imaging lens 110, an illumination source 120, a detector 130, and optical elements 140.

[0048] In some embodiments, the optical imaging system 100 may include a spatial light modulator 150.

[0049] Illumination source 120 emits excitation beams 121 and 122. Optical element 140 is configured such that at least a portion of the excitation beams 121 and 122 is at least partially reflected when incident on optical element 140 and guided to imaging lens 110.

[0050] The optical properties of sample 10 allow for optical imaging at the wavelength of illumination source 120. When excited by excitation beams 121 and 122, sample 10 can emit light according to the detection mode or detection scheme. For example, sample 10 can emit light via fluorescence, Raman scattering, and Rayleigh scattering. Each of these schemes may require different configurations of illumination source 120, detector 130, and optical elements 140.

[0051] Optical element 140 is configured to provide an optical path toward detector 130 for light collected from sample 10 via imaging lens 110, the optical path being separate from the optical paths of excitation beams 121, 122. Examples of optical element 140 may include beam splitters, polarizing beam splitters, dichroic mirrors, and polychromatic mirrors; however, optical element 140 is not limited to these examples.

[0052] In some embodiments, when sample 10 includes fluorescent molecules, optical element 140 may be configured as dichroic or polychromatic, configured to reflect light at wavelengths of excitation beams 121, 122 incident on optical element 140 and transmit light at at least one wavelength of fluorescence emitted from sample 10. Fluorescence collected by imaging lens 110 may reach detector 130 after transmission at optical element 140.

[0053] In some embodiments, when the sample 10 is to be detected via scattering, the optical element 140 may be configured as a beam splitter or polarization beam splitter at the wavelengths of the excitation beams 121, 122 and the scattered light from the sample 10. Both the reflected excitation beams 121, 122 and the scattered light may be transmitted through the optical element 140 before reaching the detector 130.

[0054] Examples of imaging lenses 110 include oil immersion lenses, air lenses, aspherical lenses, and achromatic lenses; however, imaging lenses 110 are not limited to these examples. Imaging lenses 110 can be configured to provide tight focusing (such as diffraction-limited spots) of excitation beams 121, 122, and simultaneously provide efficient collection of emissions from sample 10.

[0055] In some embodiments, the direction of the incident excitation beam 121, 122 can be along... Figure 1 The position of sample 10 is adjusted in the z-direction so that a portion or part of sample 10 is within the focal depth of imaging lens 110 or equivalently within the focal volume of imaging lens 110, close to the focal plane 111 of imaging lens 110. This portion of sample 10 will be referred to as being at the “focal point” or “focal plane” of imaging lens 110.

[0056] The range of sample 10 in the z-direction can exceed the depth of focus of imaging lens 110. In this case, optical slicing can be performed by effectively collecting signals from the focal plane 111 while suppressing background signals from the defocus plane 113. After obtaining the z-direction slice image, i.e., the optical slice image or the optical slice image as a planar image in the xy plane within the depth of focus of imaging lens 110, the z-direction position of sample 10 can be adjusted to obtain further z-direction slice images to finally construct a three-dimensional optical image of sample 10.

[0057] In some embodiments, the illumination source 120 may be arranged such that the excitation beam 121 is located at the focal point of the imaging lens 110 when it enters the imaging lens 110.

[0058] For example, the excitation beam 121 can be fed into a substantially collimated imaging lens 110 such that the excitation beam 121 is focused at the focal plane 111 or the focal point of the imaging lens 110. In this case, the width of the excitation beam 121 in the xy plane is minimized near the sample 10 or at the focal plane 111. For the remainder of this specification, this illumination mode will be referred to as confocal illumination.

[0059] In some embodiments, the illumination source 120 may be arranged such that the excitation beam 122 is focused at or near the back focusing plane 112 of the imaging lens 110, and illuminates the sample 10 upon entering the imaging lens 110, such that a relatively large area in the xy plane is simultaneously illuminated. In this case, the excitation beam 122 is not focused at the focusing plane 111 of the imaging lens 110. This illumination mode will be referred to as wide-field illumination in this specification.

[0060] In some implementations, wide-field illumination can be combined with spatial light modulator 150 to enable multiple focused illuminations at focal plane 111 or focused illuminations with arbitrary structures in the xy plane. For example, by using spatial light modulator 150, illumination beam 122 can be arranged as multiple focused spots or a linefoci array at focal plane 150. In this case, spatial light modulator 150 can be positioned such that the plane of spatial light modulator 150 from which excitation beam 122 is reflected is imaged at focal plane 111.

[0061] The spatial light modulator 150 can be any optical element capable of applying spatial variation modulation to the transverse mode of the excitation beam 122. Examples of the spatial light modulator 150 may include a mask pattern, a liquid crystal device, and a digital micromirror device (DMD) mounted on a translation stage; however, the spatial light modulator 150 is not limited to these examples.

[0062] At the focal point of the imaging lens 110, or within the focusing space of the imaging lens 110, a portion of the sample 10 is optically imaged onto the detector 130, such as... Figure 1 As shown.

[0063] Parts of the sample 10 outside the focal point of the imaging lens 110 or outside the focal space of the imaging lens 110 (e.g., at the defocus plane 113) are not imaged onto the detector 130. Figure 1 The signal from a portion of the sample 10 at the defocus plane 113 is imaged at a plane that is displaced from the detection plane of the detector 130, and the signal from the defocus plane 113 is displaced from the optical axis and occupies a wider area on the detector 130 as it is projected onto the detector 130.

[0064] It should be understood that, in addition to Figure 1 In addition to the components described herein, additional optical devices for imaging may be introduced as needed. For example, when the imaging lens 110 is corrected to infinity, a barrel lens may be included within the detector 130 or in the beam path between the optical element 140 and the detector, such that the sample interface 150 and the sample 10 are optically imaged on the detector 130.

[0065] In some implementations, detector 130 may be a single-pixel detector, such as an avalanche photodiode (APD), a photomultiplier tube (PMT), or a superconducting nanowire single-photon detector (SNSPD). A single effective area of ​​detector 130 can be used to detect all light collected by imaging lens 110 and transmitted to detector 130. This type of detector can be used with confocal illumination, where excitation beam 121 is tightly focused onto sample 10. In this case, because the signal region at the plane of detector 130 is larger than the effective area of ​​detector 130, the portion of sample 10 at the defocus plane 113 results in a weaker signal intensity than the portion of sample at the focusing plane 111.

[0066] In some implementations, detector 130 may be a multi-pixel detector or a multi-array detector, such as a CCD, EMCCD, and sCMOS. This type of detector can be used with wide-field illumination, where excitation beam 122 illuminates sample 10 over a wider domain, and the light collected over the illuminated area is optically imaged onto detector 130, across multiple pixels. In this case, the portion of sample 10 at defocus plane 113 results in a signal distributed across a larger number of pixels and shifted from the optical axis compared to the signal of the portion of sample 10 at focal plane 111. In this case, the portion of sample 10 at defocus plane 113 results in a signal with lower intensity at a given pixel of detector 130 compared to the signal of the portion of sample 10 at focal plane 111.

[0067] Figure 2 This is a flowchart of a method for obtaining optical slice images of a sample.

[0068] In step 210, the sample 10 is illuminated by patterned illumination beams 121 and 122.

[0069] To obtain an image, patterned illumination illuminates a predetermined portion of the sample to be imaged on detector 130. Illumination beams 121, 122 of any pattern on the xy plane at focal plane 111 can be used, as long as the portion of the sample to be imaged on detector 130 is covered by repeating illumination at one or more of the lateral translation positions of the patterned illumination beams 121, 122.

[0070] As will be discussed in the next step, multiple images will be obtained by shifting and patterning the illumination beams 121, 122. In one of the multiple images, the patterned illumination beams 121, 122 cover a predetermined portion of the region of the sample to be imaged on the detector 130.

[0071] Patterned illumination beams 121 and 122 are focused at focal plane 111, such that the depth of focus of the patterned illumination beams 121 and 122 along the z-axis provides varying degrees of illumination to sample 10. The intensity is maximized at focal plane 111. This contrasts with some conventional optical slicing methods that employ patterned illumination.

[0072] In some embodiments, the illumination beam 121 may be in the form of confocal illumination. In this case, to obtain one of a plurality of images, the illumination beam 121 can be patterned by selectively or periodically modulating the amplitude of the illumination beam 121 while raster scanning the focused spot of the illumination beam 121 at the focal plane 111 on a z-axis slice of the sample 10 in the xy plane. The amplitude modulation of the illumination beam 121 can be achieved by a spatial light modulator 150 or an additional amplitude modulator (not shown), such as an acousto-optic modulator placed in the beam path of the illumination beam 121.

[0073] At step 220, multiple images of sample 10 are obtained by shifting the patterned illumination beams 121 and 122.

[0074] After acquiring an image or frame, the patterned illumination beams 121 and 122 can be moved laterally or shifted to a new position to acquire another image or frame, so that the patterned illumination beams 121 and 122 illuminate the previously unilluminated parts of the sample 10.

[0075] Multiple images or frames are obtained by repeatedly translating and moving patterned illumination beams 121 and 122.

[0076] In some implementations, the illumination beam 122 may be in the form of a linear focal array at the focal plane 111 of the spatial light modulator 150.

[0077] In some implementations, the illumination beam 122 may be in the form of a focused beam array using a spatial light modulator 150.

[0078] In some embodiments, the illumination beam 122 can be arranged as any combination of a linear focal point and a focused spot on the focusing plane 111. Images or frames are recorded at each location of the patterned illumination beams 121, 122 by collecting the emission from the sample 10 via the imaging lens 110 and passing it through the detection detector 130.

[0079] In some implementations, the entire area of ​​the sample 10 imaged on the detector 130 can be covered by moving the patterned illumination beams 121, 122 a predetermined number of times.

[0080] In some implementations, when using a single pixel detector 130, each image or frame is obtained by translating the sample 10 along the xy plane.

[0081] The collection or arrangement of multiple images obtained in step 220 will also be referred to as image stacking.

[0082] Method 200 is not limited to line scanning or raster scanning. Any pattern modulation that covers the entire field of view in a single scan can be used. Non-periodic random patterns can also be used.

[0083] For each pixel (x,y) of an image or frame j, where j is from 1 to n, and n is the number of frames obtained by stacking each image.

[0084] Because the patterned illumination beams 121, 122 are arranged such that the illumination intensity varies for each image or each frame j, a portion of the sample 100 of each pixel (x, y) to be imaged onto image j has a lateral extent or size in the xy plane that is smaller than the period or interval of the distance between each illumination spot or line or any other illumination area of ​​the patterned illumination 121, 122 at the sample plane.

[0085] In some embodiments, the illumination beam 122 may be in the form of a linear focal array at the focusing plane 111, and two sets of orthogonal line data may be obtained by arranging two sets of mutually orthogonal linear focal points in the xy plane.

[0086] In some implementations, before calculating the optical slice images as discussed below, each image in the image stack can be corrected to reduce image artifacts in the optical slice images by taking into account imperfections in the illumination and imaging systems. From the image Ni from the image stack, N can be used to... i,c =N i / c to obtain the corrected image N i,c Where c is the correction factor, c = median (N) i / N m ), where N m It is the median image, or the median image through a low-pass filter, or the average image, or the average image through a low-pass filter.

[0087] At step 230, a threshold is evaluated for each pixel based on multiple images.

[0088] The threshold value t takes into account the signal values ​​of all acquired images and the desired z-axis slicing degree.

[0089] By considering n pixel values ​​p j (x,y) is used to determine the focused light.

[0090] pixel value pj (x,y) shows the distribution within a given image stack, which contains information about the ratio of the signal (signal of interest) from the focus space at focus plane 111 to the signal (background) from outside the focus space at defocus plane 113.

[0091] For example, pixels primarily containing signals from within the focus space and pixels primarily containing signals from outside the focus space may result in different distributions of pixel values ​​with different statistical properties: pixels corresponding to the in-focus object on the sample will have signal levels that change rapidly as the pattern is scanned and have a large intensity difference, while pixels corresponding to the out-of-focus object will have signal levels that change slowly with a lower amplitude. As a result, when the signal levels of pixels corresponding to the in-focus object are plotted relative to the frame index, the graph will have narrower peaks, while the signals from pixels focusing on the out-of-focus object will have wider peaks.

[0092] Signal values ​​that will be included in the optical slice image are referred to as “focus signals” in this specification, and signal values ​​that will not be included in the optical slice image are referred to as “defocus signals”.

[0093] A threshold can be used to determine the focused light for each pixel (x, y). The focused signal is estimated as n pixel values ​​p. j The sum of the differences between each pixel value in (x,y) and the threshold. If this sum results in a negative number for any pixel (x,y), then all pixels with a negative sum can be set to zero, as in adjusting the contrast of an image in a traditional image viewing program.

[0094] The sum of the resulting focus signals across image stacking is used for the final optical slice image f. t The intensity value of (x,y).

[0095] f t (x,y): The resulting optical slice image (where t is the threshold value). The choice of the threshold value t determines the degree of optical slicing.

[0096] f t (x,y)=(p j (x,y)-t) is the summation over all j.

[0097] Determine the threshold value t for each pixel.

[0098] In some implementations, through the background function I bg The evaluation of (x,y) or the threshold, multiplied by a single constant slice factor CSF, gives a threshold value t for each pixel. The degree of slicing in the resulting optically sliced ​​image can be controlled by varying the constant slice factor CSF. Background function I bg(x, y) corresponds to a statistical method that is consistently applied to all pixels. The background function I is evaluated for each pixel. bg (x,y). A single constant slice factor is common to all pixels and varies with different slice degrees.

[0099] In this case, the intensity of the resulting optical slice image is given by the following formula:

[0100]

[0101] In some implementations, it can be achieved by receiving p j (x,y), I bg The resulting optical slice image can be constructed using any function of (x,y) and the constant slice factor CSF. In particular, the function constructing the optical slice image can be defined such that the constant slice factor CSF cannot be decomposed. For example, the intensity of the resulting optical slice image can instead be given by:

[0102]

[0103] or

[0104]

[0105] Typically, the resulting optical slice image can be obtained by receiving p j (x,y), I bg (x,y) can be constructed from any function of CSF and can have properties that CSF cannot be decomposed from functions.

[0106] Background Function I bg Some examples of (x,y) include the following:

[0107] (1)I bg (x,y)={p j The median of 'j' on (x,y)}

[0108] (2)I bg (x,y)={p j The median of (x,y)} at 'j' is then used to perform a Gaussian blur on the adjacent pixels (i.e., blurring at 'x,y').

[0109] (3)I bg (x,y)={p j The average value over 'j' of (x,y)} is then used to perform a Gaussian blur on the adjacent pixels (i.e., blurring over 'x,y').

[0110] (4)I bg (x,y)=({p jThe average value at 'j' of (x,y)}-({p j (Standard deviation of (x,y)} on 'j')

[0111] (5)I bg (x,y)={p j The Pth percentile at 'j' in (x,y)}

[0112] The threshold value t (especially the background function I evaluated above) bg (x,y) can be used to calculate the pixel values ​​p of the stacked frames. j It is sensitive to the distribution of (x,y).

[0113] In some implementations, the background function I bg (x, y) can be a statistical function whose output depends on the skewness of the distribution of the input data. For example, the background function I bg (x, y) can be determined as pixel values ​​p that are sensitive to distribution skewness. j The median of (x,y). Optical slice images can be obtained by assigning greater weight to pixels j with a larger ratio of focused to defocused light.

[0114] In contrast, the mean depends only on the sum and quantity of elements, and is therefore insensitive to skewness, and may therefore not provide any meaningful optical slices. For another example, the background function I... bg (x,y) can include any nth-order moment of the input data.

[0115] The defocus signal is digitally selected from each pixel value to change the degree of optical slicing. This is achieved by determining a constant slice factor (CSF). The constant slice factor (CSF) is a value that determines the degree of optical slicing across the entire image and is the same for all pixels.

[0116] Once multiple images are acquired to form a frame stack and the background function is evaluated for each pixel from the frame stack, the degree of optical slicing can be continuously changed or tuned by altering the constant slice factor CSF, allowing for the acquisition of different levels of optical slicing. Therefore, a single set of scans can provide multiple images of optical slices at different levels. Furthermore, the continuous tuning of the optical slices is independent of the pattern of the patterned illumination beams 121, 122.

[0117] In step 240, an optical slice image is constructed.

[0118] As described above, an optically sliced ​​image is generated by subtracting multiple backgrounds from the image and summing them. Specifically, at each pixel, f t (x,y)=(p j (x,y)-t) is the summation over all j. This is achieved by using the background function I.bg (x,y) is multiplied by a constant slice factor CSF to give the value t of the threshold.

[0119] This step is used to construct an optical slice image using the median as a threshold, corresponding to cases (1) and (5) when P = 50, which will be... Figure 3a and Figure 3b Detailed examples are provided below.

[0120] Figure 3a This is a schematic diagram illustrating the construction of an optical slice image.

[0121] Figure 3a An image stack 300 obtained at step 220 is shown, comprising 10 individual images or frames 300-1, 300-2, 300-3, 300-4, 300-5, 300-6, 300-7, 300-8, 300-9, and 300-10. Each image 300-1, 300-2, 300-3, 300-4, 300-5, 300-6, 300-7, 300-8, 300-9, and 300-10 was obtained by illuminating the sample 10 using an array of linear focal points 311, as shown in the illumination pattern image 310.

[0122] exist Figure 3a In the example shown in illumination pattern image 310, the patterned illumination beams 121, 122 at the focal plane 111 comprise a linear parallel array of linear focal points 311 extending from the positive y-direction to the positive x-direction in the xy-plane at the focal plane 111, having a negative slope in the xy-plane, as... Figure 3a As shown in the diagram, the linear foci 311 are regularly spaced apart and parallel to each other.

[0123] Each individual image 300-1, 300-2, 300-3, 300-4, 300-5, 300-6, 300-7, 300-8, 300-9, and 300-10 is obtained by translating the array of linear focal points 311 10 times, thus producing 10 images. For each image, the patterned illumination beams 121 and 122 at the focal plane 111 are moved approximately one-tenth of the distance between two adjacent linear focal points 311, such that shifting the patterned illumination beams 121 and 122 11 times corresponds to the original pattern of the illumination beams 121 and 122.

[0124] exist Figure 3aIn the example, detector 130 is a multi-pixel detector. In the following description, detector 130 will be assumed to be a multi-pixel detector such as a CCD to explain the construction of optical slice images. However, the same operating principle applies to the use of single-pixel detectors such as photomultiplier tubes (PMTs) or avalanche photodiodes (APDs), where each pixel of the multi-pixel detector is considered a single-pixel detector.

[0125] For a given pixel 301, in Figure 3a The white squares marked in each image 300-1, 300-2, 300-3, 300-4, 300-5, 300-6, 300-7, 300-8, 300-9, and 300-10 correspond to one pixel of detector 130. This pixel 301 is imaged onto a portion of sample 10 at focal plane 111 via imaging lens 110.

[0126] The lateral extent of this portion of the sample imaged onto the xy plane of pixel 301 is less than or equal to or less than the period of the linear focal point 311 or the distance between two adjacent linear focal points 311. Therefore, when the patterned illumination beams 121, 122 are laterally shifted parallel to the xy plane, the intensity of the illumination beams 121, 122 on the portion of the sample corresponding to pixel 310 changes.

[0127] Signal diagram 320 illustrates the signal collected and recorded at pixel 301. The x-axis 321 of signal diagram 320 represents the frame number, where each frame contains individual images 300-1, 300-2, 300-3, 300-4, 300-5, 300-6, 300-7, 300-8, 300-9, and 300-10, thus having 10 distinct points. The y-axis 322 of signal diagram 320 represents the signal value, which corresponds to the intensity of the signal received at pixel 301 of detector 130. As discussed above, since the intensity of illumination beams 121 and 122 varies for each frame, the signal generated from the portion of the sample imaged onto pixel 301 varies accordingly. The relationship between the signal intensity and the intensity of illumination beams 121 and 122 depends on the optical imaging mode and the specific sample. However, it should generally be understood that a greater illumination intensity results in a greater signal intensity or pixel value at pixel 301. The units of the y-axis 322, which represent signal strength, are specific to the type of detector 130. Examples of units include total number of photoelectrons, total number of photons, voltage, or photocurrent; however, units are not limited to these examples.

[0128] exist Figure 3aIn the example, signal graph 320 shows that the signal value, or pixel value, is maximized at frame 5. This means that at the positions of the patterned illumination beams 121, 122 corresponding to frame 5, one of the linear focal points 311 is closest to the portion of sample 10 imaged at pixel 301. The signal value at each frame includes the signal generated from the portion of sample 10 within the focus space at focus plane 111 (which is the signal of interest), and also includes the signal generated from the portion of the sample outside the focus space at defocus plane 113 (which is the background). Of the 10 frames plotted in signal graph 320, the ratio of the signal of interest to the background is also maximized at frame 5. Figure 1 As shown, the background signal from the defocus plane 113, which is imaged at a plane shifted from the detector 130, is distributed across a larger number of pixels of the detector 130, while the signal of interest from the focus plane 111 is concentrated on a single pixel of the detector 130.

[0129] Figure 3b A diagram illustrating the construction of an optical slice image is shown.

[0130] By connecting the signal values ​​in signal diagram 320, a signal curve 323 for each pixel (e.g., pixel 301) of image stack 300 can be obtained.

[0131] As discussed in step 230, in order to determine the focus signal, a threshold value of 324 is set. Figure 3b The middle line is represented as a horizontal line. Figure 3b In the example, the threshold value is set to the median of the 10 pixel values ​​of pixel 301 in image stack 300, and the constant slice factor CSF is set to 1.

[0132] As discussed in step 230, in order to evaluate the focus signal, the threshold value 324 is subtracted from each pixel value.

[0133] The degree of optical slicing in the optically sliced ​​image can be adjusted by correspondingly adjusting the threshold value 324. For example, the constant slice factor CSF can be multiplied by the initial threshold value 324, such as the median pixel value, to adjust the threshold value 324. For example, when the constant slice factor CSF is greater than 1, the value subtracted from the pixel value is larger, so the final focus signal value is smaller than when the threshold value 324 is the initial threshold value 324, which corresponds to a higher degree of slicing.

[0134] The slice level can be gradually changed from an open aperture equivalent configuration to a closed aperture equivalent configuration.

[0135] Figure 3c This is a schematic diagram illustrating the construction of image stacking from two sets of images.

[0136] Unlike the direct computation of optical slice images as discussed in steps 230 and 240, the obtained two sets of multiple images 331, 332 can be processed to produce a stack of sparse multifocal images 333. This can be achieved, for example, by performing pairwise minimum intensity projection between two sets of orthogonal line data 331, 332. As another example, this can be achieved by performing M... k =N i,1 +N j,2 -|N i,1 -N j,2 |To obtain a stack of sparse multifocal images 333, where N i,1 and N j,2 It is the image of two sets of orthogonal line data 331 and 332, and M k This is an image formed by stacking multifocal images. The methods for forming the stack of multifocal images 333 are not limited to these examples. The stack of multifocal images 333 can be used for further imaging resolution enhancement algorithms, such as Image Scanning Microscopy (ISM), Multifocal Excited, Pinholed, Scaled and Summed (MPSS), Multifocal Structured Imaging Microscopy (MSIM), Structured Imaging Microscopy (SIM), Super-Resolution Microscopy, and Deconvolution Microscopy.

[0137] Figure 4a An optical slice image showing varying degrees of optical slice detail is presented.

[0138] From such Figure 2 The stacking of frames obtained from a set of scans, as described in Figure 3, provides access to optical slice images at various degrees. That is, the same original image data can be processed again to obtain optical slice images at different degrees. Figure 4a An example of an image is shown, starting from an open pinhole equivalent configuration (OS level 1) and progressively closing the pinhole size within 10 steps to a maximized slice (OS level 10) corresponding to higher x, y, z resolution and a smaller depth of field. Imaging was performed using a 561 nm laser excitation and a 660–700 nm filter via fluorescence to a fixed Cos-7 unit. The imaging lens 110 used was a 100 x 1.39 NA objective. The detector 130 used was an sCMOS camera. The camera exposure time was 33 ms for each image. With 2 × 2 pixel binning, the size of each effective pixel was 0.117 μm, and the field of view of the image was 50 × 80 μm.

[0139] Figure 4b An example of using a higher degree of optical slicing to improve image resolution is shown.

[0140] Imaging of a 0.1 μm diameter tetraspeck using a 561 nm laser excitation source 120 is performed, and a filter for collecting fluorescence signals in the 660-700 nm range is placed in the detection path. The imaging lens 110 is a 100×, 1.39 NA oil immersion objective. The detector 130 is an sCMOS camera with a 33 ms camera exposure time and 2×2 pixel binning. Each binned pixel corresponds to 0.117 μm in both the x and y directions.

[0141] The three images in the first row, namely image 401, image 402, and image 403, were obtained using three different levels of optical slices (OS level 1, OS level 2, and OS level 3). The field of view of the images is 50 μm × 80 μm in the xy plane parallel to the sample plane.

[0142] The three images 411, 412, and 413 in the second row are portions of the first to third images 401, 402, and 403, respectively. They are images of a single fluorescent bead with a field of view of 1.3 μm × 1.3 μm in the xy-plane. Images 411, 412, and 413 in the second row show that the lateral resolution in the xy-plane improves with increasing slice size. For example, the image of bead 413 corresponding to OS level 3 exhibits a smaller lateral size than the image of bead 412 corresponding to OS level 2.

[0143] The three images 421, 422, and 423 in the third row are images obtained from a single fluorescent bead in the yz plane, acquired by repeatedly capturing the image while changing the position of the focusing plane in the z-direction, which is the axial direction of the imaging lens 110. The field of view of images 421, 422, and 423 is 1.3 μm × 2 μm in the yz plane. Each pixel is 0.117 μm in the y-direction and 40 nm in the z-direction. Fifty images were captured in 40 nm steps in the z-direction. Images 421, 422, and 423 in the third row show that the axial resolution in the z-direction improves with higher slice degrees. For example, the image of bead 423 corresponding to OS level 3 exhibits a smaller axial range than the image of bead 422 corresponding to OS level 2.

[0144] Figure 4a and Figure 4b The data presented in this paper demonstrate that the method and apparatus described herein are capable of obtaining different levels of optical slices in a single scan without prior knowledge of the pattern of the illumination beams 121 and 122.

[0145] Figure 5a and Figure 5b This is a schematic diagram illustrating an exemplary embodiment of an optical imaging system that combines structured illumination and wide-field illumination.

[0146] Optical imaging system 500 includes Figure 1 An integrated combination of the optical imaging system described in [reference to a document] and the compact microscope described in [reference to a document] such as US10,330,904B2.

[0147] The optical imaging system 500 includes an imaging lens 510, an illumination source 520, a detector 530, optical elements 540, and a spatial light modulator 550, such as... Figure 1 As shown. In particular, the optical imaging system 500 can operate in a mode for structured illumination or in a mode for standard wide-field illumination.

[0148] The optical imaging system 500 can switch between two illumination modes by moving two mirrors on slider 560 while keeping the rest of the optics in the proper position. Figure 5a An optical imaging system 500 in structured illumination mode is shown. Figure 5b An optical imaging system 500 in standard wide-field illumination mode is shown.

[0149] Both modes share the illumination source 520. The illumination light emitted from the illumination source 520 is a combination of one or more beams emitted from fiber-coupled lasers 520-1, 520-2, 520-3, and 520-4. For example, lasers with four different wavelengths of 405nm, 488nm, 561nm, and 635nm can be collimated by a lens and combined into the same beam path by a dichroic mirror.

[0150] In such Figure 5a In the structured illumination pattern shown, slider 560 is positioned such that a collimated beam emitted from illumination source 520 is projected onto spatial light modulator 550, such as a digital micromirror device (DMD). The plane of spatial light modulator 550 is imaged onto the focal plane of imaging lens 510 at a magnification of approximately 55 through additional optics between imaging lens 510 and spatial light modulator 550. For example, Figure 5a An additional lens 570 placed for this purpose is shown.

[0151] In such Figure 5b In the wide-field mode shown, slider 560 is positioned such that a collimated beam emitted by illumination source 520 is guided to lens 570, which focuses the beam onto the back focusing plane of imaging lens 510. Lens 570 and a mirror are on a second slider 580, configured to shift the illumination beam parallel to the optical axis of imaging lens 510 to achieve angular illumination, such as objective-based HILO (Highly Tilt and Laminated Optics) illumination and objective-based TIRF (Total Internal Reflection Fluorescence) illumination.

[0152] The detection channel is the same for both lighting modes. Figure 5a and Figure 5b In the example, optical element 540 is a dichroic mirror configured to reflect the excitation beam emitted by illumination source 520 toward imaging lens 510 and transmit the fluorescence of interest toward detector 530. The detection path is split into two paths by long-pass mirror 541, for example, with a cutoff wavelength of 620 nm. Light reflected at long-pass mirror 541 passes through optical filter 542, for example, a transmission filter of 505-545 nm and 575-620 nm, and is imaged onto a camera by the lens. Light transmitted at long-pass mirror 541 passes through optical filter 543, for example, a transmission filter of 660-710 nm, and is imaged onto another portion of detector 530, such as an sCMOS device, CCD, or EMCCD.

[0153] The optical imaging apparatus 100, 500 and method 200 described herein offer the following advantages:

[0154] The method disclosed in this specification can obtain image data with only a single scan of the sample, which includes multiple positions of the patterned illumination beams 121, 122 on the sample. The complete range of the optical section can be obtained.

[0155] The methods disclosed in this specification do not require pre-calibration of any type of lattice fitting or imaging device.

[0156] The degree of optical slicing can be rapidly changed by altering the constant slice factor (CSF) without having to rescan the sample or change the hardware configuration as with traditional methods, thus avoiding any sample degradation caused by rescanning, such as photobleaching.

[0157] The level or extent of the slices can be continuously tuned according to the sample, regardless of physical constraints.

[0158] Because optical slicing is achieved in an optical path without a physical aperture, the efficiency of emitted light collection is not compromised.

[0159] The imaging speed or the number of patterns required to acquire the image can be adjusted by the user.

[0160] Users do not need to select the level of optical slices or related parameters before or during the experiment.

[0161] The embodiments of the invention shown in the accompanying drawings and described above are merely exemplary embodiments and are not intended to limit the scope of the invention, which is defined by the following claims. Any combination of non-mutually exclusive features described herein is within the scope of the invention.

Claims

1. A method for obtaining an optical slice image of a sample based on multiple images of the sample, the method comprising constructing the optical slice image by means of the following steps: Multiple signal values ​​are obtained from the multiple images, and the multiple signal values ​​correspond to the same pixel in the multiple images; The threshold value of the pixel is evaluated based on the multiple signal values ​​and the desired slicing degree of the optical slice image; as well as Pixel values ​​are evaluated by integrating a subset of the signal values ​​that are above the threshold. The plurality of images include optical images of the sample obtained by illuminating the sample at a plurality of lateral positions at a focal plane using a focused patterned illumination beam, wherein each image is obtained when the patterned illumination beam is at a corresponding lateral position among the lateral positions, such that the intensity of the patterned illumination beam on a portion of the sample at the focal plane varies with respect to each of the plurality of lateral positions.

2. The method according to claim 1, wherein, The threshold values ​​for evaluating the pixel include: A slice factor is provided for determining the slice degree of the desired optical slice image. The threshold value of the pixel is evaluated based on the functional relationship between the pixel's threshold value, the slice factor, and the background function evaluated at that pixel; and Specifically, the portion of the plurality of signal values ​​is evaluated by subtracting the threshold value from each of the plurality of signal values ​​or by dividing each of the plurality of signal values ​​by the threshold value.

3. The method according to claim 2, wherein, The threshold value of the pixel is the product of the slice factor and the background function evaluated at that pixel.

4. The method according to claim 2, wherein, The background function is a statistical function, and the output of the statistical function depends on the skewness of the distribution of the input data; and Specifically, the background function is evaluated at the pixel by performing the statistical function on the plurality of signal values.

5. The method according to claim 2, wherein, Evaluating the background function at the pixel involves one of the following: (1) Calculate the median of the plurality of signal values ​​for the pixel; (2) Calculate the median of the plurality of signal values ​​for the pixel, and then perform Gaussian blur on the adjacent pixels; (3) Calculate the average value of the plurality of signal values ​​for the pixel; (4) Calculate the average of the plurality of signal values ​​for the pixel, and then perform Gaussian blur on the adjacent pixels; (5) Calculate the average of the multiple signal values ​​and subtract the standard deviation for each signal value; (6) Determine the Pth percentile signal value of the pixel.

6. The method according to any one of claims 1 to 5, wherein, The step of evaluating a pixel value includes summing the differences between each signal value in the pixel's signal values ​​and a threshold value, and setting the pixel value to zero if the sum is negative.

7. The method according to any one of claims 1 to 5, the method comprising an imaging step, the imaging step involving: A patterned illumination beam is provided through an imaging lens, such that the patterned illumination beam is focused at the focal plane of the imaging lens; Obtain the multiple images of the sample. in, The acquisition includes: The patterned illumination beam is provided at a plurality of lateral positions on the focal plane, and each image is acquired when the patterned illumination beam is at a corresponding lateral position among the plurality of lateral positions, such that the intensity of the patterned illumination beam on the portion of the sample at the focal plane varies with respect to each of the plurality of lateral positions, and The signal collected via the imaging lens is detected using a detector.

8. The method according to claim 7, in, The patterned illumination beam is focused at the focal plane, and The patterned illumination beam includes a periodic pattern having a spatial period defined in at least one direction within the focal plane.

9. The method according to claim 8, wherein, The patterned illumination beam comprises a periodic array of linear focal points focused at the focal plane.

10. The method according to claim 8, wherein, The patterned illumination beam comprises an array of focused light spots focused on the focal plane.

11. The method according to claim 8, wherein, The patterned illumination beam comprises a combination of a linear focal point and a focused spot focused at the focal plane.

12. An apparatus for obtaining an optical slice image of a sample, the apparatus comprising: Imaging lens; An illumination source is configured to provide a patterned illumination beam through the imaging lens, such that the patterned illumination beam is focused at the focal plane of the imaging lens; A detector configured to detect signals collected from the sample via the imaging lens; The control unit is configured as follows: Multiple images of the sample are obtained by providing the patterned illumination beam at multiple lateral positions on the focal plane, such that the intensity of the patterned illumination beam on a portion of the sample at the focal plane varies for each of the multiple lateral positions. as well as Constructing an optical slice image based on the plurality of images using the following steps: Multiple signal values ​​are obtained from the multiple images, and the multiple signal values ​​correspond to the same pixel in the multiple images; The threshold value of the pixel is evaluated based on the multiple signal values ​​and the desired slicing degree of the optical slice image; as well as Pixel values ​​are evaluated by integrating a subset of the signal values ​​that are above the threshold.

13. The apparatus according to claim 12, wherein, The control unit is also configured to: Receive a slice factor for determining the slice degree of the desired optical slice image; The threshold value of the pixel is evaluated based on the functional relationship between the threshold value of the pixel and the slice factor and the background function evaluated at that pixel. as well as The portion of the plurality of signal values ​​is evaluated by subtracting the threshold value from each of the plurality of signal values ​​or by dividing each of the plurality of signal values ​​by the threshold value.

14. The apparatus according to claim 13, wherein, The threshold value of the pixel is the product of the slice factor and the background function evaluated at that pixel.

15. The apparatus of claim 13, further comprising: A spatial modulator, configured to provide the patterned illumination beam focused at the focal plane. The patterned illumination beam includes a periodic pattern having a spatial period defined in at least one direction within the focal plane.

16. The apparatus according to claim 15, wherein, The patterned illumination beam comprises a periodic array of linear focal points focused at the focal plane.

17. The apparatus according to claim 15, wherein, The patterned illumination beam comprises an array of focused light spots focused on the focal plane.

18. The apparatus according to claim 15, in, The patterned illumination beam comprises a combination of a linear focal point and a focused spot focused at the focal plane.

19. The apparatus according to any one of claims 13 to 18, wherein, The control unit is also configured to: The background function is evaluated at the pixel by performing a statistical function on the plurality of signal values. The statistical function receives multiple data values ​​and outputs a value based on the skewness of the multiple data values.

20. The apparatus according to claim 19, wherein, The statistical function outputs the median of the received multiple data values, such that the background function at the pixel is the median of the multiple signal values.

21. The apparatus according to any one of claims 12 to 18, wherein, The detector is either a single-pixel detector or a multi-pixel detector.

Images