Programmable multi-point illuminator, confocal filter, confocal microscope, and method for operating a confocal microscope

Abstract

To fill a gap between a single-point microscope and a multiple-point microscope.SOLUTION: A programmable multiple-point illuminator for an optical microscope M comprises: light sources 1 and 2; and a spatial light modulator SLM to modulate a light beam from the light source. The modulated light beam scans across a sample placed under a microscope objective lens 21, and emits sample fluorescence. The SLM comprises: a first acousto-optic deflector 8; and a second acousto-optic deflector 9, the two acousto-optic deflectors being arranged in cascade to provide respective deflection in different directions, whereby the SLM is enabled to scan two-dimensionally. The SLM also comprises a telescope relay 10 that conjugates a first modulation plane 81 with a second modulation plane 91. The illuminator includes an arbitrary waveform generator 13 that is configured to synthesize holograms, and is arranged to simultaneously inject respective holograms into the first acousto-optic deflector 8 and the second acousto-optic deflector 9 in order for the SLM to modulate the light beam in response to the hologram.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a programmable multi-point illuminator for an optical microscope, and a confocal filter for turning the microscope into a confocal microscope. The present disclosure also relates to a method of operating a confocal microscope.

[0002] The illuminator comprises a light source and a spatial light modulator for modulating a light beam from the light source, the modulated light beam being intended to be scanned across a sample placed under a microscope objective, the sample typically comprising a fluorophore. The expression "the sample is placed under the microscope objective" means that the light beam is focused into the sample by the objective (this expression should not be understood to mean that the sample is always under the objective). [Background technology]

[0003] Confocal microscopy is the reference technique for the visualization of samples in all areas of cell biology and is widely recognized as one of the most important inventions ever made in light microscopy. Confocal microscopes have gained tremendous popularity in recent years, with most universities and scientific institutions around the world, as well as an increasing number of individual laboratories, owning a confocal microscope.

[0004] There are essentially two different styles of confocal microscopes: single-point and multipoint scanning instruments. They are often used in combination with fluorescent tags (fluorescent molecules or fluorophores) that selectively label structures of interest and respond to the illuminating laser by emitting light at a longer wavelength (Stokes shift). This wavelength shift allows easy separation of the excitation and emission light trains using dichroic mirrors and filters.

[0005] Single-point confocal microscopy is based on a single laser beam that scans the sample stepwise, point by point, resulting in high-resolution, high-contrast, optically sectioned images after the light emitted from the sample is filtered by a tiny pinhole aperture conjugated to the laser spot. Light emitted by excited fluorophores above and below the focal plane is blocked by the pinhole and does not reach the detector, minimizing the optical haze that plagues non-confocal microscopes when imaging thick samples.

[0006] However, this point-by-point scanning method necessarily entails an apparently slow image acquisition, which is the main drawback of single-point confocals: living samples such as cells often need to be rectified (i.e., unstitched) to obtain images free of subject blur, as the instrument cannot resolve the temporal dynamics of many cellular phenomena.

[0007] Faster scanning has therefore become an important vector in the development of modern confocal microscopy. However, scanning with a single laser spot cannot be made arbitrarily fast, since a fast scanning speed means that the laser spot can only illuminate the sample point during a very short exposure time, on the order of microseconds. To compensate for this small excitation time, the laser power impinging on the sample must be increased, thereby saturating the fluorophores very quickly. An increment in laser power above the saturation threshold does not lead to an equivalent increment in the fluorescence emission rate, so the total amount of photons reaching the detector decreases with decreasing exposure time, thus limiting the scanning speed to approximately a few frames per second.

[0008] The only technical solution allowing fast confocal operation is to use several laser spots that scan the sample in parallel.

[0009] In response to this need, multipoint confocals were developed to simultaneously scan a sample using thousands of laser beamlets, thus reaching frame rates in the range of hundreds of frames per second. An added advantage of splitting the total power over many laser foci is that these instruments are considerably gentler on biological samples (1 / 15 less damaging than single-point confocals under comparable conditions), minimizing photobleaching and phototoxicity.

[0010] However, commercial implementations of the multi-point scanning principle are based on a disk covered with rows of tiny microlenses and pinholes (Nipkow disks) that rotate at high speed, making the system inflexible and optically inefficient. Indeed, high-speed spinning disk microscopes cannot scan any region of interest in a sample and typically use high-magnification, high-numerical aperture lenses to match a single objective lens. Also, typical Nipkow disks have an optical efficiency of about 4% and require powerful excitation lasers, which are costly.

[0011] A further difficulty is the confocal attenuation resulting from crosstalk between pinholes, especially in thick samples (crosstalk is due to leakage of the detected signal from other optical signals). Spurious light excited by one laser spot can reach the detector (e.g., a camera) through an adjacent pinhole, resulting in a significantly lower image resolution when compared to images produced by single-point confocal in the same conditions.

[0012] US Pat. No. 6,299,366 aims to suppress optical crosstalk between illumination spots generated to inspect a sample. US Pat. No. 6,299,366 discloses a diffractive optical element (DOE) placed in front of the objective lens. The DOE creates copies of the spot outputs without changing the spot spacing to ensure sufficient separation between the spots. However, US Pat. No. 6,299,366 assumes only simple and relatively inflexible spot patterns (it merely discloses one programmable acousto-optical deflector to provide adjustable spot spacing), thus reproducing to some extent the shortcomings of the Nipkow disk.

[0013] In summary, the two modulations of confocal microscopy have distinct advantages and disadvantages relative to each other that make them specialized and incompatible. In general, the user needs access to two types of equipment at once. Summary of the Invention

[0014] The aim of this disclosure is to bridge the gap between single-point and multipoint microscopy, enabling laboratories to envision unconstrained operation with just one confocal microscope.

[0015] Another objective of the present disclosure is to go beyond conventional multi-point microscopy by generating freeform illumination patterns that are significantly more complex than simple spot arrays.

[0016] In a first aspect, a programmable multi-point illuminator for an optical microscope comprises a light source and a spatial light modulator (SLM) for modulating a light beam from the light source. The modulated light beam is intended to be scanned across a sample placed under a microscope objective, the sample comprising fluorophores. The SLM comprises a first acousto-optic deflector (AOD) and a second acousto-optic deflector, the first AOD having a first modulation surface and the second AOD having a second modulation surface, the two acousto-optic deflectors being arranged in cascade to provide respective modulations (i.e., deflections) in different directions (e.g., the respective directions of polarization of the two AODs may be orthogonal), such that the spatial light modulator scans in two dimensions across the sample. The SLM further comprises a telescope relay for conjugating the first modulation plane with the second modulation plane. The illuminator also comprises an arbitrary waveform generator (AWG) configured to synthesize a radio frequency (RF) signal calculated with a digital holography algorithm, the synthesized signal being referred to as a hologram (such a hologram comprises an encoded recording of a light wave including its amplitude and phase characteristics), and the SLM is arranged to simultaneously inject a first such hologram into the first AOD and inject a second such hologram into the second AOD for modulating the light beam in response to the hologram.

[0017] Thus, the illuminator can illuminate the sample with a precise, purposefully designed and selected 2D light pattern; the selected light pattern can be very simple (e.g., by a few spots) or very complex (e.g., by a dense and intricate spot arrangement), but can be generated similarly in either case, thus bridging the gap between single-point and multi-point microscopy.

[0018] In one example, the programmable illuminator can comprise a laser device (light source) capable of dynamically projecting multiple light spots onto a microscope sample in parallel with precise positioning according to any variable and arbitrary pattern. The programmable illuminator is based on AOD technology. The AOD is essentially an ultrafast optical deflector that can impart changes in the direction of a light beam across the device. The AOD comprises a purposefully cut optical crystal and a piezoelectric transducer that is attached to one end of the crystal and can generate sound waves within it.

[0019] The modulation plane (or pivot plane) of an AOD is an imaginary plane within the AOD crystal where an incoming collimated light beam appears to be deflected, resulting in an outgoing collimated light beam traveling at a different angle. The modulation plane can be found by forward projecting the propagation direction of the incoming light beam and back projecting the propagation direction of the corresponding outgoing light beam, where the back and forward beams meet at a plane (the modulation plane) inside the crystal.

[0020] Two optically conjugated AODs produce a separable joint modulation function, ie the product of the modulation function of the first AOD (eg in the X direction) and the modulation function of the second AOD (eg in the Y direction).

[0021] It is known in the art to apply a simple sinusoidal radio frequency (RF) signal (see FIG. 2A) to a piezoelectric transducer. The vibration of the transducer sends an acoustic signal to the crystal that spatially modulates the refractive index in a periodic manner, generating a diffraction grating. By varying the frequency of the controlling RF signal, the period of the diffraction grating is modified and the deflection of the laser can be rapidly changed so that the beam can be redirected to a different spatial location.

[0022] In one example, the AWG is intended to be configured to synthesize holograms of arbitrary complexity (see Fig. 2B, Fig. 3 and Fig. 4), allowing the illuminator to scan across the sample with two-dimensional light patterns of arbitrary complexity. The AWG can synthesize electrical signals with arbitrary shapes, i.e., the time evolution of the output voltage can be specified by the user in a completely general way within the bandwidth of the instrument. In this sense, the AWG is a generalization of the voltage-controlled oscillator (VCO), which can only generate sine functions of variable frequency, and the function generator, which generates several different waveforms (sine, square, sawtooth, etc.), only within a choice predefined by the design.

[0023] In general, an AWG consists of a digital device, such as a field programmable gate array (FPGA), that can mathematically synthesize waveforms, and a high-speed digital-to-analog converter circuit that ultimately produces a time-varying electrical signal.

[0024] In one example, the illuminator can include a scan lens positioned after the spatial light modulator to project a reconstruction of the desired illumination pattern onto an intermediate image plane, which together with the microscope tube lens forms a 4f optical system that conjugates the modulation plane of the acousto-optic deflector with the input pupil of the microscope objective lens, which is a Fourier transform lens that is charged by focusing the light beam onto a Fourier reconstruction plane that intersects the sample, so that centering of the hologram is not critical due to the shifting nature of the Fourier transform.

[0025] In a second aspect, a confocal filter for an optical microscope having such an illuminator comprises an image sensor comprising an electronic multi-pixel detector configured to enable real-time implementation of one digital pinhole around the image of any excited fluorescent location (by a fluorophore) in the sample, the filter further comprising a relay system for focusing the fluorescent light emitted by the sample onto the image sensor.

[0026] Confocal filters, together with programmable illuminators, make optical microscopes truly confocal, since digital post-processing of images can mimic the effect of a physical pinhole. For example, CMOS image sensors that can selectively read out programmable sets of pixels (such as rows) can make image filtering fast enough.

[0027] In one example, a programmable illuminator and confocal filter set may provide a means to synchronize the AWG with the image sensor in order to properly construct the confocal emission image.

[0028] In a third aspect, a method of operating a confocal microscope with a programmable illuminator and confocal filters comprises: causing a light source to emit a first light beam of a particular diameter; expanding the first light beam into a second light beam having a predetermined diameter to define an illumination window on the first AOD; injecting a first hologram into a first AOD for modulating a second light beam and converting it into a third light beam; imaging a third light beam onto a second modulation surface; collimating the third light beam with zero modulation such that the diameter of the third light beam is the diameter of the second light beam and defines an illumination window on the second AOD; injecting a second hologram into the second AOD to modulate the third light beam and convert it into a fourth light beam; imaging the fourth light beam onto an input pupil of a microscope objective; focusing a fourth light beam onto a reconstruction plane that intersects the sample; collecting fluorescent light emitted from the sample and focusing the light onto an image sensor; Includes.

[0029] This method is calculating a first composite radio frequency signal with a first digital holography algorithm and a second composite radio frequency signal with a second digital holography algorithm (wherein the first digital holography algorithm and the second digital holography algorithm may be the same algorithm); The method may further include using the AWG to synthesize a first hologram that is injected into the first AOD from the first calculated signal and a second hologram that is injected into the second AOD from the second calculated signal.

[0030] In operation, the AWG injects a mathematically designed composite RF signal into the AOD cell, which results in further modification of the laser beam, as opposed to the simple deflection of a conventional AOD. , any complex light pattern desired can be generated. For example, a laser beam can be split into several sub-beams whose spatial positions can be controlled individually. This principle can be used to parallelize the illumination of a confocal microscope, similar to what a rotating disk achieves with a perforated Nipkow disk.

[0031] The sample is scanned by varying the array of illumination spots until it is fully exposed, which simply requires sending a new control signal to the AOD. In contrast to spinning disk confocal microscopes, the sampling pattern is here fully programmable, as it is not based on fixed apertures etched on a solid substrate with mechanical motion.

[0032] The difficulty with this approach is the confocal filtering of the out-of-focus light, which cannot be performed by the AOD itself. A way to overcome this is to create a virtual pinhole, as explained above (and in more detail below).

[0033] For applications where a sample needs to be illuminated by non-separable light patterns, two procedures are disclosed herein for constructing a general light distribution as the sum of several mathematically separable patterns. An important aspect to consider is that the patterns are time-multiplexed, i.e., they are generated by irradiance-sensitive integration devices (such as cameras or the sample itself) at different times and are summed thereby. Thus, the patterns to be added can only have positive values, and subtraction can be performed by optical means, but requires coherent superposition, which is not done here. The disclosed algorithm is as follows: a) Decomposition into rows

[0034] A simple way to decompose a two-dimensional NxM image into separable patterns is to divide the image into lines (i.e. N rows or M columns). For simplicity, assume that the image is divided into N rows. If an AOD deflecting in the X direction reconstructs the intensity of the line, and an AOD modulating in the Y direction at the same time deflects that line to the appropriate Y position (these two AODs are orthogonal), the image can be constructed line-by-line.

[0035] The drive signal for the Y-direction AOD is then a continuous sinusoidal pattern whose frequency is varied in steps to define the reconstruction position of the line, and the X-direction AOD is a hologram that encodes the inverse Fourier transform of the particular line being reconstructed at that time. b) Decomposition into separable 2D sub-images: Non-negative Singular Value Decomposition (NNSVD)

[0036] Singular Value Decomposition (SVD) is a well-known factorization method that decomposes an NxM matrix A into a sum of cross products, i.e., A=Σ1 k ω i u i v i T where k is the rank of A. If we consider an image as a matrix, the SVD operation automatically produces a decomposition into separable patterns (u and v vectors of the cross product). However, these patterns generally contain positive as well as negative values, which cannot be achieved by AOD projection.

[0037] Fortunately, nonnegative singular value decomposition (NNSVD) can be defined and computed by an iterative algorithm, as disclosed in W. Liu et al., "Nonnegative Singular Value Decomposition for Microarray Data Analysis of Spermatogenesis," Proceedings of the 5th International Conference on Information Technology and Application in Biomedicine, 225-228 (2008).

[0038] The NNSVD algorithm decomposes the image into separable sub-images and captures them with the AOD illuminator. It has proven to be very useful for displaying images in a linear fashion, making it possible in certain cases to increase the reconstruction speed while compressing the image information. The algorithm approximates a diagonalization of the image A with all positive values, according to A ≈ X W Y.

[0039] The diagonal matrix W is composed of the eigenvalues ​​of the diagonal coefficients w ii is a one-dimensional vector xi andy i If columns xi and yi are normalized, then w ii represents the total power (or energy) of the decomposed sub-images. This property is useful when judging the importance of sub-images. To speed up the reconstruction and improve efficiency, sub-images with negligible intensity can be discarded. [Brief description of the drawings]

[0040] Non-limiting examples of the present disclosure are described below with reference to the accompanying drawings. [Figure 1] FIG. 2 is a diagram showing the arrangement of an optical microscope M. [Diagram 2] FIG. 1 illustrates a comparison between two radio frequency signals. [Diagram 3] FIG. 1 is a diagram illustrating a 2D acousto-optic deflector. [Figure 4] FIG. 1 shows a comparison between two holograms. [Diagram 5] FIG. 2 shows an array of light spots. [Figure 6] FIG. 1 illustrates confocal filtering. [Figure 7] FIG. 1 shows a comparison between three imaging results. [Figure 8] FIG. 1 shows a comparison between six image reconstructions. [Figure 9] FIG. 13 is a diagram showing a hologram transition. [Figure 10] FIG. 13 shows a comparison between different phase alignments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] FIG. 1 shows an illumination system comprising a first laser source 1 and a second laser source 2, a dichroic device 4 for combining the respective beams from the laser sources in one direction, a first acousto-optic deflector (AOD) 8, an inverted telescope 5 in front of the first AOD 8, a second AOD 9, a telescope relay 10 between the first AOD 8 and the second AOD 9, and a scanning lens 17 after the second AOD 9.

[0042] FIG. 1 also shows the following parts of the optical microscope M: a tube lens 19 and an input pupil 20 to an objective lens 21 (microscope objective lens).

[0043] The illumination system of FIG. 1 further comprises an arbitrary wave generator (AWG) 13, a first radio frequency (RF) amplifier 14 between the AWG 13 and the first AOD 8, a second radio frequency (RF) amplifier 15 between the AWG 13 and the second AOD 9, a two-dimensional (2-D) image sensor 26, a synchronizer 16 between the image sensor 26 and the AWG 13, a relay system 25 in front of the image sensor 26, a fluorescence filter 24 in front of the relay system 25, and a dichroic mirror 23 between the tube lens 19 and the fluorescence filter 24 (the dichroic mirror 23 is also disposed between the scan lens 17 and the tube lens 19).

[0044] Any laser source 1 or 2 can be either continuous or pulsed. Pulsed lasers (femtosecond) can in some applications be used to trigger multiphoton absorption phenomena at the reconstruction plane, for the purpose of exciting fluorescence or inducing photopolymerization. When using pulsed lasers as illumination sources, additional optical elements such as prism 3 can be used to avoid or compensate for large-scale velocity dispersion inside the AOD. In contrast, continuous wave (CW) lasers are preferred for general fluorescence microscopy, as they emit one uninterrupted light beam and have much lower peak power values ​​than pulsed or ultrafast lasers that can adversely affect the sample, and they present a simple design that makes them easier to manufacture and maintain.

[0045] In general, several laser sources can provide illumination simultaneously or sequentially at selected wavelengths for multicolor applications and can be unidirectionally combined by a dichroic device 4. For example, several light sources are required for excitation in multicolor microscopy, where at least two fluorescent molecular dyes are used to label different sample structures. Upon laser illumination, these structures emit light in different wavelength ranges, so that they can be visualized individually. Furthermore, two lasers with different wavelengths may be required for the excitation and depletion steps in super-resolution techniques such as STED (Stimulated Emission Depletion) or RESOLFT (Reversible Saturable Optical Fluorescence Transition) microscopy. Moreover, multiple lasers can be used to study the colocalization of two or more molecular species within the same biological structure, visualized by labeling them with dyes that respond to different excitation wavelengths.

[0046] In operation, the single or combined laser beams are expanded by the inverted telescope 5 from an initial laser diameter D1 to an illumination window of diameter D2 on the first AOD 8. The size D2 of the illumination window, and therefore the magnification of the beam expander 5, is carefully chosen as this controls the field of view versus frame rate tradeoff of the illumination system.

[0047] The two AODs 8 and 9 provide modulation of the light beam in two orthogonal directions (X and Y), i.e. they constitute a spatial light modulator. These AODs are high-resolution, high-deflection angle devices (preferably providing a spot with a resolution of more than 500x500), with a large square input window, preferably more than 8x8 mm, and with acousto-optical properties as similar as possible. When illuminated with several lasers, the AODs are configured to function in Bragg regime simultaneously for the entire set of wavelengths of interest.

[0048] Furthermore, when addressed by a particular set of RF signals within the bandwidth of the AOD device, the two AODs must provide overlapping deflection ranges for the entire set of wavelengths. The AODs are mounted on tilt-tip optomechanical mounts (not shown) so that they are oriented at the appropriate angle with respect to the incident laser beam to achieve good diffraction efficiency for the entire bandwidth and any wavelengths of interest.

[0049] The two orthogonal AODs are optically conjugated by a telescope relay 10, which comprises two identical lenses in a 4f configuration to image the modulation plane 81 (i.e. the pivot plane of the beam deflection) of the first AOD 8 onto the modulation plane 91 of the second AOD 9 with unity magnification. Furthermore, the relay 10 simultaneously keeps the laser beam collimated (with zero modulation) with diameter D2 at the illumination window on the second AOD 9. The two optically conjugated AODs then achieve an optical multiplication of the modulation functions as follows: h(x,y)=f(x)·g(y) (see FIG. 3) without distortions caused by the light propagating from the first AOD 8 to the second AOD 9.

[0050] The AOD device is connected to a dual-channel arbitrary waveform generator (AWG) 13 via radio frequency (RF) amplifiers 14 and 15. The AWG digitally synthesizes two, usually discrete, RF signals calculated in the manner of digital holography. These synthesized pixelated radio signals (see Fig. 3A) induce complex spatial variations of the refractive index in the AOD cell by the acousto-optic effect, which then modulates the light beam in a manner similar to a Fourier spatial light modulator. The resulting complex refractive index modulation becomes an acoustically induced hologram.

[0051] The AWG 13 is capable of generating signals of a bandwidth matching that of the AOD devices (8, 9) and incorporates a memory bank capable of storing enough pre-computed drive signals to jointly represent a complex light pattern. In effect, as opposed to a true two-dimensional spatial light modulator, the AOD device described herein comprises two one-dimensional light modulating devices in cascade to produce a mathematically separable two-dimensional light pattern. Only turns can be generated (i.e. the product of a function of X with a function of Y, h(x, y) = f(x) g(y), see Figure 3). This is usually sufficient to implement an excitation pattern consisting of an array of light spots that can be scanned in parallel over the sample (as explained above, the sample is placed under the objective lens 21 of the microscope M).

[0052] If the illuminator is required to project more complex light patterns (e.g. to excite arbitrary regions of interest, for photostimulation or photobleaching), these can be modelled using an appropriate mathematical algorithm (represented by reference numeral 11 in Fig. 1), i.e. H(x,y)=Σ i h i (x, y) = Σ i f i (x)·g i (y) can be obtained by series decomposition into separable functions.

[0053] The samples themselves sum the series terms (or in other applications by synchronization with a final detector, typically AWG 13, reference numeral 16) due to the cumulative effect.

[0054] In any case, in order to correctly form the desired light pattern, it must be taken into account that the light modulation in the AOD device is generated by the progression of acoustic waves that propagate from the piezoelectric transducer at one end of the acousto-optical crystal to the muffler at the other end and interact with the laser beam within a finite time interval when crossing the illuminated optical window of the AOD. This necessarily requires performing the reconstruction in the Fourier plane for the conjugated modulation planes (81 and 91) of the two AODs 8 and 9, so that the centering of the hologram is not critical due to the shifting properties of the Fourier transform. To implement this, the modulation plane needs to be conjugated with the input pupil 20 of the microscope objective 21, which here acts as a Fourier transform lens.

[0055] First, a scan lens 17 projects a reconstruction of the desired excitation pattern onto an intermediate image plane 18. This scan lens 17, together with the tube lens 19 of the microscope M, forms a 4-f system that optically conjugates modulation planes 81 and 91 (which are then mutually conjugated by the telescope relay 10) with the input pupil 20 of the microscope objective 21, which focuses the laser beam onto a final reconstruction plane 22 that intersects with the sample (not shown), thereby exciting the fluorescence of the sample (i.e., the fluorophores therein).

[0056] The field of view of the microscope objective 21 (which is a Fourier transform lens) must be larger than the slope of the highest spatial frequency Fourier component contained in the displayed hologram. Pupil coincidence may be necessary to avoid vignetting of the Fourier components in the microscope objective 21. Since the microscope objective 21 must generally be a highly corrected optical system consisting of several optical surfaces, its input pupil 20 may be internal to the system and not directly accessible. Since the modulation planes 81 and 91 are also internal to the AOD devices 8 and 9, respectively, a relay system formed by the scan lens 17 and the tube lens 19 is used to couple the modulation planes to the input pupil 20 of the microscope objective 21.

[0057] In addition, the scan lens 17 and tube lens 19 match the input pupil size D3 with the optical window size D2 of the AOD in order to use the full numerical aperture of the microscope objective 21 and thereby optimize the sectioning capability and resolution. The ratio D3 / D2 then determines the magnification of the telescope system formed by the scan lens 17 and tube lens 19 and thus the deflection angle of the Fourier components of the wavefront diffracted by the AOD and thus the field of view of the illuminator on the sample plane. Since the time τ that the sound wave needs to cross the illuminated window is τ=D2 / v, where v is the speed of sound in the crystal, D2 is also related to the maximum repetition rate at which the hologram can be updated (i.e. the maximum frame rate of the illuminator) and as a result needs to be chosen carefully.

[0058] Fluorescence emitted by the sample traveling in the opposite direction to the excitation laser is captured by microscope objective 2. The emitted light is collected by AWG 13 and focused to an intermediate image plane 18. After this, a dichroic mirror 23 and a fluorescence filter 24 select the emission wavelength, and a relay system 25 focuses the emission light onto a 2D image sensor 26. Electronic post-processing (see Fig. 5) of several individual frames obtained by methodically shifting the multi-spot illumination array can produce the final confocal image, as shown in Figs. 6 and 7. Correct construction of the final image requires precise synchronization (16) between the sensor 26 and the AWG 13.

[0059] The image sensor may be a CMOS multi-pixel detector that allows for arbitrary reading area at high speed, allowing real-time implementation of a digital mask (digital pinhole) around each emitting focus. Various algorithms that improve both lateral and axial resolution, such as photon reallocation, may also be implemented.

[0060] Figures 3A and 3B show the formation of a two-dimensional modulation pattern from two one-dimensional modulation signals injected into orthogonal AOD cells. When the two AOD devices are conjugated via relay 10, the first X-varying modulation plane is projected onto the second Y-varying modulation plane. The result is an external vector multiplication of the discrete signals encoded in the two planes.

[0061] Figure 4 shows the last step of the synthesis process of the holographic radio frequency signal injected into the AOD cell. First (and not shown in Figure 4), the illumination pattern is described in a digital data array. The index of each array element represents the location of the physical illumination spot in the sample plane, and the value of each array element represents its intensity accordingly. If the illumination pattern I(x,y) is separable, it can be described by the multiplication of two discrete functions, I(x,y)=I1(x)·I2(y). If not, the decomposition can be performed according to the procedure disclosed above.

[0062] Since the wavefront of the laser beam is modulated by the AOD and then transformed into the desired illumination pattern by the optical Fourier transform, the calculation of the required wavefront modulation can be performed via the inverse Fourier transform of the illumination pattern. Since the illumination pattern is described digitally, the required amplitude and phase modulation can be calculated by the discrete Fourier transform (DFT). The position of the illumination pattern elements (spots) is controlled by the frequency at which the DFT is evaluated.

[0063] To obtain the required spatial amplitude and phase modulation by the AOD, an electronic drive signal is required that induces a corresponding acousto-optic modulation of the AOD crystal. Finally, the relationship between the drive RF signal and the resulting spatial amplitude and phase modulation is simple, i.e., the carrier frequency f c A piecewise defined sinusoidal drive signal having a piecewise varying amplitude and phase corresponding to the calculated amplitude and phase results in the desired wavefront modulation (an accurate approximation) (reference A in FIG. 4).

[0064] Furthermore, to maximize the optical efficiency, an iterative Gerchberg-Saxton algorithm (reference B in FIG. 4) is preferably used to convert the full composite hologram (left graph in FIG. 4) into a kinoform (or phase hologram, which is most often the case, right graph in FIG. 4). A typical result of RF frequencies synthesized by this procedure is shown in the right graph in FIG. 4, which represents a high-efficiency hologram (reference C in FIG. 4) injected into one of the AODs 8 or 9 (the left graph in FIG. 4 represents a relatively low-efficiency hologram).

[0065] Figure 2A shows an oscilloscope trace of a pure sinusoidal RF signal of frequency = 75 MHz that typically drives an AOD when used as a beam deflector. In contrast, Figure 2B shows a much more complex signal calculated with the digital holography technique described above, designed to generate any of the light distributions disclosed herein.

[0066] This holographic RF signal allows any light distribution to be obtained on the reconstruction plane 22. When the illuminator of the present disclosure is used to excite fluorescence from a microscope specimen, a convenient pattern is formed by a regular array of light spots (e.g., a square matrix of 32 × 32 light spots). By modifying the X and Y holograms, as shown in Figure 5, the array can be shifted stepwise until the sample is fully exposed. The entire sample plane can be scanned by an illumination pattern consisting of a regular array of light spots that are shifted stepwise horizontally and vertically until the gaps between the spots are filled. The shift increment can be selected as the radius of the individual light spots that matches the Rayleigh criterion.

[0067] In other words, the increments in the X and Y directions can be chosen according to the resolution of the optical system, i.e., to match the radius of the point spread function at the reconstruction plane 22 (e.g., a 16x16 shift). The specimen (in the sample) emits fluorescence in response to the individual excitation spot arrays, which is captured by the sensor 26 in synchronization with the AWG 13 (16).

[0068] Figure 6 shows one of the fluorescence images emitted in response to the illumination light. The enlarged inset shows that the resulting image is also an array of light spots, whose intensity is related to the local density of fluorescent molecules at the focal plane, surrounded by a cloud of scattered light, mainly coming from out-of-focus planes. This scattered light, if not removed, will significantly reduce the contrast and resolution of the final image, as shown in Figure 7. Confocal filtering is performed on a single emission frame. Fluorescence photons coming from planes above and below the plane of interest create a halo of light around the excitation focus, which can be removed by digitally processing the image, i.e. only the information within a small circle around each light spot is kept and the rest of the image is deleted.

[0069] Figure 7 again shows the individual response of a sample to one of the excitation spot arrays. Fluorescence images of the actin network of a chicken embryo immunolabeled with phalloidin+TRITC are shown. From left to right: a single emission frame (the excitation pattern consists of a square array of 32x32 laser spots; Figure 7A), a widefield (non-confocal) image (Figure 7B), a confocal image obtained after pinhole filtering and addition of the entire set of single emission frames (Figure 7C).

[0070] If the final image is composed as the addition of these individual responses, without filtering out-of-focus light, the image in Figure 7B is obtained. In contrast, if the individual fluorescence frames are digitally pinholeed, by retaining only the light in a small circle around the excitation focus before final composition (see the right-most image in Figure 6), the result in Figure 7C is obtained, clearly showing the improved resolution and contrast. Spot array illumination allows this operation to be performed and justifies its use.

[0071] Figure 8 shows the reconstruction of a fluorescence image according to a scheme of alternately capturing fluorescence, filtering out-of-focus light and shifting the illumination pattern to a new position according to the layout in Figure 5. The quality of the final reconstructed image is comparable to that produced by a conventional confocal microscope based on a physical pinhole. Reconstruction of the final image as a sum of digitally pinhole-depleted emission frames. The excitation spot array is shifted to 16x16=256 locations completely scanning the sample.

[0072] Figure 9 illustrates the problems that arise when several holograms need to be displayed in succession (e.g., with separable resolution). First, the hologram needs to be repeated several cycles in succession by the AWG 13 before switching to the next hologram in the series, in order to efficiently modulate the light beam. Effectively, if a moving hologram is displayed only once, it will only be correctly reconstructed at one point in time, when it has completely passed through the optical window. Before that point, it is only partially visible, since it is still passing through the exposed laser field. After that point, the first hologram starts to leave the illuminated field and the next hologram starts to become visible (Fig. 9), temporarily sharing part of the AOD window, which causes crosstalk. Diffraction patterns generated during the transition between successive holograms degrade the quality of the reconstruction.

[0073] Conversely, the reconstruction is optimal while the signal of one hologram is continuously repeated, since the circular shift produced by the mobility of the hologram (due to the repetition, a part of the hologram disappearing from one end of the illuminated window apparently reappears via the other end) does not affect the reconstruction due to the Fourier transform properties.

[0074] To solve this problem, each hologram signal can be repeated for a finite number of instances that ensures a reasonable reconstruction time. The ratio of the total time a single hologram is displayed to the transition time between two holograms determines the quality of the reconstruction. As an additional measure, blanking signal periods can be introduced that separate each pair of hologram repetitions to completely suppress temporal mixing. However, if more repetitions and blanking periods are used, the reconstruction time will increase, and as a result a trade-off between frame rate and quality must be implemented depending on the needs of the application.

[0075] An additional problem with the hologram sequences is related to the erroneous multiplication of X and Y holograms. Note that both the X and Y windows display two consecutive holograms during a hologram transition (X hologram 1 and X hologram 2 for AODx 8, and Y hologram 1 and Y hologram 2 for AODy 9), leading to the formation of erroneous products (between X-hologram 1 and Y-hologram 2, and between X-hologram 2 and Y-hologram 1). This effect is minimized if the X and Y sequences are synchronized.

[0076] However, depending on several factors, such as the difference in the speed of sound in the two AO crystals, or the spatial centering of the illumination fields of the two AOD devices, an RF signal (injected simultaneously into the two devices) will generate a moving hologram, reaching the laser beam at two different times.

[0077] This has the effect of temporarily extending the transient period during which a hologram in the X axis multiplies the wrong hologram in the Y axis (e.g. the first X hologram in a sequence multiplies the second Y hologram in a sequence), thus inducing artifacts in the final reconstructed image.

[0078] Therefore, a key feature of the AWG 13 may be to incorporate a means to advance or delay the hologram sequence in one channel relative to the other channel (relative phase delay control) to compensate for these acoustic path differences in the AOD cells, so that the signal in AODx 8 can be precisely aligned with the signal in AODy 9, as shown in Figure 10.

[0079] The triangular spot patterns are not separable, however, it is possible to compute separable holograms that, when repeated indefinitely, produce the spot patterns of Figures 10b and 10c.

[0080] When these two identical holograms are displayed consecutively during the camera integration time, taking into account the repetition and blanking periods as discussed above, and with a suitable phase delay between the two AWG channels, Fig. 10a correctly shows a correctly formed, inseparable triangular spot array. In contrast, Figs. 10d and 10e show the actual results of our setup. Figure 1 shows an inaccurate reconstruction of the triangular spot array due to settings for phase delays that do not pre-compensate for differences in the actual acoustic paths.

[0081] While only some examples are disclosed herein, other alternatives, modifications, uses, and / or equivalents thereof are possible. Moreover, all possible combinations of the examples described are also encompassed. Thus, the scope of the disclosure should not be limited by the specific examples, but should be determined only by a fair reading of the claims that follow. If reference signs related to the drawings are placed within parentheses in the claims, they are intended only to enhance the understanding of the claims and should not be construed as limiting the scope of the claims. [Prior art documents] [Patent documents]

[0082] [Patent Document 1] U.S. Patent No. 9,395,340

Claims

1. 1. A programmable multi-point illuminator for an optical microscope, comprising: a continuous laser or multiple continuous wave lasers, focused into a single light beam and capable of emitting an uninterrupted light beam (1, 2); and a spatial light modulator intended to modulate the single or combined light beam from said continuous wave lasers and to scan said modulated light beam across a sample placed under a microscope objective (21), said sample comprising a fluorophore, The spatial light modulator comprises a first acousto-optic deflector (8) and a second acousto-optic deflector (9), the first acousto-optic deflector having a first modulation surface (81) and the second acousto-optic deflector having a second modulation surface (91), the two acousto-optic deflectors being arranged in cascade to provide respective deflections in different directions, the spatial light modulator is capable of scanning in two dimensions across the sample, the first modulation surface and the second modulation surface are conjugated with an input pupil (20) of the microscope objective (21), and the illuminator is configured to synthesize a hologram. and an arbitrary waveform generator (13) configured to simultaneously inject such a first hologram into the first acousto-optic deflector and inject such a second hologram into the second acousto-optic deflector so that the spatial light modulator modulates a single or combined light beam in response to the hologram, wherein the microscope objective lens (21) acts as a Fourier transform lens with respect to the conjugated modulation plane (81, 91) and focuses the single or combined light beam into a Fourier reconstruction plane that intersects the sample.

2. 10. The illuminator of claim 1, wherein the first and second acousto-optic deflectors are high resolution and high deflection angle devices providing a spot with a resolution greater than 500x500.

3. 3. A confocal filter for an optical microscope with an illuminator as described in claim 1 or 2, comprising an image sensor (26) with an electronic multi-pixel detector configured to enable real-time implementation of one digital pinhole around an image of any excited fluorescence location within the sample, and further comprising a relay system (25) for focusing the fluorescence emitted by the sample onto the image sensor.

4. A confocal microscope comprising the confocal filter described in claim 3, and comprising means (16) for synchronizing the waveform generator (13) with the image sensor (26) to correctly construct a confocal luminescence image.

5. 5. A method of operating a confocal microscope according to claim 4, comprising the steps of: causing said single or combined continuous wave lasers (1, 2) to emit a first light beam of a specific diameter (D1); expanding the first light beam into a second light beam having a predetermined diameter (D2) to define an illumination window on the first acousto-optic deflector (8); injecting a first hologram into the first acousto-optic deflector to modulate the second light beam and convert it into a third light beam; imaging the third light beam onto the second modulation surface (91); Collimating the third light beam with zero modulation such that the diameter (D2) of the third light beam is the diameter of the second light beam and defines an illumination window on the second acousto-optic deflector (9); injecting a second hologram into the second acousto-optic deflector to modulate the third light beam and convert it into a fourth light beam; imaging the fourth light beam onto the input pupil (20) of the microscope objective lens (21); focusing the fourth light beam onto a Fourier reconstruction plane (22) intersecting the sample such that centering of the first and second holograms is not critical due to the shifting properties of the Fourier transform; repeating the first and second holograms in succession for several cycles before switching to another hologram to prevent the first and second holograms from being only partially displayed as they pass through each illuminator window; collecting fluorescent light emitted from the sample and focusing the fluorescent light onto the image sensor (26); The method includes:

6. The method of claim 5 further comprising the step of introducing a blank period before switching to another hologram.

Images