Microscope apparatus generating structured illumination and operation method thereof

Abstract

A microscope apparatus generating structured illumination is disclosed. One embodiment may comprise: a light source unit for modulating an optical signal so as to generate a modulated optical signal, and outputting the generated modulated optical signal; an illumination unit for performing beam scanning on the basis of the outputted modulated optical signal so that a structured illumination pattern is generated on a sample; a detection unit for collecting an optical signal from the sample and acquiring a high-quality image of the sample on the basis of the collected optical signal; and a control unit for controlling the beam scanning so that the illumination unit generates the structured illumination pattern on the sample, and performing at least one among synchronization between the light source unit and the detection unit, and synchronization between the illumination unit and the detection unit.

Description

Microscope device for generating structured illumination and method for operating the same

[0001] The following examples relate to a microscope device for generating structural illumination and a method of operating the same.

[0002] Structured illumination microscopy illuminates a sample with a specific pattern (e.g., an interference pattern) and acquires images based on the acquired images, enabling optical sectioning and enhanced spatial resolution. Conventional structured illumination microscopy can utilize a diffraction grating or a spatial light modulator (SLM) to generate structured illumination.

[0003] As a related prior art, there is Korean Patent Publication No. 10-2018-0033379 (Title of invention: Structured illumination microscope, Applicant: Daegu Gyeongbuk Institute of Science and Technology).

[0004] The background technology described above is something that the inventor possessed or acquired in the process of deriving the disclosure of the present application, and cannot necessarily be said to be a publicly known technology disclosed to the general public prior to the present application.

[0005] A microscope device for generating structured illumination according to one embodiment includes a light source unit for generating a modulated light signal by modulating an optical signal and outputting the generated modulated light signal; an illumination unit for performing beam scanning based on the outputted modulated light signal to generate a structured illumination pattern on a sample; a detection unit for collecting an optical signal from the sample and obtaining an image of the sample based on the collected optical signal; and a control unit for controlling the beam scanning so that the illumination unit generates the structured illumination pattern on the sample and performing at least one of synchronization between the light source unit and the detection unit and synchronization between the illumination unit and the detection unit.

[0006] The above lighting unit can generate a line beam for the beam scanning using the output modulated optical signal, and illuminate the generated line beam onto the sample through a plurality of lenses, thereby generating a line structure illumination pattern on the focal plane of the sample.

[0007] The above lighting unit may include a first lens that focuses the output modulated optical signal into a line beam; a scanner that adjusts the position of the optical axis of the line beam to perform the beam scanning; a scan lens that focuses the line beam; a tube lens onto which the focused line beam is incident; and an objective lens that focuses the line beam passing through the tube lens and illuminates the sample.

[0008] The control unit can synchronize the scanner performing the beam scanning with the image sensor so that the start and end of the frame of the image sensor of the detection unit are respectively aligned with the start and end of the structural lighting pattern.

[0009] The detection unit may include a rolling shutter-based image sensor. The control unit may synchronize the image sensor with a scanner performing the beam scanning so that the start and speed of the rolling line of the image sensor are respectively aligned with the start and speed of the structured illumination pattern.

[0010] The width of the above rolling line may correspond to a multiple of the product of the spatial period of the above structural illumination pattern and the first magnification. The first magnification may represent a magnification determined based on the magnifications of each of the objective lens and the tube lens within the microscope device.

[0011] The spatial period of the above-mentioned structured illumination pattern may be set to a value close to the size of a light spot that is continuously repeated in the above-mentioned structured illumination pattern. The width of the above-mentioned rolling line may correspond to a value obtained by multiplying the set spatial period by a first magnification. The duty cycle of the above-mentioned modulated optical signal may be smaller than a time corresponding to the width of one pixel of the above-mentioned image sensor. The above-mentioned first magnification may represent a magnification determined based on the magnifications of each of the objective lens and the tube lens in the above-mentioned microscope device.

[0012] The detection unit may include a line sensor. The control unit may synchronize the light source unit and the line sensor so that an image corresponding to one cycle of the structured lighting pattern is acquired within one frame of the line sensor. The sensor width of the line sensor may be greater than the width of one structured light within the structured lighting pattern.

[0013] The control unit controls the light source unit so that a time delay occurs in the modulated optical signal, the lighting unit performs the beam scanning based on the modulated optical signal with the time delay and generates a structured illumination pattern within the sample, and the detection unit collects a first signal from the sample and can obtain an image having a first phase of the sample based on the collected first signal.

[0014] The control unit controls the light source unit so that a time delay occurs again in the modulated optical signal in which the time delay occurs, the lighting unit performs the beam scanning based on the modulated optical signal in which the time delay occurs again and generates a structured illumination pattern within the sample, and the detection unit collects a second signal from the sample and can obtain an image having a second phase of the sample based on the collected second signal.

[0015] The above control unit can obtain a high-quality image of the sample using the obtained images.

[0016] The control unit controls the lighting unit so that the lighting unit generates a different structural lighting pattern for each frame of the detection unit on the sample, receives an image for each frame from the detection unit, and can generate a high-quality image using the image for each frame.

[0017] In one embodiment, a method of operating a microscope device that generates structured illumination includes the steps of modulating an optical signal to generate a modulated optical signal and outputting the generated modulated optical signal; performing beam scanning based on the outputted modulated optical signal to generate a structured illumination pattern on a sample; collecting an optical signal from the sample; and acquiring an image of the sample based on the collected optical signal.

[0018] The generating step may include generating a line beam for the beam scanning using the output modulated optical signal, and illuminating the sample with the generated line beam through a plurality of lenses to generate a line structure illumination pattern on the focal plane of the sample.

[0019] The method of operating the above microscope device may further include a step of synchronizing the image sensor with a scanner that performs the beam scanning so that the start and end of each frame of the image sensor that acquires the image are aligned with the start and end of the structured illumination pattern, respectively.

[0020] The above microscope device may include a rolling shutter-based image sensor. The operating method of the microscope device may further include a step of synchronizing the image sensor with a scanner that performs the beam scanning so that the start and speed of the rolling line of the image sensor are respectively aligned with the start and speed of the structured illumination pattern.

[0021] The width of the above rolling line may correspond to a multiple of the product of the spatial period of the above structural illumination pattern and the first magnification. The first magnification may represent a magnification determined based on the magnifications of each of the objective lens and the tube lens within the microscope device.

[0022] The spatial period of the above-mentioned structured illumination pattern may be set to a value close to the size of a light spot that is continuously repeated in the above-mentioned structured illumination pattern. The width of the above-mentioned rolling line may correspond to a value obtained by multiplying the set spatial period by a first magnification. The duty cycle of the above-mentioned modulated optical signal may be smaller than a time corresponding to the width of one pixel of the above-mentioned image sensor. The above-mentioned first magnification may represent a magnification determined based on the magnifications of each of the objective lens and the tube lens in the above-mentioned microscope device.

[0023] The above microscope device may include a line sensor. The operating method of the microscope device may further include a step of synchronizing the light source unit outputting the modulated light signal with the line sensor so that an image corresponding to one cycle of the structured illumination pattern is acquired within one frame of the line sensor.

[0024] The sensor width of the above line sensor may be greater than the width of one structural light within the above structural light pattern.

[0025] The operating method of the above microscope device may further include the steps of generating a time delay in the modulated optical signal, performing the beam scanning based on the modulated optical signal with the time delay to generate a structured illumination pattern within a sample, collecting a first signal from the sample, and obtaining an image having a first phase of the sample based on the collected first signal; generating a time delay again in the modulated optical signal with the time delay, performing the beam scanning based on the modulated optical signal with the time delay again to generate a structured illumination pattern within the sample, collecting a second signal from the sample, and obtaining an image having a second phase of the sample based on the collected second signal; and obtaining a high-quality image of the sample using the obtained images.

[0026] The operating method of the above microscope device may further include a step of generating a different structural illumination pattern on the sample for each frame of the image sensor to acquire an image for each frame; and a step of generating a high-quality image using the image for each frame.

[0027] The embodiment can generate periodic structured illumination on a sample by using temporal amplitude modulation of a light source and beam scanning without using a diffraction grating and SLM, thereby improving the simplicity and robustness of a microscope system and improving the controllability of structured illumination generation, thereby easily implementing structured illumination in a confocal detection method, thereby improving optical sectioning performance.

[0028] Figure 1 is a block diagram illustrating the configuration of a microscope device according to one embodiment.

[0029] FIG. 2 is a drawing illustrating an example of a microscope device according to one embodiment.

[0030] FIG. 3 is a drawing illustrating another example of a microscope device according to one embodiment.

[0031] FIG. 4 is a drawing illustrating another example of a microscope device according to one embodiment.

[0032] FIGS. 5A and 5B are block diagrams illustrating a light source unit of a microscope device according to one embodiment.

[0033] Figures 6 to 9 are drawings illustrating examples of light source modulation according to one embodiment.

[0034] FIGS. 10 to 12 are drawings illustrating examples of structural lighting patterns according to one embodiment.

[0035] FIG. 13 is a drawing explaining control for continuous shooting of structural lighting in wide-field detection according to one embodiment.

[0036] FIG. 14 is a diagram illustrating synchronization and speed control of a camera and a scanner for structural lighting imaging in a rolling shutter-based line field method detection according to one embodiment.

[0037] FIGS. 15a and 15b are drawings explaining rolling line width setting conditions of a camera for structural lighting imaging in a rolling shutter-based line field method detection according to one embodiment.

[0038] FIG. 16 is a diagram illustrating control for high-frequency structured illumination imaging in a rolling shutter-based line field method detection according to one embodiment.

[0039] FIG. 17 is a diagram illustrating control for structural illumination imaging in a line field method detection based on a multi-line camera according to one embodiment.

[0040] FIG. 18 is a diagram explaining the sensor width in a line field method detection based on a multi-line camera according to one embodiment.

[0041] FIGS. 19 and 20 are drawings illustrating examples of control for phase change according to one embodiment.

[0042] Figure 21 is a comparison image before and after application of structural lighting according to one embodiment.

[0043] FIG. 22 is a drawing illustrating control for changing the structural illumination of a microscope device according to one embodiment.

[0044] FIG. 23 is a flowchart illustrating a method of operating a microscope device for generating structural illumination according to one embodiment.

[0045] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0046] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0047] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0048] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0050] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0051]

[0052] Figure 1 is a block diagram illustrating the configuration of a microscope device according to one embodiment.

[0053] Referring to FIG. 1, a microscope device (100) according to one embodiment may include a light source unit (110), an illumination unit (120), a detection unit (130), and a control unit (140). The illumination unit (120) may generate (or form) periodic structured illumination on a sample, and may be otherwise expressed as a structured illumination unit. Although not shown in FIG. 1, the microscope device (100) may further include a stage on which a sample is placed (or positioned). This stage may be moved in the x-axis, y-axis, and z-axis directions by the control unit (140).

[0054] The light source unit (110) may include, but is not limited to, a light source such as a laser diode, a CW (Continuous Wave) laser, a pulse laser, or an LED (light emitting diode), for example.

[0055] The light source unit (110) can generate a modulated light signal by modulating an optical signal of the light source, and can output the modulated light signal. For example, the light source unit (110) can perform a first modulation (e.g., amplitude modulation (AM)) on the optical signal to generate a modulated light signal of a specific waveform. The modulated light signal can have, for example, a pulse waveform, a sine waveform, a triangle waveform, etc., but is not limited thereto.

[0056] The illumination unit (120) can perform beam scanning based on the output modulated optical signal to generate a structured illumination pattern (or periodic structured illumination pattern) on the sample. For example, the illumination unit (120) can generate a line beam for beam scanning using the output modulated optical signal. The illumination unit (120) can illuminate the generated line beam onto the sample through a plurality of lenses (e.g., a scan lens, a tube lens, an objective lens, etc.) to generate a periodic structured illumination pattern (e.g., a line structured illumination pattern, etc.) on the focal plane of the sample.

[0057] When illumination is performed on a sample, an optical signal may be reflected from the sample or an optical signal may be generated that transmits through the sample. The detection unit (130) can collect an optical signal from the sample (e.g., an optical signal reflected from the sample or an optical signal transmitted through the sample) and obtain an image of the sample based on the collected optical signal.

[0058] The control unit (140) can control at least one of the light source unit (110), the lighting unit (120) (e.g., a scanner), or the detection unit (130) (e.g., a camera (or image sensor)). As will be described later, the control unit (140) can control the light source unit (110) so that a time delay occurs in the modulated optical signal. The control unit (140) can control the scanner to change the structured illumination pattern cycle. The control unit (140) can control the light source unit (110), the scanner, and the camera according to conditions to implement line confocal structured illumination.

[0059] The control unit (140) can control the illumination unit (120) so that the illumination unit (120) can perform beam scanning. For example, the illumination unit (120) can include a first lens capable of generating a line beam for performing beam scanning. The control unit (140) can control the scanner so that the position of the optical axis of the line beam is adjusted. According to this control, the line beam can be periodically illuminated on the sample, and periodic structured illumination (e.g., a striped pattern) can be generated on the sample.

[0060] The control unit (140) can synchronize the lighting unit (120) and the detection unit (130). For example, the detection unit (130) can include a rolling shutter-based image sensor. The control unit (140) can synchronize the beam scanning of the lighting unit (120) with the rolling scanning of the rolling shutter-based image sensor.

[0061] The control unit (140) can generate high-quality and / or high-resolution images of the sample using the images received from the detection unit (130). As will be described later, the control unit (140) can receive images of various phases of the sample (e.g., optical sectioning images) from the detection unit (130) and generate high-quality images (e.g., enhanced optical sectioned images) using the received images.

[0062] Conventional structured illumination microscope devices can use a diffraction-based grating element, a Deformable Mirror Device (DMD), a Spatial Light Modulator (SLM), etc. to generate periodic structured illumination on a sample. A microscope device (100) according to one embodiment can generate periodic structured illumination on a sample by using modulation (e.g., AM) of a light source and beam scanning without using a diffraction-based grating element, a DMD, and an SLM. Accordingly, one embodiment can simplify the microscope system compared to existing structured illumination microscope systems, make it more robust than existing structured illumination microscope systems, and improve system controllability. According to one embodiment, a line confocal-based structured illumination microscope can be simply implemented with improved controllability.

[0063] The microscope device (100) according to one embodiment can be applied to a wide-field fluorescence microscope device and a line confocal fluorescence microscope device, respectively, to implement a wide-field structured illumination fluorescence microscope device and a line confocal structured illumination fluorescence microscope device, respectively. Without being limited thereto, the microscope device (100) according to one embodiment can be applied to any imaging device (or microscope device) in which the intensity of a signal reflected from a sample changes depending on the illumination intensity of the sample. For example, the microscope device (100) can be applied to a multiphoton (e.g., two-photon, three-photon, etc.) imaging device (or multiphoton microscope device), an optical microscope device (e.g., bright field microscope device, dark field microscope device, phase contrast microscope device, polarizing microscope device, etc.), a photoluminescence (PL) imaging device, a light sheet microscope device, a thermal reflectance imaging device, a photothermal reflectance imaging device (or a photothermal reflectance microscope device), etc.

[0064]

[0065] FIG. 2 is a drawing illustrating an example of a microscope device according to one embodiment.

[0066] An example of the microscope device (100) of FIG. 1 is shown in FIG. 2.

[0067] The microscope device (200) illustrated in FIG. 2 may correspond to a wide-field microscope device capable of generating a periodic structured illumination pattern on a sample (213) by using modulation of a light source unit (110) and beam scanning of an illumination unit (120). In other words, the microscope device (200) illustrated in FIG. 2 may be a microscope device for wide-field detection without a confocal function.

[0068] Referring to FIG. 2, the microscope device (200) may include a light source unit (110), a first lens (201), a scanner (203), a scan lens (205), a tube lens (207), a first mirror (209), an objective lens (211), a tube lens (215), a first image sensor (217), and a control unit (140).

[0069] The first lens (201), scanner (203), scan lens (205), tube lens (207), first mirror (209), and objective lens (211) of FIG. 2 may be included in the lighting unit (120) of FIG. 1.

[0070] The objective lens (211), the tube lens (215), and the first image sensor (217) of FIG. 2 may be included in the detection unit (130) of FIG. 1. In the example illustrated in FIG. 2, the illumination unit (120) and the detection unit (130) may share the objective lens (211). Without being limited thereto, in some embodiments, the detection unit (130) may not share the objective lens (211) with the illumination unit (120) and may include a separate objective lens. The detection unit (130) may collect the optical signal of the sample through the separate objective lens.

[0071] The microscope device (200) can irradiate a modulated light signal output by the light source unit (110) to a sample (213) at the bottom of the objective lens (211) through the first lens (201) (e.g., a cylindrical lens or a Powell lens), a scanner (203), a scan lens (205), a tube lens (207), and an objective lens (211). For example, the microscope device (200) can generate a line beam that scans in a direction orthogonal to the focal plane of the scan lens (205) using the first lens (201) and the scanner (203), and can reduce (or focus) the line beam according to the illumination magnification (Mi) through the tube lens (207) and the objective lens (211) and then illuminate the sample (213). Here, the illumination magnification (Mi) may represent a magnification determined through the magnification of the tube lens (207) and the magnification of the objective lens (211). The optical signal reflected from the sample (213) may pass through the objective lens (211), the first mirror (209), and the tube lens (215) and be transmitted to the first image sensor (217). The first image sensor (217) may acquire an image of the sample based on the signal passed through the objective lens (211), the first mirror (209), and the tube lens (215). For example, the optical signal reflected from the sample (213) may be magnified according to the detection magnification (Md) through the objective lens (211) and the tube lens (215) and may be detected by the first image sensor (217), and the first image sensor (217) may acquire an image of the sample (213) based on the detected optical signal. Hereinafter, the microscope device (200) will be described in more detail with reference to FIG. 2.

[0072] The control unit (140) can synchronize the first image sensor (217) and the scanner (203) with each other.

[0073] The light source unit (110) can modulate an optical signal to output a modulated optical signal. For example, the control unit (140) can control the light source unit (110) so that the light source unit (110) can output a modulated optical signal.

[0074] The first lens (201) can focus the output modulated optical signal into a line beam.

[0075] The scanner (203) can adjust the position of the optical axis of a signal (e.g., a line beam) passing through the first lens (201) to perform beam scanning or the position of the point at which the line beam is illuminated (or irradiated) to the sample (213). The scanner (203) can include, but is not limited to, an x-axis galvano mirror, an acousto-optic deflector, etc. The scanner (203) can deflect a line beam traveling along the optical axis.

[0076] The scan lens (205) can focus a signal (e.g., a deflected line beam) that has passed through the scanner (203). The signal that has passed through the scan lens (205) can be incident on the tube lens (207).

[0077] The tube lens (207) can allow a signal (e.g., a focused deflected line beam) passing through the scan lens (205) to enter the objective lens (211).

[0078] The objective lens (211) can focus a signal passing through the tube lens (207) and illuminate the sample (213). The signal focused by the objective lens (211) can be axially illuminated on the sample (213), and the position of the optical axis can be adjusted by the scanner (203). Accordingly, a periodic structured illumination pattern (e.g., a line structured illumination pattern (or stripe pattern)) can be generated (or formed) on the sample (213).

[0079] When the objective lens (211) focuses the signal passing through the tube lens (207) and illuminates the sample (213), for example, an optical signal may be reflected from the sample (213). The objective lens (211) can collect the optical signal reflected from the sample (213).

[0080] The reflected light signal can pass through the objective lens (211), the first mirror (209) can transmit the signal that has passed through the objective lens (211), and the signal that has passed through the first mirror (209) can be incident on the tube lens (215). The first image sensor (217) can detect the signal that has passed through the tube lens (215). The light signal reflected from the sample (213) can be magnified by the detection magnification (Md) by the objective lens (211) and the tube lens (215) and detected by the first image sensor (217). The detection magnification (Md) can represent a magnification determined by the magnification of the objective lens (211) and the magnification of the tube lens (215).

[0081] The first image sensor (217) can acquire an image (e.g., a two-dimensional fluorescence image) of the sample (213) based on the detected signal. The acquired image may correspond to an image having a 0-degree phase, for example. The first image sensor (217) may include, but is not limited to, a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor, for example.

[0082] As will be described later with reference to FIGS. 19 and 20, the control unit (140) can control the light source unit (110) so that a time delay occurs in the modulated optical signal. The illumination unit (120) can perform beam scanning based on the modulated optical signal in which the time delay occurs, and can generate a structured illumination pattern within the sample (213). The detection unit (130) can collect a first signal from the sample (213), and can obtain an image (or an image delayed by the first phase) of the sample (213) with a first phase (e.g., 120 degrees) based on the collected first signal. The control unit (140) can control the light source unit (110) so that a time delay occurs again in the modulated optical signal in which the time delay occurs. The illumination unit (120) can perform beam scanning based on the modulated optical signal in which the time delay occurs again, and can generate a structured illumination pattern within the sample (213). The detection unit (130) can collect a second signal from the sample, and based on the collected second signal, can obtain an image having a second phase (e.g., 240 degrees) of the sample (or an image delayed by the second phase).

[0083] As mentioned above, images having three different phases were given as examples, but this is only an example, and the microscope device (200) can acquire images having three or more different phases.

[0084] The control unit (140) can receive images of various phases of the sample from the detection unit (130) and can use the received images to generate a high-quality image (e.g., an enhanced optical sectioned image).

[0085] The control unit (140) can move the stage of the sample (213) in the z-axis direction, and the signal reflected from the sample (213) of the moved stage can pass through the optical path of the objective lens (211), the first mirror (209), and the tube lens (215), and the first image sensor (217) can detect the signal that has passed through the optical path of the objective lens (211), the first mirror (209), and the tube lens (215) to obtain an image of the sample (213). In addition, the first image sensor (217) can obtain phase-delayed images similar to that described above. The control unit (140) can generate a high-quality image on the moved z-axis by using the images obtained by the first image sensor (217).

[0086] The microscope device (200) can acquire images of various phases at each designated position along the z-axis, and can use the acquired images to generate high-quality images at each designated position along the z-axis. The microscope device (200) can generate a three-dimensional image of a sample using the high-quality images at each designated position along the z-axis.

[0087]

[0088] FIG. 3 is a drawing illustrating another example of a microscope device according to one embodiment.

[0089] Another example of the microscope device (100) of FIG. 1 is shown in FIG. 3.

[0090] The microscope device (300) illustrated in FIG. 3 may correspond to a line confocal microscope device capable of generating a periodic structured illumination pattern on a sample (317) by using modulation of a light source unit (110) and beam scanning of an illumination unit (120). As will be described later, the microscope device (300) of FIG. 3 may include a line sensor (321) (e.g., a multi-line camera or a multi-line sensor). The microscope device (300) of FIG. 3 may be expressed differently as a microscope device for detection using a line field method based on a multi-line camera.

[0091] Referring to FIG. 3, the microscope device (300) may include a light source unit (110), a first lens (301), a first mirror (303), a scanner (307), a scan lens (309), a tube lens (313), an objective lens (315), a second lens (319), a line sensor (321), and a control unit (140).

[0092] In one embodiment, the first lens (301), the first mirror (303), the scanner (307), the scan lens (309), the tube lens (313), and the objective lens (315) of FIG. 3 may be included in the illumination unit (120) of FIG. 1, and the scanner (307), the scan lens (309), the tube lens (313), the objective lens (315), the second lens (319), and the line sensor (321) of FIG. 3 may be included in the detection unit (130). In the example illustrated in FIG. 3, the illumination unit (120) and the detection unit (130) may share the scanner (307), the scan lens (309), the tube lens (313), and the objective lens (315).

[0093] The microscope device (300) can irradiate a modulated light signal output by the light source unit (110) to a sample (317) through a first lens (301) (e.g., a cylindrical lens or a Powell lens), a first mirror (303), a scanner (307), a scan lens (309), a tube lens (313), and an objective lens (315). For example, the microscope device (300) can generate a line beam that scans in a direction orthogonal to the focal plane of the scan lens (309) using the first lens (301) and the scanner (307), and can reduce (or focus) the line beam according to the illumination magnification (Mi) through the second mirror (311), the tube lens (313), and the objective lens (315) and then illuminate the sample (317).

[0094] The microscope device (300) can acquire an image of the sample by transmitting an optical signal reflected from the sample (317) to the line sensor (321) through an optical path of the objective lens (315), the tube lens (313), the scan lens (309), the scanner (307), the first mirror (303), and the second lens (319) (e.g., a condenser lens). For example, the optical signal reflected from the sample (317) can be magnified according to the detection magnification (Md) by the objective lens (315) and the tube lens (313), and descanned by the scan lens (309) and the scanner (307) so that the optical signal can be aligned to the optical axis and incident on the second lens (319). The signal aligned to the optical axis can be focused on the line sensor (321) after passing through the second lens (319). Signals within the focal plane (in-focus) excluding signals outside the focal plane (out-focus) of the sample (317) can be focused on the line sensor (321). The line sensor (321) can detect signals within the focal plane to obtain an image of the sample (317). Hereinafter, the microscope device (300) will be described in more detail with reference to FIG. 3.

[0095] The control unit (140) can synchronize the line sensor (321) and the scanner (307) with each other.

[0096] The light source unit (110) can modulate an optical signal to output a modulated optical signal. For example, the control unit (140) can control the light source unit (110) so that the light source unit (110) can output a modulated optical signal.

[0097] The first lens (301) can focus the output modulated optical signal into a line beam.

[0098] The first mirror (303) can pass a signal that has passed through the first lens (301). The first mirror (303) can include, for example, a dichroic mirror.

[0099] The scanner (307) can adjust the position of the optical axis of the signal passing through the first lens (301) to perform beam scanning or the position of the point where the signal passing through the first lens (301) is illuminated (or irradiated) to the sample (213). The scanner (307) can include, but is not limited to, an x-axis galvano mirror, an acousto-optic deflector, etc. The scanner (307) can deflect a line beam traveling along the optical axis.

[0100] The scan lens (309) can focus a signal (e.g., a deflected line beam) that has passed through the scanner (307). The signal that has passed through the scan lens (309) can be incident on the tube lens (313).

[0101] The objective lens (315) can focus the signal passing through the tube lens (313) and illuminate the sample (317). The focused signal can illuminate the sample (317) in the axial direction, and the position of the optical axis can be adjusted by the scanner (307). Accordingly, a periodic structured illumination pattern (e.g., a line structured illumination pattern (or stripe pattern)) can be generated (or formed) on the sample (317).

[0102] When the objective lens (315) focuses the signal passing through the tube lens (313) and illuminates the sample (317), for example, an optical signal may be reflected from the sample (317). The objective lens (315) can collect the optical signal reflected from the sample (317).

[0103] The reflected optical signal can pass through the optical path of the objective lens (315), the tube lens (313), the scan lens (309), the scanner (307), the first mirror (303), and the second lens (319). At this time, the reflected optical signal can be magnified by the detection magnification (Md) by the objective lens (315) and the tube lens (313), and the detection magnification (Md) can represent a magnification determined by the magnification (or focal length) of the objective lens (315) and the magnification (or focal length) of the tube lens (313).

[0104] The line sensor (321) can detect a signal transmitted through the optical path of the objective lens (315), the tube lens (313), the scan lens (309), the scanner (307), the first mirror (303), and the second lens (319). The line sensor (321) may include a multi-line sensor. The number of pixels of the line sensor (321) may be, for example, 2048×128 or 2048×1, but is not limited thereto.

[0105] The line sensor (321) can obtain an image (e.g., a two-dimensional image) of the sample (213) based on the detected signal.

[0106] The control unit (140) can move the stage of the sample (317) in the z-axis direction, and the signal reflected from the sample (317) of the moved stage can be transmitted to the line sensor (321) through the optical path of the objective lens (315), the tube lens (313), the scan lens (309), the scanner (307), the first mirror (303), and the second lens (319). The line sensor (321) can detect the signal transmitted through the optical path to obtain an image of the sample (317). The control unit (140) can obtain a three-dimensional image (e.g., a three-dimensional high-resolution and high-resolution image) of the sample by using the images obtained by the line sensor (321).

[0107]

[0108] FIG. 4 is a drawing illustrating another example of a microscope device according to one embodiment.

[0109] Another example of the microscope device (100) of FIG. 1 is shown in FIG. 4.

[0110] The microscope device (400) illustrated in FIG. 4 can generate a periodic structured illumination pattern on a sample (213) by using modulation of a light source unit (110) and beam scanning of an illumination unit (120). As will be described later, the microscope device (400) of FIG. 4 can include a rolling shutter-based image sensor, and thus can be expressed differently as a microscope device for detection using a rolling shutter-based line field method.

[0111] Referring to FIG. 4, the microscope device (400) may include a light source unit (110), a first lens (401), a scanner (403), a scan lens (405), a tube lens (407), a first mirror (409), an objective lens (411), a tube lens (415), a second image sensor (417), and a control unit (140).

[0112] The first lens (401), scanner (403), scan lens (405), tube lens (407), and objective lens (411) of FIG. 4 may be included in the lighting unit (120) of FIG. 1, and the objective lens (411), tube lens (415), and second image sensor (417) of FIG. 4 may be included in the detection unit (130) of FIG. 1. In the example illustrated in FIG. 4, the lighting unit (120) and the detection unit (130) may share the objective lens (411).

[0113] The second image sensor (417) may be a rolling shutter-based image sensor.

[0114] The microscope device (400) can irradiate a modulated light signal output by the light source unit (110) to a sample (413) through a first lens (401) (e.g., a cylindrical lens or a Powell lens), a scanner (403), a scan lens (405), a tube lens (407), a first mirror (409), and an objective lens (411). For example, the microscope device (400) can generate a line beam that scans in a direction orthogonal to the focal plane of the scan lens (405) using the first lens (401) and the scanner (403), and can reduce (or focus) the line beam according to the illumination magnification (Mi) through the tube lens (407), the first mirror (409), and the objective lens (411) and then illuminate the sample (413).

[0115] An optical signal reflected from the sample (413) can pass through the objective lens (411), the first mirror (409), and the tube lens (415) to be transmitted to the second image sensor (417), and the second image sensor (417) can obtain an image of the sample based on the transmitted signal. For example, an optical signal reflected from the sample (413) can be magnified according to a detection magnification (Md) through the objective lens (411) and the tube lens (415) and detected by the second image sensor (417), and the second image sensor (417) can obtain an image of the sample (413) based on the detected optical signal. Hereinafter, the microscope device (400) will be described in more detail with reference to FIG. 4.

[0116] The control unit (140) can synchronize the second image sensor (417) and the scanner (403) with each other. The control unit (140) can synchronize the rolling scanning of the second image sensor (417) and the beam scanning of the scanner (403).

[0117] The light source unit (110) can modulate an optical signal to output a modulated optical signal. For example, the control unit (140) can control the light source unit (110) so that the light source unit (110) can output a modulated optical signal.

[0118] The first lens (401) can focus the output modulated optical signal into a line beam.

[0119] The scanner (403) can adjust the position of the optical axis of a signal (e.g., a line beam) passing through the first lens (401) to perform beam scanning or the position of the point at which the line beam is illuminated (or irradiated) to the sample (413). The scanner (403) can include, but is not limited to, an x-axis galvano mirror, an acousto-optic deflector, etc. The scanner (403) can deflect the line beam traveling along the optical axis.

[0120] The scan lens (405) can focus a signal (e.g., a deflected line beam) that has passed through the scanner (403). The signal that has passed through the scan lens (405) can be incident on the tube lens (407).

[0121] The tube lens (407) can input a signal (e.g., a focused deflected line beam) that has passed through the scan lens (405) to the objective lens (411).

[0122] The objective lens (411) can focus a signal passing through the tube lens (407) and illuminate the sample (413). The focused signal can illuminate the sample (413) axially, and the position of the optical axis can be adjusted by the scanner (403). Accordingly, a periodic structured illumination pattern (e.g., a line structured illumination pattern (or stripe pattern)) can be generated (or formed) on the sample (413).

[0123] When the objective lens (411) focuses the signal passing through the tube lens (407) and illuminates the sample (413), for example, an optical signal may be reflected from the sample (413). The objective lens (411) can collect the optical signal reflected from the sample (413).

[0124] The reflected light signal can pass through the objective lens (411), the first mirror (409) can transmit the signal that passed through the objective lens (411), and the signal that passed through the first mirror (09) can be incident on the tube lens (415). The second image sensor (417) can detect the signal that passed through the tube lens (415). The light signal reflected from the sample (413) can be magnified by the detection magnification (Md) by the objective lens (411) and the tube lens (415) and detected by the second image sensor (417). The detection magnification (Md) can represent a magnification determined by the magnification (or focal length) of the objective lens (411) and the magnification (or focal length) of the tube lens (415).

[0125] The second image sensor (417) can acquire an image (e.g., a two-dimensional image) of the sample (413) based on the detected signal.

[0126] The microscope device (400) can acquire images of various phases at each position along the z-axis while moving the stage in the z-axis direction similarly to the microscope device (200) of FIG. 2. The microscope device (400) can acquire high-quality images (e.g., enhanced optical sectioned images) of the sample (413) by using the images of various phases at each position along the z-axis. The microscope device (400) can generate a three-dimensional image of the sample (413) by using the high-quality images at each position along the z-axis.

[0127]

[0128] FIGS. 5A and 5B are block diagrams illustrating a light source unit of a microscope device according to one embodiment.

[0129] In the example illustrated in FIG. 5a, the light source unit (110) may include a light source (505), and the light source (505) may include a modulator (510). In the example illustrated in FIG. 5b, the modulator (510) may not be included in the light source (505) and may be located at the output terminal of the light source unit (110).

[0130] A modulator (510) can perform modulation (e.g., AM) on an optical signal (501) to generate a modulated optical signal (503). Examples of optical source modulation (or modulated optical signal (503)) will be described with reference to FIGS. 6 to 9.

[0131]

[0132] Figures 6 to 9 are drawings illustrating examples of light source modulation according to one embodiment.

[0133] In the example illustrated in FIG. 6, the modulated optical signal (600) may be a pulse waveform.

[0134] The modulated optical signal (600) is t on It can be output during t off It may not be output during. For example, t on During which light can be output continuously, t off During this time, light may not be output. T on The longer it is, the longer the light can be output.

[0135] T on may correspond to the duty cycle, and the duty ratio may correspond to t on / T may be.

[0136] T on + t off may be the period (T) of the modulated optical signal (600).

[0137] The light source unit (110) can output a modulated optical signal (600), and the lighting unit (120) can perform beam scanning based on the modulated optical signal (600) to provide pulsed illumination to the sample.

[0138] In one embodiment, the control unit (140) may adjust the duty ratio to 1. This may be the case when no modulation is performed. When the duty ratio is 1, the microscope device (100) may provide uniform illumination to the sample, as in the example illustrated in FIG. 7. In the example illustrated in FIG. 7, the light source unit (110) may output an unmodulated uniform light signal (700), and the lighting unit (120) may provide uniform illumination to the sample.

[0139] The modulated optical signal (503) is not limited to the modulated optical signal (600) of the pulse waveform illustrated in FIG. 6.

[0140] In the example illustrated in FIG. 8, the modulated optical signal (800) may be a sinusoidal waveform.

[0141] The light source unit (110) can output a modulated optical signal (800), and the lighting unit (120) can perform beam scanning based on the modulated optical signal (800) to provide sinusoidal structural illumination to the sample.

[0142] In the example illustrated in FIG. 9, the modulated optical signal (900) may have a periodic arbitrary shape.

[0143] The light source unit (110) can output a modulated light signal (900), and the lighting unit (120) can provide periodic, arbitrary-shaped structural illumination to the sample.

[0144]

[0145] FIGS. 10 to 12 are drawings illustrating examples of structural lighting patterns according to one embodiment.

[0146] Referring to FIG. 10, a line structure illumination pattern (or stripe pattern, pulse wave structure illumination pattern) (1000) is illustrated in the xy plane of a sample (e.g., sample (213) of FIG. 2, sample (317) of FIG. 3, sample (413) of FIG. 4).

[0147] The line structure lighting pattern (1000) may be an example of a periodic structure lighting pattern.

[0148] When the light source unit (110) outputs the modulated light signal (600) of FIG. 6, the lighting unit (120) can irradiate a line-shaped beam onto the sample, and the stage of the sample can move in the axial direction (e.g., the y-axis direction). Accordingly, a line-structured illumination (1010) can be generated on the sample.

[0149] A scanner (e.g., scanner (203) of FIG. 2, scanner (307) of FIG. 3, scanner (403) of FIG. 4) can deflect a beam incident on the scanner by changing the optical axis of the beam incident on the scanner. The illumination unit (120) can irradiate the deflected beam on the sample, and the stage of the sample can move in the axial direction (e.g., the y-axis direction). Accordingly, a line structure illumination (1011) can be generated on the sample. In this way, the microscope device (100) can perform beam scanning based on the modulated optical signal (600) to periodically generate line structure illumination (1010 to 1014) on the sample. A line structure illumination pattern (1000) can be formed on the sample.

[0150] In the example shown in Fig. 10, the width (or stripe width) (d) of the line structure lights (1010 to 1014) on ) is the scan speed of the beam scanning (e.g., scan speed within the sample) (vs) and t on can correspond to the result of multiplication. For example, "d on = v s × t on "may be. The spacing (or stripe spacing) of the line structure lights (1010 to 1014) (d off ) is the scan speed (v s ) and t off can correspond to the result of multiplication. For example, "d off = v s × t off "It could be.

[0151] The line structured lighting (1010 to 1014) may have repeated light spots. For example, as in the example illustrated in FIG. 11, the line structured lighting (1010) may have repeated light spots continuously. In FIG. 11, the light spots are illustrated as dotted lines. When the line beam has a Gaussian shape, the light spots may have a Gaussian shape, as in the example illustrated in FIG. 11. The interval between the peak of the first light spot (1101-1) and the peak of the last light spot (1101-n) of the line structured lighting (1010) is d as described above. on It may correspond to . The interval between the peak of the last light spot (1101-n) of the line structured lighting (1010) and the peak of the first light spot (1102-1) of the line structured lighting (1011) is d as described above. off may apply.

[0152] In Fig. 12, when the sinusoidal structured illumination described through Fig. 8 is provided to the sample, a sinusoidal structured illumination pattern (1200) can be formed on the sample. The spatial period (d) of the sinusoidal structured illumination pattern (1200) T ) is the in-sample scan speed of beam scanning (v s ) and the result of multiplication of the period (T) of the modulated optical signal (800) of FIG. 8.

[0153] In Fig. 12, the dotted lines may represent light spots. In the structural lighting pattern (1200), light spots may be continuously repeated.

[0154]

[0155] FIG. 13 is a drawing explaining control for continuous shooting of structural lighting in wide-field detection according to one embodiment.

[0156] Referring to FIG. 13, the control unit (140) of the microscope device (200) of FIG. 2 can synchronize the detection unit (130) (e.g., the first image sensor (217)) and the illumination unit (120) (e.g., the scanner (203)) with each other to align the start point of the camera frame with the start point of the sinusoidal structured illumination pattern (or the modulated light signal (800)) and to align the end point of the camera frame with the end point of the sinusoidal structured illumination pattern (or the modulated light signal (800)). T frame During this time, sinusoidal structural illumination can be provided to the sample and imaging can be performed by a camera (or first image sensor (217)).

[0157] In the example shown in Fig. 13, the control unit (140) is T frame If this is terminated, the time required to reload the camera or scanner is T off During this time, the light source (110) can be turned off so that the modulated light signal (800) is not output, and the camera can be turned off so that no shooting is performed. Accordingly, photobleaching and / or phototoxicity of the sample can be reduced.

[0158]

[0159] FIG. 14 is a diagram illustrating synchronization and speed control of a camera and a scanner for structural lighting imaging in a rolling shutter-based line field method detection according to one embodiment.

[0160] Referring to FIG. 14, the control unit (140) of the microscope device (400) of FIG. 4 aligns the starting point of the rolling line of the second image sensor (417) with the starting point of the pulse wave structured illumination pattern (e.g., the line structured illumination pattern (1000) of FIG. 10) and the rolling speed (v) of the second image sensor (417). rolling ) and scan speed (v s ), the second image sensor (417) and the scanner (403) can be synchronized with each other.

[0161] In the example shown in Fig. 14, the rolling speed (v) of the second image sensor (417) rolling ) is the sample scan speed (v) of the beam scanning of the scanner (403). s ) and detection magnification (M d ) can be corresponded to the result of the multiplication of rolling speed (v rolling ) can represent the movement speed of the window of the second image sensor (417). The rolling speed of the second image sensor (417) is M d × v s It could be.

[0162]

[0163] FIGS. 15a and 15b are drawings explaining the rolling line width setting conditions of a camera for structural lighting imaging in a rolling shutter-based line field method detection according to one embodiment.

[0164] Referring to Fig. 15a, the rolling line width (d w ) (or the width of the window of the second image sensor (417)) is the spatial period (d) of the pulse wave structured illumination pattern within the sample (e.g., the line structured illumination pattern (1000) of FIG. 10). T ) and detection magnification (M d ) can be a multiple of the value multiplied by d w =n(M d × d T ) can be. Here, n can be a natural number (1, 2, 3, ...).

[0165] Referring to Figure 15b, the rolling line width (d w ) (or the width of the window of the second image sensor (417)) is the spatial period (d) of the sinusoidal structured illumination pattern within the sample (e.g., the sinusoidal structured illumination pattern (1200) of FIG. 12). T ) and detection magnification (M d ) can be a multiple of the value multiplied by d w =n(M d × d T ) can be. Here, n can be a natural number (1, 2, 3, ...).

[0166] The pulse wave structured illumination pattern illustrated in Fig. 15a and the sinusoidal structured illumination pattern illustrated in Fig. 15b are only examples of periodic structured illumination patterns. When a periodic structured illumination pattern is formed on a sample, the rolling line width (d) is determined according to the description of Figs. 15a and 15b. w ) can be set.

[0167]

[0168] FIG. 16 is a diagram illustrating control for high-frequency structured illumination imaging in a rolling shutter-based line field method detection according to one embodiment.

[0169] Referring to Figure 16, the spatial period (d) of the structural lighting pattern T ) is the size of the light spot (d S ) or set to a value close to the size of the light spot (d S ) can be set to the same value as the rolling line width (d) of the second image sensor (417). w ) is a set high frequency period (e.g. d T ) in the detection ratio (M d ) can be set to a value multiplied by the duty cycle (t) of the light source (110). on ) is the width of one pixel (d) of the second image sensor (417). P ) can be set to less than the time corresponding to t on ≤ d p / (M d ×V s ) may be, and the duty cycle is t on / T may be.

[0170]

[0171] FIG. 17 is a diagram illustrating control for structural illumination imaging in a line field method detection based on a multi-line camera according to one embodiment.

[0172] Referring to FIG. 17, the control unit (140) of the microscope device (300) of FIG. 3 can synchronize the light source unit (110) and the camera (e.g., line sensor (321)) so that an image of one cycle of the structured illumination pattern can be captured in one camera frame. In the example shown in FIG. 17, the control unit (140) synchronizes the start / end point of the cycle (T) of the modulated light signal (600) (or the cycle of the pulse wave structured illumination pattern) with the frame time (T) of the line sensor (321). Multi-line ) can be synchronized so that the start / end points of the light source unit (110) and the line sensor (321) match. In other words, in the case of the microscope device (300) of FIG. 3, temporal synchronization between the cycle of the structural illumination pattern and the photographing cycle of the line sensor (321) can be achieved by the control unit (140).

[0173] In the example shown in Fig. 17, t on ≤ t exposure < T may be. Here, t exposure can represent the shutter exposure time of the line sensor (321).

[0174] The modulated optical signal (600) of the pulse waveform in FIG. 17 is only an example of a periodic modulated optical signal, and the contents described through FIG. 17 can be applied when the light source unit (110) outputs a periodic modulated optical signal.

[0175]

[0176] FIG. 18 is a diagram explaining the sensor width in a line field method detection based on a multi-line camera according to one embodiment.

[0177] In the example illustrated in FIG. 18, the light source unit (110) of the microscope device (300) of FIG. 3 can output a modulated light signal (600) of a pulse waveform, and the illumination unit (120) can generate a periodic structured illumination pattern (e.g., a pulse wave structured illumination pattern (1000) of FIG. 10) corresponding to the modulated light signal (600) on the sample.

[0178] The detection unit (130) can collect a signal corresponding to one structured illumination pattern (e.g., line structured illumination (1010) of FIG. 10) from the sample. The spatial width (d) of the collected signal (i.e., the signal corresponding to one structured illumination pattern) on ) is the sensor width (d) of the line sensor (321) w ) may be larger than a portion of the collected signal may be lost. Accordingly, the sensor width (d) of the line sensor (321) should be adjusted so that all collected signals (i.e., signals corresponding to one structural lighting pattern) can be detected by the line sensor (321). w ) is M d ×d on It can be bigger.

[0179] In the example illustrated in Fig. 18, the modulated optical signal (600) of the pulse waveform is only an example, and the contents described through Fig. 18 can be applied when the light source unit (110) outputs a periodic modulated optical signal.

[0180]

[0181] FIGS. 19 and 20 are drawings illustrating examples of control for phase change according to one embodiment.

[0182] Referring to FIG. 19, the light source unit (110) can output a modulated light signal (1910). In the example illustrated in FIG. 19, the period of the modulated light signal (1910) can be T, and the duty ratio can be 0.1.

[0183] When the light source unit (110) outputs a modulated light signal (1910), the lighting unit (120) can perform beam scanning based on the modulated light signal (1910). A periodic structured illumination pattern corresponding to the modulated light signal (1910) can be generated in the sample. The periodic structured illumination pattern corresponding to the modulated light signal (1910) can have light spots that are continuously repeated, and an example thereof is illustrated in (a) of FIG. 20.

[0184] The detection unit (130) can collect an optical signal reflected from a sample and obtain an image of the sample (e.g., an image with a phase of 0 degrees) based on the collected optical signal.

[0185] The control unit (140) can control the light source unit (110) so that a time delay occurs in the modulated optical signal (1910). The light source unit (110) can output a modulated optical signal (1920) with a time delay (or a modulated optical signal (1920) with a time delay applied). If the modulated optical signal (1910) is, for example, m(t), the modulated optical signal (1920) with a time delay can be, for example, m(t—T / 3).

[0186] When the light source unit (110) outputs a modulated optical signal (1920), the lighting unit (120) can perform beam scanning based on the modulated optical signal (1120). A periodic structured illumination pattern corresponding to the modulated optical signal (1920) can be generated in the sample. The periodic structured illumination pattern corresponding to the modulated optical signal (1920) can have light spots that are continuously repeated, and an example thereof is illustrated in (b) of FIG. 20.

[0187] The detection unit (130) can collect an optical signal (hereinafter referred to as a “first signal”) from a sample, and can obtain an image having a first phase based on the collected first signal. The first phase may be, for example, 120 degrees, but is not limited thereto. The periodic structured illumination pattern illustrated in (b) of FIG. 20 may have a time delay of, for example, T / 3, and the first signal may also have a time delay of T / 3. This time delay may cause a phase change of the first phase. The detection unit (130) can obtain an image having a first phase based on the collected first signal.

[0188] The control unit (140) can control the light source unit (110) so that a time delay occurs again in the modulated optical signal (1920) in which a time delay occurs. The light source unit (110) can output a modulated optical signal (1930) in which a time delay occurs again. If the modulated optical signal (1920) is, for example, m(t―T / 3), the modulated optical signal (1930) in which a time delay occurs again can be, for example, m(t―2T / 3).

[0189] When the light source unit (110) outputs a modulated optical signal (1930), the lighting unit (120) can perform beam scanning based on the modulated optical signal (1930). A periodic structured illumination pattern corresponding to the modulated optical signal (1930) can be generated in the sample. The periodic structured illumination pattern corresponding to the modulated optical signal (1930) can have light spots that are continuously repeated, and an example thereof is illustrated in (c) of FIG. 20.

[0190] The detection unit (130) can collect an optical signal (hereinafter referred to as a “second signal”) from the sample, and can obtain an image having a second phase based on the collected second signal. The second phase may be, for example, 240 degrees, but is not limited thereto. The periodic structured illumination pattern illustrated in (c) of FIG. 20 may have a time delay of, for example, 2T / 3, and the second signal may also have a time delay of 2T / 3. This time delay may cause a phase change of the second phase. The detection unit (130) can obtain an image having a second phase based on the collected second signal.

[0191] The control unit (140) can obtain a high-quality image (e.g., optical sectioning enhancement) by using the images obtained by the detection unit (130) (e.g., an image with a phase of 0 degrees, an image with a first phase, and an image with a second phase). An example of this is shown in (a) of FIG. 21.

[0192]

[0193] Figure 21 is a comparison image before and after application of structural lighting according to one embodiment.

[0194] Fig. 21 (a) shows an example of a high-quality image with improved optical sectioning obtained by using a microscope device (400) with an image having a phase of 0 degrees, an image having a first phase, and an image having a second phase. Fig. 21 (b) shows an example of an image obtained only by the conventional rolling shutter-based line confocal microscope device of Fig. 7 based on uniform illumination. The image shown in Fig. 21 (a) can express optical sectioning more clearly than the image shown in Fig. 21 (b).

[0195] Although images of three phases are described through FIGS. 19 to 21, this is only an example, and the microscope device (100) can acquire images of three or more different phases.

[0196]

[0197] FIG. 22 is a drawing illustrating control for changing the structural illumination of a microscope device according to one embodiment.

[0198] Referring to FIG. 22, the microscope device (100) can form different structural illumination on the sample frame by frame. In the example shown in FIG. 22, the microscope device (100) forms the first camera frame (T frame ) can provide uniform illumination to the sample and acquire an image. After Toff has elapsed, the microscope device (100) can provide periodic structured illumination (e.g., sinusoidal structured illumination, etc.) to the sample during the second camera frame and acquire an image.

[0199] The microscope device (100) can generate high-quality images by using images acquired from each of different frames.

[0200]

[0201] FIG. 23 is a flowchart illustrating a method of operating a microscope device for generating structural illumination according to one embodiment.

[0202] Referring to FIG. 23, in step 2310, the microscope device (100) can generate a modulated optical signal by modulating an optical signal through a light source unit (110), and output the generated modulated optical signal.

[0203] In step 2320, the microscope device (100) can perform beam scanning based on the output modulated optical signal to generate a structured illumination pattern on the sample. For example, the microscope device (100) can generate a line beam for beam scanning using the output modulated optical signal, and can generate a structured illumination pattern (e.g., a pulsed wave structured illumination pattern, a sinusoidal wave structured illumination pattern, etc.) on the focal plane of the sample by illuminating the generated line beam on the sample through a plurality of lenses (e.g., a tube lens, an objective lens, etc.).

[0204] In step 2330, the microscope device (100) can collect an optical signal from the sample. For example, the microscope device (100) can collect an optical signal reflected from the sample in which the structured illumination pattern is generated or an optical signal transmitted through the sample.

[0205] In step 2340, the microscope device (100) can acquire a high-quality image based on the collected optical signal.

[0206] In one embodiment, the microscope device (100) (e.g., the microscope device (200) of FIG. 2) acquires an image from an image sensor (e.g., the first image sensor (217)) at the start of a frame (e.g., T of FIG. 13). frame ) and end (e.g. T in Fig. 13) frame The scanner (e.g., scanner (203) of FIG. 2) and the image sensor (e.g., first image sensor (217)) that perform beam scanning can be synchronized so that each of the start and end points of the structural lighting pattern is aligned with each of the start and end points of the structural lighting pattern.

[0207] In one embodiment, the microscope device (100) (e.g., the microscope device (400) of FIG. 4) may include a rolling shutter-based image sensor (e.g., the second image sensor (417) of FIG. 4). The microscope device (100) (e.g., the microscope device (400) of FIG. 4) (or the control unit (140)) may control the start and speed of the rolling line of the image sensor (417) (e.g., the V of FIG. 14). rolling ) can be synchronized with the scanner (403) and the image sensor (417) that perform beam scanning so that each is aligned with the start and speed of the structural lighting pattern. The width of the rolling line (e.g., d in FIGS. 15a and 15b) w ) is the spatial period of the structural illumination pattern (e.g., dT in Figs. 15a and 15b) and the first magnification (e.g., M in Figs. 15a and 15b). d ) may correspond to a multiple of the value multiplied by. Here, the first magnification may represent a magnification determined based on the magnification of each of the objective lens and the tube lens in the microscope device (400).

[0208] In one embodiment, for high frequency structured illumination imaging of a microscope device (100) (e.g., microscope device (400) of FIG. 4), the spatial period of the structured illumination pattern (e.g., d of FIG. 16) T ) is the size of a light spot that is continuously repeated in a structural lighting pattern (e.g., d in Fig. 16). s ) can be set to a value close to the width of the rolling line (e.g. d in Fig. 16). w ) is a set spatial period (e.g. d in Fig. 16 T ) and the first magnification, and the duty cycle of the modulated optical signal (e.g., t described through FIG. 16) on ) is the width of one pixel of a rolling shutter-based image sensor (e.g., d as described in Fig. 16). p ) may be less than the corresponding time.

[0209] In one embodiment, the microscope device (100) (e.g., the microscope device (300) of FIG. 3) may include a line sensor (321). The microscope device (100) (e.g., the microscope device (300) of FIG. 3) (or the control unit (140)) may display an image corresponding to one cycle (e.g., T of FIG. 17) of the structured illumination pattern in one frame (e.g., T of FIG. 17) of the line sensor (321). Multi-line ) can be synchronized with the light source unit (110) that outputs a modulated light signal so that it can be acquired within the line sensor (321). The sensor width of the line sensor (321) (e.g., d in FIG. 18) w ) may be larger than the width of one structural light within the structural light pattern (e.g., don in Fig. 18).

[0210] In one embodiment, the microscope device (100, 200, 300, 400) can cause a time delay to be generated in a modulated optical signal, perform beam scanning based on the modulated optical signal with the time delay to generate a structured illumination pattern within a sample, collect a first signal (e.g., a signal reflected from the sample or a signal transmitted through the sample) from the sample, and obtain an image having a first phase of the sample based on the collected first signal. The microscope device (100, 200, 300, 400) can cause a time delay again to be generated in the modulated optical signal with the time delay, perform beam scanning based on the modulated optical signal with the time delay again to generate a structured illumination pattern within the sample, collect a second signal (e.g., a signal reflected from the sample or a signal transmitted through the sample) from the sample, and obtain an image having a second phase of the sample based on the collected second signal. The microscope device (100, 200, 300, 400) can obtain a high-quality image of a sample using the acquired images (e.g., an image with a phase of 0 degrees, an image with a first phase, an image with a second phase).

[0211] In one embodiment, the microscope device (100, 200, 300, 400) can generate different structured illumination patterns on a sample for each frame of the image sensor, thereby acquiring images according to the different structured illumination patterns for each frame. The microscope device (100, 200, 300, 400) can generate high-quality images using the acquired images.

[0212] The matters described through FIGS. 1 to 22 can be applied to the operating method of the microscope device (100) of FIG. 23, and thus a detailed description thereof is omitted.

[0213]

[0214] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0215] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0216] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0217] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0218] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0219] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0220]

Claims

1. In a microscope device that generates structured illumination, A light source unit that modulates an optical signal to generate a modulated optical signal and outputs the generated modulated optical signal; An illumination unit that performs beam scanning based on the output modulated optical signal to generate a structured illumination pattern on the sample; A detection unit that collects an optical signal from the sample and obtains an image of the sample based on the collected optical signal; and A control unit that controls the beam scanning so that the lighting unit generates the structural lighting pattern on the sample, and performs at least one of synchronization between the light source unit and the detection unit and synchronization between the lighting unit and the detection unit. including, Microscope device.

2. In paragraph 1, The above lighting unit, By using the above output modulated light signal, a line beam for the beam scanning is generated, and the generated line beam is illuminated on the sample through a plurality of lenses, thereby generating a line structure illumination pattern on the focal plane of the sample. Microscope device.

3. In paragraph 1, The above lighting unit, A first lens that focuses the output modulated optical signal into a line beam; A scanner that adjusts the position of the optical axis of the line beam to perform the beam scanning; A scan lens that focuses the above line beam; a tube lens into which the above-mentioned focused line beam is incident; and An objective lens that focuses the line beam passing through the above tube lens and illuminates the sample. including, Microscope device.

4. In paragraph 1, The above control unit, Synchronizing the scanner performing the beam scanning and the image sensor so that the start and end of the frame of the image sensor of the detection unit are respectively aligned with the start and end of the structural lighting pattern, Microscope device.

5. In paragraph 1, The above detection unit includes a rolling shutter-based image sensor, The control unit synchronizes the scanner performing the beam scanning with the image sensor so that the start and speed of the rolling line of the image sensor are respectively aligned with the start and speed of the structural lighting pattern. Microscope device.

6. In paragraph 5, The width of the above rolling line corresponds to a multiple of the product of the spatial period of the above structural lighting pattern and the first magnification, The above first magnification represents a magnification determined based on the magnification of each of the objective lens and the tube lens in the microscope device. Microscope device.

7. In paragraph 5, The spatial period of the above structural lighting pattern is set to a value close to the size of a light spot that is continuously repeated in the above structural lighting pattern, The width of the above rolling line corresponds to the product of the above-set spatial period and the first magnification, The duty cycle of the above modulated optical signal is less than the time corresponding to the width of one pixel of the image sensor, The above first magnification represents a magnification determined based on the magnification of each of the objective lens and the tube lens in the microscope device. Microscope device.

8. In paragraph 1, The above detection unit includes a line sensor, The control unit synchronizes the light source unit and the line sensor so that an image corresponding to one cycle of the structural lighting pattern is acquired within one frame of the line sensor. Microscope device.

9. In paragraph 8, The sensor width of the above line sensor is larger than the width of one structural light within the above structural light pattern. Microscope device.

10. In paragraph 1, The control unit controls the light source unit so that a time delay occurs in the modulated optical signal, the lighting unit performs the beam scanning based on the modulated optical signal in which the time delay occurs and generates a structured illumination pattern within the sample, and the detection unit collects a first signal from the sample and obtains an image having a first phase of the sample based on the collected first signal. The control unit controls the light source unit so that a time delay occurs again in the modulated optical signal in which the time delay occurs, the lighting unit performs the beam scanning based on the modulated optical signal in which the time delay occurs again and generates a structured illumination pattern within the sample, and the detection unit collects a second signal from the sample and obtains an image having a second phase of the sample based on the collected second signal. The above control unit obtains a high-quality image of the sample using the obtained images. Microscope device.

11. In paragraph 1, The above control unit, The lighting unit controls the lighting unit to generate a different structural lighting pattern for each frame of the detection unit on the sample, receives an image for each frame from the detection unit, and generates a high-quality image using the image for each frame. Microscope device.

12. In a method of operating a microscope device that generates structured illumination, A step of modulating an optical signal to generate a modulated optical signal and outputting the generated modulated optical signal; A step of generating a structured illumination pattern on a sample by performing beam scanning based on the output modulated optical signal; a step of collecting an optical signal from the sample; and A step of acquiring an image of the sample based on the collected optical signal. including, Method of operation of a microscope device.

13. In paragraph 12, The above generating steps are: A step of generating a line beam for the beam scanning using the output modulated optical signal, and illuminating the sample with the generated line beam through a plurality of lenses to generate a line structure illumination pattern on the focal plane of the sample. including, Method of operation of a microscope device.

14. In paragraph 12, A step of synchronizing the scanner performing the beam scanning and the image sensor so that the start and end of each frame of the image sensor acquiring the image are aligned with the start and end of the structural lighting pattern, respectively. including more, Method of operation of a microscope device.

15. In paragraph 12, The above microscope device includes a rolling shutter based image sensor, A step of synchronizing the scanner performing the beam scanning and the image sensor so that the start and speed of the rolling line of the image sensor are respectively aligned with the start and speed of the structural lighting pattern. including more, Method of operation of a microscope device.

16. In paragraph 15, The width of the above rolling line corresponds to a multiple of the product of the spatial period of the above structural lighting pattern and the first magnification, The above first magnification represents a magnification determined based on the magnification of each of the objective lens and the tube lens in the microscope device. Method of operation of a microscope device.

17. In paragraph 15, The spatial period of the above structural lighting pattern is set to a value close to the size of a light spot that is continuously repeated in the above structural lighting pattern, The width of the above rolling line corresponds to the product of the above-set spatial period and the first magnification, The duty cycle of the above modulated optical signal is less than the time corresponding to the width of one pixel of the image sensor, The above first magnification represents a magnification determined based on the magnification of each of the objective lens and the tube lens in the microscope device. Method of operation of a microscope device.

18. In paragraph 12, The above microscope device includes a line sensor, A step of synchronizing the light source unit outputting the modulated light signal and the line sensor so that an image corresponding to one cycle of the above structural lighting pattern is acquired within one frame of the line sensor. including more, Method of operation of a microscope device.

19. In paragraph 18, The sensor width of the above line sensor is larger than the width of one structural light within the above structural light pattern. Method of operation of a microscope device.

20. In paragraph 12, A step of generating a time delay in the modulated optical signal, performing beam scanning based on the modulated optical signal with the time delay to generate a structured illumination pattern within a sample, collecting a first signal from the sample, and obtaining an image having a first phase of the sample based on the collected first signal; A step of causing a time delay to occur again in the modulated optical signal in which the time delay has occurred, performing the beam scanning based on the modulated optical signal in which the time delay has occurred again to generate a structured illumination pattern within the sample, collecting a second signal from the sample, and obtaining an image having a second phase of the sample based on the collected second signal; and A step of obtaining a high-quality image of the sample using the acquired images. including more, Method of operation of a microscope device.

21. In paragraph 12, A step of generating a different structural illumination pattern for each frame of the image sensor on the sample to acquire the image for each frame; and A step of generating a high-quality image using the above frame-by-frame images including more, Method of operation of a microscope device.

Images