Fourier ptychography microscopy and method for using thereof
The miniaturized Fourier Typography microscope addresses the limitations of existing live cell imaging by using a reduced distance and small pixels to synthesize low-resolution images for high-resolution, large-area imaging of live cells, facilitating continuous observation without fluorescent labels.
Patent Information
- Application Number
- KR1020210137121
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-10-15
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2041-10-15
AI Technical Summary
Existing live cell imaging equipment is expensive, limited by imaging speed, and requires cell fixation, making it unsuitable for continuous observation of living cells, and fluorescent labels may affect cell growth.
A miniaturized Fourier Typography microscope with a reduced distance between the illumination source and the sample, using a lighting display with small pixels to generate low-resolution images that are synthesized in the frequency domain for high-resolution imaging, enabling live cell observation without fluorescent labels.
Enables high-resolution, large-area imaging of live cells over time, maintaining spatial coherence and facilitating continuous observation without affecting cell growth, suitable for cell-based drug evaluation systems.
Smart Images

Figure 112021118054433-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a Fourier Typography microscope and a method of using the same. Specifically, the present invention relates to a Fourier Typography microscope capable of live cell imaging when mounted in a cell culture system and a method of using the same. Background Technology
[0002] Recently, there has been an increasing demand for live cell imaging equipment that can be utilized in cell experiments across fields such as biology, medicine, and pharmacy. By utilizing live cell imaging equipment, researchers can elucidate the mechanisms by which drugs act in vivo to treat diseases and assess the efficacy and toxicity of drug candidates. Furthermore, it is possible to detect and analyze molecular reactions related to intracellular signal transduction and to utilize it for studying the characteristics of stem cells. In particular, during the new drug development process, screening of new drug candidates can be effectively implemented by using live cell imaging to monitor and quantitatively analyze the long-term response of cell lines to target drugs.
[0003] Currently commercialized equipment for cell-based High Content Analysis (HCA) and High Content Screening (HCS) primarily utilizes fluorescence imaging to measure various target molecules within cells, tissues, and living organisms. These devices mainly employ laser scanning or spinning disc confocal microscopy methods and are capable of imaging various fluorescent labels at high resolution.
[0004] However, equipment using these fluorescent labels is very expensive and has limitations in imaging speed because it utilizes high-magnification lenses, precision optics, and laser light sources, and requires the mechanical, precise movement of samples for wide-area observation. Furthermore, most immunofluorescence labels used in fluorescence imaging require cell fixation during the labeling process, which may make them unsuitable for the continuous observation of living cell lines.
[0005] In addition, even in the case of fluorescent labels capable of live-cell imaging, further investigation is needed to determine whether they are suitable for HCS experiments because they affect cell growth. Therefore, imaging methods that continuously and quantitatively measure the morphological and physical information of cell lines to evaluate drug responses by imaging live cells without fluorescent labels are expected to be useful in cell-based / imaging-based drug evaluation systems.
[0006] For label-free live cell imaging, a quantitative phase imaging (QPI) method may be required to obtain refractive index and thickness information of light-nonabsorbing transparent samples from the sample's phase information. Fourier Ptychography Microscopy (FPM), a type of QPI technology, is a computational microscopy imaging technique that computationally reconstructs quantitative phase information through structural illumination and simultaneously increases the microscope's observation area and resolution using a massive space-bandwidth product. Prior art literature
[0007] Published Patent Application No. 2016-0065742 (Date of publication: June 9, 2016) The problem to be solved
[0008] The objective of the present invention is to provide a Fourier Tichography microscope that can be miniaturized by shortening the distance between the illumination source and the sample.
[0009] In addition, another objective of the present invention is to provide a method for using a Fourier typography microscope that can be miniaturized by shortening the distance between the illumination and the sample.
[0010] The objects of the present invention are not limited to those mentioned above, and other unmentioned objects and advantages of the present invention may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims. means of solving the problem
[0011] A Fourier typography microscope according to some embodiments of the present invention for solving the above problem comprises: a light display including a first pixel that generates a first incident light and a second pixel that generates a second incident light; a transparent sample container positioned on the light display at a first distance from the light display, wherein the sample container has a live cell on it and the angle of incidence of the first incident light reaching the sample container and the angle of incidence of the second incident light are different; an optical system positioned on the sample container at a second distance greater than the first distance, receiving a second light generated by the first light generated from the light display passing through the sample container to generate a first low-resolution image and a second low-resolution image, wherein the first low-resolution image and the second low-resolution image each correspond to the first incident light and the second incident light, respectively; and an image restoration module that synthesizes the first low-resolution image and the second low-resolution image in the frequency domain and inversely transforms them back into the spatial domain to generate a high-resolution image.
[0012] In addition, the diameter of the first pixel may be greater than 0 and less than 20 micrometers.
[0013] Additionally, the lighting display includes a third pixel closest to the first pixel, and the distance between the first pixel and the third pixel may be greater than 0 and less than 50 micrometers.
[0014] In addition, the lighting display may be a display of a type including at least one of an LCD type, an OLED type, a micro-LED type, and a mini-LED type.
[0015] In addition, the first distance may be greater than 0 and less than or equal to 10 millimeters.
[0016] Additionally, the cell culture housing includes a sealed culture chamber inside, and further includes a cell culture control unit that controls the culture environment inside the culture chamber to culture the live cell, and the lighting display, the transparent sample container, and the optical system may be located inside the culture chamber.
[0017] Additionally, the high-resolution image includes a first high-resolution image and a second high-resolution image corresponding to different points in time, and the Fourier typography microscope may further include a time-series data generation module that generates live cell time-series data over time using the first high-resolution image and the second high-resolution image.
[0018] Additionally, the area of the illumination FOV (field of view) of the first pixel measured in the first plane is smaller than the area of the imaging FOV of the optical system measured in the first plane, and the first plane can be defined perpendicular to the straight line connecting the illumination display and the optical system, and spaced apart by the first distance on the illumination display.
[0019] Additionally, the lighting display may display a lighting pattern including at least one light pattern and at least one dark pattern, wherein the light pattern includes at least one light-emitting pixel and the dark pattern includes at least one dark pixel.
[0020] Additionally, the image restoration module may include a Fourier transform module that converts the first low-resolution image and the second low-resolution image into a first frequency spectrum and a second frequency spectrum in the frequency domain, respectively; a stitch module that synthesizes the first frequency spectrum and the second frequency spectrum in the frequency domain to generate a high-resolution frequency spectrum; and an inverse transform module that converts the high-resolution frequency spectrum into the high-resolution image in the spatial domain.
[0021] A method for using a Fourier typography microscope according to some embodiments of the present invention for solving other problems described above comprises, at a first time point, receiving a second light generated by a first light generated from an illumination display passing through a sample container to generate a plurality of first time point low-resolution images, wherein the sample container accommodates a live cell, and synthesizing the plurality of first time point low-resolution images in the frequency domain to generate a first high-resolution image, and at a second time point later than the first time point, receiving the second light generated by the first light generated from the illumination display passing through the sample container to generate a plurality of second time point low-resolution images, synthesizing the plurality of second low-resolution images in the frequency domain to generate a second high-resolution image, and generating live cell time-series data regarding changes over time of the live cell using the first high-resolution image and the second high-resolution image.
[0022] Additionally, the first light includes a first incident light and / or a second incident light having different angles of incidence reaching the sample container, and generating the first high-resolution image may include generating a first low-resolution image corresponding to the first incident light and generating a second low-resolution image corresponding to the second incident light, wherein the plurality of low-resolution images includes the first low-resolution image and the second low-resolution image, and each of the first low-resolution image and the second low-resolution image is converted into the frequency domain to generate a first low-resolution frequency spectrum and a second low-resolution frequency spectrum, the first low-resolution frequency spectrum and the second low-resolution frequency spectrum are combined to generate a first high-resolution frequency spectrum, and the first high-resolution frequency spectrum is inversely converted into the spatial domain to generate the first high-resolution image.
[0023] In addition, the position of the sample container can be continuously fixed between the first time point and the second time point.
[0024] In addition, the live cell can be continuously cultured between the first time point and the second time point.
[0025] Additionally, the lighting display and the sample container are separated by a first distance, the sample container and the optical system are separated by a second distance, and the first distance may be smaller than the second distance. Effects of the invention
[0026] The Fourier typography microscope and the method of use according to the present invention can be mounted inside a cell culture vessel through miniaturization, thereby enabling the measurement of data on changes over time in live cells.
[0027] In addition, the small pixel size of the lighting display makes it easy to maintain spatial coherence, allowing for the acquisition of higher resolution images.
[0028] In addition to the above, the specific effects of the present invention are described together with the specific details for implementing the invention below. Brief explanation of the drawing
[0029] FIG. 1 is a conceptual diagram illustrating a Fourier tychography microscope according to some embodiments of the present invention. Figure 2 is an enlarged view illustrating the positional relationship between the illumination display, sample container, and objective lens of Figure 1. Figure 3 is a conceptual diagram illustrating the illumination FOV (Field of View) of a pixel of the illumination display of Figure 1 and the imaging FOV of an optical system. Figure 4 is a planar conceptual diagram for comparing the illumination FOV and imaging FOV of Figure 3. Figure 5 is a conceptual diagram for explaining the lighting display of Figure 1 in detail. Figure 6 is an example diagram illustrating a lighting pattern by the lighting display of Figure 1. Figure 7 shows various examples of lighting patterns by the lighting display of Figure 1. Figure 8 is a block diagram for explaining the image restoration module of Figure 1 in detail. Figure 9 is an example of a high-resolution image of Figure 1. FIG. 10 is a conceptual diagram illustrating a Fourier tychography microscope according to some embodiments of the present invention. FIGS. 11 to 16 are example diagrams for explaining the live cell time series data of FIG. 10. FIG. 17 is a flowchart illustrating a method of using a Fourier typography microscope according to some embodiments of the present invention. FIG. 18 is a flowchart for explaining in detail the steps for generating the first high-resolution image of FIG. 17. Specific details for implementing the invention
[0030] Terms and words used in this specification and claims shall not be interpreted as being limited to their general or dictionary meanings. In accordance with the principle that an inventor may define the concept of a term or word to best describe their invention, they shall be interpreted in a meaning and concept consistent with the technical spirit of the invention. Furthermore, since the embodiments described in this specification and the configurations illustrated in the drawings are merely one embodiment of the invention and do not represent the entire technical spirit of the invention, it should be understood that various equivalents, modifications, and applicable examples capable of replacing them may exist at the time of filing this application.
[0031] The terms first, second, A, B, etc., as used in this specification and claims may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.
[0032] The terms used in this specification and claims are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" should be understood as not precluding the existence or addition of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification.
[0033] In this specification and claims, where it is stated that one component is "connected" to another component, this should be understood to include cases where the components are connected through an intermediate component, as well as cases where they are directly connected. Conversely, only when it is stated that a component is "directly connected" or "directly connected" should it be understood that a component is connected to another component without any intermediate components. Likewise, other expressions describing the relationships between components should be understood in the same sense.
[0034] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains.
[0035] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0036] In addition, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to one another.
[0038] Hereinafter, with reference to FIGS. 1 to 9, a Fourier tychography microscope according to some embodiments of the present invention will be described.
[0039] FIG. 1 is a conceptual diagram illustrating a Fourier tychography microscope according to some embodiments of the present invention.
[0040] Referring to FIG. 1, a Fourier typography microscope according to some embodiments of the present invention includes an illumination display (100), a sample container (200), an optical system (300), and an image restoration module (400).
[0041] The lighting display (100) can generate light. The light generated by the lighting display (100) can pass through the sample container (200) and be transmitted to the optical system (300). The lighting display (100) can irradiate light having different angles of incidence onto the sample container (200) so that the optical system (300) and the image restoration module (400) can subsequently measure a high-resolution image (HRIm).
[0042] The sample container (200) may be positioned on the lighting display (100). The sample container (200) may be transparent so that light generated from the lighting display (100) can pass through. The sample container (200) may contain a sample inside. In this case, the sample may be a live cell, that is, a living cell. The sample container (200) may serve to fix the live cell in order to observe the live cell.
[0043] The optical system (300) can receive light generated from the lighting display (100) and transmitted through the sample container (200). The optical system (300) can receive the transmitted light to generate a low-resolution image (LRIm). There are multiple low-resolution images (LRIm), and each low-resolution image (LRIm) can have different spatial frequency information of the sample due to different angles of incidence.
[0044] At this time, the optical system (300) may be positioned on the sample container (200). However, the present embodiment is not limited thereto. That is, if the lighting display (100), the sample container (200), and the optical system (300) are positioned sequentially along the path of light, it includes cases where the relationship is not vertical as in FIG. 1.
[0045] The optical system (300) may include an objective lens (310), a tube lens (320), and an image sensor (330). The objective lens (310) may be the component of the optical system (300) located closest to the sample container (200). The objective lens (310) may serve to magnify the image of the sample container (200). The objective lens (310) may include a plurality of lenses, but is not limited thereto.
[0046] The objective lens (310) may use a lens with a relatively low magnification or a relatively low numerical aperture (NA) to enable large-area imaging. Generally, using a lens with a low magnification or a low numerical aperture allows for the acquisition of a low-resolution image over a large area. Conversely, using a lens with a high magnification or a high numerical aperture allows for the acquisition of a high-resolution image, but it is limited to a narrow area, so a large-area image cannot be acquired.
[0047] This embodiment enables large-area imaging by using an objective lens (310) with low magnification or a low numerical aperture, and subsequently allows the low-resolution image (LRIm) to be restored into a high-resolution image (HRIm) by an image restoration module (400), thereby enabling high image resolution to be obtained. Accordingly, the low-resolution image (LRIm) may be an image that displays a relatively wide area at a relatively low resolution as a large-area image.
[0048] The tube lens (320) can be used in conjunction with the objective lens (310) to assist in image formation. Parallel light generated through the objective lens (310) can form an image through the tube lens (320).
[0049] The image sensor (330) can detect an image generated by the tube lens (320). The image sensor (330) can generate current through a photoelectric element and process the current through a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). However, the present embodiment is not limited thereto.
[0050] The optical system (300) may additionally include various filters between the designed lenses. For example, at least one of a bandpass filter, a prism, and a polarizing filter may be added to the optical system (300). Through this, the present embodiment may have at least one function of a fluorescence microscope, a hyperspectral microscope, and a birefringence microscope. However, the present embodiment is not limited thereto.
[0051] The image restoration module (400) receives a low-resolution image (LRIm) and restores it to generate a high-resolution image (HRIm) with better resolution than the low-resolution image (LRIm). At this time, the low-resolution image (LRIm) is a plurality of images having different spatial frequency information, and the image restoration module (400) can generate a high-resolution image (HRIm) by synthesizing these low-resolution images (LRIm) in the frequency domain. A specific method for generating the high-resolution image (HRIm) will be explained in more detail later.
[0052] Figure 2 is an enlarged view illustrating the positional relationship between the illumination display, sample container, and objective lens of Figure 1.
[0053] Referring to FIG. 2, the lighting display (100) may include a first pixel (Px1) and a second pixel (Px2). The first pixel (Px1) and the second pixel (Px2) may be placed at different locations on the lighting display (100). Although FIG. 2 describes only two pixels, the present embodiment is not limited thereto. That is, it is entirely possible for one or three or more pixels to generate light at different locations.
[0054] A lighting display (100) can irradiate a first light (L1) onto a sample container (200), and the first light (L1) may include a first incident light (Li1) and a second incident light (Li2). At this time, a first pixel (Px1) may generate the first incident light (Li1). A second pixel (Px2) may generate the second incident light (Li2). At this time, the first incident light (Li1) and the second incident light (Li2) may be irradiated onto the sample container (200) simultaneously, or may be irradiated onto the sample container (200) with a time difference.
[0055] The first incident light (Li1) and the second incident light (Li2) may have different angles of incidence depending on the positions of the first pixel (Px1) and the second pixel (Px2). That is, the angle of incidence at which the first incident light (Li1) is irradiated onto the sample container (200) and the angle of incidence at which the second incident light (Li2) is irradiated onto the sample container (200) may be different from each other.
[0056] The first light (L1) can be converted into the second light (L2) by passing through the sample container (200). The second light (L2) may include the first transmitted light (Lt1) and the second transmitted light (Lt2). In this case, the first transmitted light (Lt1) may be generated by the first incident light (Li1) passing through the sample container (200). Similarly, the second transmitted light (Lt2) may be generated by the second incident light (Li2) passing through the sample container (200). The second light (L2) may be received by the objective lens (310).
[0057] The first distance (D1) between the illumination display (100) and the sample container (200) may be smaller than the second distance (D2) between the sample container (200) and the objective lens (310). Conventional Fourier Ptychographic Microscopy (FPM) uses an LED array for illumination. In the case of such an LED array, spatial coherence can be obtained only if the size of the LED elements is sufficiently small and the distance between the sample container (200) and the LED array is sufficiently far. This is because, given the nature of FPM which obtains high-resolution images through low-resolution images with spatial frequency differences, spatial coherence is necessary to obtain low-resolution images in which spatial frequency differences occur due to changes in the illumination angle.
[0058] Therefore, in the case of conventional FPMs, spatial coherence can be achieved by separating the sample and the light source by more than 100mm, which can result in a very long distance between the sample and the light source, and naturally, this distance between the sample and the light source can be greater than the distance between the sample and the optical system. This limitation on the distance between the sample and the light source has been the biggest obstacle to the miniaturization of FPMs.
[0059] In contrast, in some embodiments of the present invention, the first distance (D1) between the illumination display (100) and the sample container (200) in the Fourier Typography microscope may be significantly reduced to be smaller than or equal to 10 mm, which is 1 / 10 of the existing FPM. That is, the first distance (D1) may be greater than 0 and smaller than or equal to 10 mm. Accordingly, the first distance (D1) may be smaller than the second distance (D2), and the size of the Fourier Typography microscope according to some embodiments of the present invention may be manufactured on a greatly reduced scale.
[0060] This may be due to the fact that, as the size of the pixels of the lighting display (100) is small and the spacing between pixels is narrow, it is sufficiently possible to generate a low-resolution image (LRIm) having a spatial frequency difference according to the angle of incidence even if the size of the first distance (D1) is not increased.
[0061] Accordingly, the Fourier typography microscope according to some embodiments of the present invention can be designed and used in a relatively small size, and thus can be easily mounted even in equipment with limited size, such as a cell culture system. This will be explained in more detail later.
[0062] FIG. 3 is a conceptual diagram for explaining the illumination FOV (Field of View) of a pixel of the illumination display of FIG. 1 and the imaging FOV of an optical system, and FIG. 4 is a planar conceptual diagram for comparing the illumination FOV and imaging FOV of FIG. 3.
[0063] Referring to FIGS. 1, 3, and 4, the area of the illumination FOV (FOVil) of the first pixel (Px1) of the illumination display (100) may be smaller than the area of the imaging FOV (FOVim) of the optical system (300). In this case, the area of the illumination FOV (FOVil) and the area of the imaging FOV (FOVim) may be defined on a first plane (PL1). The first plane (PL1) may be a virtual plane defined to face the optical system (300) and the illumination display (100) at the location of the sample container (200). That is, the first plane (PL1) may be positioned perpendicular to the straight line connecting the illumination display (100) and the optical system (300).
[0064] This may be a phenomenon that occurs as the size of the pixels of the lighting display (100) decreases and the first distance (D1) decreases. At this time, the area of the lighting FOV of a pixel other than the first pixel (Px1) may also be smaller than the area of the imaging FOV (FOVim) of the optical system (300).
[0065] In the case of an LED array of a conventional FPM, as the LED is sufficiently large, the area of the illumination FOV is larger than the area of the imaging FOV (FOVim), so the difference in relative illumination angles at different locations within the imaging FOV (FOVim) area can be ignored.
[0066] In contrast, in a Fourier typography microscope according to some embodiments of the present invention, as the area of the illumination FOV (FOVil) becomes smaller than the area of the imaging FOV (FOVim), a difference in relative illumination angle may occur at different locations within the imaging FOV (FOVim) area. Accordingly, in a Fourier typography microscope according to some embodiments of the present invention, this difference in illumination angle can be calculated and transmitted to an image restoration module (400). Through this, the image restoration module (400) can generate a high-resolution image (HRIm) with greater accuracy and higher resolution.
[0067] Figure 5 is a conceptual diagram for explaining the lighting display of Figure 1 in detail.
[0068] Referring to FIG. 5, the lighting display (100) may further include a third pixel (Px3). The third pixel (Px3) may be the pixel closest to the first pixel (Px1). Here, “closest” may mean being placed close together without any other components between them.
[0069] The distance (G) between the first pixel (Px1) and the third pixel (Px3) may be much shorter than the distance between optical elements of a conventional FPM. Accordingly, even though the first distance (D1) is shortened, a large number of illumination FOVs (FOVil) can be located within the imaging FOV (FOVim), thereby enabling the generation of multiple low-resolution images (LRIm) and contributing to the miniaturization of the entire device. For example, the distance (G) between the first pixel (Px1) and the third pixel (Px3) may be greater than 0 and less than 50 micrometers. However, the present embodiment is not limited thereto.
[0070] Pixels including the first pixel (Px1), the second pixel (Px2), and the third pixel (Px3) may be arranged at regular intervals. However, the present embodiment is not limited thereto.
[0071] The diameters of the first pixel (Px1), the second pixel (Px2), and the third pixel (Px3) can be formed to be smaller than the diameter of the optical element of the conventional FPM, i.e., the LED. For example, the diameters of the first pixel (Px1), the second pixel (Px2), and the third pixel (Px3) can be greater than 0 and less than 20 micrometers. However, the present embodiment is not limited thereto.
[0072] The first pixel (Px1), the second pixel (Px2), and the third pixel (Px3) may all have the same size. However, the present embodiment is not limited thereto. That is, the diameters of the first pixel (Px1), the second pixel (Px2), and the third pixel (Px3) may be formed differently from each other.
[0073] As the pixel size of the lighting display (100) decreases, spatial coherence can be better maintained even though the first distance (D1) decreases. That is, in the case of conventional FPM, the distance between the sample and the lighting had to be kept long to obtain a low-resolution image in which spatial coherence occurs through light of various incident angles, but in this embodiment, even though the first distance (D1) is short, as the pixel size decreases, it is still possible to obtain a low-resolution image through spatial coherence.
[0074] Figure 6 is an example diagram illustrating a lighting pattern by the lighting display of Figure 1.
[0075] Referring to FIG. 6, the lighting display (100) can generate a desired pattern of lighting across the entire display. The lighting display (100) may be used as lighting as a whole, but only a specific area of the lighting display (100) may be used as lighting. The lighting display (100) may include a pattern area (PB), and the pattern area (PB) may be a part intended to be used as actual lighting. However, the present embodiment is not limited thereto.
[0076] The lighting display (100) may include at least one light pattern (Lp) and at least one dark pattern (Dp) within a pattern area (PB). The light pattern (Lp) is a portion that emits light within the pattern area (PB), and the dark pattern (Dp) may be a portion that is darkened within the pattern area (PB).
[0077] The light pattern (Lp) may be displayed by one or more light-emitting pixels. However, the light pattern (Lp) may not consist solely of light-emitting pixels. The brightness and size of the pattern of the light pattern (Lp) can be controlled through a combination of light-emitting pixels and dark pixels.
[0078] Similarly, the dark pattern (Dp) may be represented by one or more dark pixels, but the dark pattern (Dp) may not consist solely of dark pixels. The brightness and size of the pattern of the dark pattern (Dp) can also be controlled through a combination of light-emitting pixels and dark pixels.
[0079] The lighting display (100) may be any one of a display including at least one of, for example, an LCD method, an OLED method, a micro-LED method, and a mini-LED method. However, the present embodiment is not limited thereto.
[0080] Figure 7 shows various examples of lighting patterns by the lighting display of Figure 1.
[0081] Referring to FIG. 7, the lighting display (100) can freely implement and change lighting patterns, unlike an LED array. That is, an LED array determines the position of LED elements through the arrangement of hardware, and once the position of an LED element is determined, it cannot be changed again.
[0082] In contrast, the lighting display (100) of the present embodiment is <a1>class <a2>As such, free lighting patterns can be implemented through the emission and blackout of pixels without the need for changes in hardware configuration. Accordingly, it is possible to implement not only a simple Fourier Typography microscope, but also various types of microscopes such as a dark-field microscope using a donut-shaped pattern, a phase-contrast microscope using a semicircular pattern, a bright-field microscope using a circular pattern, and a structural illumination microscope using a fringe pattern.
[0083] Figure 8 is a block diagram for explaining the image restoration module of Figure 1 in detail.
[0084] Referring to FIGS. 1 and FIGS. 8, the image restoration module (400) may include a Fourier transform module (410), a stitch module (420), and an inverse transform module (430).
[0085] The Fourier transform module (410) can receive a low-resolution image (LRIm). The low-resolution image (LRIm) may include a first low-resolution image (LRIm1) and a second low-resolution image (LRIm2). In this case, the first low-resolution image (LRIm1) may be imaged by a first incident light (Li1) and a first transmitted light (Lt1), and the second low-resolution image (LRIm2) may be imaged by a second incident light (Li2) and a second transmitted light (Lt2). That is, the first low-resolution image (LRIm1) and the second low-resolution image (LRIm2) may be images having different spatial frequency information.
[0086] The Fourier transform module (410) can transform the first low-resolution image (LRIm1) and the second low-resolution image (LRIm2) into the first low-resolution frequency spectrum (LRSp1) and the second low-resolution frequency spectrum (LRSp2), respectively. The first low-resolution frequency spectrum (LRSp1) may be the result of Fourier transforming the first low-resolution image (LRIm1). Similarly, the second low-resolution frequency spectrum (LRSp2) may be the result of Fourier transforming the second low-resolution image (LRIm2). That is, the first low-resolution frequency spectrum (LRSp1) and the second low-resolution frequency spectrum (LRSp2) may be images in the frequency domain.
[0087] The stitch module (420) can receive a first low-resolution frequency spectrum (LRSp1) and a second low-resolution frequency spectrum (LRSp2). The stitch module (420) can synthesize the first low-resolution frequency spectrum (LRSp1) and the second low-resolution frequency spectrum (LRSp2). At this time, the stitch module (420) can identify the overlapping portions of the first low-resolution frequency spectrum (LRSp1) and the second low-resolution frequency spectrum (LRSp2) and synthesize them together. Accordingly, the stitch module (420) can generate a high-resolution frequency spectrum (HRSp). The high-resolution frequency spectrum (HRSp) may be an image of the frequency domain in which the first low-resolution frequency spectrum (LRSp1) and the second low-resolution frequency spectrum (LRSp2) are synthesized.
[0088] The inverse transformation module (430) can receive a high-resolution frequency spectrum (HRSp). The inverse transformation module (430) can inversely transform the high-resolution frequency spectrum (HRSp) into the spatial domain to generate a high-resolution image (HRIm). The high-resolution image (HRIm) may be an image having a higher resolution than the first low-resolution image (LRIm1) and the second low-resolution image (LRIm2).
[0089] Although the above description was made using two low-resolution images, the present embodiment is not limited thereto. That is, there may be three or more low-resolution images.
[0090] The image restoration module (400) may include a processor for computation. It may include a server composed of a CPU and / or a GPU. However, the present embodiment is not limited thereto.
[0091] Figure 9 is an example of a high-resolution image of Figure 1.
[0092] Referring to FIG. 9, the high-resolution image (HRIm) is a large-area image of a wide area and may have a resolution higher than the resolution according to the basic performance of the optical system (300). Through this, the present embodiment can observe the shape of the live cell over a large area with high resolution, thereby enabling very efficient research.
[0093] Hereinafter, with reference to FIGS. 10 to 16, a Fourier tychography microscope according to several embodiments of the present invention will be described. Parts that overlap with the embodiments described above will be simplified or omitted.
[0094] FIG. 10 is a conceptual diagram illustrating a Fourier tychography microscope according to some embodiments of the present invention.
[0095] Referring to FIG. 10, a Fourier typography microscope according to some embodiments of the present invention may further include a cell culture device (600) and a time series data generation module (500).
[0096] The cell culture device (600) may include a cell culture device housing (610), a culture chamber (620), and a cell culture device control unit (630).
[0097] The cell culture housing (610) may be an outer wall that isolates the culture chamber (620) from the outside to create a cell culture environment. The cell culture housing (610) may surround a lighting display (100), a sample container (200), and an optical system (300).
[0098] The culture chamber (620) may be defined as a space enclosed by a cell culture housing (610). The culture chamber (620) may include a lighting display (100), a sample container (200), and an optical system (300) inside. An image restoration module (400) may be located inside the culture chamber (620), but may also be located outside the culture chamber (620). The image restoration module (400) may receive a low-resolution image (LRIm) via a data transmission line connected from the optical system (300).
[0099] The cell culture control unit (630) can control the environment of the culture chamber (620). The cell culture control unit (630) can control at least one of temperature, supply gas, and humidity to ensure that the live cells in the sample container (200) are properly cultured. The sample container (200) is fixed so that its position does not change within the cell culture unit (600) to ensure that the live cells are properly cultured.
[0100] Since a general cell culture device (600) is equipment for culturing small-scale cells, the size of the internal chamber may not be large. Therefore, in order for the Fourier Tichography microscope of this embodiment to be positioned inside the cell culture device (600), it is essential to adjust the size of the optical system (300).
[0101] In this embodiment, the first distance (D1) between the sample container (200) and the lighting display (100) can be adjusted so that it can be mounted inside the cell culture vessel (600). At this time, a magnification that satisfies an appropriate Nyquist rate can be set by considering the magnification of the objective lens (310) and the tube lens (320), and the pixel size and numerical aperture of the image sensor (330). In order to set such a magnification, the position and arrangement of the lenses must be adjusted, and in this embodiment, the optical system (300) can be designed through an optical simulation program that adjusts the position and arrangement of the lenses according to the size of the cell culture vessel (600). In this embodiment, through the optical system (300) designed in this way, a device of a size suitable for the size of the cell culture vessel (600) can be implemented.
[0102] The time series data generation module (500) can receive high-resolution images (HRIm) from the image restoration module (400). The time series data generation module (500) can receive multiple high-resolution images (HRIm) at different points in time. The time series data generation module (500) can generate live cell time series data (Dts) using multiple high-resolution images (HRIm).
[0103] A live cell located inside a sample container (200) can be cultured by a cell culture device (600). Accordingly, the present embodiment can generate live cell time series data (Dts) that can observe changes in the live cell.
[0104] FIGS. 11 to 16 are example diagrams for explaining the live cell time series data of FIG. 10.
[0105] Referring to Figures 10 and 11, this is an example of live cell time series data that can be used to check the cell confluence status. The cell confluence status from 0h to 48h can be checked through high-resolution images (HRIm), and a graph of cell counts over time can also be provided.
[0106] Referring to FIGS. 10 and FIGS. 12, a time-lapse live cell image can be observed. Time-lapse is a method of periodically extracting some frames from video image frames captured in real time to rapidly examine changes in the image. Through the time-lapse live cell image of FIG. 12 and the cell confluency graph over time, the Fourier Tichography microscope of the present embodiment can provide live cell time-series data (Dts) that can rapidly confirm changes in the live cell.
[0107] Figures 13 and 14 show the intermediate process of measuring cell motility. As shown in Figure 13, the boundaries of the live cells are first detected to identify the live cells and verify their number. Subsequently, as shown in Figure 14, the boundaries of each live cell are colored, and their movement over time can be displayed in three dimensions. Through this live cell time-series data (Dts), the movement of each live cell can be intuitively understood.
[0108] Figure 15 shows an image sequence. That is, high-resolution images (HRIm) generated periodically or non-periodically are arranged in sequence to allow for detailed observation of changes in the cells. Figure 16 is live cell time series data (Dts) that tracks live cells by displaying the movement of each cell in different colors on a two-dimensional plane and indicates the motility of the live cells.
[0109] The Fourier typography microscope according to the present embodiment can image live cells without fluorescent labeling, enabling continuous measurement of morphological and physical information of cell lines without the influence of cell growth. Furthermore, the use of small pixels facilitates the maintenance of spatial coherence, thereby enabling easy high-resolution imaging. Additionally, the time-series changes of live cells can be generated as various data, which can be usefully applied in various fields.
[0111] Hereinafter, a method of using a Fourier Tichography microscope according to several embodiments of the present invention will be described with reference to FIGS. 1 to 18. Parts that overlap with the embodiments described above will be simplified.
[0112] FIG. 17 is a flowchart for explaining a method of using a Fourier tychography microscope according to some embodiments of the present invention, and FIG. 18 is a flowchart for explaining in detail the step of generating the first high-resolution image of FIG. 17.
[0113] First, referring to FIG. 17, a plurality of first time point low-resolution images are generated at a first time point (S100).
[0114] Specifically, referring to FIGS. 1 and 2, the optical system (300) can receive light generated from the illumination display (100) and transmitted through the sample container (200). The optical system (300) can receive the transmitted light to generate a low-resolution image (LRIm). There are multiple low-resolution images (LRIm), and each low-resolution image (LRIm) may be an image with different spatial frequency information due to light having different angles of incidence. At this time, multiple low-resolution images (LRIm) generated at a first time point are defined as multiple first time point low-resolution images.
[0115] Again, referring to FIG. 17, a plurality of first low-resolution images at a first time point are synthesized in the frequency domain to generate a first high-resolution image (S200).
[0116] Specifically, referring to FIG. 18, a first low-resolution image corresponding to a first incident light is generated (S210), and a second low-resolution image corresponding to a second incident light is generated (S220). Steps S210 and S220 can be performed in parallel.
[0117] Specifically, with reference to FIGS. 1, 2 and 8, when the first incident light (Li1) generated by the first pixel (Px1) passes through the sample container (200), it becomes the first transmitted light (Lt1), and the optical system (300) can generate a first low-resolution image through the first transmitted light (Lt1).
[0118] Similarly, when the second incident light (Li2) generated by the second pixel (Px2) passes through the sample container (200), it becomes the second transmitted light (Lt2), and the optical system (300) can generate a second low-resolution image through the second transmitted light (Lt2). A plurality of first low-resolution images may include a first low-resolution image and a second low-resolution image.
[0119] Again, referring to FIG. 18, a first low-resolution frequency spectrum is generated (S230), and a second low-resolution frequency spectrum is generated (S240). Steps S230 and S240 can each be performed in parallel with each other. Step S230 can be performed following Step S210, and Step S240 can be performed following Step S220.
[0120] Specifically, referring to FIG. 8, the Fourier transform module (410) can transform a first low-resolution image (LRIm1) and a second low-resolution image (LRIm2) into a first low-resolution frequency spectrum (LRSp1) and a second low-resolution frequency spectrum (LRSp2), respectively. The first low-resolution frequency spectrum (LRSp1) may be the result of Fourier transforming the first low-resolution image (LRIm1). Likewise, the second low-resolution frequency spectrum (LRSp2) may be the result of Fourier transforming the second low-resolution image (LRIm2).
[0121] Again, referring to FIG. 18, a first high-resolution frequency spectrum is generated by synthesizing the first and second low-resolution frequency spectra (S250).
[0122] Specifically, referring to FIG. 8, the stitch module (420) can synthesize a first low-resolution frequency spectrum (LRSp1) and a second low-resolution frequency spectrum (LRSp2). Accordingly, the stitch module (420) can generate a high-resolution frequency spectrum (HRSp).
[0123] Again, referring to FIG. 18, a first high-resolution image is generated by inversely transforming the first high-resolution frequency spectrum (S260).
[0124] Specifically, referring to FIG. 8, the inverse transform module (430) can inversely transform the high-resolution frequency spectrum (HRSp) into the spatial domain to generate a high-resolution image (HRIm). The high-resolution image (HRIm) may be an image having a higher resolution than the first low-resolution image (LRIm1) and the second low-resolution image (LRIm2).
[0125] Again, referring to FIG. 17, a plurality of second time point low-resolution images are generated at a second time point (S300).
[0126] Specifically, with reference to FIGS. 1 and FIGS. 2, a plurality of low-resolution images (LRIm) can be generated at a second time point in the same manner as the low-resolution image at a first time point. At this time, the generated plurality of low-resolution images (LRIm) are defined as the low-resolution images (LRIm) at the second time point.
[0127] Again, referring to FIG. 17, a plurality of second low-resolution images at a second time point are synthesized in the frequency domain to generate a second high-resolution image (S400).
[0128] Specifically, with reference to FIGS. 1 and FIGS. 2, a plurality of low-resolution images (LRIm) can be generated at a second time point in the same manner as the low-resolution image at a first time point. At this time, the generated plurality of low-resolution images (LRIm) are defined as the low-resolution images (LRIm) at the second time point.
[0129] Again, referring to FIG. 18, live cell time series data is generated using the first high-resolution image and the second high-resolution image (S500).
[0130] Specifically, with reference to FIGS. 10 to 16, the time series data generation module (500) can generate live cell time series data (Dts) using a plurality of high-resolution images (HRIm). The live cell time series data (Dts) can be visualized and represented by at least one of cell confluence, cell count, time-lapse image, and cell motility.
[0131] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment. Explanation of the symbols
[0132] 100: Lighting display 200: Sample container 300: Optical system 310: Objective lens 320: Tube lens 330: Image sensor 400: Image Restoration Module 410: Fourier Transform Module 420: Stitch Module 430: Inverse transformation module 500: Time Series Data Generation Module 600: Cell culture device 610: Cell culture housing 620: Culture chamber 630: Cell culture control unit
Claims
Claim 1 A Fourier Tichography microscope comprising: a lighting display including a first pixel that generates a first incident light and a second pixel that generates a second incident light; a transparent sample container positioned on the lighting display at a distance from the lighting display, wherein a live cell is positioned in the sample container and the angle of incidence of the first incident light reaching the sample container and the angle of incidence of the second incident light are different; an optical system including an objective lens, wherein the sample container and the objective lens are separated by a second distance greater than the first distance, and the first light generated from the lighting display passes through the sample container to receive the second light generated to produce a first low-resolution image and a second low-resolution image, wherein the first low-resolution image and the second low-resolution image correspond to the first incident light and the second incident light, respectively; and an image restoration module that synthesizes the first low-resolution image and the second low-resolution image in the frequency domain and inversely transforms them back into the spatial domain to generate a high-resolution image. Claim 2 In claim 1, a Fourier typography microscope in which the diameter of the first pixel is greater than 0 and less than 20 micrometers. Claim 3 In claim 1, the illumination display includes a third pixel closest to the first pixel, and the distance between the first pixel and the third pixel is greater than 0 and less than 50 micrometers, in a Fourier typography microscope. Claim 4 In claim 1, the lighting display is a Fourier Typography microscope that is a display of at least one of an LCD method, an OLED method, a micro-LED method, and a mini-LED method. Claim 5 In claim 1, the first distance is greater than 0 and less than or equal to 10 millimeters, a Fourier typography microscope. Claim 6 In claim 1, the cell culturer housing including a sealed culture chamber inside, and further comprising a cell culturer control unit for controlling a culture environment inside the culture chamber to culture the live cell, wherein the lighting display, the transparent sample container, and the optical system are a Fourier Typography microscope located inside the culture chamber. Claim 7 A Fourier tychography microscope according to claim 6, wherein the high-resolution image includes a first high-resolution image and a second high-resolution image corresponding to different points in time, and further includes a time-series data generation module that generates live cell time-series data over time using the first high-resolution image and the second high-resolution image. Claim 8 A Fourier Tichography microscope according to claim 1, wherein the area of the illumination FOV (field of view) of the first pixel measured in the first plane is smaller than the area of the imaging FOV of the optical system measured in the first plane, and the first plane is spaced apart from the illumination display by the first distance and is defined perpendicular to the straight line connecting the illumination display and the optical system. Claim 9 In claim 1, the illumination display can display an illumination pattern including at least one light pattern and at least one dark pattern, wherein the light pattern includes at least one light-emitting pixel and the dark pattern includes at least one dark pixel, a Fourier Typography microscope. Claim 10 In claim 1, the image restoration module comprises a Fourier transform module that converts the first low-resolution image and the second low-resolution image into a first frequency spectrum and a second frequency spectrum, respectively, in the frequency domain, a stitch module that synthesizes the first frequency spectrum and the second frequency spectrum in the frequency domain to generate a high-resolution frequency spectrum, and an inverse transform module that converts the high-resolution frequency spectrum into the high-resolution image in the spatial domain, thereby forming a Fourier tychography microscope. Claim 11 A method for using a Fourier Tichography microscope, comprising: at a first time point, a first light generated from a lighting display passes through a sample container and a second light generated therefrom is received by an optical system to generate a plurality of first time point low-resolution images, wherein the sample container accommodates a live cell, and the plurality of first time point low-resolution images are synthesized in the frequency domain to generate a first high-resolution image; at a second time point later than the first time point, the optical system receives the second light generated from the first light generated from the lighting display passes through the sample container and generates a plurality of second time point low-resolution images, and the plurality of second low-resolution images are synthesized in the frequency domain to generate a second high-resolution image; and using the first high-resolution image and the second high-resolution image to generate live cell time-series data regarding the change of the live cell over time, wherein the lighting display and the sample container are separated by a first distance, and the sample container and the objective lens of the optical system are separated by a second distance, and the first distance is smaller than the second distance. Claim 12 A method for using a Fourier Tichography microscope according to claim 11, wherein the first light comprises a first incident light and / or a second incident light having different angles of incidence reaching the sample container, and generating the first high-resolution image comprises generating a first low-resolution image corresponding to the first incident light and generating a second low-resolution image corresponding to the second incident light, wherein the plurality of low-resolution images includes the first low-resolution image and the second low-resolution image, and each of the first low-resolution image and the second low-resolution image is converted into the frequency domain to generate a first low-resolution frequency spectrum and a second low-resolution frequency spectrum, the first low-resolution frequency spectrum and the second low-resolution frequency spectrum are combined to generate a first high-resolution frequency spectrum, and the first high-resolution frequency spectrum is inversely converted into the spatial domain to generate the first high-resolution image. Claim 13 In claim 11, a method of using a Fourier Typography microscope in which the position of the sample container is continuously fixed between the first time point and the second time point. Claim 14 In paragraph 13, the above live cell is a Fourier Typography microscope method in which it is continuously cultured between the first time point and the second time point. Claim 15 delete
Citation Information
Patent Citations
Multiple Fourier ptychography imaging system and method
JP2018504627A
Array-level Fourier ptychography imaging
JP2018511815A
