Observation device and observation method

Abstract

An observation device (1A) for observing an object (S) to be observed is provided with a light source (11), a beam splitter (12), mirrors (13, 14), a beam splitter (15), a modulation unit (21), lenses (31-34), an imaging unit (41), and a processing unit (42). A modulation unit (21) inputs first branched light (L1) arriving from a mirror (13), spatially amplitude modulates the input first branched light (L1) on the basis of a modulation pattern, and sets the first branched light as a plurality of light components having mutually different wave number components in the x direction. The modulation pattern of the modulation unit (21) moves in a predetermined direction (x-axis direction) that intersects the propagation direction (z-axis direction) of the first branched light (L1). The processing unit (42) generates, on the basis of the detection signals repeatedly output from the imaging unit (41), a complex amplitude image when the plurality of light components are irradiated onto the object (S) to be observed, respectively. As a result, a device capable of stably observing an object to be observed is realized.

Description

Technical Field

[0001] This disclosure relates to an apparatus and method for observing an object. Background Technology

[0002] Non-Patent Document 1 discloses an invention of an apparatus and method for observing an object. The observation apparatus described in this document branches the light output from a light source into a first branch light and a second branch light. The first branch light is irradiated onto the object to be observed along multiple light irradiation directions, and the first branch light passing through the object to be observed is combined with the second branch light to cause interference between the first branch light and the second branch light.

[0003] Then, based on the intensity images (two-dimensional interference images) of the interference light arriving at the camera's imaging surface in multiple illumination directions of the first branch light, the observation device can acquire a complex amplitude image of the first branch light on the imaging surface. This observation device can image the observed object non-stainingly and non-invasively.

[0004] Existing technical documents

[0005] Non-patent literature

[0006] Non-patent document 1: OSAMU YASUHIKO et al., "Multiple-scattering suppressiverefractive index tomography for the label-free quantitative assessment of multicellular spheroids: supplement", Biomedical Optics Express, Vol.13,No.2, Supplemental Document, 2022

[0007] Non-patent literature 2: W. Choi, C. Fang-Yen, K. Badizadegan, S. Oh, N. Lue, RR Dasari, and MS Feld, "Tomographic phase microscopy", Nature Methods, Vol.4, No.9, pp.717-719, 2007

[0008] Non-patent document 3: Y. Sung, W. Choi, C. Fang-Yen, K. Badizadegan, RRDasari, and MS Feld, "Optical diffraction tomography for high resolution live cell imaging", Optics Express, Vol.17, No.1, pp.266-277, 2009 Summary of the Invention

[0009] The technical problem that the invention aims to solve

[0010] In the invention disclosed in Non-Patent Document 1, in order to illuminate the object of observation with a first branch light along multiple illumination directions, the orientation of the reflecting surface of the mirror that reflects the first branch light and illuminates the object of observation is scanned. The orientation of the reflecting surface of the mirror is stabilized by PID control. Depending on the control parameters at this time, the orientation of the reflecting surface of the mirror may sometimes become unstable or vibrate due to external disturbances. This can introduce noise into the measurement.

[0011] The purpose of this implementation is to provide an apparatus and method capable of stably observing an object.

[0012] Technical means to solve the problem

[0013] The embodiment is an observation device. The observation device is a device for observing an object and includes: (1) a light source that outputs light; (2) a branching section that branches the light into a first branch light and a second branch light; (3) a modulation section that spatially modulates the amplitude of the first branch light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branch light, and sets the first branch light to include multiple light components with different wavenumber components in the predetermined direction; (4) an illumination section that illuminates the object to be observed at a position optically conjugate to the first branch light after amplitude modulation by the modulation section onto the object to be observed; (5) a wave combining section that combines the first branch light and the second branch light after passing through the object to be observed and outputs the combined wave light; (6) a camera. The unit has an imaging surface positioned optically conjugate to the object being observed, which receives combined light and repeatedly outputs a detection signal representing a two-dimensional interference image; and (7) a processing unit that generates a complex amplitude image of the object being observed when multiple light components are respectively irradiated by the detection signal repeatedly output from the imaging unit; and (8) a processing unit that generates an image of the object being observed at each moment formed by the first branch light arriving at the imaging surface based on the temporal data of the two-dimensional interference image generated according to the detection signal repeatedly output from the imaging unit, extracts components that change over time at frequencies corresponding to the wavenumber components of the multiple light components in the specified directions of each of the multiple light components, and generates a complex amplitude image of the object being observed when multiple light components are respectively irradiated by the multiple light components.

[0014] The implementation method is an observation method. The observation method is a method for observing an object, comprising: (1) a branching step, which branches light output from a light source into a first branch light and a second branch light; (2) a modulation step, which uses a modulation unit that spatially modulates the amplitude of the first branch light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branch light, and sets the first branch light to include multiple light components with different wavenumber components in the predetermined direction; (3) an illumination step, which illuminates the object to be observed at a position optically conjugate to the modulation unit with the first branch light after amplitude modulation; (4) a wave combining step, which combines the first branch light and the second branch light after passing through the object to be observed and outputs the combined light; (5) an imaging step, which uses... The image includes an imaging unit having an imaging surface positioned optically conjugate to the object being observed, receiving combined light on the imaging surface and repeatedly outputting a detection signal representing a two-dimensional interference image; and (6) a processing step, generating a complex amplitude image of the object being observed when multiple light components are respectively irradiated based on the detection signal repeatedly output from the imaging unit; and (7) in the processing step, generating an image of the object being observed at each moment formed by the first branch light arriving at the imaging surface based on the temporal data of the two-dimensional interference image generated according to the detection signal repeatedly output from the imaging unit, extracting components that change over time at frequencies corresponding to the wavenumber components of the multiple light components in their respective predetermined directions, and generating a complex amplitude image of the object being observed when multiple light components are respectively irradiated.

[0015] The effects of the invention

[0016] The observation apparatus and observation method according to the embodiments enable stable observation of the object to be observed. Attached Figure Description

[0017] Figure 1 This is a diagram showing the structure of the observation device 1A.

[0018] Figure 2 This is a diagram showing the structure of the observation device 1B.

[0019] Figure 3 This is a diagram showing the structure of the observation device 1C.

[0020] Figure 4 This is a diagram showing the structure of the observation device 1D.

[0021] Figure 5 It represents the wavenumber vector {(k) in wavenumber space}. x,n , k y,n )} n A diagram showing an example of the distribution.

[0022] Figure 6 (a) and (b) represent based on Figure 5The wavenumber vector {(k) in the wavenumber space shown x,n , k y,n )} n The distribution of M(k) is based on equation (4). x , k y The resulting modulation pattern m(x, y) is a graph.

[0023] Figure 7 (a) and (b) represent based on Figure 5 The wavenumber vector {(k) in the wavenumber space shown x,n , k y,n )} n The distribution of M(k) is based on equation (5). x , k y The resulting modulation pattern m(x, y) is a graph.

[0024] Figure 8 (a) to (c) represent the wavenumber vectors {(k) in wavenumber space}. x,n , k y,n )} n Other examples of the distribution are shown in the graph.

[0025] Figure 9 (a) and (b) represent the wavenumber vector {(k) in wavenumber space}. x,n , k y,n )} n Other examples of the distribution are shown in the graph.

[0026] Figure 10 It represents the wavenumber vector {(k) used in the simulation}. x,n , k y,n )} n A graph showing the distribution of [something].

[0027] Figure 11 It means based on Figure 10 The wavenumber vector {(k)} shown x,n , k y,n )} n The distribution of the modulation pattern m(x, y) is used to create the graph.

[0028] Figure 12 (a) to (c) represent the u values ​​generated at various times during the simulation. out A graph of the real part of the image of (x, y, t).

[0029] Figure 13 (a) to (c) are graphs representing the complex amplitude images of each irradiation direction generated in the simulation.

[0030] Figure 14It is a graph representing the phase differential image of the observed object generated in the simulation.

[0031] Figure 15 It is a graph representing the refractive index distribution of the observed object generated in the simulation.

[0032] Figure 16 It represents the wavenumber vector {(k) used in the simulation}. x,n , k y,n )} n A graph showing the distribution of [something].

[0033] Figure 17 It means based on Figure 16 The wavenumber vector {(k)} shown x,n , k y,n )} n The distribution of the modulation pattern m(x, y) is used to create the graph.

[0034] Figure 18 It represents the wavenumber component k in the x-direction of each light component. x,n With frequency f n A diagram showing the correspondence between them.

[0035] Figure 19 This indicates that the frequency f after aliasing is... n A graph arranged in ascending order of values.

[0036] Figure 20 (a) to (c) represent the u values ​​generated at various times during the simulation. out A graph of the real part of the image of (x, y, t).

[0037] Figure 21 (a) to (c) are graphs representing the complex amplitude images of each irradiation direction generated in the simulation.

[0038] Figure 22 It is a graph representing the phase differential image of the observed object generated in the simulation.

[0039] Figure 23 It is a graph representing the refractive index distribution of the observed object generated in the simulation. Detailed Implementation

[0040] Hereinafter, embodiments of the observation apparatus and observation method will be described in detail with reference to the accompanying drawings. Furthermore, in the description of the drawings, the same elements are labeled with the same reference numerals, and repeated descriptions are omitted. The present invention is not limited to these examples, but is intended to include all modifications within the meaning and scope of the claims and their equivalents.

[0041] Figure 1This diagram shows the structure of the observation device 1A. The observation device 1A is a device for observing the object S, and includes: a light source 11, a beam splitter 12, mirrors 13 and 14, a beam splitter 15, a modulation unit 21, lenses 31-34, an imaging unit 41, and a processing unit 42. Furthermore, the observation method using the observation device 1A includes a branching step, a modulation step, an illumination step, a beam combining step, an imaging step, and a processing step.

[0042] For ease of explanation, an orthogonal xyz coordinate system is shown in this figure. The x-axis is parallel to the direction of movement of the modulation pattern of the modulation unit 21. The z-axis is parallel to the optical axes of lenses 32 and 33.

[0043] Light source 11 outputs light. Preferably, light source 11 is a laser source that outputs laser light with a single optical frequency, such as a He-Ne laser source or a semiconductor laser source. Light source 11 may include a beam expander to increase the beam diameter.

[0044] The light source 11 can also be an incoherent light source such as an SLD (Superluminescent diode) light source or an SC (SuperContinuum) light source. The light source 11 is preferably a light source with a long temporal coherence length, but it can also be a spatially incoherent LED or halogen lamp combined with a pinhole or other means to improve spatial coherence.

[0045] Beam splitter 12 is a branch that is optically connected to light source 11. Beam splitter 12 takes in light output from light source 11 and branches the input light into a first branch light L1 and a second branch light L2 (branching step). Beam splitter 12 outputs the first branch light L1 to mirror 13 and the second branch light L2 to mirror 14. Beam splitter 12 can be a semi-transparent and semi-reflective mirror.

[0046] Mirror 13 is optically connected to beam splitter 12. Mirror 13 receives the first branch light L1 output from beam splitter 12 and reflects the first branch light L1 to output to modulation unit 21.

[0047] Mirror 14 is optically connected to beam splitter 12. Mirror 14 receives the second branch light L2 output from beam splitter 12 and reflects the input second branch light L2 to output to beam splitter 15.

[0048] Beam splitter 15 receives first branch light L1 reflected by mirror 13 and arriving via modulation unit 21 and the observed object S, and receives second branch light L2 reflected by mirror 14. Beam splitter 15 is a beam combiner that combines the input first branch light L1 and second branch light L2 and outputs the combined light to camera unit 41 (beam combining step).

[0049] The first branch light L1 and the second branch light L2 traveling from the beam splitter 15 to the camera unit 41 are not parallel to each other, but are formed at a certain angle. The beam splitter 15 can be a semi-transparent and semi-reflective mirror.

[0050] The optical system extending from beam splitter 12 through mirrors 13 and 14 to beam splitter 15 constitutes a Mach-Zehnder interferometer. Along the optical path of the first branch light L1 from mirror 13 to beam splitter 15, a modulation unit 21, lens 31, lens 32, lens 33, and lens 34 are arranged sequentially. Lenses 31 to 34 can be spherical lenses. The object of observation S is positioned between lens 32 and lens 33.

[0051] The modulation unit 21 is optically connected to the mirror 13. The modulation unit 21 receives a first branch light L1 arriving from the mirror 13, spatially modulates the amplitude of the input first branch light L1 based on a modulation pattern, and outputs the modulated first branch light L1 to the lens 31 (modulation step). The modulation pattern of the modulation unit 21 moves in a predetermined direction (x-axis direction) that intersects the propagation direction (z-axis direction) of the first branch light L1.

[0052] Lenses 31 and 32 are illumination units optically connected to the modulation unit 21. Lenses 31 and 32 receive the first branch light L1, which is amplitude modulated by the modulation unit 21 and then outputs and arrives at the object S located at an optically conjugate position relative to the modulation unit 21, and illuminate the object S with the first branch light L1 (illumination step). The optical axis of lens 32 is parallel to the z-axis.

[0053] Lenses 33 and 34 allow the first branch of light L1, after passing through the object S, to be incident on the imaging surface of the imaging unit 41, forming an image of the object S on the imaging surface of the imaging unit 41. Lens 33 is the objective lens, and the optical axis of lens 33 is parallel to the z-axis.

[0054] The camera unit 41 is optically connected to the beam splitter 15. The camera unit 41 is a surface sensor such as a CCD sensor or a CMOS sensor. The imaging surface of the camera unit 41 is positioned optically conjugate relative to the object being observed, S. The camera unit 41 receives the combined light output from the beam splitter 15 at the imaging surface and repeatedly outputs a detection signal (imaging step) representing the two-dimensional interference image generated by the interference of the first branch light L1 and the second branch light L2.

[0055] The processing unit 42 is electrically connected to the imaging unit 41. The processing unit 42 receives detection signals repeatedly output from the imaging unit 41 and processes the timing data (processing steps) of the two-dimensional interference image generated based on the detection signals. Through this processing, the processing unit 42 creates a complex amplitude image of the observed object S. Furthermore, the processing unit 42 can also create a phase differential image of the observed object S and a refractive index distribution image of the observed object S.

[0056] The processing unit 42 is, for example, a computing device (computer, etc.) that integrates a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) as a processor, RAM (Random Access Memory) or ROM (Read Only Memory) as storage media, a communication module, and an input / output module. Alternatively, the processing unit 42 may also be constructed using an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0057] The modulation unit that spatially modulates the amplitude of the first branch light can take many forms. The modulation unit can be a structure including a transmissive spatial light modulator or a reflective spatial light modulator. Furthermore, the modulation unit can also be a structure including a phase-modulated spatial light modulator and a polarizer. Figures 2-4 This is a diagram showing the structure of an observation device with a modulation section having other structural examples.

[0058] Figure 2 This is a diagram showing the structure of observation device 1B. It is different from observation device 1A ( Figure 1 Compared to the structure of the observation device 1B, the observation device 1B ( Figure 2 The difference is that it has a beam splitter 16 and a modulation unit 22 instead of a mirror 13 and a modulation unit 21.

[0059] Beam splitter 16 reflects the first branch light L1 arriving from beam splitter 12 toward modulation unit 22, and allows the first branch light L1 arriving from modulation unit 22 to pass through lens 31. Modulation unit 22 is a reflective spatial light modulator capable of spatially modulating the amplitude of light.

[0060] The modulation unit 22 receives the first branch light L1 arriving from the beam splitter 16, and spatially modulates the amplitude of the first branch light L1 based on the modulation pattern to reflect it towards the beam splitter 16. The modulation unit 22 may be a DMD (Digital Mirror Device), in which case it can have a modulation pattern corresponding to the density distribution of the pixels that cause the light to be reflected towards the beam splitter 16.

[0061] Figure 3 This is a diagram showing the structure of observation device 1C. It is related to observation device 1B (…). Figure 2 Compared to the structure of the observation device 1C, the observation device 1C ( Figure 3The difference is that instead of the modulation unit 22 (amplitude modulation type spatial light modulator), it has a modulation unit that includes a phase modulation type spatial light modulator 23 and a polarizer 24.

[0062] Beam splitter 16 reflects the first branch light L1 arriving from beam splitter 12 toward spatial light modulator 23, and allows the first branch light L1 arriving from spatial light modulator 23 to pass through polarizer 24. Spatial light modulator 23 is a reflective spatial light modulator capable of spatially modulating the phase of light.

[0063] The spatial light modulator 23 receives the first branch light L1 arriving from the beam splitter 16, and spatially modulates the phase of the first branch light L1 based on the modulation pattern, reflecting it back to the beam splitter 16. The polarizer 24 receives the first branch light L1 reflected by the spatial light modulator 23 and arriving at the beam splitter 16, and selectively transmits the linearly polarized component of the first branch light L1 in a certain direction to the lens 31.

[0064] Spatial light modulator 23 is, for example, a reflective electrically addressed spatial light phase modulator LCOS-SLM (LiquidCrystal on Silicon Spatial Light Modulator). LCOS-SLM directly controls the voltage of the liquid crystal by applying CMOS technology to the addressing section, enabling high-precision and high-speed phase modulation.

[0065] The spatial light modulator 23 has a modulation surface arranged with multiple pixel regions that modulate light differently according to the polarization orientation of their respective input light and output light. Linearly polarized light of a specific orientation (e.g., an orientation at 45 degrees relative to the orientation direction of the liquid crystal in the spatial light modulator 23) is input to the spatial light modulator 23. The polarizer 24 outputs linearly polarized light from the light output from the spatial light modulator 23 at an orientation different from the aforementioned specific orientation (e.g., an orientation at 135 degrees relative to the orientation direction of the liquid crystal in the spatial light modulator 23).

[0066] By employing this structure, even if the spatial light modulator 23 is a phase-modulated type, the spatial light modulator 23 and the polarizer 24 can still form a modulation section that spatially modulates the amplitude of the first branch light L1. The object S is located at a position that is optically conjugate to the modulation surface of the spatial light modulator 23.

[0067] Figure 4 This is a diagram showing the structure of observation device 1D. It is related to observation device 1B (…). Figure 2 Compared to the structure of the observation device 1D, the observation device 1D ( Figure 4 The difference is that, instead of modulation unit 22, modulation unit 25 is provided, which is composed of a rotating reflector with a modulation pattern.

[0068] The modulation unit 25 can move the modulation pattern in a predetermined direction (x-axis direction) intersecting the propagation direction (z-axis direction) of the first branch light L1 by rotating the rotating reflector. Alternatively, a reflector that moves in the x-axis direction can be used instead of a rotating reflector. Furthermore, in the observation device 1A ( Figure 1 In the structure, a rotating transmission plate with a modulation pattern can be used instead of the modulation unit 21, or a transmission plate that moves in the x-axis direction can be used.

[0069] Next, the modulation unit ( Figure 1 Modulation section 21 in Figure 2 Modulation section 22 in Figure 3 The modulation section includes a spatial light modulator 23 and a polarizer 24. Figure 4 The amplitude modulation of the first branch light performed by the modulation unit 25 in the middle will be explained.

[0070] The modulation unit spatially modulates the amplitude of the first branch light based on a modulation pattern that moves in the x-direction. The modulation unit sets the wavenumber component k to include the x-direction through this amplitude modulation. x,n The first branch of light consists of multiple (N) distinct light components. Each light component is a plane wave. The first branch of light u is output from the modulation unit. in (x, y, t) is represented by the coherence sum of N light components as shown in equation (1) below. The wavenumber vector of the nth light component among the N light components is (k x,n , k y,n The amplitude of the nth light component is U. in (k) x,n , k y,n V is the speed at which the modulation pattern moves. t is the time variable.

[0071]

[0072] The modulation pattern of the modulation section used to generate the first branch light represented by equation (1) needs to satisfy the following conditions. Since the magnitude of the amplitude modulation is a real number between 0 and 1, when the modulation pattern representing the distribution of the magnitude of the amplitude modulation is m(x, y), m(x, y) satisfies the following relationship (2), and the Fourier transform of m(x, y) is M(k x , k y ) satisfies the following relationship (3).

[0073]

[0074]

[0075] M(k) x , ky ) represents the distribution of the wavenumber vector in the wavenumber space of the light output by amplitude modulation by the modulation unit, and therefore has a discrete distribution. As a condition required for processing in the processing unit 42, M(k) x , k y The wavenumber component k in the x-direction of the discrete distribution of ) x,n The values ​​need to be different from each other. When the camera 41 operates at a frame rate f s When K images are captured, the frequency resolution when performing a Fourier transform on the time axis is increased by f. s / K is given, therefore the wavenumber component k x,n As long as the distance is greater than or equal to that size, it is acceptable. m(x, y) is designed to satisfy all of the above conditions.

[0076] As a specific example of the design, the modulation pattern m(x, y) can be designed by performing the following steps 1 to 3. In step 1, to make the wavenumber component k in the x-direction... x,n They are different from each other, in wavenumber space k x The method of determining {k} without repetition within the range >0 is used. x,n} n For each k x,n Arbitrarily assign k y,n And determines the wavenumber vector {(k x,n , k y,n )} n .

[0077] Next, in step 2, in order to satisfy the condition of equation (3) above, so that k in wavenumber space x Within the range of <0, centered at the origin and k x The range >0 is point-symmetric, determining the wavenumber vector {(k)} x,n , k y,n )} n Furthermore, M(k) is determined in a manner that satisfies either equation (4) or (5) below. x,n , k y,n The amplitude and phase of ). (5) In equation (5), φn is an arbitrary real number. In wavenumber space, it is not related to the wavenumber vector {(k x,n , k y,n )} n Within the corresponding range, M(k) x , k y =0.

[0078]

[0079]

[0080] Next, in step 3, for M(k) x, k y Fourier transform is performed to obtain m(x, y). Then, in order to satisfy the condition of the above (2), m(x, y) is scaled by the following (6) to obtain the modulation pattern m(x, y) for generating the modulation section of the first branch light represented by the above (1).

[0081]

[0082] Figure 5 It represents the wavenumber vector {(k) in wavenumber space}. x,n , k y,n )} n A diagram illustrating an example of the distribution. In this diagram, the wavenumber vector {(k... x,n , k y,n )} n The corresponding positions are indicated by black dots. Figure 6 It means based on Figure 5 The wavenumber vector {(k) in the wavenumber space shown x,n , k y,n )} n The distribution of M(k) is based on equation (4). x , k y The resulting modulation pattern m(x, y) is a graph. Figure 7 It means based on Figure 5 The wavenumber vector {(k) in the wavenumber space shown x,n , k y,n )} n The distribution of M(k) is based on equation (5). x , k y The resulting modulation pattern m(x, y) is a graph.

[0083] exist Figure 6 and Figure 7 In each of them, the magnitude of amplitude modulation is represented by intensity, and a portion of (a) is magnified and represented in (b). To effectively utilize the dynamic range of the imaging unit 41, it is preferable that the light intensity distribution be as flat as possible; therefore, from this perspective... Figure 7 The modulation pattern m(x, y) shown is more preferred.

[0084] Figure 8 and Figure 9 It represents the wavenumber vector {(k) in wavenumber space}. x,n , k y,n )} n Other examples of the distribution are plotted. In these plots, the wavenumber vector {(k...} x,n , k y,n )} nThe corresponding positions are also indicated by black dots.

[0085] Figure 8 In the example shown in (a), for a square lattice in wavenumber space at k x After the direction is shifted by a shear, a wavenumber vector {(k)} is set at each grid point. x,n , k y,n )} n ( Figure 5 The examples shown are the same. Figure 8 In the example shown in (b), wavenumber vectors {(k) are set at each grid point of a square grid after rotating it by a certain angle around the origin of wavenumber space. x,n , k y,n )} n . Figure 8 In the example shown in (c), the wavenumber vector {(k)} is set at each non-periodic discrete location in the wavenumber space. x,n , k y,n )} n In these examples, in the wavenumber vector {(k x,n , k y,n )} n does not contain k x,n Same vector.

[0086] Figure 9 The example shown in (a) is Figure 8 A variation of the example shown in (a) makes the wavenumber vector {(k)} x,n , k y,n )} n k is a subset of the wavenumber vectors (the three wavenumber vectors enclosed by ellipses in the diagram). x,n Equal. During processing in processing unit 42, k x,n Equal wavenumber vectors cannot be separated from each other, but other wavenumber vectors can be separated from each other, therefore it is also possible to be... Figure 9 The wavenumber vector {(k)} shown in (a) x,n , k y,n )} n The distribution of .

[0087] Figure 9 In the example shown in (b), the wavenumber vector {(k)} is set at discrete positions on a straight line in wavenumber space. x,n , k y,n )} n In this example of the graph, k y,n =0. In this example, in the wavenumber vector {(k x,n , k y,n )} n does not contain k x,nThe same vector. However, in the wavenumber vector {(k x,n , k y,n )} n In the case of a one-dimensional distribution like this ( Figure 9 (b)), and wavenumber vector {(k) x,n , k y,n )} n The case of a two-dimensional distribution ( Figure 8 (a) to (c) Figure 9 Compared to (a), the configurable wavenumber vector {(k)} x,n , k y,n )} n The number N is relatively small.

[0088] To improve the accuracy of the three-dimensional image reconstruction of the observed object S, the wavenumber vector {(k x,n , k y,n )} n The more N values, the better. Furthermore, to reduce the anisotropy of the resolution of the three-dimensional image of the observed object S, the wavenumber vector {(k...} is preferred. x,n , k y,n )} n They are uniformly distributed in the two-dimensional wavenumber space. From these perspectives, the wavenumber vector {(k...} x,n , k y,n )} n The case of a two-dimensional distribution ( Figure 8 (a) to (c) Figure 9 (a) The ratio of wavenumber vector {(k) x,n , k y,n )} n The case of a one-dimensional distribution ( Figure 9 (b) is preferred.

[0089] Next, the acquisition of the two-dimensional interference image by the imaging unit 41 in the imaging step will be explained. When the first branch light u shown in equation (1) above... in When (x, y, t) light shines on the object S being observed, the first branch of light u reaches the imaging surface of the imaging unit 41. out (x, y, t) is represented by the following equation (7).

[0090]

[0091] In this formula, o(x, y) represents the image of the observed object S at a position optically conjugate to the imaging surface of the imaging unit 41. The imaging unit 41 receives the first branch light u represented by this formula (7) at the imaging surface. out(x, y, t), and a second branch light incident as a plane wave along a direction inclined relative to the incident direction of the first branch light, repeatedly output a detection signal representing a two-dimensional interference image generated by the interference of these first and second branch lights.

[0092] Next, the processing performed by the processing unit 42 in the processing steps will be explained. The processing unit 42 receives the detection signal repeatedly output from the camera unit 41 and performs the following processing: Based on the detection signal at each time moment, a two-dimensional interferometric image (i.e., time-series data of the two-dimensional interferometric image) is generated at each time moment. Then, using fringe analysis, based on the two-dimensional interferometric image at each time moment t, a first branch light u reaching the imaging plane of the camera unit 41 is generated at each time moment t. out The image of the object S formed by (x, y, t) is the image of the object S located at an optically conjugate position relative to the imaging plane.

[0093] From the first branch of light u out The image of the observed object S formed by (x, y, t) is expressed as equation (8) below when using equation (1) above.

[0094]

[0095] Equation (8) is expressed in terms of the frequency f in equation (9) below. n (and the wavenumber component k in the x-direction of each of the N light components) x,n The sum of the components that change over time (corresponding frequency). For u represented by equation (8) out (x, y, t) at discrete times {t i} i Sampling is performed. Then, samples are extracted from them at frequency f. n When time-varying components occur, as shown in equation (10) below, the light components (wavenumber vector (k) irradiating the observed object S can be obtained. x,n , k y,n ), amplitude U in (k) x,n , k y,n The complex amplitude image at time N, i.e., the complex amplitude image of each of the N illumination directions.

[0096]

[0097]

[0098] Furthermore, a phase differential image of the observed object S can be generated based on these N illumination directions. Moreover, a three-dimensional refractive index distribution image of the observed object S can be generated based on this phase differential image. The methods described in Non-Patent Document 2 or Non-Patent Document 3 can be used when generating these images.

[0099] The method described in Non-Patent Document 2 uses the Fourier Slice theorem, which is applicable when light scattering in the object S can be neglected. The method described in Non-Patent Document 3 uses the Fourier Diffraction theorem, taking into account light scattering in the object S to generate a three-dimensional refractive index distribution image of the object S.

[0100] Next, use Figures 10-15 The simulation results are explained below. The numerical apertures of lenses 32 and 33 are each set to 1.0. The wavelength of the light output from light source 11 in vacuum is set to λ0, and the moving speed V of the modulation pattern in the x-direction is set to 1000λ0 / s. The frame rate of the imaging unit 41 is set to 3000fps. The number of two-dimensional interference images captured by the imaging unit 41 is set to 312.

[0101] Figure 10 It represents the wavenumber vector {(k) used in the simulation}. x,n , k y,n )} n A graph showing the distribution of [something]. Figure 11 It means based on Figure 10 The wavenumber vector {(k)} shown x,n , k y,n )} n A diagram of the modulation pattern m(x, y) created based on the distribution of the data. Figure 11 In this context, the magnitude of amplitude modulation is represented by intensity.

[0102] Figures 12-15 It is a graph representing the various images generated in the simulation.

[0103] Figure 12 It represents u at each time step generated in the simulation. out A graph of the real part of the image of (x, y, t). (a) Represents u at t=0 ms. out (a) The real part of the image of (x, y, t). (b) Represents u at t=50ms. out (x, y, t) represents the real part of the image of (x, y, t). (c) represents u at t = 100 ms. out The real part of the image of (x, y, t).

[0104] Figure 13 This is a graph representing the complex amplitude images generated in the simulation for each illumination direction. (a) shows the complex amplitude images at the incident angle (θ). x , θ y The light component (f) of (-35.2°, -25.6°) n(a) represents the real part of the complex amplitude image when the object is illuminated at an incident angle (θ = -769.0 Hz). x , θ y The light component (f) of (-11.6°, -46.2°) n (c) represents the real part of the complex amplitude image when the object is illuminated at an incident angle (θ = -269.2Hz). x , θ y The light component (f) of (0.8°, -16.7°) n The real part of the complex amplitude image when the observed object is illuminated at 19.2 Hz.

[0105] Figure 14 This is a graph representing the phase differential image of the observed object generated in the simulation. The left end represents the phase differential image in the xz section, and the other parts represent the phase differential images in the xy sections at three positions along the z direction.

[0106] Figure 15 This is a graph representing the refractive index distribution of the observed object generated in the simulation. The top row shows the refractive index distribution image of the exact solution, and the bottom row shows the refractive index distribution image generated by the simulation. Furthermore, the left end shows the projection image of the maximum value in the y-direction, and the other parts show the refractive index distribution images in the xy sections at three locations in the z-direction.

[0107] According to this embodiment, in order to illuminate the object to be observed along multiple illumination directions, it is not necessary to mechanically move the mirror that reflects the first branch light to illuminate the object. Therefore, the instability or vibration of the mirror's reflecting surface caused by external disturbances is suppressed, and the object can be observed stably. When using a DMD as the modulation unit, the mirrors of each pixel are mechanically moved, but the mirror orientation of each pixel is not PID controlled; instead, only one of two states is selected, thus resulting in high stability.

[0108] Furthermore, in the invention disclosed in Non-Patent Document 1, the orientation of the mirror's reflective surface is scanned by mechanically moving the mirror, thus requiring a response time of approximately 10 megaseconds. Consequently, the camera's frame rate is limited to approximately 100 fps. Since the first branch light needs to illuminate approximately 300 directions towards the observed object, the measurement time takes several seconds.

[0109] In contrast, according to this embodiment, by using a modulation unit capable of rapidly moving the modulation pattern, it is possible to observe the object in a short time and also to observe dynamic changes in the object. For example, when using a DMD as the modulation unit, high-speed modulation at tens of kHz is possible, which is therefore preferred.

[0110] In the processing of the processing unit 42 described so far, according to the Nyquist theorem, in order to extract the frequency f from the above equation (8) n (Equation (9)) The time-varying component occurs only if the camera frame rate of the camera unit 41 is fast enough. The Nyquist theorem states that in order to accurately reconstruct the waveform of a signal, it is necessary to sample the signal at a speed more than twice that of the signal's maximum frequency component.

[0111] That is, let the numerical aperture of the objective lens 32 on the irradiation side be NA. in If the wavelength of light is λ0 and the wavenumber of light is k0, then k x,n The maximum value k x,max The frequency f is expressed by the following equation (11). n The maximum value f max It is expressed by the following equation (12). According to the Nyquist theorem, the frame rate f of the camera unit 41 is... s It needs to be as shown in equation (13) below for 2f max above.

[0112]

[0113]

[0114]

[0115] However, the above equation (8) does not need to be reproduced accurately without distortion, and the camera frame rate f of the camera unit 41 s No need for 2f max That is, as long as the components of light irradiating the observed object S (wavenumber vector (k)) can be obtained... x,n , k y,n ), amplitude U in (k) x,n ,k y,n The complex amplitude image at the time of (10) is obtained (the right side of the above equation). The camera frame rate f of the camera unit 41 is... s Less than 2f max In the case of undersampling, the aliasing phenomenon is utilized as described below.

[0116] Aliasing refers to the phenomenon where aliasing occurs at a rate lower than the Nyquist rate (2f). max The sampling rate f s When a continuous-time signal is sampled, the result of performing a discrete-time Fourier transform on the resulting discrete-time signal will be reduced back to [-f]. s / 2, f s The phenomenon occurs within the frequency range of [ / 2]. When the sampling rate is f... s For frequency f nWhen sampling the signal, the frequency f observed through aliasing is... n,alias It is expressed by the following equation (14). mod f s Therefore, f s For the remainder operation of the rule, give the division by f s The remainder when the time is equal to or less. For example, if we let f be the remainder when the time is equal to or less n =80Hz, f s =100Hz, then f n,alias =-20Hz.

[0117]

[0118] The frequencies observed through aliasing will differ from the actual frequencies. However, if the observed frequencies correspond one-to-one with the actual frequencies, then it is possible to obtain the various light components (wavenumber vectors (k)) illuminating the observed object S. x,n , k y,n ), amplitude U in (k) x,n , k y,n The complex amplitude image at time (right side of equation (10) above). In order to make the observed frequency correspond one-to-one with the actual frequency, it is only necessary to make the wavenumber component k in the x direction... x,n Divided by the frame rate f of the camera unit 41 s The remainders at each time point must be different.

[0119] In the case of aliasing, instead of the above equation (10), the following equation (15) is used to extract the frequency f. n The corresponding frequency f n,alias When time-varying components occur, the light components (wavenumber vector (k)) illuminating the observed object S can be obtained. x,n ,k y,n ), amplitude U in (k) x,n , k y,n The complex amplitude image at time N, i.e., the complex amplitude image of each of the N illumination directions.

[0120]

[0121] Next, use Figures 16-23 The simulation results using aliasing are explained. The numerical apertures of lenses 32 and 33 are each 1.0. The wavelength of the light output from light source 11 in vacuum is λ0, and the moving speed V of the modulation pattern in the x-direction is 1000λ0 / s. The frame rate of the imaging unit 41 is 1000fps. The number of two-dimensional interference images captured by the imaging unit 41 is 117.

[0122] Figure 16 It represents the wavenumber vector {(k) used in the simulation}. x,n , ky,n )} n A graph showing the distribution of [something]. Figure 17 It means based on Figure 16 The wavenumber vector {(k)} shown x,n , k y,n )} n A diagram of the modulation pattern m(x, y) created based on the distribution of the data. Figure 17 In this context, the magnitude of amplitude modulation is represented by intensity.

[0123] Figure 18 It represents the wavenumber component k in the x-direction of each light component. x,n With frequency f n A diagram showing the correspondence between the two. This diagram illustrates the correspondence before and after aliasing. Figure 19 This indicates that the frequency f after aliasing is... n Graphs arranged in ascending order of value. As shown in these graphs, the wavenumber component k in the x-direction... x,n Divided by the frame rate f of the camera unit 41 s The remainders at different times are different.

[0124] Figures 20-23 This is a diagram representing various images generated in a simulation that utilizes aliasing.

[0125] Figure 20 It represents u at each time step generated in the simulation. out A graph of the real part of the image of (x, y, t). (a) Represents u at t=0 ms. out (a) The real part of the image of (x, y, t). (b) Represents u at t=50ms. out (x, y, t) represents the real part of the image of (x, y, t). (c) represents u at t = 100 ms. out The real part of the image of (x, y, t).

[0126] Figure 21 This is a graph representing the complex amplitude images generated in the simulation for each illumination direction. (a) shows the complex amplitude images at the incident angle (θ). x , θ y The light component (f) of (-1.1°, 10.8°) n (a) The real part of the complex amplitude image when the object is illuminated at an incident angle (θ = -25.6Hz). (b) Represents the real part of the complex amplitude image when the object is illuminated at an incident angle (θ = -25.6Hz). x , θ y The light component (f) of (43.0°, -10.8°) n (c) represents the real part of the complex amplitude image when the observed object is illuminated at an incident angle (θ = 910.2 Hz, aliased frequency = -89.7 Hz). x , θ yThe light component (f) of (24.9°, -10.8°) n The real part of the complex amplitude image when the observed object is illuminated at 230.7 Hz.

[0127] Figure 22 This is a graph representing the phase differential image of the observed object generated in the simulation. The left end represents the phase differential image in the xz section, and the other parts represent the phase differential images in the xy sections at three positions along the z direction.

[0128] Figure 23 This is a graph representing the refractive index distribution of the observed object generated in the simulation. The top row shows the refractive index distribution image of the exact solution, and the bottom row shows the refractive index distribution image generated by the simulation. Furthermore, the left end shows the projection image of the maximum value in the y-direction, and the other parts show the refractive index distribution images in the xy sections at three locations in the z-direction.

[0129] As the simulation results show, the object can be observed even when aliasing is used. Furthermore, by using aliasing, the range of selectable modulation pattern movement speed V and camera frame rate of camera unit 41 is increased.

[0130] The observation device and observation method are not limited to the above-described embodiments and structural examples, and can be modified in various ways.

[0131] The observation device of the first embodiment described above is an apparatus for observing an object, comprising: (1) a light source that outputs light; (2) a branching section that branches the light into a first branch light and a second branch light; (3) a modulation section that spatially modulates the amplitude of the first branch light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branch light, and is configured as a first branch light containing multiple light components with different wavenumber components in the predetermined direction; (4) an illumination section that illuminates the object to be observed at a position optically conjugate to the first branch light after amplitude modulation by the modulation section onto the object; and (5) a wave combining section that combines the first branch light and the second branch light after passing through the object and outputs the combined wave. (6) An imaging unit having an imaging surface positioned optically conjugate to the object being observed, receiving combined light on the imaging surface and repeatedly outputting a detection signal representing a two-dimensional interference image; and (7) a processing unit that, based on the detection signal repeatedly output from the imaging unit, generates a complex amplitude image when multiple light components are respectively irradiated onto the object being observed; and (8) a processing unit that, based on the timing data of the two-dimensional interference image generated according to the detection signal repeatedly output from the imaging unit, generates an image of the object being observed at each moment formed by the first branch light reaching the imaging surface, extracts components whose frequency changes over time at frequencies corresponding to the wavenumber components of the multiple light components in the specified directions of each of the multiple light components, and generates a complex amplitude image when multiple light components are respectively irradiated onto the object being observed.

[0132] The observation device of the second method can be configured such that, in the structure of the first method, the modulation unit is set as a first branch light containing multiple light components whose wavenumbers are distributed in two dimensions in the wavenumber space.

[0133] The observation device of the third type can be configured such that, in the structure of the first or second type, the modulation unit is set as a first branch light containing multiple light components whose remainders when the wavenumber component of the specified direction is divided by the frame rate of the imaging unit are different from each other.

[0134] The fourth type of observation device can be configured such that, in any of the first to third types of structures, the modulation unit spatially modulates the amplitude of the first branch light based on a modulation pattern in which the values ​​of two positions symmetrical to each other at their origins in the Fourier transform of the modulation pattern are in a complex conjugate relationship.

[0135] The observation device of the fifth type can be configured such that, in any of the structures of the first to fourth types, the modulation unit includes a spatial light modulator capable of variably setting a modulation pattern.

[0136] The observation device of the sixth method can be configured such that, in any of the structures of the first to fifth methods, the processing unit generates a phase differential image of the object being observed based on the complex amplitude image when multiple light components are respectively irradiated onto the object being observed.

[0137] The observation device of the seventh method can be configured such that, in the structure of the sixth method, the processing unit generates a refractive index distribution image of the object to be observed based on the phase differential image.

[0138] The observation method of the first embodiment described above is a method for observing an object, comprising: (1) a branching step, which branches light output from a light source into a first branch light and a second branch light; (2) a modulation step, which uses a modulation unit that spatially modulates the amplitude of the first branch light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branch light, and sets the first branch light to include multiple light components with different wavenumber components in the predetermined direction; (3) an illumination step, which illuminates the object to be observed at a position optically conjugate to the modulation unit with the first branch light after amplitude modulation; (4) a wave combining step, which combines the first branch light and the second branch light after passing through the object to be observed and outputs the combined light; (5) a photographing step. (6) In the processing step, an imaging unit having an imaging surface configured at an optically conjugate position relative to the object being observed receives combined light on the imaging surface and repeatedly outputs a detection signal representing a two-dimensional interference image; and (7) in the processing step, based on the detection signal repeatedly output from the imaging unit, a complex amplitude image is generated when multiple light components are respectively irradiated onto the object being observed; and (8) in the processing step, based on the time-series data of the two-dimensional interference image generated according to the detection signal repeatedly output from the imaging unit, an image of the object being observed at each moment formed by the first branch light arriving at the imaging surface is generated, and components that change over time at frequencies corresponding to the wavenumber components of the respective specified directions of the multiple light components are extracted, and a complex amplitude image is generated when multiple light components are respectively irradiated onto the object being observed.

[0139] The second observation method can be configured such that, in the structure of the first method, during the modulation step, a first branch light is defined as containing multiple light components whose wavenumbers are distributed in two dimensions in the wavenumber space.

[0140] The third observation method can be configured such that, in the first or second method, during the modulation step, a first branch light is provided, consisting of multiple light components whose remainders differ when the wavenumber component in the specified direction is divided by the frame rate of the camera.

[0141] The fourth observation method can be configured such that, in any of the first to third methods, during the modulation step, the values ​​of two positions symmetrical to each other at their origins in the Fourier transform pattern of the modulation pattern become a complex conjugate relationship, and the amplitude of the first branch light is spatially modulated.

[0142] The fifth observation method can be configured such that, in any of the first to fourth methods, during the modulation step, a modulation unit including a spatial light modulator capable of variably setting a modulation pattern is used to spatially modulate the amplitude of the first branch light.

[0143] The sixth observation method can be configured such that, in any of the first to fifth methods, during the processing step, a phase differential image of the observed object is generated based on the complex amplitude image when multiple light components are respectively irradiated onto the observed object.

[0144] The seventh observation method can be configured such that, in the structure of the sixth method, a refractive index distribution image of the observed object is generated based on the phase differential image in the processing step.

[0145] Industrial availability

[0146] The implementation method can be used as a device and method for stably observing the object being observed.

[0147] Explanation of reference numerals in the attached figures

[0148] 1A~1D…Observation device, 11…Light source, 12…Beam splitter (branching section), 13, 14…Mirror, 15…Beam splitter (combining section), 16…Beam splitter, 21, 22…Modulation section, 23…Spatial light modulator, 24…Polarizer, 25…Modulation section, 31~34…Lens, 41…Imaging section, 42…Processing section.

Claims

1. An observation device, wherein, It is a device for observing objects. have: A light source, which outputs light; A branch section that splits the light into a first branch light and a second branch light; The modulation unit spatially modulates the amplitude of the first branch light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branch light, and is configured as a first branch light containing multiple light components with different wavenumber components in the predetermined direction. An irradiation unit irradiates the object of observation, which is located at an optically conjugate position relative to the modulation unit, with the first branch light modulated by the modulation unit. The wave combiner combines the first branch light and the second branch light after passing through the observed object and outputs the combined light. The camera unit has an imaging surface positioned optically conjugate relative to the object being observed, which receives the combined light and repeatedly outputs a detection signal representing a two-dimensional interference image. and The processing unit generates complex amplitude images of the observed object when the multiple light components are respectively irradiated onto the object based on the detection signals repeatedly output from the imaging unit. The processing unit generates an image of the observed object at each moment formed by the first branch light arriving at the imaging surface, based on the time-series data of the two-dimensional interference image generated from the detection signal repeatedly output from the imaging unit, extracts components that change over time at frequencies corresponding to the wavenumber components in the predetermined directions of the plurality of light components, and generates a complex amplitude image when the plurality of light components are respectively irradiated onto the observed object.

2. The observation device as claimed in claim 1, wherein, The modulation section is configured as a first branch of light containing multiple light components whose wavenumbers are distributed in two dimensions in the wavenumber space.

3. The observation device as described in claim 1 or 2, wherein, The modulation unit is configured as a first branch of light containing multiple light components whose remainders when the wavenumber component in the specified direction is divided by the frame rate of the camera unit are different from each other.

4. The observation device as described in any one of claims 1 to 3, wherein, The modulation unit spatially modulates the amplitude of the first branch light based on the modulation pattern in which the values ​​of two positions symmetrical to each other at their origins in the Fourier transform of the modulation pattern are in a complex conjugate relationship.

5. The observation device as described in any one of claims 1 to 4, wherein, The modulation unit includes a spatial light modulator capable of variably setting the modulation pattern.

6. The observation device as described in any one of claims 1 to 5, wherein, The processing unit generates a phase differential image of the observed object based on the complex amplitude image when the multiple light components are respectively irradiated onto the observed object.

7. The observation device as claimed in claim 6, wherein, The processing unit generates a refractive index distribution image of the observed object based on the phase differential image.

8. An observation method, wherein, It is a method of observing the object being observed. have: The branching step involves splitting the light output from the light source into a first branch light and a second branch light. The modulation step uses a modulation unit that spatially modulates the amplitude of the first branch light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branch light, wherein the first branch light comprises a plurality of light components whose wavenumber components in the predetermined direction are different from each other. The illumination step involves illuminating the object of observation, located at an optically conjugate position relative to the modulation unit, with the first branch light, whose amplitude has been modulated by the modulation unit, onto the object of observation. The beam combining step combines the first branch light and the second branch light after passing through the observed object and outputs the combined beam; The imaging step uses an imaging unit having an imaging surface positioned optically conjugate relative to the object being observed, which receives the combined light and repeatedly outputs a detection signal representing a two-dimensional interference image. and The processing step generates complex amplitude images of the observed object when the multiple light components are respectively irradiated by the detection signals repeatedly output from the camera unit. In the processing step, based on the time-series data of the two-dimensional interference image generated from the detection signal repeatedly output from the camera unit, an image of the observed object formed by the first branch light arriving at the camera surface at each moment is generated, and components that change over time at frequencies corresponding to the wavenumber components of the respective predetermined directions of the plurality of light components are extracted, and a complex amplitude image is generated when the plurality of light components are respectively irradiated onto the observed object.

9. The observation method as described in claim 8, wherein, In the modulation step, the first branch light is defined as containing multiple light components whose wavenumbers are distributed in two dimensions in the wavenumber space.

10. The observation method as described in claim 8 or 9, wherein, In the modulation step, a first branch of light is defined as a plurality of light components whose remainders when the wavenumber component in the specified direction is divided by the frame rate of the camera are different from each other.

11. The observation method according to any one of claims 8 to 10, wherein, In the modulation step, the first branch light is spatially amplitude modulated based on the modulation pattern in which the values ​​of two positions symmetrical to each other at their origins in the Fourier transform of the modulation pattern are in a complex conjugate relationship.

12. The observation method according to any one of claims 8 to 11, wherein, In the modulation step, the modulation unit, which includes a spatial light modulator capable of variably setting the modulation pattern, spatially modulates the amplitude of the first branch light.

13. The observation method according to any one of claims 8 to 12, wherein, In the processing step, a phase differential image of the observed object is generated based on the complex amplitude image when the multiple light components are respectively irradiated onto the observed object.

14. The observation method as described in claim 13, wherein, In the processing step, a refractive index distribution image of the observed object is generated based on the phase differential image.

Images