Observation device and observation method

Abstract

To provide a device that allows stable observation of an observation object.SOLUTION: An observation device 1A is a device for observing an observation object S, and comprises a light source 11, a beam splitter 12, mirrors 13, 14, a beam splitter 15, a modulation unit 21, lenses 31-34, an imaging unit 41, and a processing unit 42. The modulation unit 21 receives input of first branch light L1 arriving from the mirror 13, and applies spatial amplitude modulation to the input first branch light L1 on the basis of a modulation pattern to obtain first branch light including a plurality of light components different from each other in wavenumber components in an x direction. The modulation pattern of the modulation unit 21 moves in a predetermined direction (x-axis direction) intersecting a propagation direction (z-axis direction) of the first branch light L1. The processing unit 42 creates complex amplitude images when the observation object S is irradiated with the plurality of light components on the basis of detection signals repeatedly output from the imaging unit 41.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an apparatus and method for observing an object. [Background technology]

[0002] Non-Patent Document 1 discloses an invention of an apparatus and method for observing an object to be observed. The observation apparatus described in this document branches light output from a light source into a first branched light and a second branched light, irradiates the object to be observed along each of a plurality of light irradiation directions with the first branched light, combines the first branched light and the second branched light that have passed through the object to be observed, and causes the first branched light and the second branched light to interfere with each other. Then, this observation apparatus can obtain a complex amplitude image of the first branched light on the imaging plane based on an intensity image (two-dimensional interference image) of the interference light that has reached the imaging plane of the camera for each of a plurality of light irradiation directions of the first branched light. This observation apparatus can image the object to be observed non-stained and non-invasively. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] OSAMU YASUHIKO, et al, "Multiple-scattering suppressive refractive index tomography for the label-free quantitative assessment of multicellular spheroids:supplement," Biomedical Optics Express, Vol.13, No.2, (2022). [Non-Patent Document 2] W. Choi, C. Fang-Yen, K. Badizadegan, S. Oh, N. Lue, RR Dasari, and MS Feld, "Tomographicphase microscopy," Nat Meth 4(9), 717-719 (2007). [Non-Patent Document 3] Y. Sung, W. Choi, C. Fang-Yen, K. Badizadegan, RR Dasari, and MS Feld, "Optical diffraction tomography for high resolution live cell imaging," Optics Express 17(1), 266 (2009). Summary of the Invention [Problem to be solved by the invention]

[0004] In the invention disclosed in Non-Patent Document 1, in order to irradiate the first branched light to the observation object along each of a plurality of irradiation directions, the orientation of the reflective surface of a mirror that reflects the first branched light and irradiates the observation object is scanned. The orientation of the reflective surface of the mirror is stabilized by PID control. Depending on the control parameters at this time, the orientation of the reflective surface of the mirror may become unstable or vibrate due to external disturbance. This may cause noise in the measurement.

[0005] The present invention has been made to solve the above problems, and has an object to provide an apparatus and method that enable stable observation of an observation target object. [Means for solving the problem]

[0006] The observation device of the present invention is a device for observing an observation object. A first aspect of the observation device of the present invention includes: (1) a light source that outputs light; (2) a branching unit that branches the light into a first branched light and a second branched light; (3) a modulation unit that spatially amplitude-modulates the first branched light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branched light to produce a first branched light including a plurality of light components having different wave number components in the predetermined direction; (4) an irradiation unit that irradiates an observation object located at a position optically conjugate to the modulation unit with the first branched light after being amplitude-modulated by the modulation unit; (5) a combining unit that combines the first branched light and the second branched light after passing through the observation object and outputs a combined light; (6) an imaging unit that has an imaging surface located at a position optically conjugate to the observation object, receives the combined light at the imaging surface, and repeatedly outputs a detection signal representing a two-dimensional interference image; and (7) a processing unit that generates a complex amplitude image when each of the plurality of light components is irradiated onto the observation object based on the detection signal repeatedly output from the imaging unit. The processing unit generates an image of the object to be observed at each time by the first branched light that has reached the imaging surface based on time series data of a two-dimensional interference image generated from detection signals repeatedly output from the imaging unit, extracts components that change over time at a frequency corresponding to the wave number component in the specified direction of each of the multiple light components, and generates a complex amplitude image when each of the multiple light components is irradiated onto the object to be observed.

[0007] In a second aspect of the observation device of the present invention, in addition to the first aspect, the modulation section generates a first branched light including a plurality of light components whose wave numbers are distributed two-dimensionally in wave number space.

[0008] In a third aspect of the observation device of the present invention, in addition to the first or second aspect, the modulation unit generates a first branched light including a plurality of light components having mutually different surpluses when the wave number component in the predetermined direction is divided by the frame rate of the imaging unit.

[0009] In a fourth aspect of the observation device of the present invention, in addition to any of the first to third aspects, the modulation section spatially amplitude-modulates the first branched light based on a modulation pattern in which values ​​of two positions that are symmetrical with respect to the origin in a Fourier transform pattern of the modulation pattern are in a complex conjugate relationship.

[0010] In a fifth aspect of the observation device of the present invention, in addition to any one of the first to fourth aspects, the modulation section includes a spatial light modulator capable of variably setting the modulation pattern.

[0011] In a sixth aspect of the observation device of the present invention, in addition to any of the first to fifth aspects, the processing unit generates a phase differential image of the object to be observed based on a complex amplitude image obtained when each of the multiple light components is irradiated onto the object to be observed.

[0012] In a seventh aspect of the observation apparatus of the present invention, in addition to the sixth aspect, the processing unit generates a refractive index distribution image of the observation object based on the phase differential image.

[0013] The observation method of the present invention is a method for observing an object to be observed. A first aspect of the observation method of the present invention includes: (1) a branching step of branching light output from a light source into a first branched light and a second branched light; (2) a modulation step of using a modulation unit that spatially amplitude-modulates the first branched light based on a modulation pattern that moves in a predetermined direction intersecting the propagation direction of the first branched light to generate the first branched light including a plurality of light components having different wave number components in the predetermined direction; (3) an irradiation step of irradiating an observation object located at a position optically conjugate to the modulation unit with the first branched light after being amplitude-modulated by the modulation unit; (4) a combining step of combining the first branched light and the second branched light after passing through the observation object and outputting a combined light; (5) an imaging step of receiving the combined light at an imaging surface using an imaging unit having an imaging surface located at a position optically conjugate to the observation object and repeatedly outputting a detection signal representing a two-dimensional interference image; and (6) a processing step of generating a complex amplitude image when each of the plurality of light components is irradiated onto the observation object based on the detection signal repeatedly output from the imaging unit. In the processing step, an image of the object to be observed at each time is generated by the first branched light that has reached the imaging surface based on time series data of a two-dimensional interference image generated from detection signals repeatedly output from the imaging unit, and a complex amplitude image is generated when each of the plurality of light components is irradiated onto the object to be observed by extracting components that change over time at a frequency corresponding to the wave number component in the specified direction of each of the plurality of light components.

[0014] In a second aspect of the observation method of the present invention, in addition to the first aspect, the modulation step generates a first branched light including a plurality of light components whose wave numbers are distributed two-dimensionally in wave number space.

[0015] In a third aspect of the observation method of the present invention, in addition to the first or second aspect, in the modulation step, the first branched light includes a plurality of light components whose surpluses when the wave number component in the predetermined direction is divided by the frame rate of the imaging unit are different from each other.

[0016] In a fourth aspect of the observation method of the present invention, in addition to any one of the first to third aspects, in the modulation step, the first branched light is spatially amplitude-modulated based on a modulation pattern in which values ​​of two positions that are symmetrical with respect to the origin in a Fourier transform pattern of the modulation pattern are in a complex conjugate relationship.

[0017] In a fifth aspect of the observation method of the present invention, in addition to any one of the first to fourth aspects, in the modulation step, the first branched light is spatially amplitude-modulated using a modulation section including a spatial light modulator capable of variably setting a modulation pattern.

[0018] In a sixth aspect of the observation method of the present invention, in addition to any one of the first to fifth aspects, in the processing step, a phase differential image of the object to be observed is generated based on complex amplitude images obtained when each of a plurality of light components is irradiated onto the object to be observed.

[0019] In a seventh aspect of the observation method of the present invention, in addition to the sixth aspect, a refractive index distribution image of the observation object is generated based on the phase differential image in the processing step. Effect of the Invention

[0020] According to the present invention, an object to be observed can be stably observed. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram showing the configuration of an observation apparatus 1A. [Diagram 2] FIG. 2 is a diagram showing the configuration of the observation device 1B. [Diagram 3] FIG. 3 is a diagram showing the configuration of the observation apparatus 1C. [Figure 4] FIG. 4 is a diagram showing the configuration of the observation device 1D. [Diagram 5] FIG. 5 is a diagram showing an example of the distribution of wave number vectors {(kx,n,ky,n)}n in wave number space. [Figure 6]FIG. 6 is a diagram showing a modulation pattern m(x, y) obtained from M(kx, ky) created according to equation (4) based on the distribution of wave number vectors {(kx, n, ky, n)}n in the wave number space shown in FIG. [Figure 7] FIG. 7 is a diagram showing a modulation pattern m(x, y) obtained from M(kx, ky) created according to equation (5) based on the distribution of wave number vectors {(kx, n, ky, n)}n in the wave number space shown in FIG. [Figure 8] FIG. 8 is a diagram showing another example of the distribution of wave number vectors {(kx,n,ky,n)}n in wave number space. [Figure 9] FIG. 9 is a diagram showing another example of the distribution of wave number vectors {(kx,n,ky,n)}n in wave number space. [Figure 10] FIG. 10 is a diagram showing the distribution of wave vectors {(kx,n,ky,n)}n used in the simulation. [Figure 11] FIG. 11 is a diagram showing a modulation pattern m(x, y) created based on the distribution of wave vectors {(kx, n, ky, n)} n shown in FIG. [Figure 12] FIG. 12 is a diagram showing the real part of the image of uout(x, y, t) at each time generated in the simulation. [Figure 13] FIG. 13 is a diagram showing complex amplitude images in each irradiation direction generated by the simulation. [Figure 14] FIG. 14 is a diagram showing a phase differential image of an object to be observed generated by simulation. [Figure 15] FIG. 15 is a diagram showing a refractive index distribution image of an object to be observed generated by simulation. [Figure 16] FIG. 16 is a diagram showing the distribution of wave vectors {(kx,n,ky,n)}n used in the simulation. [Figure 17] FIG. 17 is a diagram showing a modulation pattern m(x, y) created based on the distribution of wave vectors {(kx, n, ky, n)} n shown in FIG. [Figure 18]FIG. 18 is a diagram showing the correspondence relationship between the wave number component kx,n in the x direction of each light component and the frequency fn. [Figure 19] FIG. 19 is a diagram showing frequencies fn after aliasing arranged in order from smallest to largest. [Figure 20] FIG. 20 is a diagram showing the real part of the image of uout(x, y, t) at each time generated in the simulation. [Figure 21] FIG. 21 is a diagram showing complex amplitude images in each irradiation direction generated by simulation. [Figure 22] FIG. 22 is a diagram showing a phase differential image of an object to be observed generated by simulation. [Diagram 23] FIG. 23 is a diagram showing a refractive index distribution image of an observation object generated by simulation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated descriptions are omitted. The present invention is not limited to these examples, but is indicated by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0023] 1 is a diagram showing the configuration of an observation device 1A. The observation device 1A is a device for observing an observation target S, and includes a light source 11, a beam splitter 12, mirrors 13 and 14, a beam splitter 15, a modulation section 21, lenses 31 to 34, an imaging section 41, and a processing section 42. In addition, an observation method using this observation device 1A includes a branching step, a modulation step, an irradiation step, a combining step, an imaging step, and a processing step.

[0024] For ease of explanation, an xyz orthogonal coordinate system is shown in this figure. The x-axis is parallel to the direction of movement of the modulation pattern of the modulation unit 21. The z-axis is parallel to the optical axis of the lenses 32 and 33.

[0025] The light source 11 outputs light. Preferably, the light source 11 is a laser light source that outputs laser light having a single optical frequency, such as a He-Ne laser light source or a semiconductor laser light source. The light source 11 may include a beam expander that expands the beam diameter of the light. The light source 11 may be an incoherent light source such as an SLD (Superluminescent Diode) light source or an SC (Super Continuum) light source. The light source 11 is preferably a light source with a long temporal coherence length, but may be a spatially incoherent LED or halogen lamp combined with a pinhole or the like that improves spatial coherence.

[0026] The beam splitter 12 is a branching section optically connected to the light source 11. The beam splitter 12 receives the light output from the light source 11 and branches the received light into a first branched light L1 and a second branched light L2 (branching step). The beam splitter 12 outputs the first branched light L1 to a mirror 13 and outputs the second branched light L2 to a mirror 14. The beam splitter 12 may be a half mirror.

[0027] The mirror 13 is optically connected to the beam splitter 12. The mirror 13 receives the first branched light L1 output from the beam splitter 12, reflects the received first branched light L1, and outputs the reflected light to the modulation unit 21.

[0028] The mirror 14 is optically connected to the beam splitter 12. The mirror 14 receives the second branched light L2 output from the beam splitter 12, reflects the received second branched light L2, and outputs the reflected light to the beam splitter 15.

[0029] The beam splitter 15 inputs the first branched light L1 that has been reflected by the mirror 13 and arrived via the modulation section 21 and the observation object S, etc., and also inputs the second branched light L2 that has been reflected by the mirror 14 and arrived. The beam splitter 15 is a multiplexing section, which multiplexes the input first branched light L1 and second branched light L2 and outputs the multiplexed light to the imaging section 41 (multiplexing step). The traveling directions of the first branched light L1 and the second branched light L2 from the beam splitter 15 to the imaging section 41 are not parallel to each other but form a certain angle. The beam splitter 15 may be a half mirror.

[0030] The optical system from the beam splitter 12 through the mirrors 13 and 14 to the beam splitter 15 constitutes a Mach-Zehnder interferometer. On the optical path of the first split light L1 from the mirror 13 to the beam splitter 15, a modulation unit 21, a lens 31, a lens 32, a lens 33, and a lens 34 are arranged in this order. The lenses 31 to 34 may be spherical lenses. The observation object S is arranged between the lenses 32 and 33.

[0031] The modulation unit 21 is optically connected to the mirror 13. The modulation unit 21 inputs the first branched light L1 arriving from the mirror 13, spatially amplitude-modulates the input first branched light L1 based on a modulation pattern, and outputs the modulated first branched light L1 to the lens 31 (modulation step). The modulation pattern of the modulation unit 21 moves in a predetermined direction (x-axis direction) intersecting with the propagation direction (z-axis direction) of the first branched light L1.

[0032] The lenses 31 and 32 are an irradiation unit optically connected to the modulation unit 21. The lenses 31 and 32 input the first branched light L1 that has been amplitude-modulated by the modulation unit 21 and then output and reaches the first branched light L1, and irradiate the observation object S located at a position optically conjugate with the modulation unit 21 (irradiation step). The optical axis of the lens 32 is parallel to the z-axis.

[0033] The lenses 33 and 34 allow the first branched light L1 after passing through the observation object S to be incident on the imaging surface of the imaging unit 41, so that an image of the observation object S can be formed on the imaging surface of the imaging unit 41. The lens 33 is an objective lens, and the optical axis of the lens 33 is parallel to the z-axis.

[0034] The imaging unit 41 is optically connected to the beam splitter 15. The imaging unit 41 is an area sensor such as a CCD sensor or a CMOS sensor. The imaging surface of the imaging unit 41 is disposed at a position optically conjugate with the observation object S. The imaging unit 41 receives the combined light output from the beam splitter 15 on the imaging surface, and repeatedly outputs a detection signal representing a two-dimensional interference image due to interference between the first branched light L1 and the second branched light L2 (imaging step).

[0035] The processing unit 42 is electrically connected to the imaging unit 41. The processing unit 42 inputs detection signals repeatedly output from the imaging unit 41, and processes time series data of a two-dimensional interference image generated from the detection signals (processing step). The processing unit 42 creates a complex amplitude image of the observation object S through this processing. Furthermore, the processing unit 42 can also create a phase differential image of the observation object S, and can also create a refractive index distribution image of the observation object S.

[0036] The processing unit 42 is, for example, a calculation device (such as a computer) incorporating a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a storage medium such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a communication module, and an input / output module, etc. Also, for example, the processing unit 42 may be configured with an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0037] The modulation section that spatially amplitude-modulates the first split light can take various forms. The modulation section may be configured to include a transmissive spatial light modulator, or may be configured to include a reflective spatial light modulator. The modulation section may also be configured to include a phase-modulation spatial light modulator and a polarizer. Figures 2 to 4 are diagrams showing the configuration of an observation device equipped with a modulation section of another configuration example.

[0038] FIG. 2 is a diagram showing the configuration of the observation device 1B. Compared with the configuration of the observation device 1A (FIG. 1), the observation device 1B (FIG. 2) is different in that it includes a beam splitter 16 and a modulation unit 22 instead of the mirror 13 and the modulation unit 21. The beam splitter 16 reflects the first branched light L1 arriving from the beam splitter 12 to the modulation unit 22, and transmits the first branched light L1 arriving from the modulation unit 22 to the lens 31. The modulation unit 22 is a reflective spatial light modulator that can spatially modulate the amplitude of light. The modulation unit 22 inputs the first branched light L1 arriving from the beam splitter 16, spatially modulates the amplitude of the first branched light L1 based on a modulation pattern, and reflects it to the beam splitter 16. The modulation unit 22 may be a DMD (Digital Mirror Device), and in this case, it can have a modulation pattern according to the density distribution of pixels that reflect light to the beam splitter 16.

[0039] FIG. 3 is a diagram showing the configuration of the observation device 1C. Compared with the configuration of the observation device 1B (FIG. 2), the observation device 1C (FIG. 3) is different in that it includes a modulation section including a phase modulation type spatial light modulator 23 and a polarizer 24 instead of the modulation section 22 (amplitude modulation type spatial light modulator). The beam splitter 16 reflects the first branched light L1 arriving from the beam splitter 12 to the spatial light modulator 23, and transmits the first branched light L1 arriving from the spatial light modulator 23 to the polarizer 24. The spatial light modulator 23 is of a reflective type that can spatially phase-modulate light. The spatial light modulator 23 inputs the first branched light L1 arriving from the beam splitter 16, spatially phase-modulates the first branched light L1 based on the modulation pattern, and reflects it to the beam splitter 16. The polarizer 24 inputs the first branched light L1 that has been reflected by the spatial light modulator 23 and has reached the polarizer 24 via the beam splitter 16, and selectively transmits a linearly polarized component of the first branched light L1 in a certain direction to the lens 31.

[0040] The spatial light modulator 23 is, for example, a reflective, electrically addressed spatial light phase modulator, LCOS-SLM (Liquid Crystal on Silicon Spatial Light Modulator). The LCOS-SLM can perform highly accurate and fast-response phase modulation by directly controlling the voltage of the liquid crystal by applying CMOS technology to the address section. The spatial light modulator 23 has a modulation surface on which a plurality of pixel regions are arranged, each of which performs different modulation depending on the polarization direction of the input light and outputs the light. Linearly polarized light in a specific direction (for example, 45 degrees with respect to the orientation direction of the liquid crystal in the spatial light modulator 23) is input to the spatial light modulator 23. The polarizer 24 outputs linearly polarized light in a direction different from the specific direction (for example, 135 degrees with respect to the orientation direction of the liquid crystal in the spatial light modulator 23) from the light output from the spatial light modulator 23.

[0041] With this configuration, even if the spatial light modulator 23 is of a phase modulation type, the spatial light modulator 23 and the polarizer 24 can form a modulation section that spatially amplitude-modulates the first split light L1. The observation object S is located at a position optically conjugate with the modulation surface of the spatial light modulator 23.

[0042] FIG. 4 is a diagram showing the configuration of the observation device 1D. Compared with the configuration of the observation device 1B (FIG. 2), the observation device 1D (FIG. 4) is different in that it includes a modulation unit 25 consisting of a rotary reflector having a modulation pattern instead of the modulation unit 22. This modulation unit 25 can move the modulation pattern in a predetermined direction (x-axis direction) intersecting with the propagation direction (z-axis direction) of the first branched light L1 by rotating the rotary reflector. Instead of the rotary reflector, a reflection plate that moves in the x-axis direction may be used. Also, in the configuration of the observation device 1A (FIG. 1), instead of the modulation unit 21, a rotary transmission plate having a modulation pattern or a transmission plate that moves in the x-axis direction may be used.

[0043] Next, the amplitude modulation of the first branched light by the modulation section (modulation section 21 in FIG. 1, modulation section 22 in FIG. 2, modulation section including spatial light modulator 23 and polarizer 24 in FIG. 3, modulation section 25 in FIG. 4) in the modulation step will be described. The modulation section spatially amplitude-modulates the first branched light based on a modulation pattern moving in the x direction. By this amplitude modulation, the modulation section modulates the wave number component k in the x direction. x,n The first branched light u output from the modulator is a multiple (N) optical component with different wavelengths. Each optical component is a plane wave. in (x, y, t) is expressed as the coherent sum of N optical components as shown in the following equation (1). The wave vector of the nth optical component among the N optical components is (k x,n ,k y,n ), and the amplitude of the nth light component is U in (k x,n ,k y,n ) where V is the moving speed of the modulation pattern. t is the time variable.

[0044]

number

[0045] The modulation pattern of the modulation section for generating the first branched light expressed by the formula (1) needs to satisfy the following conditions. Since the magnitude of the amplitude modulation is a real number between 0 and 1, when the modulation pattern representing the distribution of the magnitude of the amplitude modulation is m(x, y), m(x, y) satisfies the relationship of the formula (2) below, and the Fourier transform of m(x, y) is M(k x ,k y ) satisfies the relationship of the following equation (3). x ,k y ) represents the distribution of wave vectors in the wave number space of the light that is amplitude modulated and output by the modulator, and therefore has a discrete distribution. x ,k y ) in the discrete distribution of x-direction wave number component k x,n are required to be different values ​​as a necessary condition for processing in the processing unit 42. s When K images are taken, the frequency resolution when they are Fourier transformed on the time axis is f s Since it is given by / K, the wave number component k x,n It is sufficient that m(x,y) is separated by this amount or more. m(x,y) is designed to satisfy the above conditions.

[0046]

number

[0047]

number

[0048] As a specific example of the design, a modulation pattern m(x,y) can be designed by performing the following steps 1 to 3. In step 1, the wave number component k in the x direction is x,n In order to make the k x >0, so that there are no overlaps in the range {k x,n} n Determine each k x,n For any ky,n and assign the wave vector {(k x,n ,k y,n )} n Determine.

[0049] Next, in step 2, in order to satisfy the condition of the above equation (3), k in the wave number space is x <0, centered at the origin, k x >0, the wave vector {(k x,n ,k y,n )} n Also, M(k x,n ,k y,n Determine the amplitude and phase of φ in Eq. (5). n is an arbitrary real number. In the wave number space, the wave number vector {(k x,n ,k y,n )} n In the range that does not correspond to M(k x ,k y )=0.

[0050]

number

[0051]

number

[0052] Furthermore, in step 3, M(k x ,k y ) is Fourier transformed to obtain m(x, y), and then m(x, y) is scaled by the following equation (6) so as to satisfy the condition of equation (2) above, thereby obtaining the modulation pattern m(x, y) of the modulation section for generating the first branched light expressed by equation (1) above.

[0053]

number

[0054] Figure 5 shows the wave vector {(k x,n ,k y,n )} n In this figure, the wave vector {(k x,n ,k y,n )} n The positions corresponding to the wave vectors {(k x,n ,k y,n )} n Based on the distribution of M(k x ,k y 7 shows a modulation pattern m(x,y) obtained from the wave number vector {(k x,n ,k y,n )} n Based on the distribution of M(k x ,k y 6 and 7 show a modulation pattern m(x, y) obtained from the amplitude modulation shown in FIG. 6 by shading, and a part of FIG. 6 (a) is enlarged and shown in FIG. 7 (b). In order to effectively utilize the dynamic range of the image captured by the image capture unit 41, it is preferable that the light intensity distribution is as flat as possible, and from this point of view, the modulation pattern m(x, y) shown in FIG. 7 is more preferable.

[0055] Figures 8 and 9 show the wave vector {(k x,n ,k y,n )} n 13A and 13B are diagrams showing other examples of the distribution of wave vectors {(k x,n ,k y,n )} n The corresponding positions are indicated by black circles.

[0056] The example shown in Fig. 8(a) is a square lattice in wave number space, k x After shearing in the direction, each lattice point is assigned a wave vector {(k x,n ,k y,n )} n(The same applies to the example shown in Fig. 5.) In the example shown in Fig. 8(b), the wave vector {(k x,n ,k y,n )} n The example shown in FIG. 8(c) is a non-periodic wave number space with a discrete wave number vector {(k x,n ,k y,n )} n In these examples, the wave vector {(k x,n ,k y,n )} n In x,n However, no identical items are included.

[0057] The example shown in FIG. 9(a) is a modification of the example shown in FIG. 8(a), and is a wave vector {(k x,n ,k y,n )} n k of some of the wave vectors (three wave vectors enclosed by ellipses in the figure) x,n During processing by the processing unit 42, k x,n Wave vectors with the same k cannot be separated from each other, but other wave vectors can be separated from each other, so we have the wave vectors {(k x,n ,k y,n )} n It may be a distribution of

[0058] The example shown in FIG. 9(b) shows that the wave vector {(k x,n ,k y,n )} n In the example shown in this figure, k y,n =0. In this example, the wave vector {(k x,n ,k y,n )} n In x,n However, the wave vector {(k x,n ,k y,n )} nWhen is distributed one-dimensionally (Figure 9(b)), the wave vector {(k x,n ,k y,n )} n In comparison with the case where is distributed two-dimensionally (Figs. 8(a)-(c), Fig. 9(a)), the number of configurable wave vectors {(k x,n ,k y,n )} n The number N is small.

[0059] In order to improve the accuracy of the 3D image reconstruction of the observation object S, the configurable wave vector {(k x,n ,k y,n )} n The larger the number N of the wave number vectors, the more preferable it is. In order to reduce the anisotropy of the resolution of the three-dimensional image of the object S to be observed, the wave number vector {(k x,n ,k y,n )} n From this viewpoint, it is preferable that the wave vector {(k x,n ,k y,n )} n When is distributed one-dimensionally (Figure 9(b)), the wave vector {(k x,n ,k y,n )} n is distributed two-dimensionally (FIGS. 8(a) to (c), FIG. 9(a)).

[0060] Next, acquisition of a two-dimensional interference image by the imaging unit 41 in the imaging step will be described. in When the light (x, y, t) is irradiated onto the observation object S, the first branched light u out (x, y, t) are expressed by the following formula (7). In this formula, o(x, y) represents an image of the observation object S at a position optically conjugate with respect to the imaging plane of the imaging unit 41. The imaging unit 41 receives the first split light u out The optical axis of the optical fiber is (x, y, t) and the second branched light incident as a plane wave along a direction inclined to the incident direction of the first branched light are received on an imaging plane, and a detection signal representing a two-dimensional interference image due to interference between the first branched light and the second branched light is repeatedly output.

[0061]

number

[0062] Next, the content of the processing by the processing unit 42 in the processing step will be described. The processing unit 42 inputs the detection signal repeatedly output from the imaging unit 41 and performs the following processing. Based on the detection signal at each time, a two-dimensional interference image at each time (i.e., time-series data of the two-dimensional interference image) is generated. Then, based on the two-dimensional interference image at each time t, the first branched light u that has reached the imaging plane of the imaging unit 41 at each time t is calculated by a fringe analysis method. out An image of the observation object S is generated by (x, y, t), that is, an image of the observation object S located at a position optically conjugate with respect to the imaging plane. The first split light u out The image of the observation object S by (x, y, t) can be expressed by the following equation (8) using the above equation (1).

[0063]

number

[0064] This equation (8) is expressed by the frequency f n (The wave number component k in the x direction of each of the N light components x,n The u out (x,y,t) is expressed as a discrete time interval {t i} i Then, we sample at frequency f n When the time-varying components are extracted, each light component (wave vector (k x,n ,k y,n ), amplitude U in (k x,n ,k y,n )), that is, complex amplitude images for each of the N irradiation directions, are obtained.

[0065]

number

[0066]

number

[0067] Also, based on these N irradiation directions, a phase differential image of the observation object S can be generated. Furthermore, based on this phase differential image, a three-dimensional refractive index distribution image of the observation object S can be generated. In generating these images, the method described in Non-Patent Document 2 or Non-Patent Document 3 can be used. The method described in Non-Patent Document 2 uses the Fourier Slice theorem and is applicable when light scattering in the observation object S can be ignored. The method described in Non-Patent Document 3 uses the Fourier Diffraction theorem and generates a three-dimensional refractive index distribution image of the observation object S taking into account the light scattering in the observation object S.

[0068] Next, the simulation results will be described with reference to Figs. 10 to 15. The numerical aperture of each of the lenses 32 and 33 was set to 1.0. The wavelength in a vacuum of the light output from the light source 11 was set to λ0, and the moving speed V of the modulation pattern moving in the x direction was set to 1000 λ0 / s. The imaging frame rate of the imaging unit 41 was set to 3000 fps. The number of two-dimensional interference images captured by the imaging unit 41 was set to 312.

[0069] Figure 10 shows the wave vector {(k x,n ,k y,n )} n FIG. 11 shows the distribution of the wave vector {(k x,n ,k y,n )} n 11 is a diagram showing a modulation pattern m(x, y) created based on the distribution of . In FIG. 11, the magnitude of the amplitude modulation is indicated by shading.

[0070] 12 to 15 are diagrams showing various images generated in the simulation. Figure 12 shows the u out The real part of the image of (x, y, t). (a) shows the real part of the image of u at t = 0 ms. out (b) shows the real part of the image of (x,y,t). out (c) shows the real part of the image of (x,y,t). out Denote the real part of the image of (x,y,t).

[0071] FIG. 13 is a diagram showing complex amplitude images for each irradiation direction generated by simulation. (a) shows the incident angle (θ x ,θ y )=(-35.2°,-25.6°) light component (f n (b) shows the real part of the complex amplitude image when the observation object is irradiated with the incident angle (θ x ,θ y )=(-11.6°,-46.2°) light component (f n (c) shows the real part of the complex amplitude image when the observation object is irradiated with the incident angle (θ x ,θ y )=(0.8°,-16.7°) light component (f n The figure shows the real part of the complex amplitude image when the observation object is illuminated with a frequency of 19.2 Hz.

[0072] 14 is a diagram showing phase differential images of an object to be observed generated by simulation. The leftmost image shows a phase differential image in an xz cross section, and the others show phase differential images in an xy cross section at three positions in the z direction.

[0073] 15 is a diagram showing refractive index distribution images of an observation object generated by simulation. The upper part shows a refractive index distribution image of an exact solution, and the lower part shows a refractive index distribution image generated by simulation. The leftmost part shows a maximum intensity projection image in the y direction, and the others show refractive index distribution images in the xy cross section at three positions in the z direction.

[0074] According to this embodiment, since the first branched light is irradiated to the observation object along each of the multiple irradiation directions, it is not necessary to mechanically move the mirror that reflects the first branched light and irradiates the observation object, so that the instability and vibration of the orientation of the reflecting surface of the mirror due to external disturbances are suppressed, and the observation object can be stably observed. When a DMD is used as the modulation section, the mirror of each pixel is mechanically moved, but the orientation of the mirror of each pixel is not PID controlled, but only one of two states is selected, so that stability is high.

[0075] In addition, in the invention disclosed in Non-Patent Document 1, the scanning of the orientation of the reflecting surface of the mirror is performed by mechanically moving the mirror, so a response time of about 10 ms is required. For this reason, the frame rate of the camera is limited to about 100 fps. Since the light irradiation direction of the first branched light to the observation object needs about 300, the measurement time takes several seconds. In contrast, according to the present embodiment, by using a modulation unit that can move the modulation pattern at high speed, the observation object can be observed in a short time, and dynamic changes in the observation object can also be observed. For example, when a DMD is used as the modulation unit, it is preferable because high-speed modulation of several tens of kHz is possible.

[0076] In the processing by the processing unit 42 described above, the frequency f n In order to extract the time-varying components using equation (9), it was assumed that the imaging frame rate of the imaging unit 41 was sufficiently fast. The Nyquist theorem states that in order to accurately reconstruct the waveform of a signal, it is necessary to sample the signal at a speed at least twice as fast as the maximum frequency component of the signal. In other words, the numerical aperture of the lens 32, which is the objective lens on the irradiation side, is set to NA in Let the wavelength of light be λ0 and the wave number of light be k0. Then, k x,n The maximum value of k x,max is expressed by the following equation (11), and the frequency f n The maximum value of f maxAccording to the Nyquist theorem, the image capturing frame rate f s is expressed as 2f max It is necessary that the above is true.

[0077]

number

[0078]

number

[0079]

number

[0080] However, the above formula (8) does not need to be reproduced accurately without distortion, and the imaging frame rate f s is 2F max That is, each light component (wave vector (k x,n ,k y,n ), amplitude U in (k x,n ,k y,n )) is irradiated to obtain a complex amplitude image (the right side of the above formula (10)). s 2F max If it is less than this (undersampling condition), aliasing is exploited as described below.

[0081] Aliasing is the process of reducing the Nyquist rate (2f max ) sampling rate f s When a continuous-time signal is sampled, the discrete-time Fourier transform of the discrete-time signal obtained by sampling is [-f s / 2,f s / 2]. n A signal with sampling rate f sIf we sample at , the frequency f observed due to aliasing n,alias is expressed by the following equation (14). mod f s f s is a modulo operation, and f s Gives the remainder when divided by f. For example, n = 80Hz, f s = 100Hz, then f n,alias =-20Hz.

[0082]

number

[0083] Due to aliasing, the observed frequency will differ from the actual frequency. However, if the observed frequency and the actual frequency correspond one-to-one, each light component (wave vector (k x,n ,k y,n ), amplitude U in (k x,n ,k y,n )) can be used to obtain a complex amplitude image (the right side of the above equation (12)) when the light is irradiated. In order to ensure that the observed frequency and the actual frequency correspond one-to-one, the wave number component k x,n The frame rate f of the imaging unit 41 is s All we need to do is make sure that the remainders when divided by are different.

[0084] When aliasing is used, the frequency f is calculated by the following equation (15) instead of the above equation (10). n The frequency f n,alias When the time-varying components are extracted, each light component (wave vector (k x,n ,k y,n ), amplitude U in (k x,n ,k y,n )), that is, complex amplitude images for each of the N irradiation directions, are obtained.

[0085]

number

[0086] Next, the results of a simulation using aliasing will be described with reference to Figs. 16 to 23. The numerical aperture of each of the lenses 32 and 33 was set to 1.0. The wavelength in a vacuum of the light output from the light source 11 was set to λ0, and the moving speed V of the modulation pattern moving in the x direction was set to 1000 λ0 / s. The imaging frame rate of the imaging unit 41 was set to 1000 fps. The number of two-dimensional interference images captured by the imaging unit 41 was set to 117.

[0087] Figure 16 shows the wave vector {(k x,n ,k y,n )} n FIG. 17 shows the distribution of the wave vector {(k x,n ,k y,n )} n 17 is a diagram showing a modulation pattern m(x, y) created based on the distribution of . In FIG. 17, the magnitude of the amplitude modulation is shown by shading.

[0088] Figure 18 shows the wave number component k in the x direction of each light component. x,n and frequency f n 19 shows the correspondence between the frequency f n As shown in these figures, the wave number component k in the x direction is x,n The frame rate f of the imaging unit 41 is s When divided by, the remainders are different.

[0089] 20 to 23 are diagrams showing various images generated in a simulation in which aliasing is utilized. Figure 20 shows the u out The real part of the image of (x, y, t). (a) shows the real part of the image of u at t = 0 ms. out(b) shows the real part of the image of (x,y,t). out (c) shows the real part of the image of (x,y,t). out Denote the real part of the image of (x,y,t).

[0090] FIG. 21 shows complex amplitude images for each irradiation direction generated by simulation. (a) shows the incident angle (θ x ,θ y )=(-1.1°,10.8°) light component (f n (b) shows the real part of the complex amplitude image when the observation object is irradiated with the incident angle (θ x ,θ y )=(43.0°,-10.8°) light component (f n (c) shows the real part of the complex amplitude image when the observation object is illuminated with the incident angle (θ x ,θ y )=(24.9°,-10.8°) light component (f n The figure shows the real part of the complex amplitude image when the observation object is illuminated with a frequency of 100 Hz (=230.7 Hz).

[0091] 22 is a diagram showing phase differential images of an object to be observed generated by simulation. The leftmost image shows a phase differential image in an xz cross section, and the others show phase differential images in an xy cross section at three positions in the z direction.

[0092] 23 is a diagram showing refractive index distribution images of an observation object generated by simulation. The upper part shows a refractive index distribution image of an exact solution, and the lower part shows a refractive index distribution image generated by simulation. The leftmost part shows a maximum intensity projection image in the y direction, and the others show refractive index distribution images in the xy cross section at three positions in the z direction.

[0093] As shown in the simulation results, even when aliasing is used, the object can be observed. In addition, by using aliasing, the range of selection of the moving speed V of the modulation pattern and the imaging frame rate of the imaging unit 41 becomes wider. [Explanation of symbols]

[0094] Reference Signs: 1A to 1D: observation device, 11: light source, 12: beam splitter (branching section), 13, 14: mirror, 15: beam splitter (combining section), 16: beam splitter, 21, 22: modulation section, 23: spatial light modulator, 24: polarizer, 25: modulation section, 31 to 34: lenses, 41: imaging section, 42: processing section.

Claims

1. An apparatus for observing an observation object, comprising: A light source that outputs light; a branching unit that branches the light into a first branched light and a second branched light; a modulation unit that spatially amplitude-modulates the first branched light based on a modulation pattern that moves in a predetermined direction intersecting a propagation direction of the first branched light, to generate a first branched light including a plurality of light components having wave number components different from each other in the predetermined direction; an illumination unit that illuminates the object of observation located at a position optically conjugate with the first split light that has been amplitude-modulated by the modulation unit; a multiplexing unit that multiplexes the first split light and the second split light that have passed through the observation object and outputs multiplexed light; an imaging unit having an imaging surface arranged at a position optically conjugate with the object to be observed, receiving the combined light at the imaging surface, and repeatedly outputting a detection signal representing a two-dimensional interference image; a processing unit that generates a complex amplitude image when the object to be observed is irradiated with each of the plurality of light components based on the detection signals repeatedly output from the imaging unit; Equipped with the processing unit generates an image of the object of observation at each time by the first branched light that has reached the imaging surface based on time series data of the two-dimensional interference image generated from the detection signal repeatedly output from the imaging unit, extracts a component that changes over time at a frequency corresponding to a wave number component in the predetermined direction of each of the plurality of light components, and generates a complex amplitude image when the object of observation is irradiated with each of the plurality of light components. Observation equipment.

2. The modulation unit generates a first branched light including a plurality of optical components whose wave numbers are two-dimensionally distributed in a wave number space. The observation device according to claim 1.

3. the modulation unit generates a first branched light including a plurality of light components having different surpluses when a wave number component in the predetermined direction is divided by a frame rate of the imaging unit, The observation device according to claim 1.

4. the modulation unit spatially amplitude-modulates the first split light based on a modulation pattern in which values ​​at two positions that are symmetric with respect to an origin in a Fourier transform pattern of the modulation pattern have a complex conjugate relationship. The observation device according to claim 1.

5. The modulation unit includes a spatial light modulator capable of variably setting the modulation pattern. The observation device according to claim 1.

6. The processing unit generates a phase differential image of the observation object based on a complex amplitude image obtained when the observation object is irradiated with each of the plurality of light components. The observation device according to claim 1.

7. The processing unit generates a refractive index distribution image of the observation object based on the phase differential image. The observation device according to claim 6.

8. A method for observing an object to be observed, comprising the steps of: a branching step of branching light output from a light source into a first branched light and a second branched light; a modulation step of spatially amplitude-modulating the first branched light based on a modulation pattern moving in a predetermined direction intersecting a propagation direction of the first branched light, to obtain a first branched light including a plurality of light components having different wave number components in the predetermined direction; an irradiation step of irradiating the object of observation located at a position optically conjugate with the first split light that has been amplitude-modulated by the modulation unit; a combining step of combining the first split light and the second split light that have passed through the observation object and outputting a combined light; an imaging step of receiving the combined light on an imaging surface of an imaging unit arranged at a position optically conjugate with the object to be observed, and repeatedly outputting a detection signal representing a two-dimensional interference image; a processing step of generating a complex amplitude image when the object to be observed is irradiated with each of the plurality of light components based on the detection signals repeatedly output from the imaging unit; Equipped with In the processing step, an image of the object of observation at each time is generated by the first branched light that has reached the imaging surface based on time series data of the two-dimensional interference image generated from the detection signal repeatedly output from the imaging unit, and a component that changes over time at a frequency corresponding to a wave number component in the predetermined direction of each of the plurality of light components is extracted, and a complex amplitude image is generated when each of the plurality of light components is irradiated onto the object of observation. Observation method.

9. In the modulating step, the first branched light includes a plurality of optical components whose wave numbers are two-dimensionally distributed in a wave number space. The observation method according to claim 8.

10. In the modulating step, a remainder obtained by dividing the wave number component in the predetermined direction by a frame rate of the imaging unit is set to a first branched light including a plurality of light components different from each other. The observation method according to claim 8.

11. In the modulating step, the first split light is spatially amplitude-modulated based on a modulation pattern in which values ​​at two positions that are symmetric with respect to an origin in a Fourier transform pattern of the modulation pattern have a complex conjugate relationship. The observation method according to claim 8.

12. In the modulating step, the first split light is spatially amplitude-modulated using the modulating unit including a spatial light modulator capable of variably setting the modulation pattern. The observation method according to claim 8.

13. In the processing step, a phase differential image of the observation object is generated based on a complex amplitude image obtained when the observation object is irradiated with each of the plurality of light components. The observation method according to claim 8.

14. generating a refractive index distribution image of the object to be observed based on the phase differential image in the processing step; The observation method according to claim 13.

Images