Observation device and observation method

Abstract

To provide an observation device that allows observation of an object to be observed while reducing the influence of multiple scattered light even when a moving object to be observed is a multiple scatterer.SOLUTION: An observation device 1A comprises a light source 10, a beam splitter 21, a mirror 22, a mirror 23, a cylindrical lens 24, a lens 25, a cylindrical lens 26, a beam splitter 27, a lens 28, a frequency shifter 30, an imaging unit 40, and an analysis unit 50. The analysis unit 50 creates complex amplitude images in a plurality of light irradiation directions of object light based on time-series data of an interference intensity image output from the imaging unit 40, and creates a three-dimensional complex differential interference image of an object to be observed S based on the complex amplitude images in the plurality of light irradiation directions. The analysis unit 50 determines the three-dimensional refractive index distribution of the object to be observed S based on the three-dimensional complex differential interference image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an observation device and an observation method. [Background technology]

[0002] The observation device described in Patent Document 1 splits light output from a light source into object light and reference light, combines the object light that has passed through a moving observation object with the reference light whose optical frequency has been shifted by the heterodyne frequency, and heterodyne interferes the object light and the reference light. This observation device can obtain time series data of a complex amplitude image of the object light on the imaging plane based on time series data of an intensity image of the interference light that has reached the imaging plane of the camera. This observation device can image a moving observation object non-stained and non-invasively, and can be used, for example, to observe moving cells with a flow cytometer. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2013 / 065796 Summary of the Invention [Problem to be solved by the invention]

[0004] Although the observation device described in Patent Document 1 can be applied to the observation of a single cell as an observation object, it is difficult to apply it to the observation of multiple scattering bodies such as three-dimensional cell tissues called spheroids and organoids. This is because when there is a lot of multiple scattered light generated in the observation object, the influence of the multiple scattered light appears significantly in the acquired image.

[0005] Light scattering is a phenomenon in which the direction of light travel is changed by interacting with an object. In particular, when the spatial non-uniformity of the refractive index of the object increases, the light interacts with the object multiple times while passing through the object. Light that has interacted with the object multiple times in this way is called multiple scattered light. In contrast, light that has interacted with the object only once is called single scattered light. Multiple scattered light is known to be an obstacle to measurement, causing increased speckle and a deterioration in the single-scattering to multi-scattering ratio (SMR).

[0006] Speckle occurs when light is temporally and spatially coherent, but the interference of multiple scattered light causes large spatial changes in intensity or phase. To suppress speckle, a light source that outputs light that is temporally or spatially incoherent can be used. For example, normal bright-field microscopes such as phase-contrast microscopes use spatially and temporally incoherent light sources such as halogen lamps and light-emitting diodes to obtain images without speckle.

[0007] The deterioration of SMR occurs when multiple scattered light becomes dominant over single scattered light, and the single scattered light is buried in the multiple scattered light. As the observed object becomes larger and the observation depth becomes deeper, the single scattered light component exponentially decreases, while in contrast, the multiple scattered light component increases. Single scattered light is easy to use for measuring the structure of an object because its scattering direction has a direct correspondence with the structure of the object. On the other hand, multiple scattered light has a complex relationship with the structure of the object, making it difficult to extract information about the structure of the object. Therefore, it is known that imaging techniques using single scattered light fail to measure when single scattered light is buried in multiple scattered light (i.e., when SMR deteriorates).

[0008] Suppression of SMR deterioration is possible by a technique called gating, which selectively detects single-scattered light among single-scattered and multiple-scattered light. Gating suppresses multiple-scattered light, so it is possible to suppress speckle while suppressing SMR deterioration. Gating is achieved using degrees of freedom such as space, time, and polarization. Confocal microscopy is an example of spatial gating. Optical Coherence Tomography (OCT) is an example of temporal and spatial gating.

[0009] The observation device described in Patent Document 1 does not remove the effect of multiple scattered light, so when there is a lot of multiple scattered light generated in the observation object, speckles increase in the acquired image and the SMR deteriorates. Therefore, although the observation device described in Patent Document 1 can be applied to the observation of single cells that generate little multiple scattered light, it is difficult to apply it to the observation of multiple scattering bodies such as three-dimensional cell tissues that generate a lot of multiple scattered light.

[0010] The present invention has been made to solve the above-mentioned problems, and has an object to provide an observation device and an observation method that can observe an object by reducing the influence of multiple scattered light, even when the moving object is a multiple scatterer. [Means for solving the problem]

[0011] The observation device of the first aspect of the present invention includes: (1) an interference optical system that splits light output from a light source into object light and reference light, irradiates the object light onto an observation object moving along a first direction, and combines and outputs the object light and reference light that have passed through the observation object; (2) a frequency shifter that is provided on the optical path of the object light or the reference light between the splitting and the combination of the interference optical system, and changes the optical frequencies of the object light and the reference light by only the heterodyne frequency; (3) an imaging unit that has an imaging surface that receives the object light and reference light output from the interference optical system, captures an interference intensity image due to interference between the object light and the reference light on the imaging surface, and outputs time series data of the interference intensity image; and (4) an analysis unit that performs analysis based on the time series data of the interference intensity image. The interference optical system includes an illumination optical system that, when illuminating the object light onto the observation object, focuses and illuminates the object light onto a linear region extending in a second direction perpendicular to the first direction on the observation object, and an imaging optical system that, when guiding the object light that has passed through the observation object to an imaging plane, places the observation object and the imaging plane in a positional relationship of a Fourier transform with respect to each other in the first direction and places the observation object and the imaging plane in a positional relationship of a conjugate with each other in the second direction, and causes the object light and the reference light to be incident onto the imaging plane coaxially with each other. The analysis unit generates complex differential interference images for each of a plurality of light illumination directions and a plurality of positions when the object light is focused and illuminated onto the observation object by the illumination optical system based on the time series data of the interference intensity image, and generates a three-dimensional phase image of the observation object based on these complex differential interference images.

[0012] The observation device of the second aspect of the present invention comprises: (1) an interference optical system that splits light output from a light source into object light and reference light, irradiates the object light onto an observation object moving along a first direction, and combines and outputs the object light and reference light that have passed through the observation object; (2) an imaging unit that has an imaging surface that receives the object light and reference light output from the interference optical system, captures an interference intensity image due to interference between the object light and the reference light on the imaging surface, and outputs time series data of the interference intensity image; and (3) an analysis unit that performs analysis based on the time series data of the interference intensity image. The interference optical system includes an illumination optical system that, when illuminating the object light onto the observation object, focuses and illuminates the object light onto a linear region extending in a second direction perpendicular to the first direction on the observation object, and an imaging optical system that, when guiding the object light that has passed through the observation object to an imaging plane, places the observation object and the imaging plane in a positional relationship of Fourier transform with respect to each other in the first direction and places the observation object and the imaging plane in a positional relationship of conjugate with each other in the second direction, and causes the object light and the reference light to be incident on the imaging plane from different directions. The analysis unit generates complex differential interference images for each of a plurality of light illumination directions and a plurality of positions when the object light is focused and illuminated onto the observation object by the illumination optical system based on the time series data of the interference intensity image, and generates a three-dimensional phase image of the observation object based on these complex differential interference images.

[0013] In one aspect of the first or second aspect of the present invention, the analysis unit includes: (a) an interference intensity image acquisition unit that acquires time series data of the interference intensity image from the imaging unit; (b) a first complex amplitude image generation unit that generates a complex amplitude image for each of the multiple light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generation unit that generates a complex amplitude image for each of the multiple positions based on the complex amplitude image generated by the first complex amplitude image generation unit for each of the multiple light irradiation directions; (d) a two-dimensional phase image generation unit that generates a complex differential interference image for each of the multiple positions based on the complex amplitude image for each of the multiple light irradiation directions based on the complex amplitude image for each of the multiple light irradiation directions and generates a two-dimensional phase image based on the complex differential interference image for each of the multiple light irradiation directions; and (e) a three-dimensional phase image generation unit that generates a three-dimensional phase image based on the two-dimensional phase image for each of the multiple positions.

[0014] In another aspect of the first or second aspect of the present invention, the analysis unit includes: (a) an interference intensity image acquisition unit that acquires time series data of the interference intensity image from the imaging unit; (b) a first complex amplitude image generation unit that generates a complex amplitude image for each of a plurality of light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generation unit that generates a complex amplitude image for each of a plurality of positions based on the complex amplitude image generated by the first complex amplitude image generation unit for each of the plurality of light irradiation directions; (d) a two-dimensional phase image generation unit that divides the complex amplitude images for each of the plurality of positions into a plurality of batches, corrects the phases of the complex amplitude images included in each of the plurality of batches based on the light irradiation direction, and then generates a complex amplitude sum image representing the sum of these corrected complex amplitude images, generates a complex differential interference image for each of the plurality of batches based on the complex amplitude sum image for each of the plurality of batches, and generates a two-dimensional phase image based on the complex differential interference image for each of the plurality of batches; and (e) and a three-dimensional phase image generating unit that generates a three-dimensional phase image based on the two-dimensional phase images at each of the multiple positions.

[0015] In yet another aspect of the first or second aspect of the present invention, the analysis unit includes: (a) an interference intensity image acquisition unit that acquires time series data of the interference intensity image from the imaging unit; (b) a first complex amplitude image generation unit that generates a complex amplitude image for each of the multiple light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generation unit that generates a complex amplitude image for each of the multiple positions based on the complex amplitude image for each of the multiple light irradiation directions; (d) a phase conjugate calculation unit that performs a phase conjugate calculation on the complex amplitude image for each of the multiple light irradiation directions before, during, or after processing by the second complex amplitude image generation unit, to generate a complex amplitude image for each of the multiple light irradiation directions when the relationship of light irradiation and imaging of the observation object is reversed; and (e) The imaging system includes (f) a two-dimensional phase image generating unit that generates a complex differential interference image for each of the multiple light irradiation directions based on the complex amplitude image for each of the multiple light irradiation directions generated by the second complex amplitude image generating unit or the phase conjugate calculation unit for each of the multiple positions, and generates a two-dimensional phase image based on the complex differential interference image for each of the multiple light irradiation directions, and (f) a three-dimensional phase image generating unit that generates a three-dimensional phase image based on the two-dimensional phase image for each of the multiple positions. When a phase image generated based on a complex amplitude image before the calculation by the phase conjugate calculation unit is defined as a first phase image, and a phase image generated based on a complex amplitude image obtained by performing calculation by the phase conjugate calculation unit is defined as a second phase image, the two-dimensional phase image generating unit generates a two-dimensional phase image mainly based on the first phase image for positions relatively close to the imaging unit among the multiple positions, and generates a two-dimensional phase image mainly based on the second phase image for positions relatively far from the imaging unit.

[0016] In an aspect of the first or second aspect of the present invention, the analysis section further includes a refractive index distribution calculation section that determines a three-dimensional refractive index distribution of the observation object based on the three-dimensional phase image.

[0017] In yet another aspect of the first or second aspect of the present invention, the analysis unit includes: (a) an interference intensity image acquisition unit that acquires time series data of the interference intensity image from the imaging unit; (b) a first complex amplitude image generation unit that generates a complex amplitude image for each of the multiple light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generation unit that generates a complex amplitude image for each of the multiple positions between the first position and the second position based on the complex amplitude image of the first position in relation to the distance from the imaging unit along the light propagation path for each of the multiple light irradiation directions; (d) a two-dimensional phase image generation unit that generates a complex differential interference image for each of the multiple light irradiation directions based on the complex amplitude image for each of the multiple light irradiation directions for each of the multiple positions, and generates a two-dimensional phase image based on the complex differential interference image for each of the multiple light irradiation directions; (e) a three-dimensional phase image generation unit that generates a three-dimensional phase image for the range from the first position to the second position based on the two-dimensional phase image for each of the multiple positions; and (f) and (g) a third complex amplitude image generating unit that generates a complex amplitude image of the second position based on the complex amplitude image of the first position and the three-dimensional refractive index distribution for each of a plurality of light irradiation directions. The analysis unit sequentially performs processing of the second complex amplitude image generating unit, the two-dimensional phase image generating unit, the three-dimensional phase image generating unit, the refractive index distribution calculating unit, and the third complex amplitude image generating unit based on the complex amplitude image generated by the first complex amplitude image generating unit.

[0018] In one aspect of the present invention, the two-dimensional phase image generating section preferably generates the two-dimensional phase image based on the sum of complex differential interference images in each of a plurality of light irradiation directions.

[0019] The observation method of the first aspect of the present invention includes: (1) an interference step of causing interference between the object light and the reference light by an interference optical system that splits light output from a light source into object light and reference light, irradiates the object light onto an object to be observed moving along a first direction, and combines and outputs the object light and reference light that have passed through the object to be observed; (2) a modulation step of causing the optical frequencies of the object light and the reference light to differ from each other by only the heterodyne frequency by a frequency shifter provided on the optical path of the object light or the reference light between the splitting and combining of the interference optical system; (3) an imaging step of capturing an interference intensity image due to interference between the object light and the reference light on the imaging plane by an imaging unit having an imaging plane that receives the object light and reference light output from the interference optical system, and outputting time series data of the interference intensity image; and (4) an analysis step of performing analysis based on the time series data of the interference intensity image. The interference optical system includes an irradiation optical system that, when irradiating the object light onto the observation object, focuses and irradiates the object light onto a linear region extending in a second direction perpendicular to the first direction on the observation object, and an imaging optical system that, when guiding the object light that has passed through the observation object to an imaging plane, places the observation object and the imaging plane in a positional relationship of a Fourier transform with respect to each other in the first direction and places the observation object and the imaging plane in a positional relationship of a conjugate with each other in the second direction, and causes the object light and the reference light to be incident on the imaging plane coaxially with each other. In the analysis step, complex differential interference images are generated for each of a plurality of light irradiation directions and a plurality of positions when the object light is focused and irradiated onto the observation object by the irradiation optical system based on the time series data of the interference intensity image, and a three-dimensional phase image of the observation object is generated based on these complex differential interference images.

[0020] The observation method of the second aspect of the present invention includes: (1) an interference step of causing interference between the object light and the reference light by an interference optical system that splits light output from a light source into object light and reference light, irradiates the object light onto an object to be observed moving along a first direction, and combines and outputs the object light and reference light that have passed through the object to be observed; (2) an imaging step of capturing an interference intensity image resulting from interference between the object light and the reference light on the imaging plane by an imaging unit having an imaging plane that receives the object light and reference light output from the interference optical system, and outputting time series data of the interference intensity image; and (3) an analysis step of performing analysis based on the time series data of the interference intensity image. The interference optical system includes an illumination optical system that, when illuminating the object light onto the observation object, focuses and illuminates the object light onto a linear region extending in a second direction perpendicular to the first direction on the observation object, and an imaging optical system that, when guiding the object light that has passed through the observation object to an imaging plane, makes the observation object and the imaging plane have a positional relationship of Fourier transform with respect to each other in the first direction and a positional relationship of conjugate with respect to the observation object and the imaging plane with respect to each other in the second direction, and makes the object light and the reference light enter the imaging plane from different directions. The analysis step generates complex differential interference images for each of a plurality of light illumination directions and a plurality of positions when the object light is focused and illuminated onto the observation object by the illumination optical system based on the time series data of the interference intensity image, and generates a three-dimensional phase image of the observation object based on these complex differential interference images.

[0021] In one aspect of the first or second aspect of the present invention, the analysis step includes: (a) an interference intensity image acquisition step of acquiring time series data of the interference intensity image from an imaging unit; (b) a first complex amplitude image generation step of generating a complex amplitude image for each of a plurality of light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generation step of generating a complex amplitude image for each of a plurality of positions based on the complex amplitude image generated by the first complex amplitude image generation step for each of the plurality of light irradiation directions; (d) a two-dimensional phase image generation step of generating a complex differential interference image for each of the plurality of positions based on the complex amplitude image for each of the plurality of light irradiation directions and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; and (e) a three-dimensional phase image generation step of generating a three-dimensional phase image based on the two-dimensional phase image for each of the plurality of positions.

[0022] In another aspect of the first or second aspect of the present invention, the analyzing step includes: (a) an interference intensity image acquiring step of acquiring time series data of an interference intensity image from an imaging unit; (b) a first complex amplitude image generating step of generating a complex amplitude image for each of a plurality of light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generating step of generating a complex amplitude image for each of a plurality of positions based on the complex amplitude image generated by the first complex amplitude image generating step for each of the plurality of light irradiation directions; (d) a two-dimensional phase image generating step of dividing the complex amplitude images for each of the plurality of light irradiation directions into a plurality of batches for each of the plurality of positions, correcting the phases of the complex amplitude images included in each of the plurality of batches based on the light irradiation direction, and then generating a complex amplitude sum image representing the sum of these corrected complex amplitude images, generating a complex differential interference image for each of the plurality of batches based on the complex amplitude sum image for each of the plurality of batches, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of batches; and (e) and a three-dimensional phase image generating step of generating a three-dimensional phase image based on the two-dimensional phase images at each of the multiple positions.

[0023] In still another aspect of the first or second aspect of the present invention, the analyzing step includes: (a) an interference intensity image acquiring step of acquiring time series data of an interference intensity image from an imaging unit; (b) a first complex amplitude image generating step of generating a complex amplitude image for each of a plurality of light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generating step of generating a complex amplitude image for each of a plurality of positions based on the complex amplitude image for each of the plurality of light irradiation directions; (d) a phase conjugate calculation step of performing a phase conjugate calculation on the complex amplitude image for each of the plurality of light irradiation directions before, during or after the processing in the second complex amplitude image generating step to generate a complex amplitude image for each of the plurality of light irradiation directions when the relationship of light irradiation and imaging of the observation object is reversed; and (e) The method includes (f) a two-dimensional phase image generating step of generating a complex differential interference image for each of the plurality of positions based on the complex amplitude image for each of the plurality of light irradiation directions generated by the second complex amplitude image generating step or the phase conjugate calculation step, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions, and (f) a three-dimensional phase image generating step of generating a three-dimensional phase image based on the two-dimensional phase image for each of the plurality of positions. When a phase image generated based on a complex amplitude image before the calculation by the phase conjugate calculation step is defined as a first phase image, and a phase image generated based on a complex amplitude image obtained by performing the calculation by the phase conjugate calculation step is defined as a second phase image, the two-dimensional phase image generating step generates a two-dimensional phase image mainly based on the first phase image for a position relatively close to the imaging unit among the plurality of positions, and generates a two-dimensional phase image mainly based on the second phase image for a position relatively far from the imaging unit.

[0024] In one aspect of the first or second aspect of the present invention, the analyzing step further includes a refractive index distribution calculating step of determining a three-dimensional refractive index distribution of the observation object based on the three-dimensional phase image.

[0025] In still another aspect of the first or second aspect of the present invention, the analyzing step includes: (a) an interference intensity image acquiring step of acquiring time series data of an interference intensity image from an imaging unit; (b) a first complex amplitude image generating step of generating a complex amplitude image for each of a plurality of light irradiation directions based on the time series data of the interference intensity image; (c) a second complex amplitude image generating step of generating a complex amplitude image for each of a plurality of positions between a first position and a second position based on a complex amplitude image of a first position in relation to a distance from the imaging unit along a light propagation path for each of the plurality of light irradiation directions; (d) a two-dimensional phase image generating step of generating a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude image for each of the plurality of light irradiation directions for each of the plurality of positions, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; (e) a three-dimensional phase image generating step of generating a three-dimensional phase image for the range between the first position and the second position based on the two-dimensional phase image for each of the plurality of positions; and (f) The method includes (g) a refractive index distribution calculation step of calculating a three-dimensional refractive index distribution of the object to be observed between the first position and the second position based on the three-dimensional phase image, and (g) a third complex amplitude image generation step of generating a complex amplitude image of the second position based on the complex amplitude image of the first position and the three-dimensional refractive index distribution for each of a plurality of light irradiation directions. The analysis step sequentially performs the processes of the second complex amplitude image generation step, the two-dimensional phase image generation step, the three-dimensional phase image generation step, the refractive index distribution calculation step, and the third complex amplitude image generation step based on the complex amplitude image generated by the first complex amplitude image generation step.

[0026] In one aspect of the present invention, in the two-dimensional phase image generating step, the two-dimensional phase image is preferably generated based on a sum of complex differential interference images in each of a plurality of light irradiation directions.

[0027] The program of the present invention is for causing a computer to execute each step of the observation method of the present invention.The recording medium of the present invention is a computer-readable medium having the program of the present invention recorded thereon. Effect of the Invention

[0028] According to the present invention, even when a moving observation object is a multiple scattering medium, the influence of multiple scattered light can be reduced and the observation object can be observed. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a diagram showing the configuration of an observation apparatus 1A. [Diagram 2] FIG. 2 is a diagram showing the configuration of the observation device 1B. [Diagram 3] 3A and 3B are diagrams illustrating the imaging optical system including the lens 25, the cylindrical lens 26, and the lens 28 in each of the configurations of the observation device 1A (FIG. 1) and the observation device B (FIG. 2). Fig. 3(a) is a diagram viewed in the y direction, and Fig. 3(b) is a diagram viewed in the x direction. [Figure 4] Fig. 4 is a diagram for explaining the processing contents of the first complex amplitude image generating unit 52 of the analysis unit 50 of the observation device 1A (Fig. 1). Fig. 4(a) shows time series data I(kx, y, t) of an interference intensity image output from the imaging unit 40. Fig. 4(b) shows data I(kx0, y0, t) in which time t is a variable at a certain light irradiation direction kx0 and a certain second direction position y0. [Diagram 5] Fig. 5 is a diagram for explaining the processing contents of the first complex amplitude image generating unit 52 of the analysis unit 50 of the observation device 1A (Fig. 1). Fig. 5(a) shows the result of the Fourier transform of I(kx0, y0, t). Fig. 5(b) shows the result of cutting out and translating a part of the result of the Fourier transform of Fig. 5(a). Fig. 5(c) shows the result of the Fourier transform of the result of Fig. 5(b). [Figure 6] Fig. 6 is a diagram for explaining the processing contents of the first complex amplitude image generating unit 52 of the analysis unit 50 of the observation device 1B (Fig. 2). Fig. 6(a) shows time series data I(kx, y, t) of an interference intensity image output from the imaging unit 40. Fig. 6(b) shows data I(kx0, y, t0) in which a second direction position y at a certain light irradiation direction kx0 and a certain time t0 is a variable. [Figure 7]Fig. 7 is a diagram for explaining the processing contents of the first complex amplitude image generating unit 52 of the analysis unit 50 of the observation device 1B (Fig. 2). Fig. 7(a) shows the result of the Fourier transform of I(kx0, y, t0). Fig. 7(b) shows the result of cutting out and translating a part of the result of the Fourier transform of Fig. 7(a). Fig. 7(c) shows the result of the Fourier transform of the result of Fig. 7(b). [Figure 8] 8(a) and (b) are diagrams for explaining object light output from an observation target S in the configurations of the observation device 1A (FIG. 1) and the observation device B (FIG. 2), respectively. [Figure 9] 9(a) and (b) are diagrams for explaining object light output from an observation target S in the configurations of the observation device 1A (FIG. 1) and the observation device B (FIG. 2), respectively. [Figure 10] FIG. 10 is a flowchart illustrating the process of the first complex amplitude image generating unit 52 of the analyzing unit 50. As shown in FIG. [Figure 11] FIG. 11 is a flowchart of the observation method according to the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining the kernel function g. [Figure 13] FIG. 13 is a flow chart of a first aspect of the two-dimensional phase image generating step S4. [Figure 14] FIG. 14 is a flowchart of a second aspect of the two-dimensional phase image generating step S4. [Figure 15] FIG. 15 is a diagram for explaining the kernel function. [Figure 16] FIG. 16 is a flowchart of a third aspect of the two-dimensional phase image generating step S4. [Figure 17] FIG. 17 is a diagram showing the configuration of an observation apparatus 1C. [Figure 18] FIG. 18 is a diagram showing the configuration of an observation apparatus 1D. [Figure 19] FIG. 19 is a flowchart of the refractive index distribution measuring method B. [Figure 20]FIG. 20 is a diagram for explaining the order of the processes and images in the second complex amplitude image generating step S63 and the two-dimensional phase image generating step S65. [Figure 21] FIG. 21 is a diagram for explaining the order of the processes and images in the second complex amplitude image generating step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generating step S65. [Figure 22] FIG. 22 is a diagram for explaining the order of the processes and images in the second complex amplitude image generating step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generating step S65. [Figure 23] FIG. 23 is a diagram for explaining the order of the processes and images in the second complex amplitude image generating step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generating step S65. [Figure 24] FIG. 24 is a diagram for explaining the order of the processes and images in the three-dimensional phase image generating step S66 and the refractive index distribution calculating step S67. [Diagram 25] FIG. 25 is a diagram for explaining an overview of the phase conjugate calculation, showing input light and output light when an interference intensity image is captured by the imaging section. [Figure 26] FIG. 26 is a diagram for explaining an outline of the phase conjugate calculation, showing input light and output light when the relationship between light irradiation and imaging is reversed. [Figure 27] FIG. 27 is a diagram for explaining the image division, phase conjugate calculation, and image combination in the phase conjugate calculation step S64. [Figure 28] FIG. 28 is a diagram showing the configuration of an observation apparatus 1E. [Figure 29] FIG. 29 is a diagram showing the configuration of the observation apparatus 1F. [Diagram 30] FIG. 30 is a flowchart of the refractive index distribution measuring method C. [Diagram 31] FIG. 31 is a flowchart of the refractive index distribution measuring method C. [Diagram 32] FIG. 32 is a diagram for explaining the relationship between a region including an observation object and the first to Jth blocks. [Diagram 33] FIG. 33 is a diagram for explaining the procedure of processing in the first to Jth blocks. [Diagram 34] FIG. 34 is a diagram for explaining the processing contents of BPM. [Diagram 35] FIG. 35 is a flowchart of the third complex amplitude image generating step S77. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated descriptions are omitted. The present invention is not limited to these examples, but is indicated by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0031] (First embodiment) As the configuration of the device of the first embodiment, an observation device 1A and an observation device 1B will be described. Fig. 1 is a diagram showing the configuration of the observation device 1A. This observation device 1A includes a light source 10, a beam splitter 21, a mirror 22, a mirror 23, a cylindrical lens 24, a lens 25, a cylindrical lens 26, a beam splitter 27, a lens 28, a frequency shifter 30, an imaging unit 40, and an analysis unit 50.

[0032] The observation device 1A observes an observation object S moving between the cylindrical lens 24 and the lens 25 along a first direction. For example, the observation object S is a cell flowing along a flow path in a flow cytometer. For convenience of explanation, an xyz Cartesian coordinate system is shown in this figure. The moving direction of the observation object S (the direction of the arrow in the figure) is defined as the x direction (first direction). The direction of the optical axis of the cylindrical lens 24 and the lens 25 is defined as the z direction. The direction perpendicular to both the x direction and the z direction is defined as the y direction (second direction).

[0033] The light source 10 outputs spatially and temporally coherent light, and is preferably a laser light source. The beam splitter 21 is optically connected to the light source 10 and splits the light output from the light source 10 into an object light and a reference light. The beam splitter 21 outputs the object light to a mirror 23 and outputs the reference light to a frequency shifter 30.

[0034] The frequency shifter 30 is optically connected to the beam splitter 21 and receives the reference light arriving from the beam splitter 21. The frequency shifter 30 shifts the optical frequency of the reference light by the heterodyne frequency and outputs the shifted reference light. The mirror 22 is optically connected to the frequency shifter 30 and reflects the reference light arriving from the frequency shifter 30 to the beam splitter 27.

[0035] The mirror 23 is optically connected to the beam splitter 21 and reflects the object light arriving from the beam splitter 21 to the cylindrical lens 24. The cylindrical lens 24 is optically connected to the mirror 23 and is an irradiation optical system that irradiates the object light arriving from the mirror 23 onto the observation object S. The cylindrical lens 24 focuses and irradiates the object light onto a linear region extending in the y direction (second direction) on the observation object S. The x-component k of the wave vector of the light irradiated onto the observation object S x The value of is in a certain range, and the y component k y The value of is a certain constant.

[0036] The object light that is irradiated onto the observation object S by the cylindrical lens 24 and passes through the observation object S passes through the lens 25, the cylindrical lens 26, and the lens 28 that constitute the imaging optical system in this order, and reaches the imaging surface of the imaging section 40.

[0037] The beam splitter 27 is provided between the cylindrical lens 26 and the lens 28. The beam splitter 27 inputs the object light arriving from the cylindrical lens 26 and the reference light arriving from the mirror 22, combines the object light and the reference light, and outputs the combined light. At this time, the beam splitter 27 outputs the object light and the reference light so that they are coaxial with each other.

[0038] The optical system from the light source 10 to the imaging unit 40 constitutes an interference optical system, which causes the object light that has passed through the observation target S to interfere with the reference light (interference step). The frequency shifter 30 causes the optical frequencies of the object light and the reference light to differ from each other by the heterodyne frequency (modulation step). The frequency shifter 30 may be constituted by, for example, an acousto-optical element. The frequency shifter 30 may be provided on the optical path of the reference light or on the optical path of the object light.

[0039] The imaging unit 40 has an imaging surface that receives the object light and reference light output from the interference optical system, captures an interference intensity image due to interference between the object light and the reference light on the imaging surface, and outputs time series data of the interference intensity image obtained by the imaging to the analysis unit 50 (imaging step).

[0040] The analysis unit 50 is electrically connected to the imaging unit 40, and performs a required analysis by inputting time series data of the interference intensity image output from the imaging unit 40 (analysis step). The analysis unit 50 generates a complex amplitude image for each of a plurality of light irradiation directions of the object light based on the time series data of the interference intensity image, and generates a three-dimensional complex differential interference image of the observation object S based on the complex amplitude images for each of the plurality of light irradiation directions. Furthermore, the analysis unit 50 obtains a three-dimensional refractive index distribution of the observation object S based on the three-dimensional complex differential interference image. Details of the analysis unit 50 will be described later.

[0041] Fig. 2 is a diagram showing the configuration of observation device 1B. This observation device 1B (Fig. 2) differs from the observation device 1A (Fig. 1) in that it does not include a frequency shifter 30, and that the object light and the reference light are incident on the imaging surface of the imaging section 40 from different directions. Due to this difference, some of the processing contents of the analysis section 50 are different.

[0042] Fig. 3 is a diagram for explaining an imaging optical system including a lens 25, a cylindrical lens 26, and a lens 28 in each of the configurations of the observation device 1A (Fig. 1) and the observation device B (Fig. 2). Fig. 3(a) is a diagram viewed in the y direction, and Fig. 3(b) is a diagram viewed in the x direction. In the x direction, the pupil plane of the lens 25 and the imaging plane of the imaging section 40 are in a mutually conjugate positional relationship, that is, the observation object S and the imaging plane of the imaging section 40 are in a mutually Fourier transform positional relationship. In the y direction, the observation object S and the imaging plane of the imaging section 40 are in a mutually conjugate positional relationship.

[0043] The analysis section 50 of each of the observation devices 1A (FIG. 1) and B (FIG. 2) performs required calculations based on the time-series data of the interference intensity image output from the imaging section 40. The analysis section 50 may be a computer. The analysis section 50 includes an interference intensity image acquisition section 51, a first complex amplitude image generation section 52, a second complex amplitude image generation section 53, a two-dimensional phase image generation section 54, a three-dimensional phase image generation section 55, a refractive index distribution calculation section 56, a display section 57, and a storage section 58.

[0044] The interference intensity image acquiring unit 51 acquires time-series data of the interference intensity image output from the imaging unit 40. The interference intensity image acquiring unit 51 includes a CPU, and also has an input port for inputting the data of the interference intensity image output from the imaging unit 40.

[0045] The first complex amplitude image generating unit 52, the second complex amplitude image generating unit 53, the two-dimensional phase image generating unit 54, the three-dimensional phase image generating unit 55, and the refractive index distribution calculating unit 56 perform processing based on the time series data of the interference intensity image, and include processing devices such as a CPU, a GPU, a DSP, or an FPGA. The display unit 57 displays the image to be processed, the image in the middle of processing, and the image after processing, and includes, for example, a liquid crystal display. The storage unit 58 stores data of various images, and includes a hard disk drive, a flash memory, a RAM, a ROM, and the like. The first complex amplitude image generating unit 52, the second complex amplitude image generating unit 53, the two-dimensional phase image generating unit 54, the three-dimensional phase image generating unit 55, the refractive index distribution calculating unit 56, and the storage unit 58 may be configured by cloud computing.

[0046] The storage unit 58 also stores a program for causing the interference intensity image acquisition unit 51, the first complex amplitude image generation unit 52, the second complex amplitude image generation unit 53, the two-dimensional phase image generation unit 54, the three-dimensional phase image generation unit 55, and the refractive index distribution calculation unit 56 to execute each process. This program may be stored in the storage unit 58 at the time of manufacture or shipment of the observation device, or may be acquired via a communication line after shipment and stored in the storage unit 58, or may be recorded in a computer-readable recording medium 2 and stored in the storage unit 58. The recording medium 2 may be any type of medium, such as a flexible disk, a CD-ROM, a DVD-ROM, a BD-ROM, or a USB memory.

[0047] The processing of the interference intensity image acquisition unit 51, the first complex amplitude image generation unit 52, the second complex amplitude image generation unit 53, the two-dimensional phase image generation unit 54, the three-dimensional phase image generation unit 55 and the refractive index distribution calculation unit 56 will be described in detail later.

[0048] The observation device 1A (FIG. 1) is provided with a frequency shifter 30 to obtain an interference intensity image by time modulation. In contrast, the observation device 1B (FIG. 2) obtains an interference intensity image by spatial modulation by making the object light and the reference light incident on the imaging surface of the imaging section 40 from different directions. The observation device 1A (FIG. 1) and the observation device B (FIG. 2) differ in the processing of the first complex amplitude image generating section 52 of the analysis section 50 due to the difference in the optical systems.

[0049] 4 and 5 are diagrams for explaining the processing contents of the first complex amplitude image generating unit 52 of the analysis unit 50 of the observation device 1A (FIG. 1). The time series data of the interference intensity image output from the imaging unit 40 (FIG. 4(a)) is x and the second direction position y, and is an interference intensity image at each time t on a plane consisting of I(k x ,y,t). x , y, t) is the complex amplitude of the object light, and u ref is the complex amplitude of the reference light. x0 And data I(k x0 ,y0,t) (Fig. 4(b)) is Fourier transformed with respect to t to obtain the light irradiation direction k x0 Frequency spectrum I(k x0 , y0,f) (Figure 5(a)) is obtained, where f is the frequency.

[0050]

number

[0051] This frequency spectrum has three peaks spaced apart by the heterodyne frequency. Of the three peaks, the central peak is due to the first and second terms on the right-hand side of equation (1), and the two peaks on the left and right are due to the third and fourth terms on the right-hand side of equation (1). The area around one of the two peaks on the left and right in this frequency spectrum (the area enclosed by the dashed rectangle in Figure 5(a)) is extracted and translated so that the peak frequency becomes 0 (Figure 5(b)). Then, by Fourier transforming this with respect to frequency f, we obtain x0 And the complex amplitude u(k x0 , y0,t) are obtained. All light irradiation directions k x By performing similar processing for the second direction position y, the time series data u(k x ,y,t) is obtained.

[0052] 6 and 7 are diagrams for explaining the processing contents of the first complex amplitude image generating unit 52 of the analysis unit 50 of the observation device 1B (FIG. 2). The time-series data of the interference intensity image output from the imaging unit 40 (FIG. 6(a)) is x and the second direction position y, and is an interference intensity image at each time t on a plane consisting of I(k x , y, t). x0 And data I(k x0 ,y,t0) (Figure 6(b)) is Fourier transformed with respect to y to obtain the light irradiation direction k x0 Spatial wavenumber spectrum I(k x0 ,k y ,t0) (Figure 7(a)). y is the spatial wavenumber in the y direction.

[0053] In this spatial wavenumber spectrum, three peaks appear, spaced apart by a wavenumber difference according to the angle between the object light and the reference light incident on the imaging surface of the imaging unit 40. Of the three peaks, the central peak is due to the first and second terms on the right-most side of equation (1), and the two peaks on the left and right are due to the third and fourth terms on the right-most side of equation (1). The periphery of one of the two peaks on the left and right in this spatial wavenumber spectrum (the range enclosed by the dashed rectangle in FIG. 7(a)) is cut out and translated so that the peak frequency becomes 0 (FIG. 7(b)). This is then used as the spatial wavenumber k y When the Fourier transform is performed on a certain light irradiation direction k x0 And the complex amplitude u(k x0 , y, t0) are obtained. All light irradiation directions k x By performing the same processing for time t, the time series data u(k x ,y,t) is obtained.

[0054] As described above, in both the observation apparatus 1A (FIG. 1) and the observation apparatus B (FIG. 2), the first complex amplitude image generating unit 52 of the analysis unit 50 converts the time series data I(k x , y, t), time series data u(k x , y, t) can be obtained. Furthermore, the first complex amplitude image generating unit 52 of the analyzing unit 50 can obtain the time series data u(k x , y, t), a complex amplitude image is generated for each of a plurality of light irradiation directions.

[0055] 8 and 9 are diagrams for explaining object light output from the observation target S in the configurations of the observation device 1A (FIG. 1) and the observation device B (FIG. 2), respectively.

[0056] If the object is stationary and the wave vector k d The incident wave on the observation object has a single spatial wave number represented by the wave vector k inIn this case, the wave vector k of the object light output from the object to be observed is out As shown in the following equation (2), vector k d and vector k in It is expressed as the sum of and.

[0057]

number

[0058] If the object being observed is moving with a constant velocity vector V and the object being observed has a wave vector k d The incident wave on the observation object has a single spatial wave number represented by the wave vector k in In this case, the wave vector k of the object light output from the object to be observed is out As shown in equation (2) above, vector k d and vector k in In addition, a Doppler shift occurs in the object light emitted from the object to be observed, and the Doppler frequency ω is expressed by the vector k d and vector V. That is, as shown in FIG. 8(a), the Doppler frequency ω is the horizontal axis, and the wave vector k of the output object light is out On the plane whose vertical axis is the magnitude of the object light ω and k emitted from the observation object, out is represented by one point.

[0059]

number

[0060] If the object being observed is moving with a constant velocity vector V and has various spatial wave numbers, and the incident wave on the object has a wave vector k in In this case, the wave vector k of the object light output from the object to be observed is outis expressed by the above equation (2), and the Doppler frequency ω is expressed by the above equation (3). However, if the object being observed has various spatial wave numbers k d As shown in FIG. 8(b), the Doppler frequency ω is plotted on the horizontal axis, and the wave vector k of the output object light is plotted on the horizontal axis. out On the plane whose vertical axis is the magnitude of the object light ω and k emitted from the observation object, out is on a straight line.

[0061] Assume that the object being observed is moving with a constant velocity vector V, the object being observed has various spatial wave numbers, and the incident wave to the object being observed has various wave number vectors. In this case, the object being observed is incident on various spatial wave numbers k d and the incident wave has various wave vectors k in As shown in FIG. 9(a), the Doppler frequency ω is plotted on the horizontal axis, and the wave vector k of the output object light is plotted on the horizontal axis. out On the plane whose vertical axis is the magnitude of the object light ω and k emitted from the observation object, out is inside the domain of the parallelogram.

[0062] In the observation apparatus 1A (FIG. 1) and the observation apparatus B (FIG. 2), the ω and k of the object light output from the observation target S are out is expressed as ω-k as shown in Fig. 9(a). out In the plane, it is within the area of ​​a parallelogram. out max ,k out min ,k in max ,k in min is a value determined by the NA of the optical system in front of or behind the object of observation. On the diagonal dashed lines shown in Figure 9(a) (three are shown in this figure), k in is constant. By using the relationship in equation (3) above, the ω-k out From the plane, k in Fig. 9(b) in -k outThe first complex amplitude image generating unit 52 of the analyzing unit 50 utilizes this to generate a complex amplitude image u(k x , y, t), a complex amplitude image is generated for each of a plurality of light irradiation directions.

[0063] That is, as shown in FIG. 10, the first complex amplitude image generating unit 52 calculates u(k x ,y,t) with respect to t and Fourier transform U(k x , y, ω) is calculated. Next, using the relationship in equation (3) above, U(k x ,y,ω) to U(k x ,y,k in ), where k out k x Then, U(k x ,y,k in ) to k x u(x,y,k in ) is required.

[0064] In this manner, the first complex amplitude image generating unit 52 of the analyzing unit 50 converts the time series data I(k x , y, t), time series data u(k x , y, t) can be obtained, and further, the time series data u(k x , y, t) based on multiple light irradiation directions k in A respective complex amplitude image u(x,y) can be generated.

[0065] 11 is a flowchart of the observation method of the first embodiment. This observation method can be performed using either the observation device 1A or the observation device 1B. This observation method includes an interference intensity image acquisition step S1, a first complex amplitude image generation step S2, a second complex amplitude image generation step S3, a two-dimensional phase image generation step S4, a three-dimensional phase image generation step S5, and a refractive index distribution calculation step S6.

[0066] The processing of interference intensity image acquisition step S1 is performed by an interference intensity image acquisition unit 51. The processing of first complex amplitude image generation step S2 is performed by a first complex amplitude image generation unit 52. The processing of second complex amplitude image generation step S3 is performed by a second complex amplitude image generation unit 53. The processing of two-dimensional phase image generation step S4 is performed by a two-dimensional phase image generation unit 54. The processing of three-dimensional phase image generation step S5 is performed by a three-dimensional phase image generation unit 55. The processing of refractive index distribution calculation step S6 is performed by a refractive index distribution calculation unit 56.

[0067] In the interference intensity image acquisition step S1, the interference intensity image acquisition unit 51 acquires time series data of the interference intensity image output from the imaging unit 40. In the first complex amplitude image generation step S2, the first complex amplitude image generation unit 52 generates the time series data I(k x , y, t), time series data u(k x , y, t), and then the time series data u(k x , y, t) based on multiple light irradiation directions k in Generate a respective complex amplitude image u(x,y).

[0068] In the second complex amplitude image generating step S3, the second complex amplitude image generating unit 53 generates a complex amplitude image for each of the multiple z-direction positions based on the complex amplitude image for the reference position (z=0) generated by the first complex amplitude image generating unit 52 for each of the multiple light irradiation directions. The complex amplitude image u(x, y, 0) for the reference position is transformed into a complex amplitude image U(k x ,k y ,0), the complex amplitude image u(x,y,d) at the position z=d and the two-dimensional Fourier transform U(k x ,k y , d) are expressed by the following formula, where i is the imaginary unit and k0 is the wave number of light in the object being observed.

[0069]

number

[0070]

number

[0071] In the two-dimensional phase image generating step S4, the two-dimensional phase image generating unit 54 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images in each of the multiple light irradiation directions generated by the second complex amplitude image generating unit 53. The two-dimensional phase image generated here corresponds to a phase image centered on the focused z-direction position. The two-dimensional phase image generating step S4 will be described later in detail.

[0072] Note that the processes in and after the two-dimensional phase image generating step S4 may be performed after all of the complex amplitude images at the multiple positions for each of the multiple light irradiation directions are generated in the second complex amplitude image generating step S3. Alternatively, a complex amplitude image at one z-direction position may be generated for each of the multiple light irradiation directions in the second complex amplitude image generating step S3, and a two-dimensional phase image at that position may be generated in the two-dimensional phase image generating step S4. The unit process may be repeated while scanning the z-direction position. The latter case is preferable in that the amount of image data to be stored in the storage unit 58 can be reduced.

[0073] In a three-dimensional phase image generating step S5, the three-dimensional phase image generating section 55 generates a three-dimensional phase image based on the two-dimensional phase images at each of the multiple positions generated by the two-dimensional phase image generating section 54. The three-dimensional phase image generated here is an image whose variables are positions x, y in the two-dimensional phase image and position z of the two-dimensional phase image.

[0074] In the refractive index distribution calculation step S6, the refractive index distribution calculation unit 56 obtains the three-dimensional refractive index distribution of the observation object by deconvolution based on the three-dimensional phase image generated by the three-dimensional phase image generation unit 55. The refractive index distribution of the observation object is n(x,y,z), the electrical susceptibility distribution is f(x,y,z), and the refractive index of the background medium is n m Then, there is a relationship between the two as shown in the following formula (6). The three-dimensional phase image Φ(x,y,z) generated by the three-dimensional phase image generating unit 55 is expressed by the convolution of the kernel function g(x,y,z) and the electric susceptibility distribution f(x,y,z) as shown in the following formula (7). Therefore, the three-dimensional refractive index distribution n(x,y,z) of the observation object can be obtained by deconvolution based on the three-dimensional phase image Φ(x,y,z).

[0075]

number

[0076]

number

[0077] The kernel function g is based on the Green's function corresponding to the solution of the wave equation. Figure 12 is a diagram explaining the kernel function g. In this diagram, the central position where the value of the kernel function g is the largest is the origin, the vertical direction is the z-axis, and the horizontal direction is the direction perpendicular to the z-axis.

[0078] Next, the two-dimensional phase image generating step S4 in the observation method of the first embodiment will be described in detail. In the two-dimensional phase image generating step S4, the two-dimensional phase image generating section 54 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images in each of the multiple light irradiation directions generated by the second complex amplitude image generating section 53. The two-dimensional phase image generating step S4 can have three aspects described below.

[0079] 13 is a flow chart of a first aspect of the two-dimensional phase image generating step S4. In this first aspect, the two-dimensional phase image generating step S4 corrects the phases of the complex amplitude images in each of the multiple light irradiation directions based on the light irradiation direction for each of the multiple positions in step S11, generates a complex amplitude sum image representing the sum of these corrected complex amplitude images, and generates a two-dimensional phase image based on the complex amplitude sum image in step S12.

[0080] The process of step S11 is based on CASS (Collective Accumulation of Single Scattering) technology. Among light that is irradiated to an object along a certain light irradiation direction and passes through the object, the spatial frequency distribution of single scattered light that has interacted with the object only once shifts depending on the light irradiation direction, whereas the spatial frequency distribution of multiple scattered light that has interacted with the object multiple times changes randomly depending on the light irradiation direction. CASS technology utilizes the difference in the light irradiation direction dependency of the spatial frequency distribution of single scattered light and multiple scattered light.

[0081] That is, in step S11, the phase of the complex amplitude image for each of the multiple light irradiation directions is corrected based on the light irradiation direction (i.e., the spatial frequency distribution of the complex amplitude image in the spatial frequency domain is translated according to the light irradiation direction), so that the spatial frequency distribution of the single scattered light component in the complex amplitude image has a shape and arrangement that is independent of the light irradiation direction, while the spatial frequency distribution of the multiple scattered light component in the complex amplitude image has a random shape and arrangement. Then, in step S11, a complex amplitude sum image that represents the sum of the multiple complex amplitude images after these corrections is generated (i.e., synthetic aperture processing is performed), so that the single scattered light components in the complex amplitude images are coherently added together, while the multiple scattered light components in the complex amplitude images are offset by each other.

[0082] Therefore, the complex amplitude sum image generated in step S11 has a reduced effect of multiple scattered light, and the three-dimensional refractive index distribution finally obtained in the refractive index distribution calculation step S6 also has a reduced effect of multiple scattered light, suppressing speckles and improving SMR.

[0083] 14 is a flow chart of a second aspect of the two-dimensional phase image generating step S4. In this second aspect, the two-dimensional phase image generating step S4 generates, for each of the multiple positions, a complex differential interference image for each of the multiple light irradiation directions based on a complex amplitude image for each of the multiple light irradiation directions in step S21, generates a phase differential image based on the sum of the complex differential interference images for each of the multiple light irradiation directions in step S22, and generates a two-dimensional phase image based on the phase differential image in step S23.

[0084] If the complex amplitude image at the position z=d is u(x,y,d), the complex differential interference image q(x,y,d) generated in step S21 is expressed by the following formula (8). At least one of δx and δy is non-zero. If δx≠0, δy=0, a complex differential interference image q with the x direction as the shear direction is obtained. If δx=0, δy≠0, a complex differential interference image q with the y direction as the shear direction is obtained. If δx≠0, δy≠0, a complex differential interference image q with a shear direction different from both the x and y directions is obtained. Note that the complex differential interference image q(x,y,d) may be calculated by formula (8) after converting the complex amplitude image u(x,y,d) as shown in the following formula (9).

[0085]

number

[0086]

number

[0087] The sum of the complex differential interference images q for each of the multiple light irradiation directions is q sum(x, y, d), the phase differential image φ(x, y, z) generated in step S22 is q sum The phase of (x, y, d) is expressed by the following equation (10): In step S23, a two-dimensional phase image can be generated by integrating or deconvolving this phase differential image φ(x, y, z).

[0088]

number

[0089] In step S21, a complex differential interference image may be generated for each of a plurality of different shear directions on the complex amplitude image. In this case, the two-dimensional phase image generating step S4 generates a complex differential interference image for each of a plurality of different shear directions on the image based on the complex amplitude images for each of the plurality of light irradiation directions for each of the plurality of positions in step S21, generates a phase differential image for each of the plurality of shear directions based on the sum of the complex differential interference images for each of the plurality of light irradiation directions in step S22, and generates a two-dimensional phase image based on the phase differential images for each of the plurality of shear directions in step S23.

[0090] In step S22, the differential phase image generated based on the sum of the complex differential interference images for each of the multiple light irradiation directions has a reduced effect of multiple scattered light. The three-dimensional refractive index distribution finally obtained in step S6 of calculating the refractive index distribution also has a reduced effect of multiple scattered light and suppressed speckle. In addition, when a complex differential interference image is generated for each of the multiple shear directions different from each other on the complex amplitude image in step S21, it is possible to suppress the appearance of line-shaped noise in the two-dimensional phase image obtained in step S23.

[0091] Here, a case has been described in which a two-dimensional phase image is generated by integrating or deconvolving the phase differential image in step S23. However, the phase differential image can also be treated as a two-dimensional phase image. In this case, without performing step S23, a three-dimensional refractive index distribution of the observation object can be obtained from the phase differential image (two-dimensional phase image) generated in step S22 by using a kernel (FIG. 15) including the kernel used in the deconvolution in step S23 in the deconvolution in the refractive index distribution calculation step S67. The kernel shown in FIG. 15 is obtained by convolution integration of the kernel shown in FIG. 12 and the kernel used in the deconvolution in step S23.

[0092] 16 is a flow chart of a third aspect of the two-dimensional phase image generating step S4. In this third aspect, the two-dimensional phase image generating step S4 divides the complex amplitude images of each of the multiple light irradiation directions into multiple batches for each of the multiple positions in step S31, corrects the phase of the complex amplitude image included in each of the multiple batches based on the light irradiation direction, and generates a complex amplitude sum image representing the sum of these corrected complex amplitude images, generates a complex differential interference image for each of the multiple batches based on the complex amplitude sum image for each of the multiple batches in step S32, generates a phase differential image based on the sum of the complex differential interference images for each of the multiple batches in step S33, and generates a two-dimensional phase image based on the phase differential image in step S34.

[0093] The process of step S31 in the third aspect corresponds to dividing the complex amplitude images in each of the multiple light irradiation directions into multiple batches, and then performing the process of step S11 in the first aspect for each of the multiple batches. The processes of steps S32 and S33 in the third aspect correspond to performing the processes of steps S21 and S22 in the second aspect for each of the multiple batches. The process of step S34 in the third aspect corresponds to performing the process of step S23 in the second aspect.

[0094] In step S32, a complex differential interference image may be generated for each of a plurality of different shear directions on the complex amplitude image. In this case, the two-dimensional phase image generating step S4 generates a complex differential interference image for each of a plurality of batches for each of a plurality of different shear directions on the image based on a complex amplitude sum image of each of the plurality of batches in step S32, generates a phase differential image for each of the plurality of shear directions based on the sum of the complex differential interference images of each of the plurality of batches in step S33, and generates a two-dimensional phase image based on the phase differential image for each of the plurality of shear directions in step S34.

[0095] The suppression of speckle in the third embodiment is comparable to that in the first and second embodiments. The improvement of SMR in the third embodiment is intermediate between that in the first and second embodiments.

[0096] Here, too, a case has been described in which a two-dimensional phase image is generated by integrating or deconvolving the phase differential image in step S34. However, the phase differential image can also be treated as a two-dimensional phase image. In this case, without performing step S34, a three-dimensional refractive index distribution of the observation object can be obtained from the phase differential image (two-dimensional phase image) generated in step S33 by using a kernel including the kernel used in the deconvolution in step S34 in the deconvolution in the refractive index distribution calculation step S6.

[0097] According to this embodiment, even if the moving observation object is a multiple scattering medium, the influence of multiple scattered light can be reduced and the observation object can be observed.

[0098] Second embodiment As the configuration of the device of the second embodiment, an observation device 1C and an observation device 1D will be described. The observation device 1C shown in Fig. 17 is different from the observation device 1A shown in Fig. 1 in that the optical system from the light source 10 to the image capture unit 40 is common, but the analysis unit 60 is provided instead of the analysis unit 50. The observation device 1D shown in Fig. 18 is different from the observation device 1B shown in Fig. 2 in that the optical system from the light source 10 to the image capture unit 40 is common, but the analysis unit 60 is provided instead of the analysis unit 50.

[0099] The analysis unit 60 is electrically connected to the imaging unit 40, and inputs time series data of the interference intensity image output from the imaging unit 40. The analysis unit 60 generates a complex amplitude image for each of a plurality of light irradiation directions of the object light based on the time series data of the interference intensity image, and generates a three-dimensional complex differential interference image of the observation object S based on the complex amplitude images for each of the plurality of light irradiation directions. Furthermore, the analysis unit 60 obtains a three-dimensional refractive index distribution of the observation object S based on the three-dimensional complex differential interference image. The analysis unit 60 may be a computer. The analysis unit 60 includes an interference intensity image acquisition unit 61, a first complex amplitude image generation unit 62, a second complex amplitude image generation unit 63, a phase conjugate calculation unit 64, a two-dimensional phase image generation unit 65, a three-dimensional phase image generation unit 66, a refractive index distribution calculation unit 67, a display unit 68, and a storage unit 69.

[0100] The interference intensity image acquiring unit 61 acquires time-series data of the interference intensity image output from the imaging unit 40. The interference intensity image acquiring unit 61 includes a CPU, and also has an input port for inputting the data of the interference intensity image output from the imaging unit 40.

[0101] The first complex amplitude image generating unit 62, the second complex amplitude image generating unit 63, the phase conjugate calculation unit 64, the two-dimensional phase image generating unit 65, the three-dimensional phase image generating unit 66, and the refractive index distribution calculation unit 67 perform processing based on the time series data of the interference intensity image, and include processing devices such as a CPU, a GPU, a DSP, or an FPGA. The display unit 68 displays the image to be processed, the image in the middle of processing, and the image after processing, and includes, for example, a liquid crystal display. The storage unit 69 stores data of various images, and includes a hard disk drive, a flash memory, a RAM, a ROM, and the like. The first complex amplitude image generating unit 62, the second complex amplitude image generating unit 63, the phase conjugate calculation unit 64, the two-dimensional phase image generating unit 65, the three-dimensional phase image generating unit 66, the refractive index distribution calculation unit 67, and the storage unit 69 may be configured by cloud computing.

[0102] The storage unit 69 also stores programs for causing the interference intensity image acquisition unit 61, the first complex amplitude image generation unit 62, the second complex amplitude image generation unit 63, the phase conjugate calculation unit 64, the two-dimensional phase image generation unit 65, the three-dimensional phase image generation unit 66, and the refractive index distribution calculation unit 67 to execute each process. This program may be stored in the storage unit 69 at the time of manufacturing or shipping the observation device 1A, or may be acquired via a communication line after shipping and stored in the storage unit 69, or may be stored in the storage unit 69 after being recorded on a computer-readable recording medium 2. The recording medium 2 may be any type of medium, such as a flexible disk, a CD-ROM, a DVD-ROM, a BD-ROM, or a USB memory.

[0103] Details of the processing of each of the interference intensity image acquisition unit 61, first complex amplitude image generation unit 62, second complex amplitude image generation unit 63, phase conjugate calculation unit 64, two-dimensional phase image generation unit 65, three-dimensional phase image generation unit 66 and refractive index distribution calculation unit 67 will be described later.

[0104] 19 is a flowchart of the observation method of the second embodiment. This observation method can be performed using either the observation device 1C or the observation device 1D. This observation method includes an interference intensity image acquisition step S61, a first complex amplitude image generation step S62, a second complex amplitude image generation step S63, a phase conjugate calculation step S64, a two-dimensional phase image generation step S65, a three-dimensional phase image generation step S66, and a refractive index distribution calculation step S67.

[0105] The processing of the interference intensity image acquisition step S61 is performed by the interference intensity image acquisition unit 61. The processing of the first complex amplitude image generation step S62 is performed by the first complex amplitude image generation unit 62. The processing of the second complex amplitude image generation step S63 is performed by the second complex amplitude image generation unit 63. The processing of the phase conjugate calculation step S64 is performed by the phase conjugate calculation unit 64. The processing of the two-dimensional phase image generation step S65 is performed by the two-dimensional phase image generation unit 65. The processing of the three-dimensional phase image generation step S66 is performed by the three-dimensional phase image generation unit 66. The processing of the refractive index distribution calculation step S67 is performed by the refractive index distribution calculation unit 67.

[0106] In an interference intensity image acquisition step S61, the interference intensity image acquisition unit 61 acquires time series data of the interference intensity image output from the imaging unit 40. In a first complex amplitude image generation step S62, the first complex amplitude image generation unit 62 generates the time series data I(k x , y, t), time series data u(k x , y, t), and then the time series data u(k x , y, t) based on multiple light irradiation directions k in Generate a respective complex amplitude image u(x,y).

[0107] In a second complex amplitude image generation step S63, the second complex amplitude image generation unit 63 generates a complex amplitude image for each of the multiple z-direction positions based on the complex amplitude image of the reference position (z=0) generated by the first complex amplitude image generation unit 62 for each of the multiple light irradiation directions.

[0108] The interference intensity image acquisition step S61, the first complex amplitude image generation step S62, and the second complex amplitude image generation step S63 in the observation method of the second embodiment perform processing similar to the interference intensity image acquisition step S1, the first complex amplitude image generation step S2, and the second complex amplitude image generation step S3 in the observation method of the first embodiment, respectively.

[0109] The phase conjugate calculation step S64 is performed after the processing of the second complex amplitude image generating step S63. The phase conjugate calculation step S64 may be performed before the processing of the second complex amplitude image generating step S63 (described later). In addition, when the second complex amplitude image generating step S63 generates a complex amplitude image of a certain z position from a complex amplitude image of a reference position through multiple stages, the phase conjugate calculation step S64 may be performed between a certain stage and the next stage among the multiple stages (described later). In the phase conjugate calculation step S64, the phase conjugate calculation unit 64 performs a phase conjugate calculation on the complex amplitude images of each of the multiple irradiation directions to generate complex amplitude images of each of the multiple irradiation directions when the relationship between light irradiation and imaging of the observation object is reversed.

[0110] The phase conjugate calculation is a calculation for a complex amplitude image based on the phase conjugate method, which calculates a transmission matrix that represents the relationship between light irradiation and light output in an object, and includes the inverse matrix calculation and coordinate transformation. The phase conjugate method is also called phase conjugation, time reversal method, time reversal, digital phase conjugation, digital phase conjugate method, etc. Details will be described later.

[0111] In a two-dimensional phase image generating step S65, the two-dimensional phase image generating section 65 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images in each of the multiple light irradiation directions generated by the second complex amplitude image generating section 63 or the phase conjugate calculation section 64. The two-dimensional phase image generated here corresponds to a phase image centered on the focused z-direction position.

[0112] In the two-dimensional phase image generation step S65, when a phase image generated based on the complex amplitude image before the processing of the phase conjugate calculation step S64 is designated as a first phase image, and a phase image generated based on the complex amplitude image obtained by performing the processing of the phase conjugate calculation step S64 is designated as a second phase image, among the multiple positions, for positions relatively close to the imaging unit, a two-dimensional phase image is generated mainly based on the first phase image, and for positions relatively far from the imaging unit, a two-dimensional phase image is generated mainly based on the second phase image.

[0113] Note that the processes after the phase conjugate calculation unit 64 may be performed after all complex amplitude images at the multiple positions for each of the multiple light irradiation directions are generated in the second complex amplitude image generating step S63. Alternatively, a complex amplitude image at one z direction position may be generated for each of the multiple light irradiation directions in the second complex amplitude image generating step S63, and a two-dimensional phase image at that position may be generated in the two-dimensional phase image generating step S65. The unit process may be repeated while scanning the z direction position. The latter case is preferable because it allows the amount of image data to be stored in the storage unit 69 to be reduced.

[0114] In a three-dimensional phase image generating step S66, the three-dimensional phase image generating section 66 generates a three-dimensional phase image based on the two-dimensional phase images at each of the multiple positions generated by the two-dimensional phase image generating section 65. The three-dimensional phase image generated here is an image whose variables are positions x, y in the two-dimensional phase image and position z of the two-dimensional phase image.

[0115] In a refractive index distribution calculation step S67, the refractive index distribution calculation section 67 obtains the three-dimensional refractive index distribution of the observation object by deconvolution based on the three-dimensional phase image generated by the three-dimensional phase image generation section 66.

[0116] A two-dimensional phase image generating step S65, a three-dimensional phase image generating step S66, and a refractive index distribution calculating step S67 in the observation method of the second embodiment perform processing similar to the two-dimensional phase image generating step S4, the three-dimensional phase image generating step S5, and the refractive index distribution calculating step S6 in the observation method of the first embodiment, respectively.

[0117] FIG. 20 is a diagram for explaining the order of the processes and images in the second complex amplitude image generating step S63 and the two-dimensional phase image generating step S65. This diagram shows an embodiment in which the process in the phase conjugate calculation step S64 is not performed. In this embodiment, in the second complex amplitude image generating step S63, for each of the multiple light irradiation directions, a complex amplitude image for each of multiple z-direction positions (z=z1, z2, z3 in this diagram) is generated based on the complex amplitude image of the reference position (z=0) generated in the first complex amplitude image generating step S62 by the free propagation equations (4) and (5) above. Then, in the two-dimensional phase image generating step S65, for each of the multiple positions, a complex differential interference image is generated based on the complex amplitude images for each of the multiple light irradiation directions generated in the second complex amplitude image generating step S63, and further a phase differential image is generated.

[0118] 21 to 23 are diagrams for explaining the order and images of the processes in the second complex amplitude image generating step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generating step S65. These diagrams show the manner in which the process in the phase conjugate calculation step S64 is performed before, during, or after the process in the second complex amplitude image generating step S63.

[0119] The first mode shown in Fig. 21 corresponds to the flowchart of Fig. 19. In this first mode, the phase conjugate calculation step S64 is performed after the processing of the second complex amplitude image generating step S63. In the second complex amplitude image generating step S63, for each of the multiple light irradiation directions, a complex amplitude image is generated at each of multiple z direction positions (z = z1, z2, z3 in this figure) based on the complex amplitude image at the reference position (z = 0) generated in the first complex amplitude image generating step S62 by the free propagation equations (4) and (5) above.

[0120] In the first embodiment, subsequently, in a phase conjugate calculation step S64, a phase conjugate calculation is performed on the complex amplitude images in each of the multiple irradiation directions for each of the multiple positions, and a complex amplitude image in each of the multiple irradiation directions when the relationship between light irradiation and imaging of the observation object is reversed is generated. Then, in a two-dimensional phase image generation step S65, a complex differential interference image is generated for each of the multiple positions based on the complex amplitude images in each of the multiple light irradiation directions generated in the phase conjugate calculation step S64, and further a phase differential image is generated.

[0121] 22, a phase conjugate calculation step S64 is performed before the processing of the second complex amplitude image generating step S63. In the phase conjugate calculation step S64, for each of the plurality of light irradiation directions, a phase conjugate calculation is performed on the complex amplitude image of the reference position (z=0) generated in the first complex amplitude image generating step S62, and a complex amplitude image for each of the plurality of irradiation directions when the relationship between light irradiation and imaging of the observation object is reversed is generated.

[0122] In the second aspect, subsequently, in a second complex amplitude image generating step S63, for each of the multiple light irradiation directions, a complex amplitude image for each of multiple z direction positions (z=z1, z2, z3 in this figure) is generated based on the complex amplitude image for the reference position (z=0) generated in the phase conjugate calculation step S64 by the free propagation equations (4) and (5) above. Then, in a two-dimensional phase image generating step S65, for each of the multiple positions, a complex differential interference image is generated based on the complex amplitude images for each of the multiple light irradiation directions generated in the second complex amplitude image generating step S63, and further a phase differential image is generated.

[0123] In the third aspect shown in FIG. 23, when the second complex amplitude image generation step S63 generates complex amplitude images for each of a plurality of positions from a complex amplitude image for a reference position through two stages, a phase conjugate calculation step S64 is performed between the first and second stages of the two stages.

[0124] In the third aspect, in the first stage of the second complex amplitude image generating step S63, for each of the multiple light irradiation directions, a complex amplitude image for each of multiple z direction positions (z=z1, z3, z5 in this figure) is generated based on the complex amplitude image for the reference position (z=0) generated in the first complex amplitude image generating step S62 by the free propagation equations (4) and (5) above. Then, in the phase conjugate calculation step S64, a phase conjugate calculation is performed on the complex amplitude images for each of the multiple irradiation directions, and a complex amplitude image for each of the multiple irradiation directions when the relationship between light irradiation and imaging of the observation object is reversed is generated.

[0125] In the third aspect, furthermore, in the second stage of the second complex amplitude image generating step S63, a complex amplitude image for each of the z direction positions (z=z2, z4, z6) is generated for each of the multiple light irradiation directions based on the complex amplitude image for the z direction position (z=z1, z3, z5) generated in the phase conjugate calculation step S64 by the free propagation equations (4) and (5) above. Then, in the two-dimensional phase image generating step S65, a complex differential interference image is generated for each of the multiple positions based on the complex amplitude images for each of the multiple light irradiation directions generated in the second complex amplitude image generating step S63, and further a phase differential image is generated.

[0126] The number of times the phase conjugate calculation is performed on the complex amplitude image in the phase conjugate calculation step S64 differs between the first, second and third modes. The overall processing time of the phase conjugate calculation step S64 is shorter in the third mode than in the first mode, and is even shorter in the second mode.

[0127] FIG. 24 is a diagram for explaining the order of each process and images in the three-dimensional phase image generating step S66 and the refractive index distribution calculating step S67. In the three-dimensional phase image generating step S66, a three-dimensional phase image is generated based on the two-dimensional phase images at each of the multiple positions generated in the two-dimensional phase image generating step S65. At this time, for a position relatively close to the imaging unit, a two-dimensional phase image generated based on the complex amplitude image before the processing of the phase conjugate calculation step S64 (a two-dimensional phase image generated in the manner of FIG. 20) is mainly adopted. On the other hand, for a position relatively far from the imaging unit, a two-dimensional phase image generated based on the complex amplitude image after the processing of the phase conjugate calculation step S64 (a two-dimensional phase image generated in any of the manners of FIG. 21 to FIG. 23) is mainly adopted. Subsequently, in the refractive index distribution calculating step S67, a three-dimensional refractive index distribution of the observation object is obtained by deconvolution based on the three-dimensional phase image generated in the three-dimensional phase image generating step S66.

[0128] There are three modes for generating two-dimensional phase images at each position in the z direction: A phase image generated based on the complex amplitude image before the processing of phase conjugate calculation step S64 (a phase image generated in the mode of FIG. 20) is defined as a first phase image φ1. A phase image generated based on the complex amplitude image after the processing of phase conjugate calculation step S64 (a phase image generated in any mode of FIG. 21 to FIG. 23) is defined as a second phase image φ2. A weighting function α whose differential coefficient with respect to variable z, which indicates the distance from the imaging unit along the light propagation path, is 0 or less is used. The value of the weighting function is 0 or more and 1 or less.

[0129] In the first embodiment, the weighting function α is set so that z is smaller than the threshold z th In the following range, the value is a positive value (for example, 1), and in other ranges, the value is 0. That is, the two-dimensional phase image is expressed by the following formula (11).

[0130]

number

[0131] In the second aspect, the weighting function α changes continuously in value in at least a partial range in the z direction. That is, the two-dimensional phase image is expressed by the following equation (12).

[0132]

number

[0133] In the third aspect, the weighting function α has a value according to the position (x, y) on a plane perpendicular to the optical axis (z direction). That is, the two-dimensional phase image is expressed by the following equation (13).

[0134]

number

[0135] Next, the contents of the phase conjugate calculation in the phase conjugate calculation step S64 will be described with reference to FIG. 25 and FIG.

[0136] FIG. 25 shows the input light U in (k in ) and the output light u out (r out ) is a diagram showing the U in (k in ) is the wave number k of the light irradiated to the object of observation in represents the complex amplitude of u out (r out ) is the position r of the light emitted from the object being observed out represents the complex amplitude of U in (k in ) and u out (r out The relationship between the column vector U in The nth element of U in (k in n ) is the wave number k in n represents the complex amplitude of the plane wave of the column vector u out The nth element u of out (rout n ) at position r out n represents the complex amplitude of the light observed at the out ,k in ) is U in (k in ) and u out (r out ) and is called the transmission matrix. Such a transmission matrix can represent the scattering process of light in an object to be observed. The matrix T(r out ,k in Element T in the n1th row and n2th column of n1,n2 is the wave number k in n2 When a plane wave with amplitude 1 is input at position r out n1 represents the complex amplitude of the light observed at

[0137]

number

[0138] Figure 26 shows the input light U when the relationship between light irradiation and imaging is reversed. out (k out ) and the output light u in (r in ) is shown in FIG. out (k out ) is the wave number k of the light irradiated to the object of observation out represents the complex amplitude of u in (r in ) is the position r of the light emitted from the object being observed in represents the complex amplitude of U out (k out ) and u in (r in The relationship between the column vector U out The nth element of U out (k out n ) is the wave number k out n represents the complex amplitude of the plane wave of the column vector u inThe nth element u of in (r in n ) at position r in n represents the complex amplitude of the light observed at the in ,k out ) is U out (k out ) and u in (r in ) and is the transmission matrix when the relationship between light illumination and imaging is reversed.

[0139]

number

[0140] U in (k in ) is expressed as u in (r in ) is expressed as the Fourier transform of U out (k out ) is expressed as u out (r out ) is expressed by the Fourier transform of the equations (14) to (17). When the relationship between light irradiation and imaging is reversed, the transmission matrix S(r in ,k out ) is the matrix representing the inverse Fourier transform and the transmission matrix T(r out ,k in ) is expressed by the following equation (18).

[0141]

number

[0142]

number

[0143]

number

[0144] In the phase conjugate calculation step S64, first, the transmission matrix T(r out ,k in ) is obtained. Next, this transmission matrix T(r out ,k in ) and the above equation (18), the transmission matrix S(r in ,k out ) is calculated. Then, the transmission matrix S(r in ,k out ) to obtain a complex amplitude image when the relationship between light irradiation and imaging is reversed.

[0145] When an interference intensity image is captured by the imaging unit for each of a plurality of light irradiation directions, the input light vector U in the n-th light irradiation direction is in n (k in ) is expressed by the following equation (19), where only the value of the n-th element is 1 and the values ​​of the other elements are 0. in n (k in ) for output light u out n (r out ) is expressed by the following formula (20): This formula (20) corresponds to the complex amplitude obtained in the nth light irradiation direction.

[0146]

number

[0147]

number

[0148] From this formula (19) and the above formula (14), the following formula (21) is obtained. Then, by similarly calculating for each of the multiple light irradiation directions, the following formula (22) is obtained. In this way, the transmission matrix T(rout ,k in ) can be obtained. Furthermore, from this equation (22) and the above equation (18), the transmission matrix S(r in ,k out ) can be obtained.

[0149]

number

[0150]

number

[0151] Input light U in the nth light irradiation direction among multiple light irradiation directions when the relationship between light irradiation and imaging is reversed out n (k out ) is expressed by the following equation (23), where only the value of the n-th element is 1 and the values ​​of the other elements are 0. From this equation, the input light U out n (k out ) versus output light u in n (r in ) is expressed by the following equation (24). This equation (24) represents the complex amplitude when the relationship between light irradiation and imaging is reversed. In this way, it is possible to obtain a complex amplitude image when the relationship between light irradiation and imaging is reversed.

[0152]

number

[0153]

number

[0154] When the relationship between light illumination and imaging is reversed, the transmission matrix S(r in ,k out) is calculated using the transmission matrix T(r out ,k in ) must be calculated. Therefore, the transmission matrix T must be a square matrix in which the number of row elements is equal to the number of column elements. In other words, the matrix dimension in the light illumination sideband space for the object of observation in the interference intensity image acquisition step S61 must be equal to the number of pixels in the complex amplitude image.

[0155] To make them equal to each other, the dimension of the matrix in the light irradiation side wavenumber space for the object to be observed in the interference intensity image acquisition step S61 may be made to match the number of pixels, or only a part of the image obtained by the imaging unit may be used for subsequent processing. However, since the number of pixels of the image obtained by the imaging unit is generally, for example, 1024×1024, it is not easy to make the dimension of the matrix in the light irradiation side wavenumber space for the object to be observed the same as the number of pixels. In addition, using only a part of the image obtained by the imaging unit for subsequent processing is not preferable because it leads to a decrease in resolution.

[0156] Therefore, as shown in Fig. 27, in the phase conjugate calculation step S64, it is preferable to divide the complex amplitude image into a plurality of partial images each having the same number of pixels as the dimension of the matrix in the light irradiation side wavenumber space for the observation object, perform the phase conjugate calculation for each of the plurality of partial images, and then combine the plurality of partial images. At this time, any two or more of the plurality of partial images may have a common area.

[0157] In this embodiment as well, even if the moving observation object is a multiple scattering medium, the influence of multiple scattered light can be reduced and the observation object can be observed.

[0158] Third embodiment As the configuration of the apparatus of the third embodiment, the observation apparatus 1E and the observation apparatus 1F will be described. The observation apparatus 1E shown in Fig. 28 is different from the observation apparatus 1A shown in Fig. 1 in that the optical system from the light source 10 to the image capturing section 40 is common, but the analysis section 70 is provided instead of the analysis section 50. The observation apparatus 1F shown in Fig. 29 is different from the observation apparatus 1B shown in Fig. 2 in that the optical system from the light source 10 to the image capturing section 40 is common, but the analysis section 70 is provided instead of the analysis section 50.

[0159] The analysis unit 70 is electrically connected to the imaging unit 40, and inputs time series data of the interference intensity image output from the imaging unit 40. The analysis unit 70 generates a complex amplitude image for each of a plurality of light irradiation directions of the object light based on the time series data of the interference intensity image, and generates a three-dimensional complex differential interference image of the observation object S based on the complex amplitude images for each of the plurality of light irradiation directions. Furthermore, the analysis unit 70 obtains a three-dimensional refractive index distribution of the observation object S based on the three-dimensional complex differential interference image. The analysis unit 70 may be a computer. The analysis unit 70 includes an interference intensity image acquisition unit 71, a first complex amplitude image generation unit 72, a second complex amplitude image generation unit 73, a two-dimensional phase image generation unit 74, a three-dimensional phase image generation unit 75, a refractive index distribution calculation unit 76, a third complex amplitude image generation unit 77, a display unit 78, and a storage unit 79.

[0160] The interference intensity image acquiring unit 71 acquires time-series data of the interference intensity image output from the imaging unit 40. The interference intensity image acquiring unit 71 includes a CPU, and also has an input port for inputting the data of the interference intensity image output from the imaging unit 40.

[0161] The first complex amplitude image generating unit 72, the second complex amplitude image generating unit 73, the two-dimensional phase image generating unit 74, the three-dimensional phase image generating unit 75, the refractive index distribution calculating unit 76, and the third complex amplitude image generating unit 77 perform processing based on the time series data of the interference intensity image, and include processing devices such as a CPU, a GPU, a DSP, or an FPGA. The display unit 78 displays the image to be processed, the image in the middle of processing, and the image after processing, and includes, for example, a liquid crystal display. The storage unit 79 stores data of various images, and includes a hard disk drive, a flash memory, a RAM, a ROM, and the like. The first complex amplitude image generating unit 72, the second complex amplitude image generating unit 73, the two-dimensional phase image generating unit 74, the three-dimensional phase image generating unit 75, the refractive index distribution calculating unit 76, the third complex amplitude image generating unit 77, and the storage unit 79 may be configured by cloud computing.

[0162] The storage unit 79 also stores a program for causing the interference intensity image acquisition unit 71, the first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 to execute each process. This program may be stored in the storage unit 79 at the time of manufacturing or shipping the observation device 1A, or may be acquired via a communication line after shipping and stored in the storage unit 79, or may be recorded in a computer-readable recording medium 2 and stored in the storage unit 79. The recording medium 2 may be any type such as a flexible disk, a CD-ROM, a DVD-ROM, a BD-ROM, or a USB memory.

[0163] Details of the processing of each of the interference intensity image acquisition unit 71, the first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 will be described later.

[0164] 30 and 31 are flowcharts of the observation method of the third embodiment. FIG. 31 shows a part of the flowchart of FIG. 30. This observation method is possible when either the observation device 1E or the observation device 1F is used. This observation method includes an interference intensity image acquisition step S71, a first complex amplitude image generation step S72, a second complex amplitude image generation step S73, a two-dimensional phase image generation step S74, a three-dimensional phase image generation step S75, a refractive index distribution calculation step S76, and a third complex amplitude image generation step S77.

[0165] The processing of the interference intensity image acquisition step S71 is performed by the interference intensity image acquisition unit 71. The processing of the first complex amplitude image generation step S72 is performed by the first complex amplitude image generation unit 72. The processing of the second complex amplitude image generation step S73 is performed by the second complex amplitude image generation unit 73. The processing of the two-dimensional phase image generation step S74 is performed by the two-dimensional phase image generation unit 74. The processing of the three-dimensional phase image generation step S75 is performed by the three-dimensional phase image generation unit 75. The processing of the refractive index distribution calculation step S76 is performed by the refractive index distribution calculation unit 76. The processing of the third complex amplitude image generation step S77 is performed by the third complex amplitude image generation unit 77.

[0166] In the interference intensity image acquisition step S71, the interference intensity image acquisition unit 71 acquires time series data of the interference intensity image output from the imaging unit 40. In the first complex amplitude image generation step S72, the first complex amplitude image generation unit 72 generates the time series data I(k x , y, t), time series data u(k x , y, t), and then the time series data u(k x , y, t) based on multiple light irradiation directions k in Generate a respective complex amplitude image u(x,y).

[0167] In a second complex amplitude image generation step S73, the second complex amplitude image generation unit 73 generates complex amplitude images for each of a plurality of z-direction positions between the first position and the second position based on a complex amplitude image of a first position in relation to a distance from the imaging unit 40 along the light propagation path for each of a plurality of light irradiation directions.

[0168] In a two-dimensional phase image generating step S74, the two-dimensional phase image generating section 74 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images in each of the multiple light irradiation directions generated by the second complex amplitude image generating section 73. The two-dimensional phase image generated here corresponds to a phase image centered on the focused z-direction position.

[0169] In a three-dimensional phase image generating step S75, the three-dimensional phase image generating section 75 generates a three-dimensional phase image between the first position and the second position based on the two-dimensional phase images at each of the multiple positions generated by the two-dimensional phase image generating section 74. The three-dimensional phase image generated here is an image whose variables are positions x, y in the two-dimensional phase image and position z of the two-dimensional phase image.

[0170] In the refractive index distribution calculation step S76, the refractive index distribution calculation unit 76 obtains the three-dimensional refractive index distribution of the object to be observed between the first position and the second position by deconvolution based on the three-dimensional phase image generated by the three-dimensional phase image generation unit 75.

[0171] The interference intensity image acquiring step S71, the first complex amplitude image generating step S72, the second complex amplitude image generating step S73, the two-dimensional phase image generating step S74, the three-dimensional phase image generating step S75, and the refractive index distribution calculating step S76 in the observation method of the third embodiment perform processing that is approximately similar to the interference intensity image acquiring step S1, the first complex amplitude image generating step S2, the second complex amplitude image generating step S3, the two-dimensional phase image generating step S4, the three-dimensional phase image generating step S5, and the refractive index distribution calculating step S6 in the observation method of the first embodiment, respectively.

[0172] In a third complex amplitude image generating step S77, the third complex amplitude image generating unit 77 generates a complex amplitude image of the second position for each of the multiple light irradiation directions based on the complex amplitude image of the first position used in the second complex amplitude image generating step S73 and the three-dimensional refractive index distribution of the object to be observed between the first position and the second position calculated in the refractive index distribution calculating step S76.

[0173] In step S83 including the second complex amplitude image generating step S73, the two-dimensional phase image generating step S74, the three-dimensional phase image generating step S75 and the refractive index distribution calculating step S76, the three-dimensional refractive index distribution of the observation object between the first position and the second position is obtained based on the complex amplitude image of the first position with respect to the distance from the imaging unit 40 along the light propagation path. Each process of step S83 and the third complex amplitude image generating step S77 is repeated. This will be described with reference to Figs. 30 to 33.

[0174] FIG. 32 is a diagram for explaining the relationship between the region including the observation object and the first to Jth blocks. As shown in this diagram, the region including the observation object is divided into the first to Jth blocks in order based on the distance from the imaging unit along the light propagation path (z direction). In this diagram, J=3. The jth block among the first to Jth blocks is located at z=z j-1 From z=z j In each j-th block, the area closest to the imaging unit is z=z j-1 The position z (near end) is the first position, and the position z = z j The position (far end) is defined as the second position.

[0175] 33 is a diagram for explaining the procedure of processing in the first to Jth blocks. As shown in this diagram, for each jth block, in step S83, a complex amplitude image and a two-dimensional phase image are generated for each of a plurality of z direction positions between the first position and the second position based on the complex amplitude image at the first position, and a three-dimensional phase image is generated between the first position and the second position, and further a three-dimensional refractive index distribution is obtained. For each jth block, in a third complex amplitude image generation step S77, a complex amplitude image at the second position is generated based on the complex amplitude image at the first position and the three-dimensional refractive index distribution calculated in step S83.

[0176] The complex amplitude image at the second position of the (j+1)th block generated in the third complex amplitude image generating step S77 is used as a complex amplitude image at the first position of the next jth block, and the processes of step S83 and the third complex amplitude image generating step S77 are performed for the jth block. Once the three-dimensional refractive index distributions are obtained for each of the first to Jth blocks, they are combined to obtain the three-dimensional refractive index distribution of the entire object to be observed.

[0177] 30 and 31, in step S81 after the first complex amplitude image generating step S72, j=0 is set, and in the following step S82, the value of j is incremented by 1 to j=1, and the processes of step S83 and third complex amplitude image generating step S77 are performed for the first block. That is, for the first block closest to the imaging unit, the process of step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, and refractive index distribution calculation step S76) and third complex amplitude image generating step S77 are performed in order, with the position z=z0 (near end) closest to the imaging unit as the first position and the position z=z1 (far end) farthest from the imaging unit as the second position, based on the complex amplitude image generated in the first complex amplitude image generating step S72. Then, the process returns to step S82.

[0178] For the j-th block (here, j is 2 or more and less than J), the closest z=z j-1 The position z (near end) is the first position, and the position z = z j The position (the far end) is set as the second position, and the processes of step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, and refractive index distribution calculation step S76) and third complex amplitude image generating step S77 are performed in order. Then, the process returns to step S82.

[0179] For the J-th block, which is the final block farthest from the imaging unit, the z=z block closest to the imaging unit is selected based on the complex amplitude image generated for the (J-1)-th block in the third complex amplitude image generation step S77. J-1 The position z (near end) is the first position, and the position z = z J The position (far end) is set as the second position, and the processing of step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, and refractive index distribution calculating step S76) is performed.

[0180] The Jth block may be determined to be the final block in step S84 after step S83, and may end without proceeding to the third complex amplitude image generating step S77. Note that the Jth block may be determined to be the final block after the 3D phase image generating step S75, and may end without proceeding to the refractive index distribution calculating step S76, in which case a 3D phase image of the entire observation object is obtained.

[0181] The region including the object to be observed may be divided into two blocks based on the distance from the imaging unit along the light propagation path (z direction), in which case the above-mentioned process for the first block and the process for the Jth block in the final stage may be performed. Also, the region including the object to be observed may not be divided into a plurality of blocks, in which case each of the processes in step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, and refractive index distribution calculating step S76) and the third complex amplitude image generating step S77 may be performed only once in sequence.

[0182] Next, the third complex amplitude image generating step S77 will be described in detail. When the object to be observed is irradiated with light to obtain an interference intensity image, in each j-th block, j The light wavefront of ) propagates inside the jth block and reaches the first position (z=z j-1 ) and then propagates to the imaging unit. In the third complex amplitude image generating step S77, the first position (z=z j-1 ) is propagated backward through the jth block to the second position (z=z j That is, in the third complex amplitude image generating step S77, the optical wavefront at the first position (z=z j-1 ) and the refractive index distribution of the jth block, the second position (z=z j ) is generated. In this process, a method is used to numerically calculate the propagation of the light wavefront taking into account the refractive index distribution of the medium. Known numerical calculation methods for such propagation in non-uniform media include BPM (Beam Propagation Method) and SSNP (Split-Step Non-Paraxial). The following describes the process using BPM in the third complex amplitude image generation step S77.

[0183] Fig. 34 is a diagram explaining the processing contents of BPM. This diagram shows an arbitrary jth block. As shown in this diagram, the jth block is divided into M slices (seven slices in this diagram) (1st to Mth slices) based on the distance from the imaging unit along the light propagation path (z direction). The thickness of each slice is about the wavelength.

[0184] The thickness of each slice may be constant. Here, the thickness of each slice is set to a constant value Δz. The m-th slice among the 1st to M-th slices of the j-th block is located at the position (z j-1 +(m-1)Δz) to position (z j-1 +mΔz). The first position of the jth block (z=z j-1 ) to the second position (z=z j ), a phase change according to the refractive index distribution is applied in sequence to the first to Mth slices, and the light wavefront is propagated backward by Δz.

[0185] In addition, the thickness Δz of each slice in the processing of the third complex amplitude image generation step S77 may be different from or may be the same as the position interval when generating complex amplitude images for each of the multiple z-direction positions between the first position and the second position in the processing of the second complex amplitude image generation step S73.

[0186] The phase change o(x,y,z) imparted to the light wavefront when propagating backward through a slice of thickness Δz at position z is expressed by the following equation (25). v is the wave number of light in a vacuum. δn(x,y,z) is the refractive index distribution n(x,y,z) of the object at position z and the refractive index n of the background (medium). b and is expressed by the following equation (26). Also, cos θ is expressed by the following equation (27).

[0187]

number

[0188]

number

[0189]

number

[0190] The position of the mth slice (z=z j-1 If the complex amplitude of light at the m-th slice (z+Δz) is u(x,y,z), the complex amplitude of light at the m-th slice (z+Δz) after the light has propagated backward inside the m-th slice is expressed by the following equation (28). x ,k y ;Δz) is expressed by the following equation (29). Equation (28) is a Fourier transform of the product of the complex amplitude u(x,y,z) of light and the phase change o(x,y,z), and the result of this Fourier transform is multiplied by P(k x ,k y ;Δz) is inverse Fourier transformed to obtain the complex amplitude u(x,y,z+Δz) of the light at position (z+Δz) after propagating through a slice of thickness Δz. Δz is a function that calculates the light propagation of Δz.

[0191]

number

[0192]

number

[0193] The propagation of the light wavefront in each slice of the jth block is expressed by the following equations (30) to (32). That is, at the first position (z=z j-1 ) is the complex amplitude of light at u(x,y,z j-1 ), the complex amplitude u(x,y,z) of the light after propagating through the first slice of the jth block is j-1 +Δz) is expressed by the following equation (30). The complex amplitude of the light after propagating through the (m-1)th slice of the jth block is expressed as u(x, y, z j-1+(m-1)Δz), the complex amplitude u(x,y,z) of the light after propagating through the m-th slice of the j-th block is j-1 +mΔz) is expressed by the following equation (31). The complex amplitude of the light after propagating through the (M-1)th slice of the jth block is expressed as u(x, y, z j-1 +(M-1)Δz), the second position (z=z j ) the complex amplitude u(x,y,z j ) is expressed by the following equation (32).

[0194]

number

[0195]

number

[0196]

number

[0197] In this way, in the third complex amplitude image generating step S77, the first position (z=z j-1 ) is propagated back through the jth block slice by slice to arrive at the second position (z=z j ) optical wavefront can be obtained.

[0198] FIG. 35 is a flowchart of the third complex amplitude image generating step S77. In step S41, the position z is set to the first position of the jth block (z=z j-1) in step S42. The interaction between the complex amplitude u(x, y, z) of the light at position z and the phase change o(x, y, z) is calculated. In step S43, the wavefront of the light after the interaction is propagated a distance Δz to calculate the complex amplitude u(x, y, z+Δz) of the light at position z+Δz. In step S44, z to which Δz has been added is set as the new z. In step S44, when position z becomes the second position of the jth block (z=z j If it is determined that the position z has not yet reached the second position of the jth block (z=z j If it is determined that the jth block has reached the second position (z=z), the process of the third complex amplitude image generating step S77 is terminated. j ) is the complex amplitude at

[0199] In this embodiment as well, even if the moving observation object is a multiple scattering medium, the influence of multiple scattered light can be reduced and the observation object can be observed.

[0200] Note that an observation apparatus and an observation method using self-interference may also be used, in which case the observation apparatus does not need to include an interference optical system. [Industrial Applicability]

[0201] The observation device or observation method of the present embodiment can observe an observation object by reducing the influence of multiple scattered light even if the moving observation object is a multiple scatterer, and can be applied to, for example, a flow cytometer. Even if the observation object flowing along a flow path in a flow cytometer is a three-dimensional cell tissue such as a spheroid or an organoid, the three-dimensional cell tissue can be observed by reducing the influence of multiple scattered light.

[0202] When the observation device of this embodiment is used in combination with a three-dimensional culture analyzer in a flow cytometer, the observation device can obtain a three-dimensional refractive index distribution for each three-dimensional cell tissue flowing along the flow path, and the three-dimensional culture analyzer can determine the type based on the three-dimensional refractive index distribution. This makes it possible to determine the number and proportion of three-dimensional cell tissues of each type. Furthermore, by providing a branch flow path, the three-dimensional cell tissues can be flowed in different directions for each type, allowing separation and selection of the three-dimensional tissues.

[0203] Furthermore, when performing rotational culture (forming a three-dimensional culture by flowing cells along a flow path such as a flow cytometer), the gradual formation of a three-dimensional culture can be observed by using the observation device or observation method of this embodiment. [Explanation of symbols]

[0204] Reference Signs List 1A to 1F... Observation device, 10... Light source, 21... Beam splitter, 22, 23... Mirror, 24... Cylindrical lens, 25... Lens, 26... Cylindrical lens, 27... Beam splitter, 28... Lens, 30... Frequency shifter, 40... Imaging unit, 50... Analysis unit, 51... Interference intensity image acquisition unit, 52... First complex amplitude image generation unit, 53... Second complex amplitude image generation unit, 54... Two-dimensional phase image generation unit, 55... Three-dimensional phase image generation unit, 56... Refractive index distribution calculation unit, 57... Display unit, 58... Storage unit, 60... Analysis unit, 61... Interference intensity image acquisition unit, 62...first complex amplitude image generation unit, 63...second complex amplitude image generation unit, 64...phase conjugate calculation unit, 65...two-dimensional phase image generation unit, 66...three-dimensional phase image generation unit, 67...refractive index distribution calculation unit, 68...display unit, 69...memory unit, 70...analysis unit, 71...interference intensity image acquisition unit, 72...first complex amplitude image generation unit, 73...second complex amplitude image generation unit, 74...two-dimensional phase image generation unit, 75...three-dimensional phase image generation unit, 76...refractive index distribution calculation unit, 77...third complex amplitude image generation unit, 78...display unit, 79...memory unit.

Claims

1. an interference optical system that splits light output from a light source into object light and reference light, irradiates an observation object moving along a first direction with the object light, and combines and outputs the object light and the reference light that have passed through the observation object; a frequency shifter that is provided on an optical path of the object light or the reference light between the branching and the multiplexing of the interference optical system, and that makes the optical frequencies of the object light and the reference light different from each other by a heterodyne frequency; an imaging unit having an imaging surface that receives the object light and the reference light output from the interference optical system, capturing an interference intensity image due to interference between the object light and the reference light on the imaging surface, and outputting time series data of the interference intensity image; an analysis unit that performs analysis based on time-series data of the interference intensity image; Equipped with The interference optical system includes: an illumination optical system that condenses and illuminates the object light onto a line-shaped region extending in a second direction perpendicular to the first direction on the observation object when illuminating the object light onto the observation object; an imaging optical system that optically forms a Fourier transform image of the object of observation on the imaging plane in the first direction when guiding the object light that has passed through the object of observation to the imaging plane, and that has a conjugate positional relationship between the object of observation and the imaging plane in the second direction; Including, The object light and the reference light are made to be incident on the imaging surface so as to be coaxial with each other; the analysis unit generates complex differential interference images for a plurality of light irradiation directions and a plurality of positions when the object light is focused and irradiated onto the observation object by the irradiation optical system based on the time series data of the interference intensity image, and generates a three-dimensional phase image of the observation object based on these complex differential interference images. Observation equipment.

2. an interference optical system that splits light output from a light source into object light and reference light, irradiates an observation object moving along a first direction with the object light, and combines and outputs the object light and the reference light that have passed through the observation object; an imaging unit having an imaging surface that receives the object light and the reference light output from the interference optical system, capturing an interference intensity image due to interference between the object light and the reference light on the imaging surface, and outputting time series data of the interference intensity image; an analysis unit that performs analysis based on time-series data of the interference intensity image; Equipped with The interference optical system includes: an illumination optical system that condenses and illuminates the object light onto a line-shaped region extending in a second direction perpendicular to the first direction on the observation object when illuminating the object light onto the observation object; an imaging optical system that optically forms a Fourier transform image of the object of observation on the imaging plane in the first direction when guiding the object light that has passed through the object of observation to the imaging plane, and that has a conjugate positional relationship between the object of observation and the imaging plane in the second direction; Including, The object light and the reference light are incident on the imaging surface from different directions; the analysis unit generates complex differential interference images for a plurality of light irradiation directions and a plurality of positions when the object light is focused and irradiated onto the observation object by the irradiation optical system based on the time series data of the interference intensity image, and generates a three-dimensional phase image of the observation object based on these complex differential interference images. Observation equipment.

3. The analysis unit is an interference intensity image acquisition unit that acquires time-series data of the interference intensity image from the imaging unit; a first complex amplitude image generating unit that generates a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating unit configured to generate a complex amplitude image for each of a plurality of positions based on the complex amplitude image generated by the first complex amplitude image generating unit for each of the plurality of light irradiation directions; a two-dimensional phase image generating unit that generates a complex differential interference image for each of the plurality of positions based on the complex amplitude image for each of the plurality of light irradiation directions, and generates a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; a three-dimensional phase image generating unit that generates a three-dimensional phase image based on the two-dimensional phase images at each of the plurality of positions; Equipped with 3. The observation device according to claim 1 or 2.

4. The analysis unit is an interference intensity image acquisition unit that acquires time-series data of the interference intensity image from the imaging unit; a first complex amplitude image generating unit that generates a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating unit configured to generate a complex amplitude image for each of a plurality of positions based on the complex amplitude image generated by the first complex amplitude image generating unit for each of the plurality of light irradiation directions; a two-dimensional phase image generating unit that divides the complex amplitude images in each of the plurality of light irradiation directions into a plurality of batches for each of the plurality of positions, corrects the phases of the complex amplitude images included in each of the plurality of batches based on the light irradiation direction, and then generates a complex amplitude sum image representing a sum of the complex amplitude images after the correction, generates a complex differential interference image for each of the plurality of batches based on the complex amplitude sum image for each of the plurality of batches, and generates a two-dimensional phase image based on the complex differential interference image for each of the plurality of batches; a three-dimensional phase image generating unit that generates a three-dimensional phase image based on the two-dimensional phase images at each of the plurality of positions; Equipped with 3. The observation device according to claim 1 or 2.

5. The analysis unit is an interference intensity image acquisition unit that acquires time-series data of the interference intensity image from the imaging unit; a first complex amplitude image generating unit that generates a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating unit configured to generate a complex amplitude image for each of a plurality of positions based on the complex amplitude image for each of the plurality of light irradiation directions; a phase conjugate calculation unit that performs a phase conjugate calculation on the complex amplitude images in each of the plurality of light irradiation directions before, during, or after processing by the second complex amplitude image generation unit, to generate complex amplitude images in each of the plurality of light irradiation directions when a relationship between light irradiation and imaging of the observation object is reversed; a two-dimensional phase image generating unit that generates a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude images for each of the plurality of light irradiation directions generated by the second complex amplitude image generating unit or the phase conjugate calculation unit for each of the plurality of positions, and generates a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; a three-dimensional phase image generating unit that generates a three-dimensional phase image based on the two-dimensional phase images at each of the plurality of positions; Equipped with the two-dimensional phase image generating unit generates the two-dimensional phase image mainly based on the first phase image for positions relatively close to the imaging unit among the plurality of positions, and generates the two-dimensional phase image mainly based on the second phase image for positions relatively far from the imaging unit, when a phase image generated based on the complex amplitude image before the operation by the phase conjugate calculation unit is defined as a first phase image, and a phase image generated based on the complex amplitude image obtained by the operation by the phase conjugate calculation unit is defined as a second phase image.

3. The observation device according to claim 1 or 2.

6. The analysis unit further includes a refractive index distribution calculation unit that calculates a three-dimensional refractive index distribution of the observation object based on the three-dimensional phase image. The observation device according to any one of claims 3 to 5.

7. The analysis unit is an interference intensity image acquisition unit that acquires time-series data of the interference intensity image from the imaging unit; a first complex amplitude image generating unit that generates a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating unit configured to generate, for each of the plurality of light irradiation directions, a complex amplitude image of each of a plurality of positions between the first position and a second position based on a complex amplitude image of a first position with respect to a distance from the imaging unit along a light propagation path; a two-dimensional phase image generating unit that generates a complex differential interference image for each of the plurality of positions based on the complex amplitude image for each of the plurality of light irradiation directions, and generates a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; a three-dimensional phase image generating unit that generates a three-dimensional phase image from the first position to the second position based on the two-dimensional phase images of the respective positions; a refractive index distribution calculation unit that calculates a three-dimensional refractive index distribution of the object to be observed between the first position and the second position based on the three-dimensional phase image; a third complex amplitude image generating unit configured to generate a complex amplitude image at the second position based on the complex amplitude image at the first position and the three-dimensional refractive index distribution for each of the plurality of light irradiation directions; Equipped with based on the complex amplitude image generated by the first complex amplitude image generation unit, the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit sequentially perform processing.

3. The observation device according to claim 1 or 2.

8. the two-dimensional phase image generating unit generates the two-dimensional phase image based on a sum of the complex differential interference images in each of the plurality of light irradiation directions. The observation device according to any one of claims 3 to 7.

9. an interference step of causing the object light and the reference light to interfere with each other by an interference optical system that splits light output from a light source into object light and reference light, irradiates the object light onto an observation object moving along a first direction, and combines and outputs the object light and the reference light that have passed through the observation object; a modulation step of making the optical frequencies of the object light and the reference light different from each other by a heterodyne frequency using a frequency shifter provided on an optical path of the object light or the reference light between the branching and the multiplexing of the interference optical system; an imaging step of capturing an interference intensity image due to interference between the object light and the reference light on an imaging surface by an imaging unit having an imaging surface that receives the object light and the reference light output from the interference optical system, and outputting time series data of the interference intensity image; an analysis step of performing an analysis based on time series data of the interference intensity image; Equipped with The interference optical system includes: an illumination optical system that condenses and illuminates the object light onto a line-shaped region extending in a second direction perpendicular to the first direction on the observation object when illuminating the object light onto the observation object; an imaging optical system that optically forms a Fourier transform image of the object of observation on the imaging plane in the first direction when guiding the object light that has passed through the object of observation to the imaging plane, and that has a conjugate positional relationship between the object of observation and the imaging plane in the second direction; Including, The object light and the reference light are made to be incident on the imaging surface so as to be coaxial with each other; In the analyzing step, complex differential interference images are generated for each of a plurality of light irradiation directions and a plurality of positions when the object light is focused and irradiated onto the observation object by the irradiation optical system based on the time series data of the interference intensity image, and a three-dimensional phase image of the observation object is generated based on these complex differential interference images. Observation method.

10. an interference step of causing the object light and the reference light to interfere with each other by an interference optical system that splits light output from a light source into object light and reference light, irradiates the object light onto an observation object moving along a first direction, and combines and outputs the object light and the reference light that have passed through the observation object; an imaging step of capturing an interference intensity image due to interference between the object light and the reference light on an imaging surface by an imaging unit having an imaging surface that receives the object light and the reference light output from the interference optical system, and outputting time series data of the interference intensity image; an analysis step of performing an analysis based on time series data of the interference intensity image; Equipped with The interference optical system includes: an illumination optical system that condenses and illuminates the object light onto a line-shaped region extending in a second direction perpendicular to the first direction on the observation object when illuminating the object light onto the observation object; an imaging optical system that optically forms a Fourier transform image of the object of observation on the imaging plane in the first direction when guiding the object light that has passed through the object of observation to the imaging plane, and that has a conjugate positional relationship between the object of observation and the imaging plane in the second direction; Including, The object light and the reference light are incident on the imaging surface from different directions; In the analyzing step, complex differential interference images are generated for each of a plurality of light irradiation directions and a plurality of positions when the object light is focused and irradiated onto the observation object by the irradiation optical system based on the time series data of the interference intensity image, and a three-dimensional phase image of the observation object is generated based on these complex differential interference images. Observation method.

11. The analyzing step includes: an interference intensity image acquiring step of acquiring time series data of the interference intensity image from the imaging unit; a first complex amplitude image generating step of generating a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating step of generating a complex amplitude image for each of a plurality of positions based on the complex amplitude image generated by the first complex amplitude image generating step for each of the plurality of light irradiation directions; a two-dimensional phase image generating step of generating a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude image for each of the plurality of light irradiation directions for each of the plurality of positions, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; a three-dimensional phase image generating step of generating a three-dimensional phase image based on the two-dimensional phase images of the respective positions; Equipped with The observation method according to claim 9 or 10.

12. The analyzing step includes: an interference intensity image acquiring step of acquiring time series data of the interference intensity image from the imaging unit; a first complex amplitude image generating step of generating a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating step of generating a complex amplitude image for each of a plurality of positions based on the complex amplitude image generated by the first complex amplitude image generating step for each of the plurality of light irradiation directions; a two-dimensional phase image generating step of dividing the complex amplitude images in each of the plurality of light irradiation directions into a plurality of batches for each of the plurality of positions, correcting the phases of the complex amplitude images included in each of the plurality of batches based on the light irradiation direction, and then generating a complex amplitude sum image representing a sum of the complex amplitude images after the correction, generating a complex differential interference image for each of the plurality of batches based on the complex amplitude sum image for each of the plurality of batches, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of batches; a three-dimensional phase image generating step of generating a three-dimensional phase image based on the two-dimensional phase images of the respective positions; Equipped with The observation method according to claim 9 or 10.

13. The analyzing step includes: an interference intensity image acquiring step of acquiring time series data of the interference intensity image from the imaging unit; a first complex amplitude image generating step of generating a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating step of generating a complex amplitude image for each of a plurality of positions based on the complex amplitude image for each of the plurality of light irradiation directions; a phase conjugate calculation step of performing a phase conjugate calculation on the complex amplitude images in each of the plurality of light irradiation directions before, during or after the processing in the second complex amplitude image generation step, to generate complex amplitude images in each of the plurality of light irradiation directions when a relationship between light irradiation and imaging of the observation object is reversed; a two-dimensional phase image generating step of generating a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude images for each of the plurality of light irradiation directions generated by the second complex amplitude image generating step or the phase conjugate calculation step for each of the plurality of positions, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; a three-dimensional phase image generating step of generating a three-dimensional phase image based on the two-dimensional phase images of the respective positions; Equipped with the two-dimensional phase image generating step generates the two-dimensional phase image mainly based on the first phase image for positions relatively close to the imaging unit among the plurality of positions, and generates the two-dimensional phase image mainly based on the second phase image for positions relatively far from the imaging unit, when a phase image generated based on the complex amplitude image before the operation in the phase conjugate operation step is defined as a first phase image, and a phase image generated based on the complex amplitude image obtained by the operation in the phase conjugate operation step is defined as a second phase image, The observation method according to claim 9 or 10.

14. The analyzing step further includes a refractive index distribution calculating step of calculating a three-dimensional refractive index distribution of the observation object based on the three-dimensional phase image. The observation method according to any one of claims 11 to 13.

15. The analyzing step includes: an interference intensity image acquiring step of acquiring time series data of the interference intensity image from the imaging unit; a first complex amplitude image generating step of generating a complex amplitude image for each of the plurality of light irradiation directions based on time series data of the interference intensity image; a second complex amplitude image generating step of generating, for each of the plurality of light irradiation directions, a complex amplitude image of each of a plurality of positions between the first position and a second position based on a complex amplitude image of a first position in relation to a distance from the imaging unit along a light propagation path; a two-dimensional phase image generating step of generating a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude image for each of the plurality of light irradiation directions for each of the plurality of positions, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; a three-dimensional phase image generating step of generating a three-dimensional phase image between the first position and the second position based on the two-dimensional phase images of the respective positions; a refractive index distribution calculation step of calculating a three-dimensional refractive index distribution of the object to be observed between the first position and the second position based on the three-dimensional phase image; a third complex amplitude image generating step of generating a complex amplitude image at the second position based on the complex amplitude image at the first position and the three-dimensional refractive index distribution for each of the plurality of light irradiation directions; Equipped with performing the processes of the second complex amplitude image generating step, the two-dimensional phase image generating step, the three-dimensional phase image generating step, the refractive index distribution calculating step, and the third complex amplitude image generating step in sequence based on the complex amplitude image generated by the first complex amplitude image generating step; The observation method according to claim 9 or 10.

16. In the two-dimensional phase image generating step, the two-dimensional phase image is generated based on a sum of the complex differential interference images in each of the plurality of light irradiation directions. The observation method according to any one of claims 11 to 15.

17. A program for causing a computer to execute each step of the observation method according to any one of claims 9 to 16.

18. A computer-readable recording medium having the program according to claim 17 recorded thereon.

Images