Observation device and observation method

Abstract

To provide an observation device that allows observation of an object to be observed while reducing the influence of multiple scattered light.SOLUTION: An observation device 1A comprises a light source 11, a mirror 22, a condenser lens 24, an objective lens 25, a beam splitter 41, an imaging unit 43, and an analysis unit 70. The analysis unit 70 includes an interference intensity image acquisition unit 71, a first complex amplitude image creation unit 72, a second complex amplitude image creation unit 73, a two-dimensional phase image creation unit 74, a three-dimensional phase image creation unit 75, a refractive index distribution calculation unit 76, and a third complex amplitude image creation unit 77. The analysis unit 70 changes the direction of a reflection surface of the mirror 22 to irradiate an object to be observed S with light along a plurality of light irradiation directions, acquires, from the imaging unit 43, an interference intensity image for each of the plurality of light irradiation directions, and performs predetermined processing based on these interference intensity images to determine the three-dimensional refractive index distribution of the object to be observed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an observation device and an observation method. [Background technology]

[0002] In recent years, advances have been made in the technology for creating three-dimensional cellular tissues known as spheroids and organoids. Research is also progressing on the application of these three-dimensional cellular tissues to drug discovery, regenerative medicine, and other fields. These three-dimensional cellular tissues are optically transparent, multiple-scattering bodies. A wide variety of imaging techniques have been proposed for imaging such optically transparent scattering bodies. Among these, imaging techniques that use fluorescent probes include confocal microscopy, multiphoton microscopy, and light-sheet microscopy. Meanwhile, optical coherence tomography (OCT) is a well-known non-staining, non-invasive imaging technique that does not use fluorescent probes.

[0003] Although non-staining and non-invasive imaging is often desirable for observational objects such as spheroids and organoids, there have been few reports of OCT being used to image these objects. This is likely due to the low resolution of OCT imaging and the difficulty in interpreting the signals obtained by OCT imaging. Therefore, it can be said that at present, there is no established gold-standard imaging technology for non-staining 3D cell tissues.

[0004] Quantitative phase imaging (QPI) is a technique that can image the optical path length of an object non-invasively and without staining. QPI is being increasingly applied in the biological field because it can obtain physical information, such as the optical path length of an object (e.g., a cell). Images acquired by QPI can be used to generate other types of images, such as differential interference contrast images and phase-contrast microscopy images. QPI is a technique that can obtain images with relatively high information content, and is expected to be applicable to higher-content analyses than those using conventional bright-field images. Furthermore, with the recent improvement in image recognition accuracy through machine learning, high-content analyses using non-staining imaging techniques have been actively researched, and non-staining imaging of multiple scattering objects is expected to play an important role in the future. However, since the images acquired by QPI are merely two-dimensional projections of the optical path length, they cannot capture true three-dimensional structures.

[0005] Optical Diffraction Tomography (ODT), described in Patent Document 1, is also known as a technology capable of imaging the optical path length of an object of observation non-staining and non-invasively. ODT is an extension of QPI into a technology capable of 3D imaging, and is capable of realizing 3D refractive index tomography of the object of observation. Cell observation using ODT makes it possible to identify organelles such as cell nuclei and mitochondria, and also enables tracking of 3D morphological changes, and is expected to enable even higher-level analysis than QPI. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2017-219826 Summary of the Invention [Problem to be solved by the invention]

[0007] However, while conventional ODT can be applied to the observation of cells consisting of a few cells, it is difficult to apply it to the observation of multiple scattering objects such as the above-mentioned three-dimensional cell tissues, because with conventional ODT, when there is a lot of multiple scattered light generated in the object to be observed, the influence of the multiple scattered light appears significantly in the acquired image.

[0008] Light scattering is a phenomenon in which the direction of light is changed by interacting with an object. In particular, when the spatial non-uniformity of the object's refractive index increases, light interacts with the object multiple times while passing through it. Light that has interacted with the object multiple times in this way is called multiple scattered light. In contrast, light that has interacted with the object only once is called single scattered light. Multiple scattered light is known to cause increased speckle and a worsening of the single-scattering to multi-scattering ratio (SMR), making it an obstacle to measurement.

[0009] Speckle occurs when light that is temporally and spatially coherent is subject to interference from multiple scattered light, resulting in large spatial variations in intensity or phase. Speckle can be suppressed by using a light source that outputs temporally or spatially incoherent light. For example, conventional bright-field microscopes, such as phase-contrast microscopes, obtain speckle-free images by using spatially and temporally incoherent light sources, such as halogen lamps or light-emitting diodes.

[0010] The deterioration of SMR occurs when multiple scattered light becomes dominant over single scattered light, and the single scattered light is buried in the multiple scattered light. As the size of the observed object increases and the observation depth increases, the single scattered light component exponentially decreases, while in contrast, the multiple scattered light component increases. Single scattered light is easy to use for measuring the structure of an object because its scattering direction has a direct correspondence with the structure of the object. On the other hand, the relationship between multiple scattered light and the structure of the object is complex, making it difficult to extract information about the object's structure. Therefore, imaging techniques using single scattered light are known to fail when single scattered light is buried in the multiple scattered light (i.e., SMR deteriorates).

[0011] Suppression of SMR degradation can be achieved by a technique called gating, which selectively detects single-scattered light among single-scattered and multiply-scattered light. Gating suppresses multiply-scattered light, so it is possible to suppress speckle while suppressing SMR degradation. Gating is achieved using degrees of freedom such as space, time, and polarization. Confocal microscopy is an example of spatial gating. OCT is an example of both temporal and spatial gating.

[0012] Because conventional ODT does not eliminate the effects of multiple scattered light, when the object being observed generates a large amount of multiple scattered light, the speckle in the acquired image increases and the SMR deteriorates. Therefore, although conventional ODT can be applied to the observation of cells consisting of a few cells, which generate little multiple scattered light, it is difficult to apply it to the observation of multiple scattering objects such as three-dimensional cellular tissues, which generate a large amount of multiple scattered light.

[0013] The present invention has been made to solve the above problems, and aims to provide an observation device and an observation method that can observe an object by reducing the influence of multiple scattered light, even when the object is a multiple scattering object. [Means for solving the problem]

[0014] The observation device of the present invention includes: (1) an interference intensity image acquisition unit that acquires an interference intensity image of a reference position for each of a plurality of light irradiation directions from an imaging unit that captures an interference intensity image of a reference position resulting from interference between a reference light and light that has been irradiated onto an observation object along each of a plurality of light irradiation directions and passed through the observation object along the plurality of light irradiation directions; (2) a first complex amplitude image generation unit that generates a complex amplitude image for each of the plurality of light irradiation directions based on the interference intensity image; (3) a second complex amplitude image generation unit that generates a complex amplitude image for each of a plurality of positions between a first position and a second position based on the complex amplitude image of the first position in relation to the distance from the imaging unit along the light propagation path for each of the plurality of light irradiation directions; (4) a two-dimensional phase image generation unit that generates a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude image for each of the plurality of light irradiation directions for each of the plurality of positions, and generates a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; and (5) a three-dimensional phase image generation unit that generates a three-dimensional phase image for the region between the first position and the second position based on the two-dimensional phase image for each of the plurality of positions. The observation device of the present invention includes a refractive index distribution calculation unit that calculates the three-dimensional refractive index distribution of the object to be observed between the first position and the second position based on the three-dimensional phase image, and (7) a third complex amplitude image generation unit that generates a complex amplitude image of the second position based on the complex amplitude image of the first position and the three-dimensional refractive index distribution for each of a plurality of light irradiation directions.The observation device of the present invention sequentially performs processing in the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit based on the complex amplitude image generated by the first complex amplitude image generation unit.

[0015] In one aspect of the present invention, the observation device (a) sequentially divides an area including an object to be observed into first and second blocks based on the distance from the imaging unit along the light propagation path; (b) for the first block, which is closest to the imaging unit, based on the complex amplitude image generated by the first complex amplitude image generation unit, performs processing in sequence by the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit, with the near end of the first block being the first position and the far end of the first block being the second position; and (c) for the second block, which is the final block farthest from the imaging unit, performs processing in sequence by the second complex amplitude image generation unit, the two-dimensional phase image generation unit, and the three-dimensional phase image generation unit, based on the complex amplitude image generated for the first block by the third complex amplitude image generation unit, with the near end of the second block being the first position and the far end of the second block being the second position.

[0016] In one aspect of the present invention, the observation device (a) divides an area including an observation object into first to Jth blocks (J is 3 or more) in order based on the distance from the imaging unit along the light propagation path, (b) for the first block closest to the imaging unit, based on the complex amplitude image generated by the first complex amplitude image generation unit, sets the near end of the first block as a first position and the far end of the first block as a second position, and sequentially performs processing by the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit, (c) (d) For the jth block (j is 2 or more and less than J), based on the complex amplitude image generated for the (j-1)th block by the third complex amplitude image generation unit, the near end of the jth block is set as the first position and the far end of the jth block is set as the second position, and the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit perform processing in sequence; and (d) for the Jth block, which is the final block farthest from the imaging unit, based on the complex amplitude image generated for the (J-1)th block by the third complex amplitude image generation unit, the near end of the Jth block is set as the first position and the far end of the Jth block is set as the second position, and the second complex amplitude image generation unit, the two-dimensional phase image generation unit, and the three-dimensional phase image generation unit perform processing in sequence.

[0017] In one aspect of the present invention, the observation device preferably performs processing by the refractive index distribution calculation unit after processing by the three-dimensional phase image generation unit for the final stage block.

[0018] In one aspect of the present invention, it is preferable that the two-dimensional phase image generating unit generates a complex differential interference image for each of a plurality of light irradiation directions for each of a plurality of different shear directions on the image based on the complex amplitude image for each of the plurality of light irradiation directions, and generates a two-dimensional phase image based on the complex differential interference images for each of the plurality of shear directions and the plurality of light irradiation directions.

[0019] In one aspect of the present invention, the two-dimensional phase image generating section preferably generates the two-dimensional phase image based on the sum of complex differential interference images in each of a plurality of light irradiation directions.

[0020] The observation method of the present invention includes: (1) an interference intensity image acquisition step of acquiring an interference intensity image of a reference position for each of a plurality of light irradiation directions from an imaging unit that has captured an interference intensity image of a reference position resulting from interference between light irradiated onto an object to be observed along each of a plurality of light irradiation directions and passing through the object to be observed and a reference light; (2) a first complex amplitude image generation step of generating a complex amplitude image for each of the plurality of light irradiation directions based on the interference intensity image; (3) a second complex amplitude image generation step of generating a complex amplitude image for each of a plurality of positions between a first position and a second position based on the complex amplitude image of a first position in relation to a distance from the imaging unit along a light propagation path for each of the plurality of light irradiation directions; (4) a two-dimensional phase image generation step of generating a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude image for each of the plurality of light irradiation directions for each of the plurality of positions, and generating a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; (5) a three-dimensional phase image generation step of generating a three-dimensional phase image for the region between the first position and the second position based on the two-dimensional phase image for each of the plurality of positions; and (6) The observation method of the present invention includes a refractive index distribution calculation step of calculating a three-dimensional refractive index distribution of the object to be observed between the first position and the second position based on the three-dimensional phase image, and (7) a third complex amplitude image generation step of generating a complex amplitude image of the second position based on the complex amplitude image of the first position and the three-dimensional refractive index distribution for each of a plurality of light irradiation directions.The observation method of the present invention then sequentially performs the processes of the second complex amplitude image generation step, the two-dimensional phase image generation step, the three-dimensional phase image generation step, the refractive index distribution calculation step, and the third complex amplitude image generation step based on the complex amplitude image generated in the first complex amplitude image generation step.

[0021] In one aspect of the present invention, the observation method includes: (a) dividing an area including an object to be observed into a first block and a second block in order based on the distance from an imaging unit along a light propagation path; (b) for the first block, which is closest to the imaging unit, based on the complex amplitude image generated in the first complex amplitude image generation step, performing processes in the order of a second complex amplitude image generation step, a two-dimensional phase image generation step, a three-dimensional phase image generation step, a refractive index distribution calculation step, and a third complex amplitude image generation step, with the near end of the first block being set as a first position and the far end of the first block being set as a second position; and (c) for the second block, which is the final block farthest from the imaging unit, performing processes in the order of a second complex amplitude image generation step, a two-dimensional phase image generation step, and a three-dimensional phase image generation step, with the near end of the second block being set as a first position and the far end of the second block being set as a second position, based on the complex amplitude image generated for the first block in the third complex amplitude image generation step.

[0022] In one aspect of the present invention, the observation method includes: (a) dividing an area including an object to be observed into first to Jth blocks (J is 3 or more) in order based on the distance from an imaging unit along a light propagation path; (b) for the first block closest to the imaging unit, based on the complex amplitude image generated in the first complex amplitude image generation step, setting the near end of the first block as a first position and the far end of the first block as a second position, sequentially performing the processes of a second complex amplitude image generation step, a two-dimensional phase image generation step, a three-dimensional phase image generation step, a refractive index distribution calculation step, and a third complex amplitude image generation step; and (c) (d) for the jth block (j is 2 or more and less than J), based on the complex amplitude image generated for the (j-1)th block in the third complex amplitude image generation step, the near end of the jth block is set as the first position and the far end of the jth block is set as the second position, and the processes of the second complex amplitude image generation step, the two-dimensional phase image generation step, the three-dimensional phase image generation step, the refractive index distribution calculation step, and the third complex amplitude image generation step are performed in sequence; and (d) for the Jth block, which is the final block farthest from the imaging unit, based on the complex amplitude image generated for the (J-1)th block in the third complex amplitude image generation step, the near end of the Jth block is set as the first position and the far end of the Jth block is set as the second position, and the processes of the second complex amplitude image generation step, the two-dimensional phase image generation step, and the three-dimensional phase image generation step are performed in sequence.

[0023] In one aspect of the present invention, the observation method preferably performs a refractive index distribution calculation step after a three-dimensional phase image generation step for the final stage block.

[0024] In one aspect of the present invention, it is preferable that in the two-dimensional phase image generating step, a complex differential interference contrast image for each of a plurality of light irradiation directions is generated for each of a plurality of different shear directions on the image based on a complex amplitude image for each of a plurality of light irradiation directions, and a two-dimensional phase image is generated based on the complex differential interference contrast images for each of the plurality of shear directions and the plurality of light irradiation directions.

[0025] In one aspect of the present invention, the two-dimensional phase image generating step preferably generates the two-dimensional phase image based on a sum of complex differential interference images in each of a plurality of light irradiation directions.

[0026] The program of the present invention causes a computer to execute each step of the observation method of the present invention. The recording medium of the present invention is a computer-readable medium having the program of the present invention recorded thereon. [Effects of the Invention]

[0027] According to the present invention, even when the object to be observed is a multiple scattering medium, the object to be observed can be observed with the influence of multiple scattered light reduced. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram showing the configuration of an observation device 1A. [Figure 2] FIG. 2 is a diagram showing the configuration of the observation device 1B. [Figure 3] FIG. 3 is a diagram showing the configuration of the observation device 1C. [Figure 4] FIG. 4 is a flowchart of the observation method. [Figure 5] FIG. 5 is a flowchart of the observation method. [Figure 6] 6(a) to 6(c) are diagrams showing an example of scanning the light irradiation direction onto the observation object S in the interference intensity image acquisition step S71. [Figure 7] FIG. 7 is a diagram illustrating the kernel function g. [Figure 8] FIG. 8 is a diagram illustrating the relationship between the region including the observation object and the first to Jth blocks. [Figure 9] FIG. 9 is a diagram for explaining the procedure of processing in the first to Jth blocks. [Figure 10] FIG. 10 is a flowchart of the two-dimensional phase image generating step S74. [Figure 11] FIG. 11 is a diagram illustrating the kernel function. [Figure 12] FIG. 12 is a diagram for explaining the processing contents of BPM. [Figure 13] FIG. 13 is a flowchart of the third complex amplitude image generating step S77. [Figure 14] FIG. 14 is an interference intensity image (at normal irradiation) acquired in interference intensity image acquisition step S71. [Figure 15] FIG. 15 is a complex amplitude image (real part, z=z0) generated based on the interference intensity image (FIG. 14) in the first complex amplitude image generating step S72. [Figure 16] FIG. 16 shows a complex amplitude image (real part, z=zn) generated based on the complex amplitude image (FIG. 15) in second complex amplitude image generating step S73. [Figure 17] FIG. 17 shows a complex differential interference image (imaginary parts for both the x-direction shear and the y-direction shear) generated based on the complex amplitude image (FIG. 16) in step S21 of the two-dimensional phase image generating step S74. [Figure 18] FIG. 18 shows differential phase images (x-direction shear and y-direction shear) generated based on the complex differential interference image (FIG. 17) in step S22 of the two-dimensional phase image generating step S74. [Figure 19] FIG. 19 shows the refractive index distribution of the first block generated based on the differential phase image (FIG. 18) in the refractive index distribution calculation step S76. [Figure 20] FIG. 20 shows a complex amplitude image after a phase change o(x, y, z) is applied to the complex amplitude image (FIG. 15) in step S42 of the third complex amplitude image generating step S77. [Figure 21] FIG. 21 shows a complex amplitude image after the complex amplitude image (FIG. 20) is propagated by Δz in step S44 of the third complex amplitude image generating step S77. [Figure 22] FIG. 22 is a complex amplitude image of the second position of the first block generated in the third complex amplitude image generating step S77. [Figure 23]FIG. 23(a) is a diagram showing the phase distribution of the optical wavefront at the first position (z=z0) of the first block. FIG. 23(b) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z1) of the first block. FIG. 23(c) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z2) of the second block. FIG. 23(d) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z3) of the third block. FIG. 23(e) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z4) of the fourth block. FIG. 23(f) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z5) of the fifth block. [Figure 24] Fig. 24(a) is an MIP image in the y direction of the refractive index distribution obtained in the comparative example, and Fig. 24(b) is an MIP image in the y direction of the refractive index distribution obtained in the example. [Figure 25] Fig. 25(a) is an MIP image in the z direction of the refractive index distribution obtained in the comparative example, and Fig. 25(b) is an MIP image in the z direction of the refractive index distribution obtained in the example. [Figure 26] Fig. 26(a) is an image of the refractive index distribution (z=22 µm) obtained in the comparative example, and Fig. 26(b) is an image of the refractive index distribution (z=22 µm) obtained in the example. [Figure 27] Fig. 27(a) is an image of the refractive index distribution (z=45.2 µm) obtained in the comparative example, and Fig. 27(b) is an image of the refractive index distribution (z=45.2 µm) obtained in the example. [Figure 28] Fig. 28(a) is an image of the refractive index distribution (z=68.8 μm) obtained in the comparative example, and Fig. 28(b) is an image of the refractive index distribution (z=68.8 μm) obtained in the example. [Figure 29] FIG. 29 is a diagram showing the configuration of the observation device 1D. DETAILED DESCRIPTION OF THE INVENTION

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate explanations will be omitted. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0030] 1 is a diagram showing the configuration of an observation device 1 A. This observation device 1 A includes a light source 11, a lens 12, a lens 21, a mirror 22, a lens 23, a condenser lens 24, an objective lens 25, a beam splitter 41, a lens 42, an imaging unit 43, and an analyzing unit 70.

[0031] The light source 11 outputs spatially and temporally coherent light, and is preferably a laser light source. The lens 12 is optically connected to the light source 11 and focuses the light output from the light source 11 onto a light input end 13 of an optical fiber 14, causing the light to be incident on the light input end 13. The optical fiber 14 guides the light incident on the light input end 13 by the lens 12 to a fiber coupler 15. The fiber coupler 15 couples light between the optical fiber 14 and optical fibers 16 and 17. The light guided by the optical fiber 14 branches into two, one branched light being guided by the optical fiber 16 and the other branched light being guided by the optical fiber 17. The light guided by the optical fiber 16 is emitted as diverging light from a light output end 18. The light guided by the optical fiber 17 is emitted as diverging light from a light output end 19.

[0032] The lens 21 is optically connected to the light emitting end 18 and collimates the light output from the light emitting end 18 as divergent light. The mirror 22 is optically connected to the lens 21 and reflects the light reaching it from the lens 21 to the lens 23. The orientation of the reflective surface of the mirror 22 is variable. The lens 23 is optically connected to the mirror 22. The condenser lens 24 is optically connected to the lens 23. The lens 23 and the condenser lens 24 preferably constitute a 4f optical system. The lens 23 and the condenser lens 24 irradiate the observation object S with light from a light irradiation direction that corresponds to the orientation of the reflective surface of the mirror 22. The objective lens 25 is optically connected to the condenser lens 24. The observation object S is disposed between the objective lens 25 and the condenser lens 24. The objective lens 25 receives the light (object light) output from the condenser lens 24 and passing through the observation object S, and outputs the light to the beam splitter 41.

[0033] The beam splitter 41 is optically connected to the objective lens 25 and also to the light output end 19. The beam splitter 41 combines light (object light) output from the objective lens 25 and reaching it with light (reference light) output from the light output end 19 and reaching it, and outputs both lights to the lens 42. The lens 42 is optically connected to the beam splitter 41, and collimates the object light and reference light arriving from the beam splitter 41, and outputs them to the imaging unit 43. The imaging unit 43 is optically connected to the lens 42, and captures an interference fringe image (interference intensity image) resulting from interference between the object light and reference light arriving from the lens 42. The incident direction of the reference light on the imaging surface of the imaging unit 43 is tilted relative to the incident direction of the object light. The position where the object light and reference light are combined by the beam splitter 41 may be after the imaging lens, but considering the effects of aberration, it is preferable to position it between the objective lens 25 and lens 42 as shown in the figure.

[0034] The analysis unit 70 is electrically connected to the imaging unit 43 and receives an interference intensity image captured by the imaging unit 43. The analysis unit 70 processes the received interference intensity image to calculate a three-dimensional refractive index distribution of the observation object S. The analysis unit 70 may be a computer. The analysis unit 70 includes an interference intensity image acquisition unit 71, a first complex amplitude image generation unit 72, a second complex amplitude image generation unit 73, a two-dimensional phase image generation unit 74, a three-dimensional phase image generation unit 75, a refractive index distribution calculation unit 76, a third complex amplitude image generation unit 77, a display unit 78, and a storage unit 79.

[0035] The interference intensity image acquisition unit 71 irradiates the observation object S with light along each of a plurality of light irradiation directions by changing the orientation of the reflecting surface of the mirror 22. The interference intensity image acquisition unit 71 also acquires an interference intensity image at a reference position for each of the plurality of light irradiation directions from the imaging unit 43. The interference intensity image acquisition unit 71 includes a CPU, and has an output port that outputs a control signal for changing the orientation of the reflecting surface of the mirror 22, and also has an input port that inputs the interference intensity image from the imaging unit 43. There is no need to move the objective lens 25 in the optical axis direction. The reference position is an image plane position that is conjugate with the imaging plane of the imaging unit 43.

[0036] The first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 perform processing based on the interference intensity image and include processing devices such as a CPU, DSP, or FPGA. The display unit 78 displays the image to be processed, the image being processed, and the image after processing, and includes, for example, an LCD display. The memory unit 79 stores data of various images and includes, for example, a hard disk drive, flash memory, RAM, ROM, etc. The first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, the third complex amplitude image generation unit 77, and the memory unit 79 may be configured using cloud computing.

[0037] The storage unit 79 also stores programs for causing the interference intensity image acquisition unit 71, the first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 to execute each process. These programs may be stored in the storage unit 79 at the time of manufacture or shipment of the observation device 1A, or may be acquired via a communication line after shipment and stored in the storage unit 79, or may be stored in the storage unit 79 after being recorded on a computer-readable recording medium 2. The recording medium 2 may be any medium such as a flexible disk, a CD-ROM, a DVD-ROM, a BD-ROM, or a USB memory.

[0038] The details of the processing of the interference intensity image acquisition unit 71, the first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 will be described later.

[0039] Fig. 2 is a diagram showing the configuration of observation apparatus 1B. Observation apparatus 1B shown in Fig. 2 includes a lens 31, a mirror 32, a lens 34, and the like in addition to the configuration of observation apparatus 1A shown in Fig. 1.

[0040] The lens 31 is optically connected to the light emitting end 19 and collimates the light (reference light) output as divergent light from the light emitting end 19. The mirror 32 is optically connected to the lens 31 and reflects the light arriving from the lens 31 to the lens 34. The lens 34 is optically connected to the mirror 32 and outputs the light arriving from the mirror 32 to the beam splitter 41. The light output from the lens 34 is temporarily condensed before reaching the beam splitter 41 and then input to the beam splitter 41 as divergent light. The beam splitter 41 combines the light (object light) output from the objective lens 25 and arriving thereat and the light (reference light) output from the lens 34 and arriving thereat, and outputs both light beams coaxially to the lens 42. The imaging unit 43 captures an interference fringe image (interference intensity image) resulting from interference between the object light and the reference light arriving from the lens 42. The direction of incidence of the reference light on the imaging surface of the imaging unit 43 is parallel to the direction of incidence of the object light.

[0041] The driver 33 moves the mirror 32 in a direction perpendicular to the reflecting surface of the mirror 32. The driver 33 is, for example, a piezoelectric actuator. This movement of the mirror 32 changes the difference in optical path length (phase difference) between the object light and the reference light from the optical branching in the fiber coupler 15 to the combination in the beam splitter 41. If this optical path length difference differs, the interference intensity image captured by the imaging unit 43 also differs.

[0042] The observation device is not limited to the configuration examples of Figures 1 and 2, and various modifications are possible. In the configurations of observation device 1A (Figure 1) and observation device 1B (Figure 2), the object light is light that has passed through the observation object S, but light reflected by the observation object S may also be the object light as in the configuration of observation device 1C (Figure 3) described below.

[0043] 3 is a diagram showing the configuration of observation device 1C. Observation device 1C includes a light source 11, a lens 12, a lens 21, a mirror 22, a lens 23, an objective lens 25, a beam splitter 41, a lens 42, an imaging unit 43, and an analyzing unit 70. The following mainly describes the differences from observation device 1A (FIG. 1).

[0044] The lens 21 is optically connected to the light emitting end 18 of the optical fiber 16 and collimates the light output from the light emitting end 18 as divergent light. The mirror 22 is optically connected to the lens 21 and reflects the light reaching it from the lens 21 to the lens 23. The orientation of the reflective surface of the mirror 22 is variable. The lens 23 is optically connected to the mirror 22. The objective lens 25 is optically connected to the lens 23. A beam splitter 41 is disposed between the lens 23 and the objective lens 25. The lens 23 and the objective lens 25 preferably constitute a 4f optical system. The lens 23 and the objective lens 25 irradiate light onto the observation object S from a light irradiation direction according to the orientation of the reflective surface of the mirror 22. The objective lens 25 receives light reflected by the observation object S (object light) and outputs the light to the beam splitter 41.

[0045] The beam splitter 41 is optically connected to the objective lens 25 and also to the light emitting end 19 of the optical fiber 17. The beam splitter 41 combines light (object light) output from the objective lens 25 and reaching it with light (reference light) output from the light emitting end 19 and reaching it, and outputs both lights to the lens 42. The lens 42 is optically connected to the beam splitter 41, and collimates the object light and reference light arriving from the beam splitter 41, and outputs them to the imaging unit 43. The imaging unit 43 is optically connected to the lens 42, and captures an interference fringe image (interference intensity image) resulting from interference between the object light and reference light arriving from the lens 42. The incident direction of the reference light on the imaging surface of the imaging unit 43 is tilted relative to the incident direction of the object light. The position where the object light and reference light are combined by the beam splitter 41 may be after the imaging lens, but considering the effects of aberration, it is preferable to position it between the objective lens 25 and lens 42 as shown in the figure.

[0046] In the configuration of the observation device 1C (FIG. 3), a mechanism for changing the optical path length of the reference light (lens 31, mirror 32, drive unit 33, and lens 34 in FIG. 2) may be provided, similarly to the observation device 1B (FIG. 2), to change the difference in the optical path lengths (phase difference) of the object light and the reference light from the light branching in the fiber coupler 15 to the combination in the beam splitter 41. In this case, the incident direction of the reference light may be parallel to the incident direction of the object light on the imaging surface of the imaging unit 43.

[0047] 4 and 5 are flowcharts of the observation method. Fig. 5 shows a part of the flowchart of Fig. 4. This observation method can be used with any of the observation apparatus 1A (Fig. 1), 1B (Fig. 2), and 1C (Fig. 3). This observation method includes an interference intensity image acquisition step S71, a first complex amplitude image generation step S72, a second complex amplitude image generation step S73, a two-dimensional phase image generation step S74, a three-dimensional phase image generation step S75, a refractive index distribution calculation step S76, and a third complex amplitude image generation step S77.

[0048] The processing of interference intensity image acquisition step S71 is performed by the interference intensity image acquisition unit 71. The processing of first complex amplitude image generation step S72 is performed by the first complex amplitude image generation unit 72. The processing of second complex amplitude image generation step S73 is performed by the second complex amplitude image generation unit 73. The processing of two-dimensional phase image generation step S74 is performed by the two-dimensional phase image generation unit 74. The processing of three-dimensional phase image generation step S75 is performed by the three-dimensional phase image generation unit 75. The processing of refractive index distribution calculation step S76 is performed by the refractive index distribution calculation unit 76. The processing of third complex amplitude image generation step S77 is performed by the third complex amplitude image generation unit 77.

[0049] In the interference intensity image acquisition step S71, the interference intensity image acquisition unit 71 irradiates the observation object S with light along each of a plurality of light irradiation directions by changing the orientation of the reflecting surface of the mirror 22. Then, the interference intensity image acquisition unit 71 acquires, from the imaging unit 43, an interference intensity image at a reference position for each of the plurality of light irradiation directions.

[0050] For convenience of explanation, an xyz Cartesian coordinate system is shown in each of FIGS. 1, 2, and 3. The z axis is parallel to the optical axis of the objective lens 25. The reference position is the image plane position that is conjugate with the imaging plane of the imaging unit 43. This position is defined as z=0. The direction of light irradiation onto the observation object S is determined by the wave vector (k x ,k y ,k z ) k x and k y It can be expressed as:

[0051] 6(a) to 6(c) are diagrams showing an example of scanning the light irradiation direction onto the observation object S in the interference intensity image acquisition step S71. In these diagrams, the horizontal axis represents k x The vertical axis is k y And k x k y The position of each circle on the plane represents the light irradiation direction. The light irradiation direction is scanned as shown in Figure 6(a). x k y They may be arranged in a rectangular grid on a plane, or as shown in FIG. 6(b), k x k y The sensors may be arranged on the circumference of a plurality of concentric circles in a plane, or may be arranged in a k x k y The scanning direction of the light may be a spirally arranged plane. In either case, scanning in the light irradiation direction is possible as long as the NA of the condenser lens 24 in the configurations of Figures 1 and 2 or the objective lens 25 in the configuration of Figure 3 allows. Either raster scanning or random scanning is possible. In the case of raster scanning, a return scan may or may not be performed.

[0052] In the first complex amplitude image generating step S72, the first complex amplitude image generating unit 72 generates a complex amplitude image based on the interference intensity image acquired by the interference intensity image acquiring unit 71 for each of the multiple light irradiation directions. In the case of the observation device 1A (FIG. 1) and the observation device 1C (FIG. 3), the first complex amplitude image generating unit 72 can generate a complex amplitude image based on one interference intensity image by the Fourier fringe analysis method. In the case of the observation device 1B (FIG. 2), the first complex amplitude image generating unit 72 can generate a complex amplitude image based on three or more interference intensity images in which the optical path length difference (phase difference) between the object light and the reference light is different from one another by the phase shift method. The complex amplitude image generated in the first complex amplitude image generating step S72 may be of the same reference position as the interference intensity image, or may be of another position generated based on the complex amplitude image of the reference position.

[0053] In the second complex amplitude image generating step S73, the second complex amplitude image generating unit 73 generates complex amplitude images for each of a plurality of z-direction positions between the first position and the second position based on the complex amplitude image at the first position in relation to the distance from the imaging unit 43 along the light propagation path for each of a plurality of light irradiation directions. The two-dimensional Fourier transform of the complex amplitude image u(x, y, 0) at the position z=0 is expressed as U(k x ,k y , 0), the complex amplitude image u(x, y, d) at the position z=d and the two-dimensional Fourier transform U(k x ,k y , d) is expressed by the following equation for free propagation, where i is the imaginary unit and k0 is the wave number of light in the object being observed.

[0054]

number

[0055]

number

[0056] In a two-dimensional phase image generating step S74, the two-dimensional phase image generating unit 74 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images for each of the multiple light irradiation directions generated by the second complex amplitude image generating unit 73. The two-dimensional phase image generated here corresponds to a phase image centered on the focused z-direction position.

[0057] Note that the processes in the two-dimensional phase image generating step S74 and subsequent steps may be performed after all complex amplitude images at each of the multiple positions for each of the multiple light irradiation directions have been generated in the second complex amplitude image generating step S73. Alternatively, a complex amplitude image at one z-direction position may be generated for each of the multiple light irradiation directions in the second complex amplitude image generating step S73, and a two-dimensional phase image at that position may be generated in the two-dimensional phase image generating step S74. This unit process may be repeated while scanning the z-direction position. The latter case is preferable because it reduces the amount of image data to be stored in the storage unit 79.

[0058] In a three-dimensional phase image generating step S75, the three-dimensional phase image generating unit 75 generates a three-dimensional phase image between the first position and the second position based on the two-dimensional phase images at each of the multiple positions generated by the two-dimensional phase image generating unit 74. The three-dimensional phase image generated here is an image in which the positions x and y in the two-dimensional phase image and the position z of the two-dimensional phase image are variables.

[0059] In the refractive index distribution calculation step S76, the refractive index distribution calculation unit 76 obtains the three-dimensional refractive index distribution of the observation object between the first position and the second position by deconvolution based on the three-dimensional phase image generated by the three-dimensional phase image generation unit 75. The refractive index distribution of the observation object is n(x, y, z), the electric susceptibility distribution is f(x, y, z), and the refractive index of the background medium is n mThen, there is a relationship between the two as shown in the following equation (3). The three-dimensional phase image Φ(x, y, z) generated by the three-dimensional phase image generation unit 75 is expressed by the convolution of the kernel function g(x, y, z) and the electric susceptibility distribution f(x, y, z), as shown in the following equation (4). Therefore, the three-dimensional refractive index distribution n(x, y, z) of the observation object can be obtained by deconvolution based on the three-dimensional phase image Φ(x, y, z).

[0060]

number

[0061]

number

[0062] The kernel function g is based on the Green's function, which corresponds to the solution of the wave equation. Figure 7 is a diagram explaining the kernel function g. In this diagram, the central position where the value of the kernel function g is the largest is the origin, the vertical direction is the z-axis, and the horizontal direction is the direction perpendicular to the z-axis.

[0063] In the third complex amplitude image generating step S77, the third complex amplitude image generating unit 77 generates a complex amplitude image of the second position for each of the plurality of light irradiation directions based on the complex amplitude image of the first position used in the second complex amplitude image generating step S73 and the three-dimensional refractive index distribution of the object to be observed between the first position and the second position calculated in the refractive index distribution calculating step S76.

[0064] In step S83, which includes second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, and refractive index distribution calculating step S76, the three-dimensional refractive index distribution of the object to be observed between the first position and the second position is calculated based on the complex amplitude image of the first position with respect to the distance from the imaging unit 43 along the light propagation path. The processes of step S83 and third complex amplitude image generating step S77 are repeatedly performed. This will be described with reference to FIGS. 4, 5, 8, and 9.

[0065] FIG. 8 is a diagram illustrating the relationship between the region including the observation object and the first to Jth blocks. As shown in this diagram, the region including the observation object is divided into the first to Jth blocks in order based on the distance from the imaging unit along the light propagation path (z direction). In this diagram, J=3. The jth block among the first to Jth blocks is the block where z=z j-1 From z=z j In each j-th block, the area closest to the imaging unit is z=z j-1 The position (near end) is the first position, and the position z=z is the furthest from the imaging unit. j The position (far end) is the second position.

[0066] 9 is a diagram illustrating the processing procedure for the first to jth blocks. As shown in this diagram, for each jth block, in step S83, a complex amplitude image and a two-dimensional phase image are generated for each of a plurality of z-direction positions between the first and second positions based on the complex amplitude image at the first position, a three-dimensional phase image is generated between the first and second positions, and a three-dimensional refractive index distribution is calculated. For each jth block, in a third complex amplitude image generation step S77, a complex amplitude image at the second position is generated based on the complex amplitude image at the first position and the three-dimensional refractive index distribution calculated in step S83.

[0067] The complex amplitude image at the second position of the (j+1)th block generated in the third complex amplitude image generating step S77 is used as the complex amplitude image at the first position of the next jth block, and the processes of step S83 and the third complex amplitude image generating step S77 are performed for the jth block. Once the three-dimensional refractive index distributions are obtained for each of the first to Jth blocks, they are combined to obtain the three-dimensional refractive index distribution of the entire object being observed.

[0068] 4 and 5, in step S81 after first complex amplitude image generating step S72, j is set to 0. In the following step S82, the value of j is incremented by 1 to j = 1, and the processes of step S83 and third complex amplitude image generating step S77 are performed for the first block. That is, for the first block closest to the imaging unit, the process of step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, and refractive index distribution calculation step S76) and third complex amplitude image generating step S77 are performed in order, based on the complex amplitude image generated in first complex amplitude image generating step S72, with the position z = z0 closest to the imaging unit (near end) set as the first position and the position z = z1 farthest from the imaging unit (far end) set as the second position. Then, the process returns to step S82.

[0069] For the j-th block (here, j is 2 or more and less than J), the closest complex amplitude image z=z j-1 The position (near end) is the first position, and the position z=z is the furthest from the imaging unit. j The position (distal end) is set as the second position, and the processes of step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, refractive index distribution calculation step S76) and third complex amplitude image generating step S77 are performed in order. Then, the process returns to step S82.

[0070] For the J-th block, which is the final block farthest from the imaging unit, the z=z closest to the imaging unit is calculated based on the complex amplitude image generated for the (J-1)-th block in the third complex amplitude image generation step S77. J-1 The position (near end) is the first position, and the position z=z is the furthest from the imaging unit. J The position (distal end) is set as the second position, and the processing of step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, refractive index distribution calculating step S76) is performed.

[0071] The Jth block may be determined to be the final block in step S84 after step S83, and the process may end without proceeding to the third complex amplitude image generating step S77. Note that the Jth block may be determined to be the final block after the 3D phase image generating step S75, and the process may end without proceeding to the refractive index distribution calculating step S76, in which case a 3D phase image of the entire observation object is obtained.

[0072] The region including the object to be observed may be divided into two blocks based on the distance from the imaging unit along the light propagation path (z direction), in which case the above-mentioned processing for the first block and the processing for the Jth block in the final stage may be performed. The region including the object to be observed does not have to be divided into multiple blocks, in which case the processing for step S83 (second complex amplitude image generating step S73, two-dimensional phase image generating step S74, three-dimensional phase image generating step S75, refractive index distribution calculating step S76) and third complex amplitude image generating step S77 may be performed only once in that order.

[0073] Next, the two-dimensional phase image generating step S74 will be described in detail. In the two-dimensional phase image generating step S74, the two-dimensional phase image generating unit 74 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images for each of the multiple light irradiation directions.

[0074] 10 is a flowchart of the two-dimensional phase image generating step S74. In the two-dimensional phase image generating step S74, for each of the multiple positions, in step S21, a complex differential interference image for each of the multiple light irradiation directions is generated based on the complex amplitude image for each of the multiple light irradiation directions, in step S22, a differential phase image is generated based on the sum of the complex differential interference images for each of the multiple light irradiation directions, and in step S23, a two-dimensional phase image is generated based on the differential phase image.

[0075] If the complex amplitude image at the position z=d is u(x, y, d), the complex differential interference image q(x, y, d) generated in step S21 is expressed by the following equation (5). At least one of δx and δy is non-zero. If δx ≠ 0 and δy = 0, a complex differential interference image q with the x direction as the shear direction is obtained. If δx = 0 and δy ≠ 0, a complex differential interference image q with the y direction as the shear direction is obtained. If δx ≠ 0 and δy ≠ 0, a complex differential interference image q with a shear direction different from both the x and y directions is obtained. Note that the complex differential interference image q(x, y, d) may be calculated using equation (5) after converting the complex amplitude image u(x, y, d) as shown in equation (6) below.

[0076]

number

[0077]

number

[0078] The sum of the complex differential interference images q for each of the multiple light irradiation directions is q sum (x, y, d), the phase differential image φ(x, y, z) generated in step S22 is q sum The phase of (x, y, d) is expressed by the following equation (7): In step S23, a two-dimensional phase image can be generated by integrating or deconvolving this differential phase image φ(x, y, z).

[0079]

number

[0080] In step S21, a complex differential interference image may be generated for each of a plurality of different shear directions on the complex amplitude image. In this case, the two-dimensional phase image generating step S74 generates, for each of a plurality of positions, a complex differential interference image for each of a plurality of different shear directions on the image based on the complex amplitude images for each of the plurality of light irradiation directions in step S21, generates a differential phase image for each of the plurality of shear directions based on the sum of the complex differential interference images for each of the plurality of light irradiation directions in step S22, and generates a two-dimensional phase image based on the differential phase images for each of the plurality of shear directions in step S23.

[0081] The differential phase image generated in step S22 based on the sum of the complex differential interference images for each of the multiple light irradiation directions has a reduced effect of multiple scattered light. Finally, the three-dimensional refractive index distribution finally obtained in refractive index distribution calculation step S76 also has a reduced effect of multiple scattered light, suppressing speckle. Furthermore, when complex differential interference images are generated for each of multiple different shear directions on the complex amplitude image in step S21, it is possible to suppress the appearance of line-shaped noise in the two-dimensional phase image obtained in step S23.

[0082] Here, the case where a two-dimensional phase image is generated by integrating or deconvolving the phase differential image in step S23 has been described. However, the phase differential image can also be treated as a two-dimensional phase image. In this case, without performing step S23, the three-dimensional refractive index distribution of the object to be observed can be obtained from the phase differential image (two-dimensional phase image) generated in step S22 by using a kernel (FIG. 11) including the kernel used in the deconvolution in step S23 in the deconvolution in refractive index distribution calculation step S76. The kernel shown in FIG. 11 is obtained by convolution integration of the kernel shown in FIG. 7 and the kernel used in the deconvolution in step S23.

[0083] Next, the third complex amplitude image generating step S77 will be described in detail. When an interference intensity image is acquired by irradiating the object to be observed with light, in each j-th block, j ) propagates through the jth block and reaches the first position (z=z j-1 ) and then propagates to the imaging unit. Therefore, in the third complex amplitude image generation step S77, the first position (z=z j-1 ) is propagated backward through the jth block to the second position (z=z j That is, in the third complex amplitude image generating step S77, the optical wavefront at the first position (z=z j-1 ) and the refractive index distribution of the jth block, the second position (z = z j ) to generate a complex amplitude image. In this process, a method is used to numerically calculate the propagation of the optical wavefront, taking into account the refractive index distribution of the medium. Known numerical calculation methods for propagation in such inhomogeneous media include BPM (Beam Propagation Method) and SSNP (Split-Step Non-Paraxial). The following describes the process using BPM in the third complex amplitude image generation step S77.

[0084] FIG. 12 is a diagram explaining the processing contents of BPM. This diagram shows an arbitrary jth block. As shown in this diagram, the jth block is divided into M slices (seven in this diagram) (1st to Mth slices) based on the distance from the imaging unit along the light propagation path (z direction). The thickness of each slice is about the wavelength.

[0085] The thickness of each slice may be constant. Here, the thickness of each slice is set to a constant value Δz. The m-th slice among the 1st to M-th slices of the j-th block is located at the position (z j-1 +(m-1)Δz) to position (z j-1 +mΔz). The first position of the jth block (z=z j-1 ) to the second position (z=z j ), a phase change according to the refractive index distribution is given to the first to Mth slices in order, and the light wavefront is propagated backward by Δz.

[0086] The thickness Δz of each slice in the processing of the third complex amplitude image generation step S77 may be different from or may be the same as the position interval when generating complex amplitude images for each of the multiple z-direction positions between the first position and the second position in the processing of the second complex amplitude image generation step S73.

[0087] The phase change o(x, y, z) given to the light wavefront when it propagates backward through a slice of thickness Δz at position z is expressed by the following equation (8). v is the wave number of light in a vacuum. δn(x,y,z) is the refractive index distribution n(x,y,z) of the object at position z and the refractive index n of the background (medium). b and is expressed by the following equation (9): cos θ is expressed by the following equation (10):

[0088]

number

[0089]

number

[0090]

number

[0091] The position of the mth slice (z=z j-1 If the complex amplitude of light at the position (z+Δz) after the light has propagated backward inside the m-th slice is u(x, y, z), the complex amplitude u(x, y, z+Δz) of the light at the position (z+Δz) after the light has propagated backward inside the m-th slice is expressed by the following equation (11). x ,k y ;Δz) is expressed by the following equation (12): Equation (11) is a Fourier transform of the product of the complex amplitude u(x, y, z) of light and the phase change o(x, y, z), and the result of this Fourier transform is x ,k y ;Δz) is inverse Fourier transformed to obtain the complex amplitude u(x,y,z+Δz) of the light at the position (z+Δz) after propagating through a slice of thickness Δz. P Δz is a function that calculates the light propagation of Δz.

[0092]

number

[0093]

number

[0094] The propagation of the light wavefront in each slice of the j-th block is expressed by the following equations (13) to (15): j-1 ) is the complex amplitude of light at u(x,y,z j-1 ), then the complex amplitude u(x,y,z) of the light after propagating through the first slice of the jth block is j-1 +Δz) is expressed by the following equation (13). The complex amplitude of the light after propagating through the (m-1)th slice of the jth block is expressed as u(x, y, z j-1+(m-1)Δz), the complex amplitude u(x,y,z j-1 +mΔz) is expressed by the following equation (14). The complex amplitude of the light after propagating through the (M-1)-th slice of the j-th block is expressed as u(x, y, z j-1 +(M-1)Δz), then the second position (z=z j ) the complex amplitude u(x,y,z j ) is expressed by the following equation (15).

[0095]

number

[0096]

number

[0097]

number

[0098] In this way, in the third complex amplitude image generating step S77, the first position (z=z j-1 ) is propagated back through the jth block slice by slice, and the second position (z=z j ) can be obtained.

[0099] FIG. 13 is a flowchart of the third complex amplitude image generating step S77. In step S41, the position z is set to the first position of the jth block (z=z j-1) is initialized to . In step S42, the interaction between the complex amplitude u(x, y, z) of the light at position z and the phase change o(x, y, z) is found. In step S43, the wavefront of the light after the interaction is propagated by a distance Δz to find the complex amplitude u(x, y, z+Δz) of the light at position z+Δz. In step S44, z to which Δz has been added is set as a new z. In step S44, when position z is set to the second position of the jth block (z=z j If it is determined that the position z has not yet reached the second position of the j-th block (z=z j If it is determined that the complex amplitude of the light has reached the second position (z=z) of the jth block, the process of the third complex amplitude image generation step S77 is terminated. j ) is the complex amplitude at

[0100] Next, an example will be described. In this example, the observation device 1A (Fig. 1) was used, and Fourier fringe analysis was adopted. In step S21, complex differential interference contrast images were generated for each of two different shear directions (vertical shear and horizontal shear) on the complex amplitude image. A spheroid of a three-dimensional culture of HepG2 derived from human liver cancer was used as the observation object. The region including the observation object was divided into five blocks. Figs. 14 to 28 show examples of images obtained in each step.

[0101] Fig. 14 is an interference intensity image (at normal irradiation) acquired in interference intensity image acquisition step S71. Fig. 15 is a complex amplitude image (real part, z = z0) generated based on the interference intensity image (Fig. 14) in first complex amplitude image generation step S72. Fig. 16 is a complex amplitude image (real part, z = z) generated based on the complex amplitude image (Fig. 15) in second complex amplitude image generation step S73. n )

[0102] Fig. 17 shows a complex differential interference image (imaginary parts for both x-direction shear and y-direction shear) generated based on the complex amplitude image (Fig. 16) in step S21 of the two-dimensional phase image generation step S74. Fig. 18 shows a differential phase image (x-direction shear and y-direction shear) generated based on the complex differential interference image (Fig. 17) in step S22 of the two-dimensional phase image generation step S74. Fig. 19 shows the refractive index distribution of the first block generated based on the differential phase image (Fig. 18) in the refractive index distribution calculation step S76.

[0103] Fig. 20 is a complex amplitude image obtained after a phase change o(x, y, z) is applied to the complex amplitude image (Fig. 15) in step S42 of the third complex amplitude image generation step S77. Fig. 21 is a complex amplitude image obtained after the complex amplitude image (Fig. 20) is propagated by Δz in step S44 of the third complex amplitude image generation step S77. Fig. 22 is a complex amplitude image at the second position of the first block generated in the third complex amplitude image generation step S77.

[0104] 16 to 22 are examples of images obtained at each step for the first block. The same processing was performed for the second to fifth blocks. After calculating the refractive index distribution for the fifth block in the final stage in refractive index distribution calculation step S76, the refractive index distributions for the first to fifth blocks were combined to obtain the three-dimensional refractive index distribution of the entire object under observation.

[0105] FIG. 23(a) is a diagram showing the phase distribution of the optical wavefront at the first position (z=z0) of the first block. FIG. 23(b) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z1) of the first block. FIG. 23(c) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z2) of the second block. FIG. 23(d) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z3) of the third block. FIG. 23(e) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z4) of the fourth block. FIG. 23(f) is a diagram showing the phase distribution of the optical wavefront at the second position (z=z5) of the fifth block.

[0106] These figures show the phase distribution of the light wavefront at positions z0 to z5 when light perpendicularly irradiated onto the object of observation is propagated backward from position z0 to position z5. As shown in these figures, the phase distribution of the light wavefront becomes uniform by propagating the light backward sequentially through the first to fifth blocks.

[0107] 24 to 28 are diagrams comparing the refractive index distributions of the example and the comparative example. In the comparative example, the refractive index distribution of the object of observation was found by sequentially performing interference intensity image acquisition step S71, first complex amplitude image generation step S72, second complex amplitude image generation step S73, two-dimensional phase image generation step S74, three-dimensional phase image generation step S75, and refractive index distribution calculation step S76, without dividing the object of observation into a plurality of blocks.

[0108] Fig. 24(a) is an MIP image in the y direction of the refractive index distribution obtained in the comparative example. Fig. 24(b) is an MIP image in the y direction of the refractive index distribution obtained in the example. Fig. 25(a) is an MIP image in the z direction of the refractive index distribution obtained in the comparative example. Fig. 25(b) is an MIP image in the z direction of the refractive index distribution obtained in the example. These MIP (Maximum Intensity Projection) images are images obtained by performing processing to obtain the maximum value of the refractive index along the y direction or the z direction in the refractive index distribution.

[0109] FIG. 26(a) is an image of the refractive index distribution (z=22 μm) obtained in the comparative example. FIG. 26(b) is an image of the refractive index distribution (z=22 μm) obtained in the example. FIG. 27(a) is an image of the refractive index distribution (z=45.2 μm) obtained in the comparative example. FIG. 27(b) is an image of the refractive index distribution (z=45.2 μm) obtained in the example. FIG. 28(a) is an image of the refractive index distribution (z=68.8 μm) obtained in the comparative example. FIG. 28(b) is an image of the refractive index distribution (z=68.8 μm) obtained in the example.

[0110] 24 to 28, in the refractive index distribution obtained in the comparative example, the structure disappears at positions far from the imaging unit. In contrast, in the refractive index distribution obtained in the example, the structure can be confirmed even at positions far from the imaging unit.

[0111] Note that observation devices and observation methods using self-interference may also be used. For example, the observation device 1D shown in FIG. 29 includes a light source 11, a lens 12, a lens 21, a mirror 22, a lens 23, a condenser lens 24, an objective lens 25, a mirror 44, a lens 42, an imaging unit 43, and an analysis unit 70. Compared to the configurations of the observation devices described above, the observation device 1D differs in that light output from the light source 11 is guided through an optical fiber 14 and then emitted from a light emitting end 18 without being split into two. The observation device 1D also differs in that a mirror 44 is provided instead of a beam splitter 41. The observation device 1D does not include an interference optical system. The imaging unit 43 can capture an interference intensity image of a reference position due to self-interference of light irradiated onto the observation object S along each of a plurality of light irradiation directions and passing through the observation object S. The analysis unit 70 can perform image processing similar to that of the observation devices 1A to 1C using such interference intensity images due to self-interference.

[0112] Furthermore, the three-dimensional refractive index distribution of the observation object S between the first position and the second position does not have to be a refractive index distribution based on a three-dimensional phase image, and may be acquired using a separate refractive index distribution acquisition device capable of acquiring a refractive index distribution. In this case, the observation device may include: (1) an interference intensity image acquisition unit that acquires an interference intensity image of a reference position for each of a plurality of light irradiation directions from an imaging unit that captures an interference intensity image of a reference position of light that has been irradiated onto the observation object S along each of a plurality of light irradiation directions and passed through the observation object; (2) a first complex amplitude image generation unit that generates a complex amplitude image for each of the plurality of light irradiation directions based on the interference intensity image; (3) a refractive index distribution acquisition unit that acquires a three-dimensional refractive index distribution of the observation object S between the first position and the second position in terms of the distance from the imaging unit along the light propagation path; and (4) a second complex amplitude image generation unit (corresponding to the third complex amplitude image generation unit of the observation devices 1A to 1D) that generates a complex amplitude image for the second position based on the complex amplitude image for the first position and the three-dimensional refractive index distribution for each of the plurality of light irradiation directions.

[0113] In this case, the observation method may include: (1) an interference intensity image acquisition step of acquiring an interference intensity image of a reference position for each of a plurality of light irradiation directions from an imaging unit that has captured an interference intensity image of a reference position of light that has been irradiated onto the object of observation along each of a plurality of light irradiation directions and passed through the object of observation; (2) a first complex amplitude image generation step of generating a complex amplitude image for each of the plurality of light irradiation directions based on the interference intensity image; (3) a refractive index distribution acquisition step of acquiring a three-dimensional refractive index distribution of the object of observation between a first position and a second position in terms of the distance from the imaging unit along the light propagation path; and (4) a second complex amplitude image generation step (corresponding to the third complex amplitude image generation step in the observation method using the observation devices 1A to 1D) of generating a complex amplitude image of a second position based on the complex amplitude image of the first position and the three-dimensional refractive index distribution for each of the plurality of light irradiation directions.

[0114] In this way, in this embodiment, even if the object to be observed is a multiple scattering body, the effect of multiple scattered light can be reduced to observe the object, thereby improving the depth of observation of the object to be observed. [Explanation of symbols]

[0115] 1A to 1C... observation device, 2... recording medium, 11... light source, 12... lens, 13... light input end, 14... optical fiber, 15... fiber coupler, 16, 17... optical fiber, 18, 19... light output end, 21... lens, 22... mirror, 23... lens, 24... condenser lens, 25... objective lens, 31... lens, 32... mirror, 33... drive unit, 34... lens, 41... beam splitter, 42... lens, 43... imaging unit, 44... mirror, 70... analysis unit, 71... interference intensity image acquisition unit, 72... first complex amplitude image generation unit, 73... second complex amplitude image generation unit, 74... two-dimensional phase image generation unit, 75... three-dimensional phase image generation unit, 76... refractive index distribution calculation unit, 77... third complex amplitude image generation unit, 78... display unit, 79... memory unit.

Claims

1. an interference intensity image acquisition unit that acquires, from an imaging unit that captures an interference intensity image at a reference position due to interference between light that has been irradiated onto an observation object along each of a plurality of light irradiation directions and passed through the observation object and a reference light, the interference intensity image at the reference position in each of the plurality of light irradiation directions; a first complex amplitude image generating unit that generates a complex amplitude image based on the interference intensity image for each of the plurality of light irradiation directions; a second complex amplitude image generating unit configured to generate, for each of the plurality of light irradiation directions, a complex amplitude image of each of a plurality of positions between the first position and a second position based on a complex amplitude image of a first position in relation to a distance from the imaging unit along a light propagation path; a two-dimensional phase image generating unit that generates a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude images for each of the plurality of light irradiation directions for each of the plurality of positions, and generates a two-dimensional phase image based on the complex differential interference image for each of the plurality of light irradiation directions; a three-dimensional phase image generating unit that generates a three-dimensional phase image from the first position to the second position based on the two-dimensional phase images at each of the plurality of positions; a refractive index distribution calculation unit that calculates a three-dimensional refractive index distribution of the object to be observed between the first position and the second position based on the three-dimensional phase image; a third complex amplitude image generating unit that generates a complex amplitude image at the second position based on the complex amplitude image at the first position and the three-dimensional refractive index distribution for each of the plurality of light irradiation directions; Equipped with based on the complex amplitude image generated by the first complex amplitude image generation unit, the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit perform their respective processes in sequence. Observation equipment.

2. Dividing an area including the object to be observed into a first block and a second block in order based on a distance from the imaging unit along a light propagation path, for a first block closest to the imaging unit, based on the complex amplitude image generated by the first complex amplitude image generation unit, a near end of the first block is set as the first position, a far end of the first block is set as the second position, and processing is sequentially performed by the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit; for a second block that is a final block farthest from the imaging unit, the near end of the second block is set as the first position and the far end of the second block is set as the second position based on the complex amplitude image generated for the first block by the third complex amplitude image generation unit, and the second complex amplitude image generation unit, the two-dimensional phase image generation unit, and the three-dimensional phase image generation unit perform processing in sequence. The observation device according to claim 1 .

3. Dividing an area including the object to be observed into first to Jth blocks (J is 3 or more) in order based on a distance from the imaging unit along a light propagation path, for a first block closest to the imaging unit, based on the complex amplitude image generated by the first complex amplitude image generation unit, a near end of the first block is set as the first position, a far end of the first block is set as the second position, and processing is sequentially performed by the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit; for a j-th block (j is 2 or more and less than J), based on the complex amplitude image generated for the (j-1)-th block by the third complex amplitude image generation unit, the near end of the j-th block is set as the first position and the far end of the j-th block is set as the second position, and the second complex amplitude image generation unit, the two-dimensional phase image generation unit, the three-dimensional phase image generation unit, the refractive index distribution calculation unit, and the third complex amplitude image generation unit perform processing in sequence; For a J-th block, which is the final block farthest from the imaging unit, the near end of the J-th block is set as the first position and the far end of the J-th block is set as the second position based on the complex amplitude image generated for the (J-1)-th block by the third complex amplitude image generation unit, and the second complex amplitude image generation unit, the two-dimensional phase image generation unit, and the three-dimensional phase image generation unit perform processing in sequence. The observation device according to claim 1 .

4. For the final stage block, the processing by the refractive index distribution calculation unit is performed after the processing by the three-dimensional phase image generation unit. The observation device according to claim 2 or 3.

5. The two-dimensional phase image generating unit generating a complex differential interference image for each of the plurality of light irradiation directions for each of a plurality of shear directions different from one another on the image based on the complex amplitude image for each of the plurality of light irradiation directions; generating the two-dimensional phase image based on the complex differential interference images in the plurality of shear directions and the plurality of light irradiation directions; The observation device according to any one of claims 1 to 4.

6. the two-dimensional phase image generating unit generates the two-dimensional phase image based on a sum of the complex differential interference images for each of the plurality of light irradiation directions. The observation device according to any one of claims 1 to 5.

7. an interference intensity image acquisition step of acquiring, from an imaging unit that captures interference intensity images at reference positions due to interference between light irradiated onto an observation object along each of a plurality of light irradiation directions and passing through the observation object, and a reference light, the interference intensity images at the reference positions in the plurality of light irradiation directions; a first complex amplitude image generating step of generating a complex amplitude image based on the interference intensity image for each of the plurality of light irradiation directions; a second complex amplitude image generating step of generating, for each of the plurality of light irradiation directions, a complex amplitude image of each of a plurality of positions between the first position and a second position based on a complex amplitude image of a first position in relation to a distance from the imaging unit along a light propagation path; a two-dimensional phase image generating step of generating a complex differential interference image for each of the plurality of light irradiation directions based on the complex amplitude images for each of the plurality of light irradiation directions for each of the plurality of positions, and generating a two-dimensional phase image based on the complex differential interference images for each of the plurality of light irradiation directions; a three-dimensional phase image generating step of generating a three-dimensional phase image from the first position to the second position based on the two-dimensional phase images at each of the plurality of positions; a refractive index distribution calculation step of calculating a three-dimensional refractive index distribution of the object to be observed between the first position and the second position based on the three-dimensional phase image; a third complex amplitude image generating step of generating a complex amplitude image at the second position based on the complex amplitude image at the first position and the three-dimensional refractive index distribution for each of the plurality of light irradiation directions; Equipped with performing the processes of the second complex amplitude image generating step, the two-dimensional phase image generating step, the three-dimensional phase image generating step, the refractive index distribution calculating step, and the third complex amplitude image generating step in sequence based on the complex amplitude image generated in the first complex amplitude image generating step; Observation method.

8. Dividing an area including the object to be observed into a first block and a second block in order based on a distance from the imaging unit along a light propagation path, for a first block closest to the imaging unit, based on the complex amplitude image generated in the first complex amplitude image generation step, a near end of the first block is set as the first position and a far end of the first block is set as the second position, and the processes of the second complex amplitude image generation step, the two-dimensional phase image generation step, the three-dimensional phase image generation step, the refractive index distribution calculation step, and the third complex amplitude image generation step are performed in order; for a second block that is a final block farthest from the imaging unit, the near end of the second block is set as the first position and the far end of the second block is set as the second position based on the complex amplitude image generated for the first block in the third complex amplitude image generation step, and the processes of the second complex amplitude image generation step, the two-dimensional phase image generation step, and the three-dimensional phase image generation step are sequentially performed. The observation method according to claim 7.

9. Dividing an area including the object to be observed into first to Jth blocks (J is 3 or more) in order based on a distance from the imaging unit along a light propagation path, for a first block closest to the imaging unit, based on the complex amplitude image generated in the first complex amplitude image generation step, a near end of the first block is set as the first position and a far end of the first block is set as the second position, and the processes of the second complex amplitude image generation step, the two-dimensional phase image generation step, the three-dimensional phase image generation step, the refractive index distribution calculation step, and the third complex amplitude image generation step are performed in order; for a j-th block (j is 2 or more and less than J), based on the complex amplitude image generated for the (j-1)-th block in the third complex amplitude image generation step, the near end of the j-th block is set as the first position and the far end of the j-th block is set as the second position, and the second complex amplitude image generation step, the two-dimensional phase image generation step, the three-dimensional phase image generation step, the refractive index distribution calculation step, and the third complex amplitude image generation step are performed in order; for a J-th block, which is the final block farthest from the imaging unit, based on the complex amplitude image generated for the (J-1)-th block in the third complex amplitude image generation step, the near end of the J-th block is set as the first position, the far end of the J-th block is set as the second position, and the processing of the second complex amplitude image generation step, the two-dimensional phase image generation step, and the three-dimensional phase image generation step are sequentially performed. The observation method according to claim 7.

10. For the final stage block, the refractive index distribution calculation step is performed after the three-dimensional phase image generation step. The observation method according to claim 8 or 9.

11. In the two-dimensional phase image generating step, generating a complex differential interference image for each of the plurality of light irradiation directions for each of a plurality of shear directions different from one another on the image based on the complex amplitude image for each of the plurality of light irradiation directions; generating the two-dimensional phase image based on the complex differential interference images in the plurality of shear directions and the plurality of light irradiation directions; The observation method according to any one of claims 7 to 10.

12. the two-dimensional phase image generating step generates the two-dimensional phase image based on a sum of the complex differential interference images for each of the plurality of light irradiation directions. The observation method according to any one of claims 7 to 11.

13. A program for causing a computer to execute each step of the observation method according to any one of claims 7 to 12.

14. A computer-readable recording medium on which the program according to claim 13 is recorded.

Images