Phase image acquisition system, phase image acquisition device, program, and phase image acquisition method

Abstract

This invention provides an inexpensive and versatile phase image acquisition system, phase image acquisition device, program, and phase image acquisition method that can acquire phase images using a mobile information terminal. [Solution] The phase image acquisition system comprises a portable information terminal equipped with a display for illuminating a sample, a camera for imaging the sample illuminated by the display, and a control unit, and an optical system that guides the light that illuminates the sample and passes through it to the camera. The control unit controls the display to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each of which illuminates the sample with light that has a different angle of incidence to the sample. The camera is controlled by the optical system to sequentially capture Fourier transform images of the sample when it is sequentially illuminated by each of the plurality of illumination patterns, and the plurality of Fourier transform images captured by the camera are reconstructed to generate a phase image in which the phase has been recovered.

Description

[Technical Field]

[0001] This disclosure relates to a phase image acquisition system, a phase image acquisition device, a program, and a phase image acquisition method. [Background technology]

[0002] Cells, blood cells, and other biological tissues absorb little light and have a refractive index close to that of water. Therefore, images obtained by conventional imaging have the problem of low contrast. To solve this problem, quantitative phase imaging techniques have been proposed, but these techniques require delicate and expensive equipment such as lasers and interferometers, making their use in medical settings difficult. Recently, however, a technique for extracting phase images using optical Fourier ptychography has been proposed.

[0003] Japanese Patent Publication No. 2016-530567 (Patent Document 1) discloses an ultra-high NA Fourier ptychography imaging system comprising a variable illumination device configured to irradiate a sample at multiple incident angles at different times, an optical system equipped with a high NA lens, and a radiation detector configured to acquire multiple luminance images corresponding to multiple different incident angles. This technology requires a specific illumination device, such as using an LED array in the variable illumination device. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Special Publication 2016-530567 [Overview of the project] [Problems that the invention aims to solve]

[0005] This disclosure has been made in view of the above-mentioned prior art, and aims to provide an inexpensive and versatile phase image acquisition system, phase image acquisition device, program, and phase image acquisition method that can acquire phase images using a mobile information terminal. [Means for solving the problem]

[0006] A phase image acquisition system according to one aspect of the present disclosure is a phase image acquisition system for acquiring a phase image of a sample, comprising: a portable information terminal having a display for illuminating the sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera; and an optical system that guides light illuminated by the display and transmitted through the sample to the camera, wherein the control unit controls the display to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each of which illuminates the sample with light having a different angle of incidence to the sample; controls the camera to sequentially capture Fourier transform images of the sample illuminated by each of the plurality of illumination patterns with light guided to the camera by the optical system; and reconstructs a plurality of Fourier transform images captured by the camera to generate a phase image with recovered phase. [Effects of the Invention]

[0007] According to this disclosure, there is an advantage in that phase images can be acquired inexpensively and easily using a mobile information terminal. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing an example of the configuration of a phase image acquisition system according to the first embodiment. [Figure 2] This is a schematic diagram showing an example of the configuration of a mobile information terminal. [Figure 3] This is a schematic diagram showing an example of a pattern displayed on the screen of a mobile device. [Figure 4] This is a schematic diagram illustrating an example of how patterns displayed on a mobile device screen change. [Figure 5] This is a schematic diagram showing an example of a projection optical system configuration. [Figure 6]It is a schematic diagram showing an example of illumination light when the aperture number of the projection lens is large. [Figure 7] It is a schematic diagram showing an example of illumination light when the aperture number of the projection lens is small. [Figure 8] It is a schematic diagram showing the state of shaping of illumination light when the light shaping element is a diffusion plate. [Figure 9] It is a schematic diagram showing the state of shaping of illumination light when the light shaping element is a hologram element. [Figure 10] It is a schematic diagram for explaining a method of manufacturing a hologram element. [Figure 11] It is a schematic diagram showing an example of the configuration of the stage of the phase image acquisition system. [Figure 12] It is a block diagram showing the hardware configuration of a mobile information terminal. [Figure 13] It is a chart of a table showing the relationship between a pattern and an incident angle. [Figure 14] It is a functional block diagram showing the functional configuration of a mobile information terminal according to the first embodiment. [Figure 15] It is a flowchart showing an example of the flow of information processing executed by the control unit of the mobile information terminal according to the first embodiment. [Figure 16] It is a flowchart showing an example of the flow of imaging processing shown in FIG. 15. [Figure 17] It is a flowchart showing an example of the flow of reconstruction processing shown in FIG. 15. [Figure 18] It is a functional block diagram showing the functional configuration of a mobile information terminal according to the second embodiment. [Figure 19] It is a flowchart showing an example of the flow of information processing executed by the control unit of the mobile information terminal according to the second embodiment. [Figure 20] It is a schematic diagram for explaining the focusing condition. [Figure 21] It is a flowchart showing an example of the flow of focusing processing shown in FIG. 19.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, an example of an embodiment for carrying out the technology of this disclosure will be described in detail with reference to the drawings. Components and processes that perform the same operation, action, or function are given the same reference numerals throughout the drawings, and redundant explanations may be omitted as appropriate. Each drawing is only a schematic representation to the extent that the technology of this disclosure can be fully understood. Therefore, the technology of this disclosure is not limited to the illustrated examples. Furthermore, in this embodiment, explanations of configurations not directly related to this disclosure or well-known configurations may be omitted.

[0010] <First Embodiment> Figure 1 shows the configuration of the phase image acquisition system 10 according to this embodiment. The phase image acquisition system 10 acquires phase images using Fourier ptychography. As shown in Figure 1, the phase image acquisition system 10 comprises a portable information terminal 20 and an inspection device 30. The portable information terminal 20 is a device equipped with a camera 21, a display 23 (see Figure 2), and a communication unit 22, etc. Examples of such devices include smartphones, tablet devices, or small notebook computers. Smartphones, in particular, which have a high penetration rate, are preferably used.

[0011] As shown in Figure 2, in order to perform Fourier ptychography, multiple different patterns P are sequentially displayed on the display 23 of the mobile information terminal 20 to provide illumination. As shown in Figure 3, pattern P is a light-dark image that has multiple regions 25 within the pattern, with a bright pixel displayed in one of the regions 25. The bright pixel consists of one or more pixels. The background screen of the display 23 is black.

[0012] As shown in Figure 4, displaying pattern P displays a single bright spot, and the positions of the bright spots differ for N patterns Pi (i=1 to N) where the positions of the bright pixels are different. Displaying pattern Pi allows sample 37 to be illuminated with light at a predetermined incident angle, similar to when one LED of an LED array is illuminated. Note that when there is no need to distinguish between each of the patterns Pi (i=1 to N), they are collectively referred to as pattern P.

[0013] The inspection device 30 comprises a stage 31, a drive unit 32, optical systems 80, 82, 84, 85 including optical components such as lenses, a communication unit 35, and a control unit 36. The optical system 85 is a projection optical system for projecting illumination light onto the sample 37. The inspection device 30 comprises a housing 30A, and the stage 31, drive unit 32, optical systems 80, 82, 84, 85, communication unit 35, and control unit 36 ​​are all located within the housing 30A.

[0014] The housing 30A is provided with a retaining groove 30D for holding the personal information terminal 20, and the personal information terminal 20 is held in a predetermined position. The housing 30A is also provided with a through hole 30C for exposing a portion of the display 23 of the personal information terminal 20. When measuring the sample 37, the light from the display 23 of the personal information terminal 20 passes through the through hole 30C, is relayed by the optical system 85, and illuminates the sample 37 at a predetermined angle of incidence.

[0015] The configuration of the projection optical system 85 will now be described with reference to Figures 5 to 9. As shown in Figure 5, the projection optical system 85 comprises a projection lens 60, a reflector 62, and an optical shaping element 64. The projection lens 60, the reflector 62, and the optical shaping element 64 are arranged in the order described above along the optical axis L of the light from the pattern P on the display 23, which is the light source. As the optical shaping element 64, a transmission diffuser plate, a hologram element such as a transmission type volume phase hologram, an optical diffraction element, a photonic crystal, a metamaterial, etc., can be used. The light emitted from the pattern P is imaged by the projection lens 60 and shaped by the optical shaping element 64, which is positioned on the imaging surface of the pattern P, to illuminate the entire sample 37.

[0016] In the configurations shown in Figures 6 and 7, the optical shaping element 64 is not present. When the sample 37 cannot be directly illuminated by the pattern P displayed on the display 23 of the portable information terminal 20, an image of the pattern P (pattern image) is formed near the sample 37, and the light rays directed from this image toward the sample 37 are used as illumination light. As shown in Figure 6, the light from the pattern P is focused onto the imaging plane by the projection lens 60, forming a pattern image Ip. Bright spots corresponding to the bright spots of the pattern P are formed in the pattern image Ip, and the light diffused from these point light sources illuminates the sample 37. If the numerical aperture of the projection lens 60 is sufficiently large relative to the pattern P, even without placing the optical shaping element 64, light rays with a large angle are generated for the bright spots at the edges of the pattern P, illuminating the entire sample 37.

[0017] On the other hand, increasing the numerical aperture of the projection lens 60 leads to problems such as a shorter projection distance and an increase in lens size. To solve these problems, a projection lens 60 with a small numerical aperture is used. As shown in Figure 7, when the numerical aperture of the projection lens 60 is small relative to the pattern P, the illumination angle range becomes smaller, and light rays at large angles are not generated for the bright spots at the edges of the pattern P, resulting in a situation where a part of the sample 37 is not illuminated, as enclosed by the dashed circle. This reduces the number of conditions for minimizing the comparison error with the current image used for optimizing phase image reconstruction, leading to a decrease in phase accuracy.

[0018] Therefore, in this embodiment, an optical shaping element 64 is placed on the imaging plane of the pattern P to shape the light emitted from the pattern P into light that can illuminate the entire sample 37. For example, when a transmission diffuser plate 64d is used as the optical shaping element 64, as shown in Figure 8, the light from the bright spots of the pattern image Ip is diffused by the transmission diffuser plate 64d, and the diffused light illuminates the sample 37. Therefore, even when using a projection lens 60 with a small numerical aperture, the entire sample 37 is illuminated by the light from the bright spots at the edges of the pattern P.

[0019] For example, when a hologram element 64h is used as the optical shaping element 64, as shown in Figure 9, the light from the bright spots of the pattern image Ip is diffracted by the hologram element 64h to illuminate the entire sample 37, and the diffracted light illuminates the sample 37. Therefore, even when using a projection lens 60 with a small numerical aperture, the entire sample 37 is illuminated by the light from the bright spots at the edges of the pattern P.

[0020] As shown in Figure 10, the hologram element 64h can be fabricated by simultaneously irradiating a photopolymer 66 for hologram recording with object light L1 and reference light L2 generated from the same laser light source. In the example shown in Figure 10, the object light L1 focused by the lens 68 and the parallel reference light L2 are combined by the half mirror 80, and the object light L1 and reference light L2 are irradiated coaxially onto the photopolymer 66 for hologram recording.

[0021] By using object light to illuminate the entire sample 37, when light equivalent to the reference light is shone on the hologram element 64h, light illuminating the entire sample 37 is regenerated. Since the hologram element 64h generates light illuminating the entire sample 37, the light utilization efficiency is higher compared to when using the transmission diffuser plate 64d.

[0022] In Figures 5 to 9, the pattern P on the display 23, which is the light source, is placed on the left side, so the optical path is bent by the reflector 62. However, in the system shown in Figure 1, the display 23 is placed above the sample 37, so there is no need to bend the optical path, and the reflector 62 can be omitted.

[0023] Returning to the explanation of Figure 1, the sample 37 to be measured is placed on the stage 31. The sample 37 is a sample such as biological tissue set in a sample cell such as a cuvette. In the center of the stage 31, there is a through-hole 31A that allows light that has passed through the sample 37 to pass through. The sample 37 is set at the position of this through-hole 31A.

[0024] Optical system 82 extends in a direction intersecting optical system 80, and optical system 84 extends in a direction intersecting optical system 82. Optical system 80 includes an objective lens 80A, and optical system 84 includes an imaging lens 84. A reflecting mirror 81 is positioned between optical system 80 and optical system 82, and a reflecting mirror 83 is positioned between optical system 82 and optical system 84.

[0025] Light passing through the through-hole 31A enters the optical system 80, is focused by the objective lens 80A, and is reflected by the reflector 81, causing the optical path to be bent by 90 degrees. The light reflected by the reflector 81 passes through the optical system 82, is reflected by the reflector 83, and the optical path is again bent by 90 degrees. The light reflected by the reflector 83 enters the optical system 84, is imaged by the imaging lens 84A, and exits from the optical system 84.

[0026] The housing 30A is provided with a through-hole 30B for exposing the camera 21 of the portable information terminal 20. When measuring the sample 37, the light emitted from the optical system 84 passes through the through-hole 30B and enters the camera 21.

[0027] Stage 31 is driven by a drive unit 32. The drive unit 32 can drive stage 31 in mutually orthogonal X, Y, and Z directions according to instructions from the control unit 36. Also, as shown in Figure 11, stage 31 may include a stage 31C on which a sample 37 is placed and a stage 31D on which a test sample 39 is placed. Stages 31C and 31D are rotatable around a rotation axis 31E. The drive unit 32 switches between stage 31C and stage 31D according to instructions from the control unit 36.

[0028] Furthermore, a communication unit 35 is provided on the side of the housing 30A. When measuring sample 37, the communication unit 22 of the portable information terminal 20 and the communication unit 35 of the inspection device 30 are connected by a communication cable 41. Note that the portable information terminal 20 and the inspection device 30 may be connected wirelessly.

[0029] Figure 12 is a block diagram showing the hardware configuration of the personal information terminal 20. As shown in Figure 12, the personal information terminal 20 includes a control unit 50.

[0030] The control unit 50 includes a CPU (Central Processing Unit) 50A, a ROM (Read Only Memory) 50B, a RAM (Random Access Memory) 50C, and an input / output interface (I / O) 50D. The CPU 50A, ROM 50B, RAM 50C, and I / O 50D are connected to each other via a bus 50E. The bus 50E includes a control bus, an address bus, and a data bus. A camera 21, a communication unit 22, a display 23, and a storage unit 24 are connected to the I / O 50D.

[0031] The camera 21 includes a lens 26 and an image sensor 27. The image sensor 27 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor.

[0032] The communication unit 22 is an interface for data communication with external devices such as the inspection device 30.

[0033] The display 23 is comprised of, for example, a touch panel.

[0034] The storage unit 24 is composed of, for example, non-volatile memory. As shown in Figure 12, the storage unit 24 stores information processing programs 24A and various data 24B, etc. Examples of the various data 24B include image data of captured images and phase images, pattern image data, and tables showing the relationship between patterns and incident angles (see Figure 13).

[0035] The CPU50A is an example of a processor. The term "processor" here refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) or specialized processors (e.g., GPUs: Graphics Processing Units, ASICs: Application Specific Integrated Circuits, FPGAs: Field Programmable Gate Arrays, programmable logic devices, etc.).

[0036] The information processing program 24A may be stored on a non-volatile, non-transitory recording medium, or distributed via a network, and installed on the portable information terminal 20 as appropriate.

[0037] Examples of non-volatile, non-transitional recording media include CD-ROMs (Compact Disc Read Only Memory), magneto-optical disks, HDDs (Hard Disk Drives), DVD-ROMs (Digital Versatile Disc Read Only Memory), flash memory, and memory cards.

[0038] Figure 14 is a block diagram showing the functional configuration of the CPU 50A of the portable information terminal 20. As shown in Figure 14, the CPU 50A functionally comprises an imaging processing unit 52 and a reconstruction processing unit 54. The CPU 50A functions as each of these functional units by reading and executing the information processing program 24A stored in the storage unit 24.

[0039] The imaging processing unit 52 controls the display 23 to sequentially display multiple different patterns P to illuminate the sample 37, and causes light with multiple incident angles corresponding to the multiple patterns P to be incident on the sample 37, while also controlling the camera 21 to capture a Fourier transform image formed on the image sensor 27 for each of the multiple patterns P, thereby acquiring an image.

[0040] The reconstruction processing unit 54 reconstructs multiple captured images corresponding to multiple patterns P to generate a phase image with recovered phase.

[0041] Next, the operation of the portable information terminal 20 according to this embodiment will be described with reference to Figure 15. Figure 15 is a flowchart showing an example of the processing flow by the information processing program 24A according to this embodiment. The information processing shown in Figure 15 is executed when the user instructs the execution of the information processing program 24A from the menu of the mobile information terminal 20.

[0042] Prior to executing the information processing program 24A, the user places sample 37 on stage 31C and test sample 39 on stage 31D (see Figure 11).

[0043] In step S100, the CPU 50A displays the imaging instruction screen on the display 23. The imaging instruction screen displays a button to instruct the start of phase image acquisition, and the user presses this button to start the acquisition.

[0044] Furthermore, once the instructions are complete, the user sets the portable information terminal 20 on the top surface of the housing 30A of the inspection device 30 so that the pattern display position on the display 23 coincides with the position of the through hole 30C of the inspection device 30, and the position of the camera 21 coincides with the position of the through hole 30B of the inspection device 30 (see Figure 1). For example, these alignments may be performed when the portable information terminal 20 is held in a predetermined position by the holding groove 30D. Alternatively, without placing the sample 37, the alignment pattern may be displayed on the display 23, the alignment pattern may be captured by the camera 21, and the pattern display position on the display 23 may be set based on the captured image.

[0045] Next, in step S102, the CPU 50A determines whether or not it has been instructed to start imaging. If it has been instructed to start imaging, the process proceeds to step S104. If it has not been instructed to start imaging, the determination in step S102 is repeated.

[0046] Next, in step S104, the CPU 50A determines whether or not the personal digital assistant 20 (PDA) has been set. For example, if the CPU 50A detects from the PDA's sensors (e.g., an accelerometer or gyroscope) that the PDA has been placed face down, it may determine that the PDA has been set. If the PDA has been set, the process proceeds to step S106. If the PDA has not been set, the determination in S104 is repeated.

[0047] When the start of imaging is instructed and the mobile information terminal 20 is set, in step S106, the CPU 50A performs the imaging process. After the imaging process is completed, in step S108, the CPU 50A performs the reconstruction process. After the reconstruction process is completed, in step S110, the CPU 50A displays the phase image on the display 23. This completes the process.

[0048] (Imaging and processing) Next, the imaging process will be explained with reference to Figure 16. First, in step S200, the CPU 50A outputs an instruction signal to the inspection device 30 to switch stage 31 to stage 31C. The drive unit 32 of the inspection device 30 switches stage 31 to stage 31C on which the sample 37 is placed, in accordance with the instruction from the control unit 36. As a result, the sample 37 is positioned at the imaging location.

[0049] Next, in step S202, the CPU 50A acquires image data of multiple patterns P from the storage unit 24. For example, as shown in Figure 13, the storage unit 24 stores image data of N patterns required for Fourier tychography. In this case, image data of N patterns is acquired.

[0050] Next, in step S204, the CPU 50A selects one pattern to display from among multiple patterns P.

[0051] Next, in step S206, the CPU 50A instructs the display 23 to display the selected pattern P. The pattern P is displayed on the display 23, and the sample 37 is illuminated with light at an incident angle corresponding to the pattern P.

[0052] Next, in step S208, the CPU 50A instructs the camera 21 to take an image. The camera 21 takes an image of the Fourier transform of the light that has passed through the sample 37.

[0053] Next, in step S210, the CPU 50A acquires the captured image captured by the camera 21 and stores the image data of the captured image in the storage unit 24, associating it with a pattern ID. This captured image is referred to as the "measured Fourier image" in the reconstruction process.

[0054] Next, in step S212, the CPU 50A determines whether all of the multiple patterns P have been selected. If all patterns P have been selected, the routine terminates. On the other hand, if there are any patterns P that have not been selected, the process returns to step S204. In this way, the procedure from steps S204 to S212 is repeated until all patterns P have been selected.

[0055] (Reconstruction process) Next, we will explain the reconstruction process. Several methods can be used for reconstructing phase images, but one method is the one described in non-patent literature (Nature Photonics volume 7, pages 739-745 (2013)).

[0056] In this method, the initial low-resolution intensity image obtained with a low NA and phase 0 through oversampling is set as the default value. A new low-resolution image is generated by performing an inverse transform on the spectrum obtained by Fourier transforming this image, using the spectrum obtained by applying a low-pass filter to a circular subregion. The filter applied here corresponds to the coherent transfer function of the objective lens, and the filter center position in Fourier coordinates is determined by the wave vector k, which is determined by the illumination angle.x , k y It is set to match. For example, if it is lit vertically, k x =k y This creates a circular region centered at the position = 0. Next, the amplitude of the filtered complex amplitude in Fourier coordinates is replaced with the square root of the intensity of the measured Fourier image illuminated at an angle corresponding to the wave vector at the center of the filter. This operation is performed for all the measured Fourier images, replacing them with the corresponding initial filtering region in Fourier coordinates. Then, the inversely transformed complex amplitude image is replaced with the initial predicted image, and the process described so far is repeated. This iterative process is continued until there is no change in the complex amplitude image after each process.

[0057] Figure 17 is a flowchart showing an example of the reconstruction process flow. First, in step S300, the oversampled low-resolution intensity image is set as the initial value. This low-resolution intensity image is represented by the following formula.

[0058] JPEG2026077462000002.jpg1440

[0059] Next, in step S302, the j-th calculation is started, and the intensity image set to the initial value is Fourier transformed. The Fourier transform transforms the real space into a wave vector space.

[0060] Next, in step S304, the xy wave vectors kx and ky are derived from the illumination angle and wavelength of the i-th measured Fourier image. That is, the i-th pattern P i The angle of incidence (θx) when illuminated by i θy i From the wavelength of the illumination light and the luminescence, we derive the wave vectors kx and ky.

[0061] Next, in step S306, a circular filtering range is set based on the NA of the objective lens and the wavelength centered on the Fourier coordinates (kx, ky). In the wave vector space, this circular range corresponds to the incident angles (θx i , θy i ). The amplitude of the filtered complex amplitude image on the Fourier coordinates JPEG2026077462000003.jpg1416 is replaced with the square root of the intensity of the measured Fourier image obtained by illuminating with the incident angles (θx i , θy i ). JPEG2026077462000004.jpg1423

[0062] Next, in step S308, it is determined whether the Nth has been reached. If the Nth has been reached, the process proceeds to step S310. If the Nth has not been reached, the process proceeds to step S316. In step S316, i is incremented by 1 and the process returns to step S304. The processes of steps S304 and S306 are performed for all patterns P, that is, for all incident angles, covering the entire wave vector space.

[0063] Next, in step S310, an inverse Fourier transform of the complex amplitude image with the amplitude replaced is performed to obtain a real space image. Next, in step S312, the real space image obtained in the j-th calculation is compared with the real space image obtained in the (j - 1)-th calculation to determine whether the difference is sufficiently small. If the difference is sufficiently small, the process proceeds to step S314.

[0064] If it cannot be said that the difference is sufficiently small, the process proceeds to step S318. In step S318, j is incremented by 1 and the process returns to step S302, and iterative calculations are repeated until no change is seen in the real space image. When returning to step S302, the real space image obtained in step S310 is Fourier-transformed instead of the initial image.

[0065] ​When no further changes are observed in the real-space image, in step S314, the real-space image, i.e., the inverse Fourier transform image, is used as the solution to the phase image, and phase information φ is obtained. In this way, the phase information φ is recovered through iterative calculation. Furthermore, the real-space image obtained through the reconstruction process has improved resolution compared to the original real-space image.

[0066] As described above, in the phase image acquisition system of this embodiment, a phase image can be acquired by Fourier ptychography using the display of a commercially available personal information terminal as the illumination light source and the camera of the same personal information terminal as the imaging device. By using a commercially available personal information terminal, a versatile system can be constructed, and phase images can be acquired at low cost.

[0067] Here, we have described an example of reconstruction processing using the method described in non-patent literature (Nature Photonics volume 7, pages 739-745 (2013)). However, reconstruction processing may be performed by other methods, such as simulated annealing or, for speedup, quantum annealing. In simulated annealing, the energy of the system is taken from an error function synthesized from the difference between numerous Fourier images measured at different illumination angles and a Fourier image obtained by inverse transformation from a phase image set to an arbitrary initial value, and is computationally heated to a high temperature according to the Boltzmann distribution. Then, it is computationally cooled slowly, and each phase in which the state with the minimum energy is obtained is taken as the final phase solution. Similar calculations can be accelerated by using quantum annealing with the Ising model.

[0068] <Second Embodiment> In the second embodiment, a focusing process is performed to adjust the focal position of the lens 26 of the camera 21 before the imaging process. The other parts are the same as in the first embodiment, so the same components will not be described.

[0069] Figure 18 is a block diagram showing the functional configuration of the CPU 50A of the portable information terminal 20. As shown in Figure 18, the CPU 50A functionally comprises a focusing processing unit 51, an imaging processing unit 52, and a reconstruction processing unit 54. The CPU 50A functions as each of these functional units by reading and executing the information processing program 24A stored in the storage unit 24.

[0070] Since the imaging processing unit 52 and the reconstruction processing unit 54 have already been described in the first embodiment with reference to Figure 14, only the focusing processing unit 51 will be described. The focusing processing unit 51 controls the camera 21, which includes the lens 26 and the image sensor 27, to make the focal position of the lens 26 coincide with the imaging plane of the imaging lens 84A of the optical system 84.

[0071] Next, with reference to Figure 19, the operation of the portable information terminal 20 according to the second embodiment will be described. Figure 19 is a flowchart showing an example of the processing flow by the information processing program 24A according to the second embodiment. The information processing shown in Figure 19 is the same as the flowchart of the first embodiment shown in Figure 15, except for the focusing process in step S105. Therefore, the same parts are denoted by the same reference numerals and their explanation is omitted, and only the differences will be explained.

[0072] When the imaging instruction screen is displayed on the display 23, and imaging is instructed to start from the imaging instruction screen, and the mobile information terminal 20 is set, in step S105, the CPU 50A performs focusing. After the focusing process is completed, in step S106, the CPU 50A performs imaging. After the imaging process is completed, in step S108, the CPU 50A performs reconstruction. After the reconstruction process is completed, in step S110, the CPU 50A displays the phase image on the display 23. This completes the process.

[0073] (Focusing process) Now, with reference to Figure 20, the reason for performing the focusing process will be explained. When the sample 37 is illuminated with light corresponding to pattern P, the light that passes through the sample 37 passes through the objective lens 80A, the imaging lens 84A, and the lens 26 of the camera 21, and is imaged onto the image sensor 27.

[0074] Fourier ptychography uses the Fourier transform image observed on the Fourier plane S1 located between the objective lens 80A and the imaging lens 84A. The Fourier transform image is then inversely Fourier transformed by the imaging lens 84A, and a real-space image is observed on the imaging plane S2. The light transmitted through the imaging lens 84A is then Fourier transformed again by the lens 26 of the camera 21, and the Fourier transform image is projected onto the image sensor 27.

[0075] If the image sensor 27 is placed on the Fourier plane S1, it is possible to capture a Fourier transform image, but the camera 21 of the portable information terminal 20 cannot be used. Alternatively, if a screen is placed on the Fourier plane S1, the Fourier transform image projected onto the screen can be captured, but the resolution of the captured image will be reduced.

[0076] In this embodiment, the camera 21 is equipped with an autofocus function, and the camera 21 acquires the focus position by using the autofocus function to photograph the test sample 39. The test sample 39 is a sample from which a reference image can be obtained at the focus position. As the test sample 39, for example, a sample having a pattern with a known spatial frequency can be used. The position of the lens 26 where the image obtained by imaging the test sample matches the reference image is the focus position.

[0077] Next, we will explain the focusing process in detail with reference to Figure 21. First, in step S400, the CPU 50A outputs an instruction signal to the inspection device 30 to switch stage 31 to stage 31D. The drive unit 32 of the inspection device 30 switches stage 31 to stage 31D on which the test sample 39 is placed, in accordance with the instruction from the control unit 36. This positions the test sample 39 at the imaging position.

[0078] Next, in step S402, the CPU 50A instructs the camera 21 to take an image. The camera 21 takes an image of the test sample 39 using the autofocus function (AF).

[0079] Next, in step S404, the CPU 50A acquires the captured image taken by the camera 21. The acquired image is stored in the RAM 50C.

[0080] Next, in step S406, the CPU 50A compares the captured image with a pre-prepared reference image. For example, in the case of a test sample having a pattern with a known spatial frequency, the reference image represents a specific spatial frequency. Therefore, the spatial frequency of the captured image is compared with this specific spatial frequency.

[0081] Next, in step S408, the CPU 50A determines whether the captured image and the reference image match. If they match, the process proceeds to step S410. If they do not match, the process returns to step S402 in order to retake the image. In this way, steps S402 to S408 are repeated until the captured image and the reference image match.

[0082] Next, in step S410, the CPU 50A obtains the position of the lens 26 when the captured image matches the reference image as the "focus position".

[0083] Next, in step S412, the CPU 50A turns off the autofocus function of the camera 21 and sets the acquired "focus position" for the camera 21. As a result, the position of the lens 26 is fixed to the "focus position," and the image plane S2 of the imaging lens 84A comes into focus. In this step S412, the focusing process subroutine ends.

[0084] As described above, this embodiment provides the same effects as the first embodiment, and since focusing is performed according to the mobile information terminal, it can be used with all types of mobile information terminals. Furthermore, it can absorb deviations in the focus position due to misalignment or errors of optical components.

[0085] In the second embodiment, the camera 21 is equipped with an autofocus function, and an example was described in which the camera 21 obtains the focus position by using the autofocus function to photograph the test sample 39. However, in some cases, such as when the system is designed for a specific type of mobile information terminal, the focus position can be obtained in advance. In this case, the autofocus function may be turned off from the beginning, and the pre-obtained focus position may be set as the focus position. This reduces the number of processing steps for the focusing process.

[0086] <Variation> The configurations of the phase image acquisition system, information processing device, information processing method, and program described in the above embodiments are examples and may be modified as needed without departing from the main purpose.

[0087] For example, in the above embodiment, an example was described in which the reconstruction process is performed on the local side, such as a mobile information terminal, but the reconstruction process can also be performed on an external server. Furthermore, the obtained phase image data can be transmitted to an external display and displayed on the external display.

[0088] Furthermore, in the above embodiment, the portable information terminal is assumed to be a device such as a smartphone in which the camera and display are arranged on the same plane, according to the U-shaped optical system. However, the arrangement of the camera and display is not limited to the same plane. In the case of a device in which the camera and display are arranged on different planes, the optical system can be modified to guide the light obtained from the sample to the camera.

[0089] Furthermore, the operation of the processor in the above embodiments may not be performed by a single processor, but may be performed by multiple processors located in physically separate locations working together. Also, the order of the processor's operations is not limited to the order described in each of the above embodiments, but may be changed as appropriate.

[0090] Furthermore, the program processing flow described in the above embodiment is just one example, and unnecessary steps may be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0091] Furthermore, although the above embodiment describes a case in which the process according to the embodiment is realized by a software configuration using a computer by executing a program, the embodiment is not limited to this. The embodiment may also be realized by a hardware configuration or a combination of a hardware configuration and a software configuration.

[0092] <Note> The following is further disclosed regarding the embodiments described above. (Note 1) A phase image acquisition system for acquiring a phase image of a sample, A portable information terminal comprising a display for illuminating the sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera, An optical system that guides the light illuminated by the display and transmitted through the sample to the camera, Equipped with, The control unit, The display is controlled to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample. The camera is controlled to sequentially capture Fourier transform images of the sample when it is sequentially illuminated by each of the plurality of illumination patterns using light guided to the camera by the optical system, Multiple Fourier transform images captured by the aforementioned camera are reconstructed to generate a phase image with recovered phase. Phase image acquisition system. (Note 2) The system further comprises a second optical system that guides illumination light from the display to the sample, The phase image acquisition system according to Appendix 1, wherein the second optical system comprises an imaging optical system that forms an image of the illumination pattern in front of the sample, and an optical shaping element that shapes the light directed from the image of the illumination pattern toward the sample into light that illuminates the entire sample. (Note 3) The phase image acquisition system as described in Appendix 2, wherein the optical shaping element is a transmission diffuser plate, a hologram element, an optical diffraction element, a photonic crystal, or a metamaterial. (Note 4) The phase image acquisition system according to any one of Appendix 1 to 3, wherein the control unit further controls the camera so that the focal position of the camera lens coincides with the image plane of the imaging lens included in the optical system. (Note 5) The aforementioned camera is equipped with an autofocus function. The control unit, A known test sample is imaged to obtain a reference image at the in-focus position. The position of the lens where the obtained image matches the reference image is acquired, the autofocus function is turned off, and the acquired lens position is set as the in-focus position. The phase image acquisition system described in Appendix 4. (Note 6) The aforementioned camera does not have an autofocus function, or the autofocus function is turned off. The control unit sets the previously acquired focus position as the focus position. The phase image acquisition system described in Appendix 4. (Note 7) The reconstruction is performed using simulated annealing or quantum annealing, using the phase image acquisition system described in any one of the appendices 1 to 6. (Note 8) The phase image acquisition system according to any one of the appendices 1 to 7, wherein the mobile information terminal is a mobile information terminal in which the display and the camera are arranged on the same surface. (Note 9) The phase image acquisition system according to any one of the appendices 1 to 8, wherein the aforementioned mobile information terminal is a smartphone, a tablet device, or a small notebook computer. (Note 10) A phase image acquisition device for acquiring a phase image of a sample, The system comprises a display for illuminating the sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera. The control unit, The display is controlled to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample. The camera is controlled so as to sequentially capture the Fourier transform image of the sample when it is sequentially illuminated by each of the plurality of illumination patterns, using the light guided to the camera by an optical system that guides the light that has been illuminated by the display and passed through the sample to the camera. Multiple Fourier transform images captured by the aforementioned camera are reconstructed to generate a phase image with recovered phase. Phase image acquisition device. (Note 11) A program executed in a phase image acquisition device comprising a display for illuminating a sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera, wherein the device acquires a phase image of the sample, By computer, The steps include controlling the display to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample, The steps include controlling the camera so as to sequentially capture Fourier transform images of the sample when it is sequentially illuminated by each of the plurality of illumination patterns, using light guided to the camera by an optical system that guides light illuminated by the display and transmitted through the sample to the camera, The steps include: reconstructing multiple Fourier transform images captured by the aforementioned camera to generate a phase image with recovered phase; A program that executes the command. (Note 12) A phase image acquisition method for acquiring a phase image of a sample using a phase image acquisition system comprising: a portable information terminal equipped with a display for illuminating a sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera; and an optical system for guiding light that has been illuminated by the display and passed through the sample to the camera, wherein the phase image acquisition system acquires a phase image of the sample, The control unit, The display is controlled to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample. The camera is controlled to sequentially capture Fourier transform images of the sample when it is sequentially illuminated by each of the plurality of illumination patterns using light guided to the camera by the optical system, Multiple Fourier transform images captured by the aforementioned camera are reconstructed to generate a phase image with recovered phase. Phase image acquisition method. [Explanation of Symbols]

[0093] 10 Phase Image Acquisition System 20 Mobile Information Terminals 21 Cameras 22 Communications Department 23 displays 24 Memory section 24A Information Processing Program 24B Various Data 25 areas 26 lenses 27 Image Sensor 30 Inspection equipment 30A enclosure 30B Passing hole 30C passing hole 30D retaining groove 31 stages 31A Passing hole 31C Stage 31D Stage 31E Rotation axis 32 Drive unit 35 Communications Department 36 Control Unit 37 samples 39 Test Samples 41 Communication Cable 50 Control Unit 51 Focusing Processing Unit 52 Imaging Processing Unit 54 Reconstruction Processing Unit 60 Projection Lens 62 Reflector 64 Optical Shaping Elements 64d Transmitting Diffuser 64h hologram element 66 Photopolymer 68 lenses 80 Optical system 80A objective lens 81 Reflector 82 Optical system 83 Reflector 84 Optical system 84A imaging lens 85 Optical system P pattern Ip pattern image L optical axis L1 object light L2 Reference Light S1 Fourier surface S2 imaging plane

Claims

1. A phase image acquisition system for acquiring a phase image of a sample, A portable information terminal comprising a display for illuminating the sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera, An optical system that guides the light illuminated by the display and transmitted through the sample to the camera, Equipped with, The control unit, The display is controlled to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample. The camera is controlled to sequentially capture Fourier transform images of the sample when it is sequentially illuminated by each of the plurality of illumination patterns using light guided to the camera by the optical system, Multiple Fourier transform images captured by the aforementioned camera are reconstructed to generate a phase image with recovered phase. Phase image acquisition system.

2. The system further comprises a second optical system that guides illumination light from the display to the sample, The phase image acquisition system according to claim 1, wherein the second optical system comprises an imaging optical system that forms an image of the illumination pattern in front of the sample, and an optical shaping element that shapes the light directed from the image of the illumination pattern toward the sample into light that illuminates the entire sample.

3. The phase image acquisition system according to claim 2, wherein the optical shaping element is a transmission diffuser plate, a hologram element, an optical diffraction element, a photonic crystal, or a metamaterial.

4. The phase image acquisition system according to claim 1, further comprising: the control unit controlling the camera so that the focal position of the camera lens coincides with the imaging plane of the imaging lens included in the optical system.

5. The aforementioned camera is equipped with an autofocus function. The control unit, A known test sample is imaged to obtain a reference image at the in-focus position. The position of the lens where the obtained image matches the reference image is acquired, the autofocus function is turned off, and the acquired lens position is set as the in-focus position. The phase image acquisition system according to claim 4.

6. The aforementioned camera does not have an autofocus function, or the autofocus function is turned off. The control unit sets the previously acquired focus position as the focus position. The phase image acquisition system according to claim 4.

7. The phase image acquisition system according to claim 1, wherein the reconstruction is performed using simulated annealing or quantum annealing.

8. The phase image acquisition system according to claim 1, wherein the mobile information terminal is a mobile information terminal in which the display and the camera are arranged on the same surface.

9. The phase image acquisition system according to claim 1, wherein the mobile information terminal is a smartphone, a tablet device, or a small notebook computer.

10. A phase image acquisition device for acquiring a phase image of a sample, The system comprises a display for illuminating the sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera. The control unit, The display is controlled to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample. The camera is controlled so as to sequentially capture the Fourier transform image of the sample when it is sequentially illuminated by each of the plurality of illumination patterns, using the light guided to the camera by an optical system that guides the light that has been illuminated by the display and passed through the sample to the camera. Multiple Fourier transform images captured by the aforementioned camera are reconstructed to generate a phase image with recovered phase. Phase image acquisition device.

11. A program executed in a phase image acquisition device comprising a display for illuminating a sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera, wherein the device acquires a phase image of the sample, By computer, The steps include controlling the display to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample, The steps include controlling the camera so as to sequentially capture Fourier transform images of the sample when it is sequentially illuminated by each of the plurality of illumination patterns, using light guided to the camera by an optical system that guides light illuminated by the display and transmitted through the sample to the camera, The steps include: reconstructing multiple Fourier transform images captured by the aforementioned camera to generate a phase image with recovered phase; A program that executes the command.

12. A phase image acquisition method for acquiring a phase image of a sample using a phase image acquisition system comprising: a portable information terminal equipped with a display for illuminating a sample, a camera for imaging the sample illuminated by the display, and a control unit for controlling the display and the camera; and an optical system for guiding light that has been illuminated by the display and passed through the sample to the camera, wherein the phase image acquisition system acquires a phase image of the sample, The control unit, The display is controlled to sequentially illuminate the sample by sequentially displaying a plurality of illumination patterns, each illuminating the sample with light at a different angle of incidence to the sample. The camera is controlled to sequentially capture Fourier transform images of the sample when it is sequentially illuminated by each of the plurality of illumination patterns using light guided to the camera by the optical system, Multiple Fourier transform images captured by the aforementioned camera are reconstructed to generate a phase image with recovered phase. Phase image acquisition method.

Images