Image processing method, image processing device, and imaging apparatus

Abstract

To generate an image indicating information on an excellent depth direction with a sufficient field of view while using a general imaging optical system.SOLUTION: An image processing method includes the steps of: acquiring information about light intensity distribution of each of a plurality of optical images formed by causing light from a sample 103 illuminated at a plurality of illumination angles different from each other to enter an imaging optical system 120; acquiring three-dimensional information on the sample by calculation using the information about the light intensity distribution of each of the plurality of optical images; and generating an output image in which an observation direction is different from an optical axis direction of the imaging optical system from the three-dimensional information. In the generation step, the output image is generated by performing a conversion calculation for converting the spatial spectrum of the three-dimensional information on the basis of the illumination angles and the observation direction.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for generating images of a sample observed from various observation directions based on three-dimensional information of the sample. [Background technology]

[0002] In medical and drug discovery research and development, there is a demand for observing living samples such as tissue fragments and cell aggregates. In order to judge the quality of medical treatments and cell culture conditions, not only morphological information on individual cells contained in these samples but also the cell aggregate state observed across the entire sample is important information. In addition, cells do not only distribute within a plane but also grow overlapping each other, so information on the depth direction of the sample is also important.

[0003] To meet these observation conditions, a measurement system is required that has the ability to visualize unstained samples, a resolution that allows the shape of cells to be recognized, a wide field of view that allows the entire sample to be observed, and the ability to obtain depth information that allows overlapping cells to be recognized.In addition, the measurement system must be low-cost.

[0004] A low-cost 3D refractive index measurement technique that applies ptychography technology is disclosed in Patent Document 1 and Non-Patent Document 1. In these techniques, in the optical arrangement of a normal transmission microscope, the angle of incidence of the illumination light that illuminates the sample is changed successively to acquire multiple images, and the 3D refractive index distribution of the sample is calculated by reconstruction calculation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2018-508741 [Non-patent literature]

[0006] [Non-Patent Document 1] Chao Zuo, Jiasong Sun, Jiaji Li, Anand Asund, Qian Chena, “Wide-field high-resolution 3D microscopy with Fourier ptychographic diffraction tomography” Optics and Lasers in Engineering Vol.128, pp.106003, May 2020, USA Summary of the Invention [Problem to be solved by the invention]

[0007] By using the technology disclosed in Patent Document 1 and Non-Patent Document 1, a colorless and transparent sample can be visualized three-dimensionally with a simple configuration. However, in order to obtain sufficient image quality in the depth direction, it is necessary to increase the numerical aperture (NA) of the objective lens used for observation. As a result, the field of view is limited. Objective lenses with a wide field of view and large NA lead to high costs.

[0008] The present invention provides an image processing method and image processing device that can generate an image showing good depth information with a sufficient field of view while using a general imaging optical system. [Means for solving the problem]

[0009] An image processing method according to one aspect of the present invention includes a step of acquiring information on the light intensity distribution of each of a plurality of optical images formed by making the light from a sample illuminated at a plurality of different illumination angles enter an imaging optical system, a step of acquiring three-dimensional information of the sample by performing a calculation using the information on the light intensity distribution of each of the plurality of optical images, and a generating step of generating an output image from the three-dimensional information, the observation direction of which differs from the optical axis direction of the imaging optical system. The generating step is characterized in that the output image is generated by performing a conversion calculation for converting the spatial spectrum of the three-dimensional information based on the illumination angle and the observation direction. Note that a program for causing a computer to execute processing according to the above image processing method also constitutes another aspect of the present invention.

[0010] An image processing device according to another aspect of the present invention includes an acquisition means for acquiring information on the light intensity distribution of each of a plurality of optical images formed by making the light from a sample illuminated at a plurality of different illumination angles enter an imaging optical system, and acquiring three-dimensional information of the sample by calculation using the acquired information, and a generation means for generating an output image from the three-dimensional information, the observation direction of which is different from the optical axis direction of the imaging optical system. The generation means generates the output image by performing a conversion calculation for converting the spatial spectrum of the three-dimensional information based on the illumination angle and the observation direction. Note that an imaging device having the above-mentioned image processing device and an imaging unit for illuminating a sample at a plurality of different illumination angles and capturing a plurality of optical images also constitutes another aspect of the present invention. Effect of the Invention

[0011] According to the present invention, it is possible to generate an output image that shows good information in the depth direction with a sufficient field of view while using a general imaging optical system. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an imaging apparatus according to an embodiment. [Diagram 2] FIG. 4 is a diagram showing conditions under which diffracted light is generated. [Diagram 3] 11A and 11B are diagrams showing other conditions for generating diffracted light. [Figure 4] FIG. 13 is a diagram showing characteristics of a restored spatial spectrum. [Diagram 5] FIG. 1 illustrates a virtual optical system according to an embodiment. [Figure 6] FIG. 13 is a diagram showing a spatial spectrum acquired by a virtual optical system. [Figure 7] FIG. 13 is a diagram showing the characteristics of a spatial spectrum restored by a virtual optical system. [Figure 8] 4 is a flowchart showing image processing in the embodiment. [Figure 9] 11 is a flowchart showing a generation process of an output image in the embodiment. [Figure 10] 10 is a flowchart showing another process for generating an output image in the embodiment. [Figure 11] FIG. 2 shows a sample in Example 1. [Figure 12] FIG. 4 is a diagram showing an output image generated in the first embodiment. [Figure 13] FIG. 11 is a diagram showing an imaging apparatus according to a second embodiment. [Figure 14] 10 is a flowchart showing a generation process of an output image in the second embodiment. [Figure 15] FIG. 11 is a diagram showing an imaging apparatus according to a third embodiment. [Figure 16] FIG. 13 is a diagram showing an imaging apparatus according to a fourth embodiment. [Figure 17] 13 is a flowchart showing a generation process of an output image in the fourth embodiment. [Figure 18] FIG. 13 is a diagram showing an imaging apparatus according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0014] 1 shows the configuration of an image capturing system according to an embodiment of the present invention. The image capturing system includes an image capturing unit 100, a computer 200, and a display unit 300. The image capturing unit 100 and the computer 200 constitute an image capturing apparatus.

[0015] The imaging section 100 has an illumination section 110, an imaging optical system 120, and an image sensor 106, and captures an image of a sample 103. The illumination section 110 has an illumination light source 101 and a rotating stage 102. Illumination light emitted from the illumination light source 101 is irradiated onto the sample 103 to illuminate it. The illumination light source 101 is moved (rotated) by the rotating stage 102, thereby changing the illumination angle θi, which is the angle at which the illumination light is irradiated onto the sample 103. The illumination light source 101 uses a light-emitting element such as an LED or a laser.

[0016] Of the illumination light, light that passes through the sample 103 enters the imaging optical system 120. The imaging optical system 120 has an optical axis AXL. The light that enters the imaging optical system 120 forms an optical image of the sample 103 on the image sensor 106. The imaging optical system 120 is composed of an objective lens 104 and an imaging lens 105.

[0017] The imaging element 106 is a two-dimensional photoelectric conversion element array such as a CCD sensor or a CMOS sensor, and captures an optical image formed by the imaging optical system 120. The image data generated by the imaging element 106 includes information regarding the light intensity distribution of the optical image. The information regarding the light intensity distribution may be information that directly indicates the light intensity distribution, or may be information that can be converted into the light intensity distribution. In the following description, the information regarding the light intensity distribution is simply referred to as the light intensity distribution.

[0018] The image data (information regarding the light intensity distribution) is sent to a computer 200. The computer 200 executes image processing, which will be described later, on the image data using an image processor, which is an internal image processing device.

[0019] The operations of the image sensor 106 and the rotating stage 102 are controlled by a computer 200 as a control device, whereby the illumination angle θi is changed and the light intensity distributions of multiple optical images of the sample 103 illuminated at different illumination angles θi are acquired. The computer 200 calculates (generates) an output image to be provided to a user from the light intensity distributions of the multiple acquired optical images. The calculated output image is displayed on a display unit 300 such as an LCD. The output image referred to here is an image obtained when the sample 103 is observed from an observation direction different from the optical axis direction of the imaging optical system 120 (inclined with respect to the optical axis direction).

[0020] Next, a description will be given of a process (image processing method) executed by the computer 200 according to a program. The process of this embodiment can be mathematically modeled, and therefore can be executed by installing software including the program in a computer.

[0021] First, the computer 200 as an acquisition means causes the image capturing section 100 to illuminate the sample 103 from a plurality of different illumination angles, sequentially capture a plurality of optical images from the different illumination angles, and acquires the light intensity distribution of the plurality of optical images.

[0022] Next, the computer 200, also serving as an acquisition means, calculates information on the three-dimensional refractive index distribution as three-dimensional information of the sample 103 by performing an operation using the light intensity distributions of the multiple optical images. The information on the three-dimensional refractive index distribution referred to here may be information that directly indicates the three-dimensional refractive index distribution, or may be information that can be converted into a three-dimensional refractive index distribution. In the following description, the information on the three-dimensional refractive index distribution is simply referred to as the three-dimensional refractive index distribution.

[0023] Next, the computer 200 as a generating means calculates an output image including information on the depth direction of the sample 103 from the calculated three-dimensional refractive index distribution.

[0024] Here, a specific method for calculating the three-dimensional refractive index distribution will be described. The reconstruction calculation disclosed in Patent Document 1 or Non-Patent Document 1 can be used for the calculation method of the three-dimensional refractive index distribution. For simplicity of explanation, it is assumed that the illumination light emitted from the illumination light source 101 is a monochromatic parallel light with a wavelength λ. When the illumination light is irradiated onto the sample 103, diffracted light is generated due to the refractive index distribution n(x, y, z) of the sample 103. x and y respectively indicate coordinates in two axial directions that are perpendicular to the optical axis AXL of the imaging optical system 120 and perpendicular to each other. z is the coordinate in the optical axis direction (depth direction). The intensity and direction of the generated diffracted light are determined by the illumination angle θ of the illumination light. i and the three-dimensional spatial spectrum of the refractive index n ~ (k x ,k y ,k z ) is determined by k → =(k x ,k y ,k z ) are coordinates in the x, y, and z directions in wave number space. Since wave numbers and frequencies can be converted with a constant multiple relationship, no particular distinction is made between wave numbers and frequencies in this embodiment.

[0025] The conditions for generating diffracted light will be explained with reference to FIG. 2. For the sake of simplicity, k y The following explanation will be given using the xz cross section at =0. x ,k z ) represents the amount of spectral components contained in the sample, with the darker the value being the larger.

[0026] k i → =(k x,i ,k y,i ,k z,i ) is the wave vector of the illumination light, and k d → =(k x,d ,k y,d ,k z,d ) is the wave vector of the diffracted light. Figure 2 shows that the illumination light is incident perpendicularly (in the direction of the optical axis) on the sample (θ i = 0). From diffraction theory, it is known that the condition for diffracted light to occur is expressed by the following formula 1.

[0027]

number

[0028] Here, K → =(K x ,K y ,K z ) is the wave number at which the spatial spectrum of the sample is non-zero. d → The region where the value of -k is changed within the range of possible values ​​corresponds to the dashed area in Figure 2. i → Centered on a radius of |k d → This sphere is generally called the Ewald sphere. Equation (1) states that if the spectral components of the sample exist on the Ewald sphere, k d → This means that diffracted light occurs in the direction of the diffraction angle θ d The wave vector k of the diffracted light d →Since the spectral components of the sample are non-zero at the ends of θ d Diffracted light is generated in the direction of k. Furthermore, the intensity of the diffracted light is k d → It is determined by the amount of spectral components at the ends of the spectrum.

[0029] The generated diffracted light is incident on the imaging optical system 120. At this time, the maximum angle of the diffracted light that can be acquired by the imaging optical system 120 is determined by the numerical aperture NA, and is expressed by equation (2).

[0030] -NA≦sinθ d ≦NA (2) For simplicity, the angle is shown in the xz section here, but this can be easily extended to the three-dimensional case by adding equations for the angle in the yz section or the angle in the xy section.

[0031] Since only light propagating in the z direction enters imaging optics 120, the condition in equation (3) is also imposed.

[0032] k z,d ≧0 (3) The area on the Ewald sphere that satisfies equations (2) and (3) is the spatial spectrum of the sample that contributes to imaging. If this area is F1, it can be expressed as equation (4).

[0033]

number

[0034] The region F1 is the curved portion indicated by a thick solid line in FIG. JPEG2025071885000004.jpg107

[0035] k z Next, the illumination light is incident on the sample at an angle with respect to the optical axis of the imaging optical system (θ i ≒0) will be explained with reference to FIG. 3. From the right side of equation (1), when the illumination light has an angle with respect to the optical axis direction, the center position of the Ewald sphere is located at the radius |kl → | moves along the spherical surface (curved surface). On the other hand, since the position of the imaging optical system does not change, the conditions for the diffracted light entering the imaging optical system are the same as in the case of normal incidence, as expressed by equations (2) and (3). Therefore, the area F1 shown in equation (4) also moves along the spherical surface (curved surface) of radius |k l → Move along the spherical surface of |.

[0036] When measuring, the lighting angle θ i The light intensity distribution is obtained by successively changing the illumination angle θ. The spectral components of the sample obtained by a series of measurements are the set of the thick solid lines shown in Figure 4. By utilizing this property in reverse, the three-dimensional spatial spectrum of the sample is restored (obtained) by calculation. That is, i The light intensity distribution obtained at each illumination angle θ i The spatial spectrum of the sample is reconstructed so that all the light intensity distributions are reproduced, assuming that the spectral components of the sample in the area indicated by the thick solid line determined by are caused by the spectral components of the sample in the area indicated by the thick solid line determined by the above equation. The reconstruction calculation in this case can be performed by repeating the Fourier transform or by using an optimization calculation.

[0037] Based on the above principle, the three-dimensional refractive index distribution of a sample can be obtained.

[0038] Here, we will discuss the characteristics of the reconstructed spatial spectrum. As explained above, the center of the Ewald sphere is located at a radius of |k l → |Illumination angle θ at which the measurement was performed on the sphere i Therefore, the area F1 also moves along the same path as shown by the dashed line in FIG. 4. The illumination angle θ i is ±θ i,max Assuming that the region F1 changes within the range, the formula F2 representing the dashed line in FIG. 4 along which the region F1 moves is expressed by the formula (5).

[0039]

number

[0040] In equation (5), NA i =sin(θ i,max ) The set of spectral components that contribute to the restored image is the sum of the region F1 shifted according to F2. In other words, it is the region obtained by convolving F1 and F2. Therefore, if the spectral region that contributes to the image formation is F, F can be calculated using F1 and F2 according to formula (6).

[0041]

number

[0042] In equation (6), * denotes a convolution operation. The region represented by F determines the characteristics of the reconstructed three-dimensional refractive index distribution n(x,y,z).

[0043] This region F is the area surrounded by the dotted line in Figure 4. The resolution in the depth direction is determined by the k z The width in the direction, i.e., the part surrounded by the dotted line in Fig. 4, z The resolution in the depth direction is determined by the width of the image. If the NA of the imaging optical system is reduced to widen the observation field, this width also becomes smaller, and the resolution in the depth direction decreases. Furthermore, in the area surrounded by the dotted line in Figure 4, z In the low-frequency region in the direction, there are regions that cannot be restored to a cone shape. Due to the lack of information in these regions, the calculated output image is an image that is extended in the depth direction, which is significantly different from the actual shape of the sample.

[0044] In this embodiment, output images having different observation directions for observing the sample are generated from the calculated three-dimensional refractive index distribution n(x, y, z), thereby providing information in the depth direction to the user. By calculating output images in two different observation directions, a pair of output images (parallax images) having a parallax difference with each other can be calculated. Since the parallax images are information in a form that conforms to the normal spatial recognition method of humans, the user can easily recognize information in the depth direction by showing the calculated parallax images to the left and right eyes of the user. In addition, information in the depth direction can also be provided by generating output images in which the observation direction is continuously changed or output images corresponding to the observation direction input by the user and sequentially displaying them on the display unit 300.

[0045] Furthermore, by using a frequency filter based on imaging theory to calculate the output image, it is possible to generate a high-quality output image with few artifacts.

[0046] The calculation principle of the output image in the observation direction tilted with respect to the optical axis direction of the imaging optical system will be described below. To calculate the output image, first consider a virtual imaging optical system 130 having an optical axis AXL' tilted at an angle φ with respect to the optical axis AXL of the actual imaging optical system 120 as shown in Fig. 5. The three-dimensional refractive index distribution n(x,y,z) obtained by the above-mentioned principle is converted into a refractive index distribution n'(x,y,z) restored by the virtual imaging optical system 130, and an image of the desired xy cross section is extracted from it to obtain an output image corresponding to the observation direction.

[0047] The angular range of diffracted light acquired by the virtual imaging optical system 130 is different from that of the actual imaging optical system 120. If the angle of the observation direction is φ, then changing the observation direction (φ) is equivalent to tilting the optical axis of the virtual imaging optical system 130 by the angle φ. If the numerical aperture of the virtual imaging optical system 130 is NA', then the angular range of diffracted light incident on the virtual imaging optical system 130 changes from the above formula (2) to the following formula (7).

[0048] -NA′≦sinθ d -sinφ≦NA′ (7) On the other hand, the conditions for the generation of diffracted light shown in the above formula (1) and the conditions related to the propagation direction shown in formula (3) do not change. Therefore, formula F1, which determines the spectral components that contribute to imaging, changes to F1' shown in the following formula (8).

[0049]

number

[0050] F1' corresponds to the curved portion indicated by the thick line in Fig. 6. For ease of explanation, it is assumed that the illumination conditions do not change in the virtual imaging optical system 130. Therefore, F2', which represents the trajectory along which F1' moves, is the same as F2 shown in the above formula (5).

[0051] When the virtual imaging optical system 130 is used, the method of restoring the three-dimensional refractive index distribution can be applied in the same way as the actual imaging optical system 120. Therefore, the spectral components of the restored refractive index distribution are determined by replacing F1 with F1' and F2 with F2' in the above formula (6). This is expressed by the following formula (9).

[0052]

number

[0053] The region represented by F' calculated by Equation 9 is the area surrounded by the dotted line in Figure 7. The spectral region F1' that can be acquired at one illumination angle corresponds to one of the thick solid lines in Figure 7, and the region where this thick solid line moves along the trajectory represented by F2' (dotted line in Figure 7) corresponds to the dotted line in Figure 7. The region surrounded by this dotted line is the spatial spectrum n of the three-dimensional refractive index distribution n'(x, y, z) obtained in the observation direction (φ). ~ ′(k x ,k y ,k z ) For this reason, if we generate a spatial spectrum that has values ​​in the region enclosed by the dotted line in Figure 7 and has spectral components that are zero in other regions or have smaller values ​​than the other regions, we can generate a three-dimensional refractive index distribution n'(x,y,z) in the observation direction (φ).

[0054] The spatial spectrum of the sample is obtained from the measured light intensity distributions. ~ (k x ,k y ,k z ) has been restored, so this restored spatial spectrum n ~ (k x ,k y ,k z ) and extract the area enclosed by the dotted line in Figure 7, which is determined by F', and perform an inverse Fourier transform on this. This allows the three-dimensional refractive index distribution n'(x, y, z) in the observation direction (φ) to be calculated. In other words, the restored spatial spectrum n ~ (k x ,k y ,k z ) based on F'. Then, by extracting and displaying the desired xy cross section, it is possible to provide the user with an output image in the observation direction (φ).

[0055] F' used in this filtering process is a three-dimensional frequency filter (hereinafter simply referred to as a frequency filter), and F1' used to calculate F' is the first frequency filter (hereinafter referred to as frequency filter 1), and F2' is the second frequency filter (hereinafter referred to as frequency filter 2).

[0056] In this embodiment, a case will be described in which the frequency filter F' is generated based on the numerical aperture, which is one of the imaging characteristics of the imaging optical system. However, the frequency filter F' may be generated based on other imaging characteristics of the imaging optical system, such as the focal length and magnification, without being limited to the numerical aperture.

[0057] The image processing method of this embodiment based on the above principle will be described with reference to the flowchart of Fig. 8. This image processing method is executed by the computer 200 in accordance with a program.

[0058] In step S1, the rotation stage 102 is driven to illuminate the sample 103 at an illumination angle θ i Change the setting.

[0059] In step S2, the light intensity distribution of the optical image of the sample 103 formed on the image sensor 106 by the imaging optical system 120 is acquired.

[0060] In step S3, the illumination angle θ i If the number of measurements does not reach the predetermined number, steps S1 and S2 are repeated. If the number of measurements has been reached the predetermined number, the process proceeds to step S4.

[0061] In step S4, the three-dimensional refractive index distribution n(x, y, z) of the sample 103 is reconstructed by a reconstruction calculation by the computer 200 from the acquired light intensity distributions.

[0062] In step S5, an output image in an arbitrary observation direction is generated by the computer 200 from the restored three-dimensional refractive index distribution n(x, y, z) of the sample.

[0063] In step S6, the generated output image is displayed on the display unit 300.

[0064] Next, a method for generating an output image in step S5 of FIG. 8 will be described with reference to the flowchart shown in FIG.

[0065] In step S51, the three-dimensional spatial spectrum n of the sample is calculated from the reconstructed three-dimensional refractive index distribution n(x,y,z) of the sample. ~ (k x ,k y ,k z ) is calculated. The spatial spectrum can be calculated by using a fast Fourier transform (FFT), but it can also be calculated by using a discrete cosine transform or the like, as long as it is an operation that performs frequency conversion. Usually, in step S4 of FIG. 8, the spatial spectrum n ~ (k x ,k y ,k z ) is calculated, this step can be omitted.

[0066] In step S52, the calculated three-dimensional spatial spectrum n ~ (k x ,k y ,k z ) is the spatial spectrum n ~ ′(k x ,k y ,k z ) In this step, the spatial spectrum of the sample is transformed to give a frequency characteristic denoted as F'.

[0067] In step S53, the transformed spatial spectrum of the sample n ~ ′(k x ,k y ,k z ) is converted into an image. ~ ′(k x ,k y ,k z ) into a three-dimensional refractive index distribution n'(x,y,z) in real space can be converted using an inverse fast Fourier transform (IFFT). Alternatively, an inverse discrete cosine transform or the like may be used, and any operation corresponding to the inverse transform of the frequency transform performed in step S51 may be used. After calculating n'(x,y,z), a desired xy cross section is cut out to calculate an output image to be provided to a user.

[0068] In this case, in order to reduce the calculation load, the inverse transformation process may be partially omitted. For example, when calculating the output image at the position where z = 0, due to the characteristics of the Fourier transform, it is not necessary to perform IFFT in the z direction, and it is sufficient to take the sum of the spectrum in the z direction. When calculating the output image at a certain z, exp(ik z z) and then take the sum in the z direction.

[0069] The method of converting the spatial spectrum in step S52 in FIG. 9 will be described with reference to the flowchart shown in FIG.

[0070] In step S521, the frequency filter 1 (F1') is set based on the above formula (8). Formula (8) uses the δ function, rect function, and step function to explain the principle, but it is not necessary to reproduce these functions strictly in the actual calculation process. For example, regarding the δ function, strictly speaking, the variable part of the δ function,

[0071]

number

[0072] This would require preparing an array with infinite values ​​at the coordinates where the value of equation (10) is 0, but in practice this is not necessary. This can be achieved by assigning finite values ​​to the coordinates where the value of equation (10) is 0 or to an array within an appropriate range from there. 1 can be used as the finite value, but other values ​​can also be used. Similarly, there is no need to reproduce rect functions and step functions exactly. It is sufficient to prepare a data array that reproduces part of a curved surface (for example, a sphere) as F1'.

[0073] In step S522, a frequency filter 2 (F2') is set based on the above formula (5). As in the description in step S521, the frequency filter 2 does not need to strictly follow formula (5), and it is sufficient if a data array that reproduces a part of a curved surface (e.g., a spherical surface) can be reproduced.

[0074] In step S523, a frequency filter (F') is calculated from frequency filter 1 (F1') and frequency filter 2 (F2'). This calculation may be performed by executing the convolution operation shown in equation (9) or an operation equivalent thereto.

[0075] In step S524, the spatial spectrum of the sample is filtered using the calculated frequency filter (F′). ~ ′(k x ,k y ,k z) by a frequency filter (F'). However, multiplication by a frequency filter is not necessarily required, and it is sufficient if the values ​​outside the area surrounded by the dotted line in Fig. 7 are 0 or are smaller than the values ​​within said area. For example, a process may be used in which the spatial spectrum in the area where the frequency filter (F') is 0 is converted to 0 or reduced to be smaller than the values ​​within said area.

[0076] The frequency filter 1 (F1'), frequency filter 2 (F2') and frequency filter (F') used in steps S521 to S524 are appropriately modified in accordance with the frequency conversion executed in steps S51 and S53 in FIG.

[0077] 8 to acquire light intensity distributions of a plurality of optical images with different illumination angles may not be executed by the computer 200 itself, but may be executed by a separate control device controlled by the computer 200. Moreover, the processes of steps S4 and S5 may also be executed by the computer 200 itself, or may be executed by a separate computing device or on the cloud.

[0078] The display unit 300 is preferably capable of stereoscopic display. For example, a pair of display devices corresponding to the right and left eyes of the user, such as a head mounted display, may be provided, and output images with different viewing directions may be displayed for the right and left eyes. Alternatively, a device capable of displaying output images with different viewing directions depending on the direction from the display unit, such as a 3D television, may be used to display output images with different viewing directions for the right and left eyes.

[0079] A more specific example will now be described.

[0080] [Example 1] As Example 1, an experimental example using specific numerical values ​​will be described. In this example, two objects shown in FIG. 11 were used as samples. The two objects have a cylindrical shape with a bottom diameter of 30 μm and a height of 6 μm, and the refractive index is 1.5. The left object was placed at a height of 0 μm, and the right object was placed at a height of 15 μm. These two objects were placed in a medium with a refractive index of 1.49. The light intensity distribution of these samples was acquired by the imaging device shown in FIG. 1, and the computer 200 restored the three-dimensional refractive index distribution n(x, y, z) and generated an output image in the observation direction. The numerical aperture NA of the imaging optical system 120 was set to 0.15, and the wavelength of the illumination light was set to 0.5 μm.

[0081] 8, multiple light intensity distributions at different illumination angles are obtained, and the three-dimensional refractive index distribution n(x, y, z) of the sample is reconstructed in step S4. i The light intensity distribution was measured by changing the angle within a range of ±17.5°.

[0082] Next, in step S5, an output image in the observation direction was generated from the restored three-dimensional refractive index distribution n(x, y, z). In this case, an imaging optical system tilted at angles φ=5° and −5° was assumed as the virtual imaging optical system 130. In step S4, the spatial spectrum n ~ (k x ,k y ,k z ) has already been calculated, the process of step S51 in FIG. 9 is omitted.

[0083] In step S521 of Fig. 10, frequency filter 1 (F1') was set by equation (8). Then, in step S522, frequency filter 2 (F2') was set by equation (5) with the illumination angle in the range of ±17.5°. Furthermore, in step S523, a convolution operation of frequency filter 1 (F1') and frequency filter 2 (F2') was performed to calculate frequency filter (F'). To speed up the operation, both frequency filter 1 (F1') and frequency filter 2 (F2') were frequency converted by IFFT based on the convolution theorem, and FFT was performed after taking the product of both.

[0084] Next, in step S524, n is set so that the spatial spectrum of the sample becomes 0 in the region where the value of the obtained frequency filter (F') becomes 0. ~ (k x ,k y ,k z ) to n ~ ′(k x ,k y ,k z ) was converted to

[0085] Furthermore, in step S53 of FIG. 9, the converted spatial spectrum n ~ ′(k x ,k y ,k z ) the three-dimensional refractive index distribution n'(x,y,z) was calculated by IFFT. The output images of the xy cross section at z=0 of the three-dimensional refractive index distribution n'(x,y,z) calculated in this way are shown in Figures 12(a) and (b). Figure 12(a) is the output image calculated at φ=5°, and Figure 12(b) is the output image calculated at φ=-5°.

[0086] Comparing Figures 12(a) and (b), the object on the left side, which is located at z=0, i.e., the in-focus position, does not change its position depending on the observation direction (5°, -5°). In contrast, the object on the right side, which is located at z=15 μm, i.e., the out-of-focus position, has different positions in the left and right directions in Figures 12(a) and (b). In other words, Figures 12(a) and (b) show images with parallax.

[0087] Moreover, Figure 12(c) shows the cross section at y=z=0 of the three-dimensional refractive index distribution n'(x,y,z). The dashed line shows the cross section at φ=5°, and the solid line shows the cross section at φ=-5°. The refractive index profile of the object on the left, which is in the focal position, shows almost the same shape regardless of the observation direction (φ). In contrast, the refractive index of the object on the right, which is placed in a non-focal position, shows a profile that shifts left and right depending on the observation direction (φ). This result also shows that an output image with parallax was generated.

[0088] The two output images thus generated are displayed to the right and left eyes of the user on the display unit 300, allowing the user to recognize information about the depth of the sample.

[0089] As described above, the image processing does not have to be performed by the computer 200. For example, the light intensity distribution acquired by the image capturing unit 100 may be transmitted to a computing device present on the cloud via a network, and the computing device may perform the image processing. In this manner, the image processing may be performed by a computing device capable of data communication with the image capturing unit 100.

[0090] Furthermore, the display unit 300 does not have to be directly connected to the computer 200, and a display device that is located at a different location from the computer 200 via a network may be used as the display unit 300.

[0091] In addition, the illumination angle when imagining the virtual imaging optical system 130 may be different from the illumination angle during actual measurement using the imaging optical system 120. That is, the frequency filter 2 (F2') set in step S522 may be different from F2. However, if the illumination angle deviates too much from the illumination angle used when measuring the light intensity distribution, the three-dimensional refractive index distribution n'(x, y, z) obtained by the virtual imaging optical system 130 will change significantly from the restored n(x, y, z), which is not preferable. It is preferable to set an illumination angle close to the illumination angle used when measuring the light intensity distribution for the frequency filter 2 (F2').

[0092] Moreover, the above equation (5) is based on the illumination angle θ i is ±θ i,max However, the illumination angle θ i The illumination angle θ may be changed in other ranges. i It is possible to change the formula (5) appropriately according to the range in which θ changes. For example, the rect function in the formula (5) can be changed based on the illumination angle used when measuring the light intensity distribution. As a more specific example, when the illumination angle used when measuring is from 0 to +θ i,maxConsidering the case where it is within the range of, the rect function may be replaced with a function that takes 1 in the range where kx is from 0 to NAi / λ.

[0093] Furthermore, although the numerical aperture NA′ of the virtual imaging optical system 130 shown in the above formula (7) is different from the numerical aperture NA of the imaging optical system 120 shown in formula (2), it may be set as NA′ = NA. However, in this case, the virtually tilted imaging optical system 130 will acquire diffracted light outside the diffraction angle acquired by the actual imaging optical system 120. That is, the restored three-dimensional refractive index distribution n(x, y, z) does not have the refractive index information required by the virtually tilted imaging optical system 130. Therefore, it is preferable that NA′ < NA.

[0094] Also, setting NA′ = NA - |sinφ| is more preferable because an output image in the observation direction (φ) can be calculated based on the acquired information.

[0095] [Example 2] In FIG. 5, a method of assuming a virtual imaging optical system 130 tilted with respect to the actual imaging optical system 120 was described, but other methods may also be used. In Example 2, as shown in FIG. 13, an output image in the observation direction (φ) is generated by assuming a sample 103 virtually rotated by an angle φ corresponding to the observation direction. N in the figure indicates the normal line of the surface of the sample 103 (such as a prepared slide) on the imaging optical system side, and the angle φ is the angle formed by the normal line N with respect to the optical axis AXL of the imaging optical system 120.

[0096] A flowchart of the process for generating the output image in this example is shown in FIG. 14. In FIG. 14, a step S525 for rotating the frequency filter F′ is added between the step S532 and the step S523 shown in FIG. 10.

[0097] [Example 3] In FIG. 1, an illumination unit 110 that rotates the illumination light source 101 by a rotation stage 102 is shown, but if the illumination angle with respect to the sample 103 can be changed, an illumination unit with other configurations may also be used.

[0098] In the third embodiment, as shown in FIG. 15, an illumination device in which a plurality of point light sources 107 are arranged in an array is used as the illumination unit 110. The illumination angle with respect to the sample 103 can be changed by sequentially switching the light sources that are turned on among the plurality of point light sources 107. Alternatively, a configuration in which one point light source 107 is translated may be used. A light-emitting element such as an LED may be used as the point light source. As another illumination unit 110, a device that can change the illumination angle with respect to the sample 103 by using a movable mirror such as a galvanometer mirror may be used.

[0099] Moreover, the illumination light does not have to be parallel light as described above. As shown in Fig. 15, the sample 103 may be illuminated by a spherical wave emitted from a point light source 107. In this case, calculations can be performed using the illumination angle determined by the geometric arrangement of the sample 103 and the point light source 107.

[0100] Furthermore, the illumination light does not have to be monochromatic as described above, and may be light with a range of wavelengths, such as light from an LED. In this case, calculations can be performed using a representative wavelength.

[0101] [Example 4] The imaging device as the fourth embodiment may include an observation direction input unit 400 that allows a user to input (specify) information about the observation direction, as shown in Fig. 16. The observation direction input unit 400 may be a device that allows a user to input a numerical value corresponding to the observation direction, such as a keyboard, or a device that allows a user to input (select) the observation direction by clicking, such as a mouse. A joystick may also be used, or a device that allows a user to input the observation direction or information that can be converted thereto, through a sensor that can be built into a head mounted display and detects the tilt or acceleration of the user's head.

[0102] In a method of generating an output image when the observation direction input unit 400 is provided, as shown in the flowchart of Fig. 17, step S54 for acquiring an observation direction input by a user is provided between steps S51 and S52 shown in Fig. 9. In step S54, the computer 200 acquires the observation direction input by the observation direction input unit 400. Then, in step S52, a calculation is performed based on the acquired observation direction.

[0103] At this time, the computer 200 may repeatedly execute the processes of steps S5 and S6 shown in Fig. 8. The computer 200 may periodically communicate with the observation direction input unit 400 and re-execute the processes of steps S5 and S6 based on the acquired observation direction. Furthermore, when the input observation direction is changed, the computer 200 may re-execute the processes of steps S5 and S6.

[0104] Furthermore, if a joystick is used as the observation direction input unit 400, the observation direction input unit 400 may output a signal when the inclination of the stick changes, and the computer 200 receiving this signal may execute the processing of steps S5 and S6.

[0105] Based on a signal from the observation direction input unit 400, output images with different observation directions are successively calculated and successively displayed on the display unit 300, allowing the user to confirm the state of the sample 103 according to different observation directions.

[0106] The two samples with different depth positions described in the first embodiment change their positions in the xy cross section depending on the observation direction, so the user can observe the movement of the image caused by changing the observation direction, and can recognize information about the depth direction from the movement. In such a case, the display unit 300 may be a normal display.

[0107] [Example 5] In FIG. 1, the case where light (optical image) that is transmitted through the sample 103 out of the illumination light is captured has been described, but light that is reflected from the sample out of the illumination light may also be captured.

[0108] When imaging reflected light, the component F1 that contributes to imaging among the diffracted light generated by the sample 103 differs from that when imaging transmitted light. When imaging reflected light, only the diffracted light propagating in the reverse direction enters the imaging optical system 120, so the condition of the above formula (3) changes to the condition of the following formula (11).

[0109] k z,d <= 0 (11) Therefore, if a frequency filter 1 expressed by the following equation (12) is used as F1', an output image in the observation direction (φ)φ can be generated by a calculation method similar to that used when capturing an image of transmitted light.

[0110]

number

[0111] In this way, the frequency filter 1 (F1') may be changed depending on the imaging method.

[0112] [Example 6] Although a general coaxial imaging optical system 120 is shown in Fig. 1, imaging optical systems of other configurations may be used. Fig. 18 shows the configuration of a digital holography microscope which is an imaging device according to a sixth embodiment.

[0113] In digital holography, the phase distribution of diffracted light can be obtained by interference measurement. In FIG. 18, the illumination angle with respect to sample 103 is changed by galvanometer mirror 108. From the obtained phase distribution and illumination angle information, the three-dimensional refractive index distribution n(x, y, z) of sample 103 can be reconstructed by reconstruction calculation. Since the three-dimensional refractive index distribution n(x, y, z) reconstructed in the digital holography microscope has the same imaging characteristics as in this embodiment, an output image in the observation direction can be generated by the same principle.

[0114] In this embodiment, the imaging optical system is not a coaxial optical system as in the other embodiments, but includes a reflecting surface between the objective lens 104 and the imaging lens 105. The optical axis direction of the imaging optical system in this case corresponds to the direction of the portion (AXL) from the objective lens 104 to the reflecting surface among the axes passing through the centers of the objective lens 104 and the imaging lens 105.

[0115] [Example 7] In the above embodiment, the three-dimensional information restored by the calculation is the three-dimensional refractive index distribution n(x, y, z) of the sample, but other three-dimensional information may be restored. For example, if the sample contains a substance that absorbs light, the three-dimensional information restored is the three-dimensional complex refractive index distribution For samples dominated by absorption, the reconstructed 3D information is the 3D distribution of the absorption coefficient. In either case, an output image in the observation direction can be generated by performing calculations on the reconstructed 3D information of the sample.

[0116] The above embodiments include the following methods and configurations.

[0117] (Method 1) acquiring information about the light intensity distribution of each of a plurality of optical images formed by making the light from the sample illuminated at a plurality of different illumination angles incident on an imaging optical system; acquiring three-dimensional information of the sample by performing a calculation using information on the light intensity distribution of each of the plurality of optical images; A generating step of generating an output image having an observation direction different from the optical axis direction of the imaging optical system from the three-dimensional information, An image processing method, characterized in that in the generating step, the output image is generated by performing a conversion operation to convert a spatial spectrum of the three-dimensional information based on the illumination angle and the observation direction. (Method 2) The image processing method according to method 1, characterized in that the conversion operation is a filtering process based on the illumination angle and the observation direction for the spatial spectrum. (Method 3) The image processing method according to Method 2, wherein the filtering process uses a three-dimensional frequency filter calculated based on the illumination angle and the observation direction. (Method 4) The image processing method described in Method 3, characterized in that the three-dimensional frequency filter is generated using a first frequency filter determined based on the observation direction and a second frequency filter determined based on the illumination angle. (Method 5) The image processing method according to method 4, wherein the three-dimensional frequency filter is generated by a convolution process of the first frequency filter and the second frequency filter. (Method 6) The image processing method according to Method 4 or 5, wherein the first frequency filter is part of a curved surface defined based on the observation direction. (Method 7) 7. The image processing method according to claim 6, wherein the curved surface is a spherical surface. (Method 8) 8. The image processing method according to any one of methods 4 to 7, wherein the three-dimensional frequency filter is generated based on the imaging characteristics of the imaging optical system. (Method 9) 9. The image processing method according to method 8, wherein the imaging characteristics include at least one of a numerical aperture, a focal length, and a magnification of the imaging optical system. (Method 10) 10. The image processing method according to any one of configurations 1 to 9, further comprising the step of acquiring input information about the observation direction. (Configuration 1) an acquisition means for acquiring information regarding the light intensity distribution of each of a plurality of optical images formed by making the light from the sample illuminated at a plurality of different illumination angles incident on the imaging optical system, and acquiring three-dimensional information of the sample by performing a calculation using the acquired information; a generating means for generating an output image having an observation direction different from the optical axis direction of the imaging optical system from the three-dimensional information; The image processing device according to claim 1, wherein the generating means generates the output image by performing a conversion operation for converting a spatial spectrum of the three-dimensional information based on the illumination angle and the observation direction. (Configuration 2) The image processing device according to configuration 1, an imaging unit that captures the plurality of optical images by illuminating the sample at a plurality of illumination angles different from each other, The imaging device, wherein the image processing device acquires information regarding the light intensity distribution from an output of the imaging section. (Configuration 3) The imaging device according to configuration 2, an input unit for inputting information regarding the observation direction; and a display unit that displays the output image.

[0118] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0119] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]

[0120] 100 Imaging unit 103 Samples 106 Image sensor 110 Lighting Department 120 Imaging Optical System 130 Virtual Imaging Optical System 200 Computers 300 Display

Claims

1. acquiring information about the light intensity distribution of each of a plurality of optical images formed by making the light from the sample illuminated at a plurality of different illumination angles incident on an imaging optical system; acquiring three-dimensional information of the sample by performing a calculation using information on the light intensity distribution of each of the plurality of optical images; a generating step of generating an output image having an observation direction different from the optical axis direction of the imaging optical system from the three-dimensional information; An image processing method, characterized in that in the generating step, the output image is generated by performing a conversion operation that converts a spatial spectrum of the three-dimensional information based on the illumination angle and the observation direction.

2. 2. The image processing method according to claim 1, wherein the conversion operation is a filtering process based on the illumination angle and the observation direction for the spatial spectrum.

3. 3. The image processing method according to claim 2, wherein the filtering process uses a three-dimensional frequency filter calculated based on the illumination angle and the observation direction.

4. 4. The image processing method according to claim 3, wherein the three-dimensional frequency filter is generated using a first frequency filter determined based on the observation direction and a second frequency filter determined based on the illumination angle.

5. 5. The image processing method according to claim 4, wherein the three-dimensional frequency filter is generated by a convolution process of the first frequency filter and the second frequency filter.

6. 5. The image processing method according to claim 4, wherein the first frequency filter is a part of a curved surface that is determined based on the observation direction.

7. 7. The image processing method according to claim 6, wherein the curved surface is a spherical surface.

8. 5. The image processing method according to claim 4, wherein the three-dimensional frequency filter is generated based on the imaging characteristics of the imaging optical system.

9. 9. The image processing method according to claim 8, wherein the imaging characteristics include at least one of a numerical aperture, a focal length, and a magnification of the imaging optical system.

10. 2. The image processing method according to claim 1, further comprising the step of acquiring information regarding the input observation direction.

11. an acquisition means for acquiring information regarding the light intensity distribution of each of a plurality of optical images formed by making the light from the sample illuminated at a plurality of different illumination angles incident on an imaging optical system, and acquiring three-dimensional information of the sample by performing a calculation using the acquired information; a generating means for generating an output image having an observation direction different from the optical axis direction of the imaging optical system from the three-dimensional information; The image processing device according to claim 1, wherein the generating means generates the output image by performing a conversion operation for converting a spatial spectrum of the three-dimensional information based on the illumination angle and the observation direction.

12. The image processing device according to claim 11 ; an imaging unit that captures the plurality of optical images by illuminating the sample at a plurality of illumination angles different from each other, The imaging device, wherein the image processing device acquires information regarding the light intensity distribution from an output of the imaging section.

13. The imaging device according to claim 12 ; an input unit for inputting information regarding the observation direction; and a display unit that displays the output image.

14. A program for causing a computer to execute a process according to the image processing method according to any one of claims 1 to 10.

Images