Optical aberration correction program and optical wavefront estimation program

Abstract

This optical aberration correction program causes a computer (3) to execute: a step for obtaining a plurality of phases of a provisional optical transmission function of an optical system (10), the phases corresponding respectively to a plurality of provisional optical aberration amounts of the optical system; a step for generating a plurality of three-dimensional phase modulation images by performing deconvolution on a three-dimensional source image of a sample (12) including an optical aberration of the optical system (10), for each of the plurality of provisional optical transmission function phases; a step for obtaining an optimal theoretical three-dimensional point spread function on the basis of brightness value indicators of the plurality of three-dimensional phase modulation images; and a step for calculating a three-dimensional corrected image by performing deconvolution on a separate three-dimensional source image which is different from the three-dimensional source images and which includes an optical aberration of the optical system (10) or the three-dimensional source image by an optimal optical transmission function corresponding to the optimal theoretical three-dimensional point spread function.

Description

[Technical Field]

[0001] The present disclosure relates to an optical aberration correction program and an optical wavefront estimation program. [Background technology]

[0002] Microscopes are used to acquire three-dimensional original images of samples, including three-dimensional objects such as cells. A three-dimensional original image of a sample is a stack of multiple two-dimensional original images of the sample, obtained by moving the focal plane of the microscope's observation optical system along the optical axis of the observation optical system. The multiple two-dimensional original images include images of light within the focal plane and images of light outside the focal plane, which are caused by optical aberrations in the observation optical system. The light outside the focal plane blurs the original image, reducing its contrast.

[0003] In general, a stack image of a sample is given by the convolution integral of the brightness distribution of the sample and the point spread function (PSF) of the observation optical system. Non-Patent Document 1 discloses a parametric blind deconvolution method that uses a computer to alternately and iteratively calculate the brightness distribution of the sample and the PSF of the observation optical system, and corrects the stack image of the sample while alternately and iteratively estimating the brightness distribution of the sample and the PSF of the observation optical system using maximum likelihood estimation. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] J. Markham and JA Conchello, Parametric blind deconvolution: a robust method for the simultaneous estimation of image and blur, Journal of the Optical Society of America A, October 1999, Vol.16, No.10, p.2377-2391 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the parametric blind deconvolution method of Non-Patent Document 1, the luminance distribution of the sample and the PSF of the observation optical system are alternately and iteratively calculated, and the luminance distribution of the sample and the PSF of the observation optical system are alternately and iteratively estimated. Therefore, in order to calculate a 3D corrected image in which the optical aberration of the observation optical system has been eliminated or reduced, a very large number of calculations must be performed. It takes a long time to calculate a 3D corrected image of the sample.

[0006] The present disclosure has been made in view of the above-mentioned problems, and an object of a first aspect of the present disclosure is to provide an optical aberration correction program that can calculate, in a shorter time, a three-dimensional corrected image in which optical aberrations of an optical system have been eliminated or reduced.An object of a second aspect of the present disclosure is to provide an optical wavefront estimation program that can estimate, in a shorter time, an optical wavefront affected by optical aberrations of an optical system. [Means for solving the problem]

[0007] The optical aberration correction program of the present disclosure causes a computer to execute the steps of obtaining phases of a plurality of provisional optical transfer functions of the optical system, each corresponding to a plurality of provisional optical aberration amounts of the optical system; deconvolving a three-dimensional original image of the sample, including the optical aberration of the optical system, with each of the phases of the plurality of provisional optical transfer functions to generate a plurality of three-dimensional phase-modulated images; determining an optimal theoretical three-dimensional point spread function that most closely approximates the three-dimensional point spread function of the optical system, based on an index of the brightness values ​​of the plurality of three-dimensional phase-modulated images; calculating an optimal optical transfer function of the optical system corresponding to the optimal theoretical three-dimensional point spread function; and deconvolving the three-dimensional original image or another three-dimensional original image, which includes the optical aberration of the optical system and is different from the three-dimensional original image, with the optimal optical transfer function to calculate a three-dimensional corrected image.

[0008] The optical wavefront estimation program of the present disclosure causes a computer to perform the steps of obtaining phases of multiple provisional optical transfer functions of the optical system, each corresponding to multiple provisional optical aberration amounts of the optical system; deconvolving a three-dimensional original image of a sample including optical aberrations of the optical system with each of the phases of the multiple provisional optical transfer functions to generate multiple three-dimensional phase-modulated images; determining an optimal theoretical three-dimensional point spread function that most closely approximates the three-dimensional point spread function of the optical system based on an index of the brightness values ​​of the multiple three-dimensional phase-modulated images; and calculating an optimal optical wavefront corresponding to the optimal theoretical three-dimensional point spread function. [Effects of the Invention]

[0009] According to the optical aberration correction program of the present disclosure, it is possible to calculate a three-dimensional corrected image in which optical aberrations of the optical system have been eliminated or reduced in a shorter time. According to the optical wavefront estimation program of the present disclosure, it is possible to estimate an optical wavefront affected by optical aberrations of the optical system in a shorter time. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing a microscope system according to a first embodiment, a second embodiment, and a fourth embodiment. [Figure 2]FIG. 2 is a partially enlarged cross-sectional view of the microscopes of the first and second embodiments. [Figure 3] 10A and 10B are diagrams illustrating examples of image distortion of a bright spot caused by optical aberration. [Figure 4] 3A and 3B are diagrams showing two-dimensional original images obtained by the microscopes of the first and second embodiments. [Figure 5] FIG. 1 is a block diagram illustrating the functional configuration of a computer according to the first, second, and fourth embodiments. [Figure 6] FIG. 2 is a block diagram illustrating the functional configuration of an optical aberration correction unit according to the first, second, and fourth embodiments. [Figure 7] FIG. 2 is a block diagram illustrating the functional configuration of an optical wavefront estimating unit according to the first, second, and fourth embodiments. [Figure 8] FIG. 2 is a block diagram illustrating the functional configuration of a spherical aberration estimating unit, a coma aberration estimating unit, and an astigmatism estimating unit according to the first and second embodiments. [Figure 9] 1 is a block diagram illustrating the functional configuration of an aberration-corrected image generating unit according to the first, second, and fourth embodiments. FIG. [Figure 10] FIG. 1 is a flowchart illustrating an optical aberration correcting method according to the first, second, and fourth embodiments. [Figure 11] FIG. 10 is a diagram showing a flowchart of each of a step of estimating an optimum amount of spherical aberration in the first and second embodiments, a step of estimating an optimum amount of coma aberration in the first and second embodiments, a step of estimating an optimum amount of astigmatism in the first and second embodiments, a step of estimating an optimum amount of high-order optical aberration in the first and second embodiments, and a step of estimating an optimum amount of high-order optical aberration in the third modified example of the first and second embodiments. [Figure 12] FIG. 10 is a flowchart showing steps for performing phase deconvolution on a three-dimensional original image according to the first and second embodiments. [Figure 13] FIG. 10 is a flowchart showing steps for calculating a three-dimensional corrected image according to the first and fourth embodiments. [Figure 14] FIG. 2 is a diagram showing an example of a three-dimensional original image I according to the first embodiment. [Figure 15] FIG. 10 is a diagram showing an example of a theoretical three-dimensional PSF' calculated for a first provisional amount of spherical aberration. [Figure 16] FIG. 10 is a diagram showing an example of a theoretical three-dimensional PSF' calculated for a second provisional amount of spherical aberration. [Figure 17] FIG. 10 is a diagram showing an example of a theoretical three-dimensional PSF' calculated for a third provisional amount of spherical aberration. [Figure 18] FIG. 10 is a diagram showing an example of a three-dimensional phase-modulated image I′ calculated for a first provisional amount of spherical aberration. [Figure 19] FIG. 10 is a diagram showing an example of a three-dimensional phase-modulated image I′ calculated for a second provisional amount of spherical aberration. [Figure 20] FIG. 10 is a diagram showing an example of a three-dimensional phase-modulated image I′ calculated for a third provisional amount of spherical aberration. [Figure 21] FIG. 1 is a schematic diagram showing a flowchart of an optical wavefront estimation method according to the first and fourth embodiments. [Figure 22] 1A and 1B are diagrams showing three-dimensional original images in Examples 1 and 2. [Figure 23] FIG. 2 is a diagram showing a three-dimensional corrected image in the first embodiment. [Figure 24] 10 is a diagram showing the change in full width at half maximum of the luminance of an image of a fluorescent bead in the optical axis direction (z-axis direction) before and after optical aberration correction in Example 1. FIG. [Figure 25] 10 is a diagram showing the change in full width at half maximum of the luminance of an image of a fluorescent bead in the horizontal direction (x-axis direction and y-axis direction) perpendicular to the optical axis direction before and after optical aberration correction in Example 1. FIG. [Figure 26] FIG. 10 is a flowchart illustrating an optical aberration correcting method according to a third modification of the first and second embodiments. [Figure 27] FIG. 10 is a schematic diagram showing a microscope system according to a fifth modification of the first and second embodiments. [Figure 28] FIG. 10 is a block diagram illustrating the functional configuration of an aberration-corrected image generating unit according to the second embodiment. [Figure 29]FIG. 10 is a flowchart showing steps for calculating a three-dimensional corrected image according to the second embodiment. [Figure 30] FIG. 10 is a diagram showing a three-dimensional corrected image in Example 2. [Figure 31] 10 is a diagram showing the change in full width at half maximum of the luminance of an image of a fluorescent bead in the optical axis direction (z-axis direction) before and after optical aberration correction in Example 2. FIG. [Figure 32] 10 is a diagram showing the change in full width at half maximum of the luminance of an image of a fluorescent bead in the horizontal direction (x-axis direction and y-axis direction) perpendicular to the optical axis direction before and after optical aberration correction in Example 2. FIG. [Figure 33] FIG. 10 is a schematic diagram showing a microscope system according to a third embodiment. [Figure 34] 10A and 10B are diagrams showing two-dimensional original SIM images obtained by the structured illumination microscope according to the third embodiment. [Figure 35] FIG. 1 shows an example of a two-dimensional original SIM image of a sample modulated by interference fringes having different phases. [Figure 36] FIG. 10 is a block diagram illustrating the functional configuration of a computer according to a second embodiment. [Figure 37] FIG. 11 is a block diagram illustrating the functional configuration of an image reconstruction unit according to a third embodiment. [Figure 38] FIG. 11 is a flowchart illustrating an image reconstruction method according to the third embodiment. [Figure 39] FIG. 11 is a diagram illustrating a step of extracting a three-dimensional original image Id,m of a spatial frequency component of order m from a three-dimensional original SIM image in a direction d according to the third embodiment. [Figure 40] FIG. 11 is a flowchart showing steps for generating a three-dimensional original image according to the third embodiment. [Figure 41] FIG. 11 is a flowchart showing the steps of deconvolving a three-dimensional original image according to the third embodiment. [Figure 42] FIG. 10 is a block diagram illustrating the functional configuration of a spherical aberration estimating unit, a coma aberration estimating unit, and an astigmatism estimating unit according to a fourth embodiment. [Figure 43]13 is a diagram for explaining a model for generating a theoretical optical wavefront that is affected by spherical aberration and defocus of the observation optical system in the fourth embodiment. FIG. [Figure 44] FIG. 10 is a flowchart showing a step of estimating an optimum amount of spherical aberration according to a fourth embodiment, a step of estimating an optimum amount of coma aberration according to the fourth embodiment, a step of estimating an optimum amount of astigmatism according to the fourth embodiment, and a step of estimating an optimum amount of higher-order optical aberration according to the fourth embodiment. [Figure 45] FIG. 10 is a flowchart showing a step of estimating an optimum amount of spherical aberration according to a fourth embodiment, a step of estimating an optimum amount of coma aberration according to the fourth embodiment, a step of estimating an optimum amount of astigmatism according to the fourth embodiment, and a step of estimating an optimum amount of higher-order optical aberration according to the fourth embodiment. [Figure 46] FIG. 13 is a flowchart showing steps for performing phase deconvolution on a three-dimensional original image according to the fourth embodiment. [Figure 47] FIG. 10 shows the full width at half maximum of the brightness of an image of a fluorescent bead in the optical axis direction (z-axis direction) before optical aberration correction, after optical aberration correction in embodiment 1, after optical aberration correction in embodiment 4, and after optical aberration correction in the first variant of embodiment 4. [Figure 48] FIG. 10 is a diagram showing the calculation time required to obtain a three-dimensional image with aberration corrected in the first embodiment, the fourth embodiment, and the first modified example of the fourth embodiment. [Figure 49] FIG. 13 is a diagram showing an example of a region in the original 3D image where the average value of the amplitude of high frequency components is relatively large, according to a first modification of the fourth embodiment. [Figure 50] FIG. 13 is a diagram showing another example of a region in the original 3D image where the average value of the amplitude of high frequency components is relatively large, according to the first modification of the fourth embodiment. [Figure 51] FIG. 13 is a diagram showing an image of fluorescent beads before optical aberration correction according to a second modification of the fourth embodiment. [Figure 52] FIG. 13 is a diagram showing an image of fluorescent beads after optical aberration correction according to a second modification of the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described, in which the same reference numerals are used to denote the same components, and the description thereof will not be repeated.

[0012] (Embodiment 1) A microscope system 1 according to a first embodiment will be described with reference to FIGS. 1 to 9. As shown in FIG. 1, the microscope system 1 includes a microscope 2, a computer 3, and a display device 4. The microscope 2 is capable of obtaining a three-dimensional original image 28 (see FIG. 4) of a sample 12 (three-dimensional object 12a). In this embodiment, the microscope 2 is a fluorescence microscope. The microscope 2 may be a confocal microscope, a two-photon microscope, or a scanning microscope including a super-resolution microscope such as an Airy-scan microscope.

[0013] 1 and 2, the microscope 2 includes an illumination optical system 8, a sample support stage 11, an observation optical system 10, and an imaging device 18. The microscope 2 may further include a light source 6.

[0014] The sample support 11 supports a sample 12. The sample 12 includes a three-dimensional object 12a and a surrounding medium 12b surrounding the three-dimensional object 12a. The three-dimensional object 12a is an object to be observed using the microscope 2. The three-dimensional object 12a may be, but is not limited to, a living cell, a molecule, a protein, or DNA. The three-dimensional object 12a is labeled with, for example, a fluorescent substance. The surrounding medium 12b is, for example, a culture solution or a gel.

[0015] The sample 12 is covered by a cover glass 14. Between the cover glass 14 and the objective lens 16 is an immersion medium 15. The immersion medium 15 has a refractive index greater than that of air, which has a refractive index of 1.00, and improves the resolution of the microscope 2. The immersion medium 15 may be, for example, water, glycerin, or oil.

[0016] The light source 6 outputs illumination light 7. The light source 6 is, for example, a mercury lamp, a xenon lamp, or a light-emitting diode (LED). The illumination light 7 can be absorbed by a fluorescent substance that marks the three-dimensional object 12a, exciting the fluorescent substance. The illumination light 7 illuminates the sample 12 through an illumination optical system 8. The illumination optical system 8 includes, for example, a lens 9a and a lens 9b. The illumination light 7 is collimated by the lens 9a. The illumination light 7 passes through the lens 9b and is reflected by a dichroic mirror 19. The dichroic mirror 19 reflects the illumination light 7 and transmits light 21 (e.g., fluorescence) from the sample 12 (the three-dimensional object 12a). The lens 9b focuses the illumination light 7 at a pupil plane 23 (back focal plane) of the objective lens 16. The illumination light 7 is collimated by the objective lens 16 and illuminates the sample 12, including the three-dimensional object 12a.

[0017] When the illumination light 7 is irradiated onto the three-dimensional object 12a, light 21 is emitted from the sample 12 (three-dimensional object 12a). For example, a fluorescent substance labeled on the three-dimensional object 12a is excited by the illumination light and emits the light 21. The light 21 is, for example, incoherent light such as fluorescence. The light 21 passes through the observation optical system 10 and is imaged on the imaging plane of the imaging device 18. The imaging device 18 is, for example, a CCD camera or a CMOS image sensor.

[0018] The observation optical system 10 includes an objective lens 16, an imaging lens 17, and a dichroic mirror 19. Specifically, the observation optical system 10 includes a sample 12, a cover glass 14, an immersion medium 15, the objective lens 16, the imaging lens 17, and the dichroic mirror 19. In this specification, the optical axis of the observation optical system 10 is sometimes referred to as the z-axis. The optical axis direction of the observation optical system 10 is sometimes referred to as the z-axis direction. Two axes that are perpendicular to the optical axis of the observation optical system 10 and that are perpendicular to each other are sometimes referred to as the x-axis and y-axis. Two directions that are perpendicular to the optical axis direction (z-axis direction) of the observation optical system 10 and that are perpendicular to each other are sometimes referred to as the x-axis and y-axis directions.

[0019] Due to variations in the thickness of the objective lens 16, imperfections in the surface shape of the objective lens 16, variations in the refractive index of the objective lens 16 or the sample 12, etc., the observation optical system 10 contains optical aberrations such as spherical aberration, coma aberration, and astigmatism. Due to the optical aberrations of the observation optical system 10, the original image obtained by the microscope 2 is distorted as shown in FIG. 3, and the optical wavefront 22 (see FIG. 1) of the light 21 emitted from the sample 12 is distorted. In this specification, the term "original image" refers to an image that includes the optical aberrations of the optical system (e.g., the observation optical system 10) (in other words, an image in which the optical aberrations of the optical system (e.g., the observation optical system 10) have not been corrected). In this specification, the term "optical wavefront 22" refers to the wavefront of the light 21 at the pupil plane 23 (back focal plane) of the objective lens 16. The optical wavefront 22 is a collection of points where all light rays that make up the optical wavefront 22 have the same phase.

[0020] One of the sample support stage 11 or the objective lens 16 is movable in the optical axis direction (z-axis direction) of the observation optical system 10 relative to the other of the sample support stage 11 or the objective lens 16. For example, at least one of the sample support stage 11 or the objective lens 16 is connected to a piezoelectric element or a motor (not shown). Using the piezoelectric element or motor, one of the sample support stage 11 or the objective lens 16 is moved in the optical axis direction (z-axis direction) of the observation optical system 10 relative to the other of the sample support stage 11 or the objective lens 16.

[0021] As shown in FIG. 4, the focal plane 20 of the objective lens 16 is moved along the optical axis direction (z-axis direction) of the observation optical system 10, while the sample 12 (three-dimensional object 12a) is imaged at intervals Δz. n-1 ,z n At each of the positions z1, z2, . . . , z3, a two-dimensional original image 27 of the sample 12 (three-dimensional object 12a) is obtained. n-1 ,z n A plurality of two-dimensional original images 27 are stacked over the entire image plane to obtain a three-dimensional original image 28 of the sample 12 (three-dimensional object 12a) as a stack of the plurality of two-dimensional original images 27 (stack image).

[0022] The distance Δz between the adjacent focal planes 20 is Δz<1 / (2k zmax ) is the sampling condition expressed as follows: zmax is the optical transfer function (OTF) of the observation optical system 10 in the frequency direction (k z The cutoff frequency k represents the maximum range in the direction of the zmax is determined by the numerical aperture (NA) of the objective lens 16 included in the observation optical system 10, the wavelength of the light 21, etc. In this specification, frequency means spatial frequency.

[0023] Referring to FIG. 1, a computer 3 is communicatively connected to the microscope 2. The computer 3 is, for example, a personal computer, a microcomputer, a cloud server, or a smart device (such as a smartphone or a tablet terminal). A display device 4 is communicatively connected to the computer 3. The display device 4 is, for example, a liquid crystal display device or an organic EL display device. The display device 4 displays the operating state of the microscope 2, an original two-dimensional image 27, an original three-dimensional image 28, or an image processed by the computer 3, such as a three-dimensional corrected image in which optical aberrations of the observation optical system 10 have been removed or reduced.

[0024] The computer 3 includes a processor 30 and a storage device 55. The storage device 55 is, for example, a random access memory (RAM), a read-only memory (ROM), or a hard disk. The storage device 55 stores a computer program executed by the processor 30, a two-dimensional original image 27, a three-dimensional original image 28, or an image processed by the computer 3, such as a three-dimensional corrected image in which optical aberrations of the observation optical system 10 have been removed or reduced. The computer program includes, for example, a microscope control program, a three-dimensional original image generation program, an optical aberration correction program, and an optical wavefront estimation program. The microscope control program, the three-dimensional original image generation program, the optical aberration correction program, and the optical wavefront estimation program may be stored in a non-transitory computer-readable storage medium.

[0025] The processor 30 is configured by, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The processor 30 executes a computer program stored in the storage device 55. When the processor 30 executes the computer program, the computer 3 realizes the functions shown in Figs. 5 to 9. The functional configuration of the computer 3 will be described with reference to Figs. 5 to 9.

[0026] As shown in Fig. 5, the computer 3 includes a microscope control unit 31, a three-dimensional original image generation unit 32, an optical aberration correction unit 33, and an optical wavefront estimation unit 34. As shown in Fig. 6, the optical aberration correction unit 33 includes a spherical aberration estimation unit 35, a coma aberration estimation unit 36, an astigmatism estimation unit 37, and an aberration-corrected image generation unit 38. As shown in Fig. 7, the optical wavefront estimation unit 34 includes the spherical aberration estimation unit 35, the coma aberration estimation unit 36, the astigmatism estimation unit 37, and an optical wavefront generation unit 39. As shown in Fig. 8, the spherical aberration estimating unit 35, the coma aberration estimating unit 36, and the astigmatism estimating unit 37 each include a theoretical optical wavefront generating unit 40, a theoretical three-dimensional PSF' calculating unit 41, a phase deconvolution unit 42, a three-dimensional phase modulation image generating unit 43, an optimum theoretical three-dimensional PSF' determining unit 44, and an optimum theoretical optical wavefront determining unit 45. As shown in Fig. 9, the aberration-corrected image generating unit 38 includes an amplitude change calculating unit 46, an optimum three-dimensional PSF'' calculating unit 47, a deconvolution unit 48, and a three-dimensional inverse Fourier transform unit 49.

[0027] The processor 30 executes the microscope control program, causing the computer 3 to function as a microscope control unit 31. The computer 3 is a control device for the microscope 2. The microscope control unit 31 controls a piezoelectric element or a motor (not shown) connected to at least one of the sample support stage 11 or the objective lens 16, thereby controlling the position of at least one of the sample support stage 11 or the objective lens 16. The microscope control unit 31 controls the imaging device 18, causing the imaging device 18 to capture a two-dimensional original image 27 of the sample 12 (three-dimensional object 12a).

[0028] When the processor 30 executes the 3D original image generation program, the computer 3 realizes the function of a 3D original image generation unit 32. The computer 3 is a 3D original image generation device. The 3D original image generation unit 32 performs step S11 (see FIG. 10).

[0029] When the processor 30 executes the optical aberration correction program, the computer 3 realizes the function of an optical aberration correction unit 33 (see FIGS. 5 and 6). The computer 3 is an optical aberration correction device. The optical aberration correction unit 33 performs steps S12 and S16 (see FIG. 10). The spherical aberration estimation unit 35 performs step S13 (see FIG. 10). The coma aberration estimation unit 36 ​​performs step S14 (see FIG. 10). The astigmatism estimation unit 37 performs step S15 (see FIG. 10). The aberration-corrected image generation unit 38 performs step S16 (see FIGS. 10 and 13).

[0030] The theoretical optical wavefront generation unit 40 performs steps S21 and S22 (see FIG. 11). The theoretical three-dimensional PSF' calculation unit 41 performs step S23 (see FIG. 11). The phase deconvolution unit 42 performs step S24 (see FIGS. 11 and 12). The three-dimensional phase modulation image generation unit 43 performs step S25 (see FIG. 11). The optimal theoretical three-dimensional PSF' determination unit 44 performs steps S27 and S28 (see FIG. 11). The optimal theoretical optical wavefront determination unit 45 performs step S29 (see FIG. 11). The amplitude change calculation unit 46 performs step S34 (see FIGS. 13 and 29). The optimal three-dimensional PSF'' calculation unit 47 performs step S35 (see FIGS. 13 and 29). The deconvolution unit 48 performs step S36 (see FIGS. 13 and 29). The three-dimensional inverse Fourier transform unit 49 performs step S39 (see FIGS. 13 and 29).

[0031] When the processor 30 executes the optical wavefront estimation program, the computer 3 realizes the function of an optical wavefront estimation unit 34 (see FIGS. 5 and 7). The computer 3 also functions as an optical wavefront estimation device. The optical wavefront estimation unit 34 performs steps S12 and S17 (see FIG. 21). The spherical aberration estimation unit 35 performs step S13 (see FIG. 21). The coma aberration estimation unit 36 ​​performs step S14 (see FIG. 21). The astigmatism estimation unit 37 performs step S15 (see FIG. 21). The optical wavefront generation unit 39 performs step S17 (see FIG. 21).

[0032] <Optical aberration correction method> 10 to 20, a method for eliminating or reducing the optical aberration of the observation optical system 10 contained in the three-dimensional original image 28 to obtain a three-dimensional corrected image will be described.

[0033] As shown in FIG. 10, in step S10, a two-dimensional original image 27 is acquired using the microscope 2. The microscope control unit 31 controls a piezoelectric element or motor (not shown) connected to either the sample support stage 11 or the objective lens 16. One of the sample support stage 11 or the objective lens 16 is moved in the optical axis direction (z-axis direction) of the observation optical system 10 relative to the other of the sample support stage 11 or the objective lens 16. The sample 12 (three-dimensional object 12a) is imaged while the focal plane 20 of the microscope 2 (observation optical system 10) is moved along the optical axis direction (z-axis direction) of the observation optical system 10. As shown in FIG. 4, the focal plane 20 is imaged at multiple positions z1, z2, . . . , z n-1 ,z n In each of the three-dimensional images, a two-dimensional raw image 27 of the sample 12 (three-dimensional object 12a) is obtained.

[0034] 10, in step S11, a three-dimensional original image 28 of the sample 12 is generated from a plurality of two-dimensional original images 27. Specifically, the three-dimensional original image generating unit 32 generates a three-dimensional original image 28 of the sample 12 from a plurality of positions z1, z2, . . . , z n-1 ,z n A plurality of two-dimensional original images 27 are stacked over a range of 1000 to generate a three-dimensional original image 28 of the sample 12 (three-dimensional object 12a) as a stack of the plurality of two-dimensional original images 27 (stack image).

[0035] The three-dimensional original image 28 of the sample 12 (three-dimensional object 12a) can be expressed by Equation (1). I(x, y, z) represents the luminance distribution of the three-dimensional original image 28. Hereinafter, I(x, y, z) may be referred to as the three-dimensional original image I. Figure 14 shows an example of the three-dimensional original image I. O(x', y', z') represents the luminance distribution of the sample 12 (three-dimensional object 12a). PSF(x-x', y-y', z-z') represents the point spread function (PSF) of the observation optical system 10. In other words, PSF(x-x', y-y', z-z') represents the degree of influence that a bright spot at position x', y', z' of the sample 12 (three-dimensional object 12a) has on the luminance of an image at a location (x, y, z) away from the bright spot via the observation optical system 10. As shown in equation (1), in real space, the three-dimensional original image I is given by the convolution integral of the brightness distribution of the sample 12 (three-dimensional object 12a) and the PSF.

[0036]

number

[0037] In frequency space (Fourier space), equation (1) can be expressed as equation (2). The Fourier transform FT(I) of the three-dimensional original image I is given by the product of the Fourier transform FT(O) of the sample 12 (three-dimensional object 12a) and the Fourier transform FT(PSF) of the PSF. The Fourier transform FT(I) of the three-dimensional original image I represents the frequency distribution of the three-dimensional original image I. The Fourier transform FT(O) of the sample 12 (three-dimensional object 12a) represents the frequency distribution of the sample 12 (three-dimensional object 12a). The Fourier transform FT(PSF) of the PSF is the optical transfer function (OTF) of the observation optical system 10.

[0038]

number

[0039] In frequency space, by deconvolving the three-dimensional original image I, a frequency distribution FT(O) of the sample 12 (three-dimensional object 12a) is obtained in which the optical aberration of the observation optical system 10 has been eliminated or reduced. The method of deconvolving the three-dimensional original image I is not particularly limited, but for example, it is to multiply the frequency distribution FT(I) of the three-dimensional original image I by an inverse filter of the OTF of the observation optical system 10, as shown in equation (3).

[0040]

number

[0041] By using equation (4) to perform an inverse Fourier transform on the frequency distribution FT(O) of the sample 12 (three-dimensional object 12a) in which the optical aberration of the observation optical system 10 has been eliminated or reduced, a corrected image of the sample 12 (three-dimensional object 12a) in which the optical aberration of the observation optical system 10 has been eliminated or reduced is obtained.

[0042]

number

[0043] <Step S12 of Estimating the Amount of Optical Aberration of the Observation Optical System 10> The optical aberration correction method of this embodiment is characterized by the method of deconvolving the original three-dimensional image I.

[0044] 10 , the optical aberration correction method of this embodiment includes step S12 of estimating the amount of optical aberration of the observation optical system 10. The optical aberration of the observation optical system 10 mainly occurs in the objective lens 16. The optical aberration occurring in the objective lens 16 can mainly be expressed as Seidel's optical aberration. Therefore, in this embodiment, Seidel's optical aberration (spherical aberration, coma aberration, and astigmatism) is used as the optical aberration of the observation optical system 10. Specifically, step S12 of estimating the amount of optical aberration of the observation optical system 10 includes step S13 of estimating the optimal amount of spherical aberration of the observation optical system 10, step S14 of estimating the optimal amount of coma aberration of the observation optical system 10, and step S15 of estimating the optimal amount of astigmatism of the observation optical system 10.

[0045] In a microscope 2 equipped with an objective lens 16, of the spherical aberration, coma aberration, and astigmatism of the objective lens 16, the spherical aberration of the objective lens 16 often has the greatest effect on an image, and the astigmatism of the objective lens 16 often has the smallest effect on an image. Therefore, as shown in Fig. 10, in step S12 of this embodiment, step S13 is first performed, then step S14 is performed, and finally step S15 is performed to estimate the amount of optical aberration of the observation optical system 10.

[0046] 11, step S13 of estimating the optimal spherical aberration amount of the observation optical system 10 includes step S21 of setting a provisional optical aberration amount (provisional spherical aberration amount) of the observation optical system 10, step S22 of generating a theoretical optical wavefront based on the set provisional optical aberration amount (provisional spherical aberration amount), step S23 of generating a theoretical three-dimensional point spread function (theoretical three-dimensional PSF') including the set provisional optical aberration amount (provisional spherical aberration amount), step S24 of phase deconvoluting the three-dimensional original image I, step S25 of generating a three-dimensional phase-modulated image I', step S26 of determining whether the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' have been generated for all of the multiple provisional optical aberration amounts (provisional spherical aberration amounts), step S27 of calculating an index of the brightness value of the three-dimensional phase-modulated image, step S28 of determining an optimal theoretical three-dimensional PSF', and step S29 of determining an optimal theoretical optical wavefront.

[0047] 11, in step S21 in step S13, a provisional optical aberration amount (provisional spherical aberration amount) of the observation optical system 10 is set on the computer 3. For example, one provisional optical aberration amount (provisional spherical aberration amount) of a plurality of provisional optical aberration amounts is set on the computer 3.

[0048] 11, in step S22 of step S13, the theoretical optical wavefront generating unit 40 (see FIG. 8) uses equation (5) to generate a theoretical optical wavefront W 1000 (provisional amount of spherical aberration) at the pupil coordinates (u, v) of the pupil plane 23 (back focal plane) of the objective lens 16. s (u,v) to generate the theoretical optical wavefront W sis the wavefront of light 21 output from a bright spot 13 (see FIG. 2) in the sample 12 (three-dimensional object 12a) and subjected to the provisional optical aberration (provisional spherical aberration) set in step S21 of step S13. d represents the distance between the cover glass 14 and the bright spot 13 of the sample 12 in the optical axis direction (z-axis direction) of the observation optical system 10. n1 represents the refractive index of the immersion medium 15. n2 represents the refractive index of the sample 12. θ1 represents the angle of incidence of light 21 passing through pupil coordinates (u, v) into the immersion medium 15 at the interface between the immersion medium 15 and the cover glass 14. θ1 is given by Equation (6). λ represents the wavelength of light 21. θ2 represents the angle between light 21 passing through pupil coordinates (u, v) and the optical axis of the observation optical system 10 at the bright spot 13 of the sample 12. θ2 is given by Equation (7).

[0049]

number

[0050]

number

[0051]

number

[0052] In the observation optical system 10, n1 and λ are often known, but n2 and d are unknown. Therefore, by setting n2 and d to arbitrary values, the theoretical optical wavefront W s It is possible to set a provisional amount of optical aberration (provisional amount of spherical aberration) of the observation optical system 10 that the subject will be subjected to.

[0053] 11, in step S23 of step S13, a theoretical three-dimensional PSF' of the observation optical system 10 including the provisional optical aberration amount (provisional spherical aberration amount) set in step S21 is generated. The theoretical three-dimensional PSF' calculation unit 41 (see FIG. 8) calculates the theoretical optical wavefront W sA theoretical three-dimensional PSF' is calculated from the above. In this specification, PSF' means a point spread function that mainly includes the amount of provisional optical aberration in its phase, or a point spread function in which the amount of provisional optical aberration is mainly reflected in its phase. In other words, "'" means that the amount of provisional optical aberration is mainly included in the phase, or that the amount of provisional optical aberration is mainly reflected.

[0054] Specifically, the theoretical three-dimensional PSF' calculation unit 41 uses equation (8) to calculate a pupil function P'(u,v) including a provisional optical aberration amount (provisional spherical aberration amount) on the pupil plane 23 of the objective lens 16. P(u,v) in equation (8) is given by equation (9). NA represents the numerical aperture of the objective lens 16.

[0055]

number

[0056]

number

[0057] Assuming that the light 21 travels as a spherical wave in the optical axis direction (z-axis direction) of the observation optical system 10, the defocused wavefront W of the light 21 is d is given by equation (10).

[0058]

number

[0059] The theoretical three-dimensional PSF' calculation unit 41 uses equation (11) to calculate a pupil function P'(u, v, z) corresponding to position z on the focal plane 20. The pupil function P'(u, v, z) includes not only information on the intensity of the light 21, but also information on the phase of the light 21, which mainly reflects information on the provisional optical aberration amount (provisional spherical aberration amount).

[0060]

number

[0061] The theoretical three-dimensional PSF' calculation unit 41 calculates the two-dimensional light intensity PSF at the position z on the focal plane 20 from the pupil function P'(u, v, z) using equation (12). z '(x, y, z) is calculated. In equation (12), Re() represents the real part in the parentheses. FT 2D -1 represents the two-dimensional inverse Fourier transform. For example, in equation (12), FT 2D -1 means the two-dimensional inverse Fourier transform in the uv space. In equation (12), * denotes the complex conjugate.

[0062]

number

[0063] The theoretical three-dimensional PSF' calculation unit 41 calculates two-dimensional light intensity PSFs at a plurality of positions z on the focal plane 20. z The theoretical three-dimensional PSF'(x, y, z) is calculated as a stack (assembly) of the provisional optical aberration amounts (provisional spherical aberration amounts). Information on the phase of light 21, which is mainly reflected by the provisional optical aberration amounts (provisional spherical aberration amounts), is reflected in the theoretical three-dimensional PSF'(x, y, z). Using the theoretical three-dimensional PSF'(x, y, z) and the three-dimensional original image I, it is possible to obtain the theoretical three-dimensional PSF'(x, y, z) that includes the optimal optical aberration amount (optimal spherical aberration amount) that is closest to the optical aberration amount (spherical aberration amount) of the observation optical system 10 among all the provisional optical aberration amounts (provisional spherical aberration amounts).

[0064] 11 and 12, in step S24, the three-dimensional original image I is subjected to phase deconvolution. Step S24 includes step S30 for calculating a theoretical OTF' (temporary optical transfer function) and step S40 for calculating a phase OTF of the theoretical OTF'. p '(phase of provisional optical transfer function)' (step S31), and the three-dimensional original image I is calculated as the theoretical OTF' phase OTF p In this specification, for example, a "theoretical" OTF means an OTF created on a computer 3.

[0065] 12, in step S30, the phase deconvolution unit 42 (see FIG. 8) performs a three-dimensional Fourier transform on the theoretical three-dimensional PSF′(x, y, z) using equation (13) to obtain the theoretical OTF′(k x ,k y ,k z ) (provisional optical transfer function) is calculated. 3D represents a three-dimensional Fourier transform. In this specification, OTF′ means an optical transfer function that mainly includes a provisional optical aberration amount in its phase, or an optical transfer function in which a provisional optical aberration amount is mainly reflected in its phase.

[0066]

number

[0067] The provisional optical aberration (e.g., provisional spherical aberration) is mainly the phase OTF of the theoretical OTF'. p 12, in step S31, the phase deconvolution unit 42 calculates the phase OTF of the theoretical OTF' from the theoretical OTF' using equation (14). p ' (phase of provisional optical transfer function). arg[] is a function to obtain the phase in parentheses.

[0068]

number

[0069] Referring to FIG. 12, in step S32, the phase deconvolution unit 42 converts the three-dimensional original image I into the phase OTF of the theoretical OTF′. p Specifically, the phase deconvolution unit 42 performs a three-dimensional Fourier transform on the three-dimensional original image I using equation (15) to obtain the frequency distribution D(k x ,k y ,k zThen, the phase deconvolution unit 42 calculates the frequency distribution D of the three-dimensional original image I in the frequency space by deconvolving the phase OTF of the theoretical OTF′. p For example, as shown in equation (15), the frequency distribution D of the original 3D image I is deconvolved with the phase OTF of the theoretical OTF '. p By multiplying the inverse filter of ', the frequency distribution D of the original 3D image I is converted into the phase OTF of the theoretical OTF '. p '. Thus, the frequency distribution D'(k x ,k y ,k z ) is calculated. In the phase of the frequency distribution D' of the original three-dimensional image I, the provisional optical aberration amount (provisional spherical aberration amount) is removed or reduced.

[0070]

number

[0071]

number

[0072] As shown in equation (16), phase deconvolution is performed by adjusting the cutoff frequency k xmax ,k ymax ,k zmax The reason is as follows: xmax ,k ymax ,k zmax In the frequency range beyond k, there is no signal, only noise. Therefore, the cutoff frequency k xmax ,k ymax ,k zmax When performing phase deconvolution in the frequency domain above k, noise-induced artifacts occur. To avoid these artifacts, phase deconvolution is performed at a cutoff frequency k. xmax ,k ymax ,k zmaxThis is done only in the frequency space within the

[0073] Cutoff frequency k xmax is the optical transfer function (OTF) of the observation optical system 10 in the frequency direction conjugate with the x-axis direction (k x The cutoff frequency k represents the maximum range in the direction of the ymax is the optical transfer function (OTF) of the observation optical system 10 in the frequency direction conjugate to the y-axis direction (k y The cutoff frequency k represents the maximum range in the direction of the xmax ,k ymax is determined by the numerical aperture (NA) of the objective lens 16 included in the observation optical system 10, the wavelength of the light 21, and the like.

[0074] 11, in step S25 of step S13, a three-dimensional phase-modulated image I'(x, y, z) is generated in which the provisional optical aberration amount (provisional spherical aberration amount) set in step S21 is corrected. Specifically, the three-dimensional phase-modulated image generating unit 43 (see FIG. 8) performs a three-dimensional inverse Fourier transform on D' using equation (17) to generate a three-dimensional phase-modulated image I'. FT 3D -1 means the three-dimensional inverse Fourier transform. For example, FT in Eq. (17) 3D -1 is the frequency space (k x k y k z This refers to the three-dimensional inverse Fourier transform in space.

[0075]

number

[0076] 11, in step S26 of step S13, it is determined on the computer 3 whether or not the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' have been generated for all of the plurality of provisional optical aberration amounts (provisional spherical aberration amounts). If the generation of the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' for all of the plurality of provisional optical aberration amounts (provisional spherical aberration amounts) has not yet been completed, steps S22 to S26 are repeated while changing the provisional optical aberration amounts (provisional spherical aberration amounts). For example, the provisional spherical aberration amount of the observation optical system 10 can be changed by changing at least one of n2 and d. In this way, the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' are generated for all of the plurality of provisional optical aberration amounts (provisional spherical aberration amounts). That is, by repeatedly performing steps S22 to S26 while changing the provisional optical aberration amount (provisional spherical aberration amount), a plurality of theoretical three-dimensional PSF's and a plurality of three-dimensional phase-modulated images I' are generated. Each of the plurality of theoretical three-dimensional PSF's includes a corresponding one of the plurality of provisional optical aberration amounts. Each of the plurality of three-dimensional phase-modulated images I's includes a phase OTF of the theoretical OTF' of the corresponding one of the plurality of theoretical three-dimensional PSF's. p The three-dimensional phase-modulated images I' are generated by deconvolving the original three-dimensional image I with the phase of the interim optical transfer function. Each of the multiple three-dimensional phase-modulated images I' is obtained by correcting the original three-dimensional image I with a corresponding one of the multiple interim optical aberration amounts.

[0077] 15 to 17 show examples of theoretical three-dimensional PSF' calculated for multiple provisional spherical aberration amounts. 18 to 20 show examples of three-dimensional phase-modulated images I' calculated for multiple provisional spherical aberration amounts. 15 and 18 show the theoretical three-dimensional PSF' and three-dimensional phase-modulated image I' calculated for the first provisional spherical aberration amount, respectively. 16 and 19 show the theoretical three-dimensional PSF' and three-dimensional phase-modulated image I' calculated for the second provisional spherical aberration amount, respectively. 17 and 20 show the theoretical three-dimensional PSF' and three-dimensional phase-modulated image I' calculated for the third provisional spherical aberration amount, respectively.

[0078] In the three-dimensional phase-modulated image I' corresponding to the theoretical three-dimensional PSF' that includes the optimal optical aberration (optimal spherical aberration) that is closest to the optical aberration (spherical aberration) of the observation optical system 10 among all the provisional optical aberrations (provisional spherical aberrations), the light that had been spread due to the optical aberration of the observation optical system 10 is concentrated at the center of the image of the bright spot 13, and the brightness of the three-dimensional phase-modulated image I' increases.

[0079] 11, in step S27 of step S13, the optimal theoretical three-dimensional PSF' determiner 44 (see FIG. 8) calculates an index of the luminance values ​​of the three-dimensional phase-modulated image I'. The index of the luminance values ​​of the three-dimensional phase-modulated image I' is, for example, the variance of the luminance values ​​of the three-dimensional phase-modulated image I', the standard deviation of the luminance values ​​of the three-dimensional phase-modulated image I', or the maximum value of the luminance values ​​of the three-dimensional phase-modulated image I'. The variance of the luminance values ​​of the three-dimensional phase-modulated image I' is, for example, the variance of the luminance values ​​of an image (MIP image) obtained using maximum intensity projection. Maximum intensity projection is a method of projecting three-dimensionally constructed image data in an arbitrary viewpoint direction and displaying the maximum value in the projection path on the projection surface. When calculating the variance of the luminance values ​​of the three-dimensional phase-modulated image I' related to spherical aberration using maximum intensity projection, maximum intensity projection is applied to the three-dimensional phase-modulated image I' in the x-axis direction and the y-axis direction.

[0080] 11, in step S28 of step S13, the optimal theoretical three-dimensional PSF' determiner 44 (see FIG. 8) determines an optimal theoretical three-dimensional PSF' from the multiple theoretical three-dimensional PSF's based on the luminance value index of the multiple three-dimensional phase-modulated images I'. The optimal theoretical three-dimensional PSF' includes, among the multiple theoretical three-dimensional PSF's, an optimal optical aberration amount (optimal spherical aberration amount) that is closest to the optical aberration amount (spherical aberration amount) of the observation optical system 10. The optimal theoretical three-dimensional PSF' is, among the multiple theoretical three-dimensional PSF's, most approximate to the three-dimensional point spread function of the observation optical system 10 in terms of the optical aberration amount (spherical aberration amount) of the observation optical system 10. Specifically, the optimal theoretical three-dimensional PSF' determiner 44 determines, as the optimal theoretical three-dimensional PSF', the theoretical three-dimensional PSF' corresponding to the three-dimensional phase-modulated image I' having the maximum luminance value index.

[0081] For example, in the three-dimensional phase-modulated image I' shown in Fig. 19, light is more concentrated at the center of the image of the bright spot than in the three-dimensional phase-modulated image I' shown in Figs. 18 and 20. The brightness value index of the three-dimensional phase-modulated image I' shown in Fig. 19 is larger than the brightness value index of the three-dimensional phase-modulated image I' shown in Figs. 18 and 20. The optimal theoretical three-dimensional PSF' determiner 44 determines the theoretical three-dimensional PSF' shown in Fig. 16 corresponding to the three-dimensional phase-modulated image I' shown in Fig. 19 as the optimal theoretical three-dimensional PSF'.

[0082] 11, in step S29 in step S13, the optimum theoretical optical wavefront determiner 45 (see FIG. 8) determines the optimum theoretical optical wavefront W corresponding to the optimum theoretical three-dimensional PSF′. s Determine the optimal theoretical optical wavefront W s is an optical wavefront that has the optimum optical aberration amount (optimum spherical aberration amount) closest to the optical aberration amount (spherical aberration amount) of the observation optical system 10. s is most similar to the optical wavefront 22 in terms of the amount of optical aberration (amount of spherical aberration) of the observation optical system 10. Specifically, the optimal theoretical optical wavefront determiner 45 determines the optimal theoretical optical wavefront W corresponding to the optimal theoretical three-dimensional PSF′ from equations (5) to (12) and the like. s Determine.

[0083] 10, the optical aberration correcting method of this embodiment includes step S14 of estimating the optimum amount of coma aberration of the observation optical system 10. The coma aberration estimating unit 36 ​​(see FIGS. 6 and 8) executes step S14. As shown in FIG. 11, step S14 of estimating the optimum amount of coma aberration of the observation optical system 10 is basically the same as step S13 of estimating the optimum amount of spherical aberration of the observation optical system 10, except that in the above description of step S13, spherical aberration is replaced with coma aberration and there are the following differences.

[0084] 11, in step S22 in step S14, the theoretical optical wavefront generating unit 40 uses equation (18) to generate a theoretical optical wavefront W 1 that has been subjected to the provisional optical aberration amount (provisional coma aberration amount) at the pupil coordinates (u, v) of the pupil plane 23 (back focal plane) of the objective lens 16. c (u,v) to generate the theoretical optical wavefront W c is the wavefront of the light 21 output from the bright spot 13 in the sample 12 and subjected to the provisional optical aberration amount (provisional coma aberration amount) set in step S21 in step S14. u ,c v By setting to an arbitrary value, the theoretical optical wavefront W c It is possible to set a provisional amount of optical aberration (provisional amount of coma aberration) of the observation optical system 10 that the subject is subjected to.

[0085]

number

[0086] 11, in step S23 in step S14, the theoretical optical wavefront generating unit 40 calculates a pupil function P′(u,v) including the provisional optical aberration amount (provisional coma aberration amount) of the observation optical system 10 by using equation (19). s is the optimum theoretical optical wavefront W determined in step S29 in step S13. s is.

[0087]

number

[0088] 11, in step S26 of step S14, it is determined on the computer 3 whether or not the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' have been generated for all of the plurality of provisional optical aberration amounts (provisional coma aberration amounts). If the generation of the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' for all of the plurality of provisional optical aberration amounts (provisional coma aberration amounts) has not yet been completed, steps S22 to S26 are repeated while changing the provisional optical aberration amounts (provisional coma aberration amounts). For example, the coefficient c u ,c v By changing at least one of the above, it is possible to change the provisional coma aberration amount of the observation optical system 10. In this way, a theoretical three-dimensional PSF' and a three-dimensional phase-modulated image I' are generated for all of the multiple provisional optical aberration amounts (provisional coma aberration amounts).

[0089] 11, the variance of the luminance values ​​of the three-dimensional phase-modulated image I' calculated using maximum intensity projection will be described as an example of an index of the luminance values ​​of the three-dimensional phase-modulated image I' in step S27 in step S14. When calculating the variance of the luminance values ​​of the three-dimensional phase-modulated image I' related to the x-axis coma aberration using maximum intensity projection, maximum intensity projection is applied to the three-dimensional phase-modulated image I' in the y-axis and z-axis directions. When calculating the variance of the luminance values ​​of the three-dimensional phase-modulated image I' related to the y-axis coma aberration using maximum intensity projection, maximum intensity projection is applied to the three-dimensional phase-modulated image I' in the z-axis and x-axis directions.

[0090] 11, in step S28 of step S14, the optimal theoretical three-dimensional PSF' determiner 44 (see FIG. 8) determines an optimal theoretical three-dimensional PSF' from the multiple theoretical three-dimensional PSF's based on the luminance value index of the multiple three-dimensional phase-modulated images I'. The optimal theoretical three-dimensional PSF' includes, among the multiple theoretical three-dimensional PSF's, an optimal optical aberration amount (optimum coma aberration amount) that is closest to the optical aberration amount (coma aberration amount) of the observation optical system 10. The optimal theoretical three-dimensional PSF' is, among the multiple theoretical three-dimensional PSF's, most approximate to the three-dimensional point spread function of the observation optical system 10 in terms of the optical aberration amount (coma aberration amount) of the observation optical system 10. Specifically, the optimal theoretical three-dimensional PSF' determiner 44 determines, as the optimal theoretical three-dimensional PSF', the theoretical three-dimensional PSF' corresponding to the three-dimensional phase-modulated image I' having the largest luminance value index.

[0091] 11, in step S29 in step S14, the optimum theoretical optical wavefront determiner 45 (see FIG. 8) determines the optimum theoretical optical wavefront W corresponding to the optimum theoretical three-dimensional PSF′. c Determine the optimal theoretical optical wavefront W c is an optical wavefront that has the optimum optical aberration (optimum coma aberration) closest to the optical aberration (optimum coma aberration) of the observation optical system 10. c is most similar to the optical wavefront 22 in terms of the amount of optical aberration (coma aberration) of the observation optical system 10. Specifically, the optimal theoretical optical wavefront determiner 45 determines the optimal theoretical optical wavefront W corresponding to the optimal theoretical three-dimensional PSF′ from equation (18) and the like. c Determine.

[0092] 10, the optical aberration correcting method of this embodiment includes step S15 of estimating the optimum amount of astigmatism of the observation optical system 10. The coma aberration estimating unit 36 ​​(see FIGS. 6 and 8) executes step S15. As shown in FIG. 11, step S15 of estimating the optimum amount of astigmatism of the observation optical system 10 is basically the same as step S13 of estimating the optimum amount of spherical aberration of the observation optical system 10, except that in the above description of step S13, spherical aberration is replaced with astigmatism and there are the following differences.

[0093] 11, in step S22 for generating an optical wavefront in step S15, the theoretical optical wavefront generating unit 40 uses equation (20) to generate a theoretical optical wavefront W 1000 , which is subjected to a provisional amount of optical aberration (provisional amount of astigmatism) at pupil coordinates (u, v) on the pupil plane 23 (back focal plane) of the objective lens 16. a (u,v) to generate the theoretical optical wavefront W a is the wavefront of the light 21 output from the bright spot 13 in the sample 12 and subjected to the provisional optical aberration amount (provisional astigmatism amount) set in step S21 in step S15. a By rotating the lens by a factor of , it is possible to create astigmatism for various directions in the pupil plane 23. u or angle r a By setting at least one of these to an arbitrary value, the theoretical optical wavefront W a It is possible to set a provisional amount of optical aberration (provisional amount of astigmatism) of the observation optical system 10 that the subject is subjected to.

[0094]

number

[0095] 11, in S23 in step S15, the theoretical optical wavefront generating unit 40 calculates a pupil function P'(u,v) including the provisional optical aberration amount (provisional astigmatism amount) of the observation optical system 10 by using equation (21). s and W c are the optimal theoretical optical wavefronts W determined in steps S13 and S14, respectively. s and W c is.

[0096]

number

[0097] 11, in step S26 of step S15, it is determined on the computer 3 whether or not the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' have been generated for all of the plurality of provisional optical aberration amounts (provisional astigmatism amounts). If the generation of the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' for all of the plurality of provisional optical aberration amounts (provisional astigmatism amounts) has not yet been completed, steps S22 to S26 are repeated while changing the provisional optical aberration amounts (provisional astigmatism amounts). For example, when the coefficient a u or angle r a The provisional astigmatism amount can be changed by changing at least one of the following: In this way, a theoretical three-dimensional PSF′ and a three-dimensional phase-modulated image I′ are generated for all of the provisional optical aberration amounts (provisional astigmatism amounts).

[0098] 11, the variance of the luminance values ​​of the three-dimensional phase-modulated image I' calculated using maximum intensity projection will be described as an example of an index of the luminance values ​​of the three-dimensional phase-modulated image I' in step S27 in step S15. When calculating the variance of the luminance values ​​of the three-dimensional phase-modulated image I' related to astigmatism using maximum intensity projection, maximum intensity projection is applied to the three-dimensional phase-modulated image I' in the x-axis direction and the y-axis direction.

[0099] 11, in step S28 of step S15, the optimal theoretical three-dimensional PSF' determiner 44 (see FIG. 8) determines an optimal theoretical three-dimensional PSF' from the multiple theoretical three-dimensional PSF's based on the luminance value index of the multiple three-dimensional phase-modulated images I'. The optimal theoretical three-dimensional PSF' includes, among the multiple theoretical three-dimensional PSF's, an optimal optical aberration amount (optimal astigmatism amount) that is closest to the optical aberration amount (astigmatism amount) of the observation optical system 10. The optimal theoretical three-dimensional PSF' is, among the multiple theoretical three-dimensional PSF's, most approximate to the three-dimensional point spread function of the observation optical system 10 in terms of the optical aberration amount (astigmatism amount) of the observation optical system 10. Specifically, the optimal theoretical three-dimensional PSF' determiner 44 determines, as the optimal theoretical three-dimensional PSF', the theoretical three-dimensional PSF' corresponding to the three-dimensional phase-modulated image I' having the maximum luminance value index.

[0100] 11, in step S29 in step S15, the optimum theoretical optical wavefront determiner 45 (see FIG. 8) determines the optimum theoretical optical wavefront W corresponding to the optimum theoretical three-dimensional PSF′. a Determine the optimal theoretical optical wavefront W a is an optical wavefront that has the optimum optical aberration amount (optimum astigmatism amount) closest to the optical aberration amount (amount of astigmatism) of the observation optical system 10. a is most similar to the optical wavefront 22 in terms of the amount of optical aberration (amount of astigmatism) of the observation optical system 10. Specifically, the optimal theoretical optical wavefront determiner 45 determines the optimal theoretical optical wavefront W corresponding to the optimal theoretical three-dimensional PSF′ from equation (20) and the like. a Determine.

[0101] <Step S16 of calculating a three-dimensional corrected image> 10 , the optical aberration correction method of this embodiment includes step S16 of calculating a three-dimensional corrected image. The optical aberration of the observation optical system 10 mainly changes the phase of the optical wavefront 22 and the phase of the OTF of the observation optical system 10, but also secondarily changes the amplitude of the light 21 or the amplitude of the PSF or OTF of the observation optical system 10. In this embodiment, in step S16 of calculating the three-dimensional corrected image, changes in the amplitude of the light 21 or the amplitude of the PSF or OTF of the observation optical system 10 caused by the optical aberration of the observation optical system 10 are corrected. The changes in the amplitude of the light 21 or the amplitude of the PSF or OTF of the observation optical system 10 caused by the optical aberration of the observation optical system 10 include changes in the amplitude of the wavefront generated at the interface between the sample 12 and the cover glass 14 (see FIG. 2 ) due to the spherical aberration of the observation optical system 10.

[0102] As shown in FIG. 13, step S16 of calculating the three-dimensional corrected image includes step S34 of calculating a change in the amplitude of light 21 due to spherical aberration, step S35 of calculating an optimal three-dimensional PSF'', step S36 of deconvolving the three-dimensional original image I, and step S39 of performing a three-dimensional inverse Fourier transform. Step S36 includes step S37 of calculating an optimal OTF'' from the optimal three-dimensional PSF'', and step S38 of deconvolving the three-dimensional original image I with the normalized optimal OTF''.

[0103] Step S34 of calculating the change in amplitude of light 21 due to spherical aberration will be described with reference to Fig. 13. As shown in Fig. 2, when the refractive index n2 of sample 12 is smaller than the refractive index of cover glass 14, part of light 21 is reflected as reflected light 25 at the interface between sample 12 and cover glass 14. Therefore, the spherical aberration of observation optical system 10 causes the amplitude of light 21 to change at the interface between sample 12 and cover glass 14. In step S34, amplitude change calculation unit 46 (see Fig. 9) calculates the change A in amplitude of light 21 due to this reflection at pupil plane 23 of objective lens 16 using equation (22). t Calculate (u,v).

[0104]

number

[0105] Furthermore, it is known that a phenomenon called wavefront compression occurs at the interface between the sample 12 and the cover glass 14 due to spherical aberration of the observation optical system 10 (BM Hanser et al., Journal of Microscopy, 2004, Vol. 216 Part 1, pp. 32-48). In step S34, the amplitude change calculation unit 46 calculates the change A in the amplitude of the light 21 at the pupil plane 23 of the objective lens 16 due to wavefront compression using equation (23). w Calculate (u,v).

[0106]

number

[0107] Step S35 of calculating the optimum three-dimensional PSF'' will be described with reference to FIG. 13. In this specification, PSF'' means a point spread function whose amplitude also contains the provisional optical aberration amount, or a point spread function whose amplitude also reflects the provisional optical aberration amount. The optimum three-dimensional PSF'' calculation unit 47 calculates the pupil function P''(u,v) using equation (24). W in equation (24) s , W c and W a are the optimal theoretical optical wavefronts W determined in step S29 of steps S13, S14, and S15, respectively. s , W c and W a is.

[0108]

number

[0109] The optimal three-dimensional PSF'' calculation unit 47 calculates the pupil function P''(u, v, z) corresponding to the position z of the focal plane 20 using equation (25).

[0110]

number

[0111] The optimal three-dimensional PSF″ calculation unit 47 calculates the two-dimensional light intensity PSF at the position z on the focal plane 20 from the pupil function P″(u, v, z) using equation (26). z Calculate (x,y,z).

[0112]

number

[0113] The optimal three-dimensional PSF″ calculation unit 47 calculates two-dimensional light intensity PSFs at multiple positions z on the focal plane 20. z The optimal three-dimensional PSF (x, y, z) is calculated as a stack (assembly) of the PSFs.

[0114] 13, in step S36, deconvolution unit 48 deconvolves the original three-dimensional image I. Step S36 includes step S37 of calculating an optimal OTF″ and step S38 of deconvolving the original three-dimensional image I with the optimal OTF″.

[0115] 13, in step S37, the deconvolution unit 48 performs a three-dimensional Fourier transform on the optimal three-dimensional PSF″ using equation (27) to obtain the optimal OTF″ (k x ,k y ,k z ) is calculated. The phase of the optimal OTF'' includes the optimal optical aberration amount that is closest to the optical aberration amount of the observation optical system 10. The amplitude of the optimal OTF'' includes the spherical aberration of the observation optical system 10. In this specification, OTF'' means an optical transfer function whose amplitude also includes the provisional optical aberration amount, or an optical transfer function whose amplitude also reflects the provisional optical aberration amount.

[0116]

number

[0117] 13, in step S38, the deconvolution unit 48 deconvolves the original three-dimensional image I with the optimal OTF″. Specifically, the deconvolution unit 48 performs a three-dimensional Fourier transform on the original three-dimensional image I using the above equation (15) to obtain the frequency distribution D(k x ,k y ,k z) is calculated. The deconvolution unit 48 deconvolves the frequency distribution D of the original three-dimensional image I in frequency space with the optimal OTF''. For example, the frequency distribution D of the original three-dimensional image I may be deconvolved with the optimal OTF'' by multiplying the frequency distribution D of the original three-dimensional image I by a Wiener filter of the optimal OTF'' using equation (28). In equation (28), w is a constant. In this way, the frequency distribution D''(k x ,k y ,k z ) is calculated. The optimal optical aberration amount closest to the optical aberration amount of the observation optical system 10 is removed from the phase of the frequency distribution D'' of the original three-dimensional image I. The spherical aberration is removed from the amplitude of the frequency distribution D'' of the original three-dimensional image I.

[0118]

number

[0119] The deconvolution method in step S38 is not particularly limited. For example, deconvolution may be performed by applying an inverse filter of the optimal OTF'' to the three-dimensional Fourier transform D of the three-dimensional original image I. In order to avoid artifacts, the cutoff frequency k of the optical transfer function (OTF) of the observation optical system 10 is set as shown in equation (28). xmax ,k ymax ,k zmax Deconvolution is performed only in the frequency space within

[0120] 13, in step S39, the three-dimensional inverse Fourier transform unit 49 performs a three-dimensional inverse Fourier transform on the frequency distribution D'' of the original three-dimensional image I using equation (29). In this way, a three-dimensional corrected image I''(x, y, z) in which the optical aberration of the observation optical system 10 has been removed or reduced is calculated.

[0121]

number

[0122] <Optical wavefront estimation method> A method for estimating an optical wavefront 22 according to this embodiment will be described with reference to Fig. 21. Specifically, the optical wavefront 22 is the optical wavefront of the light 21 at the pupil plane 23 (back focal plane) of the objective lens 16. At the pupil plane 23 (back focal plane) of the objective lens 16, a defocused wavefront W d is zero. The method for estimating the optical wavefront 22 of this embodiment includes step S12 of estimating the amount of optical aberration of the observation optical system 10, and step S17 of calculating an optimal optical wavefront that most closely resembles the optical wavefront 22. The optimal optical wavefront at the pupil plane 23 (back focal plane) of the objective lens 16 most closely resembles the optical wavefront 22 at the pupil plane 23 (back focal plane) of the objective lens 16 in terms of the amount of optical aberration (for example, the amount of spherical aberration, the amount of coma aberration, and the amount of astigmatism) of the observation optical system 10. In step S17 of generating the optimal optical wavefront, the optical wavefront estimating unit 34 (see FIGS. 5 and 7) calculates the optimal theoretical optical wavefront W determined in step S29 of step S12. s ,W c ,W a and equation (30), the optimum optical wavefront at the pupil plane 23 (back focal plane) of the objective lens 16 is calculated.

[0123]

number

[0124] Example 1 Example 1, which is an example of this embodiment, will be described with reference to Figures 22 and 23. In Example 1, the microscope 2 is a fluorescence microscope. The three-dimensional object 12a contained in the sample 12 is a fluorescent bead with a diameter of 0.10 µm. The light 21 is fluorescence emitted from the fluorescent bead. Because the diameter of the fluorescent bead is smaller than the diffraction limit of the observation optical system 10, the fluorescent bead can be considered as a point light source in the observation optical system 10.

[0125] The three-dimensional corrected image shown in Fig. 23 is clearer at z=0 than the three-dimensional original image I shown in Fig. 22, and the image rapidly blurs as it deviates from z=0. Furthermore, the three-dimensional corrected image shown in Fig. 23 is more similar in the way the image blurs when it is deviated from z=0 in the +z direction and when it is deviated from z=0 in the -z direction than the three-dimensional original image I shown in Fig. 22. Therefore, it can be seen that the optical aberration of the observation optical system 10 is eliminated or reduced in the three-dimensional corrected image shown in Fig. 23 compared to the three-dimensional original image I shown in Fig. 22.

[0126] 24 and 25, by performing step S13 of estimating the optimal spherical aberration amount of the observation optical system 10 and step S16 of calculating a three-dimensional corrected image, the full width at half maximum of the luminance of the image of the fluorescent beads is reduced. By performing step S14 of estimating the optimal coma aberration amount of the observation optical system 10 and step S16 of calculating a three-dimensional corrected image, the full width at half maximum of the luminance of the image of the fluorescent beads is reduced. By performing step S15 of estimating the optimal astigmatism amount of the observation optical system 10 and step S16 of calculating a three-dimensional corrected image, the full width at half maximum of the luminance of the image of the fluorescent beads is reduced. The quality of the three-dimensional corrected image is improved compared to the quality of the original three-dimensional image I.

[0127] (Modification of this embodiment) In a first variant of this embodiment, in step S12 of estimating the amount of optical aberration of the observation optical system 10, at least one of step S13 of estimating the optimal amount of spherical aberration of the observation optical system 10, step S14 of estimating the optimal amount of coma aberration of the observation optical system 10, or step S15 of estimating the optimal amount of astigmatism of the observation optical system 10 may be performed.

[0128] In a second modified example of the present embodiment, step S12 of estimating the amount of optical aberration of the observation optical system 10 may include at least two of step S13 of estimating the optimal amount of spherical aberration of the observation optical system 10, step S14 of estimating the optimal amount of coma aberration of the observation optical system 10, or step S15 of estimating the optimal amount of astigmatism of the observation optical system 10. Specifically, step S12 of estimating the amount of optical aberration of the observation optical system 10 may include at least one of step S13 of estimating the optimal amount of spherical aberration of the observation optical system 10, step S14 of estimating the optimal amount of coma aberration of the observation optical system 10, or step S15 of estimating the optimal amount of astigmatism of the observation optical system 10.

[0129] A third modified example of this embodiment will be described with reference to Fig. 26. The optical aberration of the observation optical system 10 includes not only the optical aberration of the objective lens 16 but also optical aberrations caused by the sample 12. Therefore, the optical aberration of the observation optical system 10 includes optical aberrations more complex than Seidel's optical aberrations (optical aberrations of higher orders than Seidel's optical aberrations). In this third modified example, in order to eliminate or reduce optical aberrations more complex than Seidel's optical aberrations (optical aberrations of higher orders than Seidel's optical aberrations), step S12 of estimating the amount of optical aberration of the observation optical system 10 further includes step S18 of estimating the optimal amount of high-order optical aberration of the observation optical system 10, in addition to steps S13, S14, and S15.

[0130] Step S18 for estimating the optimum amount of high-order optical aberration of the observation optical system 10 is basically the same as step S13 for estimating the optimum amount of spherical aberration of the observation optical system 10, except that in the above description of step S13, spherical aberration is replaced with high-order optical aberration, and step S18 differs in the following respects.

[0131] 11, in step S22 for generating an optical wavefront in step S18, the theoretical optical wavefront generating unit 40 calculates the theoretical optical wavefront W 1 , which is subjected to the provisional high-order optical aberrations of the observation optical system 10, at the pupil coordinates (u, v) of the pupil plane 23 (back focal plane) of the objective lens 16, using equation (31). ze (u,v) to generate the theoretical optical wavefront W ze(u, v) is the wavefront of light 21 output from bright spot 13 (see FIG. 2) in sample 12 and subjected to the provisional high-order optical aberrations set in step S21 in step S18. j represents the Zernike polynomials at the expansion mode j. The Zernike polynomials may be, for example, standard Zernike polynomials or fringe Zernike polynomials. M j represents the amplitude of the expansion mode j. M j is an unknown constant. M j By setting to an arbitrary value, the theoretical optical wavefront W ze It is possible to set the provisional amount of high-order optical aberration of the observation optical system 10 that M j By varying the theoretical optical wavefront W ze The amount of provisional high-order optical aberrations experienced by the viewing optical system 10 can vary.

[0132]

number

[0133] 11, in step S23 in step S18, the theoretical optical wavefront generating unit 40 calculates a pupil function P′(u,v) including provisional optical aberration amounts (provisional high-order optical aberration amounts) of the observation optical system 10 using equation (32). s ,W c ,W a are the optimal theoretical optical wavefronts W determined in step S29 in steps S13, S14, and S15, respectively. s ,W c ,W a is.

[0134]

number

[0135] 11, in step S26 of step S18, it is determined on the computer 3 whether or not the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' have been generated for all of the plurality of provisional optical aberration amounts (provisional high-order optical aberration amounts). If the generation of the theoretical three-dimensional PSF' and the three-dimensional phase-modulated image I' for all of the plurality of provisional optical aberration amounts (provisional high-order optical aberration amounts) has not yet been completed, steps S22 to S26 are repeated while changing the provisional optical aberration amounts (provisional high-order optical aberration amounts). For example, when M j The provisional high-order optical aberration amount can be changed by changing the value of . In this way, a theoretical three-dimensional PSF′ and a three-dimensional phase-modulated image I′ are generated for all of the multiple provisional optical aberration amounts (provisional high-order optical aberration amounts).

[0136] 11, in step S28 of step S18, the optimal theoretical three-dimensional PSF' determiner 44 (see FIG. 8) determines an optimal theoretical three-dimensional PSF' from the multiple theoretical three-dimensional PSF's based on the luminance value index of the multiple three-dimensional phase-modulated images I'. The optimal theoretical three-dimensional PSF' includes, among the multiple theoretical three-dimensional PSF's, an optimal optical aberration amount (optimal high-order optical aberration amount) that is closest to the optical aberration amount (high-order optical aberration amount) of the observation optical system 10. The optimal theoretical three-dimensional PSF' is, among the multiple theoretical three-dimensional PSF's, most approximate to the three-dimensional point spread function of the observation optical system 10 in terms of the optical aberration amount (high-order optical aberration amount) of the observation optical system 10. Specifically, the optimal theoretical three-dimensional PSF' determiner 44 determines, as the optimal theoretical three-dimensional PSF', the theoretical three-dimensional PSF' corresponding to the three-dimensional phase-modulated image I' having the largest luminance value index.

[0137] 11, in step S29 of step S18, the optimum theoretical optical wavefront determiner 45 (see FIG. 8) determines the optimum theoretical optical wavefront W corresponding to the optimum theoretical three-dimensional PSF′. ze Determine the optimal theoretical optical wavefront W ze is an optical wavefront that has the optimum optical aberration amount (optimum high-order optical aberration amount) closest to the optical aberration amount (high-order optical aberration amount) of the observation optical system 10. zeis most similar to the optical wavefront 22 in terms of the optical aberration amount (high-order optical aberration amount) of the observation optical system 10. Specifically, the optimal theoretical optical wavefront determiner 45 determines the optimal theoretical optical wavefront W corresponding to the optimal theoretical three-dimensional PSF′ from equation (31) etc. ze Determine.

[0138] In step S35 of the third modified example of this embodiment, the optimum three-dimensional PSF″ calculation unit 47 (see FIG. 9) calculates the pupil function P″(u, v) using equation (33). s , W c , W a ,W ze are the optimal theoretical optical wavefronts W determined in steps S13, S14, S15, S18, and S29, respectively. s , W c , W a ,W ze is.

[0139]

number

[0140] In step S17 of the third modified example of this embodiment, the optical wavefront estimating unit 34 (see FIGS. 5 and 7) estimates the optimal theoretical optical wavefront W determined in steps S13, S14, S15, S18, and S29. s , W c , W a ,W ze and equation (34), the optimum optical wavefront that most closely approximates the optical wavefront 22 is calculated.

[0141]

number

[0142] A fourth modified example of this embodiment will be described. An optical wavefront affected by Seidel optical aberration can also be expressed using Zernike polynomials. Therefore, in this fourth modified example of this embodiment, the optical wavefront is expressed using only Zernike polynomials. In this fourth modified example of this embodiment, step S12 for estimating the optical aberration of the observation optical system 10 does not include steps S13, S14, and S15, but does include step S18 (however, "high-order optical aberration" in step S18 should be read as "optical aberration").

[0143] In the fourth modification of this embodiment, in step S23 in step S18, the theoretical optical wavefront generation unit 40 calculates the pupil function P'(u,v) using equation (32). In step S35 of the fourth modification of this embodiment, the theoretical optical wavefront generation unit 40 calculates the pupil function P''(u,v) using equation (33). In step S17 of the fourth modification of this embodiment, the optical wavefront estimation unit 34 generates an optimal optical wavefront using equation (34). However, in the fourth modification of this embodiment, W s , W c and W a is zero.

[0144] 27, in a microscope system 1a according to a fifth modification of the present embodiment, the functions of the computer 3 according to the present embodiment may be distributed and implemented among computers 3a, 3b, and 3c and a storage device 55d, which are communicatively connected to each other via a network 58 such as the Internet. The computer 3a includes a processor 30a and a storage device 55a. The computer 3b includes a processor 30b and a storage device 55b. The computer 3c includes a processor 30c and a storage device 55c.

[0145] Specifically, the storage device 55a stores a microscope control program according to this embodiment. The processor 30a executes the microscope control program. The computer 3a functions as a microscope control unit 31 (see FIG. 5) and a three-dimensional original image generating unit 32. The computer 3a is a control device for the microscope 2. The computer 3a outputs a two-dimensional original image 27 (see FIG. 4) of the sample 12 (three-dimensional object 12a) or a three-dimensional original image 28 (see FIG. 4) of the sample 12 (three-dimensional object 12a) to at least one of the storage devices 55a, 55b, 55c, and 55d. The two-dimensional original image 27 (see FIG. 4) of the sample 12 (three-dimensional object 12a) or the three-dimensional original image 28 (see FIG. 4) of the sample 12 (three-dimensional object 12a) is stored in at least one of the storage devices 55a, 55b, 55c, and 55d.

[0146] The storage device 55b stores an optical aberration correction program according to the present embodiment. The processor 30b executes the optical aberration correction program. The computer 3b functions as the optical aberration correction unit 33. The computer 3b is an optical aberration correction device. The computer 3b outputs a three-dimensional corrected image of the sample 12 (three-dimensional object 12a) to at least one of the storage devices 55b and 55d. The three-dimensional corrected image of the sample 12 (three-dimensional object 12a) is stored in at least one of the storage devices 55b and 55d.

[0147] The storage device 55c stores an optical wavefront estimation program according to this embodiment. The processor 30c executes the optical wavefront estimation program. The computer 3c functions as an optical wavefront estimation unit 34. The computer 3c is an optical wavefront estimation device. The computer 3c outputs the estimated optical wavefront (the optimal optical wavefront that most closely resembles the optical wavefront 22) to at least one of the storage devices 55c and 55d. The estimated optical wavefront (the optimal optical wavefront that most closely resembles the optical wavefront 22) is stored in at least one of the storage devices 55c and 55d.

[0148] In the microscope system 1a according to the fifth modified example of this embodiment, the functions of the computer 3b and the computer 3c may be implemented in a single computer.

[0149] In the sixth modification of this embodiment, the three-dimensional original image used to determine the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) may be different from the three-dimensional original image 28 of the sample 12 corrected in the step of calculating the three-dimensional corrected image (e.g., steps S38 and S39). The three-dimensional original image used to determine the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) may be the three-dimensional image 28 of the entire sample 12 obtained by the microscope system 1, 1a, or may be a three-dimensional image of a partial region of the sample 12 obtained by the microscope system 1, 1a. However, the three-dimensional original image corrected in the step of calculating the three-dimensional corrected image (e.g., steps S38 and S39) is common to the three-dimensional original image 28 of the sample 12 used in the step of determining the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) (e.g., step S12) in that it is obtained through the observation optical system 10 and includes the optical aberration of the observation optical system 10.

[0150] For example, the original three-dimensional image 28 of the sample 12 used in the step (e.g., step S12) of determining the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) may be an original three-dimensional test image of a test sample such as fluorescent beads obtained through the observation optical system 10, and the original three-dimensional image corrected in the step (e.g., steps S38 and S39) of calculating the three-dimensional corrected image may be an original three-dimensional image of a sample such as a cell obtained through the observation optical system 10. In another example, the original three-dimensional image 28 of the sample 12 used in the step (e.g., step S12) of determining the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) may be an original three-dimensional image of a partial region of the sample 12 obtained through the observation optical system 10, and the original three-dimensional image corrected in the step (e.g., steps S38 and S39) of calculating the three-dimensional corrected image may be an original three-dimensional image 28 of the entire sample 12 obtained through the observation optical system 10.

[0151] The effects of this embodiment will be described. The optical aberration correction program of this embodiment causes the computer 3 to calculate the phases of a plurality of provisional optical transfer functions of the optical system (observation optical system 10) corresponding to a plurality of provisional optical aberration amounts of the optical system (phases OTF of theoretical OTF′ p a step of deconvolving a three-dimensional original image 28 of the sample 12 including the optical aberration of the optical system with each of the phases of the plurality of interim optical transfer functions to generate a plurality of three-dimensional phase-modulated images I' (e.g., steps S25, S26, S32); and a step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system based on an index of the brightness values ​​of the plurality of three-dimensional phase-modulated images I' ( For example, step S28), a step of calculating an optimal optical transfer function (optimal OTF'') of the optical system corresponding to the optimal theoretical three-dimensional point spread function (for example, steps S34, S35, S37), and a step of deconvolving the original three-dimensional image 28 of the sample 12 or another original three-dimensional image that includes optical aberrations of the optical system (observation optical system 10) and is different from the original three-dimensional image 28 of the sample 12 with the optimal optical transfer function to calculate a corrected three-dimensional image (for example, steps S38, S39).

[0152] The optical aberration correction method of this embodiment calculates the phases of a plurality of provisional optical transfer functions (phases OTF of theoretical OTF′) of the optical system corresponding to a plurality of provisional optical aberration amounts of the optical system (observation optical system 10), respectively. pthe step of deconvolving a three-dimensional original image 28 of the sample 12, including the optical aberration of the optical system, with each of the phases of the plurality of tentative optical transfer functions to generate a plurality of three-dimensional phase-modulated images I' (e.g., steps S25, S26, S32); the step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system, based on an index of the brightness values ​​of the plurality of three-dimensional phase-modulated images I' (e.g., step S28); the step of calculating an optimal optical transfer function (optimal OTF'') of the optical system corresponding to the optimal theoretical three-dimensional point spread function; and the step of deconvolving the three-dimensional original image 28 of the sample 12 or another three-dimensional original image that includes the optical aberration of the optical system (observation optical system 10) and is different from the three-dimensional original image 28 of the sample 12, with the optimal optical transfer function to calculate a three-dimensional corrected image (e.g., steps S38, S39).

[0153] In this embodiment, the luminance distribution of the sample 12 and the PSF of the optical system (observation optical system 10) are not calculated alternately and iteratively. Therefore, a three-dimensional corrected image in which optical aberrations of the optical system have been eliminated or reduced can be calculated in a shorter time. Furthermore, in this embodiment, an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system is obtained based on the index of the luminance values ​​of multiple three-dimensional phase-modulated images I'. Therefore, a more accurate optimal theoretical three-dimensional point spread function can be obtained. A three-dimensional corrected image in which optical aberrations of the optical system have been eliminated or reduced with higher accuracy can be calculated.

[0154] In the optical aberration correction program and the optical aberration correction method of the present embodiment, the step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') is a step (e.g., step S28) of determining an optimal theoretical three-dimensional point spread function from a plurality of theoretical three-dimensional point spread functions of the optical system based on an index of the luminance values ​​of a plurality of three-dimensional phase-modulated images. Each of the plurality of theoretical three-dimensional point spread functions includes a corresponding one of a plurality of provisional optical aberration amounts.

[0155] In this embodiment, the luminance distribution of the sample 12 and the PSF of the optical system (observation optical system 10) are not calculated alternately and iteratively. Therefore, a three-dimensional corrected image in which optical aberrations of the optical system have been eliminated or reduced can be calculated in a shorter time. Furthermore, in this embodiment, an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system is determined based on the luminance value indexes of multiple three-dimensional phase-modulated images I'. Therefore, the optimal theoretical three-dimensional point spread function can be determined more accurately from multiple theoretical three-dimensional point spread functions. A three-dimensional corrected image in which optical aberrations of the optical system have been eliminated or reduced with higher accuracy can be calculated.

[0156] In the optical aberration correction program and optical aberration correction method of the present embodiment, the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') includes at least two of the spherical aberration, coma aberration, and astigmatism of the optical system, so that it is possible to calculate a three-dimensional corrected image in which the optical aberration of the optical system has been removed or reduced with higher accuracy.

[0157] The optical wavefront estimation program of this embodiment causes the computer 3 to calculate the phases of a plurality of provisional optical transfer functions (phases OTF of theoretical OTF′) of the optical system corresponding to a plurality of provisional optical aberration amounts of the optical system (observation optical system 10), respectively. p The method executes the steps of: obtaining a three-dimensional original image 28 of the sample 12 including the optical aberration of the optical system with each of the phases of the plurality of tentative optical transfer functions (e.g., steps S26 and S31); generating a plurality of three-dimensional phase-modulated images I' by deconvolving the three-dimensional original image 28 of the sample 12 including the optical aberration of the optical system (e.g., steps S25, S26, and S32); obtaining an optimal theoretical three-dimensional point spread function (optimum theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system based on the index of the brightness values ​​of the plurality of three-dimensional phase-modulated images I' (e.g., step S28); and calculating an optimal optical wavefront corresponding to the optimal theoretical three-dimensional point spread function (e.g., steps S17 and S29).

[0158] The optical wavefront estimation method of this embodiment is to estimate the phases of a plurality of provisional optical transfer functions (phases OTF of theoretical OTF′) of an optical system corresponding to a plurality of provisional optical aberration amounts of the optical system (observation optical system 10), respectively. p The method includes a step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system (e.g., step S28), based on the index of the brightness values ​​of the multiple three-dimensional phase-modulated images I', and a step of calculating an optimal optical wavefront corresponding to the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') (e.g., step S17, S29).

[0159] In this embodiment, the luminance distribution of the sample 12 and the PSF of the optical system (observation optical system 10) are not calculated alternately and iteratively. Therefore, the optical wavefront 22 affected by the optical aberration of the optical system can be calculated in a shorter time. Furthermore, in this embodiment, an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system is obtained based on the index of the luminance values ​​of multiple three-dimensional phase-modulated images I'. Therefore, the optimal theoretical three-dimensional point spread function can be obtained more accurately from multiple theoretical three-dimensional point spread functions. The optical wavefront 22 affected by the optical aberration of the optical system can be estimated with higher accuracy.

[0160] In the optical wavefront estimation program and the optical wavefront estimation method of the present embodiment, the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') includes at least two of the spherical aberration, coma aberration, and astigmatism of the optical system, so that the optical wavefront 22 affected by the optical aberration of the optical system can be estimated with higher accuracy.

[0161] (Embodiment 2) The second embodiment will be described with reference to Figures 1 to 8, 10 to 12, and 28 to 32. The second embodiment is similar to the first embodiment, but differs from the first embodiment in an aberration-corrected image generation unit 38 (see Figure 28) and step S36 of deconvolving the original three-dimensional image I.

[0162] A microscope system 1 according to this embodiment will be described with reference to FIGS. 1 to 8 and 28. As shown in FIG. 28, in this embodiment, the aberration-corrected image generation unit 38 further includes a normalized optimal three-dimensional PSF''' calculation unit 50. The normalized optimal three-dimensional PSF''' calculation unit 50 performs step S40 (see FIG. 29).

[0163] 10 to 12 and 29, the optical aberration correction method of this embodiment will be described. The optical aberration of the observation optical system 10 is mainly reflected in the phase of the optical transfer function (OTF) of the observation optical system 10, but also has a small effect on the amplitude of the OTF of the observation optical system 10 (excluding the change in the amplitude of the light 21 taken into account in step S36 of the first embodiment). The optical aberration correction method of this embodiment further removes the effect of the optical aberration of the observation optical system 10 on the amplitude of the OTF of the observation optical system (excluding the change in the amplitude of the light 21 taken into account in step S36 of the first embodiment) in step S36 of deconvolving the three-dimensional original image I.

[0164] Step S36 of deconvolving the original three-dimensional image I includes, in addition to step S37 of calculating the optimal OTF'', step S40 of calculating a normalized optimal OTF''', and step S41 of deconvolving the original three-dimensional image I with the normalized optimal OTF'''.

[0165] 29, in step S40, the deconvolution unit 48 (see FIG. 28) calculates the amplitude of the optimal OTF″ using equation (35) in the case where there is no optical aberration in the observation optical system 10 (W s =W c =W a The amplitude of the optical transfer function |FT 3D(PSF(x,y,z))| is normalized. In this way, the normalized optimal OTF''' is calculated. By this normalization, only the amplitude and phase changes caused by the optimal optical aberration that is closest to the optical aberration of the observation optical system 10 are introduced into the normalized optimal OTF'''.

[0166]

number

[0167] 29, in step S41, the deconvolution unit 48 deconvolves the original three-dimensional image I with the normalized optimal OTF'''. Specifically, the deconvolution unit 48 performs a three-dimensional Fourier transform on the original three-dimensional image I using the above equation (15) to obtain the frequency distribution D(k x ,k y ,k z ) is calculated. The deconvolution unit 48 deconvolves the frequency distribution D of the original three-dimensional image I in frequency space with the normalized optimal OTF'''. For example, the frequency distribution D of the original three-dimensional image I may be deconvolved with the normalized optimal OTF''' by applying the inverse filter of the normalized optimal OTF''' to the frequency distribution D of the original three-dimensional image I using equation (36). In this way, the deconvolution unit 48 calculates the frequency distribution D'''(k x ,k y ,k z ) is calculated. From the amplitude and phase of the frequency distribution D''' of the three-dimensional original image I, the optimal optical aberration amount closest to the optical aberration amount of the observation optical system 10 is removed.

[0168]

number

[0169] The deconvolution method in step S41 is not particularly limited. For example, deconvolution may be performed by applying a Wiener filter of a normalized optimal OTF'' to the three-dimensional Fourier transform D of the three-dimensional original image I. In order to avoid artifacts, the cutoff frequency k of the optical transfer function (OTF) of the observation optical system 10 is set as shown in equation (36). xmax ,k ymax ,k zmax Deconvolution is performed only in the frequency space within

[0170] 29, in step S39, the three-dimensional inverse Fourier transform unit 49 (see FIG. 28) performs a three-dimensional inverse Fourier transform on the frequency distribution D''' of the three-dimensional original image I using equation (37). In this way, a three-dimensional corrected image I'''(x, y, z) in which the optical aberration of the observation optical system 10 has been removed or reduced is calculated.

[0171]

number

[0172] The optical aberration correcting program of the second embodiment causes the computer 3 to execute the optical aberration correcting method of the present embodiment.

[0173] Example 2 Example 2, which is one example of the present embodiment, will be described with reference to Fig. 22 and Fig. 30 to Fig. 32. In Example 2, similar to Example 1, the microscope 2 is a fluorescence microscope, and the three-dimensional object 12a contained in the sample 12 is a fluorescent bead with a diameter of 0.10 µm.

[0174] The three-dimensional corrected image shown in Fig. 30 is clearer at z=0 than the original three-dimensional image shown in Fig. 22, and the image rapidly blurs as it deviates from z=0. Furthermore, the three-dimensional corrected image shown in Fig. 30 is more similar in the way the image blurs when it is deviated from z=0 in the +z direction and when it is deviated from z=0 in the -z direction than the original three-dimensional image shown in Fig. 22. Therefore, it can be seen that the optical aberration of the observation optical system 10 is corrected more effectively in the three-dimensional corrected image shown in Fig. 30 than in the original three-dimensional image shown in Fig. 22.

[0175] 31 and 32, by performing step S13 (see FIG. 10) of estimating the optimal spherical aberration amount of the observation optical system 10 and step S16 (see FIG. 10) of calculating a three-dimensional corrected image, the full width at half maximum of the luminance of the fluorescent bead image is reduced. By performing step S14 (see FIG. 10) of estimating the optimal coma aberration amount of the observation optical system 10 and step S16 of calculating a three-dimensional corrected image, the full width at half maximum of the luminance of the fluorescent bead image is further reduced. By performing step S15 (see FIG. 10) of estimating the optimal astigmatism amount of the observation optical system 10 and step S16 of calculating a three-dimensional corrected image, the full width at half maximum of the luminance of the fluorescent bead image is further reduced. The quality of the three-dimensional corrected image is improved over the quality of the original three-dimensional image, and has been improved to near the diffraction limit.

[0176] In addition to the effects of the first embodiment, the present embodiment has the following effects. In the optical aberration correction program and the optical aberration correction method of this embodiment, the optimal optical transfer function is normalized by the amplitude of the optical transfer function of the optical system (observation optical system 10) when there is no optical aberration in the optical system (observation optical system 10).

[0177] According to the optical aberration correction method and the optical aberration correction program of the present embodiment, the influence of the optical aberration of the observation optical system 10 on the luminance of multiple three-dimensional corrected images can be further reduced or eliminated. This makes it possible to compare the luminance of multiple three-dimensional corrected images. Furthermore, it becomes possible to quantitatively grasp the degree of correction of the optical aberration of the observation optical system 10 (for example, the Strehl ratio) and compare it with the diffraction limit of the observation optical system 10. This makes it possible to expand the fields of use of three-dimensional corrected images.

[0178] (Embodiment 3) A microscope system 1b according to the third embodiment will be described with reference to Fig. 33 to Fig. 37. The microscope system 1b according to the present embodiment has a configuration similar to that of the microscope system 1 according to the first embodiment, but differs mainly in that it has a structured illumination microscope (SIM) 2b instead of the microscope 2 and in the functional configuration of the computer 3.

[0179] 33, the structured illumination microscope 2b includes a configuration similar to that of the microscope 2 of the first embodiment (see FIGS. 1 and 2), but differs from the microscope 2 of the first embodiment in the following respects. The structured illumination microscope 2b includes an illumination optical system 8b, a sample support stage 11, an observation optical system 10, an imaging device 18, and a drive device 67. The illumination optical system 8b includes a collimating lens 65, a light branching element 66, and a relay lens 70. The illumination optical system 8b may further include an optical fiber 64. The structured illumination microscope 2b may further include a light source 6b.

[0180] The light source 6b outputs illumination light 7b. The light source 6b is, for example, a semiconductor laser, and the illumination light 7b is, for example, laser light. The illumination light 7b can be absorbed by a fluorescent material that marks the three-dimensional object 12a and excite the fluorescent material. The illumination light 7b output from the light source 6b is incident on an optical fiber 64. The illumination light 7b output from the optical fiber 64 is collimated by a collimating lens 65 and then incident on a light branching element 66. The light branching element 66 is, for example, a diffraction grating. The light branching element 66 branches the illumination light 7b into multiple light beams. For example, the light branching element 66 branches the illumination light 7b into a 0th-order diffracted light, a +1st-order diffracted light, and a −1st-order diffracted light. The light branching element 66 may be a spatial light modulator such as a liquid crystal spatial light modulator.

[0181] The illumination light 7b enters a relay lens 70. The relay lens 70 includes a lens 71, a lens 72, and a lens 73. The illumination light 7b passes through the relay lens 70 and enters a dichroic mirror 19. The dichroic mirror 19 transmits the illumination light 7b and reflects light 21 (e.g., fluorescence) from the sample 12 (three-dimensional object 12a). The illumination light 7b passes through the dichroic mirror 19 and enters an objective lens 16. The objective lens 16 forms a focal plane 20 on the sample 12 (three-dimensional object 12a) that is optically conjugate with the light branching element 66. The zeroth-order diffracted light, the +1st-order diffracted light, and the −1st-order diffracted light of the illumination light 7b interfere with each other to form interference fringes 77 (or structured illumination light). The sample 12 (three-dimensional object 12a) is illuminated by the interference fringes 77 of the illumination light 7b. 35, interference fringes 77 are structured illumination patterns in which linear bright areas and linear dark areas are arranged periodically. In this specification, the direction in which the bright areas and dark areas are arranged is referred to as the periodic direction of interference fringes 77.

[0182] Specifically, the interference fringes 77 include a zeroth-order component (DC component) formed only by the zeroth-order diffracted light, a first-order component formed by interference between the zeroth-order diffracted light and ±1st-order diffracted lights (interference between the zeroth-order diffracted light and +1st-order diffracted light and interference between the zeroth-order diffracted light and −1st-order diffracted light), and a second-order component formed by interference between the ±1st-order diffracted lights. The zeroth-order component of the interference fringes 77 has one spatial frequency component (specifically, a spatial frequency of zero) where the order m is zero. The first-order component of the interference fringes 77 has two spatial frequency components where the order m is ±1. The second-order component of the interference fringes 77 has two spatial frequency components where the order m is ±2. The interference fringes 77 include five spatial frequency components (order m=0, ±1, ±2).

[0183] The bright portions of the interference fringes 77 in the sample 12 (three-dimensional object 12a) excite the fluorescent material that marks the three-dimensional object 12a. Light 21 is emitted from the excited fluorescent material. The light 21 passes through the objective lens 16, is reflected by the dichroic mirror 19, and enters the imaging lens 17. The imaging lens 17 forms an optically conjugate plane (image plane) with the focal plane 20 of the objective lens 16 on the imaging plane of the imaging device 18. The light 21 passes through the imaging lens 17 and enters the imaging device 18. Two-dimensional original SIM images 81, 82, and 83 (see FIG. 34) of the sample 12 (three-dimensional object 12a) are acquired using the imaging device 18.

[0184] 33, the driver 67 is connected to the light branching element 66. The driver 67 can move the light branching element 66 within a plane (xy plane) perpendicular to the optical axis (z axis) of the observation optical system 10. By linearly moving the light branching element 66 using the driver 67, the interference fringes 77 can be moved in the periodic direction of the interference fringes 77. In other words, the phase α of the interference fringes 77 can be changed.

[0185] The driver 67 can rotate the light branching element 66 around the optical axis (z-axis) of the observation optical system 10. By rotating the light branching element 66 around the optical axis (z-axis) of the observation optical system 10 using the driver 67, the orientation d (direction of the interference fringes 77) of the interference fringes 77 on the focal plane 20 can be changed. The driver 67 is, for example, a piezoelectric element or a motor. The driver 67 is communicatively connected to the computer 3 and can be controlled using the computer 3. When the light branching element 66 is a spatial light modulator, the driver 67 electrically changes the orientation of the pattern formed on the spatial light modulator.

[0186] One of the sample support stage 11 and the objective lens 16 can be moved in the optical axis direction (z-axis direction) of the observation optical system 10 relative to the other of the sample support stage 11 and the objective lens 16. For example, at least one of the sample support stage 11 and the objective lens 16 is connected to a piezoelectric element or a motor (not shown). Using the piezoelectric element or motor, one of the sample support stage 11 and the objective lens 16 is moved in the optical axis direction (z-axis direction) of the observation optical system 10 relative to the other of the sample support stage 11 and the objective lens 16. As shown in FIG. 34 , while the focal plane 20 is moved along the optical axis direction (z-axis direction) of the observation optical system 10, a plurality of positions z1, z2, . . . , z3 of the focal plane 20 can be measured. n-1 ,z n In this way, the sample 12 is imaged at a plurality of positions z1, z2, . . . , z n-1 ,z n In each of the focal planes 20, two-dimensional original SIM images 81, 82, and 83 of the sample 12 are acquired. The distance Δz between adjacent focal planes 20 is Δz<1 / (2k zmax ) is preferably satisfied.

[0187] Specifically, the optical branching element 66 is rotated around the optical axis (z-axis) of the observation optical system 10 using a driving device 67 to set the orientation d of the interference fringes 77. As shown in FIG. 34 , two-dimensional original SIM images 81, 82, and 83 of the sample 12 are captured at the orientation d of the interference fringes 77. The orientation d of the interference fringes 77 includes, for example, an orientation of 0° represented as d=1, an orientation of 120° represented as d=2, and an orientation of 240° represented as d=3. The angle of the orientation d (azimuth angle) is defined, for example, as the angle of the orientation d with respect to the +x direction. The number of orientations d of the interference fringes 77 is not limited to three, and may be any plural number.

[0188] The two-dimensional original SIM image 81 is an image acquired by using the imaging device 18 to image the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 at the orientation d=1. The two-dimensional original SIM image 81 is a moiré image of the sample 12 modulated by the interference fringes 77 at the orientation d=1. The two-dimensional original SIM image 82 is an image acquired by using the imaging device 18 to image the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 at the orientation d=2. The two-dimensional original SIM image 82 of the sample 12 is a moiré image of the sample 12 modulated by the interference fringes 77 at the orientation d=2. The two-dimensional original SIM image 83 is an image acquired by using the imaging device 18 to image the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 at the orientation d=3. The two-dimensional original SIM image 83 of the sample 12 is a moiré image of the sample 12 modulated by the interference fringes 77 with an orientation d=3.

[0189] 34, two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e of the sample 12 are captured for each of multiple phases α of the interference fringes 77 in each of multiple orientations d of the interference fringes 77. Specifically, two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e of the sample 12 are captured while linearly moving the light branching element 66 using the driving device 67 to change the phase α of the interference fringes 77. As shown in FIGS. 34 and 35, each of the two-dimensional original SIM images 81, 82, and 83 includes two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e.

[0190] The two-dimensional original SIM image 84a is an image acquired by using the imaging device 18 to image the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 with an orientation d and a phase α=0. The two-dimensional original SIM image 84a is a moiré image of the sample 12 modulated by the interference fringes 77 with an orientation d and a phase α=0. The two-dimensional original SIM image 84b is an image acquired by using the imaging device 18 to image the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 with an orientation d and a phase α=2π / 5. The two-dimensional original SIM image 84b is a moiré image of the sample 12 modulated by the interference fringes 77 with an orientation d and a phase α=2π / 5. The two-dimensional original SIM image 84c is an image acquired by using the imaging device 18 to image the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 with an orientation d and a phase α=4π / 5. The two-dimensional original SIM image 84c is a moiré image of the sample 12 modulated by interference fringes 77 of orientation d and phase α=4π / 5.

[0191] Two-dimensional original SIM image 84d is an image acquired by imaging sample 12 (three-dimensional object 12a) illuminated by interference fringes 77 with orientation d and phase α=6π / 5 using image capture device 18. Two-dimensional original SIM image 84d is a moiré image of sample 12 modulated by interference fringes 77 with orientation d and phase α=6π / 5. Two-dimensional original SIM image 84e is an image acquired by imaging sample 12 (three-dimensional object 12a) illuminated by interference fringes 77 with orientation d and phase α=8π / 5 using image capture device 18. Two-dimensional original SIM image 84e is a moiré image of sample 12 modulated by interference fringes 77 with orientation d and phase α=8π / 5.

[0192] The two-dimensional original SIM images 81, 82, and 83 of the sample 12 obtained as described above are reconstructed using the computer 3, thereby obtaining a three-dimensional original image of the sample 12 (three-dimensional object 12a).

[0193] The computer 3 is communicatively connected to the structured illumination microscope 2b. The display device 4 is communicatively connected to the computer 3. The display device 4 displays the operating state of the structured illumination microscope 2b, images processed by the computer 3 such as two-dimensional original SIM images 81, 82, and 83, or three-dimensional corrected images corrected for optical aberrations, etc. The storage device 55 stores computer programs executed by the processor 30, and images processed by the computer 3 such as the two-dimensional original SIM images 81, 82, and 83, or three-dimensional corrected images corrected for optical aberrations, etc.

[0194] The processor 30 executes computer programs stored in the storage device 55. The computer programs include, for example, a microscope control program, an image reconstruction program including an optical aberration correction program, and an optical wavefront estimation program. The microscope control program, the image reconstruction program including an optical aberration correction program, and the optical wavefront estimation program may be stored in a non-transitory computer-readable storage medium.

[0195] Embodiment 3 is an embodiment in which the optical aberration correction program, optical wavefront estimation program, optical aberration correction method, and optical wavefront estimation method of Embodiment 1 are applied to a structured illumination microscope 2b. When the processor 30 executes the computer program, the computer 3 realizes the functions shown in Figs. 36 and 37. The functional configuration of the computer 3 will be described with reference to Figs. 36 and 37.

[0196] 36, the computer 3 includes a microscope control unit 31, an image reconstruction unit 32b, and an optical wavefront estimation unit 34. As shown in Fig. 37, the image reconstruction unit 32b includes a three-dimensional original image generation unit 32, an optical aberration correction unit 33, a three-dimensional original SIM image generation unit 86, an image extraction unit 87, a deconvolution unit 48b, a three-dimensional inverse Fourier transform unit 49, and an image synthesis unit 88.

[0197] The processor 30 executes the microscope control program, causing the computer 3 to function as a microscope control unit 31. The computer 3 is a control device for the structured illumination microscope 2b. The microscope control unit 31 controls a piezoelectric element or a motor (not shown) connected to at least one of the sample support stage 11 or the objective lens 16 to control the position of at least one of the sample support stage 11 or the objective lens 16. The microscope control unit 31 controls the drive device 67 to control the orientation and position of the light branching element 66. The microscope control unit 31 controls the imaging device 18 to cause the imaging device 18 to capture two-dimensional original SIM images 81, 82, and 83 of the sample 12.

[0198] The processor 30 executes an image reconstruction program including an optical aberration correction program, causing the computer 3 to function as an image reconstruction unit 32b. The computer 3 is both an optical aberration correction device and an image reconstruction device. Alternatively, the computer 3 is an image reconstruction device including an optical aberration correction function. The image reconstruction unit 32b calculates a three-dimensional corrected image of the sample 12 in which the optical aberration of the observation optical system 10 has been eliminated or reduced, from the two-dimensional original SIM images 81, 82, and 83.

[0199] The three-dimensional original image generation unit 32 performs step S56 (see Figures 38 and 40). The optical aberration correction unit 33 performs steps S12 and S60 (see Figure 38). The three-dimensional original SIM image generation unit 86 performs step S53 (see Figure 38). The image extraction unit 87 performs step S55 (see Figure 38). The deconvolution unit 48b performs step S60 (see Figure 38). The three-dimensional inverse Fourier transform unit 49 performs step S65 (see Figure 38). The image synthesis unit 88 performs steps S67 and S69 (see Figure 38).

[0200] The processor 30 executes the optical wavefront estimation program, causing the computer 3 to function as an optical wavefront estimation unit 34. The computer 3 also functions as an optical wavefront estimation device. Similar to the optical wavefront estimation unit 34 of the first embodiment, the optical wavefront estimation unit 34 of the present embodiment generates an optimal optical wavefront that most closely resembles the optical wavefront 22.

[0201] <Image reconstruction method> The image reconstruction method of this embodiment will be described with reference to Figures 38 to 41. The image reconstruction method of this embodiment includes step S12 and steps S51 to S69.

[0202] Referring to FIG. 38, in step S51, two-dimensional original SIM images 81, 82, and 83 of the sample 12 (three-dimensional object 12a) are acquired using the structured illumination microscope 2b.

[0203] Specifically, the microscope control unit 31 (see FIG. 36) controls a piezoelectric element or a motor (not shown) connected to at least one of the sample support stage 11 or the objective lens 16. The position of the focal plane 20 is set to, for example, z1. The microscope control unit 31 controls a drive device 67 connected to the light branching element 66 to rotate the light branching element 66 around the optical axis (z-axis) of the observation optical system 10. The orientation of the light branching element 66 is set. The orientation d of the interference fringes 77 of the illumination light 7b is set to, for example, orientation d=1.

[0204] The microscope control unit 31 controls the drive device 67 connected to the light branching element 66 to linearly move the light branching element 66 along the direction d. The interference fringes 77 move in the periodic direction of the interference fringes 77, changing the phase α of the interference fringes 77. While the interference fringes 77 are moving, the image capture device 18 captures images of the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 at multiple phases α (e.g., at least five phases α). In this way, as shown in FIG. 34, a two-dimensional original SIM image 81 can be acquired at position z1 on the focal plane 20. The two-dimensional original SIM image 81 includes two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e.

[0205] Similarly, in each of the remaining orientations d (e.g., d = 2, 3), the interference fringes 77 are linearly moved while the imaging device 18 captures images of the sample 12 (three-dimensional object 12a) illuminated by the interference fringes 77 at multiple phases α (e.g., at least five phases α). In this manner, as shown in FIG. 34 , two-dimensional original SIM images 82 and 83 can be acquired at position z1 of the focal plane 20. The two-dimensional original SIM image 82 includes two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e. The two-dimensional original SIM image 83 includes two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e.

[0206] Similarly, all remaining focal plane 20 positions z2,...,z n In each of the above, two-dimensional original SIM images 81, 82, and 83 are obtained.

[0207] Then, in the image reconstruction method of this embodiment, the optical aberration of the observation optical system 10 contained in the two-dimensional original SIM images 81, 82, and 83 is removed or reduced, and the two-dimensional original SIM images 81, 82, and 83 are reconstructed to calculate a three-dimensional corrected image in which the optical aberration of the observation optical system 10 has been removed or reduced.

[0208] Each of the two-dimensional original SIM images 81, 82, and 83 (two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e) is a moiré image of the sample 12 (three-dimensional object 12a) modulated by the interference fringes 77 of the illumination light 7b. As described above, the interference fringes 77 include five spatial frequency components of order m. Therefore, each of the two-dimensional original SIM images 81, 82, and 83 (two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e) is a moiré image of the sample 12 (three-dimensional object 12a) modulated by the five spatial frequency components of the interference fringes 77 of order m.

[0209] In frequency space (Fourier space), the frequency distribution of the moire image of the sample 12 (three-dimensional object 12a) modulated by the spatial frequency component of the interference fringes 77 of order m corresponds to the original frequency distribution of the sample 12 (three-dimensional object 12a) shifted by the spatial frequency component of the interference fringes 77 of order m. Therefore, in the original frequency distribution of the sample 12 (three-dimensional object 12a), the cutoff frequency k of the optical transfer function (OTF) of the observation optical system 10 is xmax ,k ymax ,k zmax The original frequency range of the sample 12 (three-dimensional object 12a) (hereinafter referred to as the "super-resolution component of the sample 12 (three-dimensional object 12a)") exceeds the cutoff frequency k xmax ,k ymax ,k zmax The observation optical system 10 can transmit high-resolution images of the sample 12 (three-dimensional object 12 a) corresponding to super-resolution components of the sample 12 (three-dimensional object 12 a). Each of the two-dimensional original SIM images 81, 82, and 83 (two-dimensional original SIM images 84 a, 84 b, 84 c, 84 d, and 84 e) includes a high-resolution image of the sample 12 (three-dimensional object 12 a) corresponding to a super-resolution component of the sample 12 (three-dimensional object 12 a).

[0210] In each of the two-dimensional original SIM images 81, 82, and 83 (two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e), moiré images of five samples 12 (three-dimensional objects 12a) modulated by the spatial frequency components of the five m-order interference fringes 77 are superimposed. Therefore, it is necessary to separate and reconstruct the moiré images of five samples 12 (three-dimensional objects 12a) modulated by the spatial frequency components of the five m-order interference fringes 77. For this reason, the image reconstruction method of this embodiment includes steps S52 to S69 (see FIGS. 38, 40, and 41).

[0211] 38, in step S52, the orientation d of the interference fringes 77 is set on the computer 3. For example, the orientation d of the interference fringes 77 is set to 1 (an azimuth angle of 0°). The computer 3 selects the two-dimensional original SIM image (for example, the two-dimensional original SIM image 81) having the set orientation d from among the two-dimensional original SIM images 81, 82, and 83.

[0212] 38, in step S53, the three-dimensional original SIM image generation unit 86 (see FIG. 37) generates three-dimensional original SIM images 81g, 81h, 81i, 81j, and 81k of the orientation d and phase α set in step S52 from the two-dimensional original SIM image of the orientation d set in step S52. Specifically, with reference to FIGS. 34 and 39, the three-dimensional original SIM image generation unit 86 sorts the two-dimensional original SIM image of the orientation d set in step S52 for each phase α. For each phase α, the three-dimensional original SIM image generation unit 86 generates the two-dimensional original SIM image of the orientation d set in step S52 at multiple positions z1, z2, . . . , z on the focal plane 20. n-1 ,z n These images are stacked over the entire image plane to generate three-dimensional original SIM images 81g, 81h, 81i, 81j, and 81k of orientation d and phase α.

[0213] 38, in step S54, the order m of the spatial frequency component of interference fringes 77 is set on computer 3. For example, any one of 0, ±1 or ±2 is set as the order m.

[0214] 38, in step S55, the image extraction unit 87 (see FIG. 37) extracts the three-dimensional original image I from the three-dimensional original SIM images 81g, 81h, 81i, 81j, and 81k. d,m Extract the original 3D image I d,m is the spatial frequency component of order m of the three-dimensional original SIM images 81g-81k. d,mis a three-dimensional original image of the sample 12 modulated by interference fringes 77 having an orientation d and an order of spatial frequency m. Specifically, the image extraction unit 87 analyzes the correspondence between the way in which the brightness changes at any point in the three-dimensional original SIM images 81g, 81h, 81i, 81j, and 81k and the brightness distribution expected from the frequency and phase of the illumination light 7b, and extracts five three-dimensional original images I from the three-dimensional original SIM images 81g-81k. d,m Extract.

[0215] 38, step S12 for estimating the amount of optical aberration of the observation optical system 10 is the same as step S12 for estimating the amount of optical aberration of the observation optical system 10 in embodiment 1. However, in this embodiment, the sample 12 (three-dimensional object 12a) is illuminated by the interference fringes 77 of the illumination light 7b, whereas in embodiment 1, the sample 12 (three-dimensional object 12a) is illuminated by the illumination light 7 (see FIG. 1) without the interference fringes 77. Therefore, in this embodiment, step S56 is performed before performing step S12 for estimating the amount of optical aberration of the observation optical system 10. With reference to FIG. 40, step S56 includes steps S57, S58, and S59.

[0216] 38, in step S56, the three-dimensional original image generating unit 32 (see FIG. 37) generates the three-dimensional original image 28b from the two-dimensional original SIM images 81, 82, and 83.

[0217] 40, in step S57, the three-dimensional original image generation unit 32 selects at least one orientation d (e.g., orientation d=1). The orientation d three-dimensional original image generation unit 32 selects a two-dimensional original SIM image of the selected orientation d (e.g., two-dimensional original SIM image 81) from among the two-dimensional original SIM images 81, 82, and 83. As shown in FIG. 34, the two-dimensional original SIM image of the selected orientation d (e.g., two-dimensional original SIM image 81) includes two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e having mutually different phases α.

[0218] 40, in step S58, the three-dimensional original image generator 32 generates a two-dimensional original image 27b from the two-dimensional original SIM image of the orientation d selected in step S57. Specifically, the three-dimensional original image generator 32 generates a two-dimensional original image 27b from the two-dimensional original SIM image of the orientation d selected in step S57 at a plurality of positions z1, z2, . . . , z n For each image, two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e are added together. As shown in FIG. 35, the phases α of the two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e are different from one another. Therefore, by adding together the two-dimensional original SIM images 84a, 84b, 84c, 84d, and 84e, a two-dimensional original image 27b of the sample 12 (three-dimensional object 12a) illuminated with the illumination light 7 (see FIG. 1) without interference fringes 77 is obtained. The two-dimensional original image 27b is equivalent to the two-dimensional original image 27 (FIG. 4) of the first embodiment.

[0219] 40, in step S59, the three-dimensional original image generating unit 32 calculates the three-dimensional image at a plurality of positions z1, z2, . . . , z on the focal plane 20. n-1 ,z n A plurality of two-dimensional original images 27b are stacked over the entire area to generate a three-dimensional original image 28b of the sample 12 (three-dimensional object 12a) as a stack image of the plurality of two-dimensional original images 27b. The three-dimensional original image 28b is equivalent to the three-dimensional original image 28 (FIG. 4) of the first embodiment.

[0220] 38, in step S12, the optical aberration corrector 33 (see FIG. 37) uses the three-dimensional original image 28b to estimate the optical aberration of the observation optical system 10. Step S12 in the present embodiment is the same as step S12 in the first embodiment.

[0221] Referring to FIG. 38, the deconvolution unit 48b (see FIG. 37) converts the three-dimensional original image I extracted in step S55 into d,m 41, step S60 includes step S61 for calculating a change in the amplitude of the light 21 due to the spherical aberration of the observation optical system 10, step S62 for calculating an optimal three-dimensional PSF'', step S63 for calculating an optimal OTF'', and step S64 for calculating the three-dimensional original image I.d,m with the optimal OTF''.

[0222] Steps S61, S62, and S63 of this embodiment are similar to steps S34, S35, and S37 (see FIG. 13) of the first embodiment, respectively.

[0223] Step S64 in this embodiment is the same as step S38 in the first embodiment (see FIG. 13). Step S38 in the first embodiment is performed on the original three-dimensional image I, but step S64 in this embodiment is performed on the original three-dimensional image I. d,m That is, as in the sixth modification of the first embodiment, the three-dimensional original image I d,m is different from the original three-dimensional image 28b used in the step (e.g., step S12) of determining the optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') of the observation optical system 10 in step S12 of this embodiment. d,m is common to the three-dimensional original image 28b in that it is an image obtained through the observation optical system 10 and includes the optical aberration of the observation optical system 10. Specifically, in step S64, the deconvolution unit 48b uses equation (38) to derive the three-dimensional original image I d,m is transformed into the three-dimensional original image I d,m Frequency distribution of D d,m get.

[0224]

number

[0225] Then, the deconvolution unit 48b decomposes the three-dimensional original image I d,m Frequency distribution of D d,m is deconvolved with the optimal OTF″ of the observation optical system 10. For example, using equation (39), the three-dimensional original image I d,m Frequency distribution of D d,m The original 3D image I is obtained by multiplying it with the Wiener filter of the optimal OTF ′′.d,m Frequency distribution of D d,m may be deconvolved with the optimal OTF''. Thus, the original 3D image I d,m Frequency distribution of D d,m ''(k x ,k y ,k z ) is calculated. d,m Frequency distribution of D d,m The optimal optical aberration amount closest to the optical aberration amount of the observation optical system 10 is removed from the phase of the three-dimensional original image I. d,m Frequency distribution of D d,m Thus, in step S60, the three-dimensional original image I ′, which includes the optical aberration of the observation optical system 10 of the structured illumination microscope 2 b and is modulated by the interference fringes 77 having the direction d and the order of the spatial frequency m, is obtained. d,m is deconvolved with the optimal OTF′′.

[0226]

number

[0227] The deconvolution method in step S64 is not particularly limited. For example, d,m Frequency distribution of D d,m To avoid artifacts, the cutoff frequency k of the optical transfer function (OTF) of the observation optical system 10 can be reduced by multiplying the inverse filter of the optimal OTF″. xmax ,k ymax ,k zmax Deconvolution is performed only in the frequency space within

[0228] 38, in step S65, the three-dimensional inverse Fourier transform unit 49 converts the three-dimensional original image I d,m Frequency distribution of D d,m In this way, a three-dimensional corrected image I ′ in which the optical aberration of the observation optical system 10 has been removed or reduced is obtained. d,m''(x,y,z) is calculated. This three-dimensional corrected image I d,m '' is an image that includes a super-resolution component of the sample 12 (three-dimensional object 12a) in the direction d and is modulated with a spatial frequency component of order m in the direction d.

[0229]

number

[0230] 38, in step S66, it is confirmed whether steps S55, S60, and S65 have been performed for all orders m on the computer 3. If steps S55, S60, and S65 have not been performed for all orders m, steps S55, S60, and S65 are repeatedly performed while changing the order m until steps S55, S60, and S65 have been performed for all orders m.

[0231] Referring to FIG. 38, when steps S55, S60, and S65 are performed for all orders m, the image synthesis unit 88 (see FIG. 37) generates a three-dimensional corrected image I d,m Then, step S67 is performed in which the signals '' are inversely modulated and synthesized.

[0232] Specifically, the image synthesis unit 88 (see FIG. 37) synthesizes a three-dimensional corrected image I d,m '' is inversely modulated with a sawtooth wave pattern corresponding to the spatial frequency component of the m-th order interference fringe 77. As a result, in frequency space (Fourier space), the super-resolution component of the sample 12 (three-dimensional object 12a) that has shifted by the spatial frequency component of the m-th order interference fringe 77 can be returned (relocated) to the original position of the super-resolution component of the sample 12 (three-dimensional object 12a).

[0233] The image synthesis unit 88 then synthesizes the inversely modulated three-dimensional corrected image I d,m In this way, the image synthesis unit 88 generates a three-dimensional corrected image I of the sample 12 (three-dimensional object 12a) containing the super-resolution component in the direction d. d '' can be calculated. Three-dimensional corrected image Id The resolution in the azimuth direction d is increased, and the three-dimensional corrected image I d '' the optical aberrations of the observation optical system 10 are eliminated or reduced.

[0234] 38, in step S68, it is confirmed whether steps S53-S55, S60, and S65-S67 have been performed for all directions d on the computer 3. If steps S53-S55, S60, and S65-S67 have not been performed for all directions d, steps S53-S55, S60, and S65-S67 are repeatedly performed while changing the direction d until steps S53-S55, S60, and S65-S67 have been performed for all directions d.

[0235] Referring to FIG. 38, when steps S53-S55, S60, and S65-S67 are performed for all directions d, in step S69, the image synthesis unit 88 (see FIG. 37) synthesizes three-dimensional corrected images I for all directions d. d In this way, the image synthesis unit 88 obtains a three-dimensional corrected image I'' of the sample 12 (three-dimensional object 12a) in all directions d, which includes super-resolution components and in which the optical aberration of the observation optical system 10 has been eliminated or reduced.

[0236] The effects of this embodiment will be described. In the optical aberration correction program and optical aberration correction method of the present embodiment, the optical system is the observation optical system 10 of the structured illumination microscope 2b. Therefore, the optical aberration correction program and optical aberration correction method of the present embodiment have the effect of being able to obtain a three-dimensional corrected image with higher resolution, in addition to the effects of the optical aberration correction program and optical aberration correction method of the first embodiment.

[0237] In the optical aberration correction program and the optical aberration correction method of this embodiment, another three-dimensional original image (three-dimensional original image I) modulated by interference fringes 77 of structured illumination microscope 2b, which includes the optical aberration of observation optical system 10 of structured illumination microscope 2b and has orientation d and spatial frequency order m, is generated. d,m) is deconvolved with the optimal optical transfer function (optimal OTF''). Therefore, the optical aberration correction program and the optical aberration correction method of the present embodiment can obtain a three-dimensional corrected image with higher resolution.

[0238] In the optical wavefront estimation program and the optical wavefront estimation method of the present embodiment, the optical system is the observation optical system 10 of the structured illumination microscope 2b. Therefore, the optical wavefront estimation program and the optical wavefront estimation method of the present embodiment have the same effects as the optical wavefront estimation program and the optical wavefront estimation method of the first embodiment.

[0239] (Fourth embodiment) The microscope system 1 of the fourth embodiment will be described mainly with reference to Figures 1, 5 to 7, 9, and 42. The microscope system 1 of the present embodiment has a similar configuration to the microscope system 1 of the first embodiment, but differs from the microscope system 1 of the first embodiment mainly in the following points.

[0240] Referring to Figure 42, in this embodiment, the spherical aberration estimation unit 35, the coma aberration estimation unit 36, and the astigmatism estimation unit 37 each include a theoretical optical wavefront generation unit 40, a theoretical three-dimensional PSF' calculation unit 41, a phase deconvolution unit 42, a three-dimensional phase modulation image generation unit 43, a multiple provisional phase transfer function mPTF' generation unit 51, an optimal phase transfer function bPTF' determination unit 52, and an optimal aberration amount calculation unit 53.

[0241] The theoretical optical wavefront generation unit 40 performs steps S21, S22, and S78 (see Figures 44 and 45). The theoretical three-dimensional PSF' calculation unit 41 performs steps S23 and S79 (see Figures 44 and 45). The phase deconvolution unit 42 performs steps S24d, S73, and S74 (see Figures 44 and 45). The three-dimensional phase modulation image generation unit 43 performs steps S25 and S75 (see Figure 44). The multiple temporary phase transfer function mPTF' generation unit 51 performs step S72 (see Figure 44). The optimal phase transfer function bPTF' determination unit 52 performs steps S27 and S76 (see Figure 44). The optimal aberration amount calculation unit 53 performs step S77 (see Figure 45).

[0242] The optical aberration correction method, optical aberration correction program, optical wavefront estimation method, and optical wavefront estimation program of embodiment 4 will be described mainly with reference to Fig. 10, Fig. 13, Fig. 21, and Fig. 43 to Fig. 52. The optical aberration correction method, optical aberration correction program, optical wavefront estimation method, and optical wavefront estimation program of this embodiment are similar to the optical aberration correction method, optical aberration correction program, optical wavefront estimation method, and optical wavefront estimation program of embodiment 1, respectively, but differ in the following points.

[0243] First, in this embodiment, the theoretical optical wavefront W s In the first embodiment, when the refractive index n2 of the sample 12 is large, the model for generating the appropriate theoretical optical wavefront W s Therefore, in this embodiment, a theoretical optical wavefront W that is affected by the spherical aberration and defocus of the observation optical system 10 is generated based on the Gibson and Lanni model (S.F. Gibson and F. Lanni, Journal of the Optical Society of America A, 1991, Vol. 8, No. 10, pp. 1601-1613). s +W d 43, a theoretical optical wavefront W s +W d is given by the difference between the optical path length of the optical path 21b of the actual observation optical system 10 and the optical path length of the optical path 21a of the ideal observation optical system 10, and specifically given by equation (41).

[0244]

number

[0245] γ in equation (41) is given by equation (42).

[0246]

number

[0247] In an ideal observation optical system 10, the refractive index n liq1 and thickness t liq1 and an immersion medium 15 having a refractive index n designated according to the objective lens 16. g1 and thickness t g1 The observation optical system 10 has a cover glass 14 having a refractive index of 0.25, and the bright spot 13 is located directly below the cover glass 14. An ideal observation optical system 10 has no spherical aberration. The optical path 21a is the optical path of light that emerges from the bright spot 13 and enters the objective lens 16 at an incident angle θ.

[0248] In the actual observation optical system 10, the cover glass 14 has a refractive index n g and thickness t g and the immersion medium 15 has a refractive index n liq and thickness t liq The bright spot 13 is located at a depth t s The sample 12 has a refractive index n s The actual observation optical system 10 has spherical aberration, and the amount of spherical aberration of the observation optical system 10 is n liq and n liq1 The difference between n g and n g1 The difference between t g and t g1 The difference between t s and n s is given by n liq , n g , t g , t s and n s is a parameter relating to the amount of spherical aberration of the observation optical system 10. The optical path 21b is the optical path of light that is emitted from the bright spot 13 and enters the objective lens 16 at an incident angle θ. liq is a parameter related to defocus.

[0249] According to a paper by J. Li et al. (J. Li, F. Xue, and T. Blu, Journal of the Optical Society of America A, 2017, Vol. 34, No. 6, pp. 1029-1034), the right-hand side of equation (41) can be rewritten as equation (42), where NA represents the numerical aperture of the objective lens 16.

[0250]

number

[0251] ρ(u,v) in equation (43) is expressed by equation (44).

[0252]

number

[0253] In this embodiment, the theoretical optical wavefront W s +W d is generated as described above. Therefore, in this embodiment, some equations in the first embodiment are changed. For example, in this embodiment, equation (11) in the first embodiment is replaced with equation (45).

[0254]

number

[0255] In this embodiment, equation (19) in the first embodiment is replaced with equation (46).

[0256]

number

[0257] In this embodiment, equation (21) in the first embodiment is replaced with equation (47).

[0258]

number

[0259] In this embodiment, equation (22) in the first embodiment is replaced with equation (48).

[0260]

number

[0261] In this embodiment, equation (23) in the first embodiment is replaced with equation (49).

[0262]

number

[0263] In this embodiment, equation (25) in the first embodiment is replaced with equation (50).

[0264]

number

[0265] Secondly, as shown in FIGS. 44 to 46, this embodiment differs from the first embodiment in the method for obtaining the optimum theoretical optical wavefront and the optimum theoretical three-dimensional PSF'.

[0266] 10 and 44 to 46, step S13 for estimating the optimum spherical aberration amount of the observation optical system 10 in this embodiment will be described. In step S21 of step S13, a provisional optical aberration amount (for example, a provisional spherical aberration amount) of the observation optical system 10 is set on the computer 3. For example, the refractive index n liq , the refractive index n of the cover glass 14 g and thickness t g , the depth t of the bright spot 13 from the cover glass 14 s , and the refractive index n of sample 12 sBy setting to an arbitrary value, the provisional optical aberration amount (for example, provisional spherical aberration amount) of the observation optical system 10 is set.

[0267] In step S22 of step 13, the theoretical optical wavefront generating unit 40 (see FIG. 42) uses equations (43) and (44) to generate a theoretical optical wavefront W that is affected by the spherical aberration and defocus of the observation optical system 10. s +W d Generate.

[0268] In step S23 of step 13, the theoretical three-dimensional PSF' calculation unit 41 (see FIG. 42) calculates the theoretical optical wavefront W s +W d Specifically, the theoretical three-dimensional PSF' calculation unit 41 calculates a pupil function P'(u,v,z) corresponding to the position z on the focal plane 20 using equation (45). The theoretical three-dimensional PSF' calculation unit 41 calculates a two-dimensional light intensity PSF at the position z on the focal plane 20 from the pupil function P'(u,v,z) using equation (12). z The theoretical three-dimensional PSF' calculation unit 41 calculates the two-dimensional light intensity PSF' (x, y, z) at a plurality of positions z on the focal plane 20. z The theoretical three-dimensional PSF'(x,y,z) is calculated as a stack (assembly) of '.

[0269] 44 and 46, in step S24d, the phase deconvolution unit 42 (see FIG. 42) performs phase deconvolution on the three-dimensional original image I. Step S24d includes step S30 for calculating the theoretical OTF' and step S40 for calculating the phase OTF of the theoretical OTF'. p Step S31d of calculating the unit temporary phase transfer function uPTF'(k x ,k y ,k z ) and the three-dimensional original image I is converted into the theoretical OTF's phase OTF p Step S24d includes a step S32 of deconvolving with a temporary phase transfer function PTF'(kx ,k y ,k z The method may further include step S70 of determining whether the maximum absolute value of (π) is equal to or less than π.

[0270] In step S30, the phase deconvolution unit 42 (see FIG. 42) performs a three-dimensional Fourier transform on the theoretical three-dimensional PSF′(x, y, z) using equation (13) to obtain the theoretical OTF′(k x ,k y ,k z In step S31d, the phase deconvolution unit 42 calculates the phase OTF of the theoretical OTF' using equations (14) and (51). p '(k x ,k y ,k z ) (phase of provisional optical transfer function) and provisional phase transfer function PTF'(k x ,k y ,k z ) is calculated. x ,k y ,k z ) is the phase OTF of the theoretical OTF' p '(k x ,k y ,k z ) is the angular component of the provisional phase transfer function PTF'(k x ,k y ,k z ) is the theoretical OTF'(k x ,k y ,k z ) or the phase OTF of the theoretical OTF' p '(k x ,k y ,k z ) is calculated from

[0271]

number

[0272] In step S70, the phase deconvolution unit 42 (see FIG. 42) calculates the cutoff frequency k xmax ,k ymax ,k zmax The provisional phase transfer function PTF'(k x ,k y ,k z ) is equal to or less than π.

[0273] Provisional phase transfer function PTF'(k x ,k y ,k z ) is greater than π, the aberration-corrected image and the optical wavefront are distorted due to phase wrapping. xmax ,k ymax ,k zmax The provisional phase transfer function PTF'(k x ,k y ,k z ) is determined to be greater than π, the process returns to step S21, and another provisional amount of optical aberration (for example, another provisional amount of spherical aberration) of the observation optical system 10 is set on the computer 3.

[0274] Specifically, the refractive index n of the immersion medium 15 liq , the refractive index n of the cover glass 14 g and thickness t g , the depth t of the bright spot 13 from the cover glass 14 s , or the refractive index n of the sample 12 s By changing at least one of the above, a different provisional optical aberration amount (for example, a different provisional spherical aberration amount) is set. liq By changing the cutoff frequency k of the observation optical system 10, a different provisional optical aberration amount (for example, a different provisional spherical aberration amount) is set. xmax ,k ymax ,k zmax The provisional phase transfer function PTF'(k x ,k y ,k zSteps S21-S23, S30, S31d, and S70 are repeated until the maximum absolute value of (π) is equal to or less than π.

[0275] The phase deconvolution unit 42 (see FIG. 42) calculates the cutoff frequency k xmax ,k ymax ,k zmax The provisional phase transfer function PTF'(k x ,k y ,k z If it is determined that the maximum absolute value of the provisional phase transfer function PTF'(k x ,k y ,k z ) into the unit provisional phase transfer function uPTF'(k x ,k y ,k z ) (step S71). If step S70 is not performed, the provisional phase transfer function PTF' calculated in step S31d is set as the unit provisional phase transfer function uPTF'(k x ,k y ,k z )

[0276] Unit provisional phase transfer function uPTF'(k x ,k y ,k z ) the phase OTF of the theoretical OTF' corresponding to p ' (phase of the temporary optical transfer function) has already been obtained in step S31d. In step S32, the phase deconvolution unit 42 (see FIG. 42) converts the three-dimensional original image I into a unit temporary phase transfer function uPTF'(k x ,k y ,k z ) the phase OTF of the theoretical OTF' corresponding to p Step S32 in this embodiment is the same as step S32 in the first embodiment. Then, in step S25, the three-dimensional phase modulation image generating unit 43 (see FIG. 42) calculates the unit temporary phase transfer function uPTF'(k x ,k y ,k zThe three-dimensional phase-modulated image I'(x, y, z) generated in step S25 is generated by converting the three-dimensional original image into a unit temporary phase transfer function uPTF'(k x ,k y ,k z ) is corrected by a provisional amount of optical aberration (for example, a provisional amount of spherical aberration) corresponding to the calculated value of the aberration. Step S25 in this embodiment is the same as step S25 in the first embodiment.

[0277] In step S72, the multiple temporary phase transfer function mPTF' generator 51 (see FIG. 42) generates the unit temporary phase transfer function uPTF'(k x ,k y ,k z ) to obtain the multiple provisional phase transfer function mPTF'(k x ,k y ,k z For example, as shown in equation (52), the unit temporary phase transfer function uPTF'(k x ,k y ,k z ) by various real numbers β, the multiple provisional phase transfer functions mPTF'(k x ,k y ,k z ) is generated.

[0278]

number

[0279] In step S73, the phase deconvolution unit 42 (see FIG. 42) calculates the multiple temporary phase transfer functions mPTF′(k x ,k y ,k z ) are the phase OTFs of the theoretical OTFs', each corresponding to p '(Phase of provisional optical transfer function)' Generate the phase OTF of multiple theoretical OTFs p Each of the ' is a multiple provisional phase transfer function mPTF'(k x ,k y ,k z ) corresponding to the

[0280]

number

[0281] In step S74, the phase deconvolution unit 42 (see FIG. 42) converts the three-dimensional original image I into the phase OTFs of the plurality of theoretical OTFs' generated in step S73. p Step S74 of this embodiment is the same as step S32 of this embodiment. Then, in step S75, the three-dimensional phase-modulated image generating unit 43 (see FIG. 42) deconvolves the phases of the multiple provisional phase transfer functions mPTF'(k x ,k y ,k z ), a plurality of three-dimensional phase-modulated images I'(x, y, z) corresponding to the respective three-dimensional phase-modulated images I'(x, y, z) are generated. Step S75 of this embodiment is the same as step S25 of this embodiment. Each of the plurality of three-dimensional phase-modulated images I'(x, y, z) generated in step S75 is corrected by a provisional optical aberration amount, and this provisional optical aberration amount is corrected by a plurality of provisional phase transfer functions mPTF'(k x ,k y ,k z ) corresponds to the corresponding provisional phase transfer function.

[0282] In step S27, the optimum phase transfer function bPTF' determination unit 52 (see FIG. 42) determines the unit provisional phase transfer function uPTF'(k x ,k y ,k z ) corresponding to the three-dimensional phase modulation image I′ (step S25) and the multiple temporary phase transfer functions mPTF′(k x ,k y ,k z ) and calculates an index of the brightness value of each of the plurality of three-dimensional phase-modulated images I' (step S75). Step S27 of the present embodiment is similar to step S27 of the first embodiment.

[0283] In step S76, the optimum phase transfer function bPTF' determination unit 52 (see FIG. 42) determines the unit provisional phase transfer function uPTF'(k x ,k y ,k z ) and multiple provisional phase transfer functions mPTF'(k x ,k y ,k z ) and the phase transfer function PTF'(k x ,k y ,k z ) is the optimal phase transfer function bPTF'(k x ,k y ,k z Specifically, the optimum phase transfer function bPTF' determination unit 52 determines the unit provisional phase transfer function uPTF'(k x ,k y ,k z ) and multiple provisional phase transfer functions mPTF'(k x ,k y ,k z ), the provisional phase transfer function corresponding to the three-dimensional phase-modulated image I' in which the index of the brightness value calculated in step S27 is maximized is designated as the optimal phase transfer function bPTF'(k x ,k y ,k z For example, as shown in equation (54), the optimal phase transfer function bPTF' determiner 52 determines the provisional phase transfer function obtained by multiplying the unit provisional phase transfer function uPTF' by δ as the optimal phase transfer function bPTF'(k x ,k y ,k z ) is determined as follows.

[0284]

number

[0285] In step S77 of step S13, the optimum optical aberration calculation unit 53 (see FIG. 42) calculates the optimum optical aberration (for example, the optimum spherical aberration) of the observation optical system 10 corresponding to the optimum phase transfer function bPTF′. liqWhen the optical aberration amount (for example, spherical aberration amount) of the observation optical system 10 is set by changing liqb In equation (55), n liqu is the refractive index of the immersion medium 15 corresponding to the unit provisional phase transfer function uPTF', and n liq1 is the refractive index of the immersion medium 15 in an ideal observation optical system that has no optical aberration (for example, spherical aberration).

[0286]

number

[0287] In step S78, the theoretical optical wavefront generating unit 40 generates an optimal theoretical optical wavefront corresponding to the optimal phase transfer function bPTF' from the optimal optical aberration amount. Step S78 for generating the optimal theoretical optical wavefront is similar to step S22 for generating the theoretical optical wavefront in this embodiment.

[0288] In step S79, the theoretical three-dimensional PSF' calculation unit 41 calculates an optimal theoretical three-dimensional PSF' corresponding to the optimal phase transfer function bPTF'. Step S79 for calculating the optimal theoretical three-dimensional PSF' is similar to step S23 for calculating the theoretical three-dimensional PSF' in this embodiment.

[0289] Step S14 for estimating the optimum amount of coma aberration of the observation optical system 10 in this embodiment will be described mainly with reference to Fig. 10 and Fig. 44 to Fig. 46. Step S14 in this embodiment is similar to step S14 in embodiment 1, but differs in that it includes step S24d and steps S72 to S79.

[0290] In step S77 in step S14, the optimum optical aberration calculation unit 53 (see FIG. 42) calculates the optimum optical aberration (for example, the optimum coma aberration) of the observation optical system 10 corresponding to the optimum phase transfer function bPTF′. As in step S21 in step S14 in the first embodiment, the coefficient c u ,c v The optical aberration amount (for example, coma aberration amount) of the observation optical system 10 changes by changing at least one of the above. The optimal phase transfer function bPTF' determination unit 52 determines the provisional phase transfer function obtained by multiplying the unit provisional phase transfer function uPTF' by δ as the optimal phase transfer function bPTF'(k x ,k y ,k z ), the optimal optical aberration amount (for example, the coma aberration amount) corresponding to the optimal phase transfer function bPTF′ is determined by, for example, the coefficient c ub ,c vb In equation (56), c uu is the coefficient c corresponding to the unit provisional phase transfer function uPTF' u In equation (57), c vu is the coefficient c corresponding to the unit provisional phase transfer function uPTF' v is.

[0291]

number

[0292]

number

[0293] 44 to 46, step S15 for estimating the optimal amount of astigmatism of the observation optical system 10 in this embodiment will be described. Step S15 in this embodiment is similar to step S15 in embodiment 1, but differs in that it includes step S24d and steps S72 to S79.

[0294] In step S77 of step S15, the optimum optical aberration amount calculation unit 53 (see FIG. 42) calculates the optimum optical aberration amount (for example, the optimum astigmatism amount) of the observation optical system 10 corresponding to the optimum phase transfer function bPTF′. As in step S21 of step S15 of the first embodiment, the coefficient a u By changing the value of , the optical aberration (for example, the amount of astigmatism) of the observation optical system 10 changes. The optimal phase transfer function bPTF' determination unit 52 determines the provisional phase transfer function obtained by multiplying the unit provisional phase transfer function uPTF' by δ as the optimal phase transfer function bPTF'(k x ,k y ,k z ), the optimal optical aberration amount (for example, the amount of astigmatism) corresponding to the optimal phase transfer function bPTF' is determined as, for example, the coefficient a shown in Equation (58) ub In equation (58), a uu is the coefficient a corresponding to the unit provisional phase transfer function uPTF' u is.

[0295]

number

[0296] Step S16 (see FIGS. 10 and 13) of calculating a three-dimensional corrected image in this embodiment is similar to step S16 (see FIGS. 10 and 13) of calculating a three-dimensional corrected image in embodiment 1, except that equations (22), (23), and (25) are replaced with equations (48), (49), and (50), respectively. Step S17 (see FIG. 21) of calculating an optimal optical wavefront in this embodiment is similar to step S17 (see FIG. 21) of calculating an optimal optical wavefront in embodiment 1, except that W in equation (30) is replaced with W s (u,v) is given by the right-hand side of equation (43).

[0297] The operation of this embodiment will be described with reference to Figures 47 and 48. Referring to Figure 47, the full width at half maximum of the luminance of the aberration-corrected image of fluorescent beads obtained by the optical aberration correction method and optical aberration correction program of this embodiment is improved to the same extent as the full width at half maximum of the luminance of the aberration-corrected image of fluorescent beads obtained by the optical aberration correction method and optical aberration correction program of Embodiment 1. According to the optical aberration correction method and optical aberration correction program of this embodiment, the quality of the three-dimensional corrected image is improved compared to the quality of the original three-dimensional image, and is improved to near the diffraction limit. According to the optical wavefront estimation method and optical wavefront estimation program of this embodiment, the optical wavefront 22 affected by the optical aberration of the observation optical system 10 can be estimated with higher accuracy.

[0298] 48, the calculation time required to obtain an aberration-corrected image using the optical aberration correction method and optical aberration correction program of this embodiment is significantly shorter than the calculation time required to obtain an aberration-corrected image using the optical aberration correction method and optical aberration correction program of Embodiment 1. This is because the optical aberration correction method and optical aberration correction program of this embodiment include steps S72 to S79 ​​(see FIG. 44), which significantly reduces the number of times steps S22 and S23, which require a long calculation time, are performed. The optical aberration correction method and optical aberration correction program of this embodiment make it possible to correct the optical aberration of the observation optical system 10 contained in the three-dimensional original image in a shorter time (e.g., in real time). The optical wavefront estimation method and optical wavefront estimation program of this embodiment make it possible to estimate the optical wavefront 22 affected by the optical aberration of the observation optical system 10 in a shorter time (e.g., in real time).

[0299] A modification of this embodiment will now be described. 47 to 50, in the optical aberration correction method, the optical aberration correction program, the optical wavefront estimation method, and the optical wavefront estimation program according to the first modified example of this embodiment, the cutoff frequency k of the observation optical system 10 in the three-dimensional original image is xmax ,k ymax ,k zmaxThe optimal theoretical three-dimensional PSF' may be calculated using only the region 90 in which the average value of the amplitude of the high frequency components is relatively large. xmax ,k ymax ,k zmax The high frequency components of the original 3D image are, for example, 0.75k xmax ≦k x ≦k xmax , and 0.75k ymax ≦k y ≦k ymax , and 0.75 zmax ≦k z ≦ zmax are spatial frequency components that satisfy the following.

[0300] The optical aberration of the observation optical system 10 is more clearly apparent in region 90. Because calculations need to be performed on only a portion of the original three-dimensional image, rather than the entire original three-dimensional image, the optical aberration correction method and optical aberration correction program of the first modified example of this embodiment can further reduce the calculation time required to obtain the aberration-corrected image (see FIG. 48) while improving the quality of the three-dimensional corrected image (e.g., the full width at half maximum of the luminance of the image of the fluorescent beads) (see FIG. 47). The optical wavefront estimation method and optical wavefront estimation program of the first modified example of this embodiment can estimate the optical wavefront 22 affected by the optical aberration of the observation optical system 10 with higher accuracy and in a shorter time.

[0301] In the optical aberration correction method according to the second modification of this embodiment, similarly to the fourth modification of the first embodiment, the theoretical optical wavefront W is calculated using only the Zernike polynomials expressed by the equation (31). ze In the second modification of this embodiment, step S12 (see FIG. 10) of estimating the optical aberration of the observation optical system 10 does not include steps S13, S14, and S15, but includes step S18 (see FIG. 26) (however, "high-order optical aberration" in step S18 is replaced with "optical aberration"). The optimal phase transfer function bPTF' determination unit 52 determines the provisional phase transfer function obtained by multiplying the unit provisional phase transfer function uPTF' by δ as the optimal phase transfer function bPTF'(k x ,k y ,k z), the optimum optical aberration amount corresponding to the optimum phase transfer function bPTF' is, for example, the amplitude M of the expansion mode j of the Zernike polynomial j is given by multiplying by δ.

[0302] In the second modified example of this embodiment, similarly to the present embodiment, the calculation time required to obtain an aberration-corrected image is significantly reduced. Furthermore, in the second modified example of this embodiment, higher-order optical aberrations contained in the original three-dimensional image can be corrected in a shorter time period within a predetermined time. Therefore, as shown in FIGS. 51 and 52, the optical aberration correction method and optical aberration correction program of the second modified example of this embodiment further improve the quality of the three-dimensional corrected image. The optical wavefront estimation method and optical wavefront estimation program of the second modified example of this embodiment can estimate the optical wavefront 22 affected by the optical aberrations of the observation optical system 10 with higher accuracy.

[0303] In the third modification of this embodiment, the unit provisional phase transfer function uPTF′(k x ,k y ,k z )'s k x axis and k y By transposing the y-axis and y-axis, the unit provisional phase transfer function uPTF'(k x ,k y ,k z ) may be obtained. The unit provisional phase transfer function uPTF'(k x ,k y ,k z ), steps S22 and S23, which require a lot of calculation time, can be omitted, so that the optical aberration of the observation optical system 10 included in the original three-dimensional image can be corrected in a shorter time. Similarly, the unit temporary phase transfer function uPTF'(k x ,k y ,k z )'s k x axis and k y By transposing the y-axis and y-axis, the unit provisional phase transfer function uPTF'(k x ,k y ,k z) may be obtained. According to the optical aberration correction method and optical aberration correction program of the third modified example of this embodiment, it is possible to correct the optical aberration of the observation optical system 10 contained in the original three-dimensional image in a shorter time (for example, in real time). According to the optical wavefront estimation method and optical wavefront estimation program of the third modified example of this embodiment, it is possible to estimate the optical wavefront 22 affected by the optical aberration of the observation optical system 10 in a shorter time (for example, in real time).

[0304] In addition to the effects of the first embodiment, the present embodiment has the following effects. The optical aberration correction program of this embodiment causes the computer 3 to calculate the phases of a plurality of provisional optical transfer functions of the optical system (observation optical system 10) corresponding to a plurality of provisional optical aberration amounts of the optical system (phases OTF of theoretical OTF′ p a step of deconvolving a three-dimensional original image 28 of the sample 12 including the optical aberration of the optical system with each of the phases of the plurality of interim optical transfer functions to generate a plurality of three-dimensional phase-modulated images I' (e.g., steps S25, S32, S74, S75); and a step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system based on an index of the brightness values ​​of the plurality of three-dimensional phase-modulated images I'. The program executes a step (e.g., steps S76-S79) of calculating an optimal optical transfer function (optimal OTF'') of the optical system corresponding to the optimal theoretical three-dimensional point spread function (e.g., steps S34, S35, S37), and a step (e.g., steps S38, S39) of deconvolving the original three-dimensional image 28 of the sample 12 or another original three-dimensional image that includes optical aberrations of the optical system (observation optical system 10) and is different from the original three-dimensional image 28 of the sample 12 with the optimal optical transfer function to calculate a corrected three-dimensional image.

[0305] The optical aberration correction method of this embodiment calculates the phases of a plurality of provisional optical transfer functions (phases OTF of theoretical OTF′) of the optical system corresponding to a plurality of provisional optical aberration amounts of the optical system (observation optical system 10), respectively. pa step of deconvolving a three-dimensional original image 28 of the sample 12 including the optical aberration of the optical system with each of the phases of the plurality of tentative optical transfer functions to generate a plurality of three-dimensional phase-modulated images I' (e.g., steps S25, S32, S74, S75); and a step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system based on an index of the brightness values ​​of the plurality of three-dimensional phase-modulated images I'. The method includes steps (e.g., steps S76-S79), steps (e.g., steps S34, S35, S37) of calculating an optimal optical transfer function (optimal OTF'') of the optical system corresponding to the optimal theoretical three-dimensional point spread function, and steps (e.g., steps S38, S39) of deconvolving the original three-dimensional image 28 of the sample 12 or another original three-dimensional image that includes optical aberrations of the optical system (observation optical system 10) and is different from the original three-dimensional image 28 of the sample 12 with the optimal optical transfer function to calculate a corrected three-dimensional image.

[0306] In this embodiment, the luminance distribution of the sample 12 and the PSF of the optical system (observation optical system 10) are not calculated alternately and iteratively. Therefore, a three-dimensional corrected image in which optical aberrations of the optical system have been eliminated or reduced can be calculated in a shorter time. Furthermore, in this embodiment, an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system is obtained based on the index of the luminance values ​​of multiple three-dimensional phase-modulated images I'. Therefore, a more accurate optimal theoretical three-dimensional point spread function can be obtained. A three-dimensional corrected image in which optical aberrations of the optical system have been eliminated or reduced with higher accuracy can be calculated.

[0307] In the optical aberration correction program and the optical aberration correction method of the present embodiment, the phases of a plurality of provisional optical transfer functions (phases of the theoretical OTF′ and OTF pThe step of obtaining the theoretical OTF' (for example, steps S31d, S71-S73) includes a step of generating a unit provisional phase transfer function uPTF' of the optical system (observation optical system 10), which is one angle component of the phases of the plurality of provisional phase transfer functions (for example, step S31d, S71), a step of generating a plurality of provisional phase transfer functions mPTF' of the optical system from the unit provisional phase transfer function uPTF' (for example, step S72), and a step of obtaining the phases of the plurality of provisional phase transfer functions (phase OTF of the theoretical OTF') corresponding to the unit provisional phase transfer function uPTF' and the plurality of provisional phase transfer functions mPTF', respectively. p The step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') (e.g., steps S76-S79) includes a step of determining an optimal phase transfer function that most closely resembles the phase transfer function of the optical system from the unit tentative phase transfer function and the multiple tentative phase transfer functions based on an index of the luminance values ​​of the multiple three-dimensional phase-modulated images (e.g., step S76), a step of calculating an optimal optical aberration amount corresponding to the optimal phase transfer function (e.g., step S77), and a step of calculating an optimal theoretical three-dimensional point spread function corresponding to the optimal optical aberration amount (e.g., step S79).

[0308] This significantly reduces the number of times steps S22 and S23, which require a long calculation time, are performed. According to the optical aberration correction program and the optical aberration correction method of the present embodiment, it is possible to calculate a three-dimensional corrected image in which the aberration of the optical system (observation optical system 10) has been eliminated or reduced in a shorter time (for example, in real time) and with higher accuracy.

[0309] In the optical aberration correction program and optical aberration correction method of this embodiment, the maximum absolute value of the unit temporary phase transfer function uPTF' within the cutoff frequency of the optical system (observation optical system 10) is equal to or less than π. Therefore, distortion of the aberration-corrected image caused by phase wrapping can be suppressed. A three-dimensional corrected image in which the aberration of the optical system (observation optical system 10) has been eliminated or reduced with higher accuracy can be calculated.

[0310] In the optical aberration correction program and optical aberration correction method of the present embodiment, the original three-dimensional image that is deconvolved to generate multiple three-dimensional phase-modulated images is a region 90 of the original three-dimensional image where the average value of the amplitude of high-frequency components within the cutoff frequency of the optical system (observation optical system 10) is relatively large. Because calculations need to be performed on only a portion of the original three-dimensional image rather than the entire original three-dimensional image, the time required to correct the aberration of the original three-dimensional image can be further reduced. It is possible to calculate a three-dimensional corrected image in which the aberration of the optical system (observation optical system 10) has been eliminated or reduced in a shorter time (e.g., in real time) and with higher accuracy.

[0311] The optical wavefront estimation program of this embodiment causes the computer 3 to calculate the phases of a plurality of provisional optical transfer functions (phases OTF of theoretical OTF′) of the optical system corresponding to a plurality of provisional optical aberration amounts of the optical system (observation optical system 10), respectively. p The program executes the steps of: obtaining a three-dimensional phase-modulated image I′ (e.g., steps S31d and S71-S73); deconvolving the three-dimensional original image 28 of the sample 12, including the optical aberration of the optical system, with each of the phases of the plurality of tentative optical transfer functions to generate a plurality of three-dimensional phase-modulated images I′ (e.g., steps S25, S32, S74, and S75); obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) that most closely approximates the three-dimensional point spread function of the optical system based on the index of the brightness values ​​of the plurality of three-dimensional phase-modulated images I′ (e.g., steps S76-S79); calculating an optimal optical transfer function (optimal OTF″) of the optical system corresponding to the optimal theoretical three-dimensional point spread function (e.g., steps S34, S35, and S37); and calculating an optimal optical wavefront corresponding to the optimal theoretical three-dimensional point spread function (e.g., steps S17 and S78).

[0312] The optical wavefront estimation method of this embodiment is to estimate the phases of a plurality of provisional optical transfer functions (phases OTF of theoretical OTF′) of an optical system corresponding to a plurality of provisional optical aberration amounts of the optical system (observation optical system 10), respectively. pthe step of obtaining an optimal theoretical three-dimensional point spread function (optimum theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system based on the index of the brightness values ​​of the plurality of three-dimensional phase-modulated images I' (e.g., steps S76-S79); the step of calculating an optimal optical transfer function (optimum OTF'') of the optical system corresponding to the optimal theoretical three-dimensional point spread function (e.g., steps S34, S35, S37); and the step of calculating an optimal optical wavefront corresponding to the optimal theoretical three-dimensional point spread function (e.g., steps S17, S78).

[0313] In this embodiment, the luminance distribution of the sample 12 and the PSF of the optical system (observation optical system 10) are not calculated alternately and iteratively. Therefore, the optical wavefront 22 affected by the optical aberration of the optical system can be calculated in a shorter time. Furthermore, in this embodiment, an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF') that most closely approximates the three-dimensional point spread function of the optical system is obtained based on the index of the luminance values ​​of multiple three-dimensional phase-modulated images I'. Therefore, the optimal theoretical three-dimensional point spread function can be obtained more accurately from multiple theoretical three-dimensional point spread functions. The optical wavefront 22 affected by the optical aberration of the optical system can be estimated with higher accuracy.

[0314] In the optical wavefront estimation program and the optical wavefront estimation method of the present embodiment, the phases of a plurality of provisional optical transfer functions (phases of the theoretical OTF′, OTF pThe step of obtaining the theoretical OTF' (for example, steps S31d, S71-S73) includes a step of generating a unit provisional phase transfer function uPTF' of the optical system (observation optical system 10), which is one angle component of the phases of the plurality of provisional phase transfer functions (for example, step S31d, S71), a step of generating a plurality of provisional phase transfer functions mPTF' of the optical system from the unit provisional phase transfer function uPTF' (for example, step S72), and a step of obtaining the phases of the plurality of provisional phase transfer functions (phase OTF of the theoretical OTF') corresponding to the unit provisional phase transfer function uPTF' and the plurality of provisional phase transfer functions mPTF', respectively. p The method includes a step of generating an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) (e.g., steps S31d and S73). The step of obtaining an optimal theoretical three-dimensional point spread function (optimal theoretical three-dimensional PSF′) (e.g., steps S76-S79) includes a step of determining an optimal phase transfer function that is most approximate to the phase transfer function of the optical system from the unit tentative phase transfer function and the multiple tentative phase transfer functions based on an index of the brightness values ​​of the multiple three-dimensional phase-modulated images (e.g., step S76), a step of calculating an optimal optical aberration amount corresponding to the optimal phase transfer function (e.g., step S77), and a step of calculating an optimal theoretical optical wavefront corresponding to the optimal optical aberration amount (e.g., step S78).

[0315] This makes it possible to significantly reduce the number of times steps S22 and S23, which require a long calculation time, are performed. According to the optical wavefront estimation program and the optical wavefront estimation method of the present embodiment, it is possible to estimate the optical wavefront 22 affected by the optical aberration of the optical system (observation optical system 10) in a shorter time (for example, in real time) and with higher accuracy.

[0316] In the optical wavefront estimation program and optical wavefront estimation method of the present embodiment, the maximum absolute value of the unit temporary phase transfer function uPTF' within the cutoff frequency of the optical system (observation optical system 10) is equal to or less than π. Therefore, distortion of the optical wavefront 22 caused by phase wrapping can be suppressed. The optical wavefront 22 affected by the optical aberration of the optical system (observation optical system 10) can be estimated with higher accuracy.

[0317] In the optical wavefront estimation program and optical wavefront estimation method of the present embodiment, the original three-dimensional image that is deconvolved to generate multiple three-dimensional phase-modulated images is a region 90 of the original three-dimensional image where the average value of the amplitude of high-frequency components within the cutoff frequency of the optical system (observation optical system 10) is relatively large. Since calculations need to be performed on only a portion of the original three-dimensional image rather than the entire original three-dimensional image, the time required to calculate the optical wavefront 22 affected by the aberration of the observation optical system 10 can be further reduced. The optical wavefront 22 affected by the optical aberration of the optical system can be estimated in a shorter time (for example, in real time).

[0318] The presently disclosed embodiments 1 to 4 and their modifications should be considered to be illustrative in all respects and not restrictive. As long as there is no contradiction, at least two of the presently disclosed embodiments 1 to 4 and their modifications may be combined. For example, in the first embodiment, the theoretical optical wavefront W s In the first embodiment, step S24d and steps S72 to S79 ​​of the fourth embodiment may be adopted. The fourth embodiment may be applied to the structured illumination microscope (SIM) 2b of the third embodiment. The scope of the present disclosure is defined by the claims, rather than the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0319] 1, 1a, 1b microscope system, 2 microscope, 2b structured illumination microscope, 3, 3a, 3b, 3c computer, 4 display device, 6, 6b light source, 7, 7b illumination light, 8, 8b illumination optical system, 9a lens, 9b lens, 10 observation optical system, 11 sample support, 12 sample, 12a three-dimensional object, 12b surrounding medium, 13 bright spot, 14 cover glass, 15 immersion medium, 16 objective lens, 17 imaging lens, 18 imaging device, 19 dichroic mirror, 20 focal plane, 21 light, 22 optical wavefront, 23 pupil plane, 25 reflected light, 27, 27b two-dimensional original image, 28, 28b three-dimensional original image, 30, 30a, 30b, 30c processor, 31 microscope control unit, 32 three-dimensional original image generation unit, 32b Image reconstruction unit, 33 optical aberration correction unit, 34 optical wavefront estimation unit, 35 spherical aberration estimation unit, 36 coma aberration estimation unit, 37 astigmatism estimation unit, 38 aberration-corrected image generation unit, 39 optical wavefront generation unit, 40 theoretical optical wavefront generation unit, 41 theoretical three-dimensional PSF' calculation unit, 42 phase deconvolution unit, 43 three-dimensional phase modulation image generation unit, 44 optimum theoretical three-dimensional PSF' determination unit, 45 optimum theoretical optical wavefront determination unit, 46 amplitude change calculation unit, 47 optimum three-dimensional PSF'' calculation unit, 48, 48b deconvolution unit, 49 three-dimensional inverse Fourier transform unit, 50 normalized optimum three-dimensional PSF'' calculation unit, 51 multiple temporary phase transfer function generation unit, 52 optimum phase transfer function determination unit, 53 optimum aberration amount calculation unit, 55, 55a, 55b, 55c, 55d Memory device, 64 optical fiber, 65 collimating lens, 66 optical branching element, 67 driving device, 70 relay lens, 71, 72, 73 lens, 77 interference fringes, 81, 82, 83, 84a, 84b, 84c, 84d, 84e two-dimensional original SIM image, 81g, 81h, 81i, 81j, 81k three-dimensional original SIM image, 86 three-dimensional original SIM image generation unit, 87 image extraction unit, 88 image synthesis unit, 90 area.

Claims

1. An optical aberration correction program executed by a computer, the optical aberration correction program including: obtaining only the phases of a plurality of three-dimensional interim optical transfer functions of the optical system, the phases corresponding to a plurality of interim optical aberration quantities of the optical system; deconvolving an original three-dimensional image of the sample, including optical aberrations of the optical system, with each of the phases of the plurality of three-dimensional interim optical transfer functions to generate a plurality of three-dimensional phase-modulated images; obtaining an optimal theoretical three-dimensional point spread function that most closely approximates the three-dimensional point spread function of the optical system based on an index of the luminance values ​​of the plurality of three-dimensional phase-modulated images; calculating an optimal optical transfer function of the optical system corresponding to the optimal theoretical three-dimensional point spread function; and a step of deconvolving the original three-dimensional image or another original three-dimensional image that includes the optical aberration of the optical system and is different from the original three-dimensional image with the optimal optical transfer function to calculate a three-dimensional corrected image.

2. obtaining the optimal theoretical three-dimensional point spread function includes determining the optimal theoretical three-dimensional point spread function from a plurality of theoretical three-dimensional point spread functions of the optical system based on the index of the luminance values ​​of the plurality of three-dimensional phase-modulated images; The optical aberration correction program according to claim 1 , wherein each of the plurality of theoretical three-dimensional point spread functions includes a corresponding one of the plurality of provisional optical aberration amounts.

3. the step of obtaining only the phases of the plurality of three-dimensional interim optical transfer functions includes the steps of: generating a unit interim phase transfer function of the optical system, which is one angle component of the phases of the plurality of three-dimensional interim optical transfer functions; generating a plurality of interim phase transfer functions of the optical system from the unit interim phase transfer function; and generating the phases of the plurality of three-dimensional interim optical transfer functions corresponding to the unit interim phase transfer function and the plurality of interim phase transfer functions, respectively; 2. The optical aberration correction program according to claim 1, wherein the step of obtaining the optimal theoretical three-dimensional point spread function includes the steps of: determining an optimal phase transfer function that most closely approximates a phase transfer function of the optical system from the unit tentative phase transfer function and the plurality of tentative phase transfer functions based on the index of the luminance values ​​of the plurality of three-dimensional phase-modulated images; calculating an optimal optical aberration amount corresponding to the optimal phase transfer function; and calculating the optimal theoretical three-dimensional point spread function corresponding to the optimal optical aberration amount.

4. 4. The optical aberration correction program according to claim 1, wherein the optimum optical transfer function is normalized by an amplitude of the optical transfer function of the optical system when the optical system does not have the optical aberration.

5. 5. The optical aberration correction program according to claim 1, wherein the optimal theoretical three-dimensional point spread function includes at least two of spherical aberration, coma aberration, and astigmatism of the optical system.

6. The optical aberration correction program according to claim 1 , wherein the optical system is an observation optical system of a structured illumination microscope.

7. The optical aberration correction program of claim 6, wherein deconvolving the three-dimensional original image or another three-dimensional original image that includes the optical aberration of the optical system and is different from the three-dimensional original image with the optimal optical transfer function comprises deconvolving the other three-dimensional original image that includes the optical aberration of the observation optical system of the structured illumination microscope and is modulated by interference fringes of the structured illumination microscope having an orientation and an order of spatial frequency with the optimal optical transfer function.

8. An optical wavefront estimation program executed by a computer, the optical wavefront estimation program including: obtaining only the phases of a plurality of three-dimensional interim optical transfer functions of the optical system, the phases corresponding to a plurality of interim optical aberration quantities of the optical system; deconvolving an original three-dimensional image of the sample, including optical aberrations of the optical system, with each of the phases of the plurality of three-dimensional interim optical transfer functions to generate a plurality of three-dimensional phase-modulated images; obtaining an optimal theoretical three-dimensional point spread function that most closely approximates the three-dimensional point spread function of the optical system based on an index of the luminance values ​​of the plurality of three-dimensional phase-modulated images; and calculating an optimal optical wavefront corresponding to the optimal theoretical three-dimensional point spread function.

9. obtaining the optimal theoretical three-dimensional point spread function includes determining the optimal theoretical three-dimensional point spread function from a plurality of theoretical three-dimensional point spread functions of the optical system based on the index of the luminance values ​​of the plurality of three-dimensional phase-modulated images; The optical wavefront estimation program according to claim 8 , wherein each of the plurality of theoretical three-dimensional point spread functions includes a corresponding one of the plurality of provisional optical aberration amounts.

10. the step of obtaining only the phases of the plurality of three-dimensional interim optical transfer functions includes the steps of: generating a unit interim phase transfer function of the optical system, which is one angle component of the phases of the plurality of three-dimensional interim optical transfer functions; generating a plurality of interim phase transfer functions of the optical system from the unit interim phase transfer function; and generating the phases of the plurality of three-dimensional interim optical transfer functions corresponding to the unit interim phase transfer function and the plurality of interim phase transfer functions, respectively; 9. The optical wavefront estimation program according to claim 8, wherein the step of obtaining the optimal theoretical three-dimensional point spread function includes the steps of: determining an optimal phase transfer function that most closely approximates a phase transfer function of the optical system from the unit tentative phase transfer function and the plurality of tentative phase transfer functions based on the index of the luminance values ​​of the plurality of three-dimensional phase-modulated images; calculating an optimal optical aberration amount corresponding to the optimal phase transfer function; and calculating the optimal theoretical optical wavefront corresponding to the optimal optical aberration amount.

Images