Microscope, image processing device, image processing method, and image processing program

Abstract

[Problem] To more simply enhance resolution / sectioning in the optical axis direction of an optical system of a microscope. [Solution] The present invention comprises: a generation unit that generates a group of images including a first image of a first focal plane obtained via an optical system in a first state and a second image of a second focal plane obtained via the optical system in a second state, a first three-dimensional point image intensity distribution based on the optical system in the first state being different from a second three-dimensional point image intensity distribution based on the optical system in the second state; and an estimation unit that estimates the structure of each of a plurality of planes including a first estimation sample plane on the basis of the group of images, the first three-dimensional point image intensity distribution, and the second three-dimensional point image intensity distribution, and that outputs an estimated image. The estimation unit (i) outputs, as the estimated image, an integrated image upon performing a weighting process on the estimated structure of each of the plurality of planes. The estimation unit (ii) estimates the structure after performing different weighting processes in the optical axis direction for the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputs, as the estimated image, an integrated image in a state of a weight 1.

Description

[Technical Field]

[0001] The present invention relates to a microscope, an image processing apparatus, an image processing method, and an image processing program. [Background technology]

[0002] One technique for generating an image of a sample based on the light detected from the sample when it is illuminated with illumination light by an optical system is the technique described in Non-Patent Document 1 below. The technique described in Non-Patent Document 1 estimates the three-dimensional structure of a sample based on a two-dimensional image of the sample and the three-dimensional point image intensity distribution of the optical system. In the technique described in Non-Patent Document 1, it is necessary to use a special phase plate in order to estimate the three-dimensional structure over a wide range in the optical axis direction of the optical system (hereinafter, this direction will be appropriately referred to as the Z direction). For this reason, it is desirable to improve the resolution and sectioning ability in the optical axis direction of the microscope's optical system in a simpler way. [Prior art documents] [Patent Documents]

[0003] [Non-Patent Document 1] C. Roider et al.,Deconvolution approach for 3D scanning microscopy with helical phase engineering,Biomedical Optics Express,24,14,15456(2016) [Overview of the project]

[0004] According to an aspect of the present invention, the present invention comprises an optical system that irradiates a sample with illumination light and guides detection light from the sample to a detector, and an image processing device that generates an image of the sample based on a signal from the detector and processes the image, wherein the first three-dimensional point image intensity distribution based on the optical system in the first state is different from the second three-dimensional point image intensity distribution based on the optical system in the second state which is different from the first state. The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by placing the point detector in a first position, and the second state is formed by placing the point detector in a second position different from the first position. ii) When the detector consists of multiple detection units or multiple point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting an optical element into the optical system, and the second state is formed by removing the optical element from the optical system. iv) The first state is formed by inserting the first optical element into the optical system, and the second state is formed by inserting the second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. The provided microscope is an image processing device that includes a generation unit that generates a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination position of the illumination light, including a first image of a first focal plane based on a signal acquired through the optical system in a first state and a second image of a second focal plane that substantially coincides with the first focal plane and the optical system in the optical axis direction, based on a signal acquired through the optical system in a second state, and the relative positional relationship in the optical axis direction between the sample and the illumination position of the illumination light; and an estimation unit that estimates the structure of each of a plurality of planes in the sample, including a first estimated sample plane, along the optical axis direction, based on the group of images and a first three-dimensional point image intensity distribution and a second three-dimensional point image intensity distribution, and outputs an estimated image based on the estimated structure, wherein the estimation unit includes at least one of the following: i) Different weighting processes are applied to each of the estimated structures in the optical axis direction, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first 3D point image intensity distribution and the second 3D point image intensity distribution, the structure is estimated. The estimated structure The integrated image is output as an estimated image. According to a first aspect of the present invention, there is provided an optical system that irradiates a sample with illumination light and guides detection light from the sample to a detector, and an image processing device that generates an image of the sample based on a signal from the detector and processes the image. The first three-dimensional point image intensity distribution based on the optical system in the first state is different from the second three-dimensional point image intensity distribution based on the optical system in the second state different from the first state. The image processing device includes a first image of a first focal plane based on a signal obtained through the optical system in the first state, and a second image of a second focal plane substantially coinciding with the first focal plane in the optical axis direction based on a signal obtained through the optical system in the second state. A generation unit that generates an image group based on signals obtained without changing the relative positional relationship in the optical axis direction between the sample and the illumination light irradiation position, and based on the image group, the first three-dimensional point image intensity distribution, and the second three-dimensional point image intensity distribution, estimates the structure of each of a plurality of planes including a first estimated sample plane in the sample along the optical axis direction, and an estimation unit that outputs an estimated image based on the estimated structure. The estimation unit includes at least one of the following, and a microscope is provided. i) An image obtained by performing a weighting process on the structure of each of the plurality of estimated planes and integrating it is output as the estimated image. ii) After performing different weighting processes in the optical axis direction on the first three-dimensional point image amplitude distribution and the second three-dimensional point image amplitude distribution, the structure is estimated, and an image integrated in the state of weighting 1 is output as the estimated image.

[0005] According to an aspect of the present invention, there is provided an optical system that irradiates a sample with illumination light and guides detection light from the sample to a detector, and an image processing device that generates an image of the sample based on a signal from the detector and processes the image. The first three-dimensional point image amplitude distribution based on the optical system in the first state is different from the second three-dimensional point image amplitude distribution based on the optical system in the second state different from the first state. The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by placing the point detector in a first position, and the second state is formed by placing the point detector in a second position different from the first position. ii) When the detector consists of multiple detection units or multiple point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting an optical element into the optical system, and the second state is formed by removing the optical element from the optical system. iv) The first state is formed by inserting the first optical element into the optical system, and the second state is formed by inserting the second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. The provided microscope includes an image processing device which generates a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination position of the illumination light, and which includes a first image of a first focal plane based on a signal acquired through the optical system in a first state and a second image of a second focal plane that substantially coincides with the first focal plane and the optical system in the optical axis direction, based on a signal acquired through the optical system in a second state, and the relative positional relationship in the optical axis direction between the sample and the illumination position of the illumination light; and an estimation unit which estimates the structure of each of a plurality of planes in the sample, including a first estimated sample plane, along the optical axis direction, based on the group of images and a first three-dimensional point image amplitude distribution and a second three-dimensional point image amplitude distribution, and outputs an estimated image based on the estimated structure, wherein the estimation unit includes at least one of the following. i) Different weighting processes are applied to each of the estimated structures in the optical axis direction, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first 3D point image amplitude distribution and the second 3D point image amplitude distribution, the structure is estimated. The estimated structure The accumulated image is output as the estimated image. According to a second aspect of the present invention, a microscope is provided comprising: an optical system that irradiates a sample with illumination light and guides detection light from the sample to a detector; an image processing device that generates an image of the sample based on a signal from the detector and processes the image, wherein a first three-dimensional point image amplitude distribution based on the optical system in a first state differs from a second three-dimensional point image amplitude distribution based on an optical system in a second state different from the first state, and the image processing device includes a generation unit that generates a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination light irradiation position, and an estimation unit that estimates the structure of each of a plurality of surfaces in the sample, including a first estimated sample surface, along the optical axis direction, based on the group of images, the first three-dimensional point image amplitude distribution and the second three-dimensional point image amplitude distribution, and outputs an estimated image based on the estimated structure, wherein the estimation unit includes at least one of the following. i) The estimated image is output as the estimated image by applying weighting to the structure of each of the multiple estimated surfaces and integrating them. ii) After applying different weighting processes in the optical axis direction to the first 3D point image amplitude distribution and the second 3D point image amplitude distribution, the structure is estimated, and the integrated image obtained with weighting 1 is output as the estimated image.

[0006] According to an aspect of the present invention, in a detector of detected light from a sample when the sample is irradiated with illumination light by an optical system, signal An image processing apparatus that generates an image of a sample based on and processes the image, wherein the first three-dimensional point image intensity distribution based on the optical system of the first state is different from the second three-dimensional point image intensity distribution based on the optical system of the second state which is different from the first state, The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by placing the point detector in a first position, and the second state is formed by placing the point detector in a second position different from the first position. ii) When the detector consists of multiple detection units or multiple point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting an optical element into the optical system, and the second state is formed by removing the optical element from the optical system. iv) The first state is formed by inserting the first optical element into the optical system, and the second state is formed by inserting the second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. An image processing apparatus is provided, comprising: a generation unit that generates a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination light irradiation position of the optical system, including a first image of a first focal plane based on a signal acquired in a first state optical system and a second image of a second focal plane based on a signal acquired in a second state optical system; and an estimation unit that estimates the structure of each of a plurality of planes in the sample, including a first estimated sample plane, along the optical axis direction, based on the group of images and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputs an estimated image based on the estimated structure, wherein the estimation unit includes at least one of the following: i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first 3D point image intensity distribution and the second 3D point image intensity distribution, the structure is estimated. The estimated structureThe integrated image is output as an estimated image. According to a third aspect of the present invention, an image processing apparatus is provided that generates an image of a sample based on a detection signal in a detector of detected light from a sample when the sample is irradiated with illumination light by an optical system, and processes the image, wherein when the first three-dimensional point image intensity distribution based on the optical system in a first state is different from the second three-dimensional point image intensity distribution based on an optical system in a second state which is different from the first state, the generating unit generates a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination light irradiation position of the optical system, and includes a first image of the first focal plane based on a signal acquired in the optical system in the first state and a second image of the second focal plane based on a signal acquired in the optical system in the second state; and an estimation unit estimates the structure of each of a plurality of planes in the sample, including a first estimated sample plane, along the optical axis direction, based on the group of images, the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputs an estimated image based on the estimated structure, wherein the estimation unit includes at least one of the following. i) The estimated image is output as the estimated image by applying weighting to the structure of each of the multiple estimated surfaces and integrating them. ii) After applying different weighting processes in the optical axis direction to the first 3D point image intensity distribution and the second 3D point image intensity distribution, the structure is estimated, and the integrated image with a weight of 1 is output as the estimated image.

[0007] According to an aspect of the present invention, in a detector of detected light from a sample when the sample is irradiated with illumination light by an optical system, signal A method for generating an image of a sample and processing the image, wherein the first three-dimensional point image intensity distribution based on the optical system of the first state is different from the second three-dimensional point image intensity distribution based on the optical system of the second state which is different from the first state, The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by placing the point detector in a first position, and the second state is formed by placing the point detector in a second position different from the first position. ii) When the detector consists of multiple detection units or multiple point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting an optical element into the optical system, and the second state is formed by removing the optical element from the optical system. iv) The first state is formed by inserting the first optical element into the optical system, and the second state is formed by inserting the second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. An image processing method is provided, which includes generating a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination position of the optical system, including a first image of a first focal plane based on a signal acquired in the optical system in a first state and a second image of a second focal plane based on a signal acquired in the optical system in a second state, and estimating the structure of each of a plurality of planes in the sample, including a first estimated sample plane, along the optical axis direction, based on the group of images and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputting an estimated image based on the estimated structure, wherein the output of the estimated image includes at least one of the following: i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, the structure is estimated. The estimated structure The accumulated image is output as the estimated image. According to a fourth aspect of the present invention, an image processing method is provided for generating an image of a sample based on a detection signal in a detector of detected light from a sample when the sample is irradiated with illumination light by an optical system, wherein, when the first three-dimensional point image intensity distribution based on an optical system in a first state is different from the second three-dimensional point image intensity distribution based on an optical system in a second state which is different from the first state, the method includes generating a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination light irradiation position of the optical system, and estimating the structure of each of a plurality of surfaces in the sample, including a first estimated sample surface, along the optical axis direction, based on the group of images and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputting an estimated image based on the estimated structure, wherein the output of the estimated image includes at least one of the following. i) The estimated image is output as the estimated image by applying weighting to the structure of each of the multiple estimated surfaces and integrating them. ii) After applying different weighting processes in the optical axis direction to the first 3D point image intensity distribution and the second 3D point image intensity distribution, the structure is estimated, and the integrated image with a weight of 1 is output as the estimated image.

[0008] According to an aspect of the present invention, in a detector of detected light from a sample when the sample is irradiated with illumination light by an optical system, signal Based on this, an image of the sample is generated, and in order to process the image, the computer is instructed to consider the case where the first 3D point image intensity distribution based on the optical system of the first state is different from the second 3D point image intensity distribution based on the optical system of the second state, which is different from the first state. The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by placing the point detector in a first position, and the second state is formed by placing the point detector in a second position different from the first position. ii) When the detector consists of multiple detection units or multiple point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting an optical element into the optical system, and the second state is formed by removing the optical element from the optical system. iv) The first state is formed by inserting the first optical element into the optical system, and the second state is formed by inserting the second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. An image processing program is provided that performs the following actions: generating a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination light irradiation position of the optical system, including a first image of the first focal plane based on a signal acquired in the optical system of the first state and a second image of the second focal plane based on a signal acquired in the optical system of the second state; estimating the structure of each of a plurality of planes in the sample, including a first estimated sample plane, along the optical axis direction, based on the group of images, a first three-dimensional point image intensity distribution, and a second three-dimensional point image intensity distribution, and outputting an estimated image based on the estimated structure, wherein outputting the estimated image includes at least one of the following. i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, the structure is estimated. The estimated structureThe accumulated image is output as the estimated image. According to a fifth aspect of the present invention, an image processing program is provided that generates an image of a sample based on a detection signal in a detector of detected light from a sample when illumination light is irradiated onto the sample by an optical system, and processes the image by causing a computer to perform the following actions: generate a group of images based on signals acquired without changing the relative positional relationship in the optical axis direction between the sample and the illumination light irradiation position of the optical system, including a first image of the first focal plane based on a signal acquired in the optical system of the first state and a second image of the second focal plane based on a signal acquired in the optical system of the second state, when the first three-dimensional point image intensity distribution based on the optical system of the first state is different from the second three-dimensional point image intensity distribution based on the optical system of the second state, and estimate the structure of each of a plurality of planes on the sample, including a first estimated sample plane, along the optical axis direction, based on the group of images and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and output an estimated image based on the estimated structure, wherein the output of the estimated image includes at least one of the following. i) The estimated image is output as the sum of the structures of each of the multiple estimated surfaces, after applying a weighting process to them. ii) After applying different weighting processes in the optical axis direction to the first 3D point image intensity distribution and the second 3D point image intensity distribution, the structure is estimated, and the integrated image obtained with weighting 1 is output as the estimated image. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing an example of the configuration of a microscope according to the first embodiment. [Figure 2] This figure shows an example where the intensity profile in the Z direction of the three-dimensional point image intensity distribution differs between the optical system in the first state and the optical system in the second state according to the first embodiment. [Figure 3] This diagram schematically shows the relationship between the focal plane, the sample plane, the first plane, the second plane, and the estimated sample plane according to the first embodiment. [Figure 4] This figure shows examples of multiple images generated by the generation unit of the first embodiment. [Figure 5] This figure shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system in the first state and the optical system in the second state of the first embodiment. [Figure 6] This is a flowchart showing the flow of the image processing method according to the first embodiment. [Figure 7] This figure shows an example of an estimated image based on the structure estimated by the estimation unit of the first embodiment. [Figure 8] This is a schematic diagram showing an example of the configuration of a microscope according to the second embodiment. [Figure 9] This figure schematically shows the relationship between the focal plane, the sample plane, the first plane, the second plane, and the estimated sample plane according to the second embodiment. [Figure 10] This is a schematic diagram showing an example of the configuration of a microscope according to the third embodiment. [Figure 11] This figure shows the configuration of the detector according to the third embodiment. [Figure 12] This figure schematically shows the relationship between the focal plane, the sample plane, the first plane, the second plane, and the estimated sample plane according to the third embodiment. [Figure 13] This is a schematic diagram showing an example of the configuration of a microscope according to the fourth embodiment. [Figure 14] This figure shows the general shape of the XZ cross-section of the 3D point image intensity distribution in the first state and the general shape of the XZ cross-section of the 3D point image intensity distribution in the second state according to the fourth embodiment. [Figure 15] This figure shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system in the first state and the optical system in the second state of the fourth embodiment. [Figure 16] This is a schematic diagram showing an example of the configuration of a microscope according to the fifth embodiment. [Figure 17] This figure shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system in the first state and the optical system in the second state of the fifth embodiment. [Figure 18] This is a schematic diagram showing an example of the configuration of a microscope according to the sixth embodiment. [Figure 19]This figure shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system in the first state and the optical system in the second state of the sixth embodiment. [Figure 20] This is a schematic diagram showing an example of the configuration of a microscope according to the seventh embodiment. [Figure 21] This figure shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system in the first state and the optical system in the second state of the seventh embodiment. [Figure 22] This is a schematic diagram showing an example of the configuration of a microscope according to the 8th embodiment. [Figure 23] This is a schematic diagram showing an example of the configuration of a microscope according to the 9th embodiment. [Figure 24] This figure shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system of the first state and the optical system of the ninth embodiment. [Figure 25] This is a schematic diagram showing an example of the configuration of a microscope according to the 10th embodiment. [Figure 26] This figure shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system of the first state and the optical system of the tenth embodiment. [Figure 27] This is a schematic diagram showing an example of the configuration of a microscope according to the 11th embodiment. [Figure 28] This figure shows examples of the two-dimensional point image amplitude distribution on the first plane and the two-dimensional point image amplitude distribution on the second plane in the optical system of the first state and the optical system of the 11th embodiment. [Figure 29] This is a schematic diagram showing an example of the configuration of a microscope according to the 12th embodiment. [Figure 30] This figure shows examples of multiple images generated by the generation unit of the 12th embodiment. [Figure 31] This figure shows an example of an estimated image based on the structure estimated by the estimation unit of the 12th embodiment. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the drawings. In the drawings, the scale has been appropriately changed, such as by enlarging or emphasizing certain parts, in order to illustrate the embodiments. Some drawings use an XYZ coordinate system to explain the directions in the figures. In this XYZ coordinate system, the direction along the optical system of the microscope is defined as the Z direction. The plane perpendicular to this Z direction is defined as the XY plane. One direction in this XY plane is denoted as the X direction, and the direction perpendicular to the X direction is denoted as the Y direction.

[0011] [First Embodiment] A first embodiment will now be described. Figure 1 is a schematic diagram showing an example of the configuration of a microscope according to the first embodiment. In the following first embodiment, the microscope 1A will be described as a confocal microscope. As shown in Figure 1, the microscope 1A comprises a microscope body 10A and an image processing device 100A. The microscope body 10A comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15A.

[0012] (Microscope body configuration) The light source 11 emits illumination light L1, such as a laser. The light source 11 may be a monochromatic (single wavelength) light source or a multicolor (multiple wavelength) light source. The light source 11 may be either a laser that emits continuous oscillation light or a laser that emits pulsed light. Furthermore, the light source 11 does not have to be a laser; it may be an LED or a lamp. When a fluorescent substance is used as the sample 8, it is preferable to select a wavelength for the light source 11 that excites the fluorescent substance contained in the sample 8. When a fluorescent substance is used as the sample 8, it is also possible to select a wavelength for the light source 11 that multiphoton-excites the fluorescent substance contained in the sample 8. The light source 11 may be provided in a replaceable (attachable, detachable) manner on the microscope body 10A. The light source 11 may also be attached externally to the microscope body 10A when observing with the microscope body 10A. In this case, for example, illumination light L1 may be incident into the microscope body 10A from the light source 11 outside the microscope body 10A via an existing optical component such as an optical fiber.

[0013] The optical system 12 irradiates the sample 8 with illumination light L1 from the light source 11 and guides the detection light L2 from the sample 8 to the detector 15A. The optical system 12 includes a collimator lens 120, an objective lens 121, a pupil projection lens 122, a lens 123, a deflection unit 124, an optical path separation element 125, a focusing lens 126, etc. The collimator lens 120 converts the illumination light L1 emitted from the light source 11, such as a laser, into approximately parallel light. The optical path separation element 125 consists of a dichroic mirror or the like. The optical path separation element 125 directs the illumination light L1 that has passed through the collimator lens 120 into the deflection unit 124.

[0014] The deflection unit 124 directs the incident illumination light L1 to the objective lens 121 via the pupil projection lens 122 and lens 123. The deflection unit 124 is provided with, for example, an X-direction deflection mirror and a Y-direction deflection mirror, which are composed of, for example, a galvanometer mirror, a MEMS mirror, a resonant mirror (resonant mirror), etc. The deflection unit 124 is positioned with respect to the sample 8 via the objective lens 121, lens 123, and pupil projection lens 122, so as to be approximately conjugate to the pupil position of the objective lens 121.

[0015] The objective lens 121 irradiates the incident illumination light L1 onto the sample 8 held on the stage 2. The objective lens 121 forms an illumination region 14 on the sample 8, where the illumination light L1 is focused to a size approximately equal to the resolution limit of the objective lens 121. If the light source 11 is of a single wavelength, the illumination region 14 on the sample 8 will be an illumination region 14 illuminated by light of a single wavelength. If the light source 11 contains multiple wavelengths, the illumination region 14 on the sample 8 will be an illumination region 14 containing multiple wavelengths. As the deflection unit 124 swings in a predetermined direction, the illumination region 14 moves in a plane perpendicular to the optical axis direction of the illumination light L1.

[0016] The scanning control unit 13 controls the oscillation of the X-direction deflection mirror and Y-direction deflection mirror of the deflection unit 124 in predetermined directions, thereby scanning the illumination area 14 on the sample 8 along a plane perpendicular to the optical axis direction of the illumination light L1. Alternatively, the scanning control unit 13 may move the stage 2, which holds the sample 8, along a plane perpendicular to the optical axis direction of the illumination light L1, thereby performing relative scanning between the illumination area 14 and the sample 8 on the stage 2. Furthermore, the system may be configured to perform both scanning by the deflection unit 124 and scanning by the stage 2.

[0017] Light (detection light) L2 emitted from the sample 8 upon illumination of the illumination area 14 enters the objective lens 121. The detection light L2 is refracted by the objective lens 121 and passes through lens 123 and pupil projection lens 122 to the deflection unit 124. The detection light L2 is reflected by the deflection unit 124 and returned to almost the same optical path as the illumination light L1 (descanned) to the optical path separation element 125. The detection light L2 passes through the optical path separation element 125 and the focusing lens 126 before entering the detector 15A.

[0018] The optical system that guides the illumination light L1 emitted from the light source 11 to the sample 8 is the illumination optical system. That is, each optical component (optical path separation element 125, deflection unit 124, pupil projection lens 122, lens 123, objective lens 121, etc.) arranged on the optical path from the optical path separation element 125 to the sample 8 constitutes the illumination optical system. Furthermore, the optical system that guides the light emitted from the sample 8 to the detector 15A is the detection optical system. That is, each optical component (objective lens 121, pupil projection lens 122, lens 123, deflection unit 124, optical path separation element 125, focusing lens 126, etc.) arranged on the optical path from the sample 8 to the detector 15A constitutes the detection optical system.

[0019] The detector 15A is positioned conjugate to the illumination region 14 on the sample 8 via the detection optical system, that is, at a position where it forms an image with the illumination region 14 on the sample 8 via the detection optical system. On the light-receiving surface of the detector 15A, an image 15m of the fluorescence of the sample 8 excited in the illumination region 14 is formed as an image of the illumination region 14. Even when the deflection unit 124 is driven to perform relative scanning between the illumination region 14 and the sample 8, the image 15m on the light-receiving surface of the detector 15A remains stationary regardless of the state of the deflection unit 124. This is because the image 15m is deflected (descanned) in the opposite direction to the illumination light L1 when the detection light L2 passes through the deflection unit 124.

[0020] (Detector configuration) Detector 15A detects the image 15m formed on the light-receiving surface of detector 15A. In this embodiment, detector 15A is a point detector 151 whose light-receiving surface is sufficiently smaller than the fluorescence image 15m of sample 8. Point detector 151 includes a photoelectric conversion element (not shown) made of a semiconductor or the like. The photoelectric conversion element (not shown) of detector 15A outputs a signal (electrical signal) corresponding to the detected amount of light to the image processing device 100A. In this embodiment, detector 15A is made movable between multiple positions in a plane perpendicular to the optical axis direction of the detected light L2 by the detector control unit 16. Focusing on two of these multiple positions, detector 15A is made movable between an arbitrary first position G1 and a second position G2 in a plane perpendicular to the optical axis direction of the detected light L2. The movement of detector 15A in a plane perpendicular to the optical axis direction of the detected light L2 is controlled by the detector control unit 16. The detector 15A moves along a plane perpendicular to the optical axis, thereby detecting the amount of light in the image 15m at each of multiple positions in that plane perpendicular to the optical axis.

[0021] (3D point image intensity distribution of the optical system) Figure 2 shows an example in which the intensity profile in the Z direction of the three-dimensional point image intensity distribution differs between the optical system in the first state and the optical system in the second state according to the embodiment. When the position of the detector 15A changes in a plane perpendicular to the optical axis, its relative position to the light source 11 changes. When the relative position of the detector 15A with respect to the light source 11 changes, the three-dimensional point image intensity distribution (3D-PSF) via the optical system 12 changes. In other words, as shown in Figure 2, the three-dimensional point image intensity distribution via the optical system 12 is different in the first state A1, where the detector 15A is located at an arbitrary first position G1, and in the second state A2, where the detector 15A is located at an arbitrary second position G2 different from the first position G1 in a plane perpendicular to the optical axis. Here, the three-dimensional point image intensity distribution is a function that quantitatively represents the intensity distribution of the detected light L2 emitted from a point object present in the sample 8 and detected by the detector 15A, and the optical system in the first state and the optical system in the second state are determined by the relative relationship with the light source 11, the optical system 12, and the detector 15A, respectively. Therefore, the optical system in the first state and the optical system in the second state include the light source 11, the optical system 12, and the detector 15A. The same applies even when the configuration of the optical system 12 and the detector are different, as in other embodiments.

[0022] The image processing device 100A generates an image of the sample 8 based on the signal from the detector 15A and processes the image. The image processing device 100A consists of a computer such as a personal computer. The image processing device 100A is equipped with hardware such as a CPU and memory. The image processing device 100A functionally has the configuration shown below, through the cooperation of the CPU, memory, etc., and the image processing program stored in the memory and storage device to execute predetermined processing.

[0023] (Configuration of the image processing device) The image processing device 100A functionally includes a signal receiving unit 101, a generation unit 102, and an estimation unit 103. The signal receiving unit 101 receives a signal corresponding to the light intensity of the image 15m, which is output from the detector 15A. The generation unit 102 generates a two-dimensional image (two-dimensional image data) of the sample 8 at the focal plane F based on the signal corresponding to the light intensity from the detector 15A and the relative positional relationship between the illumination area 14 and the sample 8 when the light intensity signal was detected.

[0024] (Regarding the focal plane) Figure 3 is a schematic diagram showing the relationship between the focal plane, the sample plane, the first plane, the second plane, and the estimated sample plane according to the embodiment. The focal plane F in the optical system in each state is a plane defined by the three-dimensional point image intensity distribution of the optical system in each state. When the relative position of the detector 15A with respect to the light source 11 changes, the three-dimensional point image intensity distribution via the optical system 12 changes. Therefore, as shown in Figure 3, the position of the focal plane F in the optical system in each state can change in the optical axis direction (Z direction) depending on the position of the detector 15A with respect to the light source 11. In the example in Figure 3, the position of the detector 15A in the first state A1 (first position G1) is different from the position of the detector 15A in the second state A2 (second position G2). Therefore, the three-dimensional point image intensity distribution via the optical system is different between the first state A1 and the second state A2, and the positions of the first focal plane Fa in the first state A1 and the second focal plane Fb in the second state A2 can be different in the optical axis direction (Z direction). In addition, the positions of the first focal plane Fa and the second focal plane Fb in the optical axis direction (Z direction) may coincide. Generally, if the amount of movement of the detector 15A is equal to or less than the magnitude of the point image intensity distribution of the optical system, the first focal plane Fa and the second focal plane Fb coincide.

[0025] In this embodiment, the focal plane F is, for example, the plane (XY plane) where the center position of the Gaussian exists when a 3D Gaussian fitting is performed to find an approximate function of the 3D point image intensity distribution of the optical system 12 in each corresponding state. The focal plane F may be, for example, the plane (XY plane) at the position where the maximum brightness of the 3D point image intensity distribution of the optical system in each corresponding state exists in the Z direction, or the plane (XY plane) where the maximum value of the integrated value of the 3D point image intensity distribution of the optical system in each corresponding state exists on planes intersecting in the axial direction (Z direction). In this embodiment, the focal plane F is, for example, the XY plane in the Z coordinate where the center position of the Gaussian exists when a 3D Gaussian fitting is performed on the 3D point image intensity distribution of an optical system in a certain state.

[0026] The detector 15A detects images 15m at multiple positions in a plane perpendicular to the optical axis. Therefore, the generation unit 102 generates multiple images G of the sample 8 based on the amount of light detected by the detector 15A at multiple positions in a plane perpendicular to the optical axis. As shown in Figure 3, the generation unit 102 generates a first image Ga of the surface of the sample 8 at a position corresponding to the first focal plane Fa in the Z direction (referred to as the first sample surface Sa) based on the amount of light detected by the detector 15A in the first state A1. The generation unit 102 generates a second image Gb of the surface of the sample 8 at a position corresponding to the second focal plane Fb in the Z direction (referred to as the second sample surface Sb) based on the amount of light detected by the detector 15A in the second state A2.

[0027] (Definition of "approximate agreement" of the focal plane) Here, the focal planes F of the optical system in all states that acquire images G of multiple samples generated by the generation unit 102 are at the same or close together in the Z direction. In this embodiment, if the distribution of the Z coordinates of all focal planes F falls within a range of 10% or less of the respective Z resolution (half-width of the Z profile of the 3D point image intensity distribution) of the optical system in all states, it is said that the multiple focal planes F are approximately coincident.

[0028] (Processing in the generation unit) As described above, the generation unit 102 generates a two-dimensional image G of the sample 8 at all focal planes F based on a signal corresponding to the light intensity from the detector 15A and the relative positional relationship between the illumination area 14 and the sample 8 when the light intensity signal is detected. The two-dimensional image G is represented by the following equation (1).

[0029]

number

[0030] Here, I(x, y, z) is image G, h(x, y, z) is the three-dimensional point image intensity distribution in the optical system 12, and s(x, y, z) is the fluorescent molecule distribution in sample 8.

[0031] Figure 4 shows examples of multiple images generated by the generation unit. The image group G generated by the generation unit 102 includes, as shown in Figures 3 and 4, a first image Ga of the first focal plane Fa based on the signal from the detector 15A via the optical system 12 in the first state A1, and a second image Gb of the second focal plane Fb which substantially coincides with the first focal plane Fa in the optical axis direction (Z direction), based on the signal from the detector 15A via the optical system 12 in the second state A2. The first image Ga is an observation image acquired when the first focal plane Fa coincides with the first sample surface Sa of the sample 8. The second image Gb is an observation image acquired when the second focal plane Fb coincides with the second sample surface Sb of the sample 8. Note that m1 to m3 are objects within the sample 8.

[0032] (Processing in the estimation unit) The estimation unit 103 receives data from multiple images G generated by the generation unit 102. The multiple images G are captured when the detector 15A is positioned at multiple locations in a plane perpendicular to the optical axis direction to detect the amount of light in the image 15m, with the focal plane F of the optical system in each state aligned with the sample surface S. Here, the positions in the Z direction (Z coordinates) of the focal plane F of the optical system in all states for which multiple images G are acquired are approximately the same as those of the others, based on the definition described above.

[0033] The estimation unit 103 sets an estimated sample surface for estimating the structure of sample 8. In the example in Figure 3, the estimation unit 103 estimates the structure of sample 8 using a first estimated sample surface Q1 at a position corresponding to the first surface P1 in the Z direction, and a second estimated sample surface Q2 at a position corresponding to the second surface P2 in the Z direction. The first surface P1 is determined by approximately coinciding with the focal plane F. The first surface P1 is selected such that, when the maximum value of the z coordinate of each focal plane F of the optical system in all states in which multiple images G are acquired is Zmax and the minimum value is Zmin, the Z coordinate of the first surface P1 is between Zmax and Zmin. Furthermore, if the Z coordinates of the focal plane F of the optical system in all states in which multiple images G are acquired are the same, the Z coordinate of the first surface P1 will be the same as the Z coordinate of the focal plane F. The estimation unit 103 estimates the structure of each of the multiple estimated sample surfaces along the optical axis direction (Z direction) of the optical system 12 based on the multiple images G generated by the generation unit 102, and outputs an estimated image Ie of the first estimated sample surface Q1.

[0034] The second plane P2 is set as a non-focal plane. That is, the second plane P2 is set outside the range of the z coordinates of the focal plane F (Zmin~Zmax). Here, the estimation unit 103 preferably selects the second plane P2 such that, when the minimum value of the total Z resolution of the optical system in all states for acquiring images G of multiple samples input to the estimation unit 103 is δz, the difference between the Z coordinate of the second plane P2 (z=z2) and the Z coordinate of the first plane P1 (z=z1) is greater than 0.05δz and less than or equal to 0.5δz.

[0035] The estimation unit 103 calculates, or pre-stores, the 2D point image intensity distributions of the first plane P1 and the second plane P2 for each of the 3D point image intensity distributions of the optical system in all states for which multiple images G are acquired. In the 3D point image intensity distribution h(x,y,z) of the optical system in each of the aforementioned states, the point image intensity distribution h(x,y,z1) when the z coordinate is the z coordinate of the first plane P1 (z=z1) is defined as the 2D point image intensity distribution H1 of the first plane P1. Similarly, the point image intensity distribution h(x,y,z2) when the z coordinate is the z coordinate of the second plane P2 (z=z2) is defined as the 2D point image intensity distribution H2 of the second plane P2.

[0036] (Regarding the relative relationship of 2D point image intensity distributions across multiple surfaces) Figure 5 shows examples of the two-dimensional point image intensity distribution of the first plane P1 and the second plane P2 in the optical system of the first state of the embodiment, and the two-dimensional point image intensity distribution of the first plane P1 and the second plane P2 in the optical system of the second state. In this embodiment, as shown in Figure 5, the first relative relationship between the two-dimensional point image intensity distribution H1a of the first plane P1 and the two-dimensional point image intensity distribution H2a of the second plane P2 in the first three-dimensional point image intensity distribution based on the optical system of the first state A1 is different from the second relative relationship between the two-dimensional point image intensity distribution H1b of the first plane P1 and the two-dimensional point image intensity distribution H2b of the second plane P2 in the second three-dimensional point image intensity distribution based on the optical system of the second state A2, which is different from the first state A1.

[0037] The estimation unit 103 uses at least one set of images G to estimate the structure of each of several surfaces in the sample 8, including the first estimated sample surface Q1, along the optical axis direction (Z direction) of the optical system 12. In the example in Figure 3, the estimation unit 103 utilizes the fact that the relative relationship of the two-dimensional point image intensity distribution is different to use at least one set of images G to separately estimate the structure of the sample 8 in the first estimated sample surface Q1 and the second estimated sample surface Q2. Note that if there is no difference in the relative relationship of the two-dimensional point image intensity distribution, the structure of the sample 8 in the first estimated sample surface Q1 and the second estimated sample surface Q2 cannot be correctly estimated.

[0038] (Estimation based on the relative intensity of the 2D point image intensity distribution) The estimation unit 103 can estimate, for example, the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the difference in relative intensity between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. At this time, it is desirable that there is a predetermined difference between the first relative intensity between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2 in the optical system of the first state A1, and the second relative intensity between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2 in the optical system of the second state A2. The sum of the luminance of the two-dimensional point image intensity distribution H1a on the first surface P1 of the optical system of the first state A1 is If Let 1 be the sum of the luminances of the 2D point image intensity distribution H1b of the second surface P2 of the optical system in the first state A1. df Let 1 be the sum of the luminances of the two-dimensional point image intensity distribution H1b of the first surface P1 of the optical system in the second state A2 be I f Let 2 be the sum of the luminances of the two-dimensional point image intensity distribution H2b of the second surface P2 of the optical system in the second state A2 be I df When set to 2, it is desirable to satisfy, for example, the following equations (2) or (3).

[0039]

number

[0040]

number

[0041] (Estimation of the 3D distribution of an object) Various known algorithms can be used for object estimation in the estimation unit 103, such as the Wiener filter, gradient method, and Richardson-Lucy method. Here, for example, we will use the gradient method algorithm. The two-dimensional image Im, captured by the detector 15A located at the m-th position, is expressed as shown in equation (4) below.

[0042]

number

[0043] Here, s is a P-dimensional vector representing the three-dimensional distribution of fluorescent molecules, and A m This is an N×P matrix represented by equation (5) below (N=Nx×Ny).

[0044]

number

[0045] Here, (x n ,y nis the coordinate of the illumination area 14 of the illumination light L1, (x p , y p , z p ) is the coordinate of the sample space. h m represents the three-dimensional point image intensity distribution of the optical system when the detector 15A is arranged at the m-th position (state).

[0046] The estimation unit 103 estimates the fluorescence molecule distribution in the sample 8 by minimizing the error function F(s) represented by the following formula (6).

[0047]

Equation

[0048] Here, I mes is the image (N-dimensional vector) actually acquired by the detector 15A arranged at the m-th position. To ensure the positive value of the P-dimensional vector s, when s = ρ 2 is set, the error function F(s) of the above formula (6) is expressed as the following formula (7).

[0049]

Equation

[0050] The estimation unit 103 performs a process of finding ρ that minimizes the error function represented by the above formula (7). In the estimation unit 103, when the gradient method is used, the following formula (8) is obtained in the i-th iteration.

[0051]

Equation

[0052] Here, d i represents the gradient of the error function F(s) with respect to ρ, and α is a coefficient that specifies the amount of progress in the gradient direction. When the change rate of ρ when the number of iterations is increased becomes small, or when the value of ρ when the preset number of iterations is reached, the fluorescence molecule distribution can be obtained from s = ρ 2 .

[0053] The estimation unit 103 estimates the distribution of fluorescent molecules on the first estimated sample surface Q1 and the second estimated sample surface Q2.

[0054] The estimation unit 103 separates and estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2. However, the second estimated sample surface Q2 may be set at multiple different positions along the optical axis direction (Z direction) of the optical system 12. For example, the second estimated sample surface Q2 may be set on the + side and the - side of the optical axis direction (Z direction) relative to the first estimated sample surface Q1.

[0055] (Image processing method) Next, the image processing method for the microscope body 10A described above will be explained. In the image processing method shown below, each process performed by the image processing device 100A of the microscope body 10A is performed by the image processing device 100A based on a program pre-stored in the image processing device 100A. This program may be provided by being recorded on a computer-readable storage medium (e.g., a non-transitory tangible recording medium).

[0056] Figure 6 is a flowchart showing the flow of the image processing method according to the embodiment. As shown in Figure 6, in step S11, illumination light L1 from the light source 11 is irradiated onto the sample 8 set on the stage 2. The illumination light L1 emitted from the light source 11 is irradiated onto the sample 8 held on the stage 2 via the collimator lens 120, optical path separation element 125, deflection unit 124, pupil projection lens 122, imaging lens 123, and objective lens 121. As a result, an illumination region 14 is formed on the sample 8, where the illumination light L1 is focused by the objective lens 121. The scanning control unit 13 controls the oscillation of the deflection unit 124 to scan the illumination region 14 in two dimensions in the XY direction on the sample 8. During the period when the illumination region 14 is being scanned in two dimensions in the XY direction on the sample 8, the relative positional relationship between the sample 8 and the objective lens 121 in the Z direction is not changed (maintained and fixed), and the illumination region 14 scans within a predetermined Z plane of the sample 8. The light (detection light) L2 emitted from the sample 8 upon illumination of the illumination area passes through the objective lens 121, imaging lens 123, pupil projection lens 122, deflection unit 124, optical path separation element 125, and focusing lens 126 before entering the detector 15A.

[0057] In step S12, the detector control unit 16 moves the detector 15A to multiple positions in the XY plane, and at each position, the detector 15A detects the amount of light of the fluorescence image 15m of the sample 8 formed on the light-receiving surface of the detector 15A. The photoelectric conversion element (not shown) of the detector 15A outputs a signal (electrical signal) corresponding to the amount of light received to the image processing device 100A. Even when the detector 15A is moved to multiple positions in the XY plane, the relative positional relationship in the Z direction between the sample 8 and the objective lens 121 does not change (it is maintained and fixed), and the illumination area 14 scans within a predetermined Z plane of the sample 8.

[0058] In step S13, the image processing device 100A generates an image G of the sample 8 based on the signal from the detector 15A and processes the generated image G. The signal receiving unit 101 of the image processing device 100A receives a signal corresponding to the amount of light of the fluorescence image 15m of the sample 8, which is received by a photoelectric conversion element (not shown) and output from the detector 15A. The generation unit 102 generates a two-dimensional image G of the sample 8 at the focal plane F based on the signal corresponding to the amount of light from the detector 15A and the relative positional relationship between the illumination area 14 and the sample 8 when the light intensity signal was detected. Here, the generation unit 102 generates multiple images G (image group) based on the amount of light detected by the detector 15A at multiple positions in the XY plane. Image G is an image acquired without changing the relative positional relationship in the Z direction between the sample 8 and the objective lens 121 (optical system 12). The image group G generated by the generation unit 102 includes a first image Ga of the first focal plane Fa based on the signal from the detector 15A via the optical system 12 in a first state A1 where the detector 15A is located at a first position G1, and a second image Gb of the second focal plane Fb which substantially coincides with the first focal plane Fa in the optical axis direction (Z direction) based on the signal from the detector 15A via the optical system 12 in a second state A2 where the detector 15A is located at a second position G2. The first image Ga is acquired with the first focal plane Fa aligned with the first sample surface Sa of the sample 8. The second image Gb is acquired with the second focal plane Fb aligned with the second sample surface Sb of the sample 8.

[0059] In step S14, the estimation unit 103 estimates the structure of each of the multiple surfaces on the sample 8 along the optical axis direction (Z direction) of the optical system, based on the image group generated by the generation unit 102, and outputs an estimated image Ie based on the estimated structure. The estimation unit 103 receives data of multiple images G (image group) generated by the generation unit 102 as input. The estimation unit 103 calculates or stores in advance the 2D point image intensity distributions of the first surface P1 and the second surface P2 in the 3D point image intensity distribution based on the optical system. The first surface P1 is determined to approximately coincide with the focal plane F. The second surface P2 is a non-focal plane and differs from the first surface P1 in the optical axis direction (Z direction) of the optical system.

[0060] The first relative relationship between the two-dimensional point image intensity distribution of the first surface P1 and the two-dimensional point image intensity distribution of the second surface P2 in the first three-dimensional point image intensity distribution based on the optical system of the first state A1 is different from the second relative relationship between the two-dimensional point image intensity distribution of the first surface P1 and the two-dimensional point image intensity distribution of the second surface P2 in the second three-dimensional point image intensity distribution based on the optical system of the second state A2, which is different from the first state A1. The estimation unit 103 estimates the structure of each of several surfaces in the sample 8 along the optical axis direction (Z direction) of the optical system, including the first estimated sample surface Q1, using at least one set of images G. The sample surface S corresponding to the first surface P1 is designated as the first estimated sample surface Q1, and the sample surface S corresponding to the second surface P2 is designated as the second estimated sample surface Q2. The estimation unit 103 estimates, for example, the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative intensity of the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2.

[0061] As described above, the estimation unit 103 estimates the structure of each of the multiple estimated sample surfaces (estimated sample surfaces) and outputs an estimated image Ie based on the estimated structure. For example, the estimated image on the first estimated sample surface Q1 and the estimated image on the second estimated sample surface Q2 are each assigned different weights, and the sum (integrated) of these images is taken as Ie.

[0062]

number

[0063] Here, w k This is the k-th estimated sample surface Q. k This is the weight for the estimated image in the first estimated sample surface Q1, and can take any real value. When the weight for the estimated image in the first estimated sample surface Q1 is greater than the weight for the estimated image in the second estimated sample surface Q2, the resulting estimated image Ie has a relatively reduced structure of the non-focal plane compared to the first image Ga and the second image Gb, and it can be said that the resolution in the optical axis direction (Z direction) has improved. Furthermore, by adjusting the weight, it is possible to change the sectioning ability of the estimated image Ie. Furthermore, consider a modified 3D point image intensity distribution, given by the following equation, in which different weights are assigned to each z-value of the 3D point image intensity distribution.

[0064]

number

[0065] Here, wz is the z-weight of the 3D point image intensity distribution and can take any positive real value other than 0.

[0066] Using this, equation (5) can be rewritten as follows.

[0067]

number

[0068] Here,

number

[0069] This represents an estimated structure with different weights assigned to each z value. From this, it can be seen that using a deformed 3D point image intensity distribution allows for the estimation of a structure with different weights assigned to each z value. In this case, an estimated image Ie equivalent to the above can be obtained by integrating the estimated images on each estimated sample surface with a weight of 1.

[0070]

number

[0071] Furthermore, the flowchart showing the image processing method in Figure 6 is also applicable to other embodiments. In other embodiments, the microscope body (optical system configuration, detector type) and image processing device are different (resulting in different optical systems in the first and second states), but the flowchart can be applied by substituting the contents of each step of the flowchart with the configuration and functions of the microscope body and image processing device of each embodiment.

[0072] Figure 7 shows an example of an estimated image Ie based on the structure estimated by the estimation unit of the first embodiment. As described above, the estimation unit 103 analyzes multiple first images Ga and second images Gb (see Figure 4) acquired in the first state A1 and second state A2, respectively, to remove or reduce objects m2 and m3 that are not present on the first estimated sample surface Q1, and outputs an estimated image Ie in which object m1 present on the first estimated sample surface Q1 is emphasized.

[0073] With the microscope 1A and image processing device 100A described above, the detector 15A, consisting of point detectors 151, performs detection at multiple positions. Based on the image set, which includes a first image Ga of the first focal plane Fa based on the signal from the point detector 151 in the first state A1, and a second image Gb of the second focal plane Fb based on the signal from the point detector 151 in the second state A2, the structure of each of the multiple planes in the sample 8, including the first estimated sample plane Q1, is estimated. Therefore, the sample structure can be estimated based on more image information obtained by the point detector 151 performing detection at multiple positions. As a result, objects present on the first estimated sample plane Q1 can be estimated with greater accuracy, and an estimated image Ie with improved Z resolution and sectioning ability is obtained.

[0074] In this specification, Z resolution refers to the full width at half maximum (FWHM) of the Z profile at the (x,y) coordinates where the peak intensity of the 3D point image intensity distribution exists. Sectioning ability refers to the FWHM of the Z profile of the integrated value within the xy plane of the 3D point image.

[0075] [Second Embodiment] A second embodiment will now be described. Figure 8 is a schematic diagram showing an example of the configuration of a microscope according to the second embodiment. In this embodiment, components similar to those in the above-described embodiment may be denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 8, the microscope 1B comprises a microscope body 10B and an image processing device 100B. The microscope body 10B comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15B.

[0076] The optical system 12 of the microscope body 10B irradiates the sample 8 with illumination light L1 from the light source 11 and guides the detection light L2 from the sample 8 to the detector 15B.

[0077] In this embodiment, the detector 15B includes a plurality of point detectors 152. The plurality of point detectors 152 are arranged in a plane perpendicular to the optical axis direction of the detected light L2. The plurality of point detectors 152 are integrally mounted on a base member or the like (not shown). Here, there are no limitations on the number of point detectors 152 installed, their arrangement, etc. For example, the plurality of point detectors 152 may be arranged linearly in one direction along the plane perpendicular to the optical axis direction of the detected light L2. Alternatively, the plurality of point detectors 152 may be arranged discretely in the plane perpendicular to the optical axis direction of the detected light L2. In Figure 8, for example, three point detectors 152 are arranged along the vertical direction of the paper in Figure 8.

[0078] Each point detector 152 is equipped with a photoelectric conversion element (not shown) made of a semiconductor or the like. Each photoelectric conversion element (not shown) of each point detector 152 outputs a signal (electrical signal) corresponding to the amount of light received to the image processing device 100B. Detector 15B detects the amount of light in the image 15m at multiple positions in a plane perpendicular to the optical axis direction using each of the multiple point detectors 152.

[0079] (3D point image intensity distribution of the optical system) The multiple point detectors 152 that make up detector 15B are located at different positions in a plane perpendicular to the optical axis. As a result, the relative positions of the multiple point detectors 152 with respect to the light source 11 change. When the relative positions of detectors 15B with respect to the light source 11 change, the three-dimensional point image intensity distribution (3D-PSF) via the optical system 12 differs, as shown in Figure 2. As shown in Figure 8, if we focus on two of the multiple point detectors 152, the three-dimensional point image intensity distribution via the optical system 12 differs between the detector 152 in the first state A11 located at the first position G11 and the point detector 152 in the second state A12 located at an arbitrary second position G12 different from the first position G11 in the plane perpendicular to the optical axis.

[0080] The image processing device 100B generates multiple images of the sample 8 based on signals from multiple point detectors 152 of the detector 15B, and processes these images. The image processing device 100B includes a signal receiving unit 101, a generation unit 102, and an estimation unit 103, the functions of which are as described in the first embodiment and are also applied in this embodiment.

[0081] When the relative positions of the multiple point detectors 152 with respect to the light source 11 change, the three-dimensional point image intensity distribution via the optical system 12 changes. Therefore, as shown in Figure 9, the focal plane F in the optical system in each state can change in the optical axis direction (Z direction) depending on the positions of the multiple point detectors 152 with respect to the light source 11. In the example in Figure 9, the three-dimensional point image intensity distribution via the optical system 12 is different for the detector 15B in the first state A11, which is located at the first position G11, and for the detector 15B in the second state A12, which is located at the second position G12. The first focal plane Fa, which is the focal plane in the first state A11, and the second focal plane Fb, which is the focal plane in the second state A12, may be in different positions in the optical axis direction (Z direction), but they are at least substantially the same.

[0082] With the microscope 1B and image processing device 100B described above, the detector 15B is equipped with multiple point detectors 152, allowing for the simultaneous acquisition of numerous images. Therefore, the sample structure can be estimated based on more image information obtained from the multiple point detectors 152. As a result, objects present on the first estimated sample surface Q1 can be estimated more easily, and an estimated image Ie with improved Z resolution and sectioning ability can be obtained. Furthermore, because the detector 15B is equipped with multiple point detectors 152, light intensity can be detected at multiple positions simultaneously, enabling faster processing.

[0083] [Third Embodiment] A third embodiment will now be described. Figure 10 is a schematic diagram showing an example of the configuration of a microscope according to the third embodiment. In this embodiment, components similar to those in the above-described embodiments are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 10, the microscope 1C comprises a microscope body 10C and an image processing device 100C. The microscope body 10C comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15C. The optical system 12 of the microscope body 10C irradiates the sample 8 with illumination light L1 from the light source 11 and guides the detection light L2 from the sample 8 to the detector 15C.

[0084] (Detector configuration) Figure 11 shows the configuration of the detector according to the third embodiment. The detector 15C detects the light of the image 15m formed on the light-receiving surface of the detector 15C. As shown in Figure 11, in this embodiment, the detector 15C is a so-called two-dimensional detector (2D detector), and is configured by arranging a plurality of detection units 153 in two dimensions. That is, the plurality of detection units 153 are arranged in multiples in a first direction (left-right direction in Figure 11) along a plane perpendicular to the optical axis, and in a second direction (up-down direction in Figure 11) along a plane perpendicular to the optical axis and intersecting the first direction. Here, each detection unit 153 detects light, and its light-receiving surface is a pixel that is sufficiently smaller than the fluorescence image 15m of the sample 8. The detection unit 153 includes a photoelectric conversion element (not shown) made of a semiconductor or the like. The light-receiving surface of the detector 15C receives the light of the image 15m. Each of the multiple detection units 153 that make up the detector 15C outputs a signal (electrical signal) corresponding to the amount of light received to the image processing device 100C.

[0085] Focusing on two of the multiple detection units 153, the three-dimensional point image intensity distribution of the optical system in the first state A21, which includes the light source 11, the optical system 12, and the detection unit 153 located at the first position G21, is different from the three-dimensional point image intensity distribution of the optical system in the second state A22, which includes the light source 11, the optical system 12, and the detection unit 153 located at the second position G22.

[0086] (3D point image intensity distribution of the optical system) Figure 12 is a schematic diagram showing the relationship between the focal plane, sample plane, first plane, second plane, and estimated sample plane according to the third embodiment. When the positions of the multiple detection units 153 of the detector 15C differ in a plane orthogonal to the optical axis direction, the relative position with respect to the light source 11 changes. When the relative position of the detector 15C detection unit 153 with respect to the light source 11 changes, the three-dimensional point image intensity distribution (3D-PSF) via the optical system 12 differs, as shown in Figure 2. In other words, as shown in Figure 12, the three-dimensional point image intensity distribution of the optical system in the first state A21, which includes the light source 11, the optical system 12, and the detection unit 153 located at the first position G21, is different from the three-dimensional point image intensity distribution of the optical system in the second state A22, which includes the light source 11, the optical system 12, and the detection unit 153 located at the second position G22.

[0087] (Configuration of the image processing device) The image processing device 100C generates multiple images of the sample 8 based on signals from multiple detection units 153 of the detector 15C, and processes these images. The image processing device 100C includes a signal receiving unit 101, a generation unit 102, and an estimation unit 103, and each function is applied in this embodiment as described in the first embodiment.

[0088] (Regarding the focal plane, the definition of "approximate agreement" with the focal plane) The focal plane F in the optical system in each state is a plane defined by the three-dimensional point image intensity distribution of the optical system in each state. In this embodiment as well, the description in the first embodiment applies. Therefore, all focal planes substantially coincide.

[0089] With the microscope 1C and image processing device 100C described above, the detector 15C is equipped with multiple detection units 153 arranged in two dimensions, allowing for simultaneous detection of light intensity at multiple positions, thus enabling faster processing. Furthermore, compared to the detectors 15A and 15B in the first and second embodiments, the multiple detection units 153 arranged in two dimensions allow for obtaining more image information in the same scan time, resulting in a more accurate estimated image Ie.

[0090] [Fourth Embodiment] A fourth embodiment will now be described. Figure 13 is a schematic diagram showing an example of the configuration of a microscope according to the fourth embodiment. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 13, the microscope 1D comprises a microscope body 10D and an image processing device 100D. The microscope body 10D comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15D.

[0091] The detector 15D detects the detection light L2 from the sample 8. In this embodiment, the detector 15D is equipped with a point detector 154. The point detector 154 is equipped with a photoelectric conversion element (not shown) made of a semiconductor or the like. Each photoelectric conversion element (not shown) of the point detector 154 outputs a signal (electrical signal) corresponding to the amount of light received to the image processing device 100D. In this embodiment, the point detector 154 is fixedly installed.

[0092] In this embodiment, an optical element 17D is provided in the optical system 12. The optical element 17D switches the optical system 12 between a first state A31 and a second state A32. The optical element 17D is provided so as to be able to move forward and backward relative to the optical path of the detection light L2. The optical element 17D is configured to be switchable between a first state A31, in which the optical element 17D is located outside the optical path of the detection light L2, and a second state A32, in which the optical element 17D enters the optical path of the detection light L2. In this embodiment, a glass parallel plate 171 is provided as the optical element 17D. The glass parallel plate 171 is arranged, for example, between the optical path separation element 125 and the focusing lens 126. In the second state A32, in which the glass parallel plate 171 enters the optical path of the detection light L2, the detection light L2 shifts in a direction intersecting the optical axis direction (Z direction).

[0093] Figure 14 shows the general shape of the XZ cross-section of the 3D point image intensity distribution in the first state and the general shape of the XZ cross-section of the 3D point image intensity distribution in the second state according to the fourth embodiment. In the first state A31 and the second state A32, the detection light L2 shifts in a plane perpendicular to the optical axis direction (Z direction), which changes the 3D point image intensity distribution (3D-PSF) via the optical system 12. As shown in Figure 14, the 3D point image intensity distribution via the optical system 12 differs between the first state A31, where the glass parallel plate 171 is located outside the optical path of the detection light L2, and the second state A32, where the glass parallel plate 171 is located on the optical path of the detection light L2. The 3D point image intensity distribution in the second state A32 is tilted relative to the 3D point image intensity distribution in the first state A31.

[0094] The image processing device 100D generates an image of the sample 8 based on the signal from the point detector 154 of the detector 15D and processes the image. The generation unit 102 generates an image of the focal plane F of the optical system based on the signal from the point detector 154. The generation unit 102 generates an image G of the sample 8 based on the amount of light detected by the point detector 154 in both the first state A31, where the optical element 17D is located outside the optical path of the detected light L2, and the second state A32, where the optical element 17D has entered the optical path of the detected light L2. In the generation unit 102, based on the amount of light detected by the detector 15D in the first state A31, the generation unit 102 generates a first image Ga of the surface of the sample 8 (first sample surface Sa) at a position corresponding to the first focal plane Fa in the Z direction. In the generation unit 102, based on the amount of light detected by the detector 15D in the second state A32, a second image Gb of the surface of the sample 8 (second sample surface Sb) at a position corresponding to the second focal plane Fb in the Z direction is generated.

[0095] Figure 15 shows the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane for the optical system in the first state and the optical system in the second state of the fourth embodiment, respectively. As shown in Figure 15, the first relative relationship between the two-dimensional point image intensity distribution H31a of the first plane P1 and the two-dimensional point image intensity distribution H32a of the second plane P2 in the first three-dimensional point image intensity distribution based on the optical system of the first state A31 is different from the second relative relationship between the two-dimensional point image intensity distribution H31b of the first plane P1 and the two-dimensional point image intensity distribution H32b of the second plane P2 in the second three-dimensional point image intensity distribution based on the optical system of the second state A32, which is different from the first state A31.

[0096] (Estimation based on the relative position of the 2D point image intensity distribution) The estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative relationship between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. In this embodiment, for example, the estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative positions of the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. The centroid position vector of the two-dimensional point image intensity distribution H1a of the first surface P1 of the optical system in the first state A31 is r f Let r be 1, and the centroid position vector of the 2D point image intensity distribution H2a of the second surface P2 of the optical system in the first state A31 be r df Let r be 1. Furthermore, the relative position vector r of the 2D point image intensity distribution H2a of the 2D point image intensity distribution H2a of the 2D point image intensity distribution H1a of the 1st surface P1 of the optical system in the 1st state A31 is given by r rel Calculate 1 using the following formula (9).

[0097] r rel 1=r df 1-r f 1...(9)

[0098] The centroid position vector of the 2D point image intensity distribution H1b of the first surface P1 of the optical system in the second state A32 is r f Let 2 be the centroid position vector of the 2D point image intensity distribution H2b of the second surface P2 of the optical system in the second state A32 be r dfLet's assume it's 2. Furthermore, the relative position vector of the two-dimensional point image intensity distribution H2b of the second surface P2 of the optical system in the second state A32 with respect to the two-dimensional point image intensity distribution H1b of the first surface P1 of the optical system in the second state A32 is calculated using the following equation (10).

[0099] r rel 2=r df 2-r f 2...(10)

[0100] Here, the relative position vector r in the first state A31 is rel 1 and the relative position vector r in the second state A32 rel The difference from 2 needs to be somewhat large compared to the full width at half maximum of the two-dimensional point image intensity distribution H1a of the first plane P1 in the first state A31 and the two-dimensional point image intensity distribution H1b of the first plane P1 in the second state A32. Therefore, it is preferable that the smaller of the two-dimensional point image intensity distribution H1a of the first plane P1 in the first state A31 and the two-dimensional point image intensity distribution H1b of the first plane P1 in the second state A32 is taken as σ, and that the following equation (11) is satisfied.

[0101]

number

[0102] With the microscope 1D and image processing device 100D described above, the optical system 12 is switched between a first state A31 and a second state A32 by the optical element 17D, thereby estimating the structure of each of the multiple surfaces of the sample 8, including the first estimated sample surface Q1. The estimation method is the same as the method described above in (Estimation of the 3D distribution of objects). As a result, objects present on the first estimated sample surface Q1 can be estimated with greater accuracy, and an estimated image with improved Z resolution and sectioning ability is obtained.

[0103] In the fourth embodiment described above, a point detector 154 is used as the detector 15D, but instead of the point detector 154, a plurality of point detectors 152 as shown in the second embodiment, or a two-dimensional detector equipped with a plurality of detection units 153 arranged in two dimensions as shown in the third embodiment may be used. In the fourth embodiment described above, a glass parallel plate 171 is provided as the optical element 17D, but a phase plate may be used instead.

[0104] [Fifth Embodiment] A fifth embodiment will now be described. Figure 16 is a schematic diagram showing an example of the configuration of a microscope according to the fifth embodiment. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 16, the microscope 1E comprises a microscope body 10E and an image processing device 100E. The microscope body 10E comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15D.

[0105] In this embodiment, an optical element 17E is provided in the optical system 12. The optical element 17E switches the optical system 12 between a first state A41 and a second state A42. The optical element 17E is provided so as to be able to move forward and backward relative to the optical path of the illumination light L1. The optical element 17E is configured to be switchable between a first state A41, in which the optical element 17E is located outside the optical path of the illumination light L1, and a second state A42, in which the optical element 17E enters the optical path of the illumination light L1. In this embodiment, a glass parallel plate 172 is provided as the optical element 17E. The glass parallel plate 172 is arranged between the collimator lens 120 and the optical path separation element 125. In the second state A42, in which the glass parallel plate 172 enters the optical path of the illumination light L1, the illumination light L1 shifts in a direction intersecting the optical axis direction (Z direction).

[0106] In the first state A41 and the second state A42, the illumination light L1 is shifted in a direction that intersects the optical axis direction (Z direction), resulting in different three-dimensional point image intensity distributions (3D-PSF) through the optical system 12. In this embodiment as well, as shown in Figure 14, the three-dimensional point image intensity distribution in the second state A42 is tilted compared to the three-dimensional point image intensity distribution in the first state A41.

[0107] In the image processing device 100E, the generation unit 102 generates a first image Ga of the surface of the sample 8 (first sample surface Sa) at a position corresponding to the first focal plane Fa in the Z direction, based on the amount of light detected by the detector 15D in the first state A41. In the second state A42, the generation unit 102 generates a second image Gb of the surface of the sample 8 (second sample surface Sb) at a position corresponding to the second focal plane Fb in the Z direction, based on the amount of light detected by the detector 15D.

[0108] Figure 17 shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system in the first state and the optical system in the second state of the fifth embodiment.

[0109] The estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 in the same manner as in the fourth embodiment described above, based on the relative relationship between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. In this embodiment, for example, the estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative positions of the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2.

[0110] With the microscope 1E and image processing device 100E described above, the optical element 17E switches the optical system 12 between a first state A41 and a second state A42, thereby estimating the structure of each of the multiple surfaces of the sample 8, including the first estimated sample surface Q1. As a result, objects present on the first estimated sample surface Q1 can be estimated with greater accuracy, and an estimated image with improved Z resolution and sectioning ability is obtained.

[0111] In the fifth embodiment described above, a point detector 154 is used as the detector 15D, but instead of the point detector 154, a plurality of point detectors 152 as shown in the second embodiment, or a two-dimensional detector equipped with a plurality of detection units 153 arranged in two dimensions as shown in the third embodiment may be used. In the fifth embodiment described above, a glass parallel plate 172 is provided as the optical element 17E, but a phase plate may be used instead.

[0112] [Sixth Embodiment] A sixth embodiment will now be described. Figure 18 is a schematic diagram showing an example of the configuration of a microscope according to the sixth embodiment. In this embodiment, components similar to those in the embodiments described above are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 18, the microscope 1G comprises a microscope body 10G and an image processing device 100G. The microscope body 10G comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15D.

[0113] In this embodiment, an optical element 17G is provided in the optical system 12. The optical element 17G switches the optical system 12 between a first state A61 and a second state A62. The optical element 17G is provided so as to be able to move forward and backward relative to the optical path of the detection light L2. The optical element 17G is configured to be switchable between a first state A61, in which the optical element 17G is located outside the optical path of the detection light L2, and a second state A62, in which the optical element 17G enters the optical path of the detection light L2. In this embodiment, a cylindrical lens 174 is provided as the optical element 17G. The cylindrical lens 174 is arranged between the optical path separation element 125 and the focusing lens 126. In the second state A62, in which the cylindrical lens 174 enters the optical path of the detection light L2, the fluorescence image 15m formed on the light-receiving surface of the detector 15D is deformed.

[0114] In this embodiment, the cylindrical lens 174 includes a pair of first lenses 174a and second lenses 174b arranged along the optical axis of the optical path of the detection light L2. The first lens 174a and the second lens 174b have cylindrically bulging curved surfaces w1 and w2. The curved surface w1 of the first lens 174a is formed in a cylindrical shape with its axis in a first direction (the vertical direction of the paper in Figure 18) along a plane that intersects the optical axis. The curved surface w2 of the second lens 174b is formed in a cylindrical shape with its axis in a second direction (the direction perpendicular to the paper in Figure 18) along a plane that intersects the optical axis. As a result, the shift of the focal plane in the optical axis direction is suppressed, and the focal planes substantially coincide in the first and second states.

[0115] In the first state A61 and the second state A62, the three-dimensional point image intensity distribution (3D-PSF) via the optical system 12 changes by moving the cylindrical lens 174 in and out of the optical path of the detection light L2. The three-dimensional point image intensity distribution via the optical system 12 is different in the first state A61, where the cylindrical lens 174 is located outside the optical path of the detection light L2, and in the second state A62, where the cylindrical lens 174 is located on the optical path of the detection light L2.

[0116] The image processing device 100G generates an image of the sample 8 based on the signal from the point detector 154 of the detector 15D and processes the image. The generation unit 102 generates an image of the focal plane F of the optical system based on the signal from the point detector 154. The generation unit 102 generates an image G of the sample 8 based on the amount of light detected by the point detector 154 in both the first state A61 and the second state A62. In the first state A61, the generation unit 102 generates a first image Ga of the surface of the sample 8 (first sample surface Sa) at a position corresponding to the first focal plane Fa in the Z direction, based on the amount of light detected by the detector 15D. In the second state A62, the generation unit 102 generates a second image Gb of the surface of the sample 8 (second sample surface Sb) at a position corresponding to the second focal plane Fb in the Z direction, based on the amount of light detected by the detector 15D.

[0117] Figure 19 shows the two-dimensional point image intensity distribution on the first plane P1 and the two-dimensional point image intensity distribution on the second plane P2 in the optical system of the first state and the optical system of the sixth embodiment. As shown in Figure 19, the first relative relationship between the two-dimensional point image intensity distribution H61a of the first plane P1 and the two-dimensional point image intensity distribution H62a of the second plane P2 in the first three-dimensional point image intensity distribution based on the optical system 12 of the first state A61 is different from the second relative relationship between the two-dimensional point image intensity distribution H61b of the first plane P1 and the two-dimensional point image intensity distribution H62b of the second plane P2 in the second three-dimensional point image intensity distribution based on the optical system of the second state A62, which is different from the first state A61.

[0118] (Estimation based on the relative shape of the 2D point image intensity distribution) The estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 in the same manner as in the first embodiment, based on the relative relationship between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. In this embodiment, for example, the estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative shape of the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. The frequency distribution O1a is obtained by shifting the centroid position vector of the two-dimensional point image intensity distribution H1a on the first surface P1 of the optical system in the first state A61 so that it is at the origin, and then performing a two-dimensional Fourier transform. The frequency distribution O2a is obtained by shifting the centroid position vector of the two-dimensional point image intensity distribution H2a on the second surface P2 of the optical system in the first state A61 so that it is at the origin, and then performing a two-dimensional Fourier transform. The frequency distribution O1b is obtained by shifting the centroid position vector of the 2D point image intensity distribution H1b of the second surface P2 of the optical system in the second state A62 so that it is at the origin, and then performing a 2D Fourier transform. The frequency distribution O2b is obtained by shifting the centroid position vector of the 2D point image intensity distribution H2b of the second surface P2 of the optical system in the second state A62 so that it is at the origin, and then performing a 2D Fourier transform.

[0119] Based on the frequency distributions O1a, O2a, O1b, and O2b, the converted two-dimensional point image intensity distributions H3 and H4 are calculated using the following equations (12) and (13).

[0120]

number

[0121]

number

[0122] Here's FT -1 This represents the inverse Fourier transform. w is a decimal number to prevent division by zero, for example, 10 -5 Therefore, it is desirable that there be a sufficient difference in the relative shapes of the two-dimensional point image intensity distributions. When the correlation coefficient between H2b and H3 is ρ1, and the correlation coefficient between H2a and H4 is ρ2, it is desirable that ρ1 < 0.9 and ρ2 < 0.9 hold.

[0123] With the microscope 1G and image processing device 100G described above, the optical element 17G switches the optical system 12 between a first state A61 and a second state A62, thereby estimating the structure of each of the multiple surfaces of the sample 8, including the first estimated sample surface Q1. The estimation method is the same as the method described above in (Estimation of the 3D structure of an object). As a result, the object present on the first estimated sample surface Q1 can be estimated with greater accuracy, and an estimated image with improved Z resolution and sectioning ability is obtained.

[0124] In the sixth embodiment described above, a point detector 154 was used as the detector 15D. However, instead of the point detector 154, a plurality of point detectors 152 as shown in the second embodiment, or a two-dimensional detector equipped with a plurality of detection units 153 arranged in two dimensions as shown in the third embodiment, may be used.

[0125] [Seventh Embodiment] A seventh embodiment will now be described. Figure 20 is a schematic diagram showing an example of the configuration of a microscope according to the seventh embodiment. In this embodiment, components similar to those in the above-described embodiments may be given the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 20, the microscope 1H comprises a microscope body 10H and an image processing device 100H. The microscope body 10H comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15D.

[0126] In this embodiment, an optical element 17H is provided in the optical system 12. The optical element 17H switches the optical system 12 between a first state A71 and a second state A72. In this embodiment, pinhole members 175A and 175B are provided as optical elements 17H. The pinholes formed in pinhole member 175A and pinhole member 175B have different sizes (inner diameters). For example, the size of the pinhole in one pinhole member 175A is smaller than the size of the pinhole in the other pinhole member 175B. These pinhole members 175A and 175B are provided between the condensing lens 126 and the detector 15D so as to be able to move alternately forward and backward with respect to the optical path of the detected light L2.

[0127] The optical element 17H is configured to be switchable between a first state A71 in which the pinhole member 175A is located on the optical path of the detection light L2, and a second state A72 in which the pinhole member 175B is located on the optical path of the detection light L2. The amount of detection light L2 passing through the pinhole is different in the first state A71 and the second state A72.

[0128] The three-dimensional point image intensity distribution via the optical system 12 differs between the first state A71, where the pinhole member 175A is located on the optical path of the detection light L2, and the second state A72, where the pinhole member 175B is located on the optical path of the detection light L2.

[0129] In the image processing device 100H, the generation unit 102 generates a first image Ga of the surface of the sample 8 (first sample surface Sa) at a position corresponding to the first focal plane Fa in the Z direction, based on the amount of light detected by the detector 15D in the first state A71. In the second state A72, the generation unit 102 generates a second image Gb of the surface of the sample 8 (second sample surface Sb) at a position corresponding to the second focal plane Fb in the Z direction, based on the amount of light detected by the detector 15D.

[0130] Figure 21 shows examples of the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane in the optical system of the first state and the optical system of the seventh embodiment.

[0131] The estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative relationship between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. In this embodiment, for example, the estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative intensity of the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2, in the same manner as in the first embodiment.

[0132] With the microscope 1H and image processing device 100H described above, the optical element 17H switches the optical system 12 between a first state A71 and a second state A72, thereby estimating the structure of each of the multiple surfaces of the sample 8, including the first estimated sample surface Q1. As a result, objects present on the first estimated sample surface Q1 can be estimated with greater accuracy, and an estimated image with improved Z resolution and sectioning ability is obtained.

[0133] In the seventh embodiment described above, a point detector 154 was used as the detector 15D. However, instead of the point detector 154, a plurality of point detectors 152 as shown in the second embodiment, or a two-dimensional detector equipped with a plurality of detection units 153 arranged in two dimensions as shown in the third embodiment, may be used.

[0134] [Eighth Embodiment] An eighth embodiment will now be described. Figure 22 is a schematic diagram showing an example of the configuration of a microscope according to the eighth embodiment. In this embodiment, components similar to those in the above-described embodiments are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 22, the microscope 1I comprises a microscope body 10I and an image processing device 100I. The microscope body 10I comprises a light source 11, an optical system 12, a scanning control unit 13, and a detector 15D.

[0135] In this embodiment, an optical element 17I is provided in the optical system 12. The optical element 17I switches the optical system 12 between a first state A81 and a second state A82. In this embodiment, pinhole members 176A and 176B are provided as optical elements 17I. The pinhole members 176A and 176B are provided between the condensing lens 126 and the detector 15D so as to be able to move alternately forward and backward with respect to the optical path of the detected light L2. The pinholes formed by the pinhole member 176A and the pinhole member 176B have different shapes. For example, the shape of the pinhole of one pinhole member 176A is, for example, a circle formed in the center of the optical path of the detected light L2. The shape of the pinhole of the other pinhole member 176B is, for example, an annular shape (a so-called ring pinhole) formed radially outward from the center of the optical path of the detected light L2. Furthermore, if the pinhole shapes formed in pinhole member 176A and pinhole member 176B are different, the shape of each pinhole may be changed to another shape as appropriate.

[0136] The optical element 17I is configured to be switchable between a first state A81 in which the pinhole member 176A is located on the optical path of the detection light L2, and a second state A82 in which the pinhole member 176B is located on the optical path of the detection light L2. In the first state A81 and the second state A82, the amount of detection light L2 passing through the pinhole is different, and the three-dimensional point image intensity distribution (3D-PSF) via the optical system 12 is different. The three-dimensional point image intensity distribution of the first state A81 and the three-dimensional point image intensity distribution of the second state A82 have different profiles in the Z direction.

[0137] In the image processing device 100I, the optical system 12 is switched between a first state A81 and a second state A82 by the optical element 17I, thereby estimating the structure of each of the multiple surfaces of the sample 8, including the first estimated sample surface Q1. As a result, objects present on the first estimated sample surface Q1 can be estimated with high accuracy, and an estimated image with improved Z resolution and sectioning ability is obtained.

[0138] [Ninth Embodiment] A ninth embodiment will now be described. Figure 23 is a schematic diagram showing an example of the configuration of a microscope according to the ninth embodiment. In this embodiment, components similar to those in the above-described embodiments are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 23, the microscope 1J comprises a microscope body 10J and an image processing device 100J. In this embodiment, the microscope body 10J comprises a light source 21, an illumination optical system 22 that irradiates the sample 8 with illumination light L11 from the light source 21, and a detector 25J.

[0139] (Microscope body configuration) The light source 21 emits illumination light L11, such as a laser. The light source 21 may be a monochromatic (single wavelength) light source or a multicolor (multiple wavelength) light source. The light source 21 may be either a laser that emits continuous oscillation light or a laser that emits pulsed light. Furthermore, the light source 21 does not have to be a laser; it may be an LED or a lamp. When a fluorescent substance is used as the sample 8, it is preferable to select a wavelength for the light source 21 that excites the fluorescent substance contained in the sample 8. When a fluorescent substance is used as the sample 8, it is also possible to select a wavelength for the light source 21 that multiphoton-excites the fluorescent substance contained in the sample 8. The light source 21 may be provided in a replaceable (attachable, detachable) manner on the microscope body 10J. The light source 21 may also be attached externally to the microscope body 10J when observing with the microscope body 10J. In this case, for example, illumination light L11 may be incident into the microscope body 10J from the light source 21 outside the microscope body 10J via an existing optical component such as an optical fiber.

[0140] The optical system 22 irradiates the sample 8 with illumination light L11 from the light source 21 and guides the detection light from the sample 8 to the detector 25J. In this embodiment, the optical system 22 includes a collimator lens 220, an objective lens 221, a lens 224, an optical path separation element 225, an imaging lens 223, etc. The collimator lens 220 converts the illumination light L11 emitted from the light source 21, such as a laser, into substantially parallel light. The lens 224 focuses the substantially parallel light and guides it to the optical path separation element 225. The optical path separation element 225 consists of a dichroic mirror or the like. The optical path separation element 225 guides the illumination light L11 that has passed through the lens 224 to the objective lens 221.

[0141] The objective lens 221 irradiates the incident illumination light L11 onto the sample 8 held on the stage 2 in a single beam. The light (detection light) L22 emitted from the sample 8 due to the illumination light L11 is incident on the objective lens 221. The detection light L22 passes through the objective lens 221, the optical path separation element 225, and the imaging lens 223 before being incident on the detector 25J.

[0142] The detector 25J is positioned conjugate to the illumination region of the illumination light L11 on the sample 8 via a detection optical system. An image 25m of the fluorescence of the sample 8 excited by the illumination light L11 is formed on the light-receiving surface of the detector 25J. The detector 25J detects the light of the image 25m formed on the light-receiving surface of the detector 25J. In this embodiment, the detector 25J is configured by arranging a plurality of detection units 251 in two dimensions. Each detection unit 251 includes a photoelectric conversion element (not shown) made of a semiconductor or the like. The light-receiving surface of the detector 25J receives the light of the image 25m. Each of the plurality of detection units 251 constituting the detector 25J outputs a signal (electrical signal) corresponding to the amount of light received to the image processing device 100J.

[0143] In this embodiment, an optical element 27J is provided in the optical system 22. The optical element 27J switches the optical system 22 between a first state A91 and a second state A92. In this embodiment, an aperture 271 is provided as the optical element 27J. The aperture 271 is located between the optical path separation element 225 and the imaging lens 223, near the conjugate position of the pupil of the objective lens 221. A relay optical system may also be provided and positioned at the pupil conjugate position. The aperture 271 is configured to be switchable between the first state A91 and the second state A92 by changing the aperture diameter. In the first state A91, the aperture diameter of the aperture 271 is made larger than that of the second state A92.

[0144] If the aperture diameter of the diaphragm 271 is different between the first state A91 and the second state A92, the three-dimensional point image intensity distribution (3D-PSF) via the optical system 22 will be different. The three-dimensional point image intensity distribution via the optical system 22 is different between the first state A91 and the second state A92.

[0145] The image processing device 100J generates an image of the sample 8 based on signals from each of the multiple detection units 251 and processes the image. The image processing device 100J consists of a processing unit such as a personal computer. The image processing device 100J is equipped with hardware such as a CPU and memory. The image processing device 100J functionally has the configuration shown below, through the cooperation of the CPU, memory, etc., and the image processing program stored in the memory and storage device to execute predetermined processing.

[0146] (Configuration of the image processing device) The image processing device 100J includes a signal receiving unit 101, a generation unit 102, and an estimation unit 103, and the functions of each are as described in the first embodiment and are also applied in this embodiment.

[0147] (Regarding the relative relationship of 2D point image intensity distributions across multiple surfaces) Figure 24 shows examples of the two-dimensional point image intensity distribution on the first plane P1 and the two-dimensional point image intensity distribution on the second plane P2 in the first state optical system and the second state optical system of the ninth embodiment. In this embodiment, the relative relationship (position, shape, intensity) of the two-dimensional point image intensity distribution on the first plane P1 and the two-dimensional point image intensity distribution on the second plane P2 differs between the first state A91 and the second state A92. In other words, as shown in Figure 24, the first relative relationship between the two-dimensional point image intensity distribution H91a of the first plane P1 and the two-dimensional point image intensity distribution H92a of the second plane P2 in the first three-dimensional point image intensity distribution based on the optical system 22 in the first state A91 is different from the second relative relationship between the two-dimensional point image intensity distribution H91b of the first plane P1 and the two-dimensional point image intensity distribution of the second plane P2 in the second three-dimensional point image intensity distribution based on the optical system 22 in the second state A92, where the opening of the aperture 271 is different from that of the first state A91.

[0148] With the microscope 1J and image processing device 100J described above, even with the microscope 1J equipped with a wide-field optical system 22, an estimated image with improved Z resolution and sectioning ability can be obtained by varying the opening of the aperture 271.

[0149] [Tenth Embodiment] A tenth embodiment will now be described. Figure 25 is a schematic diagram showing an example of the configuration of a microscope according to the tenth embodiment. In this embodiment, components similar to those in the above-described embodiments are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 25, the microscope 1K comprises a microscope body 10K and an image processing device 100K. In this embodiment, the microscope body 10K comprises a light source 21, an illumination optical system 22 that irradiates the sample 8 with illumination light L11 from the light source 21, and a detector 25J.

[0150] In this embodiment, an optical element 27K is provided in the optical system 22. The optical element 27K switches the optical system 22 between a first state A93 and a second state A94. In this embodiment, an aperture 272 is provided as the optical element 27K. The aperture 272 is located between the optical path separation element 225 and the imaging lens 223, and is positioned near the conjugate position of the pupil of the objective lens 221. A relay optical system may also be provided and positioned at the pupil conjugate position. The aperture 272 is provided so as to be movable in a direction perpendicular to the optical axis direction (Z direction) of the detected light L22 while keeping the aperture diameter constant. The aperture 272 switches between the first state A93 and the second state A94 by changing its position in a direction perpendicular to the optical axis direction. For example, in the first state A93, the aperture 272 is positioned at the center of the optical axis, and in the second state A94, the aperture 272 is shifted from the center of the optical axis in a direction perpendicular to the optical axis direction (Z direction).

[0151] The position of the aperture 272 differs between the first state A93 and the second state A94, resulting in different three-dimensional point image intensity distributions (3D-PSF) via the optical system 22. In the first state A93, where the aperture 272 is located at the optical axis center, and in the second state A94, where the aperture 272 is offset from the optical axis center of the detected light L22, the three-dimensional point image intensity distribution via the optical system 22 differs. Each detection unit 251 detects the amount of light in the image 25m at multiple positions in a plane orthogonal to the optical axis direction. The three-dimensional point image intensity distribution in the second state A94 is tilted relative to the three-dimensional point image intensity distribution in the first state A93.

[0152] In the image processing device 100K, the generation unit 102 generates a first image Ga of the surface of the sample 8 at a position corresponding to the first focal plane Fa in the Z direction, based on the amount of light detected by the detector 25J in the first state A93. In the second state A94, the generation unit 102 generates a second image Gb of the surface of the sample 8 at a position corresponding to the second focal plane Fb in the Z direction, based on the amount of light detected by the detector 25J.

[0153] Figure 26 shows the two-dimensional point image intensity distribution on the first plane and the two-dimensional point image intensity distribution on the second plane for the optical system in the first state and the optical system in the second state of the 10th embodiment.

[0154] The estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 in the same manner as in the ninth embodiment described above, based on the relative relationship between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2. In this embodiment, for example, the estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative positions of the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2.

[0155] With the microscope 1K and image processing device 100K described above, even with the microscope 1K equipped with a wide-field optical system 22, by changing the position of the aperture 272 relative to the optical axis, an estimated image with improved Z resolution and sectioning ability can be obtained.

[0156] In the above-described tenth embodiment, the optical element 27K is provided with an aperture 272, but instead, a prism, phase plate, and cylindrical lens may be moved relative to the optical axis of the detection light L22 or moved forward and backward relative to the optical path of the detection light L22.

[0157] [Embodiment No. 11] An eleventh embodiment will now be described. Figure 27 is a schematic diagram showing an example of the configuration of a microscope according to the eleventh embodiment. In this embodiment, components similar to those in the above-described embodiments may be denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 27, the microscope 1L comprises a microscope body 10L and an image processing device 100L. In this embodiment, the microscope body 10L is a holographic microscope and comprises a light source 31, an optical system 32, and a detector 35L.

[0158] The light source 31 emits illumination light L31, such as a laser. The optical system 32 irradiates the sample 8 with the illumination light L31 from the light source 31 and guides the transmitted light L33 from the sample 8 to the detector 35L. In this embodiment, the optical system 32 includes a beam splitter 321, a lens 322, a deflection unit 323, a half mirror 324, a phase shift mirror 325, a beam combiner 327, a lens 328, and the like.

[0159] In the optical system 32, a portion of the illumination light L31 from the light source 31 is irradiated onto the sample 8 via the beam splitter 321. The light that passes through the sample 8 (transmitted light) L33 is incident on the beam combiner 327 via the deflection unit 323. The remaining illumination light L31 passes through the beam splitter 321 and, as reference light L34, is incident on the beam combiner 327 via the half mirror 324 and the phase shift mirror 325. In the beam combiner 327, interference occurs between the transmitted light L33 and the reference light L34, and this interference light is incident on the detector 35L as detection light L32.

[0160] An interference fringe pattern 35m is formed on the light-receiving surface of the detector 35L by the detected light L32. The detector 35L detects the light of the interference fringe pattern 35m formed on the light-receiving surface of the detector 35L. In this embodiment, the detector 35L is configured by arranging a plurality of detection units 351 in two dimensions. Each detection unit 351 includes a photoelectric conversion element (not shown) made of a semiconductor or the like. The light-receiving surface of the detector 35L receives the light of the interference fringe pattern 35m. Each of the plurality of detection units 351 constituting the detector 35L outputs a signal (electrical signal) corresponding to the intensity of the received light to the image processing device 100L. By shifting the phase-shift mirror in the direction of the optical axis and acquiring multiple images, the complex amplitude distribution of the transmitted light L33 can be reconstructed by analyzing them using the phase-shift method or the like. Since this distribution reflects the information of the complex refractive index of the sample 8, it can also be called an image of the sample 8.

[0161] In this embodiment, an optical element 37L is provided in the optical system 32. The optical element 37L switches the optical system 32 between a first state A101 and a second state A102. In this embodiment, a phase plate 371 is provided as the optical element 37L. The phase plate 371 is provided so as to be able to move forward and backward with respect to the optical path of the detection light L32. The optical element 37L is configured to be switchable between a first state A101, in which the optical element 37L is located outside the optical path of the detection light L32, and a second state A102, in which the optical element 37L enters the optical path of the detection light L32. In the second state A102, in which the phase plate 371 enters the optical path of the detection light L32, the phase of the detection light L32 changes. As the phase plate 371, for example, one that generates a helical three-dimensional point image amplitude distribution with respect to the detection light L32 can be used.

[0162] The point image amplitude distribution based on the optical system 32 differs between the first state A101, where the phase plate 371 is located outside the optical path of the detection light L32, and the second state A102, where the phase plate 371 is located on the optical path of the detection light L32.

[0163] The image processing device 100L generates an image of the sample 8 based on signals from each of the multiple detection units 351 and processes the image. The image processing device 100L consists of a processing unit such as a personal computer. The image processing device 100L includes hardware such as a CPU and memory. The image processing device 100L functionally has the configuration shown below, through the cooperation of the CPU, memory, etc., and the image processing program stored in the memory and storage device to execute predetermined processing.

[0164] (Configuration of the image processing device) The image processing device 100L functionally includes a signal receiving unit 101, a generation unit 102, and an estimation unit 103. The signal receiving unit 101 receives signals corresponding to the light intensity of the interference fringe pattern 35m output from a plurality of detection units 351. The generation unit 102 generates an image of the focal plane F in the optical system 32 based on the signals from the plurality of detection units 351.

[0165] The generation unit 102 generates multiple images G of the sample 8 based on the light intensity detected by each detection unit 351 in both the first state A101 and the second state A102. In the first state A101, the generation unit 102 generates a first image Ga of the sample 8 at a position corresponding to the focal plane F in the Z direction based on the light intensity detected by the detection unit 351. In the second state A102, the generation unit 102 generates a second image Gb of the sample 8 at a position corresponding to the second focal plane Fb in the Z direction based on the light intensity detected by the detection unit 351.

[0166] Figure 28 shows the two-dimensional point image amplitude distribution on the first plane P1 and the two-dimensional point image amplitude distribution on the second plane P2 for the optical system in the first state and the optical system in the second state of the 11th embodiment. The estimation unit 103 calculates or pre-stores the two-dimensional point image amplitude distributions of the first plane P1 and the second plane P2 for the three-dimensional point image amplitude distribution of the optical system in each state as two-dimensional images. As shown in Figure 28, in the three-dimensional point image amplitude distribution h′(x,y,z) of the optical system in each state described above, the point image amplitude distribution h′(x,y,z1) when the z coordinate is the z coordinate of the first plane P1 (z=z1) is defined as the two-dimensional point image amplitude distribution H101 on the first plane P1. Also, the point image amplitude distribution h′(x,y,z2) when the z coordinate is the z coordinate of the second plane P2 (z=z2) is defined as the two-dimensional point image amplitude distribution image H102 on the second plane P2.

[0167] The estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative relationship between the two-dimensional point image amplitude distribution on the first surface P1 and the two-dimensional point image amplitude distribution on the second surface P2. In this embodiment, for example, the estimation unit 103 estimates the structure of the sample 8 on the first estimated sample surface Q1 and the second estimated sample surface Q2 based on the relative positions of the two-dimensional point image amplitude distribution on the first surface P1 and the two-dimensional point image amplitude distribution on the second surface P2.

[0168] With the microscope 1L and image processing device 100L described above, even with the holographic microscope 1L, an estimated image with improved Z resolution and sectioning ability can be obtained by using the phase plate 371.

[0169] [Twelfth Embodiment] A twelfth embodiment will now be described. Figure 29 is a schematic diagram showing an example of the configuration of a scanning microscope according to the twelfth embodiment. In this embodiment, components similar to those in the above-described embodiments are denoted by the same reference numerals, and their descriptions may be omitted or simplified. As shown in Figure 29, the microscope 1M comprises a microscope body 10M and an image processing device 100M. The microscope body 10M comprises a light source 11, an optical system 12, a scanning control unit 13, a stage movement mechanism 18, and a detector 15C. The stage movement mechanism 18 raises and lowers the stage 2 that holds the sample 8. The stage movement mechanism 18 moves the stage 2 that holds the sample 8 in the optical axis direction (Z direction). At this time, the position of the focal plane F of the optical system 12 relative to the sample 8 changes as the sample 8 is moved in the optical axis direction by the stage movement mechanism 18, but the position of the focal plane F in the optical system 12 itself does not change.

[0170] Figure 30 shows examples of multiple images generated by the generation unit of the 12th embodiment. In the microscope 1M of this embodiment, the stage movement mechanism 18 aligns the focal plane of the optical system 12 with multiple sample surfaces that are at different positions in the optical axis direction, and the detector 15C detects the light of the image 15m at each position. In the microscope 1M, the first image Ga is acquired when the first focal plane Fa in the optical system of the first state A201 aligns with the first sample surface Sa (e.g., Z=0nm) of the sample 8. Also, in the microscope 1M, the second image Gb is acquired when the second focal plane Fb in the optical system of the second state A202 aligns with the second sample surface Sb (e.g., Z=0nm) of the sample 8. The positions of the first focal plane Fa and the second focal plane Fb in the optical axis direction (Z direction) are approximately coincident.

[0171] Furthermore, in microscope 1M, the stage movement mechanism 18 moves the sample 8 supported on stage 2 in the optical axis direction, and the third image Gc is acquired when the first focal plane Fa in the optical system 12 in the first state A201 coincides with the third sample plane Sc (e.g., Z=+400nm) of sample 8. Also in microscope 1M, the fourth image Gd is acquired when the second focal plane Fb in the optical system 12 in the second state A202 coincides with the fourth sample plane Sd (e.g., Z=+400nm) of sample 8. The third sample plane Sc and the fourth sample plane Sd are located in different positions in the optical axis direction (Z direction) from the first sample plane Sa and the second sample plane Sb, respectively. The third estimated sample plane is located in the vicinity of the third and fourth sample planes. When the first plane coincides with the first estimated sample plane, the plane that coincides with the third estimated sample plane is designated as the third plane.

[0172] The estimation unit 103 calculates or pre-stores the two-dimensional point image intensity distribution of the first plane P1, second plane P2, and third plane P3 (the second plane P2 is located between the first plane P1 and the third plane P3) in the three-dimensional point image intensity distribution via the optical system 12 in all states in which multiple images G are acquired. For example, by setting the first estimated sample plane to Z=0nm, the second estimated sample plane to Z=200nm, and the third estimated sample plane to Z=400nm, the structure of the sample 8 that does not match the focal plane (second estimated sample plane) can be reconstructed.

[0173] Figure 31 shows an example of an estimated image based on the structure estimated by the estimation unit of the 12th embodiment. The estimation unit 103 uses multiple images Ga, Gb, Gc, and Gd to estimate the structure of the sample 8 on the first estimated sample surface Q1, the second estimated sample surface Q2, and the third estimated sample surface Q3, based on the relative relationship between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the second surface P2, or the relative relationship between the two-dimensional point image intensity distribution on the first surface P1 and the two-dimensional point image intensity distribution on the third surface P3. In this case, it is desirable that there be a predetermined difference between the relative relationship (intensity, position, shape) between the two-dimensional point image intensity distribution of the first plane P1 and the two-dimensional point image intensity distribution of the second plane P2 in the optical system of the first state A201, and between the relative relationship (intensity, position, shape) between the two-dimensional point image intensity distribution of the first plane P1 and the two-dimensional point image intensity distribution of the second plane P2 in the optical system of the second state A202. Alternatively, it is desirable that there be a predetermined difference between the relative relationship (intensity, position, shape) between the two-dimensional point image intensity distribution of the first plane P1 and the two-dimensional point image intensity distribution of the third plane P3 in the optical system of the first state A201, and between the relative relationship (intensity, position, shape) between the two-dimensional point image intensity distribution of the first plane P1 and the two-dimensional point image intensity distribution of the third plane P3 in the optical system of the second state A202. Similar to the first embodiment, various known algorithms can be used for object estimation in the estimation unit 103. Here, for example, a gradient algorithm is used. The three-dimensional captured image Im, taken by the detection unit 153 located at the m-th position, is expressed as shown in equation (14) below.

[0174]

number

[0175] Here, s is a P-dimensional vector representing the three-dimensional distribution of fluorescent molecules, and A m This is an N×P matrix represented by the following equation (15) (N=Nx×Ny×Nz).

[0176]

number

[0177] Here, (x n ,y n ,z n ) is the coordinate of the illumination area 14 of the illumination light L1, and (x p ,y p ,z p ) represents the coordinates in the sample space. m This represents the three-dimensional point image intensity distribution of the optical system for the detection unit 153 at the m-th position. The estimation unit 103 estimates the distribution of fluorescent molecules in the sample 8 by minimizing the error function F(s) expressed by the following equation (16).

[0178]

number

[0179] Here, I mes This is the image (N-dimensional vector) actually acquired by detector 15A placed at the m-th position.

[0180] With the microscope 1M and image processing device 100M described above, by moving the sample 8 in the direction of the optical axis of the optical system 12 and aligning the focal plane with multiple sample surfaces S, the structure of the sample 8 on multiple estimated sample surfaces Q1 and Q3 can be estimated. Furthermore, the structure on the plane between the multiple estimated sample surfaces Q1 and Q3 can also be estimated. Therefore, the number of images taken during 3D imaging can be reduced, leading to a reduction in imaging time.

[0181] Although embodiments have been described above, the technical scope of the present invention is not limited to the embodiments described above. One or more of the requirements described above may be omitted. Furthermore, the requirements described above may be combined as appropriate. In addition, to the extent permitted by law, disclosures of all documents cited herein shall be incorporated as part of the description herein. [Explanation of Symbols]

[0182] 1A~1M...Microscope, 8...Sample, 12, 22, 32...Optical system, 15A~15D, 25J, 35L...Detector, 15m, 25m, 35...Image, 17D~17I, 27J, 27K, 37L...Optical element, 100A~100M...Image processing device, 102...Generation unit, 103...Estimation unit, 121... ...Objective lenses, 151, 152, 154...Point detectors, 153, 251, 351...Detection unit, 221...Objective lenses, 271, 272...Aperture, A1, A21, A31, A41, A61, A71, A81, A91, A93, A101, A201...First state, A2, A22, A32, A42, A62, A72, A82, A92, A94, A102, A202...Second state, F...Focal plane, Fa...First focal plane, Fb...Second focal plane, G...Image, G1, G11, G21, G201...First position, G2, G12, G22, G202...Second position, Ga, Gb, Gc, Gd...Image, Ie...Estimated image Image, L1, L11, L31... Illumination light, L2, L22, L32... Detection light, L33... Transmitted light, L34... Reference light, P1... First plane, P2... Second plane, Q1, Q2, Q3... Estimated sample plane, S... Sample plane, Sa... First sample plane, Sb... Second sample plane, Sc... Third sample plane, Sd... Fourth sample plane

Claims

1. An optical system that irradiates a sample with illumination light and guides the detection light from the sample to a detector, The system includes an image processing device that generates an image of the sample based on a signal from the detector and processes the image, The first three-dimensional point image intensity distribution based on the optical system in the first state differs from the second three-dimensional point image intensity distribution based on the optical system in a second state which is different from the first state. The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by arranging the point detector in a first position, and the second state is formed by arranging the point detector in a second position different from the first position. ii) When the detector is composed of a plurality of detection units or a plurality of point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting the optical element into the optical system, and the second state is formed by retracting the optical element from the optical system. iv) The first state is formed by inserting a first optical element into the optical system, and the second state is formed by inserting a second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. The aforementioned image processing device is A generation unit that generates a group of images based on the signal acquired without changing the relative positional relationship in the optical axis direction between the sample and the irradiation position of the illumination light, including a first image of a first focal plane based on the signal acquired through the optical system in the first state, and a second image of a second focal plane that substantially coincides with the first focal plane and the optical system in the optical axis direction, based on the signal acquired through the optical system in the second state, A microscope comprising: an estimation unit that estimates the structure of each of a plurality of surfaces in the sample, including a first estimated sample surface, along the optical axis direction, based on the aforementioned image group and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputs an estimated image based on the estimated structure; The estimation unit is a microscope that includes at least one of the following: i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, the structure is estimated, and an image obtained by integrating the estimated structure is output as the estimated image.

2. The first image is obtained by aligning the first focal plane with the first sample surface of the sample. The microscope according to claim 1, wherein the second image is obtained by aligning the second focal plane with the second sample surface of the sample.

3. The first relative relationship between the two-dimensional point image intensity distribution of the first plane and the two-dimensional point image intensity distribution of the second plane, which is different from the first plane in the direction of the optical axis, in the first three-dimensional point image intensity distribution is different from the second relative relationship between the two-dimensional point image intensity distribution of the first plane and the two-dimensional point image intensity distribution of the second plane in the second three-dimensional point image intensity distribution. The microscope according to claim 1 or claim 2, wherein the first surface is determined to substantially coincide with at least one of the first focal plane and the second focal plane.

4. The microscope according to claim 3, wherein the first estimated sample surface matches the first surface.

5. The microscope according to claim 3 or claim 4, wherein the estimated image has an image of a second estimated sample surface that matches the second surface.

6. The optical system has an objective lens, The microscope according to any one of claims 1 to 5, wherein the first focal plane and the second focal plane are the focal planes of the objective lens.

7. The first focal plane and the second focal plane are, In the optical axis direction, the plane where the maximum value of the three-dimensional point image intensity distribution of the optical system in the corresponding first and second states exists, In the optical axis direction, the plane where the maximum value of the integrated three-dimensional point image intensity distribution of the optical system in the corresponding first and second states exists, In the optical axis direction, when Gaussian fitting is performed to determine the function of the approximation curve of the three-dimensional point image intensity distribution of the optical system for each of the corresponding first and second states, the plane on which the center position of the Gaussian exists is, A microscope according to any one of claims 1 to 5, defined by any one of the following.

8. The microscope according to any one of claims 3 to 5, wherein the first relative relationship and the second relative relationship are one of the following: a relationship in intensity of the two-dimensional point image intensity distributions of the first plane and the second plane, a relationship in position of the two-dimensional point image intensity distributions of the first plane and the second plane, and a relationship in shape of the two-dimensional point image intensity distributions of the first plane and the second plane.

9. When the first state and the second state are formed by iii), The optical element is movable in relation to the illumination light or the detection light, The microscope according to any one of claims 1 to 8, wherein the entry of the optical element shifts the illumination light or the detection light in a direction intersecting the optical axis direction, thereby switching between the optical system in a first state and the optical system in a second state.

10. When the first state and the second state are formed by iv), The first optical element and the second optical element are pinholes through which the detected light passes, and the size or shape of the pinholes can be changed by switching between the first optical element and the second optical element. When the first state and the second state are formed by the method described in v), The optical element is a pinhole through which the detected light passes, and the size or shape of the pinhole can be changed. The microscope according to any one of claims 1 to 8, wherein the optical system in the first state and the optical system in the second state are switched by changing the size or shape of the pinhole.

11. When the first state and the second state are formed by the method described in v), The optical element is an aperture through which the illumination light or the detection light passes, and the aperture diameter or position of the aperture is changeable. The microscope according to any one of claims 1 to 8, wherein the optical system in the first state and the optical system in the second state are switched by changing the aperture diameter or position of the diaphragm.

12. The generation unit further generates a group of images including a third image and a fourth image based on the signal obtained by changing the relative positional relationship in the optical axis direction between the sample and the irradiation position of the illumination light, The third image is obtained by aligning the first focal plane of the optical system in the first state with the third sample surface of the sample. The fourth image is obtained by aligning the second focal plane of the optical system in the second state with the fourth sample surface of the sample. The microscope according to any one of claims 1 to 11, wherein the third sample surface and the fourth sample surface are different from the first sample surface and the second sample surface of the sample in the optical axis direction.

13. The microscope according to claim 12, wherein the estimated image includes a two-dimensional image corresponding to a plane existing along the optical axis direction from the first estimated sample plane to the third estimated sample plane which substantially coincides with the third sample plane.

14. The microscope according to claim 5, wherein the estimated image includes a two-dimensional image corresponding to at least the second estimated sample surface that matches the second surface, and a third estimated sample surface that substantially matches the third sample surface of the sample.

15. The microscope according to any one of claims 1 to 14, wherein the resolution of the estimated image in the optical axis direction is higher than the resolution of the first image and the second image in the optical axis direction.

16. An optical system that irradiates a sample with illumination light and guides the detection light from the sample to a detector, The system includes an image processing device that generates an image of the sample based on a signal from the detector and processes the image, The first three-dimensional point image amplitude distribution based on the optical system in the first state differs from the second three-dimensional point image amplitude distribution based on the optical system in a second state which is different from the first state. The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by arranging the point detector in a first position, and the second state is formed by arranging the point detector in a second position different from the first position. ii) When the detector is composed of a plurality of detection units or a plurality of point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting the optical element into the optical system, and the second state is formed by retracting the optical element from the optical system. iv) The first state is formed by inserting a first optical element into the optical system, and the second state is formed by inserting a second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. The aforementioned image processing device is A generation unit that generates a group of images based on the signal acquired without changing the relative positional relationship in the optical axis direction between the sample and the irradiation position of the illumination light, including a first image of a first focal plane based on the signal acquired through the optical system in the first state, and a second image of a second focal plane that substantially coincides with the first focal plane and the optical system in the optical axis direction, based on the signal acquired through the optical system in the second state, A microscope comprising: an estimation unit that estimates the structure of each of a plurality of surfaces in the sample, including a first estimated sample surface, along the optical axis direction, based on the aforementioned image group and the first three-dimensional point image amplitude distribution and the second three-dimensional point image amplitude distribution, and outputs an estimated image based on the estimated structure, The estimation unit is a microscope that includes at least one of the following: i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first three-dimensional point image amplitude distribution and the second three-dimensional point image amplitude distribution, the structure is estimated, and an image obtained by integrating the estimated structure is output as the estimated image.

17. An image processing apparatus that generates an image of a sample based on a signal in a detector of detected light from a sample when the sample is irradiated with illumination light by an optical system, and processes the image, If the first three-dimensional point image intensity distribution based on the optical system in the first state is different from the second three-dimensional point image intensity distribution based on the optical system in a second state which is different from the first state, The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by arranging the point detector in a first position, and the second state is formed by arranging the point detector in a second position different from the first position. ii) When the detector is composed of a plurality of detection units or a plurality of point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting the optical element into the optical system, and the second state is formed by retracting the optical element from the optical system. iv) The first state is formed by inserting a first optical element into the optical system, and the second state is formed by inserting a second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. A generation unit that generates a group of images based on the signal acquired without changing the relative positional relationship between the sample and the irradiation position of the illumination light in the optical axis direction of the optical system, including a first image of the first focal plane based on the signal acquired in the optical system in the first state and a second image of the second focal plane based on the signal acquired in the optical system in the second state, An image processing apparatus comprising: an estimation unit that estimates the structure of each of a plurality of surfaces in the sample, including a first estimated sample surface, along the optical axis direction, based on the aforementioned image group and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputs an estimated image based on the estimated structure, The estimation unit includes at least one of the following image processing devices: i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, the structure is estimated, and an image obtained by integrating the estimated structure is output as the estimated image.

18. A method for generating an image of a sample based on the signal in a detector of detected light from a sample when the sample is irradiated with illumination light by an optical system, and for processing the image, If the first three-dimensional point image intensity distribution based on the optical system in the first state is different from the second three-dimensional point image intensity distribution based on the optical system in a second state which is different from the first state, The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by arranging the point detector in a first position, and the second state is formed by arranging the point detector in a second position different from the first position. ii) When the detector is composed of a plurality of detection units or a plurality of point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting the optical element into the optical system, and the second state is formed by retracting the optical element from the optical system. iv) The first state is formed by inserting a first optical element into the optical system, and the second state is formed by inserting a second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. The method generates a set of images based on the signal acquired without changing the relative positional relationship between the sample and the irradiation position of the illumination light in the optical axis direction of the optical system, including a first image of the first focal plane based on the signal acquired in the first state and a second image of the second focal plane based on the signal acquired in the second state of the optical system. An image processing method comprising: estimating the structure of each of a plurality of surfaces in the sample, including a first estimated sample surface, along the optical axis direction, based on the aforementioned image group and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution; and outputting an estimated image based on the estimated structure, Outputting the aforementioned estimated image is an image processing method that includes at least one of the following: i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, the structure is estimated, and an image obtained by integrating the estimated structure is output as the estimated image.

19. To generate an image of the sample based on the signal in the detector of the detected light from the sample when the sample is irradiated with illumination light by the optical system, and to process the image, a computer is used. If the first three-dimensional point image intensity distribution based on the optical system in the first state is different from the second three-dimensional point image intensity distribution based on the optical system in a second state which is different from the first state, The optical system in the first state and the optical system in the second state are formed by at least one of the following: i) If the detector is a single point detector, the first state is formed by arranging the point detector in a first position, and the second state is formed by arranging the point detector in a second position different from the first position. ii) When the detector is composed of a plurality of detection units or a plurality of point detectors, the first state is formed by the detection unit or point detector located at the third position, and the second state is formed by the detection unit or point detector located at a fourth position different from the third position. iii) The first state is formed by inserting the optical element into the optical system, and the second state is formed by retracting the optical element from the optical system. iv) The first state is formed by inserting a first optical element into the optical system, and the second state is formed by inserting a second optical element in place of the first optical element into the optical system. v) The first state is formed by arranging an optical element having a first aperture in the optical system, and the second state is formed by changing the first aperture of the optical element to a second aperture different from the first aperture. The method generates a set of images based on the signal acquired without changing the relative positional relationship between the sample and the irradiation position of the illumination light in the optical axis direction of the optical system, including a first image of the first focal plane based on the signal acquired in the first state and a second image of the second focal plane based on the signal acquired in the second state of the optical system. An image processing program that performs the following actions: estimates the structure of each of a plurality of surfaces in the sample, including the first estimated sample surface, along the optical axis direction, based on the aforementioned image group and the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, and outputs an estimated image based on the estimated structure. The image processing program that outputs the estimated image includes at least one of the following: i) Different weighting processes are applied to the structure of each of the estimated multiple surfaces in the direction of the optical axis, and an image obtained by integrating the weighted structures is output as the estimated image. ii) After applying different weighting processes in the optical axis direction to the first three-dimensional point image intensity distribution and the second three-dimensional point image intensity distribution, the structure is estimated, and an image obtained by integrating the estimated structure is output as the estimated image.

Images