Sample image acquiring apparatus and sample image generating apparatus

Abstract

To provide a sample image acquisition device capable of calculating a refractive index distribution with high accuracy.SOLUTION: The sample-image acquisition apparatus includes a first optical system that forms a first optical image of a sample, a second optical system that forms a second optical image of the sample, a movement mechanism, and an image generation unit. The fluorescence image group is generated based on the first optical image, and the bright-field image group is generated based on the second optical image. The image generation unit includes a processor, and the processor calculates an estimated image of the estimated sample using the refractive index distribution of the estimated sample, calculates an optimized refractive index distribution as a final refractive index distribution using the bright-field image and the estimated image, calculates a point spread function using the final refractive index distribution, and generates an image corresponding to the fluorescence image using the point spread function and the fluorescence image of the area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a specimen image acquisition device and a specimen image generation device. [Background technology]

[0002] For example, in a microscope or endoscope, an optical system forms an optical image of a specimen, and an image of the optical image is acquired by capturing the optical image with an image sensor.

[0003] Figure 7 shows how an image is formed, an optical image, and an image of the optical image. Figure 7(a) shows how an image is formed when the specimen is a three-dimensional object. Figure 7(b) shows an optical image of the specimen in the XY cross section. Figure 7(c) shows an optical image of the specimen in the XZ cross section.

[0004] The optical axis of the optical system is the Z axis, the axis perpendicular to the Z axis is the X axis, and the axis perpendicular to both the Z axis and the X axis is the Y axis. The XY cross section is a plane that includes the X axis and the Y axis. The XZ cross section is a plane that includes the X axis and the Z axis.

[0005] The specimen is a cell mass. The cell mass is made up of multiple cells, each of which has a cell nucleus.

[0006] As shown in Figure 7(a), the specimen OBJ is a cell mass, so the specimen OBJ has thickness not only in the direction perpendicular to the optical axis AX but also in the direction parallel to the optical axis AX. An optical image IMG is formed on an image plane IP by the optical system OS.

[0007] In the specimen OBJ, only the cell nuclei are stained with a fluorescent dye. Therefore, when excitation light is irradiated onto the specimen OBJ, fluorescence is emitted only from the cell nuclei. As a result, a fluorescent image of the cell nuclei is formed as the optical image IMG.

[0008] By capturing the optical image IMG, an optical image PIC can be obtained. In the optical image PIC, only the cell nuclei are imaged.

[0009] The image plane IP is conjugate with the focal plane FP. The optical image IMG represents the optical image of the specimen OBJ located at the focal plane FP in the XY cross section. As shown by the arrows, multiple optical images can be acquired by capturing optical images while moving the specimen OBJ and the focal plane FP relatively along the optical axis AX.

[0010] When the specimen OBJ is fixed and the optical system OS is moved toward the specimen OBJ, the focal plane FP moves in the order of the top surface of the specimen OBJ, the inside of the specimen OBJ, and the bottom surface of the specimen OBJ.

[0011] 7(b) shows five optical images. Image PIC1, image PICm-1, image PICm, image PICm+1, and image PICn are optical images of the XY cross section of the specimen OBJ. For example, image PIC1 is an image of the top surface of the specimen OBJ, image PICm-1, image PICm, and image PICm+1 are images of the interior of the specimen OBJ, and image PICn is an image of the bottom surface of the specimen OBJ.

[0012] The position of the XY cross section of the specimen OBJ is different for each optical image, so the optical images are different from each other.

[0013] A row of data parallel to the X-axis is extracted from each of images PIC1 to PICn. By arranging the row of data along the Z-axis, an image of the XZ cross section of the optical image can be obtained.

[0014] Figure 7(c) is a diagram showing an image in the XZ cross section of the optical image. Image PICxz is an image of a cell nucleus. The left-right direction of the image is along the optical axis AX. The right side of the image represents an image of the top surface of the specimen OBJ, and the left side of the image represents an image of the bottom surface of the specimen OBJ.

[0015] When the shape of the cell nucleus is spherical, the shape of the XZ cross section is circular. As shown in Figure 7(c), in the image PICxz, deformation, reduction in sharpness, and reduction in brightness occur in all of the images of the cell nucleus. If deformation, reduction in sharpness, and reduction in brightness are considered to be image quality degradation, then image PICxz has experienced image quality degradation.

[0016] An optical image is obtained by capturing an optical image. When the image quality of an optical image deteriorates, it means that the optical image itself is degraded.

[0017] When the specimen is a point light source, it is desirable that the optical image is a point image. In order for a point image to be formed, the optical system must be an aberration-free optical system (hereinafter referred to as an "ideal optical system") and all of the light emitted from the point light source must be incident on the optical system.

[0018] However, because the size of an optical system is finite, it is not possible to allow all of the light emitted from a point light source to enter the optical system. In this case, the optical image is affected by diffraction. As a result, even if the optical system is ideal, a point image is not formed, but rather a spread image is formed. This spread image is called a point spread distribution.

[0019] The optical image is expressed by the following equation (A) using the point spread function. I=O*PSF (A) where: I is the optical image, O is the specimen, PSF is the point spread function, * denotes convolution, is.

[0020] If we consider the point spread function as an optical filter, then formula (A) expresses that the optical image is obtained through a filter called the point spread function. Degradation of the optical image means that the filter, i.e., the point spread function, has characteristics (hereinafter referred to as "degradation characteristics") that cause deformation, reduction in sharpness, and reduction in brightness.

[0021] In frequency space, equation (A) is expressed as equation (B) below. FI=FO×OTF (B) where: FI is the Fourier transform of the optical image, FO is the Fourier transform of the sample, OTF is the optical transfer function, is.

[0022] The OTF is the Fourier transform of the point spread function. If the point spread function has degradation characteristics, the OTF also has degradation characteristics.

[0023] By transforming equation (B), equation (B) is expressed as the following equation (C). FO=FI / OTF (C)

[0024] If we can find FI and OTF, we can find FO. Then, by performing an inverse Fourier transform on FO, we can find O, where O is the sample. This operation is called deconvolution.

[0025] The image PICxz shown in Figure 7(c) is an optical image of the cell nucleus. In the image PICxz, only the cell nucleus is imaged. Therefore, when deconvolution is performed using the image PICxz and OTF, only the image of the cell nucleus is obtained.

[0026] The specimen OBJ is a cell mass, and therefore has multiple cytoplasms and multiple nuclei. However, even after deconvolution of the image PICxz, only the image of the nuclei is obtained. Since the image of the cytoplasm is not obtained, it is difficult to say that the specimen OBJ has been obtained. The specimen can be obtained by deconvolution, but whether or not the specimen can be obtained depends on the optical image.

[0027] In terms of images, equation (A) shows that the optical image is an image obtained through a filter called a point spread function. If the point spread function has degradation characteristics, I can be considered to be the optical image with degraded image quality, and O can be considered to be the optical image before image quality degradation.

[0028] In this case, formula (C) indicates that an image of the optical image before the image quality degradation is generated from an image of the optical image after the image quality degradation. Hereinafter, the image of the optical image after the image quality degradation is referred to as a "degraded image." Furthermore, the image of the optical image before the image quality degradation can be said to be an image after the image quality degradation has been restored. Therefore, the image of the optical image before the image quality degradation is referred to as a "restored image."

[0029] To generate a restored image, it is necessary to calculate the point spread function. This will be explained using Figure 7(a). In Figure 7(a), the refractive index outside the specimen OBJ is n1, and the refractive index inside the specimen OBJ is n2.

[0030] The ideal shape is defined as the shape of the point spread function of an ideal optical system. In the ideal shape, the refractive index between the focal plane and the ideal optical system matches a predetermined refractive index.

[0031] The specimen OBJ is moved from a state where it is away from the focal plane FP toward the optical system OS. Since the optical system OS does not move, the top surface of the specimen OBJ reaches the focal plane FP. In this state (hereinafter referred to as the "first state"), only a space with a refractive index of n1 exists between the focal plane FP and the optical system OS. When a point light source is placed on the focal plane FP, the point spread function in the first state is obtained.

[0032] In the first state, the refractive index between the focal plane FP and the optical system OS is n1. If the predetermined refractive index is n1, the point spread function in the first state is obtained based only on the predetermined refractive index. Therefore, the shape of the point spread function in the first state is the same as the ideal shape.

[0033] If the specimen OBJ is moved further, the focal plane FP reaches the interior of the specimen OBJ. In this state (hereinafter referred to as the "second state"), a space with a refractive index of n1 and a space with a refractive index of n2 are located between the focal plane FP and the optical system OS. If a point light source is placed on the focal plane FP, the point spread function in the second state is obtained.

[0034] In the second state, the refractive index between the focal plane FP and the optical system OS is determined by n1 and n2. Since n1 is the predetermined refractive index, n2 is not a predetermined refractive index. In this case, the point spread function in the second state is obtained based on the predetermined refractive index and the non-predetermined refractive index. Therefore, the shape of the point spread function in the second state is different from the ideal shape.

[0035] In this way, the shape of the point spread function changes depending on the size of the space where the refractive index is n2. Therefore, when calculating the point spread function, the refractive index distribution in the specimen OBJ must be appropriately taken into consideration.

[0036] A technique for restoring an image is disclosed in Non-Patent Document 1. This restoration technique uses an optical image and a point spread function (PIF) acquired from a thick specimen. The PIF is calculated by dividing the specimen into multiple blocks and using the refractive index of a row of blocks parallel to the optical axis. [Prior art documents] [Non-patent literature]

[0037] [Non-Patent Document 1] Sreya Ghosh, Chrysanthe Preza, "Three-dimensional block-based restoration integrated with wide-field fluorescence microscopy for the investigation of thick specimens with spatially variant refractive index", J. of Biomedical Optics, 21(4), 046010 (2016) Summary of the Invention [Problem to be solved by the invention]

[0038] The above-mentioned restoration techniques use a specimen with a simple structure and a known refractive index. Therefore, it can be said that the refractive index distribution is known. However, even if the specimen structure is simple, the refractive index value of the specimen may be unknown. In this case, the refractive index distribution is unknown. Also, even if the refractive index value of the specimen is known, the specimen structure may be complex. In this case, it can also be said that the refractive index distribution is unknown.

[0039] For example, in a cell cluster, cells are randomly stacked. In this case, the structure of the specimen is complex. Therefore, even if the refractive index values ​​of the cytoplasm and the cell nucleus are known, the refractive index distribution is unknown.

[0040] Therefore, it is desirable to be able to calculate the refractive index distribution with high accuracy, regardless of whether the specimen has a complex structure or not, and whether the refractive index value is known or not. It is also desirable to be able to recover the image with high accuracy.

[0041] The present invention has been made in consideration of these problems, and aims to provide a specimen image acquisition device that can calculate the refractive index distribution with high accuracy and a specimen image generation device that can restore an image with high accuracy. [Means for solving the problem]

[0042] In order to solve the above-mentioned problems and achieve the object, a specimen image acquisition device according to at least some embodiments of the present invention comprises: a first optical system that forms a first optical image of the sample; a second optical system that forms a second optical image of the sample; A moving mechanism; an image generation unit; the first optical system has an observation optical system located on an optical path from the specimen to the first optical image; The optical axis of the first optical system and the optical axis of the second optical system are aligned at the position of the specimen, the moving mechanism changes a distance between the specimen and the first objective lens of the first optical system and a distance between the specimen and the second objective lens of the second optical system; By changing the interval, a fluorescent image group made up of a plurality of fluorescent images and a bright-field image group made up of a plurality of bright-field images are generated, a fluorescence image group is generated based on the first optical image; The bright-field images are generated based on the second optical image; The estimation sample is a sample that models the sample, The virtual observation optical system is an optical system that models the observation optical system, The image generation unit a memory for storing a set of fluorescent images and a set of bright-field images; a processor, The processor Calculating an estimated image of the estimated specimen using the refractive index distribution of the estimated specimen; Using the bright-field image and the estimated image, an optimized refractive index distribution is calculated as the final refractive index distribution. Divide the fluorescent image into multiple areas, In the estimation sample, a virtual area corresponding to the area is set, A virtual ray is set to travel from the virtual area toward the virtual observation optical system; Calculating a point spread function corresponding to the area using a final refractive index distribution included in the range through which the virtual ray travels; generating an image corresponding to each area using the point spread function and the fluorescence image of the area; All the images are combined to generate a final image corresponding to the fluorescent image; A final refractive index distribution calculation is performed for each image in the bright field image set; The generation of the final image is performed for each image in the fluorescent image group.

[0043] Furthermore, the specimen image generating device according to at least some embodiments of the present invention includes: a memory for storing a fluorescent image of the specimen and a bright-field image of the specimen; a processor, the fluorescent image is generated based on a first optical image formed by the first optical system; the bright-field image is generated based on a second optical image formed by the second optical system; the optical axis of the first optical system and the optical axis of the second optical system include a single optical axis that coincides at the position of the sample; The multiple fluorescent images and the multiple bright-field images are images taken at different positions in a direction parallel to a single optical axis, The estimation sample is a sample that models the sample, the first optical system has a first observation optical system located on an optical path from the specimen to the first optical image; the virtual observation optical system is an optical system that models the first observation optical system, The processor Calculating an estimated image of the estimated specimen using the refractive index distribution of the estimated specimen; Using the bright-field image and the estimated image, an optimized refractive index distribution is calculated as the final refractive index distribution. Divide the fluorescent image into multiple areas, In the estimation sample, a virtual area corresponding to the area is set, A virtual ray is set to travel from the virtual area toward the virtual observation optical system; Calculating a point spread function corresponding to the area using a final refractive index distribution included in the range through which the virtual ray travels; generating an image corresponding to each area using the point spread function and the fluorescence image of the area; All the images are synthesized to generate an image corresponding to the fluorescent image. [Effects of the Invention]

[0044] According to the present invention, it is possible to provide a specimen image acquisition device that can acquire a refractive index distribution with high accuracy and a specimen image generation device that can restore an image with high accuracy. [Brief explanation of the drawings]

[0045] [Figure 1] FIG. 1 is a diagram illustrating a specimen image acquisition device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an optical system of a specimen image acquisition device. [Figure 3] FIG. 1 is a diagram illustrating a specimen image acquisition device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing a light source and an aperture member. [Figure 5] FIG. 10 is a diagram showing another specimen image acquisition device according to the present embodiment. [Figure 6] 1A and 1B are diagrams illustrating a degraded image and a restored image. [Figure 7] 1A and 1B are diagrams illustrating an image formation state, an optical image, and an image of the optical image. DETAILED DESCRIPTION OF THE INVENTION

[0046] Prior to describing the examples, the effects of certain embodiments of the present invention will be described. When specifically describing the effects of the present embodiment, specific examples will be used. However, as with the examples described below, these exemplified aspects are merely a portion of the aspects included in the present invention, and there are numerous variations within these aspects. Therefore, the present invention is not limited to the exemplified aspects.

[0047] The specimen image acquisition device and the specimen image generation device of this embodiment use an optical image of the specimen. An optical image of the specimen is formed by an optical system, and the optical image of the specimen is captured by an image sensor, thereby obtaining the optical image of the specimen.

[0048] Since a specimen is a three-dimensional object, the optical image of the specimen can be expressed as an XY image, an XZ image, and a YZ image. Furthermore, since the optical image of the specimen is acquired through an optical system, the optical image of the specimen is a degraded image.

[0049] The optical axis at the position of the specimen is the Z axis, the axis perpendicular to the Z axis is the X axis, and the axis perpendicular to both the Z axis and the X axis is the Y axis. The XY cross section is a plane containing the X and Y axes. The XY image is an image in the XY cross section. The XZ cross section is a plane containing the X and Z axes. The XZ image is an image in the XZ cross section. The YZ cross section is a plane containing the Y and Z axes. The YZ image is an image in the YZ cross section.

[0050] The specimen image acquisition device of this embodiment includes a first optical system that forms a first optical image of the specimen, a second optical system that forms a second optical image of the specimen, a movement mechanism, and an image generation unit.

[0051] Fig. 1 is a diagram showing a specimen image acquisition device according to this embodiment. Fig. 1 shows the configuration of the optical system, but is not limited to this configuration. Furthermore, illumination light is represented by a dashed line, and imaging light is represented by a solid line.

[0052] The specimen image acquisition device 1 includes a first optical system 2, a second optical system 3, a moving mechanism 4, and an image generation unit 5. A specimen 6 is held on a stage 7. A first light source 8 and a second light source 9 are arranged to illuminate the specimen 6. An optical image of the specimen 6 is captured by an image sensor 10. A CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) is used as the image sensor 10.

[0053] 1, the first optical system 2 and the second optical system 3 are depicted together. Therefore, optical elements that are not used in the first optical system 2 and optical elements that are not used in the second optical system 3 are depicted. Each of the first optical system 2 and the second optical system 3 will be described.

[0054] Figure 2 shows the optical system of the specimen image acquisition device. Figure 2(a) shows the first optical system, and Figure 2(b) shows the second optical system. The same components as in Figure 1 are assigned the same numbers.

[0055] The first optical system will now be described. As shown in Fig. 2(a), the first optical system 2 includes a first observation optical system 20 and a first illumination optical system 21. The first observation optical system 20 is located on the optical path from the specimen 6 to the first optical image IM1. The first illumination optical system 21 is located on the optical path from the first light source 8 to the specimen 6.

[0056] The first observation optical system 20 includes a first objective lens 22, an absorption filter 23, and an imaging lens 24. The first illumination optical system 21 includes a collector lens 25, a relay lens 26, a relay lens 27, an excitation light filter 28, a dichroic mirror 29, and the first objective lens 22.

[0057] In the first optical system 2, the specimen 6 is illuminated via a first illumination optical system 21. A first light source 8 is disposed in the optical path of the first optical system 2. An image of the first light source 8 is formed at the pupil position of the first objective lens 22 by a collector lens 25, a relay lens 26, and a relay lens 27.

[0058] The illumination light emitted from the first light source 8 is incident on the excitation light filter 28. The excitation light filter 28 has optical properties that allow the excitation light to pass through. Therefore, the excitation light is emitted from the excitation light filter 28. The excitation light reaches the dichroic mirror 29.

[0059] The dichroic mirror 29 has optical properties that reflect the excitation light and transmit the fluorescence. Therefore, the excitation light is reflected by the dichroic mirror 29 and enters the first objective lens 22. As a result, the specimen 6 is illuminated with the excitation light via the first objective lens 22.

[0060] The specimen 6 is stained with a fluorescent dye. The excitation light is light in a wavelength range that excites the fluorescent dye. When the specimen 6 is illuminated with the excitation light, fluorescence is generated in the specimen 6. The fluorescence enters the first objective lens 22, passes through the dichroic mirror 29, and enters the absorption filter 23.

[0061] The absorption filter 23 has optical properties that allow it to transmit the fluorescence. After transmitting through the absorption filter 23, the fluorescence is collected by the imaging lens 24. As a result, a first optical image IM1 is formed on the image plane of the first observation optical system 20.

[0062] The imaging plane of the imaging element 10 is located on the image plane of the first observation optical system 20. The first optical image IM1 is captured by the imaging element 10. As a result, an image of the first optical image IM1 can be acquired. The first optical image IM1 is an optical image formed by fluorescence. Therefore, the image of the first optical image IM1 is a fluorescence image.

[0063] The second optical system will now be described. As shown in FIG. 2(b), the second optical system 3 includes a second observation optical system 30 and a second illumination optical system 31. The second observation optical system 30 is located on the optical path from the specimen 6 to the second optical image IM2. The second illumination optical system 31 is located on the optical path from the second light source 9 to the specimen 6.

[0064] The second observation optical system 30 includes a second objective lens 32 and an imaging lens 33. The second illumination optical system 31 includes a collector lens , an aperture stop 35, and a condenser lens .

[0065] In the second optical system 3, the specimen 6 is illuminated via a second illumination optical system 31. A second light source 9 is disposed in the optical path of the second optical system 3. An image of the second light source 9 is formed at the pupil position of a condenser lens 36 by a collector lens 34. An aperture stop 35 is disposed at the pupil position of the condenser lens 36.

[0066] The illumination light emitted from the second light source 9 reaches the specimen 6 as is. Therefore, bright field illumination is performed on the specimen 6. Since the illumination light is white light, the specimen 6 is illuminated with white light.

[0067] The white light transmitted through the specimen 6 is incident on the second objective lens 32 and then condensed by the imaging lens 33. As a result, a second optical image IM2 is formed on the image plane of the second observation optical system 30.

[0068] The imaging plane of the imaging element 10 is located on the image plane of the second observation optical system 30. The second optical image IM2 is captured by the imaging element 10. As a result, an image of the second optical image IM2 can be acquired. The second optical image IM2 is an optical image formed with white light. Therefore, the image of the second optical image IM2 is a bright-field image.

[0069] The absorption filter 23, the excitation light filter 28, and the dichroic mirror 29 form a fluorescence mirror unit. The fluorescence mirror unit can move from outside the optical path to inside the optical path and vice versa. When the first optical image IM1 is formed, the mirror unit is located inside the optical path. When the second optical image IM2 is formed, the mirror unit is located outside the optical path.

[0070] The optical axis AX1 of the first optical system 2 and the optical axis AX2 of the second optical system 3 coincide at the position of the specimen 6. Specifically, the optical axis of the first observation optical system 20 and the optical axis of the second observation optical system 30 coincide at the position of the specimen 6. Since the optical axis at the position of the specimen 6 is the Z axis, the specimen 6, the first observation optical system 20, and the second observation optical system 30 are located on the Z axis.

[0071] In the first optical system 2, a moving mechanism 4 is provided for the first objective lens 22. In this case, the first objective lens 22 moves in the Z-axis direction. In the second optical system 3, a moving mechanism 4 is provided for the second objective lens 32. In this case, the second objective lens 32 moves in the Z-axis direction. The moving mechanism 4 may be provided on the stage 7. In this case, the specimen 6 moves in the Z-axis direction as the stage 7 moves.

[0072] The movement mechanism 4 changes the distance between the specimen 6 and the first objective lens 22 and the distance between the specimen 6 and the second objective lens 32. In the first optical system 2, the focal position of the first observation optical system 20 and the position of the specimen 6 change relatively in the Z-axis direction. In the second optical system 3, the focal position of the second observation optical system 30 and the position of the specimen 6 change relatively in the Z-axis direction.

[0073] When the distance between the specimen 6 and the first objective lens 22 changes, the focal position of the first observation optical system 20 relative to the specimen 6 changes in the Z-axis direction. The specimen 6 is a three-dimensional object. Therefore, when the first optical image IM1 is captured while changing the distance, multiple images of the specimen 6 at different positions in the Z-axis direction are generated. Because the image generated based on the first optical image IM1 is a fluorescent image, a fluorescent image group made up of multiple fluorescent images is generated.

[0074] When the distance between the specimen 6 and the second objective lens 32 changes, the focal position of the second observation optical system 30 relative to the specimen 6 changes in the Z-axis direction. As described above, the specimen 6 is a three-dimensional object. Therefore, when the second optical image IM2 is captured while changing the distance, multiple images of the specimen 6 at different positions in the Z-axis direction are generated. Because the image generated based on the second optical image IM2 is a bright-field image, a bright-field image group composed of multiple bright-field images is generated.

[0075] Each image in the fluorescent image group and each image in the bright-field image group is an XY image. Each XY image has a different position in the Z axis direction. Therefore, an XY image, an XZ image, and a YZ image can be generated based on each image.

[0076] The image generation unit 5 is connected to the movement mechanism 4, the first light source 8, the second light source 9, and the image sensor 10. The image generation unit 5 controls the movement mechanism 4, the first light source 8, and the second light source 9. In addition, an image generated by the image sensor 10 is input to the image generation unit 5.

[0077] 3 is a diagram showing the specimen image acquisition device of this embodiment. The same components as those in FIG. 1 are assigned the same numbers and their explanations will be omitted.

[0078] The image generation unit 5 includes a memory 51, a processor 52, and an input / output unit 53. The memory 51 stores a group of fluorescent images and a group of bright-field images. The processor 52 executes various processes. Various control signals are output and images are input via the input / output unit 53. Images may also be output via the input / output unit 53.

[0079] Bright-field images, fluorescent images, estimated specimens, and estimated images are used in various processes. Bright-field images are formed from a plurality of pixels. Each pixel is assigned a value of transmitted light intensity. Fluorescent images are also formed from a plurality of pixels. Each pixel is assigned a value of fluorescent intensity.

[0080] The estimated sample is used to calculate the estimated image and the point spread function. For this purpose, a refractive index distribution is set for the estimated sample. The refractive index distribution is expressed by multiple refractive index values. By treating the estimated sample as an image, a refractive index value can be set for each pixel.

[0081] The estimated image is used for comparison with the bright-field image. Therefore, the estimated image is also an image formed by multiple pixels, and each pixel is set to a calculated value of the transmitted light intensity.

[0082] The processor 52 executes various processes, including a process for calculating the refractive index distribution and a process for restoring an image.

[0083] In image restoration, a restored image is generated from a degraded image. The degraded image is an optical image whose image quality has been degraded, and the restored image is an optical image before the image quality was degraded.

[0084] The restored image can be generated from the degraded image and the point spread function. The accuracy of the restored image depends on the accuracy of the point spread function. Therefore, it is preferable that the point spread function be calculated with high accuracy.

[0085] The refractive index distribution of the sample is used to calculate the point spread function. Therefore, the refractive index distribution of the sample must be highly accurate. By using a highly accurate refractive index distribution, the accuracy of the point spread function can be improved.

[0086] The specimen 6 to be used in the specimen image acquisition device 1 is determined by the observer. Therefore, the type of specimen 6 is known. However, the refractive index distribution of the specimen 6 is often unknown.

[0087] The process of calculating the refractive index distribution (hereinafter referred to as the "calculation process") will be described. In the calculation process, the final refractive index distribution is calculated. In order to execute the calculation process, an estimated sample is set. The estimated sample is a sample that models the sample 6. A refractive index distribution can be set in the estimated sample as refractive index information.

[0088] In the calculation process, one bright-field image is selected from multiple bright-field images. An arbitrary refractive index distribution is set for the estimated specimen. The arbitrary refractive index distribution can be considered as an initial value. Next, an estimated image of the estimated specimen is calculated using the set arbitrary refractive index distribution. Then, an optimized refractive index distribution (hereinafter referred to as the "final refractive index distribution") is calculated using the bright-field image and the estimated image.

[0089] The bright-field image and the estimated image are compared while changing the refractive index distribution until the final refractive index distribution is calculated. The comparison is stopped when the residual difference between the bright-field image and the estimated image falls below a preset standard, and the refractive index distribution at that time is taken as the final refractive index distribution.

[0090] For information on how to calculate the final refractive index distribution, refer to, for example, International Publication No. 2021 / 024420 for a deeper understanding.

[0091] The final refractive index profile is used to calculate the point spread function in the image restoration process described below. Image restoration is performed for each fluorescence image. Therefore, the calculation process is performed multiple times.

[0092] In the calculation process, each time an image is restored, a bright-field image is selected, an arbitrary refractive index distribution is set, the bright-field image is compared with the estimated image, and the final refractive index distribution is calculated. Therefore, the final refractive index distribution of the estimated specimen is updated each time an image is restored.

[0093] Using the estimated sample set eliminates the need to update the final refractive index distribution each time an image is restored. The estimated sample set has the same number of estimated samples as the number of bright-field images. Using the estimated sample set, the final refractive index distribution can be calculated for all bright-field images before image restoration.

[0094] The specimen image acquisition device of this embodiment uses bright-field images in the calculation process. Because bright-field images are based on an optical image of the specimen, the calculation of the final refractive index distribution can be considered to be performed based on the specimen. Therefore, even if the refractive index distribution is complex or the refractive index value is unknown, it can be considered that the final refractive index distribution reproduces the refractive index distribution of the specimen with high accuracy. As a result, the specimen image acquisition device of this embodiment can calculate the refractive index distribution with high accuracy, regardless of whether the specimen structure is complex or whether the refractive index value is known or not.

[0095] As described above, a bright-field image is used in the calculation process. The bright-field image is generated based on an optical image of the specimen. Therefore, it is preferable to illuminate the specimen so as to form an optical image suitable for calculating the final refractive index distribution. If partial coherent illumination is used to illuminate the specimen, an optical image suitable for calculating the final refractive index distribution can be formed.

[0096] In the second illumination optical system 31, an aperture stop 35 is disposed at the position of the pupil of a condenser lens 36. The aperture stop 35 has an opening. The size of the opening can be considered to be the size of the pupil of the condenser lens 36.

[0097] An optical image of the light source is formed at the position of the aperture stop. The magnification of the collector lens 34 is set so that the size of the optical image is smaller than the size of the aperture. Therefore, the size of the optical image is smaller than the size of the pupil of the condenser lens 36. In this case, the pupil of the condenser lens 36 is not filled with illumination light, so partial coherent illumination is performed on the specimen 6.

[0098] Figure 4 is a diagram showing a light source and an aperture member, where Figure 4(a) is a diagram showing the light source, and Figures 4(b) and 4(c) are diagrams showing the aperture member.

[0099] As shown in FIG. 4(a), the light source 60 has a substrate 61 and a light-emitting unit 62. The light-emitting unit 62 is disposed at a position away from the optical axis AX2. The shape of the light-emitting unit 62 is annular, but is not limited to this. For example, a plurality of light-emitting units may be discretely disposed on the circumference. An LED (light emitting diode) is used for the light-emitting unit 62.

[0100] The light source 60 is disposed at the position of the aperture stop 35. When using the light source 60 for illumination, no illumination light is present in a circular area including the optical axis AX2. Therefore, the pupil of the condenser lens 36 is not filled with illumination light. Therefore, by using the light source 60, partial coherent illumination can be achieved.

[0101] As shown in Fig. 4(b), the aperture member 70 has a transmitting portion 71 and a light-shielding portion 72. The transmitting portion 71 is provided at a position away from the center of the aperture member 70. The shape of the transmitting portion 71 is annular, but is not limited to an annular shape. For example, a plurality of transmitting portions may be discretely arranged on the circumference.

[0102] The aperture member 70 is disposed at the position of the aperture stop 35. In illumination using the aperture member 70, illumination light does not exist in a circular area including the optical axis AX2. Therefore, the pupil of the illumination optical system is not filled with illumination light. Therefore, by using the aperture member 70, partial coherent illumination can be achieved.

[0103] 4(c), the aperture member 80 has a first transmitting portion 81, a second transmitting portion 82, and a light-shielding portion 83. The first transmitting portion 81 and the second transmitting portion 82 are provided at positions away from the center of the aperture member 80.

[0104] The first transmission portion 81 is composed of a plurality of transmission holes arranged on a first circumference. The second transmission portion 82 is composed of a plurality of transmission holes arranged on a second circumference. The diameter of the first circumference is smaller than the diameter of the second circumference.

[0105] The aperture member 80 is disposed at the position of the aperture stop 35. In illumination using the aperture member 80, illumination light does not exist in a circular region including the optical axis AX2. Therefore, the pupil of the illumination optical system is not filled with illumination light. Therefore, by using the aperture member 80, partial coherent illumination can be achieved.

[0106] The process of restoring an image (hereinafter referred to as "restoration process") will be described. In the restoration process, a restored image is generated from a degraded image. In the specimen image acquisition device of this embodiment, an image corresponding to the fluorescent image is generated as a final image based on the fluorescent image. Therefore, the fluorescent image corresponds to the degraded image. Furthermore, the image corresponding to the fluorescent image, i.e., the final image, corresponds to the restored image.

[0107] Prior to the restoration process, the following process is performed: The fluorescent image is divided into a plurality of areas. The estimated specimen is also divided into a plurality of areas. These divisions allow one area of ​​the fluorescent image to correspond to one virtual area of ​​the estimated specimen. Therefore, a virtual area corresponding to the area of ​​the fluorescent image is set in the estimated specimen.

[0108] Furthermore, a refractive index distribution can be set for the estimated specimen. As described above, the calculation process is performed to calculate the final refractive index distribution. Therefore, the final refractive index distribution is set for the estimated specimen.

[0109] As described above, a fluorescence image is formed from a plurality of pixels, and therefore the area of ​​the fluorescence image is also formed from a plurality of pixels. Furthermore, if the estimated specimen is considered to be an image, the estimated specimen is formed from a plurality of pixels. Therefore, the virtual area is also formed from a plurality of pixels.

[0110] In addition, a virtual observation optical system is set. The virtual observation optical system is an optical system that models the observation optical system used to generate the degraded image. As described above, a fluorescent image is used as the degraded image. Therefore, an optical system that models the first observation optical system 20 is used as the virtual observation optical system.

[0111] When the above process is completed, one virtual area is selected from the multiple virtual areas. Next, a virtual ray is set. To set the virtual ray, a virtual point light source is set at the center of the selected virtual area. The virtual ray is a light ray that travels from the virtual point light source toward the virtual observation optical system. Once the virtual ray is set, the virtual area included in the range through which the virtual ray travels is identified.

[0112] Hereinafter, the virtual areas included in the range in which the virtual ray travels will be referred to as a virtual area group. The number of virtual areas included in the virtual area group will vary depending on the selected virtual area and the range of the virtual ray. The number of virtual areas included in the virtual area group will be zero or more.

[0113] For example, if the virtual area located farthest from the virtual observation optical system is selected, the virtual light rays will reach the virtual observation optical system via other virtual areas. Therefore, the number of virtual areas included in the virtual area group will be one or more.

[0114] Also, suppose that the virtual area located closest to the virtual observation optical system is selected. In this case, the virtual light rays reach the virtual observation optical system without passing through any other virtual areas. Therefore, the number of virtual areas included in the virtual area group becomes zero.

[0115] A final refractive index distribution is set for the estimated specimen. Therefore, the final refractive index distribution in the group of virtual areas can be obtained. Then, the point spread function corresponding to the selected virtual area is calculated using the final refractive index distribution in the group of virtual areas. There is a one-to-one correspondence between the virtual area of ​​the estimated specimen and the area of ​​the fluorescence image. Therefore, a point spread function corresponding to the area of ​​the fluorescence image is calculated.

[0116] The point spread function is calculated by setting a virtual point light source in the selected virtual area. The virtual point light source is located only in the center of the selected virtual area. Therefore, one point spread function is calculated for one virtual area.

[0117] Once the point spread function is calculated, an image corresponding to the area of ​​the fluorescence image is generated using the point spread function and the area of ​​the fluorescence image. As described above, the area of ​​the fluorescence image is formed by multiple pixels. Since only one point spread function is calculated, the same point spread function is used for each pixel of the area of ​​the fluorescence image.

[0118] The same process is performed for each area of ​​the fluorescent image to generate an image corresponding to each area of ​​the fluorescent image, and all images are then combined to generate a final image corresponding to the fluorescent image.

[0119] In the above process, one final image is generated from one fluorescent image. A fluorescent image group is composed of multiple fluorescent images. Therefore, the final refractive index distribution is calculated for each image in the bright-field image group, and the final image is generated for each image in the fluorescent image group.

[0120] The method for generating the final image can be better understood by reference to, for example, WO 2023 / 175860.

[0121] As described above, the specimen image acquisition device of this embodiment calculates the refractive index distribution based on the optical image of the specimen. Therefore, regardless of whether the specimen has a complex structure or not, and regardless of whether the refractive index value is known or not, it is possible to calculate the refractive index distribution with high accuracy. Furthermore, since a highly accurate refractive index distribution is used, it is possible to calculate the point spread function with high accuracy. Furthermore, since a highly accurate point spread function can be used, it is possible to restore the image with high accuracy.

[0122] In the area of ​​the fluorescence image, it is preferable to calculate the point spread function for each pixel. However, in this case, it takes a lot of time to calculate the point spread function, which results in a long recovery process.

[0123] In the specimen image acquisition device of this embodiment, one point spread function is calculated for one area of ​​a fluorescent image. Therefore, the same point spread function is used for each pixel in one area of ​​a fluorescent image. As a result, the recovery process time can be shortened.

[0124] The specimen image generating device of this embodiment includes a memory for storing a fluorescent image of a specimen and a bright-field image of the specimen, and a processor. The fluorescent image is an image generated based on a first optical image formed by a first optical system. The bright-field image is an image generated based on a second optical image formed by a second optical system.

[0125] The image generating unit 5 can be used as a specimen image generating device. However, the input / output section 53 only needs to have a function for inputting and outputting images, and does not necessarily have a function for outputting various control signals.

[0126] In the specimen image generating device, the fluorescent image and the bright-field image are generated by a device separate from the specimen image generating device, such as a microscope. This microscope has a first optical system and a second optical system. The first optical system has a first observation optical system. The first observation optical system is located in the optical path from the specimen to the first optical image.

[0127] The optical axis of the first optical system and the optical axis of the second optical system comprise a single optical axis that coincides with the position of the sample, and the multiple fluorescent images and multiple bright-field images stored in the memory are images taken at different positions in a direction parallel to the single optical axis.

[0128] The processor sets an estimated specimen and a virtual observation optical system. The estimated specimen is a model of the specimen used to acquire the image. The virtual observation optical system is an optical system that models the first observation optical system. Once these settings are complete, various processes are performed. The various processes are the same as those in the specimen image acquisition device, so explanations will be omitted.

[0129] In the specimen image acquisition device of this embodiment, it is preferable that a common objective lens is arranged in the optical path where the optical axis of the first optical system and the optical axis of the second optical system are aligned, and that the first optical image and the second optical image are formed through the common objective lens.

[0130] The final refractive index distribution is calculated using the bright-field image. Then, a point spread function corresponding to the area of ​​the fluorescence image is calculated from the final refractive index distribution. If there is a misalignment between the position of the specimen in the fluorescence image and the position of the specimen in the bright-field image, the position of the final refractive index distribution will be misaligned with the position of the specimen in the fluorescence image. Therefore, the point spread function corresponding to the area of ​​the fluorescence image cannot be accurately calculated. As a result, the image cannot be restored with high accuracy.

[0131] In the specimen image acquisition device of this embodiment, a common objective lens is arranged in the optical path where the optical axis of the first optical system and the optical axis of the second optical system coincide with each other. Therefore, when the common objective lens is the first objective lens, a first optical image is formed, and when the common objective lens is the second objective lens, a second optical image is formed.

[0132] As described above, in the specimen image acquisition device of this embodiment, the first optical image and the second optical image are formed by a common objective lens, i.e., a single objective lens. Since a single imaging element is used to capture the optical images, the first optical image and the second optical image are captured by the same imaging element. When the same imaging lens is used in the first observation optical system and the second observation optical system, the first optical image and the second optical image are formed at the same position on the imaging plane. Furthermore, in this case, since the magnification of the objective lens and the magnification of the imaging lens are the same, the size of the first optical image and the size of the second optical image are the same. Therefore, there is no discrepancy in the position or size of the optical image between the first optical image and the second optical image.

[0133] The fluorescence image is generated based on the first optical image, and the bright-field image is generated based on the second optical image. Because the first and second optical images are formed at the same position and with the same size on the imaging plane, there is no misalignment between the position of the specimen in the fluorescence image and the position of the specimen in the bright-field image. This allows the point spread function corresponding to the area of ​​the fluorescence image to be accurately calculated. As a result, the image can be restored with high accuracy.

[0134] In the specimen image acquisition device of this embodiment, it is preferable that the second optical system has an illumination optical system and satisfies the following conditional expression (1). 0.1≦NAill / NA≦1 (1) where: NAill is the numerical aperture of the illumination optical system, NA is the numerical aperture of the second objective lens, is.

[0135] As described above, the bright-field image group is made up of multiple bright-field images, each of which is located at a different position in the Z-axis direction.

[0136] If the lower limit of conditional expression (1) is exceeded, the specimen is illuminated in a state close to coherent illumination. This results in a deterioration in the resolution in the Z-axis direction of the second observation optical system. If the upper limit of conditional expression (1) is exceeded, the specimen is illuminated in a state close to incoherent illumination. This results in a decrease in the contrast of the second optical image.

[0137] In both cases, the image quality of each bright-field image is reduced. The bright-field image is used in the calculation process, i.e., to calculate the final refractive index distribution. Therefore, if the image quality of the bright-field image is reduced, it becomes difficult to calculate the refractive index distribution with high accuracy. As a result, it becomes difficult to restore the image with high accuracy.

[0138] It is preferable to satisfy the following conditional expression (1') instead of conditional expression (1). 0.2≦NAill / NA≦0.8 (1')

[0139] In the specimen image acquisition device of this embodiment, the processor preferably aligns the fluorescent image and the bright-field image before calculating the estimated image.

[0140] The first optical image is an optical image formed using fluorescent light. The second optical image is an optical image formed using white light. The wavelength bands of fluorescent light and white light are different. Therefore, an objective lens suitable for imaging using fluorescent light can be used as the first objective lens, and an objective lens suitable for imaging using white light can be used as the second objective lens.

[0141] In this case, it is preferable that the amount of decentering of the optical systems that make up the objective lenses is zero for each objective lens. However, it is difficult to achieve zero amount of decentering. It is also difficult to make the amounts of decentering the same. Therefore, even if the same imaging lens is used in the first observation optical system and the second observation optical system, and a single image sensor is used to capture the optical images, the first optical image and the second optical image are formed at different positions on the imaging plane. As a result, a misalignment occurs between the position of the specimen in the fluorescence image and the position of the specimen in the bright-field image.

[0142] Therefore, prior to acquiring the fluorescent image group and the bright-field image group, images of, for example, fluorescent beads are acquired using the first and second objective lenses.

[0143] The first optical system acquires a fluorescent image using a first objective lens. The second optical system acquires a bright-field image using a second objective lens. The amount of deviation can be determined by comparing the positions of the fluorescent beads in the fluorescent image with the positions of the fluorescent beads in the bright-field image.

[0144] By using this amount of misalignment to align the fluorescent image and the bright-field image before calculating the estimated image, it is possible to eliminate the misalignment between the fluorescent image and the bright-field image. This allows the point spread function corresponding to the area of ​​the fluorescent image to be calculated with high accuracy. As a result, the image can be restored with high accuracy.

[0145] As described above, fluorescent light and white light have different wavelength bands, but their light intensities also differ. Therefore, an image sensor suitable for fluorescent light (hereinafter referred to as the "first image sensor") and an image sensor suitable for white light (hereinafter referred to as the "second image sensor") may be used. In this case, the first image sensor captures a first optical image, and the second image sensor captures a second optical image.

[0146] To use the first and second image sensors, an optical path splitter is placed between the imaging lens and the image plane to form a first optical path toward the first image sensor and a second optical path toward the second image sensor. In the first optical path, a first optical image is formed on the imaging surface of the first image sensor. In the second optical path, a second optical image is formed on the imaging surface of the second image sensor.

[0147] Because the magnification of the objective lens and the magnification of the imaging lens are the same, the size of the first optical image and the size of the second optical image are the same. However, the first optical image and the second optical image are formed at different positions and captured by different image sensors. Therefore, it is difficult to match the position of the first optical image on the image sensor of the first image sensor with the position of the second optical image on the image sensor of the second image sensor. As a result, a misalignment occurs between the position of the specimen in the fluorescent image and the position of the specimen in the bright-field image.

[0148] Alternatively, an optical path splitter may be disposed between the objective lens and the imaging lens to form a first optical path toward the first image sensor and a second optical path toward the second image sensor. In this case, the first and second optical images are formed at different positions and captured by different image sensors. Furthermore, different imaging lenses are used for the first and second optical paths.

[0149] Therefore, it is difficult to match the position of the first optical image on the imaging plane of the first imaging element with the position of the second optical image on the imaging plane of the second imaging element, resulting in a misalignment between the position of the specimen in the fluorescent image and the position of the specimen in the bright-field image.

[0150] As described above, when two image sensors are used, a misalignment occurs between the position of the specimen in the fluorescent image and the position of the specimen in the bright-field image. However, the specimen image acquisition device of this embodiment can align the fluorescent image and the bright-field image before calculating an estimated image. This makes it possible to calculate the point spread function corresponding to the area of ​​the fluorescent image with high accuracy. As a result, the image can be restored with high accuracy.

[0151] In the specimen image acquisition device of this embodiment, when generating multiple fluorescent images, the objective lens shared with the specimen is changed at a first interval, and when generating multiple bright-field images, the objective lens shared with the specimen is changed at a second interval, and it is preferable that the following conditional expression (2) is satisfied. △Z1<△Z2 (2) where: △Z1 is the first interval, △Z2 is the second interval, is.

[0152] By satisfying conditional expression (2), the number of bright-field images generated is less than the number of fluorescent images generated. This reduces the time required to generate bright-field images. Furthermore, since an increase in the number of bright-field images generated can be suppressed, the time required to calculate the refractive index distribution can be reduced.

[0153] When conditional expression (2) is satisfied, the number of generated bright-field images differs from the number of generated fluorescent images. However, if the difference between the number of generated bright-field images and the number of generated fluorescent images is small, a significant decrease in the accuracy of the estimated final refractive index distribution can be prevented. Therefore, the image can be restored with high accuracy.

[0154] If a common objective lens is not used, the specimen and the first objective lens may be moved at a first interval, and the specimen and the second objective lens may be moved at a second interval. In this case, it is also preferable that conditional expression (2) be satisfied.

[0155] The specimen image acquisition device of this embodiment has a light source that irradiates illumination light toward the second optical system, and it is preferable that the specimen in a container filled with liquid is illuminated by the illumination light and that the following conditional expression (3) is satisfied: △Z1<△Z2<2.5×n×λc / NA 2 (3) where: n is the refractive index of the liquid, λc is the centroid wavelength of the illumination light, NA is the numerical aperture of the second objective lens, is. The centroid wavelength is expressed by the following formula:

number

[0156] If the lower limit of conditional expression (3) is not met, the number of generated bright-field images will be greater than the number of generated fluorescent images. This increases the time required to generate bright-field images. Furthermore, the increased number of generated bright-field images increases the time required to calculate the refractive index distribution.

[0157] If the upper limit of conditional expression (3) is exceeded, the number of generated bright-field images will be too small. In this case, there will be too many insufficient bright-field images corresponding to the fluorescent images. Because the final refractive index distribution is calculated from the bright-field images, there will be a shortage of the final refractive index distributions in the same number as the shortage of bright-field images.

[0158] As described above, the missing final refractive index distribution can be estimated using multiple bright-field images. However, if the number of generated bright-field images is too small, the estimation accuracy of the missing final refractive index distribution deteriorates. Therefore, it becomes difficult to calculate the final refractive index distribution with high accuracy. As a result, it becomes difficult to restore the image with high accuracy.

[0159] It is preferable to satisfy the following conditional expression (3') instead of conditional expression (3). △Z1<△Z2<1.25×n×λc / NA2 (3')

[0160] The specimen image acquisition device of this embodiment preferably has a light source that irradiates illumination light toward the second optical system, and the processor divides the bright-field image into a plurality of areas to satisfy the following conditional expression (4). 2 <dxsplit / D<200 (4) where: D = 1.22 × λc / (NA + NAill), dxsplit is the length of one side of the area in the bright-field image, λc is the centroid wavelength of the illumination light, NA is the numerical aperture of the second objective lens, NAill is the numerical aperture of the illumination optical system, is.

[0161] As described above, the restoration process uses a point spread function for each pixel in the area of ​​the fluorescent image. The point spread function is preferably calculated for each pixel. However, in the specimen image acquisition device of this embodiment, the point spread function is only calculated at the center of the virtual area.

[0162] Although this point spread function should be used only for pixels located in the center of the area of ​​the fluorescent image, the restoration process uses this point spread function for each pixel in the area of ​​the fluorescent image.

[0163] The area of ​​the fluorescence image is formed by a plurality of pixels. The plurality of pixels can be divided into central pixels and peripheral pixels. The central pixels are pixels located in the center of the area of ​​the fluorescence image. The peripheral pixels are pixels located around the central pixels.

[0164] The peripheral pixels can be divided into proximal pixels and distal pixels: proximal pixels are pixels located closer to the central pixel, and distal pixels are pixels located further away from the central pixel.

[0165] The shape of the point spread function corresponding to the peripheral pixels is different from the shape of the point spread function corresponding to the central pixel, and the difference in the shape of the point spread function increases from the proximal pixel to the distal pixel.

[0166] Among the peripheral pixels, the difference in the shape of the point spread function is small for the proximal pixels. Therefore, even if the point spread function corresponding to the central pixel is used for the proximal pixels, the deterioration in the accuracy of the restoration for the proximal pixels is slight. In contrast, the difference in the shape of the point spread function is large for the distal pixels. Therefore, if the point spread function corresponding to the central pixel is used for the distal pixels, the deterioration in the accuracy of the restoration for the distal pixels becomes significant.

[0167] If the lower limit of conditional expression (4) is not met, the size of the areas in the bright-field image will be too small. As described above, the fluorescent image is divided into multiple areas. If the number of areas in the bright-field image and the number of areas in the fluorescent image are the same, the size of the areas in the fluorescent image will be too small.

[0168] As the size of an area in a fluorescent image becomes smaller, the number of areas in the fluorescent image increases. As described above, the restoration process is performed for each area of ​​the fluorescent image. Therefore, as the number of areas in the fluorescent image increases, the restoration process takes longer.

[0169] Furthermore, if the area size in the bright-field image becomes too small, the area size in the fluorescence image will also become too small. When the area size in the fluorescence image is small, the proportion of proximal pixels in the peripheral pixels increases. Therefore, even if the point spread function corresponding to the central pixel is used for the peripheral pixels, the deterioration in the accuracy of the restoration of the peripheral pixels is slight.

[0170] Using the case where the proportion of distal pixels among the peripheral pixels is high as the standard, a comparison can be made from the perspective of improvement in restoration accuracy. Compared to when the proportion of distal pixels among the peripheral pixels is high, when the proportion of proximal pixels among the peripheral pixels is high, restoration accuracy at the peripheral pixels is significantly improved. However, the smaller the area size in the fluorescence image, the smaller the difference in the shape of the point spread function. Therefore, even if the area size in the fluorescence image is made smaller, restoration accuracy at the peripheral pixels does not improve significantly.

[0171] Furthermore, D represents the resolution in the bright-field image. A bright-field image is an XY image, and an XY image is an image in an XY cross section. Therefore, D represents the resolution in the XY cross section. The refractive index distribution is calculated using the bright-field image. Furthermore, the point spread function is calculated using the refractive index distribution. Therefore, D represents the resolution in the refractive index distribution and the resolution in the point spread function.

[0172] As mentioned above, the difference in the shape of the point spread function can be reduced by reducing the size of the area in the fluorescence image, but reducing the difference in the shape of the point spread function does not improve the accuracy of the restoration beyond the resolution of the point spread function.

[0173] In this way, if the area size in the fluorescent image becomes too small, the restoration process time will increase without any significant improvement in restoration accuracy. Therefore, it is not preferable to go below the lower limit of conditional expression (4).

[0174] If the upper limit of conditional expression (4) is exceeded, the size of the area in the bright-field image becomes too large. In this case, the proportion of the distal pixels among the peripheral pixels becomes too large. The shape of the point spread function corresponding to the distal pixels is significantly different from the shape of the point spread function corresponding to the central pixel. Therefore, the deterioration of the restoration accuracy at the peripheral pixels becomes significant. As a result, the restoration accuracy decreases throughout the entire restored image.

[0175] By satisfying conditional expression (4), it is possible to maintain the accuracy of the restoration and shorten the time required for the restoration process.

[0176] It is preferable to satisfy the following conditional expression (4') instead of conditional expression (4). 10 <dxsplit / D<75 (4’)

[0177] In the specimen image acquisition device of this embodiment and the specimen image generation device of this embodiment, it is preferable that the processor sets a point light source that radiates virtual rays in the virtual area, and calculates the point spread function using a first wavefront whose wave source is the point light source.

[0178] A point spread function is an optical image of a point light source formed by an optical system. In the specimen image acquisition device of this embodiment, the point spread function is calculated during the image restoration process. The final refractive index distribution is used in the calculation of the point spread function. Since the final refractive index distribution is set in a virtual area, a point light source is set in the virtual area. This point light source is a virtual light source.

[0179] Light can also be expressed as a wavefront. A point spread function can be considered to be formed by a wavefront emitted from a point light source propagating through object space, the optical system, and the image space. Therefore, a wavefront whose wave source is a virtual light source is set, and this wavefront is called the first wavefront. Using this first wavefront, the point spread function can be calculated.

[0180] In the specimen image acquisition device of this embodiment, it is preferable that the processor calculates a second wavefront emerging from the estimated specimen using the first wavefront and the final refractive index distribution included in the range through which the virtual ray travels, calculates a third wavefront at the focal plane of the virtual observation optical system using the second wavefront, calculates an intensity distribution corresponding to the third wavefront, and calculates a point spread function using the intensity distribution corresponding to the third wavefront.

[0181] The virtual area group is composed of virtual areas included in the range in which the virtual ray travels. The number of virtual areas included in the virtual area group is equal to or greater than zero. If the number of virtual areas included in the virtual area group is not zero, the first wavefront propagates through some virtual areas and exits from the estimated sample.

[0182] The wavefront emitted from the estimated sample is called the second wavefront. The first wavefront is the wavefront before propagating through the virtual area group, while the second wavefront is the wavefront after propagating through the virtual area group. Therefore, the shape of the wavefront differs between the first wavefront and the second wavefront. The second wavefront can be calculated based on the first wavefront and the final refractive index distribution of the virtual area group.

[0183] Since the second wavefront is incident on the virtual observation optical system, a point spread function is formed based on the second wavefront. The point spread function is conjugate with the wavefront located on the focal plane of the virtual observation optical system. However, since the wavefront located on the focal plane of the virtual observation optical system is the first wavefront, the second wavefront is not located on the focal plane of the virtual observation optical system.

[0184] The new wavefront at the focal plane of the virtual observation optical system is called the third wavefront. The third wavefront can be calculated by propagating the second wavefront to the focal plane of the virtual observation optical system. There is no estimated specimen between the position of the second wavefront and the third wavefront. Therefore, the final refractive index distribution in the virtual area is not used in calculating the third wavefront.

[0185] The third wavefront is located on the focal plane of the virtual observation optical system. Therefore, the point spread function is calculated based on the third wavefront. The point spread function represents the intensity distribution of light. Therefore, the intensity distribution corresponding to the third wavefront is calculated. Then, the point spread function can be calculated using the calculated intensity distribution.

[0186] In the specimen image acquisition device of this embodiment, it is preferable that the processor divides the virtual area into a plurality of micro areas, sets a final refractive index distribution for each micro area, calculates a point spread function corresponding to the micro area as a micro point spread function using the final refractive index distribution of the micro area, and calculates a point spread function using the micro point spread function.

[0187] If a detailed final refractive index distribution can be used, the point spread function can be calculated accurately. The more virtual areas there are or the smaller the area of ​​each virtual area, the more detailed the final refractive index distribution can be set. Therefore, one virtual area is divided into multiple minute areas. In this case, the final refractive index distribution can be set for each minute area, allowing for more detailed setting of the final refractive index distribution.

[0188] If the final refractive index distribution can be set for each minute area, the point spread function can be calculated for each minute area. In this case, the point spread function of one virtual area can be calculated using the point spread function of the minute area. Therefore, the point spread function can be calculated accurately. As a result, the image can be restored with high accuracy.

[0189] The specimen image acquisition device of this embodiment preferably has a light source that irradiates excitation light toward the first optical system, and the processor calculates the excitation light intensity at the position of the point light source, calculates the fluorescence intensity distribution using the intensity distribution corresponding to the third wavefront and the excitation light intensity, and calculates the point spread function using the fluorescence intensity distribution.

[0190] 5 is a diagram showing another specimen image acquisition device according to this embodiment. The same components as those in FIG. 1 are assigned the same numbers, and their explanations will be omitted.

[0191] The specimen image acquisition device 90 includes a first optical system 91, a second optical system 3, a moving mechanism 4, and an image generation unit 5. A first light source 92 is disposed to illuminate the specimen 6. An optical image of the specimen 6 is detected by a photodetector 93. The photodetector 93 may be a PMT (photomultiplier tube), an APD (avalanche photodiode), or a SiPM (silicon photomultiplier).

[0192] The specimen image acquisition device 90 uses the same second optical system 3 as the specimen image acquisition device 1. Therefore, only the first optical system 91 will be described. The second optical system 3 is indicated by a dotted line for reference.

[0193] For example, the optical system of a confocal laser scanning microscope or the optical system of a multiphoton excitation laser scanning microscope can be used for the first optical system 91. Here, it is assumed that the optical system of a confocal laser scanning microscope is used for the first optical system 91.

[0194] The first optical system 91 includes a first observation optical system 100 and a first illumination optical system 101. The first observation optical system 100 is located on the optical path from the specimen 6 to the first optical image IM1'.

[0195] In the specimen image acquisition device 1, the specimen 6 is illuminated at a surface, so a planar optical image is formed. In contrast, in the specimen image acquisition device 90, the specimen 6 is illuminated at a point, as will be described later, so a point-like optical image is formed. For convenience, the first optical image IM1' is represented by an arrow, just like the first optical image IM1.

[0196] The first observation optical system 100 and the first illumination optical system 101 have a common optical system that includes a dichroic mirror 110, a mirror 111, a mirror 112, a pupil projection lens 113, an imaging lens 114, a mirror 115, and an objective lens .

[0197] In Figure 5, a dashed line represents the illumination light. In a common optical system, the illumination light and the imaging light travel along the same optical path. Although a solid line representing the imaging light is not drawn, the imaging light travels along an optical path from the specimen 6 to the dichroic mirror 110.

[0198] The first observation optical system 100 includes a common optical system, a confocal lens 117, and a confocal pinhole 118. The first illumination optical system 101 includes the common optical system, and may further include a beam expander. The beam expander is disposed between the first light source 92 and the dichroic mirror 110.

[0199] The first light source 92 is a light source that irradiates excitation light toward the first optical system 91. A laser is used as the first light source 92. Since laser light is parallel light, the first light source 92 is a point light source. The laser light is used as excitation light.

[0200] The laser light emitted from the first light source 92 is reflected by the dichroic mirror 110. The laser light that has traveled through the common optical system reaches the objective lens 116. Because the laser light is parallel light, the objective lens 116 forms a spot of excitation light at the focal position.

[0201] In the specimen 6, fluorescence is generated from the position of the excitation light spot. The fluorescence travels through a common optical system, passes through a dichroic mirror 110, and is then focused by a confocal lens 117. A confocal pinhole 118 is disposed at the focusing position, so a fluorescent spot is formed on the confocal pinhole 118. The fluorescence that has passed through the confocal pinhole 118 is detected by a photodetector 93.

[0202] Since the specimen 6 is illuminated at a point, only an optical image at one point on the specimen 6 can be obtained. In the specimen image acquisition device 90, mirror 111 is deflected in the X-axis direction, and mirror 112 is deflected in the Y-axis direction. The deflection of the two mirrors scans the specimen 6 two-dimensionally. As a result, a planar fluorescent image is finally obtained, similar to the fluorescent image in the specimen image acquisition device 1.

[0203] The specimen image acquisition device 90 generates a plurality of bright field images and a plurality of fluorescent images, similar to the specimen image acquisition device 1. Therefore, the image can be restored by using the point spread function.

[0204] In the specimen image acquisition device 90, the first optical image and the second optical image are formed at different positions. Furthermore, the photodetector 93 is used to capture the first optical image, and the image sensor 10 is used to capture the second optical image. Therefore, a discrepancy occurs between the position of the specimen in the fluorescent image and the position of the specimen in the bright-field image.

[0205] However, as described above, the specimen image acquisition device 90 can also align the fluorescent image and the bright-field image before calculating the estimated image. This allows the point spread function corresponding to the area of ​​the fluorescent image to be calculated with high accuracy. As a result, the image can be restored with high accuracy.

[0206] Fluorescence is generated when a fluorescent dye is irradiated with excitation light. The intensity of the excitation light affects the intensity distribution of the fluorescence. Therefore, the point spread function is calculated using the intensity of the excitation light and the intensity distribution of the fluorescence. As described above, the intensity distribution corresponding to the third wavefront can be used to calculate the point spread function. Therefore, the fluorescence intensity distribution used to calculate the point spread function can be calculated using the intensity distribution corresponding to the third wavefront and the light intensity of the excitation light.

[0207] The point spread function used for image restoration is related to the following characteristics (a), (b), (c), and (d). The focal position is the focal position of the objective lens. The conjugate position is the position conjugate to the focal position. (a) The specimen is illuminated with a spot of excitation light formed at the focal position. (b) Fluorescence is emitted from the position of the excitation light spot. (c) A fluorescent spot is formed at the conjugated position. (d) Fluorescence is detected through a confocal pinhole placed at the conjugate position.

[0208]

number

[0209] Equation (5) can be better understood by referring to the following documents, for example: Shinichi Hayashi “Resolution doubling using confocal microscopy via analogy with structured illumination microscopy”, Japanese Journal of Applied Physics 55, 082501 (2016)

[0210] The point spread function of excitation light will now be described. The spot of excitation light is formed by an illumination optical system. However, for simplicity, it is assumed that the spot of excitation light is formed by an objective lens.

[0211] Because the light source is a point light source, an optical image of the point light source is formed at the focal position. This optical image is a spot of excitation light and also represents the point spread function of the excitation light. If the focal position is located inside the specimen, the spot of excitation light is formed inside the specimen. Therefore, the point spread function of the excitation light at the focal position must be calculated taking into account the refractive index distribution of the specimen.

[0212] In terms of wavefronts, the wavefront of the excitation light propagates from the objective lens towards the specimen and reaches the specimen's surface. The wavefront incident on the specimen surface propagates within the specimen and reaches the position of the excitation light spot. The point spread function of the excitation light can be calculated from the wavefront at the position of the excitation light spot.

[0213] To obtain the wavefront at the position of the excitation light spot, it is necessary to obtain the wavefront just before it enters the sample and the wavefront propagating within the sample. Since the wavefront is obtained by calculation, an estimated sample is used as the sample.

[0214] The estimated specimen can be represented as a structure in which thin layers are stacked with a spacing Δz. Let the total number of thin layers be L. The first layer is located farthest from the objective lens, and the Lth layer is located closest to the objective lens.

[0215] The wavefront immediately before it enters the specimen will now be described. Naturally, the wavefront immediately before it enters the specimen (hereinafter referred to as the "incident wavefront") is a wavefront that has not passed through the specimen. Furthermore, the incident wavefront is a wavefront that ultimately reaches the position of the excitation light spot. Therefore, in the absence of a specimen, a wavefront is set at the position of the excitation light spot, and the wavefront when propagated to the position of the incident wavefront is determined.

[0216] P ex (x p ,y p ,z p ), z Pin , U p (x,y) and U edge This will be explained using (x,y). P ex (x p ,y p ,z p ) represents the position of the excitation light spot. z Pin represents the Z position of the incident wavefront. U p (x, y) is the position P when no sample exists. ex (x p ,y p ,z p ) and represents a wavefront set in a plane perpendicular to the optical axis. U edge (x,y) is the wavefront U p (x,y) is the Z position z Pin The figure shows the wavefront propagating to the

[0217] When no sample exists, z = z p Wavefront U at p (x, y) is expressed by the following equation (6).

number

[0218] Wavefront U edge (x, y) is expressed by the following equation (7).

number

[0219] Wavefront U p (x,y) is the wavefront when no sample is present, so the wavefront U edge (x, y) is also a wavefront when no specimen exists. As mentioned above, the incident wavefront is also a wavefront when no specimen exists. Therefore, the wavefront U edge (x,y) coincides with the incident wavefront.

[0220] Also, the wavefront U edge (x,y) is z=z p In terms of z=z Pin The wavefront when propagating on the plane z=z Pin is the position of the incident wavefront. Therefore, the wavefront position is also edge (x,y) coincides with the incident wavefront.

[0221] However, the incident wavefront is the wavefront incident on the specimen, whereas the wavefront U edge Since (x, y) travels away from the specimen, it does not enter the specimen. Therefore, in the direction of wavefront propagation, the wavefront U edge (x,y) does not coincide with the incident wavefront.

[0222] What is needed is a wavefront that travels towards the sample. edge From (x,y), the wavefront U * edge Find (x,y). U * represents the complex conjugate of U. U * differs from U only in the sign of the imaginary part.

[0223] Wavefront U * edge (x,y) is the wavefront U edge This is a wavefront obtained by inverting (x, y) in the Z-axis direction. * edge (x, y) is the wavefront when no sample is present, and not only does it match the incident wavefront in terms of the wavefront position, but also in terms of the wavefront direction. Therefore, the wavefront U * edge (x,y) can be considered as the incident wavefront.

[0224] The wavefront propagating within the sample will now be described. A calculation method called BPM (Beam Propagation method) is used to calculate the wavefront propagating within the sample. BPM is disclosed in, for example, the following document: MD Feit, JA Fleck Jr, “Light propagation in graded-index optical fibers”, Appl. Opt. 17, 3990-3998 (1978)

[0225] In BPM, a specimen made up of stacked thin layers is used as the specimen. For each thin layer, the wavefront emerging from the thin layer is calculated based on the incident wavefront and amplitude transmittance distribution. The wavefront emerging from the thin layer propagates to the next thin layer. Therefore, the wavefront emerging from the thin layer is the wavefront that becomes the source of the wavefront incident on the next thin layer. By performing this calculation for each thin layer, the wavefront propagating within the specimen can be calculated.

[0226] As mentioned above, the estimated specimen is a specimen made up of stacked thin layers. The first layer is located farthest from the objective lens, and the Lth layer is located closest to the objective lens. Therefore, the incident wavefront propagates through the specimen from the Lth layer side to the first layer side.

[0227] Each thin layer has a final refractive index profile. Therefore, the final refractive index profile can be used to set the amplitude transmittance profile in the thin layer. The amplitude transmittance profile T in the m-th layer m (x, y) is expressed by the following equation (8).

number

[0228] For example, if the specimen is a cell clump, the cell clump is held in a container together with phosphate-buffered saline (PBS). Therefore, PBS corresponds to the medium. The refractive index of PBS is 1.333, the same as that of water, and is constant regardless of position within the container.

[0229] Wavefront G emerging from the mth layer m (x, y) is expressed by the following equation (9).

number

[0230] As mentioned above, the wavefront propagates through the specimen from the Lth layer side to the 1st layer side. m (x, y) is the wavefront emerging from the mth layer, but it is also the wavefront that is the source of the wavefront that enters the (m-1)th layer. Since the distance between two adjacent thin layers is Δz, the wavefront G m (x, y) propagates by Δz and reaches the (m-1)th layer. The wavefront that reaches the (m-1)th layer is the wavefront that is incident on the (m-1)th layer.

[0231] The wavefront propagating from one thin layer to another can be determined based on the angular spectrum method, which involves decomposing a wavefront into multiple plane waves using a Fourier transform, shifting each plane wave by a distance Δz, and then combining the multiple plane waves into a single wavefront using an inverse Fourier transform.

[0232] When the angular spectrum method is used, the wavefront U incident on the (m-1)th layer m-1 (x, y) is expressed by the following equation (10).

number

[0233] Wavefront U m-1 (x, y) is the wavefront incident on the (m-1)th layer. Therefore, the wavefront G emerging from the (m-1)th layer m-1 To find (x, y), use U in equation (9). m U instead of (x,y) m-1 Just use (x,y).

[0234] In equation (10), the wavefront incident on the (m-1)th layer is calculated using the wavefront exiting from the mth layer. When trying to calculate the wavefront incident on the Lth layer using equation (10), the wavefront on the left side is U L (x, y), the wavefront on the right side is G L+1 (x,y). Wavefront G L+1(x, y) represents the wavefront emerging from the (L+1)th layer.

[0235] However, as mentioned above, the Lth layer is located closest to the objective lens. Therefore, there is no thin layer closer to the objective lens than the Lth layer. In other words, there is no (L+1)th layer. In this case, the wavefront G L+1 Since (x,y) does not exist, the wavefront U L (x, y) cannot be calculated from equation (10).

[0236] Wavefront U L Since (x,y) is located just in front of the specimen, it can be considered as the incident wavefront. As mentioned above, the wavefront U * edge (x,y) can also be considered as an incident wavefront. Therefore, the wavefront U L (x,y) as U * edge Just use (x,y).

[0237] As shown in the following equation (11), U * edge By using (x, y), the wavefront G L (x,y) can be found.

number

[0238] wavefront G L Once (x, y) is determined, the wavefronts incident on and emerging from the thin layers located closer to the first layer than the Lth layer can be determined iteratively based on equations (9) and (10). As a result, the wavefronts can be determined sequentially from the Lth layer side toward the first layer side.

[0239] Assume that a spot of excitation light is generated on the first layer side of the nth layer. Also, the wavefront emitted from the nth layer is called wavefront G n (x,y). Wavefront G n (x, y) can be calculated based on equations (9) and (10).

[0240] Furthermore, the wavefront G1(x, y) can be calculated based on equations (9) and (10). z (x, y), the wavefront G z The absolute values ​​of (x, y) are squared and aligned along the optical axis to obtain the point spread function (PSF) of the excitation light. ex (r) can be found.

[0241] The fluorescent light point spread function will now be described. The fluorescent light spot is formed by the observation optical system. However, for simplicity, it is assumed that the fluorescent light spot is formed by the objective lens.

[0242] When a fluorescent dye is present at the position of the excitation light spot, fluorescence is emitted from the excitation light spot. The fluorescence is collected by the objective lens, and an optical image of the fluorescence is formed at the conjugate position. This optical image is the fluorescence spot and also represents the fluorescence point spread function.

[0243] When the focal position is located inside the specimen, the excitation light spot is formed inside the specimen, and the fluorescence also occurs inside the specimen. Therefore, the fluorescence point spread function must be calculated taking into account the refractive index distribution of the specimen.

[0244] In terms of wavefronts, the wavefront of the fluorescence propagates through the specimen towards the objective lens and reaches the specimen's surface. The wavefront emitted from the specimen's surface propagates through the space between the specimen and the objective lens and enters the objective lens. The wavefront that enters the objective lens reaches the conjugate position, i.e., the position of the fluorescence spot. A pinhole is placed at the conjugate position. Therefore, the point spread function can be calculated from the wavefront and pinhole at the position of the fluorescence spot.

[0245] In order to obtain the wavefront at the position of the fluorescent spot, it is necessary to obtain the wavefront emitted from the sample and the wavefront at the focal position. Since the wavefront is obtained by calculation, an estimated sample is used as the sample.

[0246] We will now explain the wavefront emitted from the specimen. As mentioned above, a spot of excitation light is generated on the first layer side of the nth layer. Fluorescence is generated from the excitation light spot, and so the fluorescence is generated within the specimen. The wavefront of the fluorescence propagates within the specimen from the first layer side toward the Lth layer side.

[0247] As mentioned above, the wavefront U m (x, y) is the wavefront incident on the mth layer, and the wavefront U m The original wavefront of (x, y) is the wavefront G emerging from the (m-1)th layer. m-1 (x, y). The distance between two adjacent thin layers is Δz, so the wavefront G m-1 (x, y) propagates by Δz and reaches the mth layer. The wavefront that reaches the mth layer is the wavefront that is incident on the mth layer.

[0248] When the angular spectrum method is used, the wavefront U incident on the mth layer m (x, y) is expressed by the following equation (12).

number

[0249] Wavefront U m (x, y) is the wavefront incident on the mth layer. Therefore, the wavefront G emerging from the mth layer m (x, y) is expressed by the following equations (13) and (14).

number

number

[0250] As described above, a spot of excitation light is generated on the first layer side of the nth layer. Fluorescence is generated from the spot of excitation light, so fluorescence is generated on the first layer side of the nth layer.

[0251] In equation (11), the wavefront incident on the mth layer is calculated using the wavefront emerging from the (m-1)th layer. When trying to calculate the wavefront incident on the nth layer using equation (11), the wavefront on the left side is U n (x, y), the wavefront on the right side is G n-1 (x,y). Wavefront G n-1 (x, y) represents the wavefront emerging from the (n-1)th layer.

[0252] A thin layer exists closer to the first layer than the nth layer. Therefore, the (n-1)th layer exists. However, the fluorescence emission point is located closer to the Lth layer than the (n-1)th layer. In this case, the wavefront does not emerge from the (n-1)th layer toward the nth layer, so the wavefront G n-1 (x, y) does not exist, so the wavefront incident on the nth layer cannot be calculated from equation (11).

[0253] The wavefront incident on the nth layer is the wavefront emitted from the fluorescent light emission point. If the fluorescent light emission point is considered as a point light source, the wavefront U emitted from the fluorescent light emission point is em (x,y) is the position P ex (x p ,y p ) and 0 elsewhere. Therefore, U n (x,y) as U em Just use (x,y).

[0254] U em By using (x, y), the wavefront G emerging from the nth layer n (x, y) is expressed by the following equation (15).

number

[0255] wavefront G nOnce (x, y) is determined, the wavefronts incident on and emerging from the thin layers located closer to the Lth layer than the nth layer can be determined iteratively based on equations (13) and (14). As a result, the wavefronts can be determined sequentially from the first layer toward the Lth layer.

[0256] As described above, the Lth layer is located closest to the objective lens. L (x,y) is the wavefront emerging from the specimen.

[0257] The wavefront at the in-focus position will now be described. As described above, the point spread function can be obtained from the wavefront at the position of the fluorescent spot. The fluorescent spot is formed at the conjugate position. The conjugate position is a position conjugate to the in-focus position. Therefore, in order to obtain the wavefront at the conjugate position, the wavefront at the in-focus position is required.

[0258] P OB (x,y,z), P FL (x,y,z), P L (x,y,z), P CON (x,y,z), U * L (x,y,z), U OB (x,y,z) and U PH The following explanation will be given using (x, y, z). In the following explanation, the z coordinate will be omitted as appropriate. P OB (x, y, z) represents the focal position of the objective lens. OB Z position z OB is the focal plane of the objective lens. P FL (x, y, z) represents the position where fluorescence is generated. z L is the wavefront G L It represents the Z position of (x,y). P CON (x,y,z) is the position P OB It represents the conjugate position of (x,y,z). CON Z position z CON is a plane conjugate to the focal plane of the objective lens. G * L (x,y) is the wavefront G L It represents the complex conjugate of (x,y). U OB (x,y) is the wavefront G * L (x,y) is position P OB represents the wavefront propagating to the z-position of U PH (x,y) is the wavefront G OB (x,y) is position P CON represents the wavefront propagating to the z-position of

[0259] P FL (x,y,z) is the position where fluorescence occurs. L is the wavefront G L Since the position is (x, y), it is the position where the fluorescence is emitted from the specimen. Therefore, regarding the position of the wavefront, L (x,y) is the wavefront U OB Does not match (x,y).

[0260] Wavefront U OB To find (x,y), we use the wavefront G L (x,y) to Z position z OB The wavefront G L The direction of travel of (x, y) is the position P OB The wavefront G is in the direction away from L (x,y) to Z position z OB In order to propagate to L The direction of travel of (x,y) needs to be reversed.

[0261] Therefore, the wavefront U L From (x,y), the wavefront U * L (x,y) is calculated. Wavefront U * L (x,y) is the wavefront U L This is the wavefront obtained by inverting (x, y) in the Z-axis direction. OB (x,y) is the wavefront U * L (x,y) to Z position z OBThe wavefront U OB (x, y) is expressed by the following equation (16).

number

[0262] position P CON (x,y) is the position P OB Since it is conjugate with (x,y), the wavefront U OB (x,y) to U PH (x,y) can be calculated. PH (x, y) is expressed by the following equation (17).

number

[0263] TIFF2026019571000016.tif29170

[0264] TIFF2026019571000017.tif24170

[0265] TIFF2026019571000018.tif29170

[0266] 6A and 6B are diagrams showing a degraded image and a restored image, respectively. Fig. 6A is a diagram showing the degraded image, and Fig. 6B is a diagram showing the restored image.

[0267] The aerial image is an XZ image. The right edge of the image represents the top surface of the specimen, and the left edge of the image represents the bottom surface of the specimen. In the entire range from the top surface to the bottom surface of the specimen, the image quality of the restored image is higher than that of the degraded image.

[0268] In the specimen image acquisition device of this embodiment, it is preferable that the processor calculates the excitation light intensity using the refractive index distribution at the excitation light wavelength.

[0269] TIFF2026019571000019.tif18170

[0270] TIFF2026019571000020.tif24170

[0271] In the specimen image acquisition device and specimen image generation device of this embodiment, before calculating the point spread spectrum, the processor preferably calculates a hypothetical point spread spectrum for each area of ​​the fluorescence image, sets the intensity peak value of the point spread spectrum when an estimated specimen is not present as a reference value, sets the intensity peak value of the hypothetical point spread spectrum as a hypothetical intensity peak value, compares the hypothetical intensity peak value with the reference value, and divides each virtual area into a target area and a non-target area, dividing the target area into areas smaller than the non-target areas. Here, the target area is an area whose hypothetical intensity peak value is less than 1 / 5 of the reference value, and the non-target area is an area whose hypothetical intensity peak value is 1 / 5 or more of the reference value.

[0272] When the specimen is a cell cluster, light scattering occurs within the cell cluster. The greater the scattering, the greater the change in the shape of the point spread function. As described above, the point spread function is calculated at the center of the virtual area. Then, in the area of ​​the fluorescence image corresponding to the virtual area, the same point spread function is used for each pixel in the restoration process.

[0273] If a point spread function with a significantly changed shape is used for peripheral pixels, the accuracy of restoration at distant pixels will decrease significantly. If the same point spread function is used for nearby pixels, the accuracy of restoration at the nearby pixels will decrease only slightly. Therefore, point spread functions with a significantly changed shape are used only for nearby pixels.

[0274] To use the point spread function only for the proximal pixels, the size of the area of ​​the fluorescence image can be reduced. The size of the area of ​​the fluorescence image is equal to the size of the virtual area. Therefore, the decision on whether to reduce the size of the area of ​​the fluorescence image can be replaced with the decision on whether to reduce the size of the virtual area.

[0275] The decision on whether to reduce the size of the virtual area can be made based on the shape of the point spread function. In a point spread function, the greater the change in shape, the smaller the intensity peak value. Therefore, the decision can be made based on the intensity peak value of the point spread function.

[0276] In order to make the judgment, a reference value is required. In the specimen image acquisition device of this embodiment and the specimen image acquisition device of this embodiment, the intensity peak value of the point spread function when no estimated specimen is present is used as the reference value.

[0277] Before calculating the point spread function, the processor calculates a virtual point spread function for each area of ​​the fluorescence image. The processor then sets the intensity peak value of the virtual point spread function as a virtual peak intensity value. The processor then compares the virtual peak intensity value with a reference value and divides each virtual area into a target area and a non-target area.

[0278] The target area is an area where the tentative intensity peak value is less than 1 / 5 of the reference value, and the non-target area is an area where the tentative intensity peak value is 1 / 5 or more of the reference value.

[0279] Compared with the non-target area, the tentative intensity peak value is lower in the target area, which means that the shape of the point spread function is significantly changed in the target area. As mentioned above, if a point spread function with a significantly changed shape is used for peripheral pixels, the accuracy of the recovery for distant pixels will be significantly reduced.

[0280] Therefore, the target area is divided into smaller areas than the non-target area. By reducing the size of the target area, the proportion of proximal pixels in the peripheral pixels can be increased. Therefore, even if the point spread function corresponding to the central pixel is used, the deterioration of the accuracy of restoration of peripheral pixels can be suppressed. As a result, the image can be restored with high accuracy. [Explanation of symbols]

[0281] 1. Specimen image acquisition device 2 First optical system 3 Second optical system 4 Moving mechanism 5 Image Generation Unit 6 specimens 7 Stages 8 1st light source 9 Second light source 10. Image sensor 20 First observation optical system 21 1st illumination optical system 22 First objective lens 23 Absorption Filter 24 Imaging lens 25 Collector lens 26 Relay Lens 27 Relay Lens 28 Excitation light filter 29 Dichroic Mirror 30 Second observation optical system 31 Second illumination optical system 32 Second objective lens 33 Imaging lens 34 Collector lens 35 Aperture diaphragm 36 Condenser Lens 51 memory 52 processors 53 Input / output section 60 light source 61 PCB 62 Light-emitting part 70 Opening material 71 Transparent part 72 Light blocking section 80 Opening material 81 1st transparent part 82 2nd transparent part 83 Light blocking section 90 Specimen image acquisition device 91 1st optical system 92 1st light source 93 Photodetector 100 first observation optical system 101 1st illumination optical system 110 Dichroic Mirror 111 Mirror 112 Mirror 113 Pupil projection lens 114 Imaging Lens 115 Mirror 116 Objective Lens 117 Confocal Lens 118 Confocal Pinhole IM1 1st optical image IM1' 1st optical image IM2 2nd optical image AX1 Optical axis of the first optical system AX2 Optical axis of the second optical system OBJ specimen FP focal plane AX optical axis OS optical system IP image plane IMG optical image PIC Optical image

Claims

1. a first optical system that forms a first optical image of the sample; a second optical system that forms a second optical image of the sample; A moving mechanism; an image generation unit; the first optical system has an observation optical system located on an optical path from the specimen to the first optical image, an optical axis of the first optical system and an optical axis of the second optical system coincide with each other at the position of the sample; the moving mechanism changes a distance between the specimen and a first objective lens of the first optical system and a distance between the specimen and a second objective lens of the second optical system; By changing the interval, a fluorescent image group made up of a plurality of fluorescent images and a bright-field image group made up of a plurality of bright-field images are generated, the fluorescence image group is generated based on the first optical image; the bright-field images are generated based on the second optical image; The estimated sample is a sample modeled from the sample, the virtual observation optical system is an optical system that models the observation optical system, The image generation unit a memory for storing the fluorescent image group and the bright-field image group; a processor, The processor: calculating an estimated image of the estimated specimen using a refractive index distribution of the estimated specimen; calculating the optimized refractive index distribution as a final refractive index distribution using the bright-field image and the estimated image; Dividing the fluorescent image into a plurality of areas; In the estimation sample, a virtual area corresponding to the area is set; setting a virtual ray traveling from the virtual area toward the virtual observation optical system; calculating a point spread function corresponding to the area using the final refractive index distribution included in the range through which the virtual ray travels; generating an image corresponding to each of the areas using the point spread function and the fluorescence image of the area; combining all of the images to generate a final image corresponding to the fluorescence image; performing the calculation of the final refractive index distribution for each image in the set of bright-field images; A specimen image acquisition device characterized in that the generation of the final image is performed for each image in the group of fluorescent images.

2. a common objective lens is disposed on an optical path in which the optical axis of the first optical system and the optical axis of the second optical system coincide with each other; 2. The specimen image acquisition device according to claim 1, wherein the first optical image and the second optical image are formed via the common objective lens.

3. the second optical system includes an illumination optical system; 2. The specimen image acquisition device according to claim 1, wherein the following conditional expression (1) is satisfied: 0.1≦NAill / NA≦1 (1) where: NAill is the numerical aperture of the illumination optical system, NA is the numerical aperture of the second objective lens, is.

4. The processor:

3. The specimen image acquisition device according to claim 1, wherein the fluorescent image and the bright-field image are aligned before the estimated image is calculated.

5. generating the plurality of fluorescent images by varying the specimen and the common objective lens at a first interval; generating the plurality of bright-field images by varying the specimen and the common objective lens at second intervals; 3. The specimen image acquisition device according to claim 2, wherein the following conditional expression (2) is satisfied: △Z1 < △Z2 (2) where: ΔZ1 is the first distance, ΔZ2 is the second distance, is.

6. a light source that irradiates illumination light toward the second optical system, the specimen in a liquid-filled container is illuminated by the illumination light; 6. The specimen image acquisition device according to claim 5, wherein the following conditional expression (3) is satisfied: △Z1<△Z2<2.5×n×λc / NA ２ (3) where: n is the refractive index of the liquid, λc is the centroid wavelength of the illumination light, NA is the numerical aperture of the second objective lens, is.

7. a light source that irradiates illumination light toward the second optical system, The processor divides the bright field image into a plurality of areas; 4. The specimen image acquisition device according to claim 2, wherein the following conditional expression (4) is satisfied: 2<dxsplit / D<200 (4) where: D=1.22×λc / (NA+NAill), dxsplit is the length of one side of the area in the bright-field image, λc is the centroid wavelength of the illumination light, NA is the numerical aperture of the second objective lens, NAill is the numerical aperture of the illumination optical system, is.

8. The processor: a point light source that emits the virtual light ray is set in the virtual area; 2. The specimen image acquisition device according to claim 1, wherein the point spread function is calculated using a first wavefront whose wave source is the point light source.

9. The processor: calculating a second wavefront emitted from the estimated sample using the first wavefront and the final refractive index distribution included in a range in which the virtual ray travels; calculating a third wavefront at a focal plane of the virtual observation optical system using the second wavefront; calculating an intensity distribution corresponding to the third wavefront; 9. The specimen image acquisition device according to claim 8, wherein the point spread function is calculated using the intensity distribution.

10. The processor: Dividing the virtual area into a plurality of small areas; The final refractive index distribution is set for each of the minute areas; calculating a point spread function corresponding to the minute area as a minute point spread function using the final refractive index distribution of the minute area; 2. The specimen image acquisition device according to claim 1, wherein the point spread function is calculated using the minute point spread function.

11. a light source that irradiates excitation light toward the first optical system, The processor: Calculating the excitation light intensity at the position of the point light source; calculating a fluorescence intensity distribution using the intensity distribution and the excitation light intensity; 10. The specimen image acquisition device according to claim 9, wherein the point spread function is calculated using the fluorescence intensity distribution.

12. The processor:

12. The specimen image acquisition device according to claim 11, wherein the excitation light intensity is calculated using a refractive index distribution at the excitation light wavelength.

13. The processor: calculating a virtual point spread function for each of the areas before calculating the point spread function; setting a peak intensity value of the point spread function when the estimated specimen does not exist as a reference value; setting the intensity peak value of the provisional point spread function as a provisional intensity peak value; comparing the tentative intensity peak value with the reference value, and dividing each of the virtual areas into a target area and a non-target area; the target area is an area where the tentative intensity peak value is less than 1 / 5 of the reference value, the non-target area is an area where the tentative intensity peak value is equal to or greater than 1 / 5 of the reference value, 2. The specimen image acquisition device according to claim 1, wherein the target area is divided into areas smaller than the non-target area.

14. a memory for storing a fluorescent image of a specimen and a bright field image of said specimen; a processor, the fluorescence image is generated based on a first optical image formed by a first optical system; the bright-field image is generated based on a second optical image formed by a second optical system; an optical axis of the first optical system and an optical axis of the second optical system include a single optical axis that coincides at a position of a sample; the plurality of fluorescent images and the plurality of bright-field images are images taken at different positions in a direction parallel to the single optical axis; The estimated sample is a sample modeled from the sample, the first optical system includes a first observation optical system located on an optical path from the specimen to the first optical image, the virtual observation optical system is an optical system that models the first observation optical system, The processor: calculating an estimated image of the estimated specimen using a refractive index distribution of the estimated specimen; calculating the optimized refractive index distribution as a final refractive index distribution using the bright-field image and the estimated image; Dividing the fluorescent image into a plurality of areas; In the estimation sample, a virtual area corresponding to the area is set; setting a virtual ray traveling from the virtual area toward the virtual observation optical system; calculating a point spread function corresponding to the area using the final refractive index distribution included in the range through which the virtual ray travels; generating an image corresponding to each of the areas using the point spread function and the fluorescence image of the area; A specimen image generating device characterized by synthesizing all of the images to generate an image corresponding to the fluorescent image.

15. The processor: a point light source that emits the virtual light ray is set in the virtual area; 15. The specimen image generating device according to claim 14, wherein the point spread function is calculated using a first wavefront whose wave source is the point light source.

16. The processor: calculating a virtual point spread function for each of the areas before calculating the point spread function; setting a peak intensity value of the point spread function when the estimated specimen does not exist as a reference value; setting the intensity peak value of the provisional point spread function as a provisional intensity peak value; comparing the tentative intensity peak value with the reference value, and dividing each of the virtual areas into a target area and a non-target area; the target area is an area where the tentative intensity peak value is less than 1 / 5 of the reference value, the non-target area is an area where the tentative intensity peak value is equal to or greater than 1 / 5 of the reference value, 15. The specimen image generating device according to claim 14, wherein the target area is divided into areas smaller than the non-target area.

Images