Sample image generation device, sample image generation method, sample image generation system, and recording medium

Abstract

Provided is a specimen image generation device with which it is possible to recover an image in a highly accurate manner.　This specimen image generation device 1 comprises a memory 2 and a processor 3. A first image is an image obtained by imaging a specimen, and a prescribed direction in the first image is the direction, from among the optical axis directions of a virtual observation optical system, in which the virtual observation optical system is present. The processor 3 acquires the first image from the memory 2, divides the first image into a plurality of areas, acquires the refractive index distribution of the specimen from the memory 2, calculates a point image intensity distribution using the refractive index distribution, generates a second image using the point image intensity distribution, synthesizes the second image, and generates a third image corresponding to the first image. In the calculation process, the point image intensity distribution of a first area is calculated using the refractive index distribution of areas included in an area group, the first area being an area for which the point image intensity distribution is to be calculated, and the area group being configured from a plurality of areas on the inside of a range in which a light beam is radiated in a prescribed direction from the first area.

Description

Technical Field

[0001] The present invention relates to a specimen image generation device, a specimen image generation method, a specimen image generation system, and a recording medium.

Background Art

[0002] For example, in a microscope or an endoscope, an optical image of a specimen is formed by an optical system. By photographing the optical image with an image pickup device, an image of the optical image is acquired.

[0003] FIG. 18 is a diagram showing the state of imaging, the optical image, and the image of the optical image. FIG. 18(a) is a diagram showing the state of imaging when the specimen is a three-dimensional object. FIG. 18(b) is a diagram showing the image of the optical image in the XY cross section of the specimen. FIG. 18(c) is a diagram showing the image in the XZ cross section of the optical image.

[0004] The optical system is an ideal optical system. The optical axis of the optical system is defined as the Z axis, the axis orthogonal to the Z axis is defined as the X axis, and the axis orthogonal to both the Z axis and the X axis is defined as the Y axis. The XY cross section is a plane including the X axis and the Y axis. The XZ cross section is a plane including the X axis and the Z axis.

[0005] The specimen is a cell mass. The cell mass is formed of a plurality of cells. Each cell has a cell nucleus.

[0006] As shown in FIG. 18(a), since the specimen OBJ is a cell mass, the specimen OBJ has a thickness not only in the direction orthogonal to the optical axis AX but also in the direction parallel to the optical axis AX. The optical image IMG is formed on the image plane IP by the optical system OS.

[0007] In the specimen OBJ, only the cell nuclei are stained with a fluorescent dye. Therefore, when the specimen OBJ is irradiated with excitation light, fluorescence is emitted only from the cell nuclei. As a result, a fluorescence image of the cell nuclei is formed as the optical image IMG.

[0008] By photographing the optical image IMG, an image PIC of the optical image can be acquired. In the image PIC of the optical image, only the cell nuclei are imaged.

[0009] The image plane IP is conjugate to the focal plane FP. The optical image IMG represents the optical image of the specimen OBJ located at the focal plane FP in the XY cross-section. As indicated by the arrows, multiple optical images can be acquired by capturing optical images while relatively moving the specimen OBJ and the focal plane FP along the optical axis AX.

[0010] When the specimen obj is fixed and the optical system OS is moved toward the specimen obj, the focal plane FP moves in the following order: the top surface of the specimen obj, the interior of the specimen obj, and the bottom surface of the specimen obj.

[0011] Figure 18(b) shows five optical images. Images PIC1, PICm-1, PICm, PICm+1, and PICn are optical images of the XY cross-section of the specimen OBJ. For example, image PIC1 is an image of the top surface of the specimen OBJ, images PICm-1, PICm, and PICm+1 are images of the interior of the specimen OBJ, and image PICn is an image of the bottom surface of the specimen OBJ.

[0012] The position of the XY cross-section in the specimen OBJ differs in each optical image. Therefore, the optical images are all different from one another.

[0013] From each image, from image PIC1 to image PICn, a column of data parallel to the X-axis is extracted. By arranging this column of data along the Z-axis, an image of the XZ cross-section of the optical image is obtained.

[0014] Figure 18(c) shows images in the XZ section of the optical image. Image PICxz is an image of the cell nucleus. The left-right direction of the image is along the optical axis AX. The right side of the image represents the image of the top surface of the specimen OBJ, and the left side of the image represents the image of the bottom surface of the specimen OBJ.

[0015] When the shape of the cell nucleus is spherical, the shape of the XZ cross-section is circular. As shown in Figure 18(c), in image PICxz, deformation, loss of sharpness, and decrease in brightness occur in all images of the cell nucleus. If deformation, loss of sharpness, and decrease in brightness are considered to be image quality degradation, then image PICxz exhibits image quality degradation.

[0016] An optical image is obtained by capturing an optical image. If there is a degradation in the image quality of an optical image, it means that degradation has occurred in the optical image itself.

[0017] When the sample is a point light source, it is desirable that the optical image be a point image. For a point image to be formed, the optical system must be aberration-free (hereinafter referred to as the "ideal optical system") and all of the light emitted from the point light source must be incident on the optical system.

[0018] However, since the size of an optical system is finite, it is not possible to allow all the light emitted from a point source to enter the optical system. In this case, the optical image is affected by diffraction. As a result, even if the optical system is ideal, a point image is not formed, and a spread image is formed. This spread image is called a point image intensity distribution.

[0019] The optical image is represented by the following equation (1) using the point image intensity distribution. I = O*PSF (1) Here, I is an optical image. O is a sample. PSF is point intensity distribution. * is convolution, That is the case.

[0020] If we consider the point image intensity distribution as an optical filter, then equation (1) shows that the optical image is obtained through this filter of the point image intensity distribution. If degradation occurs in the optical image, it means that the filter, i.e., the point image intensity distribution, has characteristics that cause deformation, a decrease in sharpness, and a decrease in brightness (hereinafter referred to as "degradation characteristics").

[0021] In the frequency space, Equation (1) is represented by the following Equation (2). FI = FO × OTF (2) Here,[[]]END]] FI is the Fourier transform of the optical image,[[]]END]] FO is the Fourier transform of the specimen,[[]]END]] OTF is the optical transfer function,[[]]END]] and

[0022] The OTF is the Fourier transform of the point spread function. When the point spread function has degradation characteristics, the OTF also has degradation characteristics.[[]]END]]

[0023] When Equation (2) is transformed, Equation (2) is represented by the following Equation (3).[[]]END]] FO = FI / OTF (3)[[]]END]]

[0024] If FI and OTF can be obtained, FO can be obtained. Then, by performing the inverse Fourier transform on FO, O can be obtained. O is the specimen. This operation is called deconvolution.[[]]END]]

[0025] The image PICxz shown in FIG. 18(c) is an image of the optical image of the cell nucleus. In the image PICxz, only the cell nucleus is imaged. Therefore, when deconvolution is performed using the image PICxz and the image of the OTF, only the image of the cell nucleus can be obtained.[[]]END]]

[0026] Since the specimen OBJ is a cell mass, it has a plurality of cytoplasm and a plurality of cell nuclei. However, in the image PICxz, even when deconvolution is performed, only the image of the cell nucleus can be obtained. Since the image of the cytoplasm cannot be obtained, it is difficult to say that the specimen OBJ has been obtained. Although the specimen can be obtained by performing deconvolution, whether the specimen can be obtained depends on the image of the optical image.[[]]END]]

[0027] Considering in terms of an image, Equation (1) represents that the image of the optical image is an image obtained through a filter called the point spread function. When the point spread function has degradation characteristics, I can be regarded as an image of the optical image with degraded image quality, and O can be regarded as an image of the optical image before the image quality degrades.[[]]END]]

[0028] In this case, equation (3) indicates that an image of the optical image before degradation is generated from an image of the optical image with degraded quality. Hereafter, the image of the optical image with degraded quality will be referred to as the "degraded image". Furthermore, the image of the optical image before degradation can be said to be the result of image restoration in the degraded image. Therefore, the image of the optical image before degradation will be referred to as the "restored image".

[0029] To generate a recovery image, it is necessary to determine the point intensity distribution. This will be explained using Figure 18(a). In Figure 18(a), the refractive index outside the sample OBJ is n1, and the refractive index inside the sample OBJ is n2.

[0030] The ideal shape is defined as the shape of the point image intensity distribution of an ideal optical system. In the ideal shape, the refractive index between the focal plane and the ideal optical system matches a predetermined refractive index.

[0031] The sample object (OBJ) is moved from a position away from the focal plane (FP) toward the optical system (OS). Since the optical system OS does not move, the top surface of the sample object (OBJ) reaches the focal plane (FP). In this state (hereinafter referred to as the "first state"), only a space with refractive index n1 exists between the focal plane (FP) and the optical system OS. When a point light source is placed on the focal plane (FP), the point image intensity distribution of the first state is obtained.

[0032] In the first state, the refractive index between the focal plane FP and the optical system OS is n1. If we assume a predetermined refractive index of n1, the point image intensity distribution in the first state is obtained based solely on this predetermined refractive index. Therefore, the shape of the point image intensity distribution in the first state is the same as the ideal shape.

[0033] Further movement of the sample OBJ causes the focal plane FP to reach the interior of the sample OBJ. In this state (hereinafter referred to as the "second state"), there are spaces with refractive index n1 and spaces with refractive index n2 between the focal plane FP and the optical system OS. When a point light source is placed on the focal plane FP, the point image intensity distribution of the second state is obtained.

[0034] In the second state, the refractive index between the focal plane FP and the optical system OS is determined by n1 and n2. Since n1 is the predetermined refractive index, n2 is not the predetermined refractive index. In this case, the point image intensity distribution in the second state is obtained based on the predetermined refractive index and the undetermined refractive index. Therefore, the shape of the point image intensity distribution in the second state differs from the ideal shape.

[0035] Thus, the shape of the point image intensity distribution changes depending on the size of the space with refractive index n². Therefore, when calculating the point image intensity distribution, the refractive index distribution in the sample OBJ must be appropriately considered.

[0036] A technique for recovering images is disclosed in Non-Patent Document 1. This recovery technique uses the optical image and point intensity distribution obtained from a thick specimen. In calculating the point intensity distribution, the specimen is divided into multiple blocks, and the refractive index of a row of blocks parallel to the optical axis is used. [Prior art documents] [Non-patent literature]

[0037] [Non-Patent Document 1] Sreya Ghosh, Chrysanthe Preza, "Three-dimensional block-based restoration integrated with wide-field fluorescence microscopy for the investigation of thick specimens with spatially variant refractive index", J. of Biomedical Optics, 21(4), 046010 (2016) [Overview of the Initiative] [Problems that the invention aims to solve]

[0038] In calculating the point image intensity distribution, a point light source is placed in a block, and the wavefront emitted from the sample is determined. The wavefront emitted from the point light source is a spherical wave. Therefore, the wavefront propagates through a row of blocks that are in contact with the block where the point light source is placed and parallel to the optical axis, as well as through blocks located around the row of blocks.

[0039] The recovery technique described above calculates the point image intensity distribution using only the refractive index of a single row of blocks. In this case, the point image intensity distribution is not calculated accurately. Therefore, it cannot be said that the image is recovered with high accuracy.

[0040] The present invention has been made in view of these problems, and aims to provide a specimen image generation device, a specimen image generation method, a specimen image generation system, and a recording medium that can recover images with high accuracy. [Means for solving the problem]

[0041] To solve the above-mentioned problems and achieve the objectives, the sample image generation apparatus according to at least some embodiments of the present invention is: Equipped with memory and a processor, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. The processor is The first acquisition process is executed to acquire the first image from memory. The acquired first image is divided into multiple areas using a division process. A second acquisition process is executed to obtain the refractive index distribution of the sample from memory. Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each divided area. Using the point image intensity distribution calculated for each area, a first generation process is performed to generate a second image corresponding to each area. A second image corresponding to each area is combined to generate a third image corresponding to the first image. In the calculation process, The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. Area 1 is the area for which the point image intensity distribution will be calculated. The area group is characterized by being composed of multiple areas located within the range from which light rays radiate in a predetermined direction, starting from the first area, and including areas outside the range extended from the first area in the predetermined direction.

[0042] Furthermore, the sample image generation system according to at least some embodiments of the present invention is An observation optical system that forms an optical image of a specimen, An image sensor that captures an optical image, The above-described sample image generation device is characterized by having the above-described sample image generation device.

[0043] Furthermore, the sample image generation system according to at least some embodiments of the present invention is Equipped with memory and a processor, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. The processor is The first acquisition process is executed to acquire the first image from memory. The acquired first image is divided into multiple areas using a division process. A second acquisition process is executed to obtain the refractive index distribution of the sample from memory. Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each divided area. Using the point image intensity distribution calculated for each area, a first generation process is performed to generate a second image corresponding to each area. A second image corresponding to each area is combined to generate a third image corresponding to the first image. In the calculation process, The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. Area 1 is the area for which the point image intensity distribution will be calculated. The area group consists of multiple areas located within the range from which light rays radiate in a predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in a predetermined direction. The processor performs machine learning processes to train the AI ​​model. Machine learning processing involves training an AI model on multiple datasets. The dataset includes the first image and the training data corresponding to the first image. The training data corresponding to the first image is characterized by being the second image corresponding to the first image.

[0044] Furthermore, the sample image generation method according to at least some embodiments of the present invention is A method for generating a sample image using the refractive index distribution of a first image and the sample, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. The first acquisition process is executed to acquire the first image from memory. The acquired first image is divided into multiple areas using a division process. A second acquisition process is executed to obtain the refractive index distribution of the sample from memory. Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each divided area. Using the point image intensity distribution calculated for each area, a first generation process is performed to generate a second image corresponding to each area. A second image corresponding to each area is combined to generate a third image corresponding to the first image. In the calculation process, The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. Area 1 is the area for which the point image intensity distribution will be calculated. The area group is characterized by being composed of multiple areas located within the range from which light rays radiate in a predetermined direction, starting from the first area, and including areas outside the range extended from the first area in the predetermined direction.

[0045] Furthermore, recording media according to at least some embodiments of the present invention are A computer-readable recording medium containing a program for generating sample images, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. On the computer, The first acquisition process involves obtaining the first image from memory, The acquired first image is divided into multiple areas using a division process, A second acquisition process obtains the refractive index distribution of the sample from memory, A calculation process is performed to calculate the point image intensity distribution for each divided area using the acquired refractive index distribution. The first generation process generates a second image corresponding to each area using the point image intensity distribution calculated for each area, The process involves combining a second image corresponding to each area to generate a third image corresponding to the first image, and then executing this process. In the calculation process, The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. Area 1 is the area for which the point image intensity distribution will be calculated. The area group is characterized by being composed of multiple areas located within the range from which light rays radiate in a predetermined direction, starting from the first area, and including areas outside the range extended from the first area in a predetermined direction. [Effects of the Invention]

[0046] According to the present invention, it is possible to provide a specimen image generation apparatus, a specimen image generation method, a specimen image generation system, and a recording medium that can recover images with high accuracy. [Brief explanation of the drawing]

[0047] [Figure 1] This figure shows the specimen image generation device, microscope system, and estimation system of this embodiment. [Figure 2] This figure shows the specimen, optical image, image of the optical image, and XY image set. [Figure 3] This is a flowchart of the processing performed by the processor. [Figure 4] This figure shows the specimen, the optical image, and the first image. [Figure 5] This figure shows the first image, the refractive index image, the first area, and the group of areas. [Figure 6] This is a diagram showing the grouping of areas. [Figure 7] This figure shows the first image, the refractive index image, and the PSF image. [Figure 8] This is a diagram showing the first, second, and third images. [Figure 9] This is a diagram showing the first and third images. [Figure 10] This diagram shows wavefront propagation. [Figure 11] This is a diagram showing the refractive index. [Figure 12] This is a diagram showing the refractive index. [Figure 13] This is a diagram showing the refractive index. [Figure 14] This is a flowchart of the calculation process. [Figure 15] This figure shows the predetermined intensity distribution and the transmission characteristics of the pinhole. [Figure 16] This is a diagram showing the training process. [Figure 17] This is a diagram showing the sample image generation system of this embodiment. [Figure 18] This diagram shows the image formation process, the optical image, and an image of the optical image. [Modes for carrying out the invention]

[0048] Prior to describing the embodiments, the effects and advantages of an embodiment according to a certain aspect of the present invention will be explained. In specifically describing the effects and advantages of this embodiment, concrete examples will be provided. However, as with the embodiments described later, these exemplified embodiments represent only a portion of the embodiments included in the present invention, and numerous variations exist. Therefore, the present invention is not limited to the exemplified embodiments.

[0049] In this embodiment of the specimen image generation device, an image of the optical image of the specimen is used. An optical image of the specimen is formed in the observation optical system, and the optical image of the specimen is captured by an image sensor, thereby obtaining an image of the optical image of the specimen.

[0050] Since the specimen is a three-dimensional object, the optical image of the specimen can be represented by an XY image, an XZ image, and a YZ image. Furthermore, since the optical image of the specimen is acquired through an observation optical system, the optical image of the specimen is a degraded image.

[0051] The optical axis of the observation optical system is defined as the Z-axis, the axis perpendicular to the Z-axis is defined as the X-axis, and the axis perpendicular to both the Z-axis and the X-axis is defined as the Y-axis. The XY section is a plane containing the X-axis and the Y-axis. The XY image is the image in the XY section. The XZ section is a plane containing the X-axis and the Z-axis. The XZ image is the image in the XZ section. The YZ section is a plane containing the Y-axis and the Z-axis. The YZ image is the image in the YZ section.

[0052] The specimen image generation apparatus of this embodiment comprises a memory and a processor. The processor performs a first acquisition process to acquire a first image from the memory, a division process to divide the acquired first image into multiple areas, a second acquisition process to acquire the refractive index distribution of the specimen from the memory, a calculation process to calculate the point image intensity distribution for each of the divided areas using the acquired refractive index distribution, a first generation process to generate a second image corresponding to each area using the point image intensity distribution calculated for each area, and synthesizes the second images corresponding to each area to generate a third image corresponding to the first image. In the calculation process, the point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first image is an image obtained by photographing a specimen, and the predetermined direction in the first image is the direction in which the virtual observation optical system exists within the optical axis direction of the virtual observation optical system. The first area is the area for which the point image intensity distribution is calculated, and the area group consists of multiple areas located inside the range in which light rays radiate in a predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in a predetermined direction.

[0053] Figure 1 shows the specimen image generation device, microscope system, and estimation system of this embodiment. Figure 1(a) shows the specimen image generation device of this embodiment. Figure 1(b) shows the microscope system. Figure 1(c) shows the estimation system.

[0054] As shown in Figure 1(a), the sample image generation device 1 comprises a memory 2 and a processor 3. The memory 2 stores the first image and the refractive index distribution of the sample.

[0055] The first image is an image obtained by photographing the specimen. The first image can be generated, for example, from multiple XY images (hereinafter referred to as the "XY image group"). Each XY image in the XY image group is an optical image of the specimen. As mentioned above, the optical image of the specimen is a degraded image. Therefore, the first image is also a degraded image.

[0056] Processor 3 performs the process of generating a sample image. This process uses the refractive index distribution of the first image and the sample. Since the first image is a degraded image, the sample image is a restored image. Therefore, processor 3 performs the process of generating the sample image from the degraded image.

[0057] In order to generate the first image with the specimen image generation device 1, the XY image group must be input to the specimen image generation device 1. The XY image group is input to the specimen image generation device 1 via the input unit 4. The XY images can be acquired, for example, by a microscope system.

[0058] As shown in Figure 1(b), the microscope system 10 includes a microscope 20 and a processing unit 30. The microscope 20 includes a main body 21, an objective lens 22, a stage 23, an reflected light illumination device 24, an imaging unit 25, and a controller 26. The processing unit 30 includes an input unit 31, a memory 32, a processor 33, and an output unit 34.

[0059] A specimen 27 is placed on stage 23. In the microscope 20, an optical image of the specimen 27 is formed on the image plane of the observation optical system. If a lens is arranged in the imaging unit 25, the observation optical system is formed by the objective lens 22, the imaging lens, and the lens of the imaging unit 25. If a lens is not arranged in the imaging unit 25, the observation optical system is formed by the objective lens 22 and the imaging lens.

[0060] The imaging unit 25 has an image sensor. An image of the optical image is acquired by capturing the optical image formed on the image plane with the image sensor. The optical image formed on the image plane is an optical image of the XY cross section of the specimen 27. Therefore, the image of the optical image is an XY image.

[0061] The objective lens 22 and the stage 23 can be moved relative to each other along the optical axis of the observation optical system. The movement of the objective lens 22 or the stage 23 can be controlled by the controller 26. The specimen 27 is a thick specimen. Therefore, by moving the objective lens 22 and the stage 23 relative to each other, XY images can be acquired for multiple cross-sections. The XY image group will now be described.

[0062] Figure 2 shows the specimen, optical image, image of the optical image, and XY image set. Figure 2(a) shows the specimen. Figures 2(b), 2(c), and 2(d) show the optical image and image of the optical image. Figure 2(e) shows the XY image set.

[0063] As shown in Figure 2(a), since sample 50 is a three-dimensional object, it can be represented by multiple block layers. In Figure 2(a), sample 50 is divided into seven block layers along the Z-axis. However, the number of block layers is not limited to seven. Sample OZ1 represents the block layer at one end, and sample OZ7 represents the block layer at the other end. Each block layer represents the XY cross-section of sample 50.

[0064] In forming the optical image of sample 50, block layers from sample OZ1 to sample OZ7 are sequentially positioned at the focal plane of the observation optical system 51. Although the optical image is planar, it is represented by block layers for ease of viewing. Furthermore, because the optical image is represented by block layers, the image of the optical image is also represented by block layers.

[0065] In forming the optical image of sample 50, sample 50 and the observation optical system 51 are moved relative to each other along the optical axis 52. In this case, sample 50 is not moved, while the observation optical system 51 is moved relative to each other along the optical axis 52.

[0066] As shown in Figure 2(b), when the specimen OZ1 is positioned at the focal plane of the observation optical system 51, an optical image IZ1 is formed. By capturing the optical image IZ1 with an image sensor, an image of the optical image PZ1 is acquired.

[0067] As shown in Figure 2(c), when the specimen OZ4 is positioned at the focal plane of the observation optical system 51, an optical image IZ4 is formed. By capturing the optical image IZ4 with an image sensor, an image of the optical image PZ4 is acquired.

[0068] As shown in Figure 2(d), when the specimen OZ7 is positioned at the focal plane of the observation optical system 51, an optical image IZ7 is formed. By capturing the optical image IZ7 with an image sensor, an image of the optical image PZ7 is acquired.

[0069] Images PZ1, PZ4, and PZ7 are XY images. Two block layers are located between image PZ1 and image PZ4. Also, two block layers are located between image PZ4 and image PZ7. If we denote the images of these block layers as images PZ2, PZ3, PZ5, and PZ6, then images PZ2, PZ3, PZ5, and PZ6 are also XY images.

[0070] Since all images from image PZ1 to image PZ7 are XY images, a set of XY images can be obtained from these images. Furthermore, by stacking these images in the direction of the optical axis 52, a three-dimensional set of XY images can be obtained.

[0071] Figure 2(e) shows a three-dimensional XY image group 53. The XY image group 53 contains brightness information in the X-axis, Y-axis, and Z-axis directions, respectively. Therefore, XY images, XZ images, and YZ images can be generated from the XY image group 53.

[0072] As described above, the first image is generated from the XY image set. Therefore, the XY, XZ, and YZ images can all be used as the first image.

[0073] Returning to Figure 1(b), let's continue the explanation. In the microscope 20, multiple objective lenses can be mounted on the revolving nosepiece. The magnification of the objective lenses can be changed by rotating the revolving nosepiece. The rotation of the revolving nosepiece can be controlled by the controller 26.

[0074] The XY image is output from the imaging unit 25 and input to the processing unit 30. Optical information, movement information, and microscope information (hereinafter referred to as "various information") are output from the controller 26 and input to the processing unit 30.

[0075] Optical information includes, for example, information on the magnification of the objective lens and information on the numerical aperture of the objective lens. Movement information includes, for example, information on the amount of stage movement per step and information on the number of stage movements, or information on the amount of objective lens movement per step and information on the number of objective lens movements.

[0076] Microscope information refers to the information of the microscope used to acquire the XY images. Types of microscopes that can be used to acquire XY images include, for example, fluorescence microscopes, scanning laser microscopes (hereinafter referred to as "LSM"), two-photon microscopes, sheet illumination microscopes, and emission microscopes.

[0077] The XY images and various information are input to the input unit 31 and then stored in the memory 32. A group of XY images is obtained from multiple XY images. The group of XY images and various information are output from the output unit 34. Therefore, the group of XY images and various information can be input to the sample image generation device 1. The group of XY images and various information are stored in the memory 2.

[0078] The XY image set and various pieces of information must be associated with each other. This association can be performed by either the processing unit 30 or the sample image generation device 1.

[0079] The XY image set and various information can be input via wired or wireless connection. Alternatively, the processing unit 30 may record the XY image set and various information onto a recording medium. In this case, the XY image set and various information are input to the sample image generation device 1 via the recording medium.

[0080] As described above, the processing performed by processor 3 uses the refractive index distribution of the first image and the sample. The set of XY images required to generate the first image is: Microscope 20 It can be generated by [method]. The refractive index distribution of the sample can be generated, for example, by an estimation device.

[0081] As shown in Figure 1(c), the estimation device 40 has an input unit 41, a memory 42, a processor 43, and an output unit 44. In the estimation device 40, the refractive index distribution of the sample is estimated by computational imaging or deep learning.

[0082] Refraction by computational imaging rate This section explains the estimation of the distribution. The estimation uses the optical image of the sample and the optical image of the estimated sample. The optical image of the estimated sample can be obtained by simulation using a virtual optical system. Since the sample is a three-dimensional object, the estimated sample is also a three-dimensional object. In this case, the optical image of the estimated sample is represented by multiple estimated XY images (hereinafter referred to as the "estimated XY image group").

[0083] The refractive index distribution can be represented by an image. In this case, the refractive index distribution of a sample can be represented by multiple distribution images (hereinafter referred to as the "distribution image group"). The refractive index distribution of an estimated sample can be represented by multiple estimated distribution images (hereinafter referred to as the "estimated distribution image group"). Distribution images and estimated distribution images, like XY images, are images of the refractive index distribution in the XY cross-section.

[0084] As described above, the optical image of the sample is used for estimation. Therefore, the estimation device 40 receives the XY image set and various information via the input unit 41. The XY image set and various information are stored in the memory 42.

[0085] In processor 43, the estimated distribution image set is estimated using computational imaging. The estimation involves comparing the XY image set with the estimated XY image set. Specifically, the refractive index values ​​in the estimated distribution image set are changed so that the difference between the XY image set and the estimated XY image set is minimized.

[0086] The difference between the two images can be expressed numerically. Therefore, the comparison of the two images and the change in refractive index value are repeated until this numerical value falls below a threshold. The estimated distribution images obtained when this numerical value falls below the threshold are defined as the distribution image group. The distribution image group represents the refractive index distribution of the sample. Therefore, the refractive index distribution of the sample is determined. The distribution image group is output from the output unit 44.

[0087] The distribution image set can be input from the estimation device 40 to the sample image generation device 1 via wired or wireless connection. Alternatively, the distribution image set may be recorded on a recording medium by the estimation device 40 and input to the sample image generation device 1 via the recording medium. The distribution image set represents the refractive index distribution of the sample. Therefore, the refractive index distribution of the sample can be input to the sample image generation device 1. The distribution image set is stored in memory 2.

[0088] The sample image generation device 1 performs the process of generating sample images. This process of generating sample images is performed by the processor 3. The process performed by the processor 3 will be described below.

[0089] Figure 3 is a flowchart of the processing performed by the processor. Figure 4 shows the specimen, optical image, and first image. Figure 4(a) is a three-dimensional view of the specimen and optical image. Figure 4(b) shows the XZ cross section of the specimen. Figure 4(c) shows the first image. Figure 4(d) shows the XZ cross section of the specimen and the first image. Components identical to those in Figure 2(a) are numbered the same and their explanations are omitted.

[0090] As described above, the first image is generated from the XY image set. The XY image set is obtained from multiple optical images. As shown in Figure 4(a), when the observation optical system 51 is moved along the optical axis 52 without moving the specimen 50, multiple optical images are formed.

[0091] At position Z1, optical image IZ1 of sample OZ1 is formed. At position Z7, optical image IZ7 of sample OZ7 is formed. Optical image 60 is formed from optical images IZ1 to IZ7. By capturing optical image 60, an XY image set can be obtained.

[0092] XY, XZ, and YZ images can be generated from a set of XY images. All of these XY, XZ, and YZ images can be used as the first image. Let's assume that the XZ image is stored in memory as the first image.

[0093] In step S10, the first acquisition process is executed. In the first acquisition process, the first image is acquired from memory.

[0094] The first image is an optical image of the specimen in the XZ section. Figure 4(b) shows the XZ section of specimen 70. Specimen 70 is a cell aggregate. The cell aggregate is made up of multiple cells. Each cell in specimen 70 has cytoplasm 71 and a cell nucleus 72.

[0095] Figure 4(c) shows the first image acquired from memory. The first image 80 is a fluorescence image. In sample 70, only the cell nuclei 72 are stained with fluorescence. In this case, only the optical image of the cell nuclei 72 is formed. Therefore, the first image 80 contains only the image 81 of the cell nuclei.

[0096] Since the first image 80 is an optical image, the first image 80 is a degraded image. If the shape of the cell nucleus is considered to be a circle, then in the image of the cell nucleus 81, the shape is elliptical. When step S10 is completed, step S20 is executed.

[0097] In step S20, a division process is performed. In the division process, the acquired first image is divided into multiple areas. As shown in Figure 4(d), the first image 80 is divided into 11 areas in both the X-axis and Z-axis directions.

[0098] In Figure 4(d), the observation optical system and light rays are conveniently illustrated to show the correspondence between sample 70 and the first image 80. Observation optical system 51' is a virtual optical system, and its optical specifications are the same as those of observation optical system 51.

[0099] In optical imaging, the top and bottom of the optical image of the specimen are inverted relative to the top and bottom of the specimen. Since the first image 80 is an image, its top and bottom can be inverted during its generation. Therefore, in Figure 4(d), the top and bottom of the specimen 70 and the top and bottom of the first image 80 are aligned.

[0100] The location of area 82 corresponds to location OP1. The location of area 83 corresponds to location OP2. When step S20 is completed, step S30 is executed.

[0101] In step S30, the second acquisition process is executed. In the second acquisition process, the refractive index distribution of the sample is obtained from memory. The acquisition of the refractive index distribution will be described later. When step S30 is completed, step S40 is executed.

[0102] In step S40, the calculation process is performed. In the calculation process, the point image intensity distribution is calculated for each divided area using the acquired refractive index distribution. Specifically, the point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. Therefore, it is necessary to determine the first area and the area group.

[0103] Figure 5 shows the first image, refractive index image, first area, and area group. Figure 5(a) shows the first image, refractive index image, and first area. Figure 5(b) shows the first example of the area group. Figure 5(c) shows the second example of the area group.

[0104] The first area is the area for which the point image intensity distribution is calculated. In step S20, the first image 80 is divided into multiple areas. Therefore, the first area and the group of areas are determined by the areas in the first image 80.

[0105] However, the first image 80 is an optical image of the specimen. The optical image of the specimen contains brightness information but not refractive index distribution information. Since the point image intensity distribution is calculated using the refractive index distribution of the area group, the first image 80 is not suitable for calculating the point image intensity distribution. In the first image 80, the first area can be determined, but the area group cannot.

[0106] As described above, the distribution image set represents the refractive index distribution of the sample. Therefore, an image corresponding to the first image (hereinafter referred to as the "refractive index image") is obtained from the distribution image set. The point image intensity distribution is calculated using the refractive index distribution. Since the refractive index image is an image of the refractive index distribution, the refractive index image is suitable for calculating the point image intensity distribution.

[0107] The refractive index image should be stored in memory 2, and then read from memory 2 when the calculation process is executed. Since the first image is the XZ image, the refractive index image is the image of the XZ cross-section.

[0108] Image 80 is divided into multiple areas. Therefore, as shown in Figure 5(a), the refractive index image 90 is also divided into multiple areas. The refractive index image 90 is divided into 11 areas in both the X-axis and Z-axis directions. For clarity, only cell nuclei are shown in the refractive index image 90.

[0109] Similar to Figure 4(d), the top and bottom of the refractive index image 90 and the top and bottom of the first image 80 coincide. Furthermore, for convenience, the observation optical system 51' and the light rays are shown to illustrate the correspondence between the refractive index image 90 and the first image 80.

[0110] In refractive index image 90, the area corresponding to area 82 is area 91. The area corresponding to area 83 is area 92. The area corresponding to area 84 is area 93.

[0111] As described above, the refractive index image 90 is suitable for calculating the point image intensity distribution. Therefore, the first area and the area group are determined using the refractive index image 90.

[0112] Let's describe the first example of an area group. Figure 5(b) shows area 91, observation optical system 51', ray 100, and optical axis 101 of the observation optical system. Since no light rays are emitted from the image, ray 100 is a virtual ray.

[0113] In the first example, the first area in the first image 80 is area 82. The area corresponding to area 82 in the refractive index image 90 is area 91. Therefore, in the refractive index image 90, area 91 is the first area.

[0114] The area group and the predetermined direction are defined as follows: The area group consists of multiple areas located inside the range from which light rays radiate in the predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in the predetermined direction. The predetermined direction in the first image is the direction in which the observation optical system exists within the optical axis direction of the virtual observation optical system.

[0115] As described above, the first area and area group are determined using the refractive index image 90. Therefore, in the above specification, the first image is replaced with the refractive index image. In this case, the area group and the predetermined direction are defined as follows.

[0116] The area group in the refractive index image consists of multiple areas located inside the range in which light rays radiate in a predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in the predetermined direction. The predetermined direction in the refractive index image is the direction in which the virtual observation optical system exists within the optical axis direction of the observation optical system.

[0117] In the refractive index image 90, the side closer to the observation optical system 51' is considered the top surface of the specimen, and the side further from the observation optical system 51' is considered the bottom surface of the specimen. Area 91 is located at the point where it intersects the optical axis 101 on the top surface 90a. Light rays 100 are emitted from area 91. The light rays 100 emitted from area 91 are incident on the observation optical system 51'.

[0118] Light ray 100 is light incident on the observation optical system 51'. The amount of light incident on the observation optical system 51' is determined by the object-side numerical aperture of the observation optical system 51'. As described above, in the specimen image generation device 1, optical information is stored in the memory 2.

[0119] The optical information includes information about the numerical aperture of the objective lens. The numerical aperture of the objective lens can be considered as the object-side numerical aperture of the observation optical system 51'. Therefore, ray 100 can be identified from the numerical aperture of the objective lens.

[0120] Figure 5(b) illustrates the predetermined direction 102 and the non-predetermined direction 103. The predetermined direction 102 and the non-pretermined direction 103 are the optical axis directions of the observation optical system 51'. Of the optical axis directions of the observation optical system 51', the observation optical system 51' is located in the predetermined direction 102, but not in the non-pretermined direction 103.

[0121] The two rays 100 are rays of synchrotron radiation emitted from area 91. The area of ​​refractive index image 90 is not located inside the region between the two rays 100. Therefore, at the location of area 91, the number of areas in the area group is zero.

[0122] A second example of an area group will be described. Figure 5(c) shows area 93, central area 94, surrounding area 95, and surrounding area 96. The same numbering is used for elements with the same configuration as in Figure 5(b), and the explanation is omitted.

[0123] In the second example, the first area in the first image 80 is area 84. The area corresponding to area 84 in the refractive index image 90 is area 93. Therefore, in the refractive index image 90, area 93 is the first area.

[0124] Area 93 is located at the point where it intersects the optical axis 101 on the bottom surface 90b. Rays 100 and 104 are emitted from Area 93. Rays 100 and 104 emitted from Area 93 enter the observation optical system 51'. Ray 104 is a hypothetical ray.

[0125] Two rays 100 and two rays 104 are rays emitted from area 93. When light scattering in the sample is very small, the rays emitted from area 93 are represented by two rays 100. Inside the area between the two rays 100 are the central area 94 and the surrounding area 95. The central area 94 and the surrounding area 95 form a group of areas. Areas that intersect with rays 100 are considered to be included in the group of areas.

[0126] The central area 94 and the surrounding area 95 are each composed of multiple areas. Therefore, the area group is composed of multiple areas.

[0127] In the area group, the central area 94 is located within the range of area 93 extended in a predetermined direction 102. The surrounding area 95 is located outside the central area 94.

[0128] When light scattering in the sample is very large, the rays emitted from area 93 are represented by two rays 104. The central area 94, peripheral area 95, and peripheral area 96 are located within the area enclosed by the two rays 104. Therefore, center Area 94, surrounding area 95, and surrounding area 96 form an area group. Areas that intersect with ray 104 are considered to be included in the area group.

[0129] The central area 94, the surrounding area 95, and the surrounding area 96 are each composed of multiple areas. Therefore, the area group is composed of multiple areas.

[0130] Figure 6 shows an area group. Figure 6(a) shows a third example of an area group. Figure 6(b) shows a fourth example of an area group.

[0131] In the third example, as shown in Figure 6(a), area 97 is the first area. Area 97 is located between the top and bottom surfaces, where it intersects with the optical axis 101.

[0132] When light scattering in the sample is very small, the rays emitted from area 97 are represented by two rays 100. Inside the area between the two rays 100 are the central area 94 and the surrounding area 95. The central area 94 and the surrounding area 95 form an area group. Comparing the third example with the second example, the third example has fewer areas in its area group.

[0133] When light scattering in the sample is very large, the rays emitted from area 97 are represented by two rays 104. Inside the area between the two rays 104 are the central area 94, the peripheral area 95, and the peripheral area 96. The central area 94, the peripheral area 95, and the peripheral area 96 form a group of areas.

[0134] In the fourth example, as shown in Figure 6(b), area 98 is the first area. Area 98 is located on the bottom surface, away from the optical axis 101.

[0135] When light scattering in the sample is very small, the rays emitted from area 98 are represented by two rays 100. Inside the area between the two rays 100 are the central area 94 and the surrounding area 95. The central area 94 and the surrounding area 95 form an area group. Comparing the fourth example with the second example, the fourth example has fewer areas in its area group.

[0136] When light scattering in the sample is very large, the rays emitted from area 98 are represented by two rays 104. Inside the area between the two rays 104 are the central area 94, peripheral area 95, and peripheral area 96. Area groups are formed by central area 94, peripheral area 95, and peripheral area 96.

[0137] In the first example, the number of areas in the area group is zero. Therefore, the refractive index distribution of each area included in the area group is not used in the calculation process. The point image intensity distribution is calculated using the refractive index of the space between the refractive index image 90 and the observation optical system 51'. The calculation for the case where the number of areas in the area group is zero is also included in the calculation process.

[0138] In the second, third, and fourth examples, areas located outside the surrounding area 96 are not included in the area group. Therefore, these areas are not used in calculating the point image intensity distribution. However, these areas may be used in calculating the point image intensity distribution. That is, all areas located on the side of the predetermined direction 102 from the first area may be considered as an area group, and the point image intensity distribution may be calculated.

[0139] The first area is the area for which the point image intensity distribution is calculated. Therefore, by changing the area targeted by the first area, the point image intensity distribution can be calculated for each of the divided areas.

[0140] In the calculation process, the point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. In the first image 80, the first area is area 84 in the second example. In the refractive index image 90, area 93 corresponds to area 84. In the refractive index image 90, the area group consists of a central area 94 and a peripheral area 95, or a central area 94, a peripheral area 95, and a peripheral area 96.

[0141] Therefore, the point image intensity distribution of area 93 is calculated using the refractive index distribution of each area constituting the central area 94 and the refractive index distribution of each area constituting the peripheral area 95, or the point image intensity distribution of area 93 is calculated using the refractive index distribution of each area constituting the central area 94, the refractive index distribution of each area constituting the peripheral area 95, and the refractive index distribution of each area constituting the peripheral area 96. The point image intensity distribution of area 93 can then be treated as the point image intensity distribution of area 84 in the first image 80.

[0142] Area 84 is the first area in the first image 80. Each area of ​​the first image 80 already contains information about the brightness of the optical image of the specimen. Therefore, an image with a point image intensity distribution (hereinafter referred to as the "PSF image") is generated separately from the first image 80.

[0143] Figure 7 shows the first image, refractive index image, and PSF image. Figure 7(a) shows the first image and refractive index image. Figure 7(b) shows the refractive index image and PSF image. Figure 7(a) is the same as Figure 5(a), so no explanation is given.

[0144] The first image 80 is divided into multiple areas. Therefore, as shown in Figure 7(b), the PSF image 110 is also divided into multiple areas. The PSF image 110 is divided into 11 areas in both the X-axis and Z-axis directions.

[0145] As can be seen from comparing Figure 7(a) and Figure 7(b), in PSF image 110, the area corresponding to area 111 is area 82. The area corresponding to area 112 is area 83. The area corresponding to area 113 is area 84. Therefore, each area in PSF image 110 has the point intensity distribution of the area corresponding to the first area in the first image 80. In Figure 7(b), the point intensity distribution is illustrated for only some areas.

[0146] Let's return to Figure 3 for explanation. Once step S40 is completed, step S50 is executed.

[0147] In step S50, the first generation process is executed. In the first generation process, a second image corresponding to each area is generated using the point image intensity distribution calculated for each area.

[0148] Figure 8 shows the first, second, and third images. Figure 8(a) shows the first and second images. Figure 8(b) shows the first and third images.

[0149] Figure 8(a) shows a portion of the first image, a portion of the PSF image, and a group of second images.

[0150] Area DEG is a part of the first image 80. Area DEG is formed by Area DEG1, Area DEG2, Area DEG3, Area DEG4, Area DEG5, and Area DEG6.

[0151] Area PSF is a part of the PSF image 110 and corresponds to Area DEG. Area PSF is formed by Area PSF1, Area PSF2, Area PSF3, Area PSF4, Area PSF5, and Area PSF6.

[0152] The second image group REC consists of images of the areas corresponding to area DEG and area PSF. The second image group REC is composed of second image REC1, second image REC2, second image REC3, second image REC4, second image REC5, and second image REC6.

[0153] The image for area DEG1 is the first image. The image for area PSF1 is the point intensity distribution. The second image, REC1, is generated from the images for area DEG1 and area PSF1.

[0154] In area DEG, the shape of the cell nucleus is elliptical. In the second image, the shape of the cell nucleus is circular. Therefore, in the first generation process, a restored image is generated from the degraded image. When step S50 is completed, step S60 is executed.

[0155] In generating the second image, it is advisable to apply a masking process to the image of each area DEG. One example of masking is blurring the edges of the image.

[0156] In step S60, a third image is generated. The third image corresponds to the first image. In generating the third image, the second images corresponding to each area are combined.

[0157] Figure 8(b) shows the first image 80 and the third image 120. The third image 120 is generated by combining the second image. The second image is generated based on the first image 80, and the third image is generated based on the second image. Therefore, the third image 120 is the image corresponding to the first image 80.

[0158] In the first image 80, the shape of the cell nucleus is elliptical. In the third image 120, the shape of the cell nucleus is circular. Therefore, the specimen image generation device 1 can generate high-quality restored images from degraded images.

[0159] When combining the second image, it is advisable to use weighting. For two adjacent second images, the influence of one second image should be halved at the boundary between the two images.

[0160] Figure 9 shows the first and third images. Figure 9(a) shows the first image. Figure 9(b) shows the third image.

[0161] The right edge of the image represents the top surface of the specimen, and the left edge represents the bottom surface of the specimen. The image quality is higher in the third image than in the first image across the entire range from the top to the bottom surface of the specimen.

[0162] As described above, the calculation of the point image intensity distribution utilizes not only the refractive index distribution of the central area but also the refractive index distribution of the surrounding area. Therefore, compared to recovery techniques that use only the refractive index distribution of the central area, the point image intensity distribution can be calculated with higher accuracy. As a result, the sample image generation device of this embodiment can recover images with high accuracy.

[0163] In the sample image generation apparatus of this embodiment, it is preferable that the processor, in the calculation process, sets a point light source within the first area and calculates the point image intensity distribution of the first area using the first wavefront with the set point light source as the wave source.

[0164] As explained in Figure 5(c), the area group is determined by the range of light radiating in a predetermined direction, starting from the first area. Light emitted from the first area passes through the area group and enters the observation optical system. The point image intensity distribution is then obtained from the light emitted from the observation optical system.

[0165] Light radiating from the first area is obtained by setting a point light source in the first area. A wavefront originating from the point light source is emitted. If this wavefront is designated as the first wavefront, the point image intensity distribution of the first area can be calculated using the first wavefront.

[0166] In the sample image generation apparatus of this embodiment, it is preferable that the processor calculates a second wavefront using the first wavefront and the refractive index distribution corresponding to each area included in the area group during the calculation process, calculates an intensity distribution corresponding to the third wavefront using the calculated second wavefront, and calculates the point image intensity distribution of the first area using the calculated intensity distribution. The second wavefront is the wavefront propagated through the sample in a predetermined direction, and the third wavefront is the wavefront at the position of the focal plane of the virtual observation optical system.

[0167] Figure 10 shows the propagation of the wavefront. Components identical to those in Figure 5(c) are numbered the same way, and their explanations are omitted.

[0168] The refractive index distribution is used to calculate the point image intensity distribution. Therefore, we will explain using the refractive index image 90. In the refractive index image 90, area 93 corresponds to the first area. Therefore, area 93 is located on the focal plane FP. Also, a point light source 130 is set in area 93.

[0169] A first wavefront WF1 is emitted from the point light source 130. The first wavefront WF1 propagates from area 93 toward the upper surface 131 of the refractive index image 90. The upper surface 131 is the outer edge of the sample. The observation optical system 132 is located on the side of the upper surface 131. Therefore, the first wavefront WF1 propagates in a predetermined direction.

[0170] Wavefront propagation can be calculated using simulation. The observation optical system 132 is a virtual optical system, for example, formed by an objective lens 133 and an imaging lens 134. The optical specifications of the observation optical system 132 are the same as those of the observation optical system 51. Optical specifications, such as magnification and numerical aperture, can be obtained based on various information.

[0171] The first wavefront WF1 propagates through the area group and reaches the upper surface 131. From the upper surface 131, the second wavefront WF2 is emitted. The second wavefront WF2 is the wavefront after propagating through the area group. The area group is formed by the central area 94 and the surrounding area 95. Therefore, the second wavefront WF2 can be calculated using the refractive index distribution of each area included in the area group.

[0172] In the observation optical system 132, the focal plane FP and the image plane IP are conjugate. In order to determine the point image intensity distribution 135 at the image plane IP, the wavefront at the focal plane FP is required. The second wavefront WF2 is located at the upper surface 131. By propagating the second wavefront WF2 to the focal plane FP, the third wavefront WF3 can be obtained as the wavefront at the focal plane FP.

[0173] observation The optical system 132 forms a Fourier optical system. The point image intensity distribution 135 corresponding to the imaging plane of the third wavefront WF3 is observation It can be calculated using the pupil function of optical system 132. The calculation formula is shown below. In the calculation formula, WF3 is the third wavefront, and P is observation Pupil function of optical system 132, U 135 I is the wavefront in the image plane. 135 This represents the intensity distribution on the image plane.

number

[0174] The area groups differ in size between Figure 10 and Figure 5(c). The extent of the surrounding area 95 in Figure 10 is larger than the sum of the surrounding areas 95 and 96 in Figure 5(c). In Figure 10, all areas located to the side of the first area in a predetermined direction are considered as an area group, and the point image intensity distribution is calculated.

[0175] To accurately calculate the point image intensity distribution 135, it is preferable to use the refractive index distributions of all areas that make up the area group. In Figure 10, since area 93 is the first area, the refractive index distributions of all areas located between area 93 and the top surface 131 are used. Using the refractive index distributions of all areas located between the first area and the top surface of the sample allows for a more accurate calculation of the point image intensity distribution.

[0176] In the sample image generation apparatus of this embodiment, it is preferable that the processor, in the calculation process, determines whether or not the wavefront propagating through the sample has reached the outer edge of the sample in a predetermined direction. The second wavefront is the wavefront at the position where it is determined that the wavefront has reached the outer edge.

[0177] The first wavefront WF1 propagates from area 93 to the upper surface 131. Therefore, it is necessary to determine whether or not the first wavefront WF1 has reached the upper surface 131 in a predetermined direction. The second wavefront WF2 is located at the upper surface 131. Therefore, the second wavefront WF2 is the wavefront at the position where it was determined to have reached the upper surface 131.

[0178] In the sample image generation apparatus of this embodiment, the second wavefront is preferably the wavefront after passing through the sample and before reaching the virtual observation optical system.

[0179] In calculating the point image intensity distribution 135, the third wavefront WF3 passes through the observation optical system 132. The second wavefront WF2 passes through the observation optical system 132 If the wavefront has passed through the upper surface 131, the point image intensity distribution 135 cannot be calculated accurately. Therefore, the second wavefront WF2 must be a wavefront emitted from the upper surface 131 and before reaching the observation optical system 132.

[0180] In the sample image generation apparatus of this embodiment, it is preferable that the processor acquires the refractive index distribution for each sub-area obtained by further dividing the divided area in the second acquisition process. Furthermore, it is preferable that the processor calculates the point image intensity distribution of the sub-area using the refractive index distribution of the sub-area, and then calculates the point image intensity distribution of the area using the point image intensity distribution of the sub-area.

[0181] Figure 11 shows a refractive index image. Figure 11(a) shows the entire refractive index image. Figure 11(b) shows a portion of the refractive index image.

[0182] To accurately calculate the point image intensity distribution, a detailed refractive index distribution should be used. The more areas there are, or the smaller the area, the more detailed the refractive index distribution can be obtained. In refractive index image 90, if area 140 is the first area, then area 141 is included in the area group.

[0183] In Figure 11(a), area 141 is formed as a single region. Therefore, area 141 is divided into multiple regions. For example, as shown in Figure 11(b), area 141 is divided into 16 smaller areas 142. In this case, a refractive index distribution can be obtained for each smaller area 142, allowing for a detailed refractive index distribution. As a result, the point image intensity distribution can be accurately calculated.

[0184] In calculating wavefront propagation, the refractive index distribution of each sub-area 142 can be used. Furthermore, the average refractive index distribution can be determined from the refractive index distribution of each sub-area 142. Therefore, in calculating wavefront propagation, the average refractive index distribution can be used as the refractive index distribution of area 141.

[0185] Furthermore, since the refractive index distribution can be obtained for each small area, the point image intensity distribution can be calculated for each small area. Therefore, the point image intensity distribution of area 141 can be calculated using the point image intensity distribution of small area 142.

[0186] In the sample image generation apparatus of this embodiment, it is preferable that the processor performs a preliminary calculation process to preliminaryly calculate the point image intensity distribution for each area divided in the division process, and then performs a division process to set the intensity peak value of the preliminaryly calculated point image intensity distribution in areas where the intensity peak value is less than 1 / 5 of the reference value to be smaller than the intensity peak value of the preliminaryly calculated point image intensity distribution in areas where the intensity peak value is 1 / 5 or more of the reference value. The reference value is the intensity peak value of the point image intensity distribution when no sample is present.

[0187] Figure 12 shows a refractive index image. Figure 12(a) shows the state after the first division. Figure 12(b) shows the state after the second division.

[0188] The point image intensity distribution is calculated using the refractive index distribution. Therefore, the shape of the point image intensity distribution differs depending on the refractive index distribution. As shown in Figure 12(a), when area 150 is the first area, there is a row of areas located above area 151. Therefore, the refractive index distribution of this row of areas is considered when calculating the point image intensity distribution.

[0189] In contrast, if area 152 is the first area, then 10 rows of areas are located above area 152 on the upper surface 151 side. Therefore, when calculating the point image intensity distribution, the refractive index distribution of the 10 rows of areas will be taken into consideration.

[0190] The more areas included in an area group, the wider the refractive index distribution becomes. In this case, for example, the number of refractive index distributions to be calculated increases. The detail of the refractive index distribution is affected by the number of areas or the area of ​​each area. When the number of areas is small or the area of ​​each area is large, the wider the refractive index distribution becomes, the greater the difference between the shape of the point image intensity distribution and the ideal shape. In some cases, the difference between the shape of the point image intensity distribution and the ideal shape may be large in the area selected as the first area. In this case, it is preferable to calculate the point image intensity distribution in the selected area based on a detailed refractive index distribution.

[0191] Therefore, a preliminary calculation process is performed. In the preliminary calculation process, a preliminary point image intensity distribution is calculated for each area divided in the division process. Subsequently, based on the intensity peak values ​​of the preliminaryly calculated point image intensity distribution, the area is divided into target area 153 and non-target area 154, as shown in Figure 12(a).

[0192] Target area 153 is an area where the intensity peak value is less than 1 / 5 of the standard. Non-target area 154 is an area where the intensity peak value is 1 / 5 or greater than the standard. The standard is the intensity peak value of the point image intensity distribution when no sample is present.

[0193] Finally, as shown in Figure 12(b), each area of ​​the target area 153 is divided into multiple areas. As a result, the size of each area of ​​the target area 153 can be set to be smaller than the size of each area of ​​the non-target area 154. In Figure 12(b), each area of ​​the target area 153 is divided into four areas.

[0194] In the sample image generation apparatus of this embodiment, the processor performs an estimation process to estimate the sample, and in the division process, the estimated sample area is defined in a direction orthogonal to a predetermined direction, and the outer edge of the estimated sample belongs to Don't It is preferable to set the size of the area to be smaller than the size of an area that is not the area of ​​the estimated sample and does not include the outer edge of the estimated sample.

[0195] In this embodiment of the sample image generation device, XY images, XZ images, and YZ images can be generated from a group of XY images. Therefore, XY images, XZ images, and YZ images can be used as the first image. The XY image is an image in a plane orthogonal to a predetermined direction.

[0196] Even in XY images, there may be significant differences between the shape of the point image intensity distribution and the ideal shape in the area selected as the first area. In this case, it is preferable to calculate the point image intensity distribution in the selected area based on a detailed refractive index distribution.

[0197] A detailed refractive index distribution can be obtained by increasing the number of areas or decreasing the area size. The number of areas can be increased by dividing a single area. To increase the number of areas, for example, only the area representing the sample can be used as the area to be divided. In this case, the area to be divided can be extracted using the outer edge.

[0198] Area division is performed using the first image. The refractive index image corresponds to the first image, so we will use the refractive index image for explanation.

[0199] Figure 13 shows a refractive index image. Figure 13(a) shows the area division for the first example. Figure 13(b) shows the area division for the second example.

[0200] As shown in Figures 13(a) and 13(b), the refractive index image 160 has an internal area 161, an external area 162, and an outer edge area 163.

[0201] The internal area 161 is the area located inside the outer edge 164. The outer edge 164 does not belong to the internal area 161. The internal area 161 represents the specimen. The external area 162 is the area located outside the outer edge 164. The outer edge 164 does not belong to the external area 162. The external area 162 does not represent the specimen. The outer edge area 163 is the area to which the outer edge 164 of the specimen belongs.

[0202] In the first example of area division, as shown in Figure 13(a), each area of ​​the internal area 161 is divided into multiple areas. Each area of ​​the external area 162 and each area of ​​the outer edge area 163 are not divided into multiple areas.

[0203] In the second example of area division, as shown in Figure 13(b), each area of ​​the internal area 161 and each area of ​​the outer edge area 163 are divided into multiple areas. Each area of ​​the external area 162 is not divided into multiple areas.

[0204] In both the first and second examples, each area of ​​the internal area 161 is divided, but each area of ​​the external area 162 is not divided. The size of each area of ​​the internal area 161 is set to be smaller than the size of the external area 162.

[0205] The internal area 161 is the area representing the sample and does not include the outer border 164. The external area 162 is the area not representing the sample and does not include the outer border 164. In both the first and second examples, the size of the area representing the sample and not including the outer border is set to be smaller than the size of the area not representing the sample and not including the outer border.

[0206] Regarding the outer edge area 163, the division of the area differs between the first and second examples. In the first example, each area of ​​the outer edge area 163 is not divided into multiple areas. In contrast, in the second example, each area of ​​the outer edge area 163 is divided into multiple areas.

[0207] In the first example, the size of each area in the outer area 163 is larger than the size of each area in the inner area 161, and the same as the size of the outer area 162. In the second example, the size of each area in the outer area 163 is the same as the size of each area in the inner area 161, and smaller than the size of the outer area 162.

[0208] In both the first and second examples, the number of areas representing the sample can be increased. In this case, the point image intensity distribution can be calculated based on a detailed refractive index distribution. As a result, images can be recovered with high accuracy even in images on planes perpendicular to a given direction.

[0209] In this embodiment of the specimen image generation device, the calculation process differs slightly depending on the type of microscope used to acquire the XY image. The cases where a fluorescence microscope and an emission microscope are used to acquire the XY image will be described below.

[0210] Figure 14 is a flowchart of the calculation process. The first acquisition process in step S10, the division process in step S20, and the second acquisition process in step S30 have been completed. Therefore, the first image and the refractive index image have been divided into multiple areas. The number of divisions is Nm and Nn.

[0211] In step S100, the variables m and n are each set to 1.

[0212] In step S110, the first area ARE(m,n) is set. The first area ARE(m,n) is the area for which the point image intensity distribution is calculated. Each area in the first image corresponds one-to-one with each area in the refractive index image. Therefore, when the first area ARE(m,n) is set in the first image, the first area ARE(m,n) is also set in the refractive index image.

[0213] In step S120, the area group is determined. The area group is used to calculate the point image intensity distribution. Each area in the area group can be determined by the first area ARE(m,n) and the object-side numerical aperture of the observation optical system. The area group is determined using the refractive index image.

[0214] Instead of using the object's numerical aperture, scattered light may be used to determine each area in the area group. Alternatively, all areas located above the first area ARE(m,n) on the upper surface of the specimen may be considered as each area in the area group.

[0215] In step S130, the refractive index distribution in the area group is determined. The point image intensity distribution is calculated using the refractive index distribution of each area included in the area group. The refractive index image has a refractive index distribution. Since the area group is determined by the refractive index image, the refractive index distribution in the area group can be determined by determining the area group.

[0216] In step S140, a point light source is set. The point light source is set in the first area ARE(m,n).

[0217] In step S150, the first wavefront is set. In step S160, the second wavefront is calculated. In step S170, the third wavefront is calculated. In step S180, the point image intensity distribution is calculated. The first wavefront, second wavefront, third wavefront, and point image intensity distribution have already been explained, so their explanation is omitted here.

[0218] The microscopes used to acquire the XY images were a fluorescence microscope and an emission microscope. In the fluorescence microscope, the excitation light uniformly illuminates the focal plane and a wide area in front of and behind it. In the emission microscope, the excitation light does not irradiate the specimen. In either case, the excitation light intensity does not need to be considered. Therefore, the point image intensity distribution can be calculated using only the intensity distribution set for the point light source.

[0219] In step S190, a PSF image PSF(m,n) is generated. The point intensity distribution calculated in step S180 is the point intensity distribution in the first area ARE(m,n). The first area ARE(m,n) changes depending on the values ​​of variables m and n. Therefore, each time a point intensity distribution is calculated, the point intensity distribution is saved in the PSF image PSF(m,n).

[0220] In step S200, the first image PIC1(m,n) is acquired. In step S210, the second image PIC2(m,n) is generated. The first and second images have already been explained, so their explanation will be omitted here. The acquisition of the first image PIC1(m,n) can be done between steps S110 and S220.

[0221] In step S220, the value of variable n is compared with the number of divisions Nn. If the value of variable n does not match the number of divisions Nn, step S230 is performed. In step S230, 1 is added to the value of variable n.

[0222] When step S230 is completed, the process returns to step S110. In step S230, the value of the variable n is increased by 1. Therefore, the point image intensity distribution is calculated for the new first area ARE(m,n).

[0223] If the value of variable n matches the number of divisions Nn in step S220, step S240 is executed. In step S240, the value of variable m is compared with the number of divisions Nm. If the value of variable m does not match the number of divisions Nm, step S250 is executed. In step S250, 1 is added to the value of variable m.

[0224] When step S250 is completed, the process returns to step S110. In step S250, the value of the variable m is increased by 1. Therefore, the point image intensity distribution is calculated for the new first area ARE(m,n).

[0225] If the value of variable m matches the number of divisions Nm in step S240, step S260 is executed. In step S260, a third image is generated. The third image has already been explained, so its explanation is omitted here.

[0226] In the sample image generation apparatus of this embodiment, it is preferable that the processor calculates the excitation light intensity at the position of a set point light source during the calculation process, calculates the fluorescence intensity distribution using the calculated intensity distribution and the calculated excitation light intensity, and calculates the point image intensity distribution of the first area using the calculated fluorescence intensity distribution.

[0227] This section describes the cases where LSM, two-photon microscope, and sheet illumination microscope were used to acquire XY images.

[0228] In LSM and two-photon microscopes, the excitation light illuminates a single point on the focal plane. For example, in the refractive index image 90 shown in Figure 10, the excitation light is incident from the upper surface 131 toward the point light source 130. In this case, the excitation light is affected by the refractive index distribution, just like the light emitted from the point light source 130.

[0229] Therefore, the excitation light intensity must be considered when using LSM and two-photon microscopy. The excitation light intensity can be determined as follows.

[0230] In Step 1, the amplitude distribution of the wavefront with the excitation light focusing position as the wave source is calculated. At this time, the refractive index distribution in refractive index image 90 is not used. The excitation light focusing position is a position within the area where the point light source is set, i.e., a position within the first area.

[0231] In Step 2, the amplitude distribution at the excitation light focus position is calculated using the wavefront amplitude distribution calculated in Step 1. The refractive index distribution in refractive index image 90 is used for this calculation.

[0232] In step 3, the excitation light intensity Iex(Pi) at the point source position is calculated using the amplitude distribution at the focal point of the excitation light calculated in step 2.

[0233] The specified intensity distribution IFL(Pi) is the intensity distribution when the microscope used to acquire the XY image is a fluorescence microscope or an emission microscope. The specified intensity distribution IFL(Pi) is the intensity distribution on the image plane calculated based on the intensity distribution set for a point light source.

[0234] If the microscope used to acquire the XY image is a fluorescence microscope, the given intensity distribution IFL(Pi) is the fluorescence intensity distribution. When using an LSM, two-photon microscope, or sheet illumination microscope, a fluorescence image is formed in the same way as with a fluorescence microscope. Therefore, even when using an LSM, two-photon microscope, or sheet illumination microscope, the given intensity distribution IFL(Pi) is the fluorescence intensity distribution.

[0235] The intensity distribution F1(Pi) is the intensity distribution at the image plane when using an LSM. When using an LSM, the intensity distribution F1(Pi) is affected by the excitation light intensity. The intensity distribution F1(Pi) is expressed by the following equation (4). F1(Pi) = Iex(Pi) × IFL(Pi) (4)

[0236] The intensity distribution F2(Pi) is the intensity distribution on the image plane when using a two-photon microscope. When using a two-photon microscope, the intensity distribution F2(Pi) is affected by the excitation light intensity. The intensity distribution F2(Pi) is expressed by the following equation (5). F2(Pi)=Iex(Pi) 2 ×IFL(Pi) (5)

[0237] If an LSM is used to acquire XY images, the point image intensity distribution can be calculated from the intensity distribution F1(Pi). If a two-photon microscope is used to acquire XY images, the point image intensity distribution can be calculated from the intensity distribution F2(Pi).

[0238] Since the given intensity distribution IFL(Pi) is a fluorescence intensity distribution, intensity distributions F1(Pi) and F2(Pi) are also fluorescence intensity distributions.

[0239] In a sheet illumination microscope, the excitation light illuminates the focal plane. For example, in the refractive index image 90 shown in Figure 10, the light is incident on a single block layer located at the focal plane FP. In this case, the refractive index distribution of the single block layer has an effect.

[0240] Therefore, in sheet illumination microscopy, the excitation light intensity must be considered. The excitation light intensity can be determined as follows.

[0241] Step 1 calculates the excitation light intensity in one block layer. The refractive index distribution in refractive index image 90 is used for this calculation.

[0242] In step 2, the excitation light intensity I'ex(Pi) within a single block layer is calculated in the area where the point light source is set, i.e., within the first area. As described above, the predetermined intensity distribution IFL(Pi) is the fluorescence intensity distribution.

[0243] The intensity distribution F3(Pi) is the intensity distribution on the image plane when using a sheet illumination microscope. When using a sheet illumination microscope, the intensity distribution F3(Pi) is affected by the excitation light intensity. The intensity distribution F3(Pi) is expressed by the following equation (6). F3(Pi) = I'ex(Pi) × IFL(Pi) (6)

[0244] When a sheet illumination microscope is used to acquire XY images, the point image intensity distribution can be calculated from the intensity distribution F3(Pi). Since the given intensity distribution IFL(Pi) is a fluorescence intensity distribution, the intensity distribution F3(Pi) is also a fluorescence intensity distribution.

[0245] In the sample image generation apparatus of this embodiment, it is preferable that the processor calculates the excitation light intensity using the refractive index distribution of the excitation light wavelength during the calculation process.

[0246] The refractive index distribution is used in the calculation of a predetermined intensity distribution IFL(Pi), the photon emission intensity Iex(Pi), and the excitation light intensity I'ex(Pi). The predetermined intensity distribution IFL(Pi) is the fluorescence intensity distribution. The wavelength of fluorescence and the wavelength of excitation light are different. Therefore, the refractive index distribution at the fluorescence wavelength is used in the calculation of the predetermined intensity distribution IFL(Pi). The refractive index distribution at the excitation light wavelength is used in the calculation of the excitation light intensity Iex(Pi).

[0247] In the sample image generation apparatus of this embodiment, the first image is an image obtained by taking a photograph with an apparatus equipped with a confocal pinhole, and in the calculation process, the processor preferably calculates the light intensity passing through the confocal pinhole using the calculated intensity distribution, calculates the fluorescence intensity using the calculated excitation light intensity and the calculated light intensity, and calculates the point image intensity distribution of the first area using the calculated fluorescence intensity.

[0248] In a non-confocal LSM, there is no pinhole in the image plane. Therefore, the intensity distribution F1(Pi) is expressed by equation (4). In contrast, in a confocal LSM, a pinhole is located in the image plane. Therefore, a given intensity distribution IFL(Pi) is affected by the aperture of the pinhole.

[0249] FIG. 15 is a diagram showing a predetermined intensity distribution and the transmission characteristics of a pinhole. The dashed line indicates the predetermined intensity distribution, and the solid line indicates the transmission characteristics of the pinhole. In the pinhole, light passes only through the aperture. Therefore, the intensity distribution I’FL(Pi) of the light emitted from the confocal pinhole is determined by the intensity distribution within the range of the rectangle indicated by the solid line among the predetermined intensity distribution IFL(Pi).

[0250] The intensity distribution F4(Pi) is the intensity distribution on the image plane when a confocal type LSM is used. When a confocal type LSM is used, the intensity distribution F4(Pi) is affected by the excitation light intensity. The intensity distribution F4(Pi) is represented by the following equation (7). F4(Pi)=Iex(Pi)×I’FL(Pi) (7)

[0251] When a confocal type LSM is used for acquiring an XY image, the point image intensity distribution may be calculated from the intensity distribution F4(Pi). Since the predetermined intensity distribution IFL(Pi) is a fluorescence intensity distribution, the intensity distribution F4(Pi) is also a fluorescence intensity distribution.

[0252] In the specimen image generation apparatus of the present embodiment, it is preferable that the processor calculates the second wavefront using the beam propagation method in the calculation process.

[0253] As shown in FIG. 10, the positions of the first wavefront WF1, the second wavefront WF2, and the third wavefront WF3 are different from each other. The wavefront propagates between the position of the first wavefront WF1 and the position of the second wavefront WF2, and between the position of the second wavefront WF2 and the position of the third wavefront WF③. The beam propagation method may be used for calculating the second wavefront WF2 and the third wavefront WF3.

[0254] In the beam propagation method, the object model is replaced with multiple thin layers. Then, the wavefront changes as light passes through each layer are calculated sequentially to compute an image of the object model. The beam propagation method is disclosed, for example, in "High-resolution 3D refractive index microscopy of multiple-scattering samples from intensity images" Optica, Vol. 6, No. 9, pp.1211-1219 (2019).

[0255] The specimen image generation system of this embodiment comprises an observation optical system for forming an optical image of a specimen, an image sensor for capturing the optical image, and the specimen image generation device of this embodiment.

[0256] According to the sample image generation system of this embodiment, images can be recovered with high accuracy.

[0257] The sample image generation system of this embodiment includes a memory and a processor. The processor performs a first acquisition process to acquire a first image from the memory, a division process to divide the acquired first image into multiple areas, a second acquisition process to acquire the refractive index distribution of the sample from the memory, a calculation process to calculate the point image intensity distribution for each of the divided areas using the acquired refractive index distribution, a first generation process to generate a second image corresponding to each area using the point image intensity distribution calculated for each area, synthesizes the second images corresponding to each area to generate a third image corresponding to the first image, and in the calculation process, the point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first image is an image obtained by photographing a sample, the predetermined direction in the first image is the direction in which the virtual observation optical system exists within the optical axis direction of the virtual observation optical system, the first area is the area for which the point image intensity distribution is calculated, and the area group consists of multiple areas located inside the range radiated by light rays, starting from the first area in a predetermined direction, and includes areas outside the range extended from the first area in a predetermined direction. The processor performs machine learning processing to train the AI ​​model, and this machine learning processing trains the AI ​​model on multiple datasets. The datasets include the first image and the training data corresponding to the first image, and the training data corresponding to the first image is the second image corresponding to the first image.

[0258] Sample image generation in this embodiment system Next, the third image is generated from the first image. The first image is the degraded image, and the third image is the restored image. If we consider the third image as training data, then the first and third images can be used as data for machine learning. Hereafter, the first image will be referred to as the image before improvement, and the third image as the improved image.

[0259] The improved images can be generated using an AI model trained with supervised machine learning (hereinafter referred to as "supervised ML").

[0260] AI models perform inference based on patterns found during data analysis in the training process, providing the ability to execute tasks on a computer system without explicit programming.

[0261] AI models can be trained continuously or periodically before performing inference processing.

[0262] Supervised machine learning (ML) AI models include algorithms that are trained on existing sample data and training data, and then make predictions about new data. Training data is also called teacher data.

[0263] Such algorithms work by building an AI model from sample and training data to make data-driven predictions or decisions that are expressed as results.

[0264] In supervised machine learning, when training is performed, sample data and training data are input, and the system learns a function that best approximates the relationship between input and output. When the trained AI model performs inference, it implements the same function when new data is input and generates the corresponding output.

[0265] Examples of commonly used supervised machine learning algorithms include logistic regression (LR), naive Bayes, random forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and support vector machines (SVM).

[0266] The training process in this embodiment can perform supervised ML processing. The training process trains or learns the AI ​​model.

[0267] When the training process is executed, a sufficient number of datasets are input into the input layer of the AI ​​model, and these datasets are propagated through the AI ​​model to the output layer.

[0268] FIG. 16 is a diagram showing a training process. The dataset includes an image before improvement and an improved image. The image before improvement is sample data. The improved image is training data or teacher data corresponding to the sample data. In FIG. 16, the sample data is shown as Image 1, Image 2, etc. The improved image is Training Data 1, Training Data 2, etc.

[0269] In the training process, optimal parameters for generating estimated data from the sample data are searched for and updated using, for example, a loss function. Estimated data is generated for the input sample data, and the difference between the generated estimated data and the training data is evaluated using a loss function, and parameters that minimize the value of the loss function are searched for.

[0270] The inference process of this embodiment can execute an inference process that outputs inference data when data for which new inference is desired is input to the trained AI model.

[0271] When the inference process is executed, an image before improvement is input to the input layer of the AI model and propagated to the output layer through the AI model.

[0272] By executing the inference process, an improved image can be generated from the image before improvement.

[0273] FIG. 17 is a diagram showing a specimen image generation system of this embodiment. FIG. 17(a) is a diagram showing a specimen image generation system of the first example. FIG. 17(b) is a diagram showing a specimen image generation system of the second example. FIG. 17(c) is a diagram showing a specimen image generation system of the third example.

[0274] As shown in Figure 17(a), in the first example of the sample image generation system, the sample image generation system 170 consists only of the sample image generation device of this embodiment. In this case, the processor 3 of the sample image generation device (hereinafter referred to as the "first processor") can perform training and inference processing.

[0275] The sample image generation device 1 may include a first processor and a second processor. The second processor is different from the first processor. Training and inference processes can be performed on the second processor.

[0276] The sample image generation device 1 may include a first processor, a second processor, and a third processor. The third processor is different from the first and second processors. The second processor can perform training processing, and the third processor can perform inference processing.

[0277] The memory 2 of the sample image generation device 1 stores the image before improvement used in the training process, the improved image, and the image before improvement used in the inference process.

[0278] As shown in Figure 17(b), in the second example of the sample image generation system, the sample image generation system 180 consists of the sample image generation device 1 of this embodiment and the learning inference device 190. The learning inference device 190 includes a memory 191 and a processor 192.

[0279] The learning inference device 190 can perform training and inference processes. In this case, the learning inference device 190 includes memory and one or more processors. The inference process can be performed on the same processor as the training process. The inference process may also be performed on a different processor than the training process.

[0280] The learning inference unit's memory 191 stores the image before improvement used in the training process, the improved image, and the image before improvement used in the inference process.

[0281] As shown in Figure 17(c), in the third example of the sample image generation system, the sample image generation system 200 consists of the sample image generation device 1 of this embodiment, a learning device 210, and an inference device 220. The learning device 210 performs training processing, and the inference device 220 performs inference processing.

[0282] The learning device 210 includes a memory 211 and a processor 212. The inference device 220 includes a memory 221 and a processor 222. The processor 212 of the learning device 210 can perform training processing, and the processor 222 of the inference device 220 can perform inference processing.

[0283] The memory 211 of the learning device 210 stores the image before improvement and the improved image used in the training process. The memory 221 of the inference device 220 stores the image before improvement used in the inference process.

[0284] The learning inference device 190 and the learning device 210 described above receive data used for training from the sample image generation device 1 via communication or a recording medium such as a USB memory, and store it in the memory provided by each device.

[0285] According to the sample image generation system of this embodiment, images can be recovered with high accuracy.

[0286] The sample image generation method of this embodiment is a sample image generation method that uses a first image and the refractive index distribution of a sample, and involves executing a first acquisition process to acquire a first image from memory, executing a division process to divide the acquired first image into a plurality of areas, executing a second acquisition process to acquire the refractive index distribution of the sample from memory, executing a calculation process to calculate the point image intensity distribution for each of the divided areas using the acquired refractive index distribution, executing a first generation process to generate a second image corresponding to each area using the point image intensity distribution calculated for each area, synthesizing the second images corresponding to each area to generate a third image corresponding to the first image, and in the calculation process, the point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first image is an image obtained by photographing a sample, the predetermined direction in the first image is the direction in which the virtual observation optical system exists within the optical axis direction of the virtual observation optical system, the first area is the area for which the point image intensity distribution is to be calculated, and the area group consists of a plurality of areas inside the range in which light rays radiate in a predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in a predetermined direction.

[0287] According to the sample image generation method of this embodiment, images can be recovered with high accuracy.

[0288] The recording medium of this embodiment is a computer-readable recording medium that stores a program for generating a sample image, and causes the computer to execute a first acquisition process for acquiring a first image from memory, a division process for dividing the acquired first image into a plurality of areas, a second acquisition process for acquiring the refractive index distribution of the sample from memory, a calculation process for calculating the point image intensity distribution for each of the divided areas using the acquired refractive index distribution, a first generation process for generating a second image corresponding to each area using the point image intensity distribution calculated for each area, and a process for synthesizing the second images corresponding to each area to generate a third image corresponding to the first image, wherein in the calculation process, the point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first image is an image obtained by photographing a sample, the predetermined direction in the first image is the direction in which the virtual observation optical system exists within the optical axis direction of the virtual observation optical system, the first area is the area for which the point image intensity distribution is calculated, and the area group consists of a plurality of areas inside the range in which light rays radiate in a predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in a predetermined direction.

[0289] According to the recording medium of this embodiment, images can be recovered with high accuracy. [Industrial applicability]

[0290] The present invention is suitable for a specimen image generation device, specimen image generation method, specimen image generation system, and recording medium that can recover images with high accuracy. [Explanation of Symbols]

[0291] 1. Sample image generation device 2 memory 3 processors 4 Input section 10 Microscope Systems 20 Microscopes 21 Main unit 22 Objective lenses 23 stages 24 Incident Illumination Device 25 Imaging Unit 26 controllers 27 specimens 30 Processing Unit 31 Input section 32 memory 33 processors 34 Output section 40 Estimation device 41 Input section 42 memory 43 processors 44 Output section 50 specimens 51 Observation Optical System 51' Observation Optical System 52 Optical axis 53 XY image group 60 Optical image 70 specimens 71 Cytoplasm 72 cell nucleus 80 First image 81 Image of a cell nucleus Areas 82, 83, and 84 90 Refractive Index Image 90a top surface 90b Bottom Areas 91, 92, 93, 97, and 98 94 Central Area Areas 95 and 96 100, 104 rays 101 Optical axis 102 predetermined direction 103 Non-designated direction 110 PSF images Areas 111, 112, and 113 120 Third image 130 point light source 131 Top surface 132 Observation Optical System 133 Objective lens 134 Imaging lens 135 Point spread intensity distribution Areas 140 and 141 142 Small Areas Areas 150 and 152 151 Top surface 153 Target Areas 154 Non-applicable areas 160 refractive index image 161 Internal Area 162 External Area 163 Outer Area 164 Outer edge of the specimen 170, 180, 200 Sample Image Generation System 190 Learning Inference Machine 191, 211, 221 memory 192, 212, 222 processors 210 Learning device 211 memory 212 processors 220 Reasoning device 221 memory 222 processors OBJ specimen AX optical axis OS optical system IP image plane IMG optical image PIC optical image FP focal plane DEG, PSF, REC area WF1 1st wave WF2 2nd wave WF3 3rd wave

Claims

1. Equipped with memory and a processor, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. The aforementioned processor, A first acquisition process is executed to acquire the first image from the memory, A division process is performed to divide the acquired first image into multiple areas. A second acquisition process is executed to obtain the refractive index distribution of the sample from the memory. Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each of the divided areas. In the calculation process described above, A point light source is set within the first area, and the point image intensity distribution of the first area is calculated using the first wavefront with the set point light source as the wave source. Using the first wavefront and the refractive index distribution corresponding to each area included in the area group, the second wavefront is calculated. The second wavefront is a wavefront obtained by propagating the sample in the predetermined direction, It is determined whether the wavefront propagating through the sample has reached the outer edge of the sample in the predetermined direction. The second wavefront is the wavefront at the position where it was determined to have reached the outer edge. Using the calculated second wavefront, the intensity distribution corresponding to the third wavefront is calculated. The third wavefront is the wavefront at the position of the focal plane of the virtual observation optical system, Using the calculated intensity distribution, the point image intensity distribution of the first area is calculated. The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first area is the area for which the point image intensity distribution is calculated, The group of areas consists of a plurality of areas located inside the range from which the light ray is emitted in the predetermined direction, starting from the first area, and includes areas located outside the range extended from the first area in the predetermined direction. A first generation process is performed to divide the first image into multiple areas and generate a second image corresponding to each area using the point image intensity distribution calculated for each area. The second images corresponding to each of the aforementioned areas are combined to generate a third image corresponding to the first image. A specimen image generation device characterized in that the third image is a recovered image of the first image.

2. In the second acquisition process, the processor acquires the refractive index distribution for each sub-area obtained by further dividing the divided area. In the calculation process, the aforementioned processor Using the refractive index distribution of the small area, the point image intensity distribution of the small area is calculated. The sample image generation device according to claim 1, characterized in that it calculates the point image intensity distribution of the area using the point image intensity distribution of the small area.

3. The aforementioned processor, A preliminary calculation process is performed to provisionally calculate the point image intensity distribution for each of the areas divided in the aforementioned division process. The division process is executed to set the area where the peak intensity value of the provisionally calculated point image intensity distribution is less than 1 / 5 of the standard to be smaller than the area where the peak intensity value of the provisionally calculated point image intensity distribution is 1 / 5 or more of the standard. The specimen image generating apparatus according to claim 1, characterized in that the aforementioned standard is the peak intensity value of the point image intensity distribution when the specimen is absent.

4. The aforementioned processor, An estimation process is performed to estimate the aforementioned sample. In the above partitioning process, In a direction perpendicular to the predetermined direction, The specimen image generating apparatus according to claim 1, characterized in that the size of the area which is the estimated area of ​​the specimen and to which the outer edge of the estimated specimen does not belong is set to be smaller than the size of the area which is not the estimated area of ​​the specimen and to which the outer edge of the estimated specimen does not belong.

5. In the calculation process, the aforementioned processor The excitation light intensity at the set point light source position is calculated, Using the calculated intensity distribution and the calculated excitation light intensity, the fluorescence intensity distribution is calculated. The specimen image generating apparatus according to claim 1, characterized in that it calculates the point image intensity distribution of the first area using the calculated fluorescence intensity distribution.

6. The sample image generating apparatus according to claim 5, characterized in that the processor calculates the excitation light intensity using the refractive index distribution of the excitation light wavelength in the calculation process.

7. The first image above was obtained by taking a picture with a device equipped with a confocal pinhole. In the calculation process, the aforementioned processor Using the calculated intensity distribution, the light intensity passing through the confocal pinhole is calculated. The fluorescence intensity is calculated using the calculated excitation light intensity and the calculated light intensity. The specimen image generating apparatus according to claim 5, characterized in that it calculates the point image intensity distribution of the first area using the calculated fluorescence intensity.

8. The sample image generation apparatus according to claim 1, characterized in that the processor calculates the second wavefront using the beam propagation method in the calculation process.

9. An observation optical system that forms an optical image of a specimen, An image sensor for capturing the aforementioned optical image, A specimen image generation system characterized by comprising the specimen image generation device described in claim 1.

10. Equipped with memory and a processor, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. The aforementioned processor, A first acquisition process is executed to acquire the first image from the memory, A division process is performed to divide the acquired first image into multiple areas. A second acquisition process is executed to obtain the refractive index distribution of the sample from the memory. Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each of the divided areas. In the calculation process described above, A point light source is set within the first area, and the point image intensity distribution of the first area is calculated using the first wavefront with the set point light source as the wave source. Using the first wavefront and the refractive index distribution corresponding to each area included in the area group, the second wavefront is calculated. The second wavefront is a wavefront obtained by propagating the sample in the predetermined direction, It is determined whether the wavefront propagating through the sample has reached the outer edge of the sample in the predetermined direction. The second wavefront is the wavefront at the position where it was determined to have reached the outer edge. Using the calculated second wavefront, the intensity distribution corresponding to the third wavefront is calculated. The third wavefront is the wavefront at the position of the focal plane of the virtual observation optical system, Using the calculated intensity distribution, the point image intensity distribution of the first area is calculated. The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first area is the area for which the point image intensity distribution is calculated, The group of areas consists of a plurality of areas located inside the range from which the light ray is emitted in the predetermined direction, starting from the first area, and includes areas located outside the range extended from the first area in the predetermined direction. A first generation process is performed to divide the first image into multiple areas and generate a second image corresponding to each area using the point image intensity distribution calculated for each area. The second images corresponding to each of the aforementioned areas are combined to generate a third image corresponding to the first image. The third image is a recovered image of the first image, The aforementioned processor performs machine learning processing to train the AI ​​model. The aforementioned machine learning process involves training the AI ​​model on multiple datasets. The dataset includes the first image and training data corresponding to the first image. A sample image generation system characterized in that the training data corresponding to the first image is the second image corresponding to the first image.

11. A method for generating a sample image using the refractive index distribution of a first image and a sample, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. The first acquisition process is executed to acquire the first image from memory. A division process is performed to divide the acquired first image into multiple areas. A second acquisition process is executed to obtain the refractive index distribution of the sample from the memory. Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each of the divided areas. In the calculation process described above, A point light source is set within the first area, and the point image intensity distribution of the first area is calculated using the first wavefront with the set point light source as the wave source. Using the first wavefront and the refractive index distribution corresponding to each area included in the area group, the second wavefront is calculated. The second wavefront is a wavefront obtained by propagating the sample in the predetermined direction, It is determined whether the wavefront propagating through the sample has reached the outer edge of the sample in the predetermined direction. The second wavefront is the wavefront at the position where it was determined to have reached the outer edge. Using the calculated second wavefront, the intensity distribution corresponding to the third wavefront is calculated. The third wavefront is the wavefront at the position of the focal plane of the virtual observation optical system, Using the calculated intensity distribution, the point image intensity distribution of the first area is calculated. The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first area is the area for which the point image intensity distribution is calculated, The group of areas consists of a plurality of areas located inside the range from which the light ray is emitted in the predetermined direction, starting from the first area, and includes areas located outside the range extended from the first area in the predetermined direction. A first generation process is performed to divide the first image into multiple areas and generate a second image corresponding to each area using the point image intensity distribution calculated for each area. The second images corresponding to each of the aforementioned areas are combined to generate a third image corresponding to the first image. A method for generating a sample image, characterized in that the third image is a recovered image of the first image.

12. A computer-readable recording medium containing a program for generating sample images, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. To the aforementioned computer, A first acquisition process to acquire the first image from memory, A division process that divides the acquired first image into multiple areas, A second acquisition process for obtaining the refractive index distribution of the sample from the memory, Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each of the divided areas. In the calculation process described above, A point light source is set within the first area, and the point image intensity distribution of the first area is calculated using the first wavefront with the set point light source as the wave source. Using the first wavefront and the refractive index distribution corresponding to each area included in the area group, the second wavefront is calculated. The second wavefront is a wavefront obtained by propagating the sample in the predetermined direction, It is determined whether the wavefront propagating through the sample has reached the outer edge of the sample in the predetermined direction. The second wavefront is the wavefront at the position where it was determined to have reached the outer edge. Using the calculated second wavefront, the intensity distribution corresponding to the third wavefront is calculated. The third wavefront is the wavefront at the position of the focal plane of the virtual observation optical system, Using the calculated intensity distribution, the point image intensity distribution of the first area is calculated. The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first area is the area for which the point image intensity distribution is calculated, The group of areas consists of a plurality of areas located inside the range from which the light ray is emitted in the predetermined direction, starting from the first area, and includes areas located outside the range extended from the first area in the predetermined direction. A first generation process is performed to divide the first image into multiple areas and generate a second image corresponding to each area using the point image intensity distribution calculated for each area. The second images corresponding to each of the aforementioned areas are combined to generate a third image corresponding to the first image. A computer-readable recording medium characterized in that the third image is a recovered image of the first image.

13. A device comprising memory and a processor, The first image is an image obtained by photographing the specimen. The predetermined direction in the first image is the direction in which the virtual observation optical system exists, within the optical axis direction of the virtual observation optical system. The aforementioned processor, A first acquisition process is executed to acquire the first image from the memory, A division process is performed to divide the acquired first image into multiple areas. A second acquisition process is executed to obtain the refractive index distribution of the sample from the memory. Using the acquired refractive index distribution, a calculation process is performed to calculate the point image intensity distribution for each of the divided areas. In the calculation process described above, A point light source is set within the first area, and the point image intensity distribution of the first area is calculated using the first wavefront with the set point light source as the wave source. Using the first wavefront and the refractive index distribution corresponding to each area included in the area group, the second wavefront is calculated. The second wavefront is a wavefront obtained by propagating the sample in the predetermined direction, The second wavefront is the wavefront after passing through the sample and before reaching the virtual observation optical system. Using the calculated second wavefront, the intensity distribution corresponding to the third wavefront is calculated. The third wavefront is the wavefront at the position of the focal plane of the virtual observation optical system, Using the calculated intensity distribution, the point image intensity distribution of the first area is calculated. The point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. The first area is the area for which the point image intensity distribution is calculated, The group of areas consists of a plurality of areas located inside the range from which the light ray is emitted in the predetermined direction, starting from the first area, and includes areas located outside the range extended from the first area in the predetermined direction. A first generation process is performed to divide the first image into multiple areas and generate a second image corresponding to each area using the point image intensity distribution calculated for each area. The second images corresponding to each of the aforementioned areas are combined to generate a third image corresponding to the first image. A specimen image generation device characterized in that the third image is a recovered image of the first image.

14. The processor, in the calculation process, The excitation light intensity at the set point light source position is calculated, Using the calculated intensity distribution and the calculated excitation light intensity, the fluorescence intensity distribution is calculated. The specimen image generating apparatus according to claim 13, characterized in that it calculates the point image intensity distribution of the first area using the calculated fluorescence intensity distribution.

Images