Refractive index distribution generation device, refractive index distribution generation method, refractive index distribution generation system, and recording medium

Abstract

Provided is a refractive index distribution generation device capable of improving the accuracy of a refractive index distribution of even a thick sample.　This refractive index distribution generation device comprises a processor and a memory. The processor executes refractive index distribution generation processing of generating a refractive index distribution for a processing target image. The refractive index distribution generation processing includes: input processing of inputting, from a memory, the processing target image, first refractive index information indicating the refractive index of a first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure; and setting processing of setting each refractive index constituting the refractive index distribution. The setting processing includes: first setting processing of setting a refractive index, which is based on the first refractive index information, at a position corresponding to a first image region of the processing target image on the basis of a signal strength; and second setting processing of setting a refractive index, which is based on the second refractive index information, at a position corresponding to an image region different from the first image region of the processing target image. The first image region is an image region corresponding to the first structure.

Description

Technical Field

[0001] The present invention relates to a refractive index distribution generation device, a refractive index distribution generation method, a refractive index distribution generation system, and a recording medium.

Background Art

[0002] A technique for estimating the refractive index distribution of a specimen using computational imaging is disclosed in Non-Patent Document 1. In this estimation technique, a specimen placed on glass is illuminated by epi-illumination. Then, two lights incident on the objective lens are detected by an image sensor. One light is the light reflected by the glass after being scattered by the specimen. The other light is the light scattered by the specimen after being reflected by the glass.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the estimation technique, the two lights incident on the objective lens are both lights reflected by the glass. The intensity of the light reflected by the glass is very small. Therefore, it is difficult to estimate the refractive index distribution with high accuracy for a thick specimen.

[0005] The present invention has been made in view of these problems, and aims to provide a refractive index distribution generating device, a refractive index distribution generating method, a refractive index distribution generating system, and a recording medium that can improve the accuracy of the refractive index distribution even with thick samples. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objectives, the refractive index distribution generating apparatus according to at least some embodiments of the present invention is as follows: A processor composed of hardware, memory composed of hardware, and equipped, The processor is, The refractive index distribution generation process is executed to generate the refractive index distribution corresponding to the image to be processed. The refractive index distribution generation process is, The input process involves inputting the image to be processed, first refractive index information indicating the refractive index of the first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure, from memory. This includes a setting process for setting each refractive index that makes up the refractive index distribution, The setup process is as follows: A first setting process in which the refractive index based on the first refractive index information is set at a position corresponding to the first image region of the image to be processed, based on the signal intensity of the unit pixel of the image to be processed, The process includes a second setting process which sets the refractive index based on the second refractive index information to a position corresponding to an image region different from the first image region of the image to be processed, The first image region is the image region corresponding to the first structure. A unit pixel consists of one or more pixels. The processor is characterized by using an image of a sample as the image to be processed and performing refractive index distribution generation processing.

[0007] Furthermore, at least some embodiments of the refractive index distribution generation system of the present invention are An observation optical system that forms an optical image of a specimen, An image sensor that captures an optical image, The present invention is characterized by comprising the refractive index distribution generating device described in claim 1.

[0008] Furthermore, at least some embodiments of the refractive index distribution generation system of the present invention are A processor composed of hardware, memory composed of hardware, and equipped, The processor is, The refractive index distribution generation process is executed to generate the refractive index distribution corresponding to the image to be processed. The refractive index distribution generation process is, The input process involves inputting the image to be processed, first refractive index information indicating the refractive index of the first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure, from memory. This includes a setting process for setting each refractive index that makes up the refractive index distribution, The setup process is as follows: A first setting process in which the refractive index based on the first refractive index information is set at a position corresponding to the first image region of the image to be processed, based on the signal intensity of the unit pixel of the image to be processed, The process includes a second setting process which sets the refractive index based on the second refractive index information to a position corresponding to an image region different from the first image region of the image to be processed, The first image region is the image region corresponding to the first structure. A unit pixel consists of one or more pixels. The processor uses the image of the sample as the image to be processed and performs refractive index distribution generation processing. The processor performs machine learning processes to train the AI ​​model. Machine learning processing involves training an AI model on multiple datasets. The dataset includes the images to be processed and the training data corresponding to the images to be processed. The training data is characterized by being a refractive index distribution generated by a refractive index distribution generation process.

[0009] Furthermore, at least some embodiments of the refractive index distribution generation method of the present invention are A method for generating a refractive index distribution corresponding to an image to be processed, Input a processing target image, first refractive index information indicating the refractive index of a first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure. Set the refractive index based on the first refractive index information at a position corresponding to the first image region of the processing target image based on the signal intensity of a unit pixel of the processing target image. Set the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the processing target image. The first image region is an image region corresponding to the first structure. A unit pixel is composed of one pixel or a plurality of pixels. The processing target image is characterized by being an image obtained by photographing a specimen.

[0010] Also, a recording medium according to at least some embodiments of the present invention is A computer-readable recording medium recording a program for generating a specimen image, An input process for inputting a processing target image, first refractive index information indicating the refractive index of a first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure from a memory, A setting process for setting each refractive index constituting a refractive index distribution is executed. In the setting process, A first setting process for setting the refractive index based on the first refractive index information at a position corresponding to the first image region of the processing target image based on the signal intensity of a unit pixel of the processing target image, A second setting process for setting the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the processing target image is executed. The first image region is an image region corresponding to the first structure. A unit pixel is composed of one pixel or a plurality of pixels. The refractive index distribution generation process is executed with an image obtained by photographing a specimen as the processing target image.

Advantages of the Invention

[0011] According to the present invention, it is possible to provide a refractive index distribution generation device, a refractive index distribution generation method, a refractive index distribution generation system, and a recording medium that can improve the accuracy of the refractive index distribution even with thick specimens. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows the refractive index distribution generation device and microscope system of this embodiment. [Figure 2] This figure shows the specimen, optical image, image of the optical image, and XY image set. [Figure 3] This figure shows the image to be processed and the refractive index distribution image. [Figure 4] This is a flowchart of the processing performed by the processor. [Figure 5] This figure shows the image to be processed and the refractive index distribution image. [Figure 6] This is a flowchart of the processing performed by the processor. [Figure 7] This is a diagram showing the image to be processed. [Figure 8] This is a flowchart of the processing performed by the processor. [Figure 9] This figure shows the image to be processed and the refractive index distribution image. [Figure 10] This is a flowchart of the processing performed by the processor. [Figure 11] This is a diagram showing the image to be processed. [Figure 12] This is a diagram showing the image to be processed. [Figure 13] This is a flowchart of the processing performed by the processor. [Figure 14] This is a diagram showing the image to be processed. [Figure 15] This is a flowchart of the processing performed by the processor. [Figure 16] This is a flowchart of the processing performed by the processor. [Figure 17] This figure shows the specimen, the optical image, and the first image. [Figure 18]This figure shows the first image, the refractive index image, the first area, and the group of areas. [Figure 19] This is a diagram showing the grouping of areas. [Figure 20] This figure shows the first image, the refractive index image, and the PSF image. [Figure 21] This is a diagram showing the first, second, and third images. [Figure 22] This is a diagram showing the first and third images. [Figure 23] This diagram shows wavefront propagation. [Figure 24] This figure shows the refractive index image, PSF image, first image, and recovery image. [Figure 25] This is a flowchart of the processing performed by the processor. [Figure 26] This is a diagram showing the training process. [Figure 27] This is a diagram showing the sample image generation system of this embodiment. [Modes for carrying out the invention]

[0013] Prior to describing the embodiments, the effects and advantages of an embodiment according to a certain aspect of the present invention will be explained. In specifically describing the effects and advantages of this embodiment, concrete examples will be provided. However, as with the embodiments described later, these exemplified embodiments represent only a portion of the embodiments included in the present invention, and numerous variations exist. Therefore, the present invention is not limited to the exemplified embodiments.

[0014] In this embodiment of the refractive index distribution generation device, an image of the optical image of the specimen is used. An optical image of the specimen is formed in the observation optical system, and the optical image of the specimen is captured by an image sensor, thereby obtaining an image of the optical image of the specimen. Since the specimen is a three-dimensional object, the image of the optical image of the specimen can be represented by an XY image, an XZ image, and a YZ image.

[0015] The optical axis of the observation optical system is defined as the Z-axis, the axis perpendicular to the Z-axis is defined as the X-axis, and the axis perpendicular to both the Z-axis and the X-axis is defined as the Y-axis. The XY section is a plane containing the X-axis and the Y-axis. The XY image is the image in the XY section. The XZ section is a plane containing the X-axis and the Z-axis. The XZ image is the image in the XZ section. The YZ section is a plane containing the Y-axis and the Z-axis. The YZ image is the image in the YZ section.

[0016] The refractive index distribution generation device of this embodiment comprises a hardware-based processor and a hardware-based memory. The processor performs a refractive index distribution generation process to generate a refractive index distribution corresponding to the image to be processed. The refractive index distribution generation process includes an input process that inputs the image to be processed, first refractive index information indicating the refractive index of a first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure from the memory, and a setting process that sets each refractive index constituting the refractive index distribution. The setting process includes a first setting process that sets the refractive index based on the first refractive index information at a position corresponding to a first image region of the image to be processed, based on the signal intensity of a unit pixel of the image to be processed, and a second setting process that sets the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the image to be processed. The first image region is an image region corresponding to a first structure, and a unit pixel consists of one or more pixels. The processor performs the refractive index distribution generation process using an image of a sample taken as the image to be processed.

[0017] Figure 1 shows the refractive index distribution generation device and microscope system of this embodiment. Figure 1(a) shows the refractive index distribution generation device of this embodiment. Figure 1(b) shows the microscope system.

[0018] As shown in Figure 1(a), the refractive index distribution generation device 1 comprises a memory 2 and a processor 3. The memory 2 stores the image to be processed, first refractive index information, and second refractive index information. The first refractive index information indicates the refractive index of the first structure. The second refractive index information indicates the refractive index of the second structure. The second structure is different from the first structure.

[0019] The image to be processed is an image obtained by photographing a specimen. The image to be processed can be generated, for example, from multiple XY images (hereinafter referred to as the "XY image group"). Each XY image in the XY image group is an optical image of the specimen.

[0020] In order to generate the image to be processed by the refractive index distribution generator 1, a set of XY images must be input to the refractive index distribution generator 1. The input of the set of XY images to the refractive index distribution generator 1 is done via the input unit 4. The XY images can be acquired, for example, by a microscope system.

[0021] As shown in Figure 1(b), the microscope system 10 includes a microscope 20 and a processing unit 30. The microscope 20 includes a main body 21, an objective lens 22, a stage 23, an reflected light illumination device 24, an imaging unit 25, and a controller 26. The processing unit 30 includes an input unit 31, a memory 32, a processor 33, and an output unit 34.

[0022] A specimen 27 is placed on stage 23. In the microscope 20, an optical image of the specimen 27 is formed on the image plane of the observation optical system. If a lens is arranged in the imaging unit 25, the observation optical system is formed by the objective lens 22, the imaging lens, and the lens of the imaging unit 25. If a lens is not arranged in the imaging unit 25, the observation optical system is formed by the objective lens 22 and the imaging lens.

[0023] The imaging unit 25 has an image sensor. An image of the optical image is acquired by capturing the optical image formed on the image plane with the image sensor. The optical image formed on the image plane is an optical image of the XY cross section of the specimen 27. Therefore, the image of the optical image is an XY image.

[0024] The objective lens 22 and the stage 23 can be moved relative to each other along the optical axis of the observation optical system. The movement of the objective lens 22 or the stage 23 can be controlled by the controller 26. The specimen 27 is a thick specimen. Therefore, by moving the objective lens 22 and the stage 23 relative to each other, XY images can be acquired for multiple cross-sections. The XY image group will now be described.

[0025] Figure 2 shows the specimen, optical image, image of the optical image, and XY image set. Figure 2(a) shows the specimen. Figures 2(b), 2(c), and 2(d) show the optical image and image of the optical image. Figure 2(e) shows the XY image set.

[0026] As shown in Figure 2(a), since sample 40 is a three-dimensional object, it can be represented by multiple block layers. In Figure 2(a), sample 40 is divided into seven block layers along the Z-axis. However, the number of block layers is not limited to seven. Sample OZ1 represents the block layer at one end, and sample OZ7 represents the block layer at the other end. Each block layer represents the XY cross-section of sample 40.

[0027] In forming the optical image of sample 40, block layers from sample OZ1 to sample OZ7 are sequentially positioned at the focal plane of the observation optical system 41. Although the optical image is planar, it is represented by block layers for ease of viewing. Furthermore, because the optical image is represented by block layers, the image of the optical image is also represented by block layers.

[0028] In forming the optical image of sample 40, sample 40 and the observation optical system 41 are moved relative to each other along the optical axis 42. Here, sample 40 is not moved, but the observation optical system 41 is moved relative to each other along the optical axis 42.

[0029] As shown in Figure 2(b), when the specimen OZ1 is positioned at the focal plane of the observation optical system 41, an optical image IZ1 is formed. By capturing the optical image IZ1 with an image sensor, an image of the optical image PZ1 is acquired.

[0030] As shown in Figure 2(c), when the specimen OZ4 is positioned at the focal plane of the observation optical system 41, an optical image IZ4 is formed. By capturing the optical image IZ4 with an image sensor, an image of the optical image PZ4 is acquired.

[0031] As shown in Figure 2(d), when the specimen OZ7 is positioned at the focal plane of the observation optical system 41, an optical image IZ7 is formed. By capturing the optical image IZ7 with an image sensor, an image of the optical image PZ7 is acquired.

[0032] Images PZ1, PZ4, and PZ7 are XY images. Two block layers are located between image PZ1 and image PZ4. Also, two block layers are located between image PZ4 and image PZ7. If we denote the images of these block layers as images PZ2, PZ3, PZ5, and PZ6, then images PZ2, PZ3, PZ5, and PZ6 are also XY images.

[0033] Since all images from image PZ1 to image PZ7 are XY images, a set of XY images can be obtained from these images. Furthermore, by stacking these images in the direction of the optical axis 42, a three-dimensional set of XY images can be obtained.

[0034] Figure 2(e) shows a three-dimensional XY image group 43. The XY image group 43 contains brightness information in the X-axis, Y-axis, and Z-axis directions, respectively. Therefore, XY images, XZ images, and YZ images can be generated from the XY image group 43.

[0035] As described above, the image to be processed is generated from the XY image set. Therefore, the XY, XZ, and YZ images can all be used as the image to be processed.

[0036] Returning to Figure 1(b), let's continue the explanation. In the microscope 20, multiple objective lenses can be mounted on the revolving nosepiece. The magnification of the objective lenses can be changed by rotating the revolving nosepiece. The rotation of the revolving nosepiece can be controlled by the controller 26.

[0037] The XY images are output from the imaging unit 25 and input to the processing unit 30. A group of XY images is obtained from multiple XY images. The group of XY images is input to the input unit 31 and then stored in the memory 32. The group of XY images is output from the output unit 34. Therefore, the group of XY images can be input to the refractive index distribution generator 1. The group of XY images is stored in the memory 2.

[0038] In the refractive index distribution generation device 1, the refractive index distribution generation process is executed by the processor 3. In the refractive index distribution generation process, a refractive index distribution corresponding to the image to be processed is generated.

[0039] Figure 3 shows the image to be processed and the refractive index distribution image. Figure 3(a) shows the image to be processed. Figure 3(b) shows a portion of the image to be processed. Figure 3(c) shows a portion of the refractive index distribution image. Figure 3(d) shows the refractive index distribution image.

[0040] Figure 3(a) shows the image to be processed. The specimen has a first structure and a second structure. The first and second structures may be adjacent to each other, or another structure may be located between them. The first structure is stained with a fluorescent dye. The second structure is not stained with a fluorescent dye.

[0041] Since the first structure is stained with a fluorescent dye, a fluorescence image of the first structure is formed. The fluorescence image of the first structure is formed by fluorescence at wavelength λ1. Since the second structure is not stained with a fluorescent dye, no fluorescence image of the second structure is formed.

[0042] The image to be processed 50 is an optical image of the specimen. The image to be processed 50 is an image acquired through an optical filter that transmits only fluorescence of wavelength λ1 (hereinafter referred to as "optical filter Fλ1"). The image to be processed 50 has a first image region 51 and a second image region 52.

[0043] The first image region 51 is the image region corresponding to the light transmitted through the optical filter Fλ1. The light that forms the fluorescence image of the first structure is fluorescence with wavelength λ1, and therefore it is transmitted through the optical filter Fλ1. Thus, the first image region 51 is the image region corresponding to the first structure. Since the image to be processed 50 contains multiple images of the first structure, the image to be processed 50 has multiple first image regions 51.

[0044] The second image region 52 is the image region corresponding to the light that did not pass through the optical filter Fλ1. Structures other than the first structure are designated as structure group A. Structure group A includes the second structure. Since no fluorescence image is formed for the second structure, there is no light that passes through the optical filter Fλ1. Therefore, the second image region 52 is the image region corresponding to structure group A. Since structure group A is different from the first structure, the second image region 52 is different from the first image region 51.

[0045] Figure 4 is a flowchart of the processing performed by the processor. Processor 3 performs the refractive index distribution generation process. The refractive index distribution generation process includes input processing and setting processing.

[0046] In step S100, input processing is performed. Step S100 comprises steps S101, S102, and S103.

[0047] In step S101, the image to be processed is input from memory. In step S102, the first refractive index information is input from memory. In step S103, the second refractive index information is input from memory. The first refractive index information indicates the refractive index of the first structure. The second refractive index information indicates the refractive index of the second structure. When step S100 is completed, step S200 is executed.

[0048] In step S200, a setting process is performed. In the setting process, each refractive index constituting the refractive index distribution is set. Step S200 comprises step S201 and step S202.

[0049] In step S201, a first setting process is executed. In the first setting process, the refractive index based on the first refractive index information is set at a position corresponding to the first image region of the image to be processed, based on the signal intensity of the unit pixel of the image to be processed. A unit pixel consists of one or more pixels. When step S201 is completed, step S202 is executed.

[0050] In step S202, the second setting process is executed. In the second setting process, the refractive index based on the second refractive index information is set to a position corresponding to an image region different from the first image region of the image to be processed.

[0051] Figure 3(b) shows a magnified view of a portion of the image being processed, along with the signal intensity in the Z-axis direction and the signal intensity in the X-axis direction. In Figure 3(b), the first image region is represented by an ellipse for clarity. However, in reality, the first image region has a complex shape.

[0052] The image to be processed 60 has a first image region 61 and a second image region 62. The second image region 62 is different from the first image region 61.

[0053] The first image region 61 is the first image region 51 in Figure 3(a). Since the first image region 51 is the image region corresponding to the first structure, the first image region 61 is the image region corresponding to the first structure. The second image region 62 is the second image region 52 in Figure 3(a). Since the second image region 52 is the image region corresponding to structure group A, the second image region 62 is the image region corresponding to structure group A.

[0054] The first image region 61 can be determined based on the signal intensity of a unit pixel in the image 60 to be processed. A unit pixel consists of one or more pixels. If the unit pixel is one pixel, the first image region 61 is determined based on the intensity signal of that one pixel. If the unit pixel is multiple pixels, the first image region 61 is determined based on the intensity signals of the multiple pixels.

[0055] In the image 60 being processed, a signal intensity greater than zero is used as the signal intensity of the unit pixel. In Figure 3(b), ΔZ and ΔX indicate the regions of unit pixels that have a signal intensity greater than zero.

[0056] In the unit pixels forming the first image region 61, the signal intensity is greater than zero. The second image region 62 is the region excluding the first image region 61. Therefore, in the unit pixels forming the second image region 62, the signal intensity is zero.

[0057] Since the first image region 61 is represented by an ellipse, the position of the first image region 61 can be determined from the position of the ellipse. The second image region 62 is the region excluding the first image region 61, so if the position of the first image region 61 is determined, the position of the second image region 62 can be determined.

[0058] In the first and second setting processes, the refractive index is set. The image to be processed is the optical image of the specimen. Since the optical image of the specimen is an image formed from brightness information, it is not possible to set the refractive index in this image. Therefore, the refractive index should be set in an image separate from the image to be processed, for example, a refractive index distribution image.

[0059] Figure 3(c) shows a magnified view of a portion of the refractive index distribution image, the refractive index distribution in the Z-axis direction, and the refractive index distribution in the X-axis direction. The refractive index distribution image 70 has a first refractive index region 71 and a second refractive index region 72.

[0060] Since the refractive index distribution image 70 is an image represented by refractive index, it is possible to set the refractive index based on the first refractive index information and the refractive index based on the second refractive index information. In order to set the refractive index, a region for setting the refractive index based on the first refractive index information and a region for setting the refractive index based on the second refractive index information are required.

[0061] The first refractive index information is information indicating the refractive index of the first structure. The image region corresponding to the first structure is the first image region 61. In order to set the refractive index based on the first refractive index information, it is sufficient to find the region in the refractive index distribution image 70 that corresponds to the first image region 61.

[0062] The second refractive index information indicates the refractive index of the second structure. The second structure is included in structure group A. The image region corresponding to structure group A is the second image region 62. To set the refractive index based on the second refractive index information, it is sufficient to find the region in the refractive index distribution image 70 that corresponds to the second image region 62.

[0063] In Figure 3(c), the first refractive index region 71 corresponds to the first image region 61. Therefore, it is sufficient to set the refractive index in the first refractive index region 71 based on the first refractive index information.

[0064] Furthermore, the second refractive index region 72 is the region corresponding to the second image region 62. Since the second image region 62 is the image region corresponding to structure group A, the second refractive index region 72 is the image region corresponding to structure group A. Structure group A includes the second structure. Therefore, it is sufficient to set the refractive index in the second refractive index region 72 based on the second refractive index information.

[0065] In order to set the refractive index based on the first refractive index information in the first refractive index region 71, it is necessary to determine the position of the first refractive index region 71. The first refractive index region 71 corresponds to the first image region 61. The position of the first image region 61 is determined based on the signal intensity of a unit pixel in the image 60 to be processed. Therefore, the position of the first refractive index region 71 can be determined based on the signal intensity of a unit pixel in the image 60 to be processed. The refractive index based on the first refractive index information can then be set at the position determined in this way.

[0066] In order to set the refractive index based on the second refractive index information in the second refractive index region 72, it is necessary to determine the position of the second refractive index region 72. The second refractive index region 72 corresponds to the second image region 62. Since the second image region 62 is the region excluding the first image region 61, if the position of the first refractive index region 71 is determined, the position of the second refractive index region 72 can be determined. The refractive index based on the second refractive index information can then be set at the position determined in this way.

[0067] Figure 3(d) shows the refractive index distribution image 80. The refractive index distribution image 80 has a first refractive index region 81 and a second refractive index region 82. The refractive index distribution image 80 is the image corresponding to the image to be processed 50. The image to be processed 50 has multiple first image regions 51. Therefore, the refractive index distribution image 80 also has multiple first refractive index regions 81.

[0068] In processor 3, refractive index distribution generation processing is performed using the image to be processed 50. The image to be processed 50 is an image of the optical image of the specimen. Processor 3 uses the image of the specimen as the image to be processed and performs refractive index distribution generation processing. As a result, the refractive index distribution generation device of this embodiment can improve the accuracy of the refractive index distribution even with thick specimens.

[0069] A specimen is, for example, a cell aggregate. A cell aggregate contains multiple cells. Cell adhesion molecules exist between adjacent cells. A cell has a nucleus, cytoplasm, and a cell membrane. If the specimen is a cell aggregate, the specimen has a cell nucleus, cytoplasm, a cell membrane, and cell adhesion molecules.

[0070] The specimen has a first structure and a second structure. The first structure is the cell nucleus, and the second structure is the cytoplasm. The cellular structures other than the cell nucleus are designated as cellular structure group A. Cellular structure group A includes the cytoplasm, the cell membrane, and cell adhesion molecules.

[0071] As described above, structure group A consists of structures other than the first structure and includes the second structure. Cell structure group A consists of cell structures other than the cell nucleus and includes the cytoplasm. Therefore, if the specimen is a cell aggregate, cell structure group A corresponds to structure group A.

[0072] When only the cell nucleus is stained with a fluorescent dye, a fluorescent image of the cell nucleus is formed. Since a cell aggregate contains multiple cell nuclei, multiple fluorescent images are formed. The first image region 51 is the image region corresponding to the first structure. Since the first structure is a cell nucleus, the first image region 51 represents an image region corresponding to multiple cell nuclei.

[0073] Cell structure group A is not stained with a fluorescent dye, so no fluorescent image is formed. Second image region 52 is the image region corresponding to structure group A. Since cell structure group A corresponds to structure group A, second image region 52 represents the image region corresponding to cell structure group A.

[0074] Since the first structure is the cell nucleus, the refractive index of the cell nucleus should be set in the first refractive index region 81. Cell structure group A corresponds to structure group A, so the refractive index of cell structure group A should be set in the second refractive index region 82. As mentioned above, cell structure group A has cytoplasm, a cell membrane, and cell adhesion molecules. Since the regions of the cell membrane and cell adhesion molecules are narrow, the refractive index of the cytoplasm should be set in the second refractive index region 82.

[0075] The specimen does not have to be a biological specimen. For example, the specimen may be foreign matter or solder on a semiconductor substrate. The refractive index distribution generation device of this embodiment can improve the accuracy of the refractive index distribution in foreign matter or solder on a semiconductor substrate.

[0076] In the refractive index distribution generating device of this embodiment, it is preferable that the first setting process sets the refractive index based on the first refractive index information at a position corresponding to the first image region composed of first unit pixels whose signal intensity value is greater than the threshold, and the second setting process sets the refractive index based on the second refractive index information at a position corresponding to the image region composed of second unit pixels whose signal intensity value is less than or equal to the threshold.

[0077] Figure 5 shows the image to be processed and the refractive index distribution image. Figure 5(a) shows the image to be processed. Figure 5(b) shows a part of the image to be processed. Figure 5(c) shows a part of the refractive index distribution image. Figure 5(d) shows the refractive index distribution image. Since Figure 5(a) is the same as Figure 3(a), the explanation for Figure 5(a) is omitted.

[0078] In the first setting process, a refractive index based on the first refractive index information is set at the position corresponding to the first image region, which is composed of first unit pixels whose signal intensity value is greater than the threshold.

[0079] In the second setting process, a refractive index based on the second refractive index information is set at the position corresponding to the image region composed of second unit pixels whose signal intensity value is below a threshold among the unit pixels.

[0080] Figure 5(b) shows a magnified view of a portion of the image being processed, along with the signal intensity in the Z-axis direction and the signal intensity in the X-axis direction. In Figure 5(b), the first image region is represented by an ellipse for clarity. However, in reality, the first image region has a complex shape.

[0081] The image to be processed 90 has a first image region 91 and a second image region 92. The second image region 92 is different from the first image region 91.

[0082] The first image region 91 is the first image region 51 in Figure 5(a). Since the first image region 51 is the image region corresponding to the first structure, the first image region 91 is the image region corresponding to the first structure. The second image region 92 is the second image region 52 in Figure 5(a). Since the second image region 52 is the image region corresponding to structure group A, the second image region 92 is the image region corresponding to structure group A.

[0083] The first image region 91 can be determined based on the signal intensity of the first unit pixel. In the image to be processed 90, the signal intensity of the first unit pixel is the signal intensity of the unit pixel that is greater than the threshold ITH. In Figure 5(b), ΔZ1 and ΔX1 indicate the region of the first unit pixel.

[0084] The second image region 92 can be determined based on the signal intensity of the second unit pixel. In the image to be processed 90, the signal intensity of the second unit pixel is the signal intensity of the unit pixel that is below the threshold ITH. In Figure 5(b), ΔZ2 and ΔX2 indicate the region of the second unit pixel.

[0085] Since the first image region 91 is represented by an ellipse, the position of the first image region 91 can be determined from the position of the ellipse. The second image region 92 is the region excluding the first image region 91, so if the position of the first image region 91 is determined, the position of the second image region 92 can be determined.

[0086] In the first and second setting processes, the refractive index is set. As mentioned above, the refractive index can be set using an image separate from the image being processed, such as a refractive index distribution image.

[0087] Figure 5(c) shows a magnified view of a portion of the refractive index distribution image, the refractive index distribution in the Z-axis direction, and the refractive index distribution in the X-axis direction. The refractive index distribution image 100 has a first refractive index region 101 and a second refractive index region 102.

[0088] Since the refractive index distribution image 100 is an image represented by refractive index, it is possible to set the refractive index based on the first refractive index information and the refractive index based on the second refractive index information. In order to set the refractive index, a region for setting the refractive index based on the first refractive index information and a region for setting the refractive index based on the second refractive index information are required.

[0089] The first refractive index information is information indicating the refractive index of the first structure. The image region corresponding to the first structure is the first image region 91. In order to set the refractive index based on the first refractive index information, it is sufficient to find the region in the refractive index distribution image 100 that corresponds to the first image region 91.

[0090] The second refractive index information indicates the refractive index of the second structure. The second structure is included in structure group A. The image region corresponding to structure group A is the second image region 92. To set the refractive index based on the second refractive index information, it is sufficient to find the region in the refractive index distribution image 100 that corresponds to the second image region 92.

[0091] In Figure 5(c), the first refractive index region 101 corresponds to the first image region 91. Therefore, the refractive index based on the first refractive index information should be set for the first refractive index region 101.

[0092] Also, the 2 The refractive index region 102 is the region corresponding to the second image region 92. Since the second image region 92 is the image region corresponding to structure group A, the second refractive index region 102 is the image region corresponding to structure group A. Structure group A includes the second structure. Therefore, it is sufficient to set the refractive index in the second refractive index region 102 based on the second refractive index information.

[0093] In order to set the refractive index based on the first refractive index information in the first refractive index region 101, it is necessary to determine the position of the first refractive index region 101. The first refractive index region 101 corresponds to the first image region 91. The position of the first image region 91 is determined based on the signal intensity of a unit pixel of the image to be processed 90, that is, a signal intensity greater than the threshold ITH. Therefore, the position of the first refractive index region 101 can be determined based on a signal intensity greater than the threshold ITH. The refractive index based on the first refractive index information can then be set at the position determined in this way.

[0094] In order to set the refractive index based on the second refractive index information in the second refractive index region 102, it is necessary to determine the position of the second refractive index region 102. The second refractive index region 102 corresponds to the second image region 92. The position of the second image region 92 is determined based on the signal intensity of a unit pixel of the image to be processed 90, that is, the signal intensity below the threshold ITH. Therefore, the position of the second refractive index region 102 can be determined based on the signal intensity below the threshold ITH. The refractive index based on the second refractive index information can then be set at the position determined in this way.

[0095] Figure 5(d) shows the refractive index distribution image 110. The refractive index distribution image 110 has a first refractive index region 111 and a second refractive index region 112. The refractive index distribution image 110 is the image corresponding to the image to be processed 50. The image to be processed 50 has multiple first image regions 51. Therefore, the refractive index distribution image 110 also has multiple first refractive index regions 111.

[0096] Processor 3 performs refractive index distribution generation processing using the image to be processed 50. The image to be processed 50 is an image of the optical image of the specimen. Processor 3 uses the image of the specimen as the image to be processed and performs refractive index distribution generation processing.

[0097] In the refractive index distribution generation apparatus of this embodiment, the refractive index distribution generation process preferably includes a first correction process that improves the brightness at deeper positions of the sample relative to the brightness at shallower positions of the sample, and the first setting process preferably sets the refractive index based on the first refractive index information based on the signal intensity of the unit pixel of the image to be processed in which the brightness at deeper positions has been improved. The image to be processed is an image in which the depth of the sample increases along the direction from one end to the other, and one end is on the opposite side of the center of the image to be processed from the other end.

[0098] Figure 6 is a flowchart of the processes performed by the processor. Processes identical to those shown in Figure 4 are omitted from the explanation.

[0099] The refractive index distribution generation process includes input processing, a first setting process, a second setting process, and a first correction process. The first correction process is performed before the first setting process.

[0100] In step S110, the first correction process is performed. The image to be processed in the first correction process is an image in which the depth of the sample increases along the direction from one end to the other. One end is on the opposite side of the center of the image to be processed from the other end. In the first correction process, the brightness of the deeper parts of the sample is improved compared to the brightness of the shallower parts of the sample.

[0101] Figure 7 shows the image to be processed. Figure 7(a) shows the image to be processed before the first correction. Figure 7(b) shows the signal intensity before the first correction. Figure 7(c) shows the image to be processed after the first correction. Figure 7(d) shows the signal intensity after the first correction. The state before the first correction is the state before the first correction process is performed. The state after the first correction is the state after the first correction process is performed.

[0102] The side closer to the observation optical system is considered the top surface of the specimen, and the side further from the observation optical system is considered the bottom surface of the specimen. Since the image to be processed 120 is an XZ image, the right edge of the image represents the top surface of the specimen, and the left edge represents the bottom surface of the specimen. The depth of the specimen increases from the right edge to the left edge of the image. The image to be processed 120 has a first image region 121 and a second image region 122.

[0103] The depth from the top surface of the specimen increases towards the left edge of the image. As shown in Figure 7(a), the first image region 121 is located to the left of the second image region 122. Therefore, the first image region 121 is located deeper than the second image region 122.

[0104] The image to be processed 120 shown in Figure 7(a) is the image before the first correction. Since this is the state before the first correction process is performed, the brightness of the first image region 121 in the image to be processed 120 is darker than the brightness of the second image region 122. Since brightness is represented by signal intensity, as shown in Figure 7(b), the signal intensity of the first image region 121 is smaller than the signal intensity of the second image region 122.

[0105] In Figure 7(a), for ease of viewing, the brightness of the first image region 121 and the brightness of the second image region 122 are depicted with uniform brightness. However, as shown in Figure 7(b), the brightness of the first image region 121 and the brightness of the second image region 122 are not uniform.

[0106] The image to be processed 130 shown in Figure 7(c) is the image in the first corrected state. The image to be processed 130 has a first image region 131 and a second image region 132. The first image region 131 corresponds to the first image region 121, and the second image region 132 corresponds to the second image region 12 2 It supports this.

[0107] Since this is the state after the first correction process has been performed, in the processed image 130, the brightness of the first image region 131 is the same as the brightness of the second image region 132. Since brightness is represented by signal intensity, as shown in Figure 7(d), the signal intensity of the first image region 131 is the same as the signal intensity of the second image region 132. The brightness of the first image region 131 and the brightness of the second image region 132 are not uniform.

[0108] The depth of the first image region 131 is greater than the depth of the second image region 132. In the image to be processed 130, the brightness at the deeper locations of the sample is improved compared to the brightness at the shallower locations of the sample. When step S110 is completed, step S201 is executed.

[0109] Step S201 is the first setting process. Since the first correction process has been performed, the first setting process is performed on the image to be processed 130. In the first setting process, the refractive index is set based on the signal intensity of the unit pixels of the image to be processed 130.

[0110] The image to be processed 130 is an image in which the brightness at deeper positions has been improved. Therefore, in step S201, the refractive index is set based on the signal intensity of the unit pixel of the image to be processed in which the brightness at deeper positions has been improved.

[0111] In processing images 120 and 130, the first image region is conveniently depicted as an ellipse. However, as is clear from the shape of the first image region in processing image 50, the actual shape of the first image region is complex.

[0112] In the refractive index distribution generation device of this embodiment, the refractive index distribution generation process includes a second correction process that corrects the first refractive index information and generates corrected refractive index information based on the signal intensity of the first unit pixel, wherein the second correction process generates corrected refractive index information in which the refractive index corresponding to the first unit pixel whose signal intensity is less than the maximum signal intensity is smaller than the refractive index corresponding to the unit pixel with the maximum signal intensity among the first unit pixels, and the first setting process preferably sets the refractive index indicated by the corrected refractive index information at a position corresponding to the first image region.

[0113] Figure 8 is a flowchart of the processes performed by the processor. Processes identical to those shown in Figure 4 are omitted from the explanation.

[0114] The refractive index distribution generation process includes input processing, a first setting process, a second setting process, and a second correction process. The second correction process is performed before the first setting process.

[0115] In step S120, a second correction process is performed. In the second correction process, the first refractive index information is corrected based on the signal intensity of the first unit pixel, and corrected refractive index information is generated.

[0116] Figure 9 shows the image to be processed and the refractive index distribution image. Figure 9(a) shows the image to be processed. Figure 9(b) shows a part of the image to be processed. Figure 9(c) shows a part of the refractive index distribution image. Figure 9(d) shows the refractive index distribution image. Figure 9(a) is the same as Figure 5(a), and Figure 9(b) is the same as Figure 5(b), so the explanation for Figures 9(a) and 9(b) is omitted.

[0117] The first unit pixel is a pixel whose signal intensity is greater than the threshold ITH among the signal intensity of unit pixels. As shown in Figure 9(b), the first image region 91 can be determined based on the signal intensity of the first unit pixel.

[0118] In Figure 9(b), ΔZ1 and ΔX1 represent the region of the first unit pixel. The region of the first unit pixel includes the unit pixel with the maximum signal intensity and the unit pixels with a signal intensity less than the maximum signal intensity.

[0119] In the second correction process, corrected refractive index information is generated. The signal intensity Imax, signal intensity ITH, refractive index n1, and refractive index n2 are used in the generation of this corrected refractive index information. Signal intensity Imax is the maximum signal intensity in the region of the first unit pixel. Signal intensity ITH is the threshold signal intensity. Furthermore, refractive index n1 is smaller than refractive index n2.

[0120] In generating corrected refractive index information, refractive index n1 is associated with signal intensity ITH, and refractive index n2 is associated with signal intensity Imax. This association allows for a one-to-one correspondence between refractive indices from n1 to n2 and signal indices from ITH to Imax. As a result, corrected refractive index information can be represented using refractive indices between n1 and n2.

[0121] For example, in the first refractive index information, let's assume that the refractive index information is formed only by the refractive index n2. In this case, there is one refractive index in the first refractive index information. In contrast, there are multiple refractive indices in the corrected refractive index information. Therefore, the corrected refractive index information can be considered as information in which the first refractive index information has been corrected with finer refractive indices. When step S120 is completed, step S201 is executed.

[0122] In step S201, the first setting process is executed. In the first setting process, the refractive index is set. As mentioned above, the refractive index can be set using an image separate from the image to be processed, for example, a refractive index distribution image.

[0123] Figure 9(c) shows a magnified view of a portion of the refractive index distribution image, the refractive index distribution in the Z-axis direction, and the refractive index distribution in the X-axis direction. The refractive index distribution image 140 has a first refractive index region 141 and a second refractive index region 142.

[0124] Since the refractive index distribution image 140 is an image represented by refractive index, it is possible to set the refractive index based on corrected refractive index information. In order to set the refractive index, a region is needed to define the region for which the refractive index based on the corrected refractive index information is to be set.

[0125] The corrected refractive index information is information obtained by correcting the first refractive index information. The first refractive index information is information indicating the refractive index of the first structure. The image region corresponding to the first structure is the first image region 91. In order to set the refractive index based on the corrected refractive index information, it is sufficient to find the region in the refractive index distribution image 140 that corresponds to the first image region 91.

[0126] In Figure 9(c), the first refractive index region 141 corresponds to the first image region 91. Similarly, the second refractive index region 142 corresponds to the second image region 92. Therefore, the refractive index based on the corrected refractive index information should be set for the first refractive index region 141.

[0127] In order to set the refractive index based on the corrected refractive index information in the first refractive index region 141, it is necessary to determine the position of the first refractive index region 141. The first refractive index region 141 corresponds to the first image region 91. The position of the first image region 91 is determined based on the signal intensity of a unit pixel of the image to be processed 90, that is, a signal intensity greater than the threshold ITH. Therefore, the position of the first refractive index region 141 can be determined based on a signal intensity greater than the threshold ITH. The refractive index based on the corrected refractive index information can then be set at the position determined in this way.

[0128] Figure 9(d) shows a refractive index distribution image 150. The refractive index distribution image 150 has a first refractive index region 151 and a second refractive index region 152. The refractive index distribution image 150 is the image corresponding to the image to be processed 50. The image to be processed 50 has multiple first image regions 51. Therefore, the refractive index distribution image 150 also has multiple first refractive index regions 151.

[0129] When the signal intensity ITH is zero, the refractive index based on the corrected refractive index information is set in the first refractive index region 71 shown in Figure 3(c). In this case, the first refractive index region 71 can be represented by refractive indices between refractive index n1 and refractive index n2.

[0130] In the refractive index distribution generation apparatus of this embodiment, the refractive index distribution generation process preferably includes a first correction process that improves the brightness at deeper positions of the sample relative to the brightness at shallower positions of the sample, and the first setting process preferably corrects the first refractive index information corresponding to the first image region of the image to be processed in which the brightness at deeper positions has been improved, and generates corrected refractive index information. The image to be processed is an image in which the depth of the sample increases along the direction from one end to the other, and one end is on the opposite side of the center of the image to be processed from the other end.

[0131] Figure 10 is a flowchart of the processes performed by the processor. The same processes shown in Figures 6 and 8 are omitted from the explanation.

[0132] In step S110, a first correction process is performed. In the first correction process, the brightness of the first image region with a deep depth from the top surface is improved compared to the brightness of the first image region with a shallow depth from the top surface. As a result, the first image region with improved brightness is obtained. When step S110 is completed, step S120 is performed.

[0133] In step S120, the second correction process is performed. The first correction process obtains the first image region after the brightness has been improved. Therefore, in the second correction process, the first refractive index information corresponding to the first image region after the brightness has been improved is corrected, and corrected refractive index information is generated. When step S120 is completed, step S201 is performed.

[0134] In step S201, the first setting process is executed. In the second correction process, corrected refractive index information is generated. Therefore, in the first setting process, the refractive index indicated by the corrected refractive index information is set at the position corresponding to the first image region after the brightness has been improved.

[0135] In the refractive index distribution generation device of this embodiment, the first setting process preferably sets the refractive index indicated by the first refractive index information at a position corresponding to the first image region composed of the first unit pixel.

[0136] Figure 11 shows the image to be processed. Figure 11(a) shows the image to be processed before processing. Figure 11(b) shows a portion of the image to be processed before processing. Figure 11(c) shows a portion of the image to be processed after processing. Figure 11(d) shows the image to be processed after processing. Since Figure 11(a) is the same as Figure 3(a) and Figure 11(b) is the same as Figure 3(b), the explanation for Figures 11(a) and 11(b) will be omitted.

[0137] As shown in Figure 11(b), in the image to be processed 90, the first image region 91 and the second image region 92 are represented by signal intensities of various magnitudes. Therefore, a binarization process is performed on the image to be processed 90.

[0138] As a result, as shown in Figure 11(c), in the processed image 160, the first image region 161 and the second image region 162 are each represented by a signal intensity of one size. The signal intensity in the first image region 161 is greater than the signal intensity in the second image region 162.

[0139] By performing a binarization process on the entire image to be processed, the processed image 170 is obtained, as shown in Figure 11(d). In the processed image 170, all first image regions 171 and second image regions 172 have been binarized.

[0140] In the first setting process, the refractive index indicated by the first refractive index information is set at the position corresponding to the binarized first image region 171.

[0141] In the refractive index distribution generation device of this embodiment, the refractive index distribution generation process preferably includes a first correction process that improves the brightness at deeper positions of the sample relative to the brightness at shallower positions of the sample, and the first setting process preferably sets the refractive index indicated by the first refractive index information at a position corresponding to the first image region of the processed image in which the brightness at deeper positions has been improved. The processed image is an image in which the depth of the sample increases along the direction from one end to the other, and one end is on the opposite side of the center of the processed image from the other end.

[0142] Step S110 may be executed before the binarization process. In step S110, the first correction process is performed. After the binarization process is completed, step S201 is executed.

[0143] In step S201, the first setting process is executed. In the first setting process, the refractive index indicated by the first refractive index information is set at the position corresponding to the binarized first image region.

[0144] In the refractive index distribution generation device of this embodiment, the first correction process preferably improves brightness more in the image region closer to the other end.

[0145] This can improve the accuracy of the refractive index distribution of the sample.

[0146] In the refractive index distribution generation device of this embodiment, the input process inputs third refractive index information indicating the refractive index of the third structure from memory, the setting process includes a third setting process that sets the refractive index based on the third refractive index information at a position corresponding to the third image region of the image to be processed, based on the signal intensity and color information of the unit pixel of the image to be processed, and preferably the second setting process sets the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the image to be processed, and at a position corresponding to an image region different from the third image region of the image to be processed. The third image region is the image region corresponding to the third structure.

[0147] Figure 12 shows the image to be processed. Figure 12(a) shows the first image region. Figure 12(b) shows the third image region.

[0148] The specimen has three structures: a first structure, a second structure, and a third structure. The first and third structures are stained with a fluorescent dye. The second structure is not stained with a fluorescent dye.

[0149] Since the first structure is stained with a fluorescent dye, a fluorescence image of the first structure is formed. The fluorescence image of the first structure is formed by fluorescence at wavelength λ1. Since the second structure is not stained with a fluorescent dye, no fluorescence image of the second structure is formed. Since the third structure is stained with a fluorescent dye, a fluorescence image of the third structure is formed. The fluorescent dye used to stain the third structure is different from the fluorescent dye used to stain the first structure. The fluorescence image of the third structure is formed by fluorescence at wavelength λ3.

[0150] The image to be processed 180 shown in Figure 12(a) is an optical image of the specimen. The image to be processed 180 is an image acquired through the optical filter Fλ1. The image to be processed 180 has a first image region 181 and an image region 182.

[0151] The first image region 181 is the image region corresponding to the light transmitted through the optical filter Fλ1. The light that forms the fluorescence image of the first structure is fluorescence with wavelength λ1, and therefore it is transmitted through the optical filter Fλ1. Thus, the first image region 181 is the image region corresponding to the first structure. Since the image to be processed 180 contains multiple images of the first structure, the image to be processed 180 has multiple first image regions 181.

[0152] Image region 182 is the image region corresponding to the light that did not pass through the optical filter Fλ1. Structures other than the first structure are designated as structure group B. Structure group B includes the second and third structures. Since no fluorescence image is formed for the second structure, there is no light that passes through the optical filter Fλ1. The light that forms the fluorescence image for the third structure is fluorescence with wavelength λ3, so it does not pass through the optical filter Fλ1. Therefore, image region 182 is the image region corresponding to structure group B. Since structure group B is different from the first structure, image region 182 is different from the first image region 181.

[0153] The processed image 190 shown in Figure 12(b) is an optical image of the specimen. The processed image 190 is an image acquired through an optical filter (hereinafter referred to as "optical filter Fλ3") that transmits only fluorescence of wavelength λ3. The processed image 190 has a third image region 191 and an image region 192.

[0154] The third image region 191 is the image region corresponding to the light transmitted through the optical filter Fλ3. The light that forms the fluorescence image of the third structure is fluorescence with wavelength λ3, and therefore it is transmitted through the optical filter Fλ3. Thus, the third image region 191 is the image region corresponding to the third structure.

[0155] Image region 192 is the image region corresponding to the light that did not pass through the optical filter Fλ3. Structures other than the third structure are designated as structure group C. Structure group C includes the first and second structures. The light that forms the fluorescence image of the first structure is fluorescence with wavelength λ1, so it does not pass through the optical filter Fλ3. No fluorescence image is formed for the second structure, so there is no light that passes through the optical filter Fλ3. Therefore, image region 192 is the image region corresponding to structure group C. Since structure group C is different from the third structure, image region 192 is different from the third image region 191.

[0156] Figure 13 is a flowchart of the processes performed by the processor. Processes identical to those shown in Figure 3 are omitted from the explanation.

[0157] In step S100, input processing is performed. Step S100 comprises steps S101, S102, S103, and S104. In step S104, third refractive index information is input from memory. Third refractive index information is information indicating the refractive index of the third structure. When step S100 is completed, step S200 is executed.

[0158] In step S200, a setting process is performed. In the setting process, each refractive index constituting the refractive index distribution is set. Step S200 comprises steps S201, S203, and S204.

[0159] In step S203, the third setting process is executed. In the third setting process, the refractive index based on the third refractive index information is set at a position corresponding to the third image region of the image to be processed, based on the signal intensity and color information of the unit pixel of the image to be processed.

[0160] In step S204, the second setting process is executed. In the second setting process, the refractive index based on the second refractive index information is set at a position corresponding to an image region different from the first image region of the image to be processed, and at a position corresponding to an image region different from the third image region of the image to be processed.

[0161] In the image 180 being processed, a signal intensity greater than zero is used as the signal intensity of the unit pixels. In the unit pixels forming the first image region 181, the signal intensity is greater than zero. Image region 182 is the region excluding the first image region 181. Therefore, in the unit pixels forming image region 182, the signal intensity is zero.

[0162] In the image 190 being processed, a signal intensity greater than zero is used as the signal intensity of the unit pixels. In the unit pixels forming the third image region 191, the signal intensity is greater than zero. Image region 192 is the region excluding the third image region 191. Therefore, in the unit pixels forming image region 192, the signal intensity is zero.

[0163] The predetermined set of shapes is defined as a set of shapes that are close to a circle and a set of shapes that are close to an ellipse. Since the first image region 181 is the region represented by the predetermined set of shapes, the position of the first image region 181 can be determined from the position of the predetermined set of shapes. Since the image region 182 is the region excluding the first image region 181, if the position of the first image region 181 is determined, the position of the image region 182 can be determined.

[0164] Since the third image region 191 is represented by a mesh pattern, the position of the third image region 191 can be determined from the position of the mesh pattern. Since image region 192 is the region excluding the third image region 191, if the position of the third image region 191 is determined, the position of image region 192 can be determined.

[0165] In the first, second, and third setting processes, the refractive index is set. The image to be processed is the optical image of the specimen. Since the optical image of the specimen is an image formed from brightness information, the refractive index can be set, for example, using a refractive index distribution image.

[0166] Since the refractive index distribution image is an image represented by refractive index, it is possible to set the refractive index based on the first refractive index information, the refractive index based on the second refractive index information, and the refractive index based on the third refractive index information. In order to set the refractive index, regions are required for setting the refractive index based on the first refractive index information, the refractive index based on the second refractive index information, and the refractive index based on the third refractive index information.

[0167] The first refractive index information is information indicating the refractive index of the first structure. The image region corresponding to the first structure is the first image region 181. In order to set the refractive index based on the first refractive index information, it is sufficient to find the region in the refractive index distribution image that corresponds to the first image region 181.

[0168] The third refractive index information indicates the refractive index of the third structure. The image region corresponding to the third structure is the third image region 191. To set the refractive index based on the third refractive index information, it is sufficient to find the region corresponding to the third image region 191 in the refractive index distribution image.

[0169] The second refractive index information indicates the refractive index of the second structure. The second structure is included in structure group B and structure group C. The image region corresponding to structure group B is image region 182, and the image region corresponding to structure group C is image region 192.

[0170] However, structure group B includes the third structure, and structure group C includes the first structure. Therefore, to find the image region containing the second structure, we can either remove the third image region 191 from image region 182, or remove the first image region 181 from image region 192. The region remaining after the removal (hereinafter referred to as "remaining region A") contains the image region corresponding to the second structure. To set the refractive index based on the second refractive index information, we can find the region corresponding to remaining region A in the refractive index distribution image.

[0171] In order to set the refractive index based on refractive index information in the refractive index distribution image, it is necessary to determine the position of the region corresponding to the first image region 181, the position of the region corresponding to the remaining region A, and the position of the region corresponding to the third image region 191.

[0172] The position of the first image region 181 is determined based on the signal intensity of a unit pixel in the image 180 to be processed. Therefore, the position of the region corresponding to the first image region 181 can be determined based on the signal intensity of a unit pixel in the image 180 to be processed. The refractive index based on the first refractive index information can then be set at the position determined in this way.

[0173] The position of the third image region 191 is determined based on the signal intensity of a unit pixel in the image 190 to be processed. Therefore, the position of the region corresponding to the third image region 191 can be determined based on the signal intensity of a unit pixel in the image 190 to be processed. The refractive index based on the third refractive index information can then be set at the position determined in this way.

[0174] The region corresponding to remaining region A can be determined by excluding the region corresponding to the first image region 181 and the region corresponding to the third image region 191 in the refractive index distribution image. Therefore, the position corresponding to remaining region A can be determined based on the position of the region corresponding to the first image region 181 and the position of the region corresponding to the third image region 191. The refractive index based on the second refractive index information can then be set at the position determined in this way.

[0175] A color image sensor or a monochrome image sensor can be used as the image sensor. As described above, the light that forms the fluorescence image of the third structure is fluorescence with wavelength λ3, and the light that forms the fluorescence image of the first structure is fluorescence with wavelength λ1. Since wavelength λ3 is different from wavelength λ1, the color of the fluorescence image of the third structure and the color of the fluorescence image of the first structure are different.

[0176] When a color image sensor is used as the image sensor, the fluorescence image of the third structure and the fluorescence image of the first structure can be distinguished by color. Therefore, the position corresponding to the third image region of the image to be processed can be determined based on the color information.

[0177] When a monoimage sensor is used as the image sensor, the fluorescence image of the third structure and the fluorescence image of the first structure cannot be distinguished by color. As described above, the third image region is the image region corresponding to the light transmitted through optical filter Fλ3, and the first image region is the image region corresponding to the light transmitted through optical filter Fλ1. The color of optical filter Fλ3 is different from the color of optical filter Fλ1. In this case, the fluorescence image of the third structure and the fluorescence image of the first structure can be distinguished by the color of the optical filter. Therefore, the position corresponding to the third image region of the image to be processed can be determined based on color information.

[0178] In the refractive index distribution generation device of this embodiment, the input process inputs fourth refractive index information indicating the refractive index of the medium surrounding the sample from memory, the refractive index distribution generation process includes a specification process that identifies the boundary between the sample and the medium, and the setting process sets the refractive index based on the fourth refractive index information at a position corresponding to the fourth image region of the image to be processed. 4 It is preferable to include a configuration process. The fourth image region is the image region corresponding to the medium.

[0179] Figure 14 shows the image to be processed. Figure 14(a) shows the first image region. Figure 14(b) shows the fourth image region.

[0180] The specimen has a first structure and a second structure. The specimen is surrounded by a medium, such as a culture medium. The first structure is stained with a fluorescent dye. The second structure and the medium are not stained with a fluorescent dye.

[0181] Since the first structure is stained with a fluorescent dye, a fluorescence image of the first structure is formed. The fluorescence image of the first structure is formed by fluorescence at wavelength λ1. Since the second structure and the medium are not stained with a fluorescent dye, no fluorescence images of the second structure or the medium are formed.

[0182] Figure 1 4The image to be processed 200 shown in (a) is an optical image of the specimen. The image to be processed 200 is an image acquired through the optical filter Fλ1. The image to be processed 200 has a first image region 201 and an image region 202.

[0183] The first image region 201 is the image region corresponding to the light transmitted through the optical filter Fλ1. The light that forms the fluorescence image of the first structure is fluorescence with wavelength λ1, and therefore it is transmitted through the optical filter Fλ1. Thus, the first image region 201 is the image region corresponding to the first structure. Since the image to be processed 200 contains multiple images of the first structure, the image to be processed 200 has multiple first image regions 201.

[0184] Image region 202 is the image region corresponding to the light that did not pass through the optical filter Fλ1. Since no fluorescence images of structure group A and the medium are formed, there is no light that passes through the optical filter Fλ1. Therefore, image region 202 is the image region corresponding to structure group A and the medium. Since structure group A and the medium are different from the first structure, image region 202 is different from the first image region 201.

[0185] Figure 1 4 The image 210 to be processed shown in (b) is an optical image of the medium. Figure 1 4 In (b), the two regions are binarized for clarity. The optical image of the medium can be estimated, for example, from a bright-field image.

[0186] The image to be processed 210 has a fourth image region 211 and an image region 212. The fourth image region 211 is the image region corresponding to the medium. The image region 212 is the image region corresponding to the specimen. The outer edge of the image region 212 can be considered to represent the general shape of the entire specimen.

[0187] Figure 15 is a flowchart of the processes performed by the processor. Processes identical to those shown in Figure 3 are omitted from the explanation.

[0188] In step S100, input processing is performed. Step S100 comprises steps S101, S102, S103, and S105. In step S105, fourth refractive index information is input from memory. The fourth refractive index information is information indicating the refractive index of the medium surrounding the sample. When step S100 is completed, step S130 is executed.

[0189] In step S130, a specific process is performed. In this process, the boundary between the sample and the medium is identified. When step S130 is completed, step S200 is performed.

[0190] In step S200, a setting process is performed. In the setting process, each refractive index constituting the refractive index distribution is set. Step S200 comprises steps S201, S202, and S205.

[0191] In step S205, the fourth setting process is executed. In the fourth setting process, the refractive index based on the fourth refractive index information is set to the position corresponding to the fourth image region of the image to be processed.

[0192] In the image 200 to be processed, a signal intensity greater than zero is used as the signal intensity of the unit pixel. In the unit pixels that form the first image region 201, the signal intensity is greater than zero. Image region 202 is the region excluding the first image region 201. Therefore, in the unit pixels that form image region 202, the signal intensity is zero.

[0193] Since the first image region 201 is a region represented by a predetermined set of elements, the position of the first image region 201 can be determined from the position of the predetermined set of elements. Since image region 202 is the region excluding the first image region 201, if the position of the first image region 201 is determined, the position of image region 202 can be determined.

[0194] Since the fourth image region 211 is the region outside the boundary between the sample and the medium, the position of the fourth image region 211 can be determined from the position of the boundary. Since image region 212 is the region excluding the fourth image region 211, if the position of the fourth image region 211 is determined, the position of image region 212 can be determined.

[0195] In the first, second, and fourth setting processes, the refractive index is set. The image to be processed is the optical image of the specimen. Since the optical image of the specimen is an image formed from brightness information, the refractive index can be set, for example, using a refractive index distribution image.

[0196] Since the refractive index distribution image is an image represented by refractive index, it is possible to set the refractive index based on the first refractive index information, the refractive index based on the second refractive index information, and the refractive index based on the fourth refractive index information. In order to set the refractive index, regions are required for setting the refractive index based on the first refractive index information, the refractive index based on the second refractive index information, and the refractive index based on the fourth refractive index information.

[0197] The first refractive index information indicates the refractive index of the first structure. The image region corresponding to the first structure is the first image region 201. To set the refractive index based on the first refractive index information, it is sufficient to find the region in the refractive index distribution image that corresponds to the first image region 201.

[0198] The fourth refractive index information indicates the refractive index of the medium. The image region corresponding to the medium is the fourth image region 211. To set the refractive index based on the fourth refractive index information, it is sufficient to find the region corresponding to the fourth image region 211 in the refractive index distribution image.

[0199] The second refractive index information indicates the refractive index of the second structure. The second structure is included in structure group A. The image region containing structure group A is image region 202. However, in image region 202, it is not possible to distinguish between the image region corresponding to structure group A and the image region corresponding to the medium.

[0200] The image region corresponding to the medium is the fourth image region 211. The image region corresponding to the first structure is the first image region 201. Therefore, to find the image region containing the second structure, we just need to subtract the first image region 201 and the fourth image region 211 from image region 202. The region remaining after the exclusion (hereinafter referred to as "remaining region B") contains the image region corresponding to the second structure. To set the refractive index based on the second refractive index information, we just need to find the region corresponding to remaining region B in the refractive index distribution image.

[0201] In order to set the refractive index based on refractive index information in the refractive index distribution image, it is necessary to determine the position of the region corresponding to the first image region 201, the position of the region corresponding to the remaining region B, and the position of the region corresponding to the fourth image region 211.

[0202] The position of the first image region 201 is determined based on the signal intensity of a unit pixel in the image 200 to be processed. Therefore, the position of the region corresponding to the first image region 201 can be determined based on the signal intensity of a unit pixel in the image 200 to be processed. The refractive index based on the first refractive index information can then be set at the position determined in this way.

[0203] The position of the fourth image region 211 is determined based on the boundary between the sample and the medium in the image 210 to be processed. Therefore, the position of the region corresponding to the fourth image region 211 can be determined based on the boundary between the sample and the medium in the image 210 to be processed. The refractive index based on the fourth refractive index information can then be set at the position determined in this way.

[0204] The region corresponding to remaining region B is obtained by excluding the region corresponding to the first image region 201 and the region corresponding to the fourth image region 211 in the refractive index distribution image. Therefore, the position corresponding to remaining region B is obtained from the position of the region corresponding to the first image region 201 and the position of the region corresponding to the fourth image region 211. The refractive index based on the second refractive index information can then be set at the position obtained in this way.

[0205] In the refractive index distribution generation device of this embodiment, it is preferable that the first refractive index information and the second refractive index information are refractive index information relating to the compositional structure of the cell.

[0206] This can improve the accuracy of the refractive index distribution of the sample.

[0207] In the refractive index distribution generation device of this embodiment, the first refractive index information is the refractive index information of the cell nucleus, and the second refractive index information is the refractive index information of the cell membrane.

[0208] This can improve the accuracy of the refractive index distribution of the sample.

[0209] In the refractive index distribution generation device of this embodiment, the third refractive index information is preferably the refractive index information of the cell adhesion molecule.

[0210] This can improve the accuracy of the refractive index distribution of the sample.

[0211] In the refractive index distribution generation device of this embodiment, the fourth refractive index information is preferably the refractive index information of the cell culture medium.

[0212] This can improve the accuracy of the refractive index distribution of the sample.

[0213] In the refractive index distribution generation apparatus of this embodiment, it is preferable that the processor performs an image generation process to generate a processed image corresponding to the image to be processed. The image generation process preferably includes a division process that divides the image to be processed into a plurality of small image regions, a point image intensity distribution calculation process that uses the refractive index distribution of the image to be processed to calculate a point image intensity distribution for each small image region, a small image generation process that uses the point image intensity distribution for each small image region to generate a small image for each small image region, and a synthesis process that synthesizes the small images for each small image region to generate a processed image.

[0214] Furthermore, in the refractive index distribution generation device of this embodiment, the point image intensity distribution calculation process calculates the point image intensity distribution for each small image region using a refractive index distribution set at a position corresponding to a fifth image region located within the range in which the wavefront propagates on the image to be processed, starting from the small image region. Preferably, the fifth image region includes an image region outside the range of a sixth image region, which is an extension of the small image region in a predetermined direction. The predetermined direction is the direction from the specimen toward the observation optical system, among the optical axis directions of the observation optical system modeled in the calculation process.

[0215] In Figure 3(a), the image 50 to be processed is an image of the XZ cross-section. In the image 50 to be processed, the right edge of the image represents the image of the top surface of the specimen, and the left edge of the image represents the image of the bottom surface of the specimen. The depth of the specimen increases from the right edge of the image to the left edge of the image.

[0216] The first image region 51 represents an image region corresponding to multiple cell nuclei. If the shape of the cell nucleus is spherical, the shape of the XZ cross-section is circular. Therefore, the shape of the XZ cross-section of the cell nucleus is inherently circular, regardless of the depth from the top surface. However, in reality, the shape of the first image region 51 deforms more from a circle as the depth from the top surface increases.

[0217] Thus, in the processed image 50, deformation, a decrease in sharpness, and a decrease in brightness occur in the entire first image region 51. If deformation, a decrease in sharpness, and a decrease in brightness are considered to be image quality degradation, then image quality degradation occurs in the processed image 50.

[0218] An optical image is obtained by capturing an optical image. If there is a degradation in the image quality of an optical image, it means that degradation has occurred in the optical image itself.

[0219] When the sample is a point light source, it is desirable that the optical image be a point image. For a point image to be formed, the optical system must be aberration-free (hereinafter referred to as the "ideal optical system") and all of the light emitted from the point light source must be incident on the optical system.

[0220] However, since the size of an optical system is finite, it is not possible to allow all the light emitted from a point source to enter the optical system. In this case, the optical image is affected by diffraction. As a result, even if the optical system is ideal, a point image is not formed, and a spread image is formed. This spread image is called a point image intensity distribution.

[0221] The optical image is represented by the following equation (1) using the point image intensity distribution. I = O*PSF (1) Here, I is an optical image. O is a sample. PSF is point intensity distribution. * is convolution, That is the case.

[0222] If we consider the point image intensity distribution as an optical filter, then equation (1) shows that the optical image is obtained through this filter of the point image intensity distribution. If degradation occurs in the optical image, it means that the filter, i.e., the point image intensity distribution, has characteristics that cause deformation, a decrease in sharpness, and a decrease in brightness (hereinafter referred to as "degradation characteristics").

[0223] In the frequency domain, equation (1) can be expressed as equation (2) below. FI = FO × OTF (2) Here, FI is the Fourier transform of an optical image. FO is the Fourier transform of the sample. OTF is an optical transfer function. That is the case.

[0224] OTF is the Fourier transform of the point image intensity distribution. If the point image intensity distribution has degradation characteristics, the OTF will also have degradation characteristics.

[0225] By rearranging equation (2), equation (2) can be expressed as equation (3) below. FO = FI / OTF (3)

[0226] If we can find FI and OTF, we can find FO. Then, by performing an inverse Fourier transform on FO, we can find O. O is the sample. This operation is called deconvolution.

[0227] Image 50, the image to be processed, is an optical image of the cell nucleus. In image 50, only the cell nucleus is imaged. Therefore, when deconvolution is performed using image 50 and the OTF image, only the image of the cell nucleus is obtained.

[0228] Since the specimen is a cell aggregate, it contains multiple cytoplasm and multiple cell nuclei. However, in the processed image 50, even after deconvolution, only images of the cell nuclei are obtained. Since images of the cytoplasm are not obtained, it is difficult to say that the specimen has been identified. Deconvolution can identify the specimen, but whether or not the specimen is identified depends on the optical image.

[0229] In terms of images, equation (1) represents that the optical image is an image obtained through a filter called the point intensity distribution. If the point intensity distribution has degradation characteristics, I can be considered as the optical image with degraded image quality, and O can be considered as the optical image before the image quality degradation.

[0230] In this case, equation (3) indicates that an image of the optical image before degradation is generated from an image of the optical image with degraded quality. Hereafter, the image of the optical image with degraded quality will be referred to as the "degraded image". Furthermore, the image of the optical image before degradation can be said to be the result of image restoration in the degraded image. Therefore, the image of the optical image before degradation will be referred to as the "restored image".

[0231] To generate a recovery image, it is necessary to determine the point intensity distribution. Let n1 be the refractive index outside the sample and n2 be the refractive index inside the sample.

[0232] The ideal shape is defined as the shape of the point image intensity distribution of an ideal optical system. In the ideal shape, the refractive index between the focal plane and the ideal optical system matches a predetermined refractive index.

[0233] The specimen is moved from a position away from the focal plane towards the optical system. Since the optical system does not move, the top surface of the specimen reaches the focal plane. In this state (hereinafter referred to as the "first state"), only a space with refractive index n1 exists between the focal plane and the optical system. When a point light source is placed at the focal plane, the point image intensity distribution of the first state is obtained.

[0234] In the first state, the refractive index between the focal plane and the optical system is n1. If we assume a predetermined refractive index of n1, the point image intensity distribution in the first state is obtained based solely on this predetermined refractive index. Therefore, the shape of the point image intensity distribution in the first state is the same as the ideal shape.

[0235] Further movement of the specimen causes the focal plane to reach the interior of the specimen. In this state (hereinafter referred to as the "second state"), there are spaces with refractive index n1 and spaces with refractive index n2 between the focal plane and the optical system. When a point light source is placed on the focal plane, the point image intensity distribution of the second state is obtained.

[0236] In the second state, the refractive index between the focal plane and the optical system is determined by n1 and n2. Since n1 is a predetermined refractive index, n2 is not a predetermined refractive index. In this case, the point image intensity distribution in the second state is obtained based on a predetermined refractive index and an unpredicted refractive index. Therefore, the shape of the point image intensity distribution in the second state differs from the ideal shape.

[0237] Thus, the shape of the point image intensity distribution changes depending on the size of the space with refractive index n². Therefore, when calculating the point image intensity distribution, the refractive index distribution in the sample must be appropriately considered.

[0238] In the refractive index distribution generation device of this embodiment, the image to be processed is a degraded image, so it is sufficient to generate a recovered image from the image to be processed. In the recovered image, the shape of the first image region is the same regardless of the depth from the top surface. Therefore, a more accurate refractive index distribution can be generated.

[0239] To generate a recovered image, it is necessary to determine the point intensity distribution. In calculating the point intensity distribution, the refractive index distribution in the sample must be appropriately considered. The process for generating a recovered image is described below. In this description, the recovered image is referred to as the sample image. Furthermore, the recovered image is the image after image recovery (the processed image).

[0240] The process of generating the recovery image is performed by processor 3. The process performed by processor 3 is described below. The first image is used in the process performed by processor 3. The first image is the image to be processed.

[0241] Figure 16 is a flowchart of the processing performed by the processor. Figure 17 shows the specimen, optical image, and first image. Figure 17(a) is a three-dimensional view of the specimen and optical image. Figure 17(b) shows the XZ cross section of the specimen. Figure 17(c) shows the first image. Figure 17(d) shows the XZ cross section of the specimen and the first image. Components identical to those in Figure 2(a) are numbered the same and their descriptions are omitted.

[0242] The first image is generated from the XY image set. The XY image set is obtained from multiple optical images. As shown in Figure 17(a), when the observation optical system 221 is moved along the optical axis 222 without moving the specimen 220, multiple optical images are formed.

[0243] At position Z1, the optical image IZ1 of sample OZ1 is formed. At position Z7, the optical image IZ7 of sample OZ7 is formed. The optical images from optical image IZ1 to optical image IZ7 form optical image 230. By capturing optical image 230, an XY image set can be obtained.

[0244] XY, XZ, and YZ images can be generated from a set of XY images. All of these XY, XZ, and YZ images can be used as the first image. Let's assume that the XZ image is stored in memory as the first image.

[0245] In step S300, the first acquisition process is executed. In the first acquisition process, the first image is acquired from memory.

[0246] The first image is an optical image of the specimen in the XZ section. Figure 17(b) shows the XZ section of specimen 240. Specimen 240 is a cell aggregate. The cell aggregate is made up of multiple cells. Each cell in specimen 240 has cytoplasm 241 and a cell nucleus 242.

[0247] Figure 17(c) shows the first image acquired from memory. The first image 250 is a fluorescence image. In specimen 240, only the cell nucleus 242 is stained with fluorescence. In this case, only the optical image of the cell nucleus 242 is formed. Therefore, the first image 250 contains only the image 251 of the cell nucleus.

[0248] Since the first image 250 is an optical image, the first image 250 is a degraded image. If the shape of the cell nucleus is considered to be a circle, then in image 251 of the cell nucleus, the shape is elliptical. When step S300 is completed, step S310 is executed.

[0249] In step S310, a division process is performed. In the division process, the acquired first image is divided into multiple areas. As shown in Figure 17(d), the first image 250 is divided into 11 areas in both the X-axis and Z-axis directions. Each area is a small image region.

[0250] In Figure 17(d), the observation optical system and light rays are conveniently illustrated to show the correspondence between sample 240 and the first image 250. Observation optical system 221' is a virtual optical system, and its optical specifications are the same as those of observation optical system 221.

[0251] In an optical imaging, the up and down of the optical image of the specimen is reversed compared to the up and down of the specimen. Since the first image 250 is an image, the up and down can be reversed when generating the first image 250. Therefore, in FIG. 17(d), the up and down of the specimen 240 and the up and down of the first image 250 are the same.

[0252] The position of area 252 corresponds to position OP1. The position of area 253 corresponds to position OP2. When step S310 ends, step S320 is executed.

[0253] In step S320, a second acquisition process is executed. In the second acquisition process, the refractive index distribution of the specimen is acquired from the memory. The acquisition of the refractive index distribution will be described later. When step S320 ends, step S330 is executed.

[0254] In step S330, a calculation process is executed. The calculation process is a point image intensity distribution calculation process. In the calculation process, the point image intensity distribution is calculated for each of the divided areas using the acquired refractive index distribution. Specifically, the point image intensity distribution of the first area is calculated using the refractive index distributions of each area included in the area group. Therefore, it is necessary to determine the first area and the area group.

[0255] FIG. 18 is a diagram showing the first image, the refractive index image, the first area, and the area group. FIG. 18(a) is a diagram showing the first image, the refractive index image, and the first area. FIG. 18(b) is a diagram showing the first example of the area group. FIG. 18(c) is a diagram showing the second example of the area group.

[0256] The first area is the area for which the point image intensity distribution is calculated. In step S310, the first image 250 is divided into a plurality of areas. Therefore, the first area and the area group are determined by the areas in the first image 250.

[0257] However, the first image 250 is an optical image of the specimen. While the optical image of the specimen contains brightness information, it does not contain information on the refractive index distribution. Since the point image intensity distribution is calculated using the refractive index distribution of the area group, the first image 250 is not suitable for calculating the point image intensity distribution. In the first image 250, the first area can be determined, but the area group cannot.

[0258] The refractive index distribution can be represented by an image. In this case, the refractive index distribution of a sample can be represented by multiple distribution images (hereinafter referred to as the "distribution image group"). The distribution image group represents the refractive index distribution of the sample. Therefore, an image corresponding to the first image (hereinafter referred to as the "refractive index image") is obtained from the distribution image group. The point image intensity distribution is calculated using the refractive index distribution. Since the refractive index image is an image of the refractive index distribution, the refractive index image is suitable for calculating the point image intensity distribution.

[0259] The refractive index image should be stored in memory 2, and then read from memory 2 when the calculation process is executed. Since the first image is the XZ image, the refractive index image is the image of the XZ cross-section.

[0260] Image 1, 250, is divided into multiple areas. Therefore, as shown in Figure 18(a), the refractive index image 260 is also divided into multiple areas. The refractive index image 260 is divided into 11 areas in both the X-axis and Z-axis directions. For clarity, only cell nuclei are shown in the refractive index image 260.

[0261] Similar to Figure 17(d), the top and bottom of the refractive index image 260 and the top and bottom of the first image 250 coincide. Furthermore, for convenience, the observation optical system 221' and the light rays are shown to illustrate the correspondence between the refractive index image 260 and the first image 250.

[0262] In refractive index image 260, the area corresponding to area 252 is area 261. The area corresponding to area 253 is area 262. The area corresponding to area 254 is area 263.

[0263] As described above, the refractive index image 260 is suitable for calculating the point image intensity distribution. Therefore, the first area and the area group are determined using the refractive index image 260.

[0264] Let's describe the first example of an area group. Figure 18(b) shows area 261, observation optical system 221', ray 270, and optical axis 271 of the observation optical system. Since no light rays are emitted from the image, ray 270 is a virtual ray.

[0265] In the first example, the first area in the first image 250 is area 252. The area corresponding to area 252 in the refractive index image 260 is area 261. Therefore, in the refractive index image 260, area 261 is the first area.

[0266] The area group and the predetermined direction are defined as follows: The area group consists of multiple areas located inside the range from which light rays radiate in the predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in the predetermined direction. The predetermined direction in the first image is the direction in which the observation optical system exists within the optical axis direction of the virtual observation optical system.

[0267] As described above, the first area and area group are determined using the refractive index image 260. Therefore, in the above specification, the first image is replaced with the refractive index image. In this case, the area group and the predetermined direction are defined as follows.

[0268] The area group in the refractive index image consists of multiple areas located inside the range in which light rays radiate in a predetermined direction, starting from the first area, and includes areas outside the range extended from the first area in the predetermined direction. The predetermined direction in the refractive index image is the direction in which the virtual observation optical system exists within the optical axis direction of the observation optical system.

[0269] In the refractive index image 260, the side closer to the observation optical system 221' is considered the top surface of the specimen, and the side further from the observation optical system 221' is considered the bottom surface of the specimen. Area 261 is located at the point where it intersects with the optical axis 271 on the top surface 260a. Light rays 270 are emitted from area 261. The light rays 270 emitted from area 261 are incident on the observation optical system 221'.

[0270] Light ray 270 is light incident on the observation optical system 221'. The amount of light incident on the observation optical system 221' is determined by the object-side numerical aperture of the observation optical system 221'. As described above, in the specimen image generation device 1, optical information is stored in memory 2.

[0271] The optical information includes information about the numerical aperture of the objective lens. The numerical aperture of the objective lens can be considered as the object-side numerical aperture of the observation optical system 221'. Therefore, the light ray 270 can be identified from the numerical aperture of the objective lens.

[0272] Figure 18(b) illustrates a predetermined direction 272 and an unpredictable direction 273. The predetermined direction 272 and the unpredictable direction 273 are the optical axis directions of the observation optical system 221'. Of the optical axis directions of the observation optical system 221', the observation optical system 221' is located in the predetermined direction 272, but not in the unpredictable direction 273.

[0273] The two rays 270 are rays of synchrotron radiation emitted from area 261. The area of ​​refractive index image 260 is not located inside the region between the two rays 270. Therefore, at the location of area 261, the number of areas in the area group is zero.

[0274] A second example of area groups will be explained. Figure 18(c) shows area 263, central area 264, peripheral area 265, and peripheral area 266. The same numbering is used for components identical to those in Figure 18(b), and the explanation is omitted. The area group is the fifth image region. The central area is the sixth image region. The peripheral areas are image regions outside the range of the sixth image region.

[0275] In the second example, the first area in the first image 250 is area 254. The area corresponding to area 254 is area 263 in the refractive index image 260. Therefore, in the refractive index image 260, area 263 is the first area.

[0276] Area 263 is located at a place where it intersects the optical axis 271 on the bottom surface 260b. Ray 270 and ray 274 are radiated from area 263. The rays 270 and 274 radiated from area 263 enter the observation optical system 221'. Ray 274 is a virtual ray.

[0277] The two rays 270 and the two rays 274 are rays radiated from area 263. When the scattering of light in the specimen is very small, the rays radiated from area 263 are represented by the two rays 270. Inside the range sandwiched by the two rays 270, a central area 264 and a peripheral area 265 are located. An area group is formed by the central area 264 and the peripheral area 265. An area that intersects ray 270 is regarded as being included in the area group.

[0278] The central area 264 and the peripheral area 265 are each composed of a plurality of areas. Therefore, the area group is composed of a plurality of areas.

[0279] In the area group, the central area 264 is located in a range extending area 263 toward the predetermined direction 272. The peripheral area 265 is located outside the central area 264.

[0280] When the scattering of light in the specimen is very large, the rays radiated from area 263 are represented by the two rays 274. Inside the range sandwiched by the two rays 274, a central area 264, a peripheral area 265, and a peripheral area 266 are located. Therefore, center An area group is formed by the area 264, the peripheral area 265, and the peripheral area 266. An area that intersects ray 274 is regarded as being included in the area group.

[0281] The central area 264, surrounding area 265, and surrounding area 266 are each composed of multiple areas. Therefore, the area group is composed of multiple areas.

[0282] Figure 19 shows an area group. Figure 19(a) shows a third example of an area group. Figure 19(b) shows a fourth example of an area group.

[0283] In the third example, as shown in Figure 19(a), area 267 is the first area. Area 267 is located between the top and bottom surfaces, where it intersects with the optical axis 271.

[0284] When light scattering in the sample is very small, the rays emitted from area 267 are represented by two rays 270. Inside the region between the two rays 270 are the central area 264 and the peripheral area 265. The central area 264 and the peripheral area 265 form an area group. Comparing the third example with the second example, the third example has fewer areas in its area group.

[0285] When light scattering in the sample is very large, the rays emitted from area 267 are represented by two rays 274. Inside the area between the two rays 274 are the central area 264, the peripheral area 265, and the peripheral area 266. The central area 264, the peripheral area 265, and the peripheral area 266 form a group of areas.

[0286] In the fourth example, as shown in Figure 19(b), area 268 is the first area. Area 268 is located on the bottom surface, away from the optical axis 271.

[0287] In the case where light scattering in the sample is very small, the rays emitted from area 268 are represented by two rays 270. Inside the range between the two rays 270 are the central area 264 and the peripheral area 265. The central area 264 and the peripheral area 265 form an area group. Comparing the fourth example with the second example, the fourth example has fewer areas in its area group.

[0288] In cases where light scattering in the sample is very large, the light rays emitted from area 268 are represented by two rays 274. Inside the area between the two rays 274 are the central area 264, the peripheral area 265, and the peripheral area 266. Area groups are formed by the central area 264, the peripheral area 265, and the peripheral area 266.

[0289] In the first example, the number of areas in the area group is zero. Therefore, the refractive index distribution of each area included in the area group is not used in the calculation process. The point image intensity distribution is calculated using the refractive index of the space between the refractive index image 260 and the observation optical system 221'. The calculation for the case where the number of areas in the area group is zero is also included in the calculation process.

[0290] In the second, third, and fourth examples, areas located outside the surrounding area 266 are not included in the area group. Therefore, these areas are not used in calculating the point image intensity distribution. However, these areas may be used in calculating the point image intensity distribution. That is, all areas located on the side of the predetermined direction 272 from the first area may be considered as an area group, and the point image intensity distribution may be calculated.

[0291] The first area is the area for which the point image intensity distribution is calculated. Therefore, by changing the area targeted by the first area, the point image intensity distribution can be calculated for each of the divided areas.

[0292] In the calculation process, the point image intensity distribution of the first area is calculated using the refractive index distribution of each area included in the area group. In the first image 250, the first area is area 254 in the second example. In the refractive index image 260, area 263 corresponds to area 254. In the refractive index image 260, the area group consists of either a central area 264 and a peripheral area 265, or a central area 264, a peripheral area 265, and a peripheral area 266.

[0293] Therefore, the point image intensity distribution of area 263 is calculated using the refractive index distribution of each area constituting the central area 264 and the refractive index distribution of each area constituting the peripheral area 265, or the point image intensity distribution of area 263 is calculated using the refractive index distribution of each area constituting the central area 264, the refractive index distribution of each area constituting the peripheral area 265, and the refractive index distribution of each area constituting the peripheral area 266. The point image intensity distribution of area 263 can then be treated as the point image intensity distribution of area 254 in the first image 250.

[0294] Area 254 is the first area in the first image 250. Each area in the first image 250 already contains information about the brightness of the optical image of the specimen. Therefore, an image with a point image intensity distribution (hereinafter referred to as the "PSF image") is generated separately from the first image 250.

[0295] Figure 20 shows the first image, refractive index image, and PSF image. Figure 20(a) shows the first image and refractive index image. Figure 20(b) shows the refractive index image and PSF image. Figure 20(a) is the same as Figure 18(a), so no explanation is given.

[0296] The first image 250 is divided into multiple areas. Therefore, as shown in Figure 20(b), the PSF image 280 is also divided into multiple areas. The PSF image 280 is divided into 11 areas in both the X-axis and Z-axis directions.

[0297] As can be seen from comparing Figure 20(a) and Figure 20(b), in PSF image 280, the area corresponding to area 281 is area 252. The area corresponding to area 282 is area 253. The area corresponding to area 283 is area 254. Therefore, each area in PSF image 280 has the point intensity distribution of the area corresponding to the first area in the first image 250. In Figure 20(b), the point intensity distribution is illustrated for only some areas.

[0298] Let's return to Figure 16 for explanation. Once step S330 is completed, step S340 is executed.

[0299] In step S340, the first generation process is executed. The first generation process is a small image generation process. In the first generation process, a second image corresponding to each area is generated using the point image intensity distribution calculated for each area. The second image is the small image.

[0300] Figure 21 shows the first, second, and third images. Figure 21(a) shows the first and second images. Figure 21(b) shows the first and third images.

[0301] Figure 21(a) shows a portion of the first image, a portion of the PSF image, and a group of second images.

[0302] Area DEG is a part of the first image 250. Area DEG is formed by Area DEG1, Area DEG2, Area DEG3, Area DEG4, Area DEG5, and Area DEG6.

[0303] Area PSF is a part of PSF image 280 and corresponds to Area DEG. Area PSF is formed by Area PSF1, Area PSF2, Area PSF3, Area PSF4, Area PSF5, and Area PSF6.

[0304] The second image group REC consists of images of the areas corresponding to area DEG and area PSF. The second image group REC is composed of second image REC1, second image REC2, second image REC3, second image REC4, second image REC5, and second image REC6.

[0305] The image for area DEG1 is the first image. The image for area PSF1 is the point intensity distribution. The second image, REC1, is generated from the images for area DEG1 and area PSF1.

[0306] In area DEG, the shape of the cell nucleus is elliptical. In the second image, the shape of the cell nucleus is circular. Therefore, in the first generation process, a restored image is generated from the degraded image. When step S340 is completed, step S350 is executed.

[0307] In generating the second image, it is advisable to apply a masking process to the images of each area DEG. One example of masking is blurring the edges of the image.

[0308] In step S350, a third image is generated. The process of generating the third image is a synthesis process. The third image is the image corresponding to the first image. In generating the third image, the second images corresponding to each area are synthesized. The third image is the processed image.

[0309] Figure 21(b) shows the first image 250 and the third image 290. The third image 290 is generated by combining it with the second image. The second image is generated based on the first image 250, and the third image is generated based on the second image. Therefore, the third image 290 is the image corresponding to the first image 250.

[0310] In the first image 250, the shape of the cell nucleus is elliptical. In the third image 290, the shape of the cell nucleus is circular. Therefore, the specimen image generation device 1 can generate high-quality restored images from degraded images.

[0311] When combining the second image, it is advisable to use weighting. For two adjacent second images, the influence of one second image should be halved at the boundary between the two images.

[0312] Figure 22 shows the first and third images. Figure 22(a) shows the first image. Figure 22(b) shows the third image.

[0313] The right edge of the image represents the top surface of the specimen, and the left edge represents the bottom surface of the specimen. The image quality is higher in the third image than in the first image across the entire range from the top to the bottom surface of the specimen.

[0314] As described above, the calculation of the point image intensity distribution utilizes not only the refractive index distribution of the central area but also the refractive index distribution of the surrounding area. Therefore, compared to recovery techniques that use only the refractive index distribution of the central area, the point image intensity distribution can be calculated with higher accuracy. As a result, images can be recovered with higher accuracy.

[0315] As explained in Figure 18(c), the area group is determined by the range of light emitted starting from the first area. Light emitted from the first area passes through the area group and enters the observation optical system. The point image intensity distribution is then obtained from the light emitted from the observation optical system.

[0316] Light radiating from the first area is obtained by setting a point light source in the first area. A wavefront originating from the point light source is emitted. If this wavefront is designated as the first wavefront, the point image intensity distribution of the first area can be calculated using the first wavefront.

[0317] Specifically, the second wavefront is calculated using the first wavefront and the refractive index distribution corresponding to each area included in the area group. The intensity distribution corresponding to the third wavefront is then calculated using the calculated second wavefront, and the point image intensity distribution of the first area is calculated using the calculated intensity distribution. The second wavefront is the wavefront propagated through the sample in a predetermined direction, and the third wavefront is the wavefront at the position of the focal plane of the virtual observation optical system.

[0318] Figure 23 shows the propagation of a wavefront. Components identical to those in Figure 18(c) are numbered the same way, and their explanations are omitted.

[0319] The refractive index distribution is used to calculate the point image intensity distribution. Therefore, we will explain using refractive index image 260. In refractive index image 260, area 263 corresponds to the first area. Therefore, area 263 is located on the focal plane FP. Also, a point light source 300 is set in area 263.

[0320] A first wavefront WF1 is emitted from the point light source 300. The first wavefront WF1 propagates from area 263 toward the upper surface 301 of the refractive index image 260. The upper surface 301 is the outer edge of the sample. The observation optical system 302 is located on the side of the upper surface 301. Therefore, the first wavefront WF1 propagates in a predetermined direction.

[0321] Wavefront propagation can be calculated using simulation. The observation optical system 302 is a virtual optical system, for example, formed by an objective lens 303 and an imaging lens 304. The optical specifications of the observation optical system 302 are the same as those of the observation optical system 221. Optical specifications, such as magnification and numerical aperture, can be obtained based on various information.

[0322] The first wavefront WF1 propagates through the area group and reaches the upper surface 301. The second wavefront WF2 is emitted from the upper surface 301. The second wavefront WF2 is the wavefront after propagating through the area group. The area group is formed by the central area 264 and the surrounding area 265. Therefore, the second wavefront WF2 can be calculated using the refractive index distribution of each area included in the area group.

[0323] In the observation optical system 302, the focal plane FP and the image plane IP are conjugate. In order to determine the point image intensity distribution 305 at the image plane IP, the wavefront at the focal plane FP is required. The second wavefront WF2 is located on the upper surface 301. By propagating the second wavefront WF2 to the focal plane FP, the third wavefront WF3 can be obtained as the wavefront at the focal plane FP.

[0324] The imaging optical system 302 forms a Fourier optical system. The point image intensity distribution 305 corresponding to the imaging plane of the third wavefront WF3 can be calculated using the pupil function of the imaging optical system 302. The calculation formula is shown below. In the calculation formula, WF3 is the third wavefront, P is the pupil function of the imaging optical system 132, and U 135 I is the wavefront in the image plane. 135 This represents the intensity distribution on the image plane.

number

[0325] In the beam propagation method, the object model is replaced with multiple thin layers. Then, the wavefront changes as light passes through each layer are calculated sequentially to compute an image of the object model. The beam propagation method is disclosed, for example, in "High-resolution 3D refractive index microscopy of multiple-scattering samples from intensity images" Optica, Vol. 6, No. 9, pp.1211-1219 (2019).

[0326] In the sample image generation device of this embodiment, since image generation processing is performed, even if the image to be processed is a degraded image, the image can be restored with high accuracy. As a result, the accuracy of the refractive index distribution of the sample can be improved.

[0327] The area groups differ in size between Figure 23 and Figure 18(c). The extent of surrounding area 265 in Figure 23 is larger than the sum of surrounding area 265 and surrounding area 266 in Figure 18(c). In Figure 23, all areas located to the side of the first area in a predetermined direction are considered as an area group, and the point image intensity distribution is calculated.

[0328] To accurately calculate the point image intensity distribution 305, it is preferable to use the refractive index distributions of all areas that make up the area group. In Figure 23, since area 263 is the first area, the refractive index distributions of all areas located between area 263 and the top surface 301 are used. Using the refractive index distributions of all areas located between the first area and the top surface of the sample allows for a more accurate calculation of the point image intensity distribution.

[0329] In the refractive index distribution generation apparatus of this embodiment, it is preferable that the processor uses the processed image as the image to be processed and performs refractive index distribution generation processing and image generation processing.

[0330] Figure 24 shows the refractive index image, PSF image, first image (degraded image), and recovered image. These images are from the process of generating the recovered image (hereinafter referred to as the "recovery process").

[0331] Each image in image group 400 is a refractive index image. Image group 400 includes refractive index image 400a, refractive index image 400b, and refractive index image 400c. Each image in image group 410 is a PSF image. Image group 410 includes PSF image 410a and PSF image 410b.

[0332] Each image in image group 420 is a first image. Image group 420 includes first image 420a and first image 420b. Each image in image group 430 is a recovery image. Image group 430 includes recovery image 430a and recovery image 430b.

[0333] Images 400, 410, 420, and 430 show images of cell nuclei. For ease of viewing, the number of cell nuclei has been reduced and their shapes have been enlarged. The images in each image group are XZ cross-sections. The right edge of the image is the top surface of the specimen, and the left edge is the bottom surface of the specimen.

[0334] A PSF image 410a is obtained from the refractive index image 400a. A recovered image 430a is obtained from the first image 420a and the PSF image 410a. Since the first image 420a is a degraded image, a recovery process is performed to obtain the recovered image 430a from the degraded image.

[0335] Comparing the first image 420a with the recovered image 430a, the shape of the cell nuclei located deeper from the top surface in the recovered image 430a is closer to a circle compared to the first image 420a.

[0336] The outline of the cell nucleus can be extracted from the recovered image 430a. Based on the extracted outline, the first image 420b and the refractive index image 400b can be generated.

[0337] From the refractive index image 400b, the PSF image 410b is obtained. From the first image 420b and the PSF image 410b, the recovered image 430b is obtained. Since the first image 420b is a degraded image, the recovered image 430b is obtained from the degraded image by performing a recovery process.

[0338] Comparing the first image 420b with the recovered image 430b, the shape of the cell nuclei located deeper from the top surface in the recovered image 430b is closer to a circle compared to the first image 420b.

[0339] The outline of the cell nucleus can be extracted from the recovered image 430a. Based on the extracted outline, the refractive index image 400c can be generated.

[0340] Comparing refractive index images 400a, 400b, and 400c, it can be seen that with each recovery process, the outline of the cell nuclei located deeper from the top surface approaches a circular shape.

[0341] Each image in image group 400 is a refractive index image. Refractive index images are generated from a group of distribution images. In the distribution image group, the refractive index distribution of the sample is represented by multiple images.

[0342] In the refractive index distribution generation device of this embodiment, a refractive index distribution generation process is performed. A refractive index distribution image is used in the refractive index distribution generation process. The refractive index distribution image is an image represented by refractive index. The refractive index distribution image is set to a refractive index based on the sample.

[0343] Thus, both the refractive index image in the recovery process and the refractive index distribution image in the refractive index distribution generation process represent the refractive index of the sample. Therefore, if each image in the image group 400 is considered a refractive index distribution image, the refractive index distribution generation device of this embodiment includes a recovery process, and thus, even with thick samples, the accuracy of the refractive index distribution can be further improved.

[0344] In the refractive index distribution generation device of this embodiment, the input process preferably involves inputting a second image of the second sample taken from memory, and the processor executes a refractive index determination process to determine the refractive index of the first structure. The refractive index determination process includes a refractive index distribution calculation process that calculates the refractive index distribution of the second sample from a plurality of second images to be processed, a second identification process that identifies a seventh image region corresponding to the first structure in the second image to be processed, and a third identification process that identifies the refractive index corresponding to the seventh image region among the refractive indices constituting the refractive index distribution of the second sample. The input process preferably involves inputting the refractive index of the first structure determined in the refractive index determination process from memory.

[0345] As described above, in the refractive index distribution generator 1, the image to be processed is stored in memory 2. The image to be processed can be acquired by the microscope system 20. The microscope system 20 can acquire a second image to be processed.

[0346] Figure 25 is a flowchart of the processing performed by the processor. Processor 3 performs the refractive index determination process. The refractive index determination process is performed before inputting the first refractive index information from memory.

[0347] In step S400, the second image to be processed is input from memory. The second image to be processed is the image acquired by photographing the second specimen. The second specimen is the same specimen used to acquire the image to be processed.

[0348] The second image to be processed may be acquired using the same observation method as the first image to be processed, or it may be acquired using a different observation method. Fluorescence images, stained images, and phase-contrast images can be used as the second image to be processed.

[0349] In step S410, a refractive index determination process is performed. In the refractive index determination process, the refractive index of the first structure is determined. Step S410 comprises steps S411, S412, S413, and S414.

[0350] In step S411, the refractive index distribution calculation process is performed. In the refractive index distribution calculation process, the refractive index distribution of the second sample is calculated from multiple second processing target images. The refractive index distribution calculation process can be performed by computational imaging.

[0351] This section describes the estimation of refractive index distribution using computational imaging. The estimation uses the optical image of the sample and the optical image of the estimated sample. The optical image of the estimated sample can be obtained through simulation using a virtual optical system. Since the sample is a three-dimensional object, the estimated sample is also a three-dimensional object. In this case, the optical image of the estimated sample is represented by multiple estimated XY images (hereinafter referred to as the "estimated XY image group").

[0352] The refractive index distribution of a sample can be represented by the distribution image set described above. The refractive index distribution of an estimated sample can be represented by multiple estimated distribution images (hereinafter referred to as the "estimated distribution image set").

[0353] Processor 3 performs estimation of the estimated distribution image set using computational imaging. The estimation involves comparing the XY image set with the estimated XY image set. Specifically, the refractive index values ​​in the estimated distribution image set are changed so that the difference between the XY image set and the estimated XY image set is minimized.

[0354] The difference between the two images can be expressed numerically. Therefore, the comparison of the two images and the change in refractive index value are repeated until this numerical value falls below a threshold. The estimated distribution images obtained when this numerical value falls below the threshold are then defined as the distribution image group. The distribution image group represents the refractive index distribution of the sample. Thus, the refractive index distribution of the sample is determined. When step S411 is completed, step S412 is executed.

[0355] In step S412, a second identification process is performed. In the second identification process, a seventh image region corresponding to the first structure is identified in the second image to be processed. When step S412 is completed, step S413 is performed.

[0356] In step S413, a third identification process is performed. In the third identification process, the refractive index corresponding to the seventh image region is identified among the refractive indices that constitute the refractive index distribution of the second sample. When step S413 is completed, step S414 is performed.

[0357] In step S414, the refractive index is input into memory. This refractive index corresponds to the seventh image region. When step S414 is completed, step S102 is executed.

[0358] As described above, in step S102, the first refractive index information is input from memory. The memory stores the refractive index corresponding to the seventh image region. The refractive index corresponding to the seventh image region is the refractive index of the first structure determined in the refractive index determination process. Therefore, the refractive index of the first structure determined in the refractive index determination process is input from memory.

[0359] The image used to identify the seventh image region is different from the image used to identify the first image region. The refractive index corresponding to the seventh image region is the refractive index estimated by computational imaging. Since both the seventh and first image regions are image regions corresponding to the first structure, the refractive index corresponding to the seventh image region can be used as the refractive index corresponding to the first image region.

[0360] In the refractive index distribution generating apparatus of this embodiment, it is preferable that the maximum depth of the second sample is less than 50 μm, and the minimum depth of the sample is 50 μm or more.

[0361] The refractive index distribution generation system of this embodiment comprises an observation optical system for forming an optical image of a specimen, an image sensor for capturing the optical image, and the refractive index distribution generation device described in claim 1.

[0362] According to the refractive index distribution generation system, the accuracy of the refractive index distribution can be improved even with thick samples.

[0363] The refractive index distribution generation system of this embodiment comprises a hardware-based processor and a hardware-based memory. The processor executes a refractive index distribution generation process to generate a refractive index distribution corresponding to an image to be processed. The refractive index distribution generation process includes an input process that inputs the image to be processed, first refractive index information indicating the refractive index of a first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure from the memory, and a setting process that sets each refractive index constituting the refractive index distribution. The setting process includes a first setting process that sets the refractive index based on the first refractive index information at a position corresponding to a first image region of the image to be processed, based on the signal intensity of a unit pixel of the image to be processed, and a second setting process that sets the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the image to be processed. The first image region is an image region corresponding to a first structure, and a unit pixel consists of one or more pixels. The processor executes the refractive index distribution generation process using an image of a specimen as the image to be processed. The processor performs machine learning processing to train the AI ​​model. This machine learning processing trains the AI ​​model on multiple datasets, which include the image to be processed and the corresponding training data. The training data consists of refractive index distributions generated by the refractive index distribution generation process.

[0364] This embodiment Refractive index distribution generation system Next, a refractive index distribution image is generated from the image to be processed. The image to be processed is a degraded image, and the refractive index distribution image is a restored image. If the refractive index distribution image is considered as training data, then both the image to be processed and the refractive index distribution image can be used as data for machine learning. Hereafter, the image to be processed will be referred to as the image before improvement, and the refractive index distribution image will be referred to as the improved image.

[0365] The improved images can be generated using an AI model trained with supervised machine learning (hereinafter referred to as "supervised ML").

[0366] AI models perform inference based on patterns found during data analysis in the training process, providing the ability to execute tasks on a computer system without explicit programming.

[0367] AI models can be trained continuously or periodically before performing inference processing.

[0368] Supervised machine learning (ML) AI models include algorithms that are trained on existing sample data and training data, and then make predictions about new data. Training data is also called teacher data.

[0369] Such algorithms work by building an AI model from sample and training data to make data-driven predictions or decisions that are expressed as results.

[0370] In supervised machine learning, when training is performed, sample data and training data are input, and the system learns a function that best approximates the relationship between input and output. When the trained AI model performs inference, it implements the same function when new data is input and generates the corresponding output.

[0371] Examples of commonly used supervised machine learning algorithms include logistic regression (LR), naive Bayes, random forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and support vector machines (SVM).

[0372] The training process in this embodiment can perform supervised ML processing. The training process trains or learns the AI ​​model.

[0373] When the training process is executed, a sufficient number of datasets are input into the input layer of the AI ​​model, and these datasets are propagated through the AI ​​model to the output layer.

[0374] Figure 26 shows the training process. The dataset includes the original image and the improved image. The original image is sample data. The improved image is training data or target data corresponding to the sample data. In Figure 26, the sample data is shown as Image 1, Image 2, etc. The improved image is Training Data 1, Training Data 2, etc.

[0375] During the training process, the optimal parameters for generating estimated data from sample data are searched for and updated, for example, using a loss function. Estimated data is generated from the input sample data, and the generated estimated data is used for training. data The difference is evaluated using a loss function, and the parameters that minimize the value of the loss function are searched for.

[0376] In this embodiment, the inference process can execute an inference process that outputs inference data when new data to be inferred is input to the trained AI model.

[0377] When the inference process is executed, the image before enhancement is input to the input layer of the AI ​​model, and the enhancement is propagated through the AI ​​model to the output layer.

[0378] By performing inference processing, it is possible to generate an improved image from the original image.

[0379] Figure 27 shows the sample image generation system of this embodiment. Figure 27(a) shows the sample image generation system of the first example. Figure 27(b) shows the sample image generation system of the second example. Figure 27(c) shows the sample image generation system of the third example.

[0380] As shown in Figure 27(a), in the first example of the sample image generation system, the sample image generation system 500 consists only of the sample image generation device of this embodiment. In this case, the processor 3 of the sample image generation device (hereinafter referred to as the "first processor") can perform training and inference processing.

[0381] The sample image generation device 1 may include a first processor and a second processor. The second processor is different from the first processor. Training and inference processes can be performed on the second processor.

[0382] The sample image generation device 1 may include a first processor, a second processor, and a third processor. The third processor is different from the first and second processors. The second processor can perform training processing, and the third processor can perform inference processing.

[0383] The memory 2 of the sample image generation device 1 stores the image before improvement used in the training process, the improved image, and the image before improvement used in the inference process.

[0384] As shown in Figure 27(b), in the second example of the sample image generation system, the sample image generation system 510 consists of the sample image generation device 1 of this embodiment and the learning inference device 520. The learning inference device 520 includes a memory 521 and a processor 522.

[0385] The learning and inference device 520 can perform training and inference processes. In this case, the learning and inference device 520 includes memory and one or more processors. The inference process can be performed on the same processor as the training process. The inference process may also be performed on a different processor than the training process.

[0386] The learning inference device's memory 521 stores the image before improvement used in the training process, the improved image, and the image before improvement used in the inference process.

[0387] As shown in Figure 27(c), in the third example of the sample image generation system, the sample image generation system 530 consists of the sample image generation device 1 of this embodiment, a learning device 540, and an inference device 550. The learning device 540 performs training processing, and the inference device 550 performs inference processing.

[0388] The learning device 540 includes a memory 541 and a processor 542. The inference device 550 includes a memory 551 and a processor 552. The processor 542 of the learning device 540 can perform training processing, and the processor 552 of the inference device 550 can perform inference processing.

[0389] The memory 541 of the learning device 540 stores the image before improvement and the improved image used in the training process. The memory 551 of the inference device 550 stores the image before improvement used in the inference process.

[0390] The learning and inference device 510 and the learning device 540 described above receive data used for training from the image generation device 1 via communication or a recording medium such as a USB memory, and store it in the memory provided by each device.

[0391] According to the refractive index distribution generation system, the accuracy of the refractive index distribution can be improved even with thick samples.

[0392] The refractive index distribution generation method of this embodiment is a method for generating a refractive index distribution corresponding to an image to be processed, and involves inputting an image to be processed, first refractive index information indicating the refractive index of a first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure, setting the refractive index based on the first refractive index information at a position corresponding to a first image region of the image to be processed based on the signal intensity of a unit pixel of the image to be processed, and setting the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the image to be processed. The first image region is an image region corresponding to a first structure, and a unit pixel consists of one or more pixels, and the image to be processed is an image of a specimen.

[0393] According to the refractive index distribution generation method of this embodiment, the accuracy of the refractive index distribution can be improved even with thick samples.

[0394] The recording medium of this embodiment is a computer-readable recording medium that stores a program for generating a sample image. The recording medium performs an input process that inputs an image to be processed, first refractive index information indicating the refractive index of a first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure from memory, and a setting process that sets each refractive index constituting the refractive index distribution. In the setting process, the recording medium performs a first setting process that sets the refractive index based on the first refractive index information at a position corresponding to a first image region of the image to be processed, based on the signal intensity of a unit pixel of the image to be processed, and a second setting process that sets the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the image to be processed. The refractive index distribution generation process is then executed using an image of a sample as the image to be processed. The first image region is an image region corresponding to a first structure, and a unit pixel consists of one or more pixels.

[0395] The present invention is suitable for a refractive index distribution generating device, refractive index distribution generating method, refractive index distribution generating system, and recording medium that can improve the accuracy of the refractive index distribution even with thick specimens. [Industrial applicability]

[0396] According to the recording medium of this embodiment, the accuracy of the refractive index distribution can be improved even with thick samples. [Explanation of Symbols]

[0397] 1 Refractive index distribution generator 2 memory 3 processors 4 Input section 10 Microscope Systems 20 Microscopes 21 Main unit 22 Objective lenses 23 stages 24 Incident Illumination Device 25 Imaging Unit 26 controllers 27 specimens 30 Processing Unit 31 Input section 32 memory 33 processors 34 Output section 40 specimens 41 Observation Optical System 42 Optical axis 43 XY image group 50, 60, Images to be processed 51, 61, First image region 52, 62, Second image region 70, 80, Refractive index distribution image 71, 81, First refractive index region 72, 82, Second refractive index region 90 Images to be processed 91 First Image Region 92 Second Image Region 100, 110 refractive index distribution images 101, 111 First refractive index region 102, 112 Second refractive index region 120, 130 Images to be processed 121, 131 First image region 122, 132 Second image region 140, 150 Refractive Index Distribution Images 141, 151 First refractive index region 142, 152 Second refractive index region 160, 170 Images to be processed 161, 171 First image region 162, 172 Second image region 180, 190 Images to be processed 181, First Image Region Image regions 182, 192 191 Third Image Region Images 200 and 210 to be processed 201 First Image Region Image regions 202, 212 211 Third Image Region 220 specimens 221 Observation Optical System 221' Observation Optical System 222 Optical axis 230 Optical image 240 specimens 241 Cytoplasm 242 cell nucleus 250 Image 1 251 Image of a cell nucleus Areas 252, 253, and 254 260 Refractive Index Image 260a top 26 0 b Bottom Areas 261, 262, 263, 267, 268 264 Central Area Areas surrounding 265 and 266 270, 274 rays 271 Optical axis 272 predetermined direction 273 Out of the designated direction 280 PSF images Areas 281, 282, and 283 290 Third image 300 point light source 301 Top surface 302 Observation Optical System 303 Objective lens 304 Imaging lens 305 Point spread intensity distribution Image group 400, 410, 420, 430 400a, 400b, 400c refractive index images 410a, 410b, 410c PSF images 420a, 420b, 420c First image 430a, 430b Recovery Images 500, 510, 530 Sample Image Generation System 520 Learning Inference Machine 521 memory 522 processors 540 Learning device 541 memory 542 processors 550 Reasoning device 551 memory 552 processors FP focal plane DEG, PSF, REC area WF1 1st wave WF2 2nd wave WF3 3rd wave

Claims

1. A processor composed of hardware, memory composed of hardware, and equipped, The aforementioned processor, The refractive index distribution generation process is executed to generate the refractive index distribution corresponding to the image to be processed. The aforementioned refractive index distribution generation process is performed by Input processing involves inputting the image to be processed, first refractive index information indicating the refractive index of the first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure, from the memory. This includes a setting process for setting each refractive index that makes up the refractive index distribution, The aforementioned configuration process is: A first setting process in which the refractive index based on the first refractive index information is set at a position corresponding to the first image region of the image to be processed, based on the signal intensity of the unit pixel of the image to be processed, The process includes a second setting process which sets the refractive index based on the second refractive index information to a position corresponding to an image region different from the first image region of the image to be processed, The first image region is an image region corresponding to the first structure, The unit pixel is composed of one or more pixels. The refractive index distribution generation device is characterized in that the processor performs the refractive index distribution generation process using an image of a sample taken as the image to be processed.

2. The first setting process sets a refractive index based on the first refractive index information at a position corresponding to the first image region, which is composed of first unit pixels among the unit pixels whose signal intensity value is greater than a threshold, The refractive index distribution generating apparatus according to claim 1, characterized in that the second setting process sets a refractive index based on the second refractive index information at a position corresponding to an image region composed of second unit pixels among the unit pixels whose signal intensity value is less than or equal to the threshold.

3. The image to be processed is an image in which the depth of the sample increases along the direction from one end to the other. The aforementioned one end is located on the opposite side from the other end, with the center of the image to be processed in between. The refractive index distribution generation process includes a first correction process that improves the brightness at deeper depths of the sample relative to the brightness at shallower depths of the sample. The refractive index distribution generating apparatus according to claim 1, characterized in that the first setting process sets the refractive index based on the first refractive index information based on the signal intensity of the unit pixel of the image to be processed in which the brightness at the deep position has been improved.

4. The refractive index distribution generation process includes a second correction process that corrects the first refractive index information and generates corrected refractive index information based on the signal intensity of the first unit pixel. The second correction process generates corrected refractive index information in which the refractive index corresponding to the first unit pixel whose signal intensity is less than the maximum signal intensity is smaller than the refractive index corresponding to the unit pixel with the maximum signal intensity among the first unit pixels. The refractive index distribution generating apparatus according to claim 2, characterized in that the first setting process sets the refractive index indicated by the corrected refractive index information at a position corresponding to the first image region.

5. The image to be processed is an image in which the depth of the sample increases along the direction from one end to the other. The aforementioned one end is located on the opposite side from the other end, with the center of the image to be processed in between. The refractive index distribution generation process includes a first correction process that improves the brightness at deeper depths of the sample relative to the brightness at shallower depths of the sample. The refractive index distribution generating apparatus according to claim 4, characterized in that the first setting process corrects the first refractive index information corresponding to the first image region of the image to be processed in which the brightness at the deep position has been improved, and generates the corrected refractive index information.

6. The refractive index distribution generating apparatus according to claim 2, characterized in that the first setting process sets the refractive index indicated by the first refractive index information at a position corresponding to the first image region composed of the first unit pixels.

7. The image to be processed is an image in which the depth of the sample increases along the direction from one end to the other. The aforementioned one end is located on the opposite side from the other end, with the center of the image to be processed in between. The refractive index distribution generation process includes a first correction process that improves the brightness at deeper depths of the sample relative to the brightness at shallower depths of the sample. The refractive index distribution generating apparatus according to claim 6, characterized in that the first setting process sets the refractive index indicated by the first refractive index information at a position corresponding to the first image region of the processed image in which the brightness at the deep position has been improved.

8. The refractive index distribution generating apparatus according to claim 3, 5, or 7, characterized in that the first correction process further improves the brightness of the image region closer to the other end.

9. The input process inputs third refractive index information, which indicates the refractive index of the third structure, from the memory. The setting process includes a third setting process that sets the refractive index based on the third refractive index information at a position corresponding to the third image region of the image to be processed, based on the signal intensity and color information of the unit pixel of the image to be processed. The aforementioned third image region is an image region corresponding to the aforementioned third structure, The refractive index distribution generating apparatus according to claim 1, characterized in that the second setting process sets the refractive index based on the second refractive index information at a position corresponding to an image region different from the first image region of the image to be processed, and at a position corresponding to an image region different from the third image region of the image to be processed.

10. The input process involves inputting fourth refractive index information, which indicates the refractive index of the medium surrounding the sample, from the memory. The refractive index distribution generation process includes a determination process to identify the boundary between the sample and the medium, The setting process includes a fourth setting process that sets the refractive index based on the fourth refractive index information to a position corresponding to the fourth image region of the image to be processed. The refractive index distribution generating apparatus according to claim 1, characterized in that the fourth image region is an image region corresponding to the medium.

11. The refractive index distribution generating apparatus according to claim 1, characterized in that the first refractive index information and the second refractive index information are refractive index information relating to the compositional structure of a cell.

12. The first refractive index information is the refractive index information of the cell nucleus, The refractive index distribution generating device according to claim 11, characterized in that the second refractive index information is refractive index information of a cell membrane.

13. The refractive index distribution generating apparatus according to claim 9, characterized in that the third refractive index information is refractive index information of cell adhesion molecules.

14. The refractive index distribution generating apparatus according to claim 10, characterized in that the fourth refractive index information is refractive index information of a cell culture medium.

15. The processor executes an image generation process to generate a processed image corresponding to the image to be processed. The aforementioned image generation process is: The aforementioned image to be processed is divided into multiple small image regions, A point image intensity distribution calculation process that uses the refractive index distribution of the image to be processed to calculate the point image intensity distribution for each of the small image regions, A small image generation process that generates a small image for each small image region using the point image intensity distribution for each small image region, The refractive index distribution generating apparatus according to claim 1, characterized by including a synthesis process that synthesizes small images for each of the small image regions to generate the processed image.

16. The point image intensity distribution calculation process calculates the point image intensity distribution for each small image region using a refractive index distribution set at a position corresponding to a fifth image region located within the range in which the wavefront propagates on the image to be processed, starting from the small image region. The fifth image region includes, for each of the small image regions, an image region outside the range of the sixth image region obtained by extending the small image region in a predetermined direction. The refractive index distribution generating apparatus according to claim 15, characterized in that the predetermined direction is the direction from the sample toward the observation optical system among the optical axis directions of the observation optical system modeled in the calculation process.

17. The refractive index distribution generating apparatus according to claim 15, characterized in that the processor performs the refractive index distribution generation process and the image generation process using the processed image as the image to be processed.

18. The aforementioned input process takes the second image to be processed, which is a photograph of the second sample, as input from the memory. The processor performs a refractive index determination process to determine the refractive index of the first structure. The refractive index determination process described above is: A refractive index distribution calculation process that calculates the refractive index distribution of the second sample from multiple images to be processed, In the second image to be processed, a second identification process is performed to identify a seventh image region corresponding to the first structure, The process includes a third identification process for identifying the refractive index corresponding to the seventh image region among the refractive indices that constitute the refractive index distribution of the second sample, The refractive index distribution generating apparatus according to claim 1, characterized in that the input processing inputs the refractive index of the first structure determined in the refractive index determination processing from the memory.

19. The maximum depth of the second sample is less than 50 μm. The refractive index distribution generating apparatus according to claim 18, characterized in that the minimum depth of the sample is 50 μm or more.

20. An observation optical system that forms an optical image of a specimen, An image sensor for capturing the aforementioned optical image, A refractive index distribution generation system characterized by comprising the refractive index distribution generation device described in claim 1.

21. A processor composed of hardware, memory composed of hardware, and equipped, The aforementioned processor, The refractive index distribution generation process is executed to generate the refractive index distribution corresponding to the image to be processed. The aforementioned refractive index distribution generation process is performed by Input processing involves inputting the image to be processed, first refractive index information indicating the refractive index of the first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure, from the memory. This includes a setting process for setting each refractive index that makes up the refractive index distribution, The aforementioned configuration process is: A first setting process in which the refractive index based on the first refractive index information is set at a position corresponding to the first image region of the image to be processed, based on the signal intensity of the unit pixel of the image to be processed, The process includes a second setting process which sets the refractive index based on the second refractive index information to a position corresponding to an image region different from the first image region of the image to be processed, The first image region is an image region corresponding to the first structure, The unit pixel is composed of one or more pixels. The processor uses the image of the sample as the image to be processed and executes the refractive index distribution generation process. The aforementioned processor performs machine learning processing to train the AI ​​model. The aforementioned machine learning process involves training the AI ​​model on multiple datasets. The dataset includes the image to be processed and the training data corresponding to the image to be processed. The refractive index distribution generation system is characterized in that the training data is the refractive index distribution generated by the refractive index distribution generation process.

22. A method for generating a refractive index distribution corresponding to an image to be processed, The image to be processed, first refractive index information indicating the refractive index of the first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure are input. The refractive index based on the first refractive index information is set at a position corresponding to the first image region of the image to be processed, based on the signal intensity of the unit pixel of the image to be processed. The refractive index based on the second refractive index information is set to a position corresponding to an image region different from the first image region of the image to be processed, The first image region is an image region corresponding to the first structure, The unit pixel is composed of one or more pixels. The refractive index distribution generation method is characterized in that the image to be processed is an image of a specimen.

23. A computer-readable recording medium containing a program for generating sample images, An input process that inputs the image to be processed, first refractive index information indicating the refractive index of the first structure, and second refractive index information indicating the refractive index of a second structure different from the first structure from memory. The process of setting each refractive index that makes up the refractive index distribution is executed. In the aforementioned setting process, A first setting process in which the refractive index based on the first refractive index information is set at a position corresponding to the first image region of the image to be processed, based on the signal intensity of the unit pixel of the image to be processed, A second setting process is performed, which sets the refractive index based on the second refractive index information to a position corresponding to an image region different from the first image region of the image to be processed. The first image region is an image region corresponding to the first structure, The unit pixel is composed of one or more pixels. A computer-readable recording medium characterized by performing a refractive index distribution generation process using an image of a specimen as the image to be processed.

24. The first refractive index information and the second refractive index information are refractive index information corresponding to the compositional structure of the cell, The processor determines the position of the first image region and the position of the second image region based on the signal intensity of the image to be processed. The first setting process sets the first refractive index information corresponding to the compositional structure of the cell at a position corresponding to the first image region, The second setting process sets the second refractive index information corresponding to the compositional structure of the cell at the position corresponding to the second image region. The refractive index distribution generating apparatus according to feature 1.

Images