Method, apparatus, and information processing program for determining the region of cells that have undergone necrosis.

Abstract

To provide a method for noninvasively determining a dead cell region in an object to be observed, particularly a region of a cell that has undergone necrosis.SOLUTION: There are disclosed a method, a device, and an information processing program for determining, by using refractive index distribution data pertaining to an object to be observed, a region of a cell that has undergone necrosis in the object to be observed.SELECTED DRAWING: None

Description

[Technical Field]

[0001] This disclosure relates to a method, apparatus, and information processing program for determining the region of a cell that has undergone necrosis. [Background technology]

[0002] Among the various forms of cell death, accidental cell necrosis is called necrosis. Necrosis occurs when cells are exposed to external stimuli, such as hypoxia, high temperature, toxins, nutrient deficiencies, and cell membrane damage. Necrosis is typically cell death that occurs in cells exposed to these stimuli in vivo. However, even when cells are cultured in three dimensions to create cell aggregates, areas with high cell density within the cell aggregate are prone to hypoxic and / or nutrient deficiencies, so cells, especially those near the center of the cell aggregate, often develop necrosis.

[0003] Generally, under conditions where individual cells can be observed, it is relatively easy to identify dead cells based on their appearance. For example, adherent cells cultured in two dimensions lose their adherence to the culture vessel and become suspended when they undergo cell death. Furthermore, suspended cells often undergo morphological changes when they undergo cell death. Therefore, dead cells can be identified by these changes in appearance.

[0004] On the other hand, in environments where cells are densely packed in three dimensions, such as cell aggregates, the suspension or morphological changes of dead cells are unlikely to occur, and detailed observation of any changes in appearance at the individual cell level is not easy. Determining cell death within cell aggregates is expected to have applications in areas such as analyzing the effects of drugs on cell death and evaluating cell aggregates themselves in the field of regenerative medicine. In particular, the development of non-invasive methods for determining the cell death region is desired.

[0005] Regarding the non-invasive determination of cell death regions, for example, Patent Document 1 discloses a cell analysis method that analyzes the state of cell death inside a spheroid by correcting multiple spheroid cross-sectional images taken at different distances from the light irradiation position, taking into account signal intensity attenuation due to the distance from the light irradiation position to the measurement position, based on the three-dimensional structure of the spheroid and the average brightness information for each pixel when light is irradiated. Patent Document 2 also discloses a spheroid evaluation method that evaluates the degree of spheroid decay from images taken of the spheroid based on information on the contour, optical density, circularity, and clarity of the spheroid region. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 6792616 [Patent Document 2] Patent No. 6122817 [Overview of the project] [Problems that the invention aims to solve]

[0007] Existing non-invasive methods for determining cell death areas struggle to identify dead cell regions with high resolution at the single-cell level, and even more so to determine the type of cell death caused by a cell. While fluorescence staining is typically used for such high-resolution cell death area identification, it requires labeling and cumbersome processing such as clearing, making deep observation difficult.

[0008] Therefore, an object of the present invention is to provide a method for non-invasively determining the region of dead cells, particularly the region of necrotic cells, in an object under observation. [Means for solving the problem]

[0009] The inventors discovered that by using refractive index distribution data of the object being observed, it is possible to determine the region of cells that have undergone necrosis within the object.

[0010] In other words, one aspect of this disclosure is a method for determining the region of necrotic cells in an object based on refractive index distribution data of the object.

[0011] Another aspect of this disclosure is a method for determining a region of necrotic cells in an object, comprising the steps of: acquiring refractive index distribution data of the object; and determining a region of necrotic cells in the object based on the refractive index distribution data of the object.

[0012] Another aspect of this disclosure is a method for evaluating the quality of a cell aggregate, comprising the steps of: obtaining refractive index distribution data of the cell aggregate; and determining, based on the refractive index distribution data of the cell aggregate, regions of cells within the cell aggregate that have undergone necrosis.

[0013] Another aspect of this disclosure is a device for determining necrotic cell regions, comprising a data acquisition unit for acquiring refractive index distribution data of an object to be observed, and a determination unit for determining regions of necrotic cells in the object based on the refractive index distribution data of the object to be observed.

[0014] Another aspect of this disclosure is an information processing program that causes a computer to perform a decision step of determining the region of necrotic cells in an object based on refractive index distribution data of the object.

[0015] Another aspect of this disclosure is an information processing program that causes a computer to perform a data acquisition step of acquiring refractive index distribution data of an object to be observed, and a determination step of determining the region of necrotic cells in the object based on the refractive index distribution data of the object to be observed.

[0016] More specifically, this disclosure relates to the following [1] to

[21] . [1] A method for determining the region of necrotic cells in an object based on refractive index distribution data of the object. [2] The method according to [1], wherein the region of cells that have undergone necrosis is determined based on the fact that the region of each cell included in the refractive index distribution data has the characteristics of a necrotic cell, and the characteristics of the necrotic cell include having a region corresponding to the nucleus. [3] The method according to [2], wherein the characteristics of the necrotic cells described above further include one or more characteristics selected from the group consisting of characteristics in which the statistical value of the refractive index of the entire cell is below a threshold and characteristics in which the statistical variability of the refractive index of the nucleus is above a threshold. [4] The method according to [2], wherein the characteristics of the necrotic cells described above further include the characteristic that the statistical value of the refractive index of the entire cell is below a threshold and the statistical variability of the refractive index of the nucleus is above a threshold. [5] The method according to any one of [2] to [4], wherein the region corresponding to the nucleus is a substantially circular or substantially spherical region and the statistical value of the refractive index is greater than the statistical value of the refractive index of the entire cell. [6] The method according to [1], wherein the region of cells that have undergone necrosis is determined by inputting the refractive index distribution data of the object into a learning model that has been trained using training data including necrotic region data of a reference object and refractive index distribution data of the reference object corresponding to the necrotic region data. [7] A method according to [1], wherein the determination of the region of a cell that has undergone necrosis is performed by inputting the refractive index distribution data of the object into a learning model that has been trained using training data including necrotic region data of a reference object and refractive index distribution data of the reference object corresponding to the necrotic region data, wherein the features used by the learning model include features that have a region corresponding to the nucleus. [8] The method according to [7], wherein the features used by the learning model further include one or more features selected from the group consisting of features where the statistical value of the refractive index of the entire cell is below a threshold and features where the statistical variability of the refractive index of the nucleus is above a threshold. [9] The method according to [7], wherein the features used by the learning model further include one or more features corresponding to the feature where the statistical value of the refractive index of the entire cell is below a threshold and the feature where the statistical variability of the refractive index of the nucleus is above a threshold.

[10] The method according to any one of [3] to [9], wherein the above statistical value is the mean or the median.

[11] The method according to any one of [1] to

[10] , wherein the above refractive index distribution data is refractive index tomographic data in a predetermined direction.

[12] A method for determining a region of cells that have undergone necrosis in an object, comprising the steps of: acquiring refractive index distribution data of the object to be observed; and determining a region of cells that have undergone necrosis by the method described in any one of [1] to

[11] .

[13] A method for evaluating the quality of a cell mass, wherein the object being observed is a cell mass, and the method comprises the steps of: obtaining refractive index distribution data of the cell mass; and determining the region of cells that have undergone necrosis within the cell mass by the method of any one of [1] to

[11] .

[14] The method according to

[13] , wherein the cell aggregate is obtained by culturing stem cells taken from an animal, including a human.

[15] The method according to

[13] , wherein the cell aggregate is obtained by culturing stem cells taken from a human.

[16] A device for determining necrotic cell regions, comprising a data acquisition unit for acquiring refractive index distribution data of an object to be observed, and a determination unit for determining a region of necrotic cells in an object to be observed by any one of the methods described in [1] to

[11] . An information processing program that causes a computer to perform a determination step of determining the region of necrotic cells in an object of observation by any one of the methods described in [1], [1], to

[11] .

[18] An information processing program that causes a computer to perform a data acquisition step of acquiring refractive index distribution data of an object to be observed, and a determination step of determining the region of cells in the object to be observed that has undergone necrosis by any one of the methods described in [1] to

[11] .

[19] The method, apparatus, or information processing program according to any one of [1] to

[12] or

[16] to

[18] , wherein the object of observation is a cell aggregate.

[20] The method, apparatus, or information processing program according to any one of [1] to

[12] or

[16] to

[18] , wherein the object being observed is a cell mass, and the region of cells that have undergone necrosis is located inside the cell mass.

[21] The method, apparatus, or information processing program according to any one of [1] to

[20] , wherein the necrosis described above is necroptosis. [Effects of the Invention]

[0017] According to this disclosure, a method, apparatus, and information processing program can be provided that can non-invasively determine, in particular, the region of necrotic cells among the dead cell regions in an object being observed. [Brief explanation of the drawing]

[0018] [Figure 1] Figure 1 (hereinafter also referred to as "Figure A01") shows the configuration of observation device 1A. [Figure 2] Figure 2 (hereinafter also referred to as "Figure A02") shows the configuration of observation device 1B. [Figure 3] Figure 3 (hereinafter also referred to as "Figure A03") shows the configuration of observation device 1C. [Figure 4] Figure 4 (hereinafter also referred to as "Figure A04") is a flowchart of refractive index distribution measurement method A. [Figure 5] Figure 5 (hereinafter also referred to as "Figure A05") (a), (b), and (c) show examples of scanning of the direction of light irradiation onto the object S during the interference intensity image acquisition step S1. [Figure 6] Figure 6 (hereinafter also referred to as "Figure A06") is a diagram illustrating the kernel function g. [Figure 7] Figure 7 (hereinafter also referred to as "Figure A07") (a) and (b) show examples of scanning the direction of light irradiation onto the object S during the interference intensity image acquisition step S1. [Figure 8] Figure 8 (hereinafter also referred to as "Figure A08") (a), (b), and (c) show examples of scanning the direction of light irradiation onto the object S during the interference intensity image acquisition step S1. [Figure 9] Figure 9 (hereinafter also referred to as "Figure A09") is a flowchart of the two-dimensional phase image generation step S4 in the refractive index distribution measurement method A1. [Figure 10] Figure 10 (hereinafter also referred to as "Figure A10") is a flowchart of the two-dimensional phase image generation step S4 in the refractive index distribution measurement method A2. [Figure 11] Figure 11 (hereinafter also referred to as "Figure A11") is a diagram illustrating the kernel function. [Figure 12] Figure 12 (hereinafter also referred to as "Figure A12") is a flowchart of the two-dimensional phase image generation step S4 in the refractive index distribution measurement method A3. [Figure 13] Figure 13 (hereinafter also referred to as "Figure B01") shows the configuration of observation device 1D. [Figure 14] Figure 14 (hereinafter also referred to as "Figure B02") shows the configuration of observation device 1E. [Figure 15] Figure 15 (hereinafter also referred to as "Figure B03") shows the configuration of observation device 1F. [Figure 16] Figure 16 (hereinafter also referred to as "Figure B04") is a flowchart of refractive index distribution measurement method B. [Figure 17] Figure 17 (hereinafter also referred to as "Figure B05") is a diagram illustrating the sequence of processes and images for the second complex amplitude image generation step S63 and the two-dimensional phase image generation step S65. [Figure 18] Figure 18 (hereinafter also referred to as "Figure B06") is a diagram illustrating the sequence and images of each process: the second complex amplitude image generation step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generation step S65. [Figure 19]Figure 19 (hereinafter also referred to as "Figure B07") is a diagram illustrating the sequence and images of each process: the second complex amplitude image generation step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generation step S65. [Figure 20] Figure 20 (hereinafter also referred to as "Figure B08") is a diagram illustrating the sequence and images of each process: the second complex amplitude image generation step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generation step S65. [Figure 21] Figure 21 (hereinafter also referred to as "Figure B09") is a diagram illustrating the sequence and images of each process in the 3D phase image generation step S66 and the refractive index distribution calculation step S67. [Figure 22] Figure 22 (hereinafter also referred to as "Figure B10") is a diagram illustrating the overview of phase conjugate operation, and shows the input and output light when the imaging unit captures an interference intensity image. [Figure 23] Figure 23 (hereinafter also referred to as "Figure B11") is a diagram illustrating the overview of phase conjugate operation, and shows the input and output light when the relationship between light irradiation and imaging is reversed. [Figure 24] Figure 24 (hereinafter also referred to as "Figure B12") is a diagram illustrating the image division, phase conjugation operation, and image merging in the phase conjugation operation step S64. [Figure 25] Figure 25 (hereinafter also referred to as "Figure C01") shows the configuration of observation device 1G. [Figure 26] Figure 26 (hereinafter also referred to as "Figure C02") shows the configuration of observation device 1H. [Figure 27] Figure 27 (hereinafter also referred to as "Figure C03") shows the configuration of observation device 1I. [Figure 28] Figure 28 (hereinafter also referred to as "Figure C04") is a flowchart of refractive index distribution measurement method C. [Figure 29] Figure 29 (hereinafter also referred to as "Figure C05") is a flowchart of refractive index distribution measurement method C. [Figure 30]Figure 30 (hereinafter also referred to as "Figure C06") is a diagram illustrating the relationship between the region containing the object to be observed and the blocks from the 1st to the Jth. [Figure 31] Figure 31 (hereinafter also referred to as "Figure C07") is a diagram illustrating the processing procedures in blocks 1 through J. [Figure 32] Figure 32 (hereinafter also referred to as "Figure C08") is a diagram illustrating the processing steps of BPM. [Figure 33] Figure 33 (hereinafter also referred to as "Figure C09") is a flowchart of the third complex amplitude image generation step S77. [Figure 34] Figure 34 (hereinafter also referred to as "Figure C10") shows the configuration of observation device 1J. [Figure 35] Figure 35 shows a typical flow chart illustrating a method for determining the region of necrotic cells, according to one aspect of this disclosure, using a threshold method based on the characteristic that the average or median refractive index of the entire cell is below a threshold. [Figure 36] Figure 36 shows a typical flow chart illustrating a method for determining the region of a cell that has undergone necrosis, according to one aspect of this disclosure, using a threshold method based on the characteristic that the statistical variation in the nuclear refractive index is greater than or equal to a threshold. [Figure 37] Figure 37 shows a typical learning flow of a model in the learning step when determining the region of a cell that has undergone necrosis, according to one aspect of this disclosure, by a machine learning method, and the necrosis region data is created from fluorescence data of a reference object. [Figure 38] Figure 38 shows a typical inference flow in the inference step when determining the region of necrotic cells according to one aspect of this disclosure, using a machine learning method, and when the necrotic region data is created from fluorescence data of a reference object. [Figure 39]Figure 39 shows an example of a method for evaluating cancer cell aggregates related to one aspect of this disclosure, in which cancer cell aggregates are evaluated as being of higher quality if the area of ​​necrotic cells within the cancer cell aggregate is large and located in the center. [Figure 40] Figure 40 shows an example of the manufacturing process for a cell aggregate obtained by culturing stem cells, relating to one aspect of this disclosure. [Figure 41] Figure 41 shows an example of a method for evaluating cell aggregates obtained by culturing stem cells, relating to one aspect of this disclosure, in which cell aggregates with smaller areas of necrotic cells within the cell aggregate are evaluated as being of higher quality. [Figure 42] Figure 42 shows representative refractive index tomographic data extracted from the refractive index distribution data of HepG2 single cells obtained in Example 1 under conditions with or without freeze-thaw. [Figure 43] Figure 43 shows representative refractive index tomographic data extracted from the refractive index distribution data of A549 cell aggregates that underwent necrosis by being left at room temperature in Example 2. [Figure 44] Figure 44 shows representative refractive index tomographic data extracted from the refractive index distribution data of HepG2 cell aggregates that underwent necrosis due to high-concentration ethanol treatment in Example 3. [Figure 45] Figure 45 shows representative refractive index tomographic data extracted from the refractive index distribution data of A549 cell aggregates that underwent necrosis due to high-concentration ethanol treatment in Example 4. [Figure 46] Figure 46 shows representative refractive index tomographic data extracted from the refractive index distribution data of HepG2 cell aggregates containing necrotic cells in Example 5. [Figure 47] Figure 47 shows the flowchart for determining the region of necrotic cells in Example 5 using a threshold method that utilizes the characteristics of necrotic cells, namely the presence of a region corresponding to the nucleus and the average refractive index of the entire cell being below a threshold. [Figure 48] Figure 48 shows the flowchart for determining the region of necrotic cells in Example 5 using a threshold method that utilizes the characteristics of necrotic cells, namely the presence of a region corresponding to the nucleus and the statistical variation of the nuclear refractive index being above a threshold. [Modes for carrying out the invention]

[0019] The embodiments will be described in detail below with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference numerals, and redundant explanations will be omitted. Actual forms are not limited to these examples.

[0020] Necrosis is cell death that occurs when cells are exposed to external factors such as hypoxia, high temperature, toxins, nutrient deficiencies, and cell membrane damage. Necrotic cells are characterized by morphological features such as cell swelling and cell membrane collapse, but unlike apoptosis, there is no significant nuclear fragmentation, and the nuclear features are preserved.

[0021] Cells undergoing necrosis first experience cell membrane collapse, followed by leakage of contents, and then cell death through a time-dependent process including nucleus collapse. Therefore, it is believed that cells in the process of necrosis undergo similar morphological changes to cells that have already undergone necrosis. Accordingly, in one embodiment, the method, apparatus, or information processing program according to one embodiment of this disclosure may determine the region of a cell in the process of necrosis instead of the region of a cell that has already undergone necrosis.

[0022] Necrosis is generally classified as (I) accidental cell death (ACD) and (II) programmed cell death (PCD) or regulated cell death (RCD), when cell death is broadly divided into two groups. On the other hand, a method, apparatus, or information processing program according to one embodiment of this disclosure determines the region of a cell that has undergone necrosis based on the characteristics of a necrotic cell. In necroptosis, a type of programmed cell death, it is known that morphological changes similar to necrosis are observed, namely cell swelling, cell membrane collapse, and maintenance of nuclear characteristics. Therefore, a method, apparatus, or information processing program according to one embodiment of this disclosure can also determine the region of a cell that has undergone necroptosis. That is, instead of determining the region of a cell that has undergone necrosis, a method, apparatus, or information processing program according to one embodiment of this disclosure may determine the region of a cell that has undergone necrosis-like cell death, for example, a region of a cell that has undergone necroptosis.

[0023] Necroptosis is a programmed form of cell death that exhibits morphological similarities to necrosis, and is a process of cellular self-destruction that is activated, for example, when apoptosis is inhibited. Cells undergoing necroptosis are characterized by morphological features such as cell swelling and cell membrane collapse, similar to necrotic cells, but do not exhibit the pronounced nuclear fragmentation seen in apoptosis.

[0024] An object of observation according to one embodiment of this disclosure is an object composed mainly of cells, but may also contain components that may be present in tissue samples, such as extracellular matrix and triglycerides. From the viewpoint of achieving both high reproducibility and ease of sample preparation, the object of observation according to one embodiment of this disclosure may be a cell culture, or it may be a cell aggregate from the viewpoint of simultaneously providing an environment close to a physiological environment. When the object of observation is a cell aggregate, it is possible to determine the region of necrotic cells within the cell aggregate, which cannot be evaluated using external appearance as an indicator. That is, the region of necrotic cells in the object of observation according to one embodiment is the region of necrotic cells within the cell aggregate. "Within the cell aggregate" means cells that form the cell aggregate but are not exposed on the surface of the cell aggregate. When the object of observation is a cell culture, the culture may consist of one type of cell or contain two or more types of cells. When the object of observation is a cell aggregate, its maximum diameter may be 100 to 200 μm.

[0025] Refractive index distribution data refers to data showing the three-dimensional distribution of refractive index for each voxel in the space containing the object being observed, or data showing the two-dimensional distribution of refractive index for each pixel in a tomographic plane in a predetermined direction in the space containing the object being observed. Furthermore, refractive index tomographic data refers to the two-dimensional distribution of refractive index for each pixel in a tomographic plane in a predetermined direction in the space containing the object being observed, as a type of refractive index distribution data.

[0026] The method for acquiring refractive index distribution data according to one embodiment of this disclosure (hereinafter also referred to as the "refractive index distribution measurement method") is not particularly limited, but one method will be described in detail below. Optical diffraction tomography (ODT) is known as a method for measuring the refractive index distribution of an object without staining or invasiveness. ODT is an evolution of quantitative phase imaging (QPI) into a three-dimensional imaging technique, and can realize three-dimensional refractive index tomography of an object.

[0027] The refractive index distribution measurement methods A to C, described below, are used. Refractive index distribution measurement method A has three forms: refractive index distribution measurement methods A1 to A3. Refractive index distribution measurement methods A1 to A3 are collectively referred to as refractive index distribution measurement method A. These refractive index distribution measurement methods A to C can realize three-dimensional refractive index tomography with reduced effects of multiple scattered light, even when the object being observed is a multiple scattering material.

[0028] In addition, optical coherence tomography (OCT) is also known as another staining and non-invasive imaging technique. However, while the resolution of OCT is approximately 10 μm, the resolution of ODT and refractive index distribution measurement methods A-C is approximately 1 μm. Furthermore, OCT does not determine the refractive index distribution, making the biological interpretation of the signals obtained through imaging difficult. In these respects, ODT and refractive index distribution measurement methods A-C are superior to OCT.

[0029] First, we will explain the refractive index distribution measurement method A (A1-A3). Figures A01-A03 show the configurations of observation devices 1A-1C that can be used when measuring the refractive index distribution using the refractive index distribution measurement method A.

[0030] Figure A01 shows the configuration of observation device 1A. This observation device 1A includes a light source 11, lens 12, lens 21, mirror 22, lens 23, condenser lens 24, objective lens 25, beam splitter 41, lens 42, imaging unit 43, and analysis unit 50, etc.

[0031] Light source 11 outputs spatially and temporally coherent light, such as a laser light source. Lens 12 is optically connected to light source 11 and focuses the light output from light source 11 onto the optical incident end 13 of optical fiber 14, causing that light to enter the optical incident end 13. Optical fiber 14 guides the light that entered the optical incident end 13 via lens 12 to fiber coupler 15. Fiber coupler 15 couples light between optical fiber 14 and optical fibers 16 and 17, splitting the light guided by optical fiber 14 into two branches. One branch is guided by optical fiber 16, and the other branch is guided by optical fiber 17. The light guided by optical fiber 16 is emitted as divergent light from the optical exit end 18. The light guided by optical fiber 17 is emitted as divergent light from the optical exit end 19.

[0032] Lens 21 is optically connected to the light output end 18 and collimates the light output from the light output end 18 as divergent light. Mirror 22 is optically connected to lens 21 and reflects the light arriving from lens 21 to lens 23. The orientation of the reflective surface of mirror 22 is variable. Lens 23 is optically connected to mirror 22. Condenser lens 24 is optically connected to lens 23. Lens 23 and condenser lens 24 constitute, for example, a 4f optical system. Lens 23 and condenser lens 24 illuminate the object to be observed S from a light irradiation direction corresponding to the orientation of the reflective surface of mirror 22. Objective lens 25 is optically connected to condenser lens 24. The object to be observed S is placed between objective lens 25 and condenser lens 24. Objective lens 25 receives light (object light) output from condenser lens 24 and passed through the object to be observed S, and outputs that light to beam splitter 41.

[0033] The beam splitter 41 is optically connected to the objective lens 25 and also to the light output end 19. The beam splitter 41 combines the light output from the objective lens 25 (object light) and the light output from the light output end 19 (reference light) and outputs both lights to the lens 42. The lens 42 is optically connected to the beam splitter 41 and collimates the object light and reference light arriving from the beam splitter 41 and outputs them to the imaging unit 43. The imaging unit 43 is optically connected to the lens 42 and captures an interference fringe image (interference intensity image) due to the interference between the object light and reference light arriving from the lens 42. The direction of incidence of the reference light is inclined with respect to the direction of incidence of the object light onto the imaging surface of the imaging unit 43. The position where the object light and the reference light are combined by the beam splitter 41 may be after the imaging lens, but considering the effects of aberrations, it is preferable that it be between the objective lens 25 and lens 42, as shown in the figure.

[0034] The analysis unit 50 is electrically connected to the imaging unit 43 and receives interference intensity images captured by the imaging unit 43 as input. The analysis unit 50 calculates the three-dimensional refractive index distribution of the object being observed S by processing the input interference intensity images. The analysis unit 50 may be a computer. The analysis unit 50 includes an interference intensity image acquisition unit 51, a first complex amplitude image generation unit 52, a second complex amplitude image generation unit 53, a two-dimensional phase image generation unit 54, a three-dimensional phase image generation unit 55, a refractive index distribution calculation unit 56, a display unit 57, and a storage unit 58.

[0035] The interference intensity image acquisition unit 51 illuminates the object to be observed S along each of the multiple light irradiation directions by changing the orientation of the reflective surface of the mirror 22. The interference intensity image acquisition unit 51 also acquires interference intensity images at a reference position from the imaging unit 43 for each of the multiple light irradiation directions. The interference intensity image acquisition unit 51 includes a CPU and has an output port that outputs a control signal for changing the orientation of the reflective surface of the mirror 22, and an input port that receives interference intensity images from the imaging unit 43. It is not necessary to move the objective lens 25 in the optical axis direction. The reference position is an image plane position that is conjugate to the imaging plane of the imaging unit 43.

[0036] The first complex amplitude image generation unit 52, the second complex amplitude image generation unit 53, the two-dimensional phase image generation unit 54, the three-dimensional phase image generation unit 55, and the refractive index distribution calculation unit 56 perform processing based on interference intensity images and include processing units such as a CPU, GPU, DSP, or FPGA. The display unit 57 displays images to be processed, images in the process of processing, and images after processing, and includes, for example, a liquid crystal display. The storage unit 58 stores data for various images and includes a hard disk drive, flash memory, RAM, and ROM. The first complex amplitude image generation unit 52, the second complex amplitude image generation unit 53, the two-dimensional phase image generation unit 54, the three-dimensional phase image generation unit 55, the refractive index distribution calculation unit 56, and the storage unit 58 may be configured using cloud computing.

[0037] The storage unit 58 also stores programs for the interference intensity image acquisition unit 51, the first complex amplitude image generation unit 52, the second complex amplitude image generation unit 53, the two-dimensional phase image generation unit 54, the three-dimensional phase image generation unit 55, and the refractive index distribution calculation unit 56 to perform their respective processes. These programs may be stored in the storage unit 58 at the time of manufacture or shipment of the observation device 1A, or they may be acquired via a communication line after shipment and stored in the storage unit 58, or they may be recorded on a computer-readable recording medium 2 and stored in the storage unit 58. The recording medium 2 can be any flexible disk, CD-ROM, DVD-ROM, BD-ROM, USB memory, etc.

[0038] Details of the processing of the interference intensity image acquisition unit 51, the first complex amplitude image generation unit 52, the second complex amplitude image generation unit 53, the two-dimensional phase image generation unit 54, the three-dimensional phase image generation unit 55, and the refractive index distribution calculation unit 56 will be described later.

[0039] Figure A02 shows the configuration of observation device 1B. In addition to the configuration of observation device 1A shown in Figure A01, observation device 1B in Figure A02 includes a lens 31, a mirror 32, a drive unit 33, and a lens 34.

[0040] Lens 31 is optically connected to the light output end 19 and collimates the light (reference light) output from the light output end 19 as divergent light. Mirror 32 is optically connected to lens 31 and reflects the light arriving from lens 31 to lens 34. Lens 34 is optically connected to mirror 32 and outputs the light arriving from mirror 32 to beam splitter 41. The light output from lens 34 is focused before reaching beam splitter 41 and then input to beam splitter 41 as divergent light. Beam splitter 41 combines the light arriving from objective lens 25 (object light) and the light arriving from lens 34 (reference light), and outputs both lights coaxially to lens 42. Imaging unit 43 captures interference fringes (interference intensity image) due to the interference between the object light and reference light arriving from lens 42. The direction of incidence of the reference light is parallel to the direction of incidence of the object light onto the imaging surface of the imaging unit 43.

[0041] The drive unit 33 moves the mirror 32 in a direction perpendicular to the reflective surface of the mirror 32. The drive unit 33 is, for example, a piezo actuator. This movement of the mirror 32 changes the difference (phase difference) in the optical path lengths of the object light and the reference light, from the optical branching at the fiber coupler 15 to the combined beam at the beam splitter 41. Different optical path length differences result in different interference intensity images captured by the imaging unit 43.

[0042] The observation device is not limited to the configuration examples shown in Figures A01 and A02, and various modifications are possible. In the configurations of observation device 1A and observation device 1B, object light transmitted through the object S is observed, but as in the configuration of observation device 1C described below, object light reflected by the object S may also be observed.

[0043] Figure A03 shows the configuration of observation device 1C. Observation device 1C includes a light source 11, lens 12, lens 21, mirror 22, lens 23, objective lens 25, beam splitter 41, lens 42, imaging unit 43, and analysis unit 50. The following will mainly describe the differences from observation device 1A (Figure A01).

[0044] Lens 21 is optically connected to the optical output end 18 of the optical fiber 16 and collimates the light output from the optical output end 18 as divergent light. Mirror 22 is optically connected to lens 21 and reflects the light arriving from lens 21 to lens 23. The orientation of the reflective surface of mirror 22 is variable. Lens 23 is optically connected to mirror 22. Objective lens 25 is optically connected to lens 23. A beam splitter 41 is positioned between lens 23 and objective lens 25. Lens 23 and objective lens 25 constitute, for example, a 4f optical system. Lens 23 and objective lens 25 irradiate the object to be observed S with light from an illumination direction corresponding to the orientation of the reflective surface of mirror 22. Objective lens 25 receives the light reflected by the object to be observed S (object light) and outputs that light to beam splitter 41.

[0045] The beam splitter 41 is optically connected to the objective lens 25 and also to the optical output end 19 of the optical fiber 17. The beam splitter 41 combines the light output from the objective lens 25 (object light) and the light output from the optical output end 19 (reference light) and outputs both lights to the lens 42. The lens 42 is optically connected to the beam splitter 41 and collimates the object light and reference light arriving from the beam splitter 41 and outputs them to the imaging unit 43. The imaging unit 43 is optically connected to the lens 42 and captures an interference fringe image (interference intensity image) due to the interference between the object light and reference light arriving from the lens 42. The direction of incidence of the reference light is inclined with respect to the direction of incidence of the object light onto the imaging surface of the imaging unit 43. The position where the object light and the reference light are combined by the beam splitter 41 may be after the imaging lens, but considering the effects of aberrations, it is preferable that it be between the objective lens 25 and lens 42, as shown in the figure.

[0046] In the configuration of observation device 1C (Figure A03), a mechanism for changing the optical path length of the reference light (lens 31, mirror 32, drive unit 33, and lens 34 in Figure A02) may be provided, similar to observation device 1B (Figure A02), to change the difference (phase difference) in the optical path lengths of the object light and the reference light from the optical branching at the fiber coupler 15 to the beam splitter 41. In this case, the incident direction of the reference light may be parallel to the incident direction of the object light onto the imaging surface of the imaging unit 43.

[0047] Figure A04 is a flowchart of refractive index distribution measurement method A. This refractive index distribution measurement method A is possible when using any of the observation devices 1A to 1C. This refractive index distribution measurement method A comprises interference intensity image acquisition step S1, first complex amplitude image generation step S2, second complex amplitude image generation step S3, two-dimensional phase image generation step S4, three-dimensional phase image generation step S5, and refractive index distribution calculation step S6.

[0048] The interference intensity image acquisition step S1 is performed by the interference intensity image acquisition unit 51. The first complex amplitude image generation step S2 is performed by the first complex amplitude image generation unit 52. The second complex amplitude image generation step S3 is performed by the second complex amplitude image generation unit 53. The two-dimensional phase image generation step S4 is performed by the two-dimensional phase image generation unit 54. The three-dimensional phase image generation step S5 is performed by the three-dimensional phase image generation unit 55. The refractive index distribution calculation step S6 is performed by the refractive index distribution calculation unit 56.

[0049] In the interference intensity image acquisition step S1, the interference intensity image acquisition unit 51 changes the orientation of the reflective surface of the mirror 22 to irradiate the object to be observed S with light along each of the multiple light irradiation directions. The interference intensity image acquisition unit 51 then acquires interference intensity images at a reference position for each of the multiple light irradiation directions from the imaging unit 43.

[0050] In each of FIGS. A01 to A03, an xyz orthogonal coordinate system is shown for convenience of explanation. The z-axis is parallel to the optical axis of the objective lens 25. The reference position is an image plane position that has a conjugate relationship with the imaging plane of the imaging unit 43. This position is set as z = 0. The light irradiation direction to the observation object S is the wave number vector (k x , k y , k z ) of which k x and k y can be represented.

[0051] FIGS. A05(a) to (c) are diagrams showing examples of scanning the light irradiation direction to the observation object S in the interference intensity image acquisition step S1. In this figure, the horizontal axis is k x and the vertical axis is k y , and the position of each circle mark in the k x k y plane represents the light irradiation direction. The scanning of the light irradiation direction may be arranged in a rectangular grid pattern in the k x k y plane as shown in FIG. A05(a), or may be arranged on the circumferences of a plurality of concentric circles in the k x k y plane as shown in FIG. A05(b), or may be arranged in a spiral pattern in the k x k y plane as shown in FIG. A05(c). In any case, the light irradiation direction can be scanned as long as the numerical aperture (NA) of the condenser lens 24 permits. Either raster scan or random scan may be used. In the case of raster scan, there may or may not be a return scan.

[0052] In the first complex amplitude image generation step S2, the first complex amplitude image generation unit 52 generates a complex amplitude image of a reference position for each of the multiple light irradiation directions based on the interference intensity image of the reference position acquired by the interference intensity image acquisition unit 51. In the case of observation device 1A (Figure A01) and observation device 1C (Figure A03), the first complex amplitude image generation unit 52 can generate a complex amplitude image based on a single interference intensity image using the Fourier fringe analysis method. In the case of observation device 1B (Figure A02), the first complex amplitude image generation unit 52 can generate a complex amplitude image based on three or more interference intensity images in which the optical path length difference (phase difference) between the object light and the reference light is different from each other, using the phase shift method.

[0053] In the second complex amplitude image generation step S3, the second complex amplitude image generation unit 53 generates complex amplitude images for each of the multiple z-direction positions based on the complex amplitude image of the reference position (z=0) generated by the first complex amplitude image generation unit 52 for each of the multiple light irradiation directions. The two-dimensional Fourier transform of the complex amplitude image u(x,y,0) of the reference position is U(k x ,k y If we set (0), then the complex amplitude image u(x,y,d) at the position z=d, and the 2D Fourier transform U(k) of this complex amplitude image u(x,y,d) are obtained. x ,k y d) is expressed by the following formula, where i is the imaginary unit and k0 is the wavenumber of light in the observed object.

[0054]

number

[0055]

number

[0056] In the two-dimensional phase image generation step S4, the two-dimensional phase image generation unit 54 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images for each of the multiple light irradiation directions generated by the second complex amplitude image generation unit 53. The two-dimensional phase image generated here corresponds to a phase image centered on the focused z-direction position. Details of the two-dimensional phase image generation step S4 will be described later.

[0057] Alternatively, in the second complex amplitude image generation step S3, after generating complex amplitude images for each of the multiple positions for each of the multiple light irradiation directions, the processing from the 2D phase image generation step S4 onward may be performed. Alternatively, in the second complex amplitude image generation step S3, a complex amplitude image for a certain z-direction position may be generated for each of the multiple light irradiation directions, and a 2D phase image for that position may be generated in the 2D phase image generation step S4. This unit processing may be repeated while scanning the z-direction positions. In the latter case, the amount of image data that the storage unit 58 needs to store can be reduced.

[0058] In the 3D phase image generation step S5, the 3D phase image generation unit 55 generates a 3D phase image based on the 2D phase image generation unit 54, which generates 2D phase images for each of the multiple positions. The 3D phase image generated here is an image in which the positions x and y in the 2D phase image and the position z in the 2D phase image are variables.

[0059] In step S6, the refractive index distribution calculation unit 56 determines the three-dimensional refractive index distribution of the object being observed by deconvolution based on the three-dimensional phase image generated by the three-dimensional phase image generation unit 55. Let n(x,y,z) be the refractive index distribution of the object being observed, f(x,y,z) be the electrical susceptibility distribution, and let n be the refractive index of the background medium. mTherefore, the following relationship (3) exists between the two. The 3D phase image Φ(x,y,z) generated by the 3D phase image generation unit 55 is expressed as a convolution of the kernel function g(x,y,z) and the electrical susceptibility distribution f(x,y,z), as shown in equation (4) below. Thus, the 3D refractive index distribution n(x,y,z) of the object to be observed can be determined by deconvolution based on the 3D phase image Φ(x,y,z).

[0060]

number

[0061]

number

[0062] Furthermore, the processes of the first complex amplitude image generation step S2, the second complex amplitude image generation step S3, the two-dimensional phase image generation step S4, the three-dimensional phase image generation step S5, and the refractive index distribution calculation step S6 may be performed each time an interference intensity image for a predetermined number of light irradiation directions is acquired in the interference intensity image acquisition step S1 (Figure A07), or each time an interference intensity image for one light irradiation direction is acquired in the interference intensity image acquisition step S1 (Figure A08).

[0063] Figures A07 and A08 show examples of scanning the direction of light irradiation onto the object S during the interference intensity image acquisition step S1. In these figures, the horizontal axis is k x Let the vertical axis be k y k x k yOn the plane, the position of each circle represents the direction of light irradiation. In the example of scanning the direction of light irradiation shown in these figures, the direction of light irradiation is sequentially changed until the direction of light irradiation when acquiring the (N+n)th interference intensity image matches the direction of light irradiation when acquiring the nth interference intensity image. n is a positive integer, and N is an integer greater than or equal to 2.

[0064] In the example shown in Figure A07, when the 1st to Nth interference intensity images are acquired in the interference intensity image acquisition step S1, the processes in steps S2 to S6 are performed based on these 1st to Nth interference intensity images (Figure A07(a)). Next, when the (N+1)th to 2Nth interference intensity images are acquired in the interference intensity image acquisition step S1, the processes in steps S2 to S6 are performed based on these (N+1)th to 2Nth interference intensity images (Figure A07(b)). Next, when the (2N+1)th to 3Nth interference intensity images are acquired in the interference intensity image acquisition step S1, the processes in steps S2 to S6 are performed based on these (2N+1)th to 3Nth interference intensity images. The process continues similarly thereafter.

[0065] In the example shown in Figure A08, when the first to Nth interference intensity images are acquired in the interference intensity image acquisition step S1, the processes in steps S2 to S6 are performed based on these first to Nth interference intensity images (Figure A08(a)). Next, when the (N+1)th interference intensity image is acquired in the interference intensity image acquisition step S1, the processes in steps S2 to S6 are performed based on the most recent N interference intensity images (the second to the (N+1)th interference intensity images) that include this (N+1)th interference intensity image (Figure A08(b)). Next, when the (N+2)th interference intensity image is acquired in the interference intensity image acquisition step S1, the processes in steps S2 to S6 are performed based on the most recent N interference intensity images (the third to the (N+2)th interference intensity images) that include this (N+2)th interference intensity image (Figure A08(c)). Similarly, once the (N+n)th interference intensity image is acquired in the interference intensity image acquisition step S1, each of the processes in steps S2 to S6 is performed based on the most recent N interference intensity images (the (1+n) to (N+n)th interference intensity images) including this (N+n)th interference intensity image.

[0066] In comparison with the example shown in Figure A07, in the example shown in Figure A08, each time an interference intensity image for one light irradiation direction is acquired in the interference intensity image acquisition step S1, the processes in steps S2 to S6 are performed based on the most recent multiple interference intensity images including that interference intensity image. Therefore, the number of images obtained per unit time by the processes in steps S2 to S6 is large.

[0067] Next, the details of the two-dimensional phase image generation step S4 in refractive index distribution measurement method A will be described. In the two-dimensional phase image generation step S4, the two-dimensional phase image generation unit 54 generates a two-dimensional phase image for each of the multiple positions based on the complex amplitude images for each of the multiple light irradiation directions generated by the second complex amplitude image generation unit 53. The two-dimensional phase image generation step S4 differs depending on which of the refractive index distribution measurement methods A1 to A3 is used.

[0068] Figure A09 is a flowchart of step S4 of the 2D phase image generation in refractive index distribution measurement method A1. In step S4 of the 2D phase image generation in refractive index distribution measurement method A1, for each of the multiple positions, in step S11, the phase of the complex amplitude image for each of the multiple light irradiation directions is corrected based on the light irradiation direction, and then a complex amplitude sum image representing the sum of these corrected complex amplitude images is generated, and in step S12, a 2D phase image is generated based on this complex amplitude sum image.

[0069] The process in step S11 is carried out using the CASS (Collective Accumulation of Single Scattering) technique (Sungsam Kang, et al, “Imaging deep within a scattering medium using collective accumulation of single-scattered waves,” NATURE PHOTONICS, Vol.9, pp.253-258 (2015)). When light is irradiated onto an object along a certain direction of light irradiation and passes through the object, the spatial frequency distribution of single-scattered light that interacts with the object only once shifts according to the direction of light irradiation, whereas the spatial frequency distribution of multiple-scattered light that interacts with the object multiple times changes randomly depending on the direction of light irradiation. The CASS technique utilizes this difference in the dependence of the spatial frequency distributions of single-scattered light and multiple-scattered light on the direction of light irradiation.

[0070] In other words, in step S11, the phase of the complex amplitude images for each of the multiple light irradiation directions is corrected based on the light irradiation direction (i.e., the spatial frequency distribution of the complex amplitude images in the spatial frequency domain is translated according to the light irradiation direction), thereby making the spatial frequency distribution of the single scattered light component in the complex amplitude image independent of the light irradiation direction in terms of shape and arrangement, while making the spatial frequency distribution of the multiple scattered light components in the complex amplitude image random in terms of shape and arrangement. Then, in step S11, a complex amplitude sum image representing the sum of these corrected complex amplitude images is generated (i.e., a synthetic aperture process is performed), thereby coherently summing the single scattered light components in the complex amplitude images, while canceling out the multiple scattered light components in the complex amplitude images.

[0071] Therefore, the complex amplitude sum image generated in step S11 has reduced effects from multiple scattered light. Finally, the 3D refractive index distribution obtained in step S6 also has reduced effects from multiple scattered light, suppressed speckle, and an improved single-scattering to multi-scattering ratio (SMR).

[0072] Figure A10 is a flowchart of step S4 of the two-dimensional phase image generation in refractive index distribution measurement method A2. Step S4 of the two-dimensional phase image generation in refractive index distribution measurement method A2 generates complex differential interference images for each of the multiple light irradiation directions based on the complex amplitude images for each of the multiple light irradiation directions in step S21, generates a phase differential image based on the sum of the complex differential interference images for each of the multiple light irradiation directions in step S22, and generates a two-dimensional phase image based on the phase differential image in step S23.

[0073] If u(x,y,d) is the complex amplitude image at position z=d, the complex differential interference image q(x,y,d) generated in step S21 is expressed by equation (5) below. At least one of δx and δy is non-zero. If δx≠0 and δy=0, a complex differential interference image q with the x-direction as the shear direction is obtained. If δx=0 and δy≠0, a complex differential interference image q with the y-direction as the shear direction is obtained. If δx≠0 and δy≠0, a complex differential interference image q with a shear direction different from both the x-direction and the y-direction is obtained. Alternatively, the complex differential interference image q(x,y,d) may be obtained by transforming the complex amplitude image u(x,y,d) as shown in equation (6) below and then using equation (5).

[0074]

number

[0075]

number

[0076]

number

[0077] In step S22, the phase differential image generated based on the sum of the complex differential interference images for each of the multiple light irradiation directions has reduced effects from multiple scattered light. Finally, the 3D refractive index distribution obtained in step S6 also has reduced effects from multiple scattered light, and speckle is suppressed. In addition, when complex differential interference images are generated for each of the multiple different shear directions on the complex amplitude image in step S21, the appearance of line-like noise in the 2D phase image obtained in step S23 can be suppressed.

[0078] Here, we have described the case in which a two-dimensional phase image is generated by integrating or deconvolving the phase differential image in step S23. However, the phase differential image can also be treated as a two-dimensional phase image. In this case, without performing step S23, the three-dimensional refractive index distribution of the object can be determined from the phase differential image (two-dimensional phase image) generated in step S22 by using a kernel (Figure A11) that includes the kernel used in the deconvolution of step S23 in the deconvolution of step S67. The kernel shown in Figure A11 is obtained by convolution and integration of the kernel shown in Figure A06 and the kernel used in the deconvolution of step S23.

[0079] Figure A12 is a flowchart of step S4 of the 2D phase image generation in refractive index distribution measurement method A3. Step S4 of the 2D phase image generation in refractive index distribution measurement method A3 involves, for each of the multiple positions, dividing the complex amplitude images for each of the multiple light irradiation directions into multiple batches in step S31, correcting the phase of the complex amplitude images contained in each batch based on the light irradiation direction, and then generating a complex amplitude sum image representing the sum of these corrected complex amplitude images in step S32, generating a complex differential interference image for each of the multiple batches based on the complex amplitude sum image for each of the multiple batches, generating a phase differential image based on the sum of the complex differential interference images for each of the multiple batches in step S33, and generating a 2D phase image based on the phase differential image in step S34.

[0080] The process in step S31 of refractive index distribution measurement method A3 corresponds to dividing the complex amplitude images for each of the multiple light irradiation directions into multiple batches, and then performing the process in step S11 of refractive index distribution measurement method A1 for each of the multiple batches. The processes in steps S32 and S33 of refractive index distribution measurement method A3 correspond to performing the processes in steps S21 and S22 of refractive index distribution measurement method A2 for each of the multiple batches. The process in step S34 of refractive index distribution measurement method A3 corresponds to performing the process in step S23 of refractive index distribution measurement method A2.

[0081] In step S32, complex differential interference images may be generated for each of several different shear directions on the complex amplitude image. In this case, step S4 generates complex differential interference images for each of several different shear directions on the image based on the complex amplitude sum image of each of the several batches in step S32, generates a phase differential image for each of the several shear directions based on the sum of the complex differential interference images of each of the several batches in step S33, and generates a two-dimensional phase image based on the phase differential images for each of the several shear directions in step S34.

[0082] The suppression of speckle in refractive index distribution measurement method A3 is comparable to that of refractive index distribution measurement methods A1 and A2. The improvement in SMR in refractive index distribution measurement method A3 is intermediate between that of refractive index distribution measurement method A1 and A2.

[0083] Here, we have explained the case in which a two-dimensional phase image is generated by integrating or deconvolving the phase differential image in step S34. However, it is also possible to treat the phase differential image as a two-dimensional phase image. In this case, without performing step S34, the three-dimensional refractive index distribution of the object can be determined from the phase differential image (two-dimensional phase image) generated in step S33 by using a kernel that includes the kernel used in the deconvolution of step S34 in the deconvolution of step S6 for calculating the refractive index distribution.

[0084] Next, the refractive index distribution measurement method B will be described. Figures B01 to B03 show the configurations of observation devices 1D to 1F that can be used when measuring the refractive index distribution using the refractive index distribution measurement method B. Observation device 1D shown in Figure B01 differs from observation device 1A shown in Figure A01 in that the optical system from the light source 11 to the imaging unit 43 is the same, but it has an analysis unit 60 instead of an analysis unit 50. Observation device 1E shown in Figure B02 differs from observation device 1B shown in Figure A02 in that the optical system from the light source 11 to the imaging unit 43 is the same, but it has an analysis unit 60 instead of an analysis unit 50. Observation device 1F shown in Figure B03 differs from observation device 1C shown in Figure A03 in that the optical system from the light source 11 to the imaging unit 43 is the same, but it has an analysis unit 60 instead of an analysis unit 50.

[0085] The analysis unit 60 is electrically connected to the imaging unit 43 and receives interference intensity images captured by the imaging unit 43 as input. The analysis unit 60 calculates the three-dimensional refractive index distribution of the object being observed S by processing the input interference intensity images. The analysis unit 60 may be a computer. The analysis unit 60 comprises an interference intensity image acquisition unit 61, a first complex amplitude image generation unit 62, a second complex amplitude image generation unit 63, a phase conjugate calculation unit 64, a two-dimensional phase image generation unit 65, a three-dimensional phase image generation unit 66, a refractive index distribution calculation unit 67, a display unit 68, and a storage unit 69.

[0086] The interference intensity image acquisition unit 61 illuminates the object to be observed S along each of the multiple light irradiation directions by changing the orientation of the reflective surface of the mirror 22. The interference intensity image acquisition unit 61 also acquires interference intensity images at a reference position from the imaging unit 43 for each of the multiple light irradiation directions. The interference intensity image acquisition unit 61 includes a CPU and has an output port that outputs a control signal for changing the orientation of the reflective surface of the mirror 22, and an input port that receives interference intensity images from the imaging unit 43. It is not necessary to move the objective lens 25 in the optical axis direction. The reference position is the image plane position that is conjugate to the imaging plane of the imaging unit 43.

[0087] The first complex amplitude image generation unit 62, the second complex amplitude image generation unit 63, the phase conjugate calculation unit 64, the two-dimensional phase image generation unit 65, the three-dimensional phase image generation unit 66, and the refractive index distribution calculation unit 67 perform processing based on interference intensity images and include processing units such as a CPU, GPU, DSP, or FPGA. The display unit 68 displays images to be processed, images in the process of processing, and images after processing, and includes, for example, a liquid crystal display. The storage unit 69 stores data for various images and includes a hard disk drive, flash memory, RAM, and ROM. The first complex amplitude image generation unit 62, the second complex amplitude image generation unit 63, the phase conjugate calculation unit 64, the two-dimensional phase image generation unit 65, the three-dimensional phase image generation unit 66, the refractive index distribution calculation unit 67, and the storage unit 69 may be configured using cloud computing.

[0088] The memory unit 69 also stores programs for causing the interference intensity image acquisition unit 61, the first complex amplitude image generation unit 62, the second complex amplitude image generation unit 63, the phase conjugate calculation unit 64, the two-dimensional phase image generation unit 65, the three-dimensional phase image generation unit 66, and the refractive index distribution calculation unit 67 to perform their respective processes. These programs may be stored in the memory unit 69 during the manufacturing or shipment of the observation devices 1D to 1F, or they may be acquired via a communication line after shipment and stored in the memory unit 69, or they may be recorded on a computer-readable recording medium 2 and stored in the memory unit 69. The recording medium 2 can be any flexible disk, CD-ROM, DVD-ROM, BD-ROM, USB memory, etc.

[0089] Details of the processing of the interference intensity image acquisition unit 61, the first complex amplitude image generation unit 62, the second complex amplitude image generation unit 63, the phase conjugate calculation unit 64, the two-dimensional phase image generation unit 65, the three-dimensional phase image generation unit 66, and the refractive index distribution calculation unit 67 will be described later.

[0090] Figure B04 is a flowchart of refractive index distribution measurement method B. This refractive index distribution measurement method B is possible when using any of the observation devices 1D to 1F. This refractive index distribution measurement method B comprises an interference intensity image acquisition step S61, a first complex amplitude image generation step S62, a second complex amplitude image generation step S63, a phase conjugate calculation step S64, a two-dimensional phase image generation step S65, a three-dimensional phase image generation step S66, and a refractive index distribution calculation step S67.

[0091] The interference intensity image acquisition step S61 is performed by the interference intensity image acquisition unit 61. The first complex amplitude image generation step S62 is performed by the first complex amplitude image generation unit 62. The second complex amplitude image generation step S63 is performed by the second complex amplitude image generation unit 63. The phase conjugate calculation step S64 is performed by the phase conjugate calculation unit 64. The two-dimensional phase image generation step S65 is performed by the two-dimensional phase image generation unit 65. The three-dimensional phase image generation step S66 is performed by the three-dimensional phase image generation unit 66. The refractive index distribution calculation step S67 is performed by the refractive index distribution calculation unit 67.

[0092] In the interference intensity image acquisition step S61, the interference intensity image acquisition unit 61 changes the orientation of the reflective surface of the mirror 22 to irradiate the object to be observed S with light along each of the multiple light irradiation directions. The interference intensity image acquisition unit 61 then acquires interference intensity images at a reference position for each of the multiple light irradiation directions from the imaging unit 43.

[0093] In the first complex amplitude image generation step S62, the first complex amplitude image generation unit 62 generates a complex amplitude image of a reference position for each of the multiple light irradiation directions based on the interference intensity image of the reference position acquired by the interference intensity image acquisition unit 61. In the case of observation device 1D (Figure B01) and observation device 1F (Figure B03), the first complex amplitude image generation unit 62 can generate a complex amplitude image based on a single interference intensity image using the Fourier fringe analysis method. In the case of observation device 1E (Figure B02), the first complex amplitude image generation unit 62 can generate a complex amplitude image based on three or more interference intensity images in which the optical path length difference (phase difference) between the object light and the reference light is different from each other, using the phase shift method.

[0094] In the second complex amplitude image generation step S63, the second complex amplitude image generation unit 63 generates complex amplitude images for each of the multiple z-direction positions for each of the multiple light irradiation directions, based on the complex amplitude image of the reference position (z=0) generated by the first complex amplitude image generation unit 62.

[0095] In refractive index distribution measurement method B, the interference intensity image acquisition step S61, the first complex amplitude image generation step S62, and the second complex amplitude image generation step S63 perform the same processing as in the interference intensity image acquisition step S1, the first complex amplitude image generation step S2, and the second complex amplitude image generation step S3 in refractive index distribution measurement method A, respectively.

[0096] The phase conjugate calculation step S64 is performed after the processing of the second complex amplitude image generation step S63. The phase conjugate calculation step S64 may also be performed before the processing of the second complex amplitude image generation step S63 (as described later). Furthermore, if the second complex amplitude image generation step S63 generates a complex amplitude image at a certain z position from a complex amplitude image at a reference position through multiple steps, the phase conjugate calculation step S64 may be performed between one of those multiple steps and the next (as described later). In the phase conjugate calculation step S64, the phase conjugate calculation unit 64 performs a phase conjugate calculation on each of the multiple complex amplitude images for each irradiation direction to generate a complex amplitude image for each of the multiple irradiation directions when the relationship between light irradiation and imaging of the object to be observed is reversed.

[0097] Phase conjugation is an operation performed on complex amplitude images based on the phase conjugate method. It involves calculating a transmission matrix that represents the relationship between light irradiation and light output in an object, and includes the calculation of its inverse matrix and coordinate transformation. The phase conjugation method is also sometimes called phase conjugation, time reversal method, time reversal, digital phase conjugation, or digital phase conjugate method. Further details will be described later.

[0098] In the two-dimensional phase image generation step S65, the two-dimensional phase image generation unit 65 generates a two-dimensional phase image for each of the multiple complex amplitude images for each of the multiple light irradiation directions generated by the second complex amplitude image generation unit 63 or the phase conjugate calculation unit 64 for each of the multiple positions. The two-dimensional phase image generated here corresponds to a phase image centered on the z-direction position where the focus is achieved.

[0099] In the two-dimensional phase image generation step S65, when the phase image generated based on the complex amplitude image before processing in the phase conjugate calculation step S64 is defined as the first phase image, and the phase image generated based on the complex amplitude image obtained after processing in the phase conjugate calculation step S64 is defined as the second phase image, the two-dimensional phase image is generated mainly based on the first phase image for positions relatively close to the imaging unit, and mainly based on the second phase image for positions relatively far from the imaging unit.

[0100] Alternatively, in the second complex amplitude image generation step S63, after generating complex amplitude images for each of the multiple positions for each of the multiple light irradiation directions, the processing from the phase conjugate calculation step S64 onward may be performed. Alternatively, in the second complex amplitude image generation step S63, a complex amplitude image for a certain z-direction position may be generated for each of the multiple light irradiation directions, and a two-dimensional phase image of that position may be generated in the two-dimensional phase image generation step S65. This unit processing may be repeated while scanning the z-direction positions. In the latter case, the amount of image data that the storage unit 69 needs to store can be reduced.

[0101] In the 3D phase image generation step S66, the 3D phase image generation unit 66 generates a 3D phase image based on the 2D phase image generation unit 65, which generates 2D phase images for each of the multiple positions. The 3D phase image generated here is an image in which the positions x and y in the 2D phase image and the position z in the 2D phase image are variables.

[0102] In step S67, the refractive index distribution calculation unit 67 determines the three-dimensional refractive index distribution of the object to be observed by deconvolution based on the three-dimensional phase image generated by the three-dimensional phase image generation unit 66.

[0103] In refractive index distribution measurement method B, the 2D phase image generation step S65, the 3D phase image generation step S66, and the refractive index distribution calculation step S67 perform the same processing as in the 2D phase image generation step S4, the 3D phase image generation step S5, and the refractive index distribution calculation step S6 in refractive index distribution measurement method A, respectively.

[0104] Figure B05 illustrates the sequence and images of the second complex amplitude image generation step S63 and the two-dimensional phase image generation step S65. This figure shows a configuration in which the phase conjugate calculation step S64 is not performed. In this configuration, in the second complex amplitude image generation step S63, for each of the multiple light irradiation directions, complex amplitude images for each of the multiple z-direction positions (z=z1, z2, z3 in this figure) are generated based on the complex amplitude image of the reference position (z=0) generated in the first complex amplitude image generation step S62, using the free propagation equations (1) and (2) above. Then, in the two-dimensional phase image generation step S65, for each of the multiple positions, a complex differential interference image is generated based on the complex amplitude images for each of the multiple light irradiation directions generated in the second complex amplitude image generation step S63, and a phase differential image is further generated.

[0105] Figures B06 to B08 illustrate the sequence and images of the second complex amplitude image generation step S63, the phase conjugate calculation step S64, and the two-dimensional phase image generation step S65. These figures show the configuration in which the phase conjugate calculation step S64 is performed before, during, or after the second complex amplitude image generation step S63.

[0106] The first embodiment shown in Figure B06 corresponds to the flowchart in Figure B04. In this first embodiment, the phase conjugate calculation step S64 is performed after the processing of the second complex amplitude image generation step S63. In the second complex amplitude image generation step S63, for each of the multiple light irradiation directions, complex amplitude images for each of the multiple z-direction positions (z=z1, z2, z3 in this figure) are generated based on the complex amplitude image of the reference position (z=0) generated in the first complex amplitude image generation step S62, using the free propagation equations (1) and (2) above.

[0107] In the first embodiment, in the phase conjugate calculation step S64, a phase conjugate calculation is performed on the complex amplitude images for each of the multiple irradiation directions for each of the multiple positions, thereby generating complex amplitude images for each of the multiple irradiation directions when the relationship between light irradiation and imaging of the object being observed is reversed. Then, in the two-dimensional phase image generation step S65, a complex differential interference image is generated for each of the multiple positions based on the complex amplitude images for each of the multiple light irradiation directions generated in the phase conjugate calculation step S64, and a phase differential image is further generated.

[0108] In the second embodiment shown in Figure B07, the phase conjugate calculation step S64 is performed before the processing of the second complex amplitude image generation step S63. In the phase conjugate calculation step S64, for each of the multiple light irradiation directions, a phase conjugate calculation is performed on the complex amplitude image of the reference position (z=0) generated in the first complex amplitude image generation step S62, thereby generating complex amplitude images for each of the multiple irradiation directions when the relationship between light irradiation and imaging of the object being observed is reversed.

[0109] In the second embodiment, in the second complex amplitude image generation step S63, for each of the multiple light irradiation directions, complex amplitude images for each of the multiple z-direction positions (z=z1, z2, z3 in this figure) are generated based on the complex amplitude image of the reference position (z=0) generated in the phase conjugate calculation step S64, using the free propagation equations (1) and (2) above. Then, in the two-dimensional phase image generation step S65, for each of the multiple positions, a complex differential interference image is generated based on the complex amplitude images for each of the multiple light irradiation directions generated in the second complex amplitude image generation step S63, and a phase differential image is further generated.

[0110] In the third embodiment shown in Figure B08, when the second complex amplitude image generation step S63 generates complex amplitude images for multiple positions from a complex amplitude image of a reference position through two steps, the phase conjugate calculation step S64 is performed between the first and second steps of those two steps.

[0111] In the third embodiment, in the first step of the second complex amplitude image generation step S63, for each of the multiple light irradiation directions, complex amplitude images for each of the multiple z-direction positions (z=z1, z3, z5 in this figure) are generated based on the complex amplitude image of the reference position (z=0) generated in the first complex amplitude image generation step S62, using the free propagation equations (1) and (2) above. Subsequently, in the phase conjugate calculation step S64, a phase conjugate calculation is performed on the complex amplitude images for each of the multiple irradiation directions to generate complex amplitude images for each of the multiple irradiation directions when the relationship between light irradiation and imaging of the object to be observed is reversed.

[0112] In the third embodiment, in the second stage of the second complex amplitude image generation step S63, for each of the multiple light irradiation directions, complex amplitude images for each z-direction position (z=z2,z4,z6) are generated based on the complex amplitude images of the z-direction positions (z=z1,z3,z5) generated in the phase conjugate calculation step S64, using the free propagation equations (1) and (2) above. Then, in the two-dimensional phase image generation step S65, for each of the multiple positions, a complex differential interference image is generated based on the complex amplitude images for each of the multiple light irradiation directions generated in the second complex amplitude image generation step S63, and a phase differential image is generated.

[0113] The number of phase conjugate operations performed on the complex amplitude image in the phase conjugate operation step S64 differs among these first, second, and third embodiments. The overall processing time for the phase conjugate operation step S64 is shorter in the third embodiment than in the first embodiment, and even shorter in the second embodiment.

[0114] Figure B09 illustrates the sequence and images of each process in the 3D phase image generation step S66 and the refractive index distribution calculation step S67. In the 3D phase image generation step S66, a 3D phase image is generated based on the 2D phase images of each of the multiple locations generated in the 2D phase image generation step S65. At this time, for locations relatively close to the imaging unit, the 2D phase image generated based on the complex amplitude image before the phase conjugate calculation step S64 (the 2D phase image generated in the manner shown in Figure B05) is mainly used. On the other hand, for locations relatively far from the imaging unit, the 2D phase image generated based on the complex amplitude image after the phase conjugate calculation step S64 (the 2D phase image generated in any of the manners shown in Figures B06 to B08) is mainly used. Subsequently, in the refractive index distribution calculation step S67, the 3D refractive index distribution of the object to be observed is determined by deconvolution based on the 3D phase image generated in the 3D phase image generation step S66. Furthermore, each refractive index distribution data that constitutes the three-dimensional refractive index distribution (for example, the two-dimensional refractive index distribution data that constitutes the three-dimensional refractive index distribution in Figure B09) can be used as refractive index tomography data.

[0115] There are three ways to generate a two-dimensional phase image for each position in the z direction. The first phase image φ1 is generated based on the complex amplitude image before the processing of the phase conjugate calculation step S64 (the phase image generated in the manner shown in Figure B05). The second phase image φ2 is generated based on the complex amplitude image after the processing of the phase conjugate calculation step S64 (the phase image generated in any of the manners shown in Figures B06 to B08). A weighting function α is used whose derivative with respect to the variable z, which represents the distance from the imaging unit along the optical propagation path, is 0 or less. The value of the weighting function is between 0 and 1.

[0116] In the first embodiment, the weight function α is such that z is a threshold z th The value is positive (e.g., 1) within the following range, and 0 in all other ranges. That is, a two-dimensional phase image can be represented by equation (8) below.

[0117]

number

[0118]

number

[0119]

number

[0120] Next, the content of the phase conjugate operation performed in phase conjugate operation step S64 will be explained using Figures B10 and B11.

[0121] Figure B10 shows the input light U when the imaging unit captures an interference intensity image. in (k in ) and output light u out (r out This is a diagram showing U. in (k in ) is the wavenumber k of the light irradiated onto the object being observed. in This represents the complex amplitude of u. out (r out ) is the position r of the light emitted from the object being observed. out This represents the complex amplitude of U. in (k in ) and u out (r out The relationship between ) and is expressed by equation (11) below. Column vector U in The nth element U in (k in n ) is wavenumber k in n Represents the complex amplitude of a plane wave. Column vector u out The nth element u out (r out n ) is at position r out n This represents the complex amplitude of light observed. It is an N x N matrix T(r out ,k in ) is U in (k in ) and u out (r out This represents a linear relationship between ) and is called the transmission matrix. Such a transmission matrix can represent the light scattering process in the observed object. Matrix T(r out ,k in Element T of the n1th row and n2nd column of ) n1,n2 is the wave number k in n2 When a plane wave with amplitude 1 is input, position r out n1 This represents the complex amplitude of light observed.

[0122]

number

[0124]

number

[0125]

number

[0126]

number

[0127] In the phase conjugate calculation step S64, first, based on the complex amplitude image, the transmission matrix T(r) obtained when the interference intensity image was captured by the imaging unit is calculated. out ,k in Next, we find this transmission matrix T(r out ,k in ) and based on equation (15) above, the transmission matrix S(r in ,k out We calculate the transmission matrix S(r). in ,k out Based on this, a complex amplitude image is obtained when the relationship between light irradiation and imaging is reversed.

[0128] When the imaging unit captures an interference intensity image for each of the multiple light irradiation directions, the input light vector U for the nth light irradiation direction in n (k in ) is expressed by equation (16) below, where only the value of the nth element is 1 and the values ​​of the other elements are 0. This input light U inn (k in ) for output light u out n (r out This is expressed by equation (17) below. Equation (17) corresponds to the complex amplitude obtained for the nth light irradiation direction.

[0129]

number

[0130]

number

[0131]

number

[0132]

number

[0133]

number

[0134]

number

[0135] Transmission matrix S(r) when the relationship between light irradiation and imaging is reversed in ,k out When determining the transmission matrix T(r), as shown in equation (15) above, out ,k in It is necessary to calculate the inverse matrix of ). Therefore, the transmission matrix T must be a square matrix in which the number of row elements and the number of column elements are equal. In other words, the dimension of the matrix in the light irradiation sidewavenumber space for the object to be observed during the interference intensity image acquisition step S61 must be equal to the number of pixels in the complex amplitude image.

[0136] To make the two equal, one can either match the dimension of the matrix in the light-irradiated wavenumber space for the object being observed during the interference intensity image acquisition step S61 to the number of pixels, or use only a portion of the image obtained by the imaging unit for subsequent processing. However, generally, the number of pixels in the image obtained by the imaging unit is, for example, 1024 × 1024, so it is not easy to make the dimension of the matrix in the light-irradiated wavenumber space for the object being observed the same as the number of pixels. Furthermore, using only a portion of the image obtained by the imaging unit for subsequent processing leads to a decrease in resolution.

[0137] Therefore, as shown in Figure B12, in the phase conjugate calculation step S64, the complex amplitude image may be divided into multiple subimages, each having the same number of pixels as the matrix dimension in the light-irradiated wavenumber space for the object being observed. Phase conjugate calculation may then be performed on each of these subimages, and the multiple subimages may then be combined. In this case, any two or more of the multiple subimages may have a common region.

[0138] Next, the refractive index distribution measurement method C will be described. Figures C01 to C03 show the configurations of observation devices 1G to 1I that can be used when measuring the refractive index distribution using the refractive index distribution measurement method C. Observation device 1G shown in Figure C01 differs from observation device 1A shown in Figure A01 in that the optical system from the light source 11 to the imaging unit 43 is the same, but it is equipped with an analysis unit 70 instead of an analysis unit 50. Observation device 1H shown in Figure C02 differs from observation device 1B shown in Figure A02 in that the optical system from the light source 11 to the imaging unit 43 is the same, but it is equipped with an analysis unit 70 instead of an analysis unit 50. Observation device 1I shown in Figure C03 differs from observation device 1C shown in Figure A03 in that the optical system from the light source 11 to the imaging unit 43 is the same, but it is equipped with an analysis unit 70 instead of an analysis unit 50.

[0139] The analysis unit 70 is electrically connected to the imaging unit 43 and receives interference intensity images captured by the imaging unit 43 as input. The analysis unit 70 calculates the three-dimensional refractive index distribution of the object being observed S by processing the input interference intensity images. The analysis unit 70 may be a computer. The analysis unit 70 comprises an interference intensity image acquisition unit 71, a first complex amplitude image generation unit 72, a second complex amplitude image generation unit 73, a two-dimensional phase image generation unit 74, a three-dimensional phase image generation unit 75, a refractive index distribution calculation unit 76, a third complex amplitude image generation unit 77, a display unit 78, and a storage unit 79.

[0140] The interference intensity image acquisition unit 71 illuminates the object to be observed S along multiple light irradiation directions by changing the orientation of the reflective surface of the mirror 22. The interference intensity image acquisition unit 71 also acquires interference intensity images at a reference position from the imaging unit 43 for each of the multiple light irradiation directions. The interference intensity image acquisition unit 71 includes a CPU and has an output port that outputs a control signal for changing the orientation of the reflective surface of the mirror 22, and an input port that receives interference intensity images from the imaging unit 43. It is not necessary to move the objective lens 25 in the optical axis direction. The reference position is an image plane position that is conjugate to the imaging plane of the imaging unit 43.

[0141] The first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 perform processing based on interference intensity images and include processing units such as CPUs, GPUs, DSPs, or FPGAs. The display unit 78 displays images to be processed, images in the process of processing, and images after processing, and includes, for example, a liquid crystal display. The storage unit 79 stores data for various images and includes a hard disk drive, flash memory, RAM, and ROM. The first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, the third complex amplitude image generation unit 77, and the storage unit 79 may be configured using cloud computing.

[0142] The memory unit 79 also stores programs for causing the interference intensity image acquisition unit 71, the first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 to perform their respective processes. These programs may be stored in the memory unit 79 during the manufacturing or shipment of the observation devices 1G to 1I, or they may be acquired via a communication line after shipment and stored in the memory unit 79, or they may be recorded on a computer-readable recording medium 2 and stored in the memory unit 79. The recording medium 2 can be any flexible disk, CD-ROM, DVD-ROM, BD-ROM, USB memory, etc.

[0143] Details of the processing of the interference intensity image acquisition unit 71, the first complex amplitude image generation unit 72, the second complex amplitude image generation unit 73, the two-dimensional phase image generation unit 74, the three-dimensional phase image generation unit 75, the refractive index distribution calculation unit 76, and the third complex amplitude image generation unit 77 will be described later.

[0144] Figures C04 and C05 are flowcharts of refractive index distribution measurement method C. Figure C05 shows a portion of the flowchart in Figure C04. This refractive index distribution measurement method C is possible when using any of the observation devices 1G to 1I. This refractive index distribution measurement method C comprises interference intensity image acquisition step S71, first complex amplitude image generation step S72, second complex amplitude image generation step S73, two-dimensional phase image generation step S74, three-dimensional phase image generation step S75, refractive index distribution calculation step S76, and third complex amplitude image generation step S77.

[0145] The interference intensity image acquisition step S71 is performed by the interference intensity image acquisition unit 71. The first complex amplitude image generation step S72 is performed by the first complex amplitude image generation unit 72. The second complex amplitude image generation step S73 is performed by the second complex amplitude image generation unit 73. The two-dimensional phase image generation step S74 is performed by the two-dimensional phase image generation unit 74. The three-dimensional phase image generation step S75 is performed by the three-dimensional phase image generation unit 75. The refractive index distribution calculation step S76 is performed by the refractive index distribution calculation unit 76. The third complex amplitude image generation step S77 is performed by the third complex amplitude image generation unit 77.

[0146] In the interference intensity image acquisition step S71, the interference intensity image acquisition unit 71 changes the orientation of the reflective surface of the mirror 22 to irradiate the object to be observed S along each of the multiple light irradiation directions. The interference intensity image acquisition unit 71 then acquires interference intensity images at a reference position for each of the multiple light irradiation directions from the imaging unit 43.

[0147] In the first complex amplitude image generation step S72, the first complex amplitude image generation unit 72 generates a complex amplitude image for each of the multiple light irradiation directions based on the interference intensity image acquired by the interference intensity image acquisition unit 71. In the case of observation device 1G (Figure C01) and observation device 1I (Figure C03), the first complex amplitude image generation unit 72 can generate a complex amplitude image based on one interference intensity image using the Fourier fringe analysis method. In the case of observation device 1H (Figure C02), the first complex amplitude image generation unit 72 can generate a complex amplitude image based on three or more interference intensity images in which the optical path length difference (phase difference) between the object light and the reference light is different from each other, using the phase shift method. The complex amplitude image generated in the first complex amplitude image generation step S72 may be at the same reference position as the interference intensity image, or it may be at a different position generated based on the complex amplitude image at the reference position.

[0148] In the second complex amplitude image generation step S73, the second complex amplitude image generation unit 73 generates complex amplitude images for each of several z-direction positions between the first and second positions, based on the complex amplitude image of the first position with respect to the distance from the imaging unit 43 along the light propagation path, for each of several light irradiation directions.

[0149] In the two-dimensional phase image generation step S74, the two-dimensional phase image generation unit 74 generates a two-dimensional phase image for each of the multiple positions based on the multiple complex amplitude images for each of the multiple light irradiation directions generated by the second complex amplitude image generation unit 73. The two-dimensional phase image generated here corresponds to a phase image centered on the z-direction position where the focus is achieved.

[0150] In the 3D phase image generation step S75, the 3D phase image generation unit 75 generates a 3D phase image between the first position and the second position based on the 2D phase images of each of the multiple positions generated by the 2D phase image generation unit 74. The 3D phase image generated here is an image in which the positions x and y in the 2D phase image and the position z in the 2D phase image are variables.

[0151] In the refractive index distribution calculation step S76, the refractive index distribution calculation unit 76 determines the three-dimensional refractive index distribution of the object to be observed between the first position and the second position by deconvolution, based on the three-dimensional phase image generated by the three-dimensional phase image generation unit 75.

[0152] In refractive index distribution measurement method C, the interference intensity image acquisition step S71, the first complex amplitude image generation step S72, the second complex amplitude image generation step S73, the two-dimensional phase image generation step S74, the three-dimensional phase image generation step S75, and the refractive index distribution calculation step S76 perform substantially the same processing as the interference intensity image acquisition step S1, the first complex amplitude image generation step S2, the second complex amplitude image generation step S3, the two-dimensional phase image generation step S4, the three-dimensional phase image generation step S5, and the refractive index distribution calculation step S6 in refractive index distribution measurement method A.

[0153] In the third complex amplitude image generation step S77, the third complex amplitude image generation unit 77 generates a complex amplitude image of the second position for each of the multiple light irradiation directions, based on the complex amplitude image of the first position used in the second complex amplitude image generation step S73 and the three-dimensional refractive index distribution of the object to be observed between the first and second positions calculated in the refractive index distribution calculation step S76.

[0154] Step S83, which includes the second complex amplitude image generation step S73, the two-dimensional phase image generation step S74, the three-dimensional phase image generation step S75, and the refractive index distribution calculation step S76, determines the three-dimensional refractive index distribution of the object to be observed between the first and second positions based on the complex amplitude image of the first position with respect to the distance from the imaging unit 43 along the light propagation path. Each process of step S83 and the third complex amplitude image generation step S77 is repeated. This will be explained using Figures C04 to C07.

[0155] Figure C06 illustrates the relationship between the region containing the object to be observed and the blocks from the 1st to the Jth. As shown in this figure, the region containing the object to be observed is divided into blocks from the 1st to the Jth in order based on the distance from the imaging unit along the light propagation path (z direction). In this figure, J=3. Of the blocks from the 1st to the Jth, the jth block is z=z j-1 From z=z j This is the region up to [the point where z=z is closest to the imaging unit]. j-1 The position (near end) is defined as the first position, and the position furthest from the imaging unit is z=z j The position (far end) is designated as the second position.

[0156] Figure C07 illustrates the processing procedure in blocks 1 through J. As shown in this figure, for each block j, in step S83, complex amplitude images and two-dimensional phase images are generated for each of multiple z-direction positions between the first and second positions based on the complex amplitude image of the first position, a three-dimensional phase image is generated between the first and second positions, and the three-dimensional refractive index distribution is further determined. For each block j, in the third complex amplitude image generation step S77, a complex amplitude image of the second position is generated based on the complex amplitude image of the first position and the three-dimensional refractive index distribution calculated in step S83.

[0157] The complex amplitude image of the second position of the (j+1)th block, generated in the third complex amplitude image generation step S77, is used as the complex amplitude image of the first position of the next jth block, and the processes of step S83 and the third complex amplitude image generation step S77 are performed for the jth block. Once the three-dimensional refractive index distribution is obtained for each of the first to jth blocks, these are combined to obtain the overall three-dimensional refractive index distribution of the object being observed. The three-dimensional refractive index distributions of each of the first to jth blocks (for example, the refractive index distribution of the first block, the refractive index distribution of the second block, and the refractive index distribution of the third block in Figure C07) can be used as refractive index tomography data.

[0158] As shown in Figures C04 and C05, in step S81 following the first complex amplitude image generation step S72, j is set to 0, and in the subsequent step S82, the value of j is increased by 1 to j=1, and the processes of step S83 and the third complex amplitude image generation step S77 are performed for the first block. That is, for the first block closest to the imaging unit, based on the complex amplitude image generated in the first complex amplitude image generation step S72, the position z=z0 closest to the imaging unit (near end) is set as the first position, and the position z=z1 furthest from the imaging unit (far end) is set as the second position, and the processes of step S83 (second complex amplitude image generation step S73, 2D phase image generation step S74, 3D phase image generation step S75, refractive index distribution calculation step S76) and the third complex amplitude image generation step S77 are performed in order. After that, the process returns to step S82.

[0159] For the j-th block (where j is 2 or more and less than J), based on the complex amplitude image generated for the (j - 1)-th block in the third complex amplitude image generation step S77, the position of z = z j-1 closest to the imaging unit (the proximal end) is taken as the first position, and the position of z = z j farthest from the imaging unit (the distal end) is taken as the second position, and the processes of step S83 (the second complex amplitude image generation step S73, the two-dimensional phase image generation step S74, the three-dimensional phase image generation step S75, the refractive index distribution calculation step S76) and the third complex amplitude image generation step S77 are sequentially performed. Then, it returns to step S82.

[0160] For the J-th block, which is the final-stage block farthest from the imaging unit, based on the complex amplitude image generated for the (J - 1)-th block in the third complex amplitude image generation step S77, the position of z = z J-1 closest to the imaging unit (the proximal end) is taken as the first position, and the position of z = z J farthest from the imaging unit (the distal end) is taken as the second position, and the process of step S83 (the second complex amplitude image generation step S73, the two-dimensional phase image generation step S74, the three-dimensional phase image generation step S75, the refractive index distribution calculation step S76) is performed.

[0161] For the J-th block, in step S84 after step S83, if it is determined to be the final-stage block, it may simply end without proceeding to the third complex amplitude image generation step S77. Note that for the J-th block, after the three-dimensional phase image generation step S75, if it is determined to be the final-stage block, it may end without proceeding to the refractive index distribution calculation step S76. In this case, a three-dimensional phase image of the entire observation object can be obtained.

[0162] Furthermore, the region containing the object to be observed may be divided into two blocks in sequence based on the distance from the imaging unit along the light propagation path (z direction). In this case, the processing for the first block and the processing for the final J block described above should be performed. Alternatively, the region containing the object to be observed does not need to be divided into multiple blocks. In this case, the processing for step S83 (second complex amplitude image generation step S73, two-dimensional phase image generation step S74, three-dimensional phase image generation step S75, refractive index distribution calculation step S76) and the processing for the third complex amplitude image generation step S77 may be performed only once in sequence.

[0163] Next, the details of the third complex amplitude image generation step S77 will be explained. When illuminating the object to be observed with light to acquire an interference intensity image, in each j-block, the second position (z=z j The wavefront of the light propagates through the interior of the j-th block to the first position (z=z j-1 It reaches ) and then propagates to the imaging unit.Therefore, in the third complex amplitude image generation step S77, the first position (z=z) is determined by numerical calculation considering the refractive index distribution of the j-th block. j-1 By back-propagating the wavefront of the ) through the interior of the j-th block, the second position (z=z j The wavefront of the ) is determined. That is, in the third complex amplitude image generation step S77, for each of the multiple light irradiation directions, the first position of the j block (z=z j-1 Based on the complex amplitude image of ) and the refractive index distribution of block j, the second position of block j (z=z j A complex amplitude image of the light wavefront is generated. In this process, a method is used to numerically calculate the propagation of the light wavefront, taking into account the refractive index distribution of the medium. Known numerical calculation methods for such non-uniform medium propagation include BPM (Beam Propagation Method) and SSNP (Split-Step Non-Paraxial). Below, the process using BPM in the third complex amplitude image generation step S77 will be described.

[0164] Figure C08 illustrates the BPM processing steps. This figure shows an arbitrary j-th block. As shown in this figure, the j-th block is divided into M slices (slices 1 to M in this figure) based on the distance from the imaging unit along the optical propagation path (z-direction). The thickness of each slice is approximately the wavelength.

[0165] The thickness of each slice may be constant. Here, the thickness of each slice is a constant value Δz. Of the 1st to Mth slices of the jth block, the mth slice is at position (z j-1 From position (z) +(m-1)Δz) j-1 Up to +mΔz). The first position of the j-th block (z=z j-1 ) to the second position (z=z j In order towards the first to the Mth slice, a phase change corresponding to the refractive index distribution is sequentially applied to cause the light wavefront to propagate backward by Δz.

[0166] Furthermore, the thickness Δz of each slice in the third complex amplitude image generation step S77 may be different from or the same as the positional interval used when generating complex amplitude images for multiple z-direction positions between the first and second positions in the second complex amplitude image generation step S73.

[0167] The phase change o(x,y,z) given to the optical wavefront when a slice of thickness Δz at position z is backpropagated is expressed by equation (22) below. In equation (22), k v n is the wavenumber of light in a vacuum. δn(x,y,z) is the refractive index distribution n(x,y,z) of the object being observed at position z and the refractive index n of the background (medium). b This is the difference between the two, and is expressed by equation (23) below. Also, cosθ is expressed by equation (24) below.

[0168]

number

[0169]

number

[0170]

number

[0171] Position of the m slice (z=z j-1 If the complex amplitude of light at +(m-1)Δz) is u(x,y,z), then the complex amplitude of light at position (z+Δz) after the light has backpropagated through the interior of the m-th slice, u(x,y,z+Δz), is given by equation (25) below. In equation (25), P(k x ,k y Δz) is expressed by equation (26) below. Equation (25) is obtained by performing a Fourier transform on the product of the complex amplitude u(x,y,z) of light and the phase change o(x,y,z), and then multiplying the result of this Fourier transform by P(k x ,k y This represents obtaining the complex amplitude u(x,y,z+Δz) of light at position (z+Δz) after propagation through a slice of thickness Δz by performing an inverse Fourier transform on the product with Δz. Δz This is a function that performs calculations for the propagation of light at Δz.

[0172]

number

[0173]

number

[0174] The propagation of the optical wavefront in each slice of block j is expressed by equations (27) to (29) below. That is, at the first position of block j (z=z j-1 The complex amplitude of light at ) is u(x,y,z j-1 If we assume that the complex amplitude u(x,y,z) of the light after it has propagated through the first slice of the j-th block is j-1 +Δz) is expressed by equation (27) below. The complex amplitude of the light after it has propagated through the (m-1) slice of the j-th block is u(x,y,z j-1Let it be +(m - 1)Δz), then the complex amplitude u(x, y, z j-1 + mΔz) after propagating through the m-th slice of the j-th block is expressed by the following equation (28). Let the complex amplitude of the light after propagating through the (M - 1)-th slice of the j-th block be u(x, y, z j-1 +(M - 1)Δz). Then, the complex amplitude u(x, y, z j ) of the light at the second position (z = z j ) after propagating through the M-th slice of the j-th block is expressed by the following equation (29).

[0175] [Number]

[0176] [Number]

[0177] [Number] In this way, in the third complex amplitude image generation step S77, by numerically calculating considering the refractive index distribution of the j-th block, the wavefront of the light wave at the first position (z = z j-1 ) is sequentially back-propagated slice by slice inside the j-th block, and the wavefront of the light wave at the second position (z = z j ) can be obtained.

[0178] FIG. C09 is a flowchart of the third complex amplitude image generation step S77. In step S41, the position z is initialized to the first position (z = z j-1 ) of the j-th block. In step S42, the interaction between the complex amplitude u(x, y, z) of the light at the position z and the phase change o(x, y, z) is obtained. In step S43, the wavefront of the light after the interaction is propagated by a distance Δz, and the complex amplitude u(x, y, z + Δz) of the light at the position z + Δz is obtained. In step S44, z added with Δz is set as the new z. In step S44, when the position z reaches the second position (z = z jIf it is determined that the position z has not yet been reached, the process returns to step S42 and steps S42 to S44 are repeated. In step S44, position z is the second position of the j-th block (z=z j If it is determined that the complex amplitude of the light has reached the second position (z=z) of the j-th block, the processing of the third complex amplitude image generation step S77 is terminated. j This represents the complex amplitude at ).

[0179] All of the refractive index distribution measurement methods A to C described above can achieve three-dimensional refractive index tomography with reduced effects of multiple scattered light, even when the object being observed is a multiple scattering material. All of the refractive index distribution measurement methods A to C can measure the refractive index distribution of a three-dimensional cultured body as the object being observed.

[0180] Alternatively, an observation device and refractive index distribution measurement method using self-interference may also be used. For example, the observation device 1J shown in Figure C10 includes a light source 11, lens 12, lens 21, mirror 22, lens 23, condenser lens 24, objective lens 25, mirror 44, lens 42, imaging unit 43, and analysis unit 70. Compared to the configuration of the observation devices described so far, the observation device 1J differs in that the light output from the light source 11 is guided by the optical fiber 14 and then emitted from the light output end 18 without being split into two. The observation device 1J also differs in that a mirror 44 is provided instead of a beam splitter 41. The observation device 1J does not have an interference optical system. The imaging unit 43 can capture interference intensity images of a reference position due to the self-interference of light that has been irradiated onto the object S along each of the multiple light irradiation directions and passed through the object S. The analysis unit 70 can perform the same image processing as described so far using such interference intensity images due to self-interference.

[0181] Furthermore, the three-dimensional refractive index distribution of the object S being observed between the first and second positions does not have to be a refractive index distribution based on a three-dimensional phase image, and may be obtained using a separate refractive index distribution acquisition device capable of acquiring refractive index distributions. In this case, the observation device may include: (1) an interference intensity image acquisition unit that acquires interference intensity images of the reference positions of each of the multiple light irradiation directions from an imaging unit that captures interference intensity images of the reference positions of light irradiated onto the object and passed through the object along each of the multiple light irradiation directions; (2) a first complex amplitude image generation unit that generates a complex amplitude image based on the interference intensity image for each of the multiple light irradiation directions; (3) a refractive index distribution acquisition unit that acquires the three-dimensional refractive index distribution of the object being observed between the first and second positions with respect to the distance from the imaging unit along the light propagation path; and (4) a second complex amplitude image generation unit (corresponding to the third complex amplitude image generation unit of observation devices 1A to 1D) that generates a complex amplitude image of the second position based on the complex amplitude image of the first position and the three-dimensional refractive index distribution for each of the multiple light irradiation directions.

[0182] In this case, the refractive index distribution measurement method may also include: (1) an interference intensity image acquisition step of acquiring interference intensity images of reference positions for each of the multiple light irradiation directions from an imaging unit that captures interference intensity images of reference positions for light irradiated onto the object and passed through the object along each of the multiple light irradiation directions; (2) a first complex amplitude image generation step of generating a complex amplitude image for each of the multiple light irradiation directions based on the interference intensity image; (3) a refractive index distribution acquisition step of acquiring the three-dimensional refractive index distribution of the object from a first position to a second position with respect to the distance from the imaging unit along the light propagation path; and (4) a second complex amplitude image generation step of generating a complex amplitude image of a second position for each of the multiple light irradiation directions based on the complex amplitude image of the first position and the three-dimensional refractive index distribution.

[0183] One aspect of this disclosure is a method for determining the region of necrotic cells in an object based on refractive index distribution data of the object.

[0184] In one embodiment of this disclosure, the determination of the region of necrotic cells is based on the fact that the region of each individual cell included in the refractive index distribution data has the characteristics of a necrotic cell. That is, the region in the object of observation that corresponds to the region of each individual cell having the characteristics of a necrotic cell is determined to be the region of a necrotic cell.

[0185] The characteristics of necrotic cells described above include the presence of at least a region corresponding to the nucleus. The nucleus is one of the organelles of eukaryotic cells, and in mammalian cells in particular, it exists in a nearly spherical shape with a radius of about 1 to 10 μm outside of the mitotic phase. Normally, there is one nucleus in each cell. The nucleus is separated from the cytoplasm by a nuclear membrane and contains structures such as the nucleolus and chromatin (a filamentous structure made up of nuclear DNA). Therefore, the region corresponding to the nucleus is, for example, a roughly circular or roughly spherical region located within the region of each cell included in the refractive index distribution data, and is a region in which the statistical value of the refractive index is greater than the statistical value of the refractive index of the entire cell. The statistical value is, for example, the mean, median, maximum, or minimum, and may be the mean or median. A roughly circular region represents a region in which the degree of roundness is above a threshold, and a roughly spherical region represents a region in which the degree of sphericity is above a threshold. The above threshold can be determined according to the type and state of the object being observed, as well as the type and amount of cells contained in the object. It can also be determined so that the region corresponding to the nucleus is identified as a region that is recognized as a nucleus based on other structural characteristics in the refractive index distribution data of a reference object different from the object being observed (sample) used to determine the region of necrotic cells. Alternatively, it can be determined so that the region corresponding to the nucleus is identified as a region fluorescently stained with known nuclear staining reagents such as 4',6-diamidino-2-phenylindole (DAPI), Hoechst33258, or Hoechst33342. The above threshold may be, for example, 60%, 65%, 70%, 75%, or 80%. Furthermore, the region corresponding to the nucleus is, for example, a roughly circular or spherical region surrounded by a high refractive index membrane structure corresponding to the nuclear membrane, located within the region of each cell included in the refractive index distribution data. Furthermore, the region corresponding to the nucleus is, for example, a roughly circular or spherical region located within the region of each cell included in the refractive index distribution data, surrounded by a membrane structure with a high refractive index corresponding to the nuclear membrane, and having a nucleolar structure with a high refractive index and a filamentous chromatin structure with a lower refractive index than the nucleolus.

[0186] In one embodiment of this disclosure, the characteristics of the necrotic cell further include one or more characteristics selected from the group consisting of the characteristics that the statistical value of the overall refractive index of the cell is below a threshold and the characteristics that the statistical variability of the refractive index of the nucleus is above a threshold. The statistical value is, for example, the mean, median, maximum, or minimum, and may be the mean or median. In cells that have undergone necrosis, the overall refractive index of the cell is lower compared to living cells because the contents of the cell leak out due to cell rupture. Also, focusing on the nucleus, the nucleolus in the nucleus has a relatively high refractive index even in cells that have undergone necrosis, so the statistical variability of the refractive index of the nucleus is large. Therefore, the region of a necrotic cell can be determined based on the fact that the region of each cell included in the refractive index distribution data has one or more characteristics selected from the group consisting of these characteristics. The method for determining the region of a necrotic cell according to this embodiment is called the "threshold method". Figure 35 shows a typical flow for determining the region of necrosis cells using the threshold method, based on the characteristic that the average or median refractive index of the entire cell is below a threshold, and Figure 36 shows a typical flow for determining the region based on the characteristic that the statistical variability of the nuclear refractive index is above a threshold. The comparison between the statistical value of the entire cell refractive index or the statistical variability of the nuclear refractive index and the threshold is performed for each cell in the region of cells that have a region corresponding to the nucleus within the region (determination region) where the object of observation exists in the refractive index distribution data. The region of cells that do not have a region corresponding to the nucleus is considered to be, for example, the region of apoptotic cells. The statistical variability in one embodiment of this disclosure is not particularly limited as long as it is an index that indicates the degree of uniformity of the data, and is selected from the group consisting of, for example, standard deviation, standard error, variance, coefficient of variation and the difference between the maximum and minimum values. For these cell-by-cell statistical variability, a histogram may be calculated for the cell population and its spread characteristics may be extracted, and the threshold described later may be determined based on the above characteristics. The thresholding method may be applied to unprocessed refractive index distribution data, or to processed data that has undergone image processing to enhance the spatial variation of the refractive index.Furthermore, one possible mechanism for the relatively high refractive index of nucleoli in necrotic cells is that, although nucleoli lack a membrane structure, they exist in a phase-separated, independent state, making leakage of their contents less likely.

[0187] The threshold values ​​for statistical values ​​and the threshold values ​​for statistical variability can be determined according to the type and state of the object being observed, as well as the type and amount of cells contained in the object. For example, known information may be used, or the region of cells that can be identified as necrotic based on cell morphology may be determined as the region of necrotic cells, or it may be calculated by unsupervised machine learning such as clustering. Alternatively, the threshold values ​​can be determined so that the region of cells that can be identified as necrotic cells is determined as the region of necrotic cells in the refractive index distribution data of a reference object that is different from the object being observed (sample) used to determine the region of necrotic cells. In this case, if the sample is a cell aggregate, the reference may be a cell culture of the same cell type as the cells forming the sample, or a cell aggregate consisting of cells of the same cell type. Furthermore, the reference may be an object that can clearly determine the region of necrotic cells. As an example, an object in which at least necrotic cells are fluorescently labeled may be an object, or as an example, an object in which at least necrotic cells and living cells are fluorescently labeled may be an object.

[0188] The method for fluorescently labeling necrotic cells is not particularly limited. If the object of observation is a cell aggregate and the culture conditions do not induce programmed cell death such as apoptosis, then fluorescent reagents capable of staining dead cells with broken cell membranes, such as ethidium homodimer 1 or propidium iodide, can be used. Furthermore, in one embodiment of this disclosure, when determining the region of necroptotic cells instead of the region of necrotic cells, for example, SMART (Shin Murai et.al. NATURE COMMUNICATIONS 9,4457 (2018)) can be used to fluorescently label the necroptotic cells.

[0189] The method for fluorescently labeling living cells is not particularly limited, and for example, calcein-AM can be used.

[0190] In one embodiment of this disclosure, the region of necrotic cells is determined by inputting the refractive index distribution data of the object being observed into a learning model that has been trained using training data including necrotic region data of a reference object and refractive index distribution data of the reference object corresponding to the necrotic region data. In one embodiment, the features used by the learning model include features corresponding to features having a region corresponding to the nucleus. The reference object being observed is an object composed mainly of cells, prepared or acquired in the same manner as the object being observed. The method for determining the region of necrotic cells according to this embodiment is called the "machine learning method".

[0191] A method for determining the region of a cell that has undergone programmed cell death using machine learning includes at least a learning step and an inference step.

[0192] In the learning step, a model is trained to infer the regions of living cells and the regions of necrotic cells from features using machine learning, with training data including necrotic region data of the reference object and refractive index distribution data of the reference object corresponding to the necrotic region data. The necrotic region data of the reference object is data that shows the spatial distribution of voxels or pixels occupied by necrotic cells in the space containing the reference object. Necrotic region data can be created from data of any reference object that can reveal the regions of necrotic cells in the reference object, but in one embodiment, the data of the reference object is fluorescence data of the reference object, that is, data that shows the spatial distribution of fluorescence intensity of voxels or pixels in the space containing the reference object.

[0193] Figure 37 shows a typical model learning flow in the learning step when necrotic region data is created from fluorescence data of a reference object. In the first stage of the learning step, refractive index distribution data and fluorescence data of a reference object that has fluorescently labeled cells that have undergone necrosis are obtained. The reference object may also have living cells that have been fluorescently stained. Fluorescent labeling of necrotic cells and living cells can be performed, for example, by the same method as described in the threshold method.

[0194] In the second stage of the learning process, features are extracted from the refractive index distribution data. These features are related to the spatial distribution of the refractive index and may be vectors or scalars, but they include features that correspond to features having at least a region corresponding to the nucleus.

[0195] In the third stage of the learning process, regions of cells that have undergone necrosis are extracted from fluorescence data to create necrosis region data. In creating necrosis region data, for example, regions of fluorescently labeled cells can be determined to be regions of cells that have undergone necrosis based on fluorescence data corresponding to the excitation and emission wavelengths of the dye that labels the cells that have undergone necrosis. Furthermore, when determining living cell regions, for example, regions of fluorescently labeled cells can be determined to be living cell regions based on fluorescence data corresponding to the excitation and emission wavelengths of the dye that labels the living cells. Alternatively, for example, regions of fluorescently labeled cells other than those determined to be regions of cells that have undergone necrosis can be determined to be living cell regions based on fluorescence data corresponding to the excitation and emission wavelengths of the dye that labels all cells.

[0196] In the fourth stage of the learning process, a model is trained using machine learning, employing training data that includes necrotic region data of the reference object and refractive index distribution data of the reference object corresponding to the necrotic region data. The model is trained to infer the regions of necrotic cells and living cells determined in the third stage, using the features extracted in the second stage as input. By using training data of necrotic cell regions and living cells that have undergone verification, such as simultaneous imaging of fluorescence data and refractive index distribution data, an improvement in judgment accuracy can be expected.

[0197] Figure 38 shows a typical inference flow for the inference step. In the first stage of the inference step, refractive index distribution data of the observed object is acquired. In the second stage, the same features extracted in the second stage of the learning step are extracted from the refractive index distribution data. In the third stage, the model trained in the learning step is used to infer the regions of necrotic cells and living cells in the observed object.

[0198] Another aspect of this disclosure is a method for determining a region of necrotic cells in an object, comprising the steps of: acquiring refractive index distribution data of the object; and determining a region of necrotic cells in the object using the refractive index distribution data of the object. The method for acquiring the refractive index distribution data is not particularly limited, and the determination of the region of necrotic cells may be by a threshold method or by a machine learning method.

[0199] Another aspect of this disclosure is a method for evaluating the quality of a cell aggregate, comprising the steps of acquiring refractive index distribution data of the cell aggregate and determining the region of necrotic cells within the cell aggregate based on the refractive index distribution data. Necrosis is typically cell death that occurs in cells exposed to stimuli such as hypoxia, high temperature, toxins, nutrient deficiency, and cell membrane damage in living organisms. Even when creating a cell aggregate by three-dimensional cell culture, regions with high cell density within the cell aggregate are prone to hypoxic and / or nutrient deficiency, so cells, especially near the center of the cell aggregate, often develop necrosis. Thus, the quality of the cell aggregate can be evaluated by determining the region of necrotic cells within the cell aggregate based on refractive index distribution data using the method for determining the region of necrotic cells according to this disclosure. That is, depending on the purpose, a cell aggregate with a larger region of necrotic cells within it can be evaluated as a higher quality cell aggregate, or a cell aggregate with a smaller region of necrotic cells within it can be evaluated as a higher quality cell aggregate.

[0200] In one embodiment, the cell aggregate is a cancer cell aggregate. A cancer cell aggregate is a cell aggregate containing cancer cells. In one embodiment, a cancer cell aggregate is a cell aggregate in which the cells contained are cancer cells. Figure 39 shows an example of a case in which the cell aggregate is a cancer cell aggregate, and the region of necrotic cells within the cell aggregate is large, and the cell aggregate located in the center is evaluated as being of higher quality. The tumor microenvironment in living organisms is generally hypoxic and / or nutrient-deficient. Therefore, the region of cells near the center of a cancer cell aggregate that is subjected to hypoxic and / or nutrient-deficient conditions is considered a useful cancer research model that simulates the tumor microtubule environment in living organisms, and the size of the region of necrotic cells within the cancer cell aggregate can be used as an indicator. In other words, the larger the region of necrotic cells within the cancer cell aggregate, and the more central the cancer cell aggregate, the more accurately it is considered to mimic the tumor microenvironment. These aggregates can be evaluated as high-quality cancer cells, and the cells in the surrounding region of the necrotic cell area can be used as a cancer research model. For example, in the example in Figure 39, cell aggregate A can be evaluated as a high-quality cancer cell aggregate, and time-lapse observation of cell death in the cells surrounding the necrotic cell area is also possible.

[0201] In one embodiment, the cell aggregate is a cell aggregate obtained by culturing stem cells, for example, a cell aggregate obtained by culturing stem cells collected from animals including humans, and for example, a cell aggregate obtained by culturing stem cells collected from humans. Figure 40 shows an example of the manufacturing process for a cell aggregate when the cell aggregate is a cell aggregate obtained by culturing stem cells, and Figure 41 shows an example of a case in which a cell aggregate is evaluated as being of higher quality if the area of ​​necrotic cells within the cell aggregate is small during the evaluation process during the preparation process. Necrosis can occur during the creation and culturing processes of cell aggregates, and when cell aggregates obtained by culturing stem cells are used in regenerative medicine, it is desirable for the area of ​​necrotic cells within the cell aggregate to be small. That is, among cell aggregates obtained by culturing stem cells, cell aggregates with a small area of ​​necrotic cells within the cell aggregate can be evaluated as high quality, and for example, in the example of Figure 41, cell aggregate A can be considered a high-quality cell aggregate and used in regenerative medicine. An example of using cell aggregates obtained by culturing stem cells taken from humans in regenerative medicine is when stem cells are extracted from fat cells taken from a subject, cultured, and the resulting cell aggregate is transplanted into the same subject.

[0202] The present disclosure will be described in detail below with reference to examples, etc., but the present disclosure is not limited to these examples. [Examples]

[0203] [Example 1: Acquisition of refractive index distribution data of HepG2 cells that have undergone necrosis by freeze-thaw treatment] To clarify the characteristics of necrotic cells, we obtained refractive index distribution data from HepG2 cells that had been necrotic by freeze-thaw treatment.

[0204] Human liver cancer-derived HepG2 cells, attached to a T25 flask, were suspended in 1 mL of Trypsin-EDTA (0.25%) and phenol red (Gibco). 5 mL of DMEM medium containing 10% fetal bovine blood was then added to inactivate trypsin. The cells were then centrifuged at 1000 rpm for 3 minutes, the supernatant was discarded, and 5 mL of DMEM medium containing 10% fetal bovine blood was added to the cell pellet to suspend the cells. The solution containing the suspended HepG2 cells was frozen at -80°C for 17.5 hours, and then thawed at room temperature for 2.5 hours (freeze-thaw cycle).

[0205] Refractive index (RF) distribution data was acquired for suspension-type necrotic HepG2 cells (with freeze-thaw) and suspension-type HepG2 cells (without freeze-thaw) obtained by the same method except for the absence of a freeze-thaw process, using observation apparatus 1A and ODT as shown in refractive index distribution measurement method A2. A 60x objective lens was used. Figure 42 shows representative refractive index tomography data extracted from the refractive index distribution data obtained under conditions with and without freeze-thaw. In the figure, the left column shows the extracted refractive index tomography data, and the right column shows the refractive index tomography data enclosed in white lines, representing the area used for calculating the mean, median, and standard deviation, either for the entire cell (hereinafter also referred to as the whole) or the nucleus. Furthermore, Table 1 shows the mean, median, and standard deviation of the refractive index of the whole and the nucleus under conditions with and without freeze-thaw, calculated based on the refractive index tomography data in Figure 42.

[0206] [Table 1]

[0207] According to Figure 42 and Table 1, in cells in which necrosis was induced by freeze-thaw cycles, a morphology suggesting leakage of contents due to cell membrane collapse was observed, and the overall refractive index was lower compared to cells without freeze-thaw cycles. On the other hand, under conditions with freeze-thaw cycles, refractive index tomography data showed a roughly circular region within the cell with a higher average refractive index than the overall cell, surrounded by a high-refractive-index nuclear membrane, and containing high-refractive-index nucleoli and low-refractive-index chromatin regions. In other words, cells that underwent necrosis did not exhibit the nuclear fragmentation seen in apoptotic cells and retained nuclear characteristics. Furthermore, under conditions with freeze-thaw cycles, the nucleoli within the nucleus had a high refractive index while the chromatin regions had a low refractive index, resulting in a larger standard deviation of nuclear refractive index in cells that underwent necrosis compared to cells without freeze-thaw cycles.

[0208] [Example 2: Acquisition of refractive index distribution data of A549 cell aggregates that underwent necrosis by being left at room temperature] To confirm whether the characteristics of necrotic cells revealed in Example 1 also apply to cell aggregates, refractive index distribution data was obtained from A549 cell aggregates that had undergone necrosis by being left at room temperature.

[0209] A549 cells, a cell line derived from human lung cancer, were placed in an EZSPHERE 6-well plate (manufactured by AGC Technoglass) at a rate of 0.75 x 10⁶ cells per well. 5 Cells were seeded at a specified density and cultured for 3 days in DMEM medium containing 10% fetal bovine blood and MEM Non-Essential Amino Acids Solution (100×) (Gibco) at a final concentration of 1×, under conditions of 5% CO2, 37°C, and 100% humidity, to form cell aggregates. The formed cell aggregates were then placed in a sealed 15 mL conical centrifuge tube containing 2 mL of the aforementioned medium, without CO2 or humidity control, and left overnight at room temperature to induce necrosis.

[0210] For the obtained cell aggregates, refractive index distribution data was acquired using observation device 1A and ODT as shown in refractive index distribution measurement method C. Representative refractive index tomography data extracted from the obtained refractive index distribution data is shown in Figure 43. In the figure, the left column shows the extracted refractive index tomography data, and the right column shows the refractive index tomography data enclosed in white lines, representing the region used for calculating the mean, median, and standard deviation of the entire cell or nucleus that was the target of analysis. Furthermore, Table 2 shows the mean, median, and standard deviation of the refractive index of the entire cell and the nucleus, calculated based on the refractive index tomography data in Figure 43.

[0211] [Table 2]

[0212] According to Figure 43 and Table 2, the necrotic cells in the cell aggregate left at room temperature had a low overall average refractive index (1.339), similar to that of the freeze-thaw condition in Example 1 (1.344), and a high standard deviation of the nuclear refractive index (0.007), similar to that of the freeze-thaw condition in Example 1 (0.007). This clearly shows that the characteristics of necrotic cells observed in single cells in Example 1 also apply to cell aggregates left at room temperature.

[0213] [Example 3: Acquisition of refractive index distribution data of HepG2 cell aggregates that have undergone necrosis by high-concentration ethanol treatment] To confirm whether the characteristics of necrotic cells also apply to cells in a cell aggregate where necrosis was induced using a different method than in Example 2, refractive index distribution data was obtained for HepG2 cell aggregates that had been necrotized by high-concentration ethanol treatment.

[0214] HepG2 cells, a cell line derived from human liver cancer, were placed in an EZSPHERE 6-well plate (AGC Technoglass Co., Ltd.) at a rate of 1 to 1.5 x 10⁶ cells per well. 5Cells were seeded at a specific density and cultured in DMEM medium containing 10% fetal bovine blood under conditions of 5% CO2, 37°C, and 100% humidity for 6 days to form cell aggregates. Ethanol (manufactured by Fujifilm Wako Pure Chemical Industries, Ltd.) was then added to the aggregates to a final concentration of 10% by volume (1.7 mol / L), and the mixture was allowed to stand for 30 minutes under conditions of 5% CO2, 37°C, and 100% humidity to induce necrosis.

[0215] For the obtained cell aggregates, refractive index distribution data was acquired using observation device 1A and ODT as shown in refractive index distribution measurement method C. Representative refractive index tomography data extracted from the obtained refractive index distribution data is shown in Figure 44. In the figure, the left column shows the extracted refractive index tomography data, and the right column shows the refractive index tomography data enclosed in white lines, representing the region used for calculating the mean, median, and standard deviation of the entire cell or nucleus that was the target of analysis. Furthermore, Table 3 shows the mean, median, and standard deviation of the refractive index of the entire cell and the nucleus, calculated based on the refractive index tomography data in Figure 44.

[0216] [Table 3]

[0217] According to Figure 44 and Table 3, the necrotic cells in the cell aggregate treated with high-concentration ethanol had a low overall average refractive index (1.341), similar to the cells in the cell aggregate treated at room temperature in Example 2 (1.339), and a high standard deviation of the nuclear refractive index (0.008), similar to the cells in the cell aggregate treated at room temperature in Example 2 (0.007). Therefore, it became clear that the characteristics of necrotic cells observed in the cell aggregate treated at room temperature in Example 2 also apply to the cell aggregate treated with high-concentration ethanol.

[0218] [Example 4: Acquisition of refractive index distribution data of A549 cell aggregates that underwent necrosis by high-concentration ethanol treatment] To confirm whether the characteristics of cells that underwent necrosis induced by high-concentration ethanol treatment apply to cells in cell aggregates other than HepG2 cell aggregates, refractive index distribution data was obtained for A549 cell aggregates that underwent necrosis induced by high-concentration ethanol treatment.

[0219] A549 cells, a cell line derived from human lung cancer, were placed in an EZSPHERE 6-well plate (manufactured by AGC Technoglass) at a rate of 1 x 10⁶ cells per well. 5 Cells were seeded at a specific density and cultured for 4 days in MEM medium containing 10% fetal bovine blood and MEM Non-Essential Amino Acids Solution (100×) (Gibco) at a final concentration of 1× under conditions of 5% CO2, 37°C, and 100% humidity to form cell aggregates. Ethanol (Fujifilm Wako Pure Chemical Industries, Ltd.) was then added to a final concentration of 10% by volume (1.7 mol / L), and the mixture was allowed to stand for 10 minutes under conditions of 5% CO2, 37°C, and 100% humidity to induce necrosis.

[0220] For the obtained cell aggregates, refractive index distribution data was acquired using observation device 1A and ODT as shown in refractive index distribution measurement method C. Representative refractive index tomography data extracted from the obtained refractive index distribution data is shown in Figure 45. In the figure, the left column shows the extracted refractive index tomography data, and the right column shows the refractive index tomography data enclosed in white lines, representing the region used for calculating the mean, median, and standard deviation of the whole or nucleus of the cells analyzed. Furthermore, Table 4 shows the mean, median, and standard deviation of the refractive index of the whole and nucleus, calculated based on the refractive index tomography data in Figure 45.

[0221] [Table 4]

[0222] As shown in Figure 45 and Table 4, cells in A549 cell aggregates treated with high-concentration ethanol also exhibited necrotic cell characteristics similar to those of cells in HepG2 cell aggregates treated with high-concentration ethanol.

[0223] [Example 5: Acquisition of refractive index distribution data of HepG2 cell aggregates containing cells that underwent necrosis under high-density cell conditions within the cell aggregate] To investigate whether it is possible to determine the region of necrotic cells in any cross-section of a cell aggregate containing a mixture of living and dead cells, we obtained refractive index distribution data of HepG2 cell aggregates containing necrotic cells under high-density cell conditions within the cell aggregate, and examined whether it is possible to determine the region of necrotic cells based on the characteristics of the necrotic cells.

[0224] HepG2 cells, a cell line derived from human liver cancer, were placed in EZSPHERE 6-well plates (AGC Technoglass Co., Ltd.) at a rate of 0.75 to 2 x 10⁶ cells per well. 5 Cells were seeded at a specific density and cultured for 4 days in DMEM medium containing 10% fetal bovine blood under conditions of 5% CO2, 37°C, and 100% humidity to form cell aggregates.

[0225] For the obtained cell aggregates, refractive index distribution data was acquired using observation device 1A and ODT as shown in refractive index distribution measurement method C. Representative refractive index tomography data extracted from the obtained refractive index distribution data is shown in Figure 46. In the figure, the left column shows the extracted refractive index tomography data, and the right column shows the refractive index tomography data enclosed in white lines, representing the region used for calculating the mean, median, and standard deviation of the entire cell or nucleus that was the target of analysis. Furthermore, Table 5 shows the mean, median, and standard deviation of the refractive index of the entire cell and the nucleus, calculated based on the refractive index tomography data in Figure 46.

[0226] [Table 5]

[0227] Based on the results in Figure 46 and Table 5, we investigated whether the region of a necrotic cell could be determined based on the characteristics of necrotic cells, specifically the presence of a region corresponding to the nucleus and the average refractive index of the entire cell being below a threshold, following the flow chart shown in Figure 47. Although cells 1-4 in Figure 46 all had a region corresponding to the nucleus, only cell 1 had a low overall refractive index (1.345). By setting the threshold for the average overall refractive index to 1.350, we were able to determine that cell 1, with an overall refractive index lower than the threshold, was a dead cell region (the region of a necrotic cell).

[0228] Furthermore, regarding the results in Figure 46 and Table 5, we investigated whether the region of a necrotic cell could be determined based on the presence of necrotic cell characteristics, specifically the characteristics of necrotic cells being the presence of a region corresponding to the nucleus and the statistical variability of the nuclear refractive index being above a threshold, following the flow chart shown in Figure 48. Although cells 1-4 in Figure 46 all had a region corresponding to the nucleus, only cell 1 had a high standard deviation of the nuclear refractive index (0.007). By setting the threshold for the nuclear standard deviation to 0.006, we were able to determine that cell 1, with a nuclear refractive index higher than the threshold, was a dead cell region (the region of a necrotic cell). [Explanation of symbols]

[0229] 1A~1J…Observation device, 2…Recording medium, 11…Light source, 12…Lens, 13…Indicating light end, 14…Optical fiber, 15…Fiber coupler, 16,17…Optical fiber, 18,19…Outdicating light end, 21…Lens, 22…Mirror, 23…Lens, 24…Condenser lens, 25…Objective lens, 31…Lens, 32…Mirror, 33…Drive unit, 34…Lens, 41…Beam splitter, 42…Lens, 43…Imaging unit, 44…Mirror, 50…Analysis unit, 51…Interference intensity image acquisition unit, 52…First complex amplitude image generation unit, 53…Second complex amplitude image generation unit, 54…2D phase image generation unit, 55…3D phase image generation unit 56...Refractive index distribution calculation unit, 57...Display unit, 58...Storage unit, 60...Analysis unit, 61...Interference intensity image acquisition unit, 62...First complex amplitude image generation unit, 63...Second complex amplitude image generation unit, 64...Phase conjugate calculation unit, 65...2D phase image generation unit, 66...3D phase image generation unit, 67...Refractive index distribution calculation unit, 68...Display unit, 69...Storage unit, 70...Analysis unit, 71...Interference intensity image acquisition unit, 72...First complex amplitude image generation unit, 73...Second complex amplitude image generation unit, 74...2D phase image generation unit, 75...3D phase image generation unit, 76...Refractive index distribution calculation unit, 77...Third complex amplitude image generation unit, 78...Display unit, 79...Storage unit.

Claims

1. A method for determining the region of cells that have undergone necrosis in an object based on refractive index distribution data of the object, The object being observed is a cell mass, The region of cells exhibiting necrosis in the observed object is the region of cells exhibiting necrosis within the cell mass. The determination of the region of cells that have undergone necrosis is based on the fact that the region of each individual cell included in the refractive index distribution data has the characteristics of a necrotic cell, and the characteristics of the necrotic cell include one or more characteristics selected from the group consisting of a characteristic having a region corresponding to the nucleus, and a characteristic in which the statistical value of the refractive index of the entire cell is below a threshold, and a characteristic in which the statistical variation of the refractive index of the nucleus is above a threshold. A method wherein the region corresponding to the nucleus is a substantially circular or substantially spherical region, and the statistical value of its refractive index is greater than the statistical value of the refractive index of the entire cell.

2. The method according to claim 1, wherein the statistical value is the mean or median.

3. The method according to claim 1, wherein the refractive index distribution data is refractive index tomographic data in a predetermined direction.

4. The method according to claim 1, wherein the determination of the region of cells that have undergone necrosis is performed by inputting the refractive index distribution data of an object into a learning model that has been trained using training data including necrotic region data of a reference object and refractive index distribution data of a reference object corresponding to the necrotic region data, wherein the feature quantities used by the learning model include feature quantities that correspond to features having a region corresponding to a nucleus.

5. A method for determining a region of cells that have undergone necrosis in an object to be observed, comprising the steps of: acquiring refractive index distribution data of the object to be observed; and determining a region of cells that have undergone necrosis by the method according to any one of claims 1 to 4.

6. A method for evaluating the quality of a cell aggregate, comprising the steps of: obtaining refractive index distribution data of the cell aggregate; and determining a region of cells that have undergone necrosis within the cell aggregate by the method according to any one of claims 1 to 4.

7. A device for determining a necrotic cell region, comprising: a data acquisition unit for acquiring refractive index distribution data of an object to be observed; and a determination unit for determining a region of necrotic cells in an object to be observed by the method described in any one of claims 1 to 4.

8. An information processing program that causes a computer to perform a decision step of determining the region of cells that have undergone necrosis in an object to be observed, by the method described in any one of claims 1 to 4.

9. An information processing program that causes a computer to perform a data acquisition step of acquiring refractive index distribution data of an object to be observed, and a determination step of determining the region of cells that have undergone necrosis in the object to be observed by the method described in any one of claims 1 to 4.

Images