Image analysis device, image analysis method, and program, and light field quantum measurement system, and light field quantum measurement method

Abstract

To measure a physical quantity in a three-dimensional space in a sample at high speed with high spatial resolution, by using a quantum sensor and a light field optical system.SOLUTION: An image analysis device 30 analyzes a light field image of a sample including at least one quantum sensor and picked up by a light field optical system in a state in which the sample is irradiated with excitation light and microwaves. The image analysis device comprises: an image reconstruction unit 32 that reconstructs images refocused to each of depth positions from the light field image of the sample; a three-dimensional reconstruction unit 33 that reconstructs a volumetric image of the sample by stacking the reconstructed images at each of the depth positions; a quantum sensor position identification unit 34 that determines the three-dimensional spatial coordinates of each of the quantum sensors in the volumetric image of the sample; a fluorescence intensity identification unit 35 that determines the fluorescence intensity of each of the quantum sensors from a fluorescence beam originating from the same quantum sensor captured in the light field image of the sample; and a physical quantity calculation unit 36 that calculates a physical quantity at the three-dimensional spatial coordinates of each of the quantum sensors from the fluorescence intensity of each of the quantum sensors.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image analysis device, an image analysis method and program, as well as a light field quantum measurement system and a light field quantum measurement method, and in particular to a quantum measurement technology that analyzes a light field image of a sample containing a quantum sensor that fluoresces under specific conditions to measure physical quantities in a three-dimensional space within the sample. [Background technology]

[0002] To understand phenomena such as disease, cell aging, cancer, and cell responses to radiation, we need sensors that respond sensitively to minute changes in cells and molecules. For example, protein-type, polymer-type, and fluorescent small molecule-type temperature measurement probes have been developed to measure intracellular temperature. Such bioimaging using fluorescence has become an essential technique for analyzing the localization and function of molecules in cells and living organisms.

[0003] To accurately understand very small changes in living organisms caused by life phenomena, ultra-sensitive sensors that respond sensitively to slight changes in cells and molecules are required. One such sensor is a quantum sensor that uses quantum effects to measure physical quantities such as magnetism, magnetic field, and temperature. Among them, fluorescent nanodiamonds (FNDs) have attracted attention as highly sensitive fluorescent labeling agents in cell imaging and trace virus detection. In order to measure molecules in cells using a quantum sensor, the size of the quantum sensor needs to be several nm, which is the molecular size, because a size of several tens of nm is too large. The inventors of the present invention have succeeded in creating the world's smallest FNDs of several nm in size, and have overcome the drawback that background light such as autofluorescence and fluorescence of impurities interferes with the fluorescence detection of quantum sensors in fluorescence imaging using FNDs, thereby improving the signal-to-background ratio (SBR value) (see, for example, Non-Patent Document 1).

[0004] On the other hand, high-speed, large-scale three-dimensional imaging is required to observe multiple biomolecules and intracellular behaviors at once in a delicate and dynamic manner using fluorescence imaging. Generally, such three-dimensional images are generated by using a confocal microscope to continuously change the focal position of the sample at a specific pitch in the depth direction (Z direction) and capture the image of the sample at each focal position. In addition, there is a spinning disk confocal microscope that enables three-dimensional imaging. In the spinning disk confocal method, a laser beam is irradiated onto a rotating disk with many pinholes to create multiple parallel beams of light, which are then used to scan the sample at high speed to form a confocal image. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Tamami Yanagi, Kiichi Kaminaga, Michiyo Suzuki, Hiroshi Abe, Hiroki Yamamoto, Takeshi Ohshima, Akihiro Kuwahata, Masaki Sekino, Tatsuhiko Imaoka, Shizuko Kakinuma, Takuma Sugi, Wataru Kada, Osamu Hanaizumi, and Ryuji Igarashi, “All-Optical Wide-Field Selective Imaging of Fluorescent Nanodiamonds in Cells, In Vivo and Ex Vivo,” ACS Nano 2021 15 (8), 12869-12879, August 2, 2021 Summary of the Invention [Problem to be solved by the invention]

[0006] Three-dimensional imaging using a confocal microscope requires scanning in the depth direction, which takes time for measurement, and is not suitable for observing high-speed events. For example, when imaging a three-dimensional space by scanning 30 times in the Z direction at 30 fps, the imaging speed for the entire three-dimensional space is 30 fps / 30 = 1 vps, and it takes one second to capture the three-dimensional space. In contrast, if the thermal diffusivity inside a cell is 1.5 × 10 -7 m 2 When the time required for heat to pass three-dimensionally through a cell with a diameter of 10 μm is assumed to be 0.5 seconds per second, it is difficult to track the thermal diffusion within the cell at the speed of three-dimensional imaging using a confocal microscope.

[0007] Another option for 3D imaging is light field microscopy (LFM). A light field (light ray space) refers to a collection of light rays in a 3D space. LFM uses a microlens array, in which many microlenses are arranged two-dimensionally, at an intermediate image plane, and records the light field by obtaining the position of the light rays passing on the aperture stop of the objective lens and the position of the light rays being acquired on the image sensor. From the light field image recorded by LFM, subaperture images equivalent to a small diameter lens can be obtained. Subaperture images are partial images of a subject captured by shifting the viewpoint, and each image has a deep depth of field due to the pinhole effect, and at the same time, they have parallax between other subaperture images. Therefore, by shifting and superimposing the subaperture images, it is possible to reconstruct an image refocused at any depth position. LFM is suitable for observing high-speed events because it can capture 3D images at high speed in a single shot without scanning in the depth direction.

[0008] Therefore, an object of the present invention is to measure physical quantities in three-dimensional space within a sample at high speed and with high spatial resolution using a quantum sensor and a light field optical system. [Means for solving the problem]

[0009] An image analysis device according to a first aspect of the present invention is an image analysis device that analyzes a light field image of a sample that includes at least one quantum sensor, captured by a light field optical system while the sample is irradiated with excitation light of a wavelength that excites the quantum sensor to fluoresce and microwaves of a frequency that causes the quantum sensor to undergo optical detection magnetic resonance, and is equipped with an image reconstruction unit that reconstructs images refocused at each depth position from the light field image of the sample, a three-dimensional reconstruction unit that stacks the reconstructed images at each depth position to reconstruct a volumetric image of the sample, a quantum sensor position identification unit that determines the three-dimensional spatial coordinates of each quantum sensor in the volumetric image of the sample, a fluorescence intensity identification unit that determines the fluorescence intensity of each quantum sensor from fluorescent light originating from the same quantum sensor captured in the light field image of the sample, and a physical quantity calculation unit that calculates the physical quantity of each quantum sensor in the three-dimensional spatial coordinates from the fluorescence intensity of each quantum sensor.

[0010] An image analysis device according to a second aspect of the present invention is the image analysis device according to the first aspect, wherein the light field image of the sample is obtained by continuously imaging the sample by sweeping the microwave frequency around the resonance frequency of the photodetection magnetic resonance of the quantum sensor, and the physical quantity calculation unit identifies the resonance frequency of the photodetection magnetic resonance of each quantum sensor from the change pattern of the fluorescence intensity of each quantum sensor, and calculates the physical quantity corresponding to the identified resonance frequency.

[0011] An image analysis device according to a third aspect of the present invention is an image analysis device according to the second aspect, in which the light field image of the sample is obtained by continuously imaging the sample while further turning the microwave on and off at a constant cycle, and the fluorescence intensity determination unit determines, as the fluorescence intensity of each quantum sensor, the relative value of the fluorescence intensity of each quantum sensor captured in the light field image when the microwave is on and off, which are successive in time series.

[0012] An image analysis device according to a fourth aspect of the present invention is the image analysis device according to the second aspect, wherein the light field image of the sample is obtained by continuously imaging the sample while further turning the microwave on and off at a constant cycle, and the image analysis device further includes a fluorescent light ray identification unit that identifies light rays in the light field image that become bright and dark with the on / off cycle of the microwave as fluorescent light rays of a quantum sensor, the image reconstruction unit reconstructs images refocused at each depth position for the identified fluorescent light rays, and the fluorescence intensity identification unit determines the intensity of the identified fluorescent light rays.

[0013] An image analysis device according to a fifth aspect of the present invention is an image analysis device according to any one of the first to fourth aspects, wherein the quantum sensor is a fluorescent nanodiamond.

[0014] An image analyzing device according to a sixth aspect of the present invention is the image analyzing device according to any one of the first to fifth aspects, wherein the physical quantity is temperature.

[0015] An image analysis method according to aspect 7 of the present invention is an image analysis method for analyzing a light field image of at least one quantum sensor captured by a light field optical system while a sample containing the quantum sensor is irradiated with excitation light of a wavelength that excites the quantum sensor to fluoresce and microwaves of a frequency that causes the quantum sensor to undergo optical detection magnetic resonance, and includes the steps of an image reconstruction unit reconstructing images refocused at each depth position from the light field image of the sample, a three-dimensional reconstruction unit stacking the reconstructed images at each depth position to reconstruct a volumetric image of the sample, a quantum sensor position identification unit determining three-dimensional spatial coordinates of each quantum sensor from the volumetric image of the sample, a fluorescence intensity identification unit determining the fluorescence intensity of each quantum sensor from the fluorescent light beam originating from the same quantum sensor captured in the light field image of the sample, and a physical quantity calculation unit calculating the physical quantity of each quantum sensor in the three-dimensional spatial coordinates from the fluorescence intensity of each quantum sensor.

[0016] A program according to aspect 8 of the present invention is a program that causes a computer to analyze a light field image of a sample containing at least one quantum sensor, captured by a light field optical system, while the sample is irradiated with excitation light of a wavelength that excites the quantum sensor to fluoresce and microwaves of a frequency that causes the quantum sensor to undergo optical detection magnetic resonance, and causes the computer to function as an image reconstruction means that reconstructs images refocused at each depth position from the light field image of the sample, a three-dimensional reconstruction means that stacks the reconstructed images at each depth position to reconstruct a volumetric image of the sample, a quantum sensor position identification means that determines the three-dimensional spatial coordinates of each quantum sensor from the volumetric image of the sample, a fluorescence intensity identification means that determines the fluorescence intensity of each quantum sensor from fluorescent light rays originating from the same quantum sensor captured in the light field image of the sample, and a physical quantity calculation means that calculates the physical quantity of each quantum sensor in the three-dimensional spatial coordinates from the fluorescence intensity of each quantum sensor.

[0017] A light field quantum measurement system according to aspect 9 of the present invention is a system for measuring a physical quantity using a quantum sensor, and includes an excitation light generator that generates excitation light of a wavelength that excites the quantum sensor to fluoresce, a microwave generator that generates microwaves of a frequency that causes the quantum sensor to undergo optical detection magnetic resonance, a light field optical system that captures a light field image of a subject, and an image analysis device according to any one of aspects 1 to 6 that analyzes the light field image of the sample captured by the light field optical system while a sample including at least one of the quantum sensors is irradiated with the excitation light and the microwaves.

[0018] A light field quantum measurement method according to aspect 10 of the present invention is a method for measuring a physical quantity using a quantum sensor, comprising the steps of: a light field optical system capturing a light field image of a sample containing at least one quantum sensor while irradiating the sample with excitation light of a wavelength that excites the quantum sensor to fluoresce and microwaves of a frequency that causes the quantum sensor to optically detect magnetic resonance; and an image analysis device according to any one of aspects 1 to 6 analyzing the light field image of the sample. Effect of the Invention

[0019] According to the present invention, by using a quantum sensor, which is an extremely small sensor, it is possible to measure physical quantities in three-dimensional space within a sample with high spatial resolution. Furthermore, the three-dimensional spatial coordinates of multiple quantum sensors within the sample can be identified from a single shot of a light field image of the sample. This eliminates the need to repeatedly scan at different depths to image the three-dimensional space, as is the case with a confocal microscope, and makes it possible to quickly measure physical quantities in three-dimensional space within a sample. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram of a light field quantum metrology system according to an embodiment of the present invention. [Diagram 2] 2 is a schematic diagram of a light field microscope, a storage device, and an image analysis device in the light field quantum metrology system of FIG. 1. [Diagram 3] 1 is a flowchart illustrating an example of a method for acquiring a light field image of a sample. [Figure 4] 1 is a flowchart illustrating an example of a method for analyzing a light field image of a sample. [Diagram 5] FIG. 1 is a schematic diagram showing the difference in intensity fluctuation between quantum sensor fluorescence captured in a light field image of a sample and other light. [Figure 6] 1A to 1C are diagrams illustrating an example of (a) a light field image, (b) a group of elemental images, and (c) a reconstructed image. [Figure 7]FIG. 2 is a diagram illustrating a geometric relationship between parameters of a light field microscope according to an example. [Figure 8] 1A and 1B are diagrams for explaining the relationship between a PQ array and superimposition of elemental images. [Figure 9] 1A to 1C are diagrams illustrating the reconstruction of images refocused at each depth position from a light field image, and further the reconstruction of a volumetric image. [Figure 10] 1 is a graph illustrating the temperature dependence of the ODMR spectrum of a fluorescent nanodiamond, which is an example of a quantum sensor. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the inventor provides the accompanying drawings and the following explanation so that those skilled in the art can fully understand the present invention, and does not intend to limit the subject matter described in the claims by them. In addition, the dimensions of each member depicted in the drawings, detailed shapes of details, etc. may differ from the actual ones.

[0022] <Embodiment> 1 is a schematic diagram of a light-field quantum metrology system according to an embodiment of the present invention. The light-field quantum metrology system 100 according to this embodiment captures a light-field image of a sample including at least one quantum sensor, analyzes the image, and measures the three-dimensional spatial coordinates of the quantum sensor in the sample and the physical quantities at those coordinates. Specifically, the light-field quantum metrology system 100 includes, as main components, a light-field microscope 10, a storage device 20, an image analyzer 30, an excitation light generator 40, a microwave generator 50, and a synchronization signal generator 60.

[0023] For example, fluorescent nanodiamonds (FNDs) can be used as quantum sensors. FNDs are artificial diamond particles with diameters ranging from a few nm to a few hundred nm, and one of their characteristics is that they contain nitrogen atoms (Nitrogen) as impurities in the crystal. This nitrogen atom and the vacancy (vacancy) next to it form an NVC (Nitrogen-Vacancy Center), which emits a photon of 637 nm when excited with green light of 532 nm (red fluorescence). The fluorescence of FNDs is highly light-resistant and does not fade or blink, making it suitable for long-term image measurement. Furthermore, NVCs contain negatively charged NVs consisting of a total of six electrons: two from the nitrogen atom, three from the carbon atom, and one in the vacancy. - There exists a cavity in which the electron spin resonates with electromagnetic waves, causing a change in the fluorescence intensity of the FND, an phenomenon known as optically detected magnetic resonance (ODMR). Specifically, when microwaves of around 2.87 GHz are irradiated onto an FND, the fluorescence intensity of the FND weakens due to ODMR activity. Since this resonance frequency is temperature-dependent, it is possible to measure temperature as a physical quantity by measuring the change in the resonance frequency of the optically detected magnetic resonance of the FND. For example, it is possible to measure the temperature of a microscopic region within the biological tissue of the nematode C. elegans by measuring the fluorescence of an FND incorporated into the nematode C. elegans as a sample.

[0024] The excitation light generator 40 is a device that generates excitation light of a wavelength that excites the quantum sensor to fluoresce. When the FND is used as the quantum sensor, the excitation light generator 40 outputs a green laser light of 532 nm.

[0025] The microwave generator 50 is a device that generates microwaves at a frequency that causes the quantum sensor to optically detect magnetic resonance. When the FND is used as a quantum sensor, the microwave generator 50 generates microwaves at about 2.87 GHz, which is the resonance frequency of the electron spin of the NVC in the FND, for example, 2.82 GHz to 2.92 GHz. The microwave generator 50 can also sweep the frequency to generate microwaves, and can output the microwaves in a pulsed form in synchronization with the synchronization signal output from the synchronization signal generator 60.

[0026] The sample is adjusted to a sample holder such as a slide glass or a petri dish and placed on the stage. An objective lens is placed below the stage, and laser light output from an excitation light generator 40 enters the objective lens via a collimator lens, a condenser lens, a dichroic mirror, etc., and the excitation light is irradiated from the tip of the objective lens onto the sample in the sample holder. Meanwhile, a coil is placed directly above the sample holder placed on the stage, and microwaves output from a microwave generator 50 are transmitted through a coaxial cable and irradiated from the coil at the tip onto the sample in the sample holder.

[0027] The synchronization signal generator 60 is a device that generates a synchronization signal for controlling the frequency sweep of the microwave generator 50 and the on / off control of the microwave generation, and for controlling the imaging timing of the light field microscope 10. Specifically, the synchronization signal generator 60 is a function generator.

[0028] The light field microscope 10, the storage device 20, and the image analysis device 30 will be described with reference to another drawing. FIG. 2 is a schematic diagram of the light field microscope 10, the storage device 20, and the image analysis device 30 in the light field quantum measurement device 100 of FIG. 1. The light field microscope 10 is an example of a light field optical system, and includes a main lens 1, a microlens array 2, and an image sensor 3. The main lens 1 may include an objective lens, an imaging lens, an absorption filter, and the like, but for convenience, the main lens 1 includes them. The microlens array 2 is configured by arranging a plurality of microlenses (lenslets) 2a two-dimensionally, and is disposed between the main lens 1 and the image sensor 3. The image sensor 3 is a CCD sensor or a CMOS sensor configured by arranging a plurality of light receiving elements (not shown) two-dimensionally, which perform photoelectric conversion of input light and output an electrical signal. The electrical signal output from each light receiving element is converted into digital data by an A / D converter (not shown), and a light field image is output from the light field microscope 10.

[0029] The storage device 20 is a collection of storage devices such as RAM, ROM, SSD, and HDD. The RAM is mainly used as a working memory when the image analysis device 30 performs image analysis. The ROM, SSD, and HDD mainly store computer programs for operating the image analysis device 30, light field images of samples that are the analysis targets of the image analysis device 30, reconstructed images and volumetric images that are generated and used in the image analysis described below, and the like, temporarily or for a long period of time. The storage device 20 may be located on a cloud.

[0030] The image analysis device 30 analyzes the light field image of the sample stored in the storage device 20 to identify the three-dimensional spatial position of the quantum sensor in the image and calculates the physical quantity at that position. Specifically, the image analysis device 30 includes a fluorescent light beam discrimination unit 31, an image reconstruction unit 32, a three-dimensional reconstruction unit 33, a quantum sensor position identification unit 34, a fluorescent intensity identification unit 35, and a physical quantity calculation unit 36. Each of these components can be realized as hardware such as an ASIC or FPGA, or as software that executes a computer program stored in the storage device 20 or a computer program stored and provided in a computer-readable non-transitory recording medium on a processor (not shown) such as a CPU or GPU. It can also be realized by appropriately combining hardware and software. The image analysis device 30 appropriately accesses various storage devices constituting the storage device 20 during image analysis to write and read data. All or part of the components of the image analysis device 30 may be located on the cloud. The analysis results by the image analysis device 30 are passed to another processing system for further processing, or the results are displayed to the user through a user interface (not shown). The storage device 20 and the image analysis device 30 may be configured as an integrated computer device.

[0031] Next, a method for acquiring a light field image of a sample to be analyzed by the image analysis device 30 will be described. FIG. 3 is a flow chart showing an example of a method for acquiring a light field image of a sample. First, a sample holder containing a sample including at least one quantum sensor is placed on a stage, and the sample is irradiated with excitation light output from the excitation light generator 40 (S1). Next, the oscillation frequency of the microwave generator 50 is set to a start value near the resonance frequency of the photodetection magnetic resonance of the quantum sensor, and a frequency sweep is started (S2). In synchronization with the synchronization signal of the synchronization signal generator 60, the microwave generator 50 turns on the microwave, and the light field microscope 10 captures a light field image of the sample in a state where the excitation light and microwave are irradiated, and stores the image in the storage device 20 (S3). Next, in synchronization with the synchronization signal of the synchronization signal generator 60, the microwave generator 50 turns off the microwave, and the light field microscope 10 captures a light field image of the sample in a state where the microwave is not irradiated but only the excitation light is irradiated, and stores the image in the storage device 20 (S4). Thereafter, the microwave generator 50 sweeps the frequency (S5), and steps S3 and S4 are repeated until the microwave frequency reaches a sweep end value near the resonant frequency of the optically detected magnetic resonance of the quantum sensor.

[0032] For example, when the microwave frequency is swept over 100 points in synchronization with a 400 Hz synchronization signal, 200 light field images are acquired in 0.5 seconds, alternating between light field images of the sample irradiated with excitation light and microwaves and light field images of the sample irradiated with excitation light only.

[0033] The microwave frequency may be swept so that it monotonically increases or decreases, and the frequency sweep step does not have to be a constant value. Frequency hopping may be used instead of frequency sweep. It is sufficient to obtain multiple light field images of the sample when irradiated with microwaves of multiple different frequencies within a certain frequency range.

[0034] In addition, depending on the performance and frame rate of the light field microscope 10, the fluorescent light of the quantum sensor captured in one light field image may be extremely weak. In such a case, the pixel values ​​of the fluorescent light of the quantum sensor may be integrated to make the light intensity a sufficiently large value, and then various analysis processes described below may be performed. Therefore, the above steps S2 to S5 may be repeated multiple times to obtain more sets of light field images so that the light intensity can be integrated in the image analysis process.

[0035] Next, a method for analyzing the acquired light field image of a sample by the image analysis device 30 will be described. Fig. 4 is a flow chart showing an example of the method for analyzing the light field image of a sample. When the image analysis device 30 reads the continuously captured light field images of the sample from the storage device 20 (S11), the fluorescent light beam identification unit 31 identifies the fluorescent light beam of the quantum sensor in the read light field image (S12).

[0036] FIG. 5 is a schematic diagram showing the difference in the intensity fluctuation of the quantum sensor fluorescence and other light captured in a light field image of a sample. As described above, when a microwave having a frequency near the resonant frequency of the optical detection magnetic resonance is irradiated on the quantum sensor, the fluorescence intensity becomes weak. Therefore, in a light field image obtained by continuously imaging a sample while turning on and off the microwave in synchronization with a constant periodic synchronization signal output from the synchronization signal generator 60, the fluorescence intensity of the quantum sensor changes microscopically in accordance with the on / off period of the microwave, while changing macroscopically in accordance with the frequency sweep of the microwave. On the other hand, the intensity of background light such as the autofluorescence emitted by the sample itself and the fluorescence of impurities changes independently of the on / off period of the microwave. Due to the difference in the characteristics of such intensity changes, the fluorescence of the quantum sensor can be distinguished from other light in the light field image of the sample that is continuously imaged. Specifically, the fluorescent light beam identification unit 31 identifies the light beam that changes in accordance with the on / off period of the microwave in the light field image of the sample as the fluorescent light beam of the quantum sensor. This enables the quantum sensor's fluorescence to be efficiently extracted from light field images that contain background light, improving the signal-to-background ratio (SBR) value.

[0037] Returning to FIG. 4, the image reconstruction unit 32 selects an arbitrary light field image from the light field images of the sample captured continuously, and reconstructs images refocused at each depth position from the selected light field image (S13). The reconstructed images are temporarily stored in the storage device 20. Note that the image reconstruction unit 32 does not need to reconstruct images for all light rays captured in the light field image of the sample, but only needs to reconstruct images refocused at each depth position for light rays identified as fluorescent light rays of the quantum sensor by the fluorescent light ray identification unit 31 in step S12. This makes it possible to reconstruct refocused images and perform three-dimensional reconstruction, which will be described later, only for the fluorescent light rays of the quantum sensor, thereby reducing the image processing load and improving the analysis speed.

[0038] Here, the characteristics of light field images and the principle of refocusing will be described. A light field image is a collection of sub-images such as multiple elemental images or microlens images captured of the same subject from multiple viewpoints that are shifted from each other. A refocused image at any depth position can be reconstructed by utilizing the parallax of the sub-images included in the light field image. FIG. 6 is a diagram showing a light field image, elemental images, and a reconstructed image according to an example. (a) The light field image is captured by placing a sheet with the characters " / 5" at a position shifted from the focal plane of the main lens 1. The light field image is a group of microlens images in which a large number of microlens images MI captured by each microlens 2a of the microlens array 2 are arranged in a square lattice shape. As shown by the dashed line, the shape of each microlens image MI is approximately circular, reflecting the circular shape of each microlens 2a. In this way, a light field image is a collection of multiple sub-images (microlens images MI in the example of FIG. 6) captured of the same subject from multiple viewpoints that are shifted from each other.

[0039] The area near the boundary of the microlens image MI is dark due to the small amount of light, and the central part of the microlens image MI is used because the area is significantly distorted due to the light rays that pass through the edge of the microlens 2a. That is, from each of the roughly circular microlens images MI, for example, a square area inscribed in the circle is cut out as an element image EI, and the element images EI are arranged in a square lattice to form the (b) element image group.

[0040] As described above, since the sheet with the letters " / 5" written on it is positioned away from the focal plane of the main lens 1, (a) in the light field image, microlens images are recorded in which the viewpoint is shifted by each microlens 2a to partially capture the letters " / 5," that is, microlens images with parallax are recorded. For example, if one focuses on the upper left corner of the letter "5," the light beam emitted from that part of the subject is recorded with a distance Δ between adjacent microlens images. As shown in (b) in the elemental image group, the distance Δ between the microlens images is Δ between the elemental images.EI Here, the pitch of the microlenses 2a is MLP, and the length of one side of the element image is EI L Then, Δ EI is expressed by the following equation (1). Δ EI =Δ-(MLP-EI L ) …(1)

[0041] The reconstructed image (c) is obtained by superimposing adjacent elemental images in the elemental image group while shifting them by an amount corresponding to the parallax. In the example of FIG. 6, adjacent elemental images are shifted by Δ EI By shifting the elemental images by an appropriate amount and superimposing them, a reconstructed image (c) is obtained, refocused at the position of the sheet with the letters " / 5" written on it. In this way, by superimposing the elemental images while appropriately changing the shift amount, it is possible to reconstruct an image refocused at any depth position based on ray tracing in the reverse direction from the acquired image.

[0042] Here, the depth of the object to be refocused, which is the deviation from the object-side focal plane NOP (Native Object Plane) of the objective lens in the main lens 1, is d obj Then, the distance Δ between adjacent microlens images of light rays originating from the same point and the depth d of that point are obj The relationship between the parameters of the light field microscope 10 and the depth d can be expressed by using the parameters of the light field microscope 10. FIG. 7 is a diagram showing a schematic diagram of the geometric relationship between the parameters of the light field microscope 10 according to an example. In the diagram, O represents the depth d obj A is the image point of the object O formed by the main lens 1, B and C are the centers of the adjacent microlenses 2a, and B' and C' are the positions of the light rays from the object O captured by the image sensor 3 via B and C. When we look at the triangles ABC and AB'C' in the figure, from the similarity of the triangles, BC:B'C'=CA:C'A This can be expressed as parameters as follows: MLP: Δ = (a + d obj M 2 ):(a+d obj M 2 -b) Solving this for Δ gives us the following equation (2). Δ=MLP(1-b / (d obj M 2 -a)) …(2) where MLP is the pitch of the microlenses 2a, M is the magnification of the objective lens of the main lens 1, a is the distance from the image-side focal plane T-NIP (Native Image Plane) of the imaging lens of the main lens 1 to the center of the microlens 2a, and b is the distance from the center of the microlens 2a to the image sensor 3.

[0043] In addition, since there is no anisotropy in two mutually perpendicular directions (X direction, Y direction) in a plane perpendicular to the depth direction (Z direction), the same Δ can be applied to both the X direction and the Y direction.

[0044] From equations (1) and (2), obj and Δ EI Therefore, at any depth position (depth d obj ), the elemental images are shifted by the amount of shift Δ EI The elemental images can be easily overlapped by referring to the PQ array described below.

[0045] FIG. 8 is a diagram for explaining the relationship between the PQ array and the overlapping of element images. Overlapping pixels are shown in a shaded manner, the left column shows the overlapping state of the element images, and the right column shows the overlapping state of the element images expanded. For convenience, the element image group is composed of four element images D, E, F, and G, two in a row and two in a row, and the size of each element image is assumed to be five pixels in length and width. The element images are arranged such that element image E is arranged to the right of element image D, element image F is arranged below element image D, and element image G is arranged below element image E, i.e., to the right of element image F. In order to refer to each pixel in the element image group, consecutive numbers starting from 1 are assigned to the rows and columns. For example, pixels belonging to element image D are referred to in the range from the first row to the fifth row and the first column to the fifth column, and pixels belonging to element image G are referred to in the range from the sixth row to the tenth row and the sixth column to the tenth column. In the following, pixel [m, n] refers to the pixel located in the mth row and the nth column in the element image group.

[0046] Z in the figure is d obj For convenience, d obj and the shift amount of the element image Δ EI are expressed by the same Z value. That is, when Z=n, d obj = n, and the shift amount of the element image Δ EI represents n pixels.

[0047] When Z=0, that is, at depth d obj If =0, the elemental images are not superimposed on each other, and the elemental images become the reconstructed image as they are.

[0048] When Z=1, that is, at depth d obj To reconstruct an image refocused to Z=1, adjacent elemental images are shifted by one pixel and then superimposed. Which pixels in the elemental image group should be superimposed can be easily determined by referring to the PQ array, which shows how the pixels are superimposed. The PQ array is a representation of the row or column numbers of pixels in the elemental image group folded over on an elemental image basis, and as shown in the figure, for example, the PQ array corresponding to Z=1 is expressed as follows: 1 2 3 4 5 6 7 8 9 10 This PQ array means that the pixel in the fifth row (fifth column) and the pixel in the sixth row (sixth column) overlap. By preparing such a PQ array for each Z value in advance and storing it in the storage device 20, it is possible to perform pixel overlap by referring to the PQ array corresponding to the Z value.

[0049] When Z=2, that is, at depth d obj To reconstruct an image refocused to Z = 2, adjacent elemental images are shifted by two pixels and overlapped. As shown in the figure, for example, the PQ array corresponding to Z = 2 is expressed as follows: 1 2 3 4 5 6 7 8 9 10 This PQ array means that the pixels in the 4th and 5th rows (4th and 5th columns) overlap with the pixels in the 6th and 7th rows (6th and 7th columns), respectively.

[0050] Note that there may be cases where three or more element images overlap in the column and row directions, in which case the PQ array will be composed of three or more rows. Also, when the matrix of element images is composed of the same number of pixels as in the above example, a common PQ array can be used for overlapping element images in the row direction and the column direction, but when the matrix of element images has a different number of pixels, a PQ array in the row direction and a PQ array in the column direction must be prepared.

[0051] The optical sectioning technology for reconstructing a high-resolution image using the above-mentioned method was invented by the first inventor of the present application and is disclosed in a separate application (Patent Application No. 2022-202566). The image reconstruction unit 32 can reconstruct a high-resolution image using the optical sectioning technology disclosed in the application.

[0052] Returning to FIG. 4, the three-dimensional reconstruction unit 33 reads out the images reconstructed by refocusing at each depth position from the storage device 20, and stacks these reconstructed images to reconstruct a volumetric image of the sample (S14). The reconstructed volumetric image is temporarily stored in the storage device 20. FIG. 9 is a diagram for explaining the reconstruction of images refocused at each depth position from the light field image, and the reconstruction of a volumetric image. Images refocused at multiple depth positions are reconstructed by shifting and superimposing elemental images in the light field image by an amount of shift corresponding to the depth position, but at depth d obj As the size of the element image increases, the shift amount Δ EI becomes larger, and the size of the reconstructed image becomes smaller accordingly. The three-dimensional reconstruction unit 33 resizes the reconstructed images, the sizes of which vary depending on the depth position, according to the depth position, and stacks them in the Z direction (depth direction) to reconstruct a volumetric image.

[0053] Returning to FIG. 4, the quantum sensor position identifying unit 34 reads out the volumetric image of the sample from the storage device 20, and determines the three-dimensional spatial coordinates of each quantum sensor in the volumetric image (S15). Specifically, the quantum sensor position identifying unit 34 identifies voxels relating to each quantum sensor in the volumetric image. That is, the identified voxels represent a three-dimensional image of each quantum sensor. Then, the quantum sensor position identifying unit 34 can determine the center of gravity of the three-dimensional image relating to each quantum sensor, and identify it as the three-dimensional spatial position of each quantum sensor.

[0054] In parallel or asynchronously with the above steps S13 to S15, the fluorescence intensity determination unit 35 determines the fluorescence intensity of each quantum sensor from the fluorescence light originating from the same quantum sensor captured in the light field image of the sample (S16). At this time, the fluorescence intensity determination unit 35 may determine the relative value of the fluorescence intensity of each quantum sensor captured in the light field image when the microwave is on and off, which are successive in time series, as the fluorescence intensity of each quantum sensor. The relative value may be the ratio or difference of the fluorescence intensity. For example, in the example of FIG. 5, the relative value of the intensity that rises and falls microscopically for each quantum sensor may be regarded as the fluorescence intensity of the quantum sensor. This makes it possible to more accurately determine the fluorescence intensity of each quantum sensor by eliminating the influence of background light and the depth position of the quantum sensor on the fluorescence intensity.

[0055] Furthermore, the fluorescence intensity specifying unit 35 obtains the fluorescence intensity of each quantum sensor for each microwave frequency point. This is to know the change pattern of how the fluorescence intensity of the quantum sensor changes due to the microwave frequency sweep. The fluorescence intensity specifying unit 35 may integrate the pixel values ​​of the fluorescence light beam of each quantum sensor for each frequency point of the swept microwave so that the analysis process can be performed with a sufficiently large light beam intensity. This makes it possible to know the change pattern of the fluorescence intensity with high accuracy.

[0056] When the fluorescence intensity of each quantum sensor is specified, the physical quantity calculation unit 36 ​​specifies the resonance frequency of the optical detection magnetic resonance of each quantum sensor from the change pattern of the fluorescence intensity of each quantum sensor, and calculates the physical quantity corresponding to the specified resonance frequency (S17). FIG. 10 is a graph illustrating the temperature dependence of the ODMR spectrum of a fluorescent nanodiamond (FND), which is an example of a quantum sensor. The horizontal axis of the graph is the frequency of the microwave irradiated to the sample, and the vertical axis is the relative value of the fluorescence intensity of the FND specified by the fluorescence intensity specification unit 35. In the graph, the relative values ​​of the fluorescence intensity of the FND at each frequency point under two different environments (when the ambient temperatures of the quantum sensor are 24° C. and 28° C.) are plotted, and the respective fitting curves are superimposed. Each fitting curve is obtained by fitting the fluorescence intensity of the FND at each frequency point to a bimodal Lorentzian function.

[0057] As can be seen from the graph, the fitting curve of the ODMR spectrum of the FND has the characteristic that as the irradiated microwave approaches the resonance frequency, it decreases significantly and takes a minimum value, and then increases slightly near the resonance frequency and takes a maximum value at the resonance frequency. Furthermore, the resonance frequency of the FND varies depending on the temperature. That is, the resonance frequency is temperature dependent. Therefore, if the resonance frequency of the FND can be determined, the temperature as a physical quantity can be determined from it. The temperature can be measured from the resonance frequency of the FND with an accuracy of about 0.1 to 1°C.

[0058] Specifically, the physical quantity calculation unit 36 ​​can determine the resonance frequency from the fitting curve by fitting the fluorescence intensity for each microwave frequency point identified by the fluorescence intensity identification unit 35 to a function for each quantum sensor. In addition to a bimodal Lorentzian function, a Gaussian function can be used for the fitting. Alternatively, the physical quantity calculation unit 36 ​​may use a machine learning model that has been trained in advance with the fluorescence intensity for each microwave frequency point as an input value and the resonance frequency as an output value, and input the fluorescence intensity of the quantum sensor for each microwave frequency point identified by the fluorescence intensity identification unit 35 into the machine learning model to predict the resonance frequency.

[0059] In addition, the corresponding temperature varies even for the same resonance frequency due to variations in the size of the quantum sensor, etc. Therefore, it is preferable to obtain the resonance frequency of the quantum sensor in advance under a predetermined temperature environment, specify the correspondence between the temperature and the resonance frequency, and use this as a reference to calculate the temperature for the resonance frequency specified from the strength of the quantum sensor contained in the sample.

[0060] The physical quantity calculation unit 36 ​​identifies the resonance frequency of each quantum sensor, calculates the corresponding temperature, and then associates the three-dimensional spatial position of each quantum sensor identified by the quantum sensor position identification unit 34 with the calculated temperature (S18). This allows the physical quantity, specifically the temperature, of each three-dimensional spatial position of the pinpoint where each quantum sensor exists in the sample to be measured.

[0061] Effect According to the light field quantum metrology system 100 of this embodiment, the physical quantities of the three-dimensional space in the sample can be measured with high spatial resolution by using quantum sensors, which are extremely small sensors. Furthermore, by capturing an image of a sample including quantum sensors with the light field microscope 10 and analyzing the light field image, the three-dimensional spatial coordinates of multiple quantum sensors in the sample can be identified from one shot of the light field image. This makes it possible to measure the physical quantities of the three-dimensional space in the sample at high speed without the need to capture an image of the three-dimensional space by repeatedly scanning at different depths as in the case of a confocal microscope.

[0062] <<Variations>> If there is no need to distinguish between the fluorescence of the quantum sensor and background light in a light field image obtained by capturing an image of a sample, the fluorescent light beam discriminating unit 31 and step S12 may be omitted.

[0063] Image analysis may be performed by treating the fluorescence intensity when the microwave is on as the fluorescence intensity of the quantum sensor without calculating the relative value of the fluorescence intensity of the quantum sensor when the microwave is on and when the microwave is off. In this case, since it is not necessary to turn the microwave of the microwave generator 50 on and off, step S4 can be omitted.

[0064] When measuring temperature using fluorescent nanodiamonds as quantum sensors, it is necessary to sweep the microwave frequency to obtain the change pattern of the fluorescent intensity of the fluorescent nanodiamonds and then identify the resonance frequency. However, when using a quantum sensor whose physical quantity is uniquely determined from the fluorescent intensity when irradiated with a microwave of a predetermined frequency, it is not necessary to sweep the microwave. In this case, steps S2 and S5 can be omitted. Furthermore, if there is no need to distinguish between the fluorescence of the quantum sensor and background light in the light field image of the sample, the synchronization signal generator 60 can be omitted.

[0065] Although an example of measuring temperature using fluorescent nanodiamonds as a quantum sensor has been explained, it is also possible to measure physical quantities other than temperature, such as electric fields and magnetic fields, by applying an external magnetic field to a sample using the same principle. For example, the following document discloses a method of measuring weak magnetic fields using fluorescent nanodiamonds as a magnetic sensor. Reference 1: Balasubramanian, G., Chan, I., Kolesov, R. et al. Nanoscale imaging magnetometry with diamond spins under ambient conditions. Nature 455, 648-651 (2008). https: / / doi.org / 10.1038 / nature07278 Reference 2: Maze, J., Stanwix, P., Hodges, J. et al. Nanoscale magnetic sensing with an individual electronic spin in diamond. Nature 455, 644-647 (2008). https: / / doi.org / 10.1038 / nature07279 Reference 3: Taylor, J., Cappellaro, P., Childress, L. et al. High-sensitivity diamond magnetometer with nanoscale resolution. Nature Phys 4, 810-816 (2008). https: / / doi.org / 10.1038 / nphys1075

[0066] As described above, the embodiment has been described as an example of the technology of the present invention. For this purpose, the attached drawings and detailed description have been provided. Therefore, among the components described in the attached drawings and detailed description, not only components essential for solving the problem but also components that are not essential for solving the problem in order to illustrate the above technology may be included. Therefore, just because those non-essential components are described in the attached drawings or detailed description, it should not be immediately recognized that those non-essential components are essential. In addition, since the above-mentioned embodiment is intended to illustrate the technology of the present invention, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0067] The light field quantum metrology system and image analysis device according to the present invention can measure physical quantities in three-dimensional space within a sample at high speed and with high spatial resolution, and are therefore useful for observing high-speed events such as biomolecules and intracellular behavior. [Explanation of symbols]

[0068] 100 Light Field Quantum Measurement System 10 Light field microscope (light field optical system) 30 Image analysis device 31 Fluorescent light identification unit 32 Image reconstruction unit 33 Three-dimensional reconstruction unit 34 Quantum sensor position determination unit 35 Fluorescence Intensity Identification Unit 36 Physical quantity calculation section 40 Excitation Light Generator 50 Microwave Generator 60 Synchronous Signal Generator

Claims

1. An image analysis device that analyzes a light field image of a sample including at least one quantum sensor, the sample being imaged by a light field optical system in a state where the sample is irradiated with excitation light having a wavelength that excites the quantum sensor with fluorescence and microwaves having a frequency that causes the quantum sensor to optically detect magnetic resonance, an image reconstruction unit that reconstructs images refocused at each depth position from the light field image of the sample; a three-dimensional reconstruction unit that stacks the reconstructed images at each depth position to reconstruct a volumetric image of the sample; a quantum sensor position determining unit for determining three-dimensional spatial coordinates of each quantum sensor in the volumetric image of the sample; a fluorescence intensity determination unit that determines the fluorescence intensity of each of the quantum sensors from a fluorescent light beam originating from the same quantum sensor captured in a light field image of the sample; a physical quantity calculation unit that calculates a physical quantity in three-dimensional spatial coordinates of each of the quantum sensors from the fluorescence intensity of each of the quantum sensors; An image analysis device comprising:

2. the light field image of the sample is obtained by continuously imaging the sample by sweeping the frequency of the microwave around a resonant frequency of the optically detected magnetic resonance of the quantum sensor; The physical quantity calculation unit identifies a resonance frequency of the optical detection magnetic resonance of each of the quantum sensors from a change pattern of the fluorescence intensity of each of the quantum sensors, and calculates the physical quantity corresponding to the identified resonance frequency. The image analysis device according to claim 1 .

3. The light field image of the sample is obtained by continuously imaging the sample while turning the microwave on and off at a constant cycle, The fluorescence intensity determination unit determines, as the fluorescence intensity of each quantum sensor, a relative value of the fluorescence intensity of each quantum sensor captured in the light field image when the microwave is on and when the microwave is off, which are successive in time series. The image analysis device according to claim 2 .

4. The light field image of the sample is obtained by continuously imaging the sample while turning the microwave on and off at a constant cycle, The image analysis device further includes a fluorescent light beam identification unit that identifies a light beam that becomes bright and dark in the light field image with an on / off cycle of the microwave as a fluorescent light beam of a quantum sensor, The image reconstruction unit reconstructs images refocused at each depth position for the identified fluorescent light beam, The fluorescence intensity identifying unit determines the intensity of the identified fluorescent light beam. The image analysis device according to claim 2 .

5. The quantum sensor is a fluorescent nanodiamond.

5. The image analysis device according to claim 1.

6. The physical quantity is temperature.

5. The image analysis device according to claim 1.

7. the quantum sensor is a fluorescent nanodiamond; The physical quantity is temperature.

5. The image analysis device according to claim 1.

8. An image analysis method for analyzing a light field image of at least one quantum sensor captured by a light field optical system in a state where a sample including the quantum sensor is irradiated with excitation light having a wavelength that excites the quantum sensor with fluorescence and a microwave having a frequency that causes optical detection magnetic resonance of the quantum sensor, the image analysis method comprising: An image reconstruction unit reconstructs images refocused at each depth position from a light field image of the sample; A three-dimensional reconstruction unit stacks the reconstructed images at each depth position to reconstruct a volumetric image of the sample; A quantum sensor position determining unit determines three-dimensional spatial coordinates of each quantum sensor from a volumetric image of the sample; A step in which a fluorescence intensity determination unit determines the fluorescence intensity of each of the quantum sensors from a fluorescence light beam originating from the same quantum sensor captured in a light field image of the sample; A physical quantity calculation unit calculates a physical quantity in three-dimensional spatial coordinates of each of the quantum sensors from the fluorescence intensity of each of the quantum sensors; The image analysis method includes:

9. A program for causing a computer to analyze a light field image of a sample including at least one quantum sensor, the sample being irradiated with excitation light having a wavelength that excites the quantum sensor with fluorescence and microwaves having a frequency that causes the quantum sensor to optically detect and magnetically resonate, the program comprising: an image reconstruction means for reconstructing images refocused at each depth position from the light field image of the sample; a three-dimensional reconstruction means for stacking the reconstructed images at each depth position to reconstruct a volumetric image of the sample; a quantum sensor position determining means for determining three-dimensional spatial coordinates of each quantum sensor from the volumetric image of the sample; a fluorescence intensity determination means for determining the fluorescence intensity of each of the quantum sensors from a fluorescent light beam originating from the same quantum sensor captured in a light field image of the sample; and a physical quantity calculation means for calculating a physical quantity in three-dimensional spatial coordinates of each of the quantum sensors from the fluorescence intensity of each of the quantum sensors; A program that makes a computer function as a

10. A system for measuring a physical quantity using a quantum sensor, an excitation light generator that generates excitation light of a wavelength that excites the quantum sensor with fluorescence; a microwave generator that generates microwaves at a frequency at which the quantum sensor optically detects magnetic resonance; a light field optical system that captures a light field image of a subject; The image analysis device according to any one of claims 1 to 4, which analyzes a light field image of a sample including at least one quantum sensor, the light field image being captured by the light field optical system in a state where the sample is irradiated with the excitation light and the microwave; A light field quantum metrology system equipped with

11. A method for measuring a physical quantity using a quantum sensor, comprising: A light field optical system captures a light field image of a sample including at least one quantum sensor in a state where the sample is irradiated with excitation light having a wavelength that excites the quantum sensor with fluorescence and microwaves having a frequency that causes the quantum sensor to optically detect and magnetically resonate; 5. The image analysis apparatus according to claim 1 , further comprising: A light field quantum metrology method comprising:

Images