Light field three-dimensional tracking control device, method, and program, and light field microscope system

Abstract

To automatically three-dimensionally track an object in a microscope system using a light field camera.SOLUTION: A light field three-dimensional tracking control device 110 controls a motor-driven stage 90 movable in XYZ directions to three-dimensionally track an object observed by a light field camera 10 through an objective lens 70 even if the object is in a sample holder 80 mounted on the motor-driven stage 90, and comprises: a light field image analysis unit 30 that analyzes a light field image obtained by imaging the object with the light field camera 10 in a time-continuous manner to track a three-dimensional spatial position of the object in the light field image; and a stage control unit 90 that controls the movement in the XYZ directions of the motor-driven stage 90 so that a relative positional relation between the objective lens 70 and the three-dimensional spatial position of the object tracked by the light field image analysis unit 30 is maintained constant.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for automatically tracking an observation object in a microscope by controlling a motorized stage, and more particularly to a technique for automatically tracking an object in three dimensions in a microscope system that uses a light field camera. [Background technology]

[0002] High-speed, large-scale three-dimensional imaging is required for delicate and dynamic observation of biomolecules and intracellular behavior. Typically, such three-dimensional images are generated by using a confocal microscope to continuously change the focal position of the sample in the depth direction (Z direction) at a predetermined pitch and capture images of the sample at each focal position (see, for example, Patent Documents 1 and 2). Another microscope that enables three-dimensional imaging is the spinning disk confocal microscope. The spinning disk confocal method uses a laser beam to illuminate a rotating disk with an array of numerous pinholes, creating multiple parallel beams of light, which are then used to rapidly scan the sample to form a confocal image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-044016 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-157158 Summary of the Invention [Problem to be solved by the invention]

[0004] Three-dimensional imaging using confocal microscopes requires scanning in the depth direction, which takes time to measure and makes it unsuitable for observing high-speed events. For example, if a three-dimensional space is imaged by 30 scans in the Z direction at 30 fps, the imaging speed for the entire three-dimensional space is 30 fps / 30 = 1 vps, which means it takes one second to capture the three-dimensional space. Because neural activity is a phenomenon on the order of milliseconds, three-dimensional imaging using confocal microscopes is insufficient in terms of speed for observing biomolecules or intracellular behavior.

[0005] Another option for 3D imaging is light field microscopy (LFM). A light field represents a collection of light rays in three-dimensional space. LFM uses a microlens array, which is a two-dimensional arrangement of many microlenses, placed at the intermediate image plane. This allows the light field to be recorded by determining the position of the light rays passing through the aperture stop of the objective lens and the position of the light rays being captured on the image sensor. From the light field image recorded by LFM, subaperture images equivalent to those captured by a small-aperture lens can be obtained. Subaperture images are partial images of the subject captured by shifting the viewpoint. Each subaperture image has a deep depth of field due to the pinhole effect, and at the same time, there is parallax between them. Therefore, by shifting and overlapping the subaperture images, it is possible to reconstruct an image refocused at any depth position.

[0006] As such, LFM is suitable for observing high-speed events because it can rapidly capture three-dimensional images in a single shot without scanning in the depth direction. However, using a high-magnification objective lens to observe an object at high magnification narrows the field of view, making it more likely for the object to fall out of the field of view, especially if the object is a moving object such as a living organism or suspended particle. Furthermore, if the object moves in the depth direction, it may fall out of the depth of field of the objective lens. This makes it difficult to observe an object with a high-magnification objective lens. Therefore, automatic three-dimensional tracking of the object is required to enable observation with a high-magnification objective lens.

[0007] Therefore, an object of the present invention is to provide a control device for automatically tracking an object in three dimensions in a microscope system that uses a light field camera, and a microscope system equipped with such a control device. [Means for solving the problem]

[0008] According to one aspect of the present invention, there is provided a light field three-dimensional tracking control device that controls a motorized stage that moves in X, Y, and Z directions to three-dimensionally track an object in a sample holder placed on the motorized stage and that is observed by a light field camera through an objective lens, the light field three-dimensional tracking control device comprising: a light field image analysis unit that analyzes light field images of the object captured continuously over time by the light field camera and tracks the three-dimensional spatial position of the object in the light field images; and a stage control unit that controls the movement of the motorized stage in the X, Y, and Z directions so that a constant relative positional relationship is maintained between the objective lens and the three-dimensional spatial position of the object tracked by the light field image analysis unit. A light field three-dimensional tracking control method and a computer program corresponding to the light field three-dimensional tracking control device are also provided.

[0009] According to another aspect of the present invention, there is provided a light field microscope system comprising an objective lens, a light field camera, a full-field illumination system that illuminates the entire field of view of the light field camera, a motorized stage that is movable in the XYZ directions and on which a sample holder that contains an object to be observed by the light field camera is placed, and the above-mentioned light field three-dimensional tracking control device. [Effects of the Invention]

[0010] The present invention enables automatic three-dimensional tracking of objects in a microscope system using a light field camera, allowing for more detailed observation of moving, microscopic objects such as living organisms with a high-magnification objective lens. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram of a light field microscope system according to an embodiment of the present invention; [Figure 2] 1 is a block diagram of a light field 3D tracking control device according to an embodiment of the present invention; [Figure 3] 1A and 1B are diagrams illustrating an example of a light field image, a group of elemental images, and a reconstructed image, respectively. [Figure 4] FIG. 1 is a diagram illustrating a geometric relationship between parameters of a light field camera according to an example. [Figure 5] 10A and 10B are diagrams illustrating the relationship between a PQ array and superimposition of element images. [Figure 6A] 1 is the first half of a flowchart for light field image analysis. [Figure 6B] This is the second half of the flowchart for light field image analysis. [Figure 7] 10A and 10B are diagrams illustrating an example of three-dimensionally reconstructing an object from an initial group of elemental images, performing bright spot labeling, setting initial coordinates of the object, and selecting a template. [Figure 8] 10A and 10B are diagrams illustrating how to specify a representative point of each bright point in a group of elemental images. [Figure 9] 10A and 10B are diagrams illustrating the tracking of representative points of each bright point in a group of elemental images. [Figure 10] FIG. 10 is a diagram illustrating bright spot labeling in an elemental image group. [Figure 11] 10A and 10B are diagrams illustrating template selection when there are multiple bright spots in one element image and bright spot labeling in the second and subsequent element image groups. [Figure 12] 10A and 10B are diagrams illustrating intervals between representative points of bright spots between adjacent elemental images in an elemental image group. [Figure 13] 10 is a graph showing the tracking performance in the Z direction. [Figure 14] FIG. 10 is a diagram showing how a light field image analysis unit tracks an object. [Figure 15] 10 is a flowchart of the motorized stage control. [Figure 16] 10 is a timing chart of a three-dimensional tracking parallel process according to an example. [Figure 17] 10 is a timing chart of three-dimensional tracking parallel processing according to another example. [Figure 18] This is a graph comparing the trajectory of a freely moving object (nematode) with the size of the field of view of a high-magnification objective lens. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present invention, and are not intended to limit the subject matter described in the claims. Furthermore, the dimensions of each component depicted in the drawings, the detailed shapes of the details, and the like may differ from the actual ones.

[0013] <Embodiment of Light Field Microscope System> FIG. 1 is a schematic diagram of a light field microscope system according to one embodiment of the present invention. The light field microscope system 100 according to this embodiment automatically tracks a moving object in three-dimensional space, such as a biomolecule, cell, or microorganism, observed through the objective lens of a light field camera, by moving a motorized stage, and keeps the object at a constant distance from the objective lens and in the center of the objective lens's field of view. Generally, the light field microscope system 100 includes a light field camera 10, a full-field illumination system 50, an objective lens 70, a motorized stage 90, a light field three-dimensional tracking controller 110, and several other optical components. For ease of explanation, it is assumed that a sample containing the object is adjusted to a sample holder 80, such as a glass slide or a petri dish, and placed on the motorized stage 90, with the objective lens 70 positioned above it.

[0014] The full-field illumination system 50 includes a light source 51 and a collimator lens 52. The light source 51 supplies light for illuminating the full field of view of the light field camera 10. The light emitted from the light source 51 may be natural light, including white light and infrared light, or laser light with a specific wavelength, such as 470 nm or 550 nm, for causing the target to fluoresce. The collimator lens 52 collimates the light emitted from the light source 51. A lens 72, a dichroic mirror 73, and an objective lens 70 are sequentially arranged ahead of the collimated light in the traveling direction. Of these, the dichroic mirror 73 is configured to transmit the wavelength range of the light from the light source 51. The light collimated by the collimator lens 52 passes through the lens 72 and the dichroic mirror 73 and enters the objective lens 70, thereby uniformly illuminating the sample holder 80 as a whole. In this way, the full-field illumination system 50 illuminates the full field of view of the light field camera 10.

[0015] The light field camera 10 includes an imaging lens (tube lens) 1, a microlens array 2, and an image sensor 3. An objective lens 70, a dichroic mirror 73, the imaging lens 1, and the microlens array 2 are sequentially arranged ahead of the sample in the sample holder 80 in the direction of fluorescence propagation. The dichroic mirror 73 reflects the wavelength range of the fluorescence from the sample. The microlens array 2 is configured by a two-dimensional array of multiple microlenses (lenslets) 2a and is positioned between the imaging lens 1 and the image sensor 3, specifically at the location of the image formed by the objective lens 70 and the imaging lens 1. The image sensor 3 is a CCD sensor or CMOS sensor, etc., configured by a two-dimensional array of multiple light-receiving elements (not shown) that photoelectrically convert input light and output electrical signals. The electrical signals output from each light-receiving element are converted into digital data by an A / D converter (not shown), and a light field image is output from the light field camera 10. The light field image is supplied to a light field three-dimensional tracking control device 110.

[0016] The motorized stage 90 includes a mounting stage (not shown) on which the sample holder 80 is placed and an actuator (not shown), and can move the mounting stage in each of three orthogonal axes, i.e., the X, Y, and Z directions, by driving the actuator in accordance with a given control signal. For example, if an instruction to move the motorized stage 90 in the X and Y directions is given, the mounting stage can be moved horizontally while maintaining a constant distance between the objective lens 70 and the sample holder 80. On the other hand, if an instruction to move the motorized stage 90 in the Z direction is given, the mounting stage can be moved vertically to adjust the distance between the objective lens 70 and the sample holder 80. Such operation of the motorized stage 90 is controlled by a light field three-dimensional tracking control device 110.

[0017] <Embodiments of Light Field Three-Dimensional Tracking Control Device> The light field 3D tracking control device 110 can be configured as a computer device equipped with a storage device and a processor. Figure 2 is a block diagram of a light field 3D tracking control device according to one embodiment of the present invention. The light field 3D tracking control device 110 according to this embodiment includes, as its main components, a storage device 20, a light field image analysis unit 30, and a stage control unit 40. These components are connected to each other so that they can communicate with each other and exchange data.

[0018] The storage device 20 is a collection of storage devices such as RAM, ROM, SSD, and HDD. The RAM is primarily used as working memory when the light field 3D tracking control device 110 performs various calculation processes, and also temporarily stores light field images captured continuously by the light field camera 10 and supplied to the light field 3D tracking control device 110, reconstructed images generated and used in the image analysis described below, 3D images, and label tables. The ROM, SSD, and HDD primarily store computer programs for operating the computer as the light field image analysis unit 30 and stage control unit 40, as well as PQ arrays referenced by the light field image analysis unit 30, for long-term storage.

[0019] The light field image analysis unit 30 is a module that analyzes light field images that are captured continuously over time by the light field camera 10 and buffered in the storage device 20, and tracks objects in the light field images. More specifically, the light field image analysis unit 30 includes the following components: an image reconstruction unit 31, a template selection unit 32, a bright point representative point identification unit 33, a bright point tracking unit 34, a bright point labeling unit 35, a three-dimensional coordinate update unit 36, and a bright point group centroid calculation unit 37.

[0020] The stage control unit 40 is a module that determines a stage control amount for maintaining a constant relative positional relationship between the object in the sample holder 80 and the objective lens 70 from the analysis result of the light field image by the light field image analysis unit 30, and controls the movement of the motorized stage 90 in the X, Y, and Z directions using the determined stage control amount. More specifically, the stage control unit 40 includes components of an XY control unit 41 and a Z control unit 42.

[0021] The light field image analysis unit 30 and the stage control unit 40 can be realized as hardware such as an ASIC or FPGA, or as software in which a computer program stored in the storage device 20 is executed by a processor (not shown) such as a CPU or GPU provided in the light field 3D tracking control device 110. They can also be realized by appropriately combining hardware and software.

[0022] The light field image to be analyzed by the light field image analysis unit 30 includes an array of elemental images, each of which is an arrangement of elemental images captured of the same subject from multiple, mutually offset viewpoints. From such an elemental image group, a refocused image at any depth can be reconstructed using the parallax of each elemental image included within the elemental image group. FIG. 3 illustrates an example of a light field image, elemental images, and a reconstructed image. (a) Light field image is captured by placing a sheet with the letters " / 5" on it at a position offset from the focal plane of the light field camera 10. The light field image is a group of microlens images MI, in which numerous microlens images MI captured by each microlens 2a of the microlens array 2 of the light field camera 10 are arranged in a square lattice pattern. As indicated by the dashed line, each microlens image MI has an approximately circular shape, reflecting the circular shape of each microlens 2a. Thus, a light field image is a collection of multiple sub-images (microlens images MI in the example of FIG. 3) captured of the same subject from multiple, mutually offset viewpoints.

[0023] The area near the boundary of the microlens image MI is dark due to the small amount of light, and the central part of the microlens image MI is used because the area is distorted significantly due to the light rays that have passed through the edge of the microlens 2a. That is, from each of the approximately circular microlens images MI, for example, a square area inscribed in the circle is cut out as an element image EI, and these element images EI are arranged in a square lattice pattern to form the (b) element image group.

[0024] As described above, since the sheet on which the characters " / 5" are written is located at a position shifted from the focal plane of the light field camera 10, (a) in the light field image, microlens images are recorded in which the viewpoint is shifted by each microlens 2a to capture a portion of the characters " / 5," i.e., microlens images with parallax. For example, if one focuses on the upper left corner of the character "5," the light beam emitted from that portion of the subject is recorded with a distance Δ between adjacent microlens images. (b) As shown in the elemental image group, the distance Δ between the microlens images is Δ between the elemental images. EI Here, the pitch of the microlenses 2a is MLP, and the length of one side of the element image is EI. L Then, Δ EI is expressed by the following equation (1). Δ EI =Δ-(MLP-EI L ) …(1)

[0025] The reconstructed image (c) is obtained by superimposing adjacent elemental images in the elemental image group by shifting them by an amount corresponding to the parallax. In the example of Figure 3, adjacent elemental images are shifted by Δ EI By shifting the elemental images by an appropriate amount and overlaying them, a reconstructed image (c) is obtained, refocused at the position of the sheet with the letters " / 5" written on it. In this way, by overlaying the elemental images with appropriate shift amounts, it is possible to reconstruct an image refocused at any depth position based on ray tracing in the reverse direction from the acquired image.

[0026] Here, the depth of the object to be refocused, which is the deviation from the object-side focal plane NOP (Native Object Plane) of the objective lens 70, is defined as d obj Then, the distance Δ between rays originating from the same point between adjacent microlens images and the depth d of the point obj The relationship between the parameters of the light field camera 10 and the objective lens 70 can be expressed using the parameters of the light field camera 10. FIG. 4 is a diagram schematically showing the geometric relationship between the parameters of the light field camera 10 according to an example. For convenience, the objective lens 70 and the imaging lens 1 shown in FIG. 1 are depicted as one unit in FIG. 4. O in the figure indicates the depth d obj A is the image point of the object O formed by the objective lens 70 and the imaging lens 1, B and C are the centers of the adjacent microlenses 2a, and B' and C' are the positions of the light rays from the object O captured by the image sensor 3 via B and C. When we look at the triangles ABC and AB'C' in the figure, from the similarity relationship between triangles, BC:B'C'=CA:C'A This can be expressed as parameters as follows: MLP:Δ=(a+d obj M 2 ):(a+d obj M 2 -b) Solving this for Δ gives the following equation (2). Δ=MLP(1-b / (d obj M 2 -a)) …(2) where MLP is the pitch of the microlenses 2a, M is the magnification of the objective lens 70, a is the distance from the image-side focal plane T-NIP (Native Image Plane) of the imaging lens 1 to the center of the microlens 2a, and b is the distance from the center of the microlens 2a to the image sensor 3. Note that in FIG. 4, a is considered to be a positive value to show the geometric relationship, but if the direction from the object side to the image side is considered to be positive, a needs to be considered to be a negative value. For this reason, in equation (2), a is finally replaced with -a.

[0027] Note that since there is no anisotropy in two mutually perpendicular directions (X direction, Y direction) in a plane perpendicular to the depth direction (Z direction), the same Δ can be applied to both the X direction and the Y direction.

[0028] From equations (1) and (2), obj and Δ EI Therefore, at any depth position (depth d obj ) to reconstruct an image refocused to the elemental images, the shift amount Δ EI The elemental images can be easily overlapped by referring to the PQ array described below.

[0029] Figure 5 illustrates the relationship between the PQ array and the overlapping of element images. The overlapping pixels are shaded. The left column shows the overlapping state of the element images, and the right column shows the expanded state of the overlapping element images. For convenience, the element image group consists of four element images, D, E, F, and G, arranged two by two, with each element image measuring five pixels vertically and horizontally. The element images are arranged as follows: element image E is to the right of element image D, element image F is below element image D, and element image G is below element image E, i.e., to the right of element image F. To reference each pixel in the element image group, consecutive numbers starting from 1 are assigned to the rows and columns. For example, pixels belonging to element image D are referenced from rows 1 to 5 and columns 1 to 5, while pixels belonging to element image G are referenced from rows 6 to 10 and columns 6 to 10. Below, pixel [m, n] refers to the pixel located at row m and column n in the element image group.

[0030] Z in the figure is d obj is a depth parameter that represents the obj and the shift amount of the element image Δ EI are expressed with the same Z value. That is, when Z=n, d obj = n, and the shift amount of the element image Δ EI represents n pixels.

[0031] When Z=0, that is, at depth dobj If ≠0, the elemental images are not superimposed on each other, and the elemental images are directly used as the reconstructed image.

[0032] When Z=1, that is, the depth d obj To reconstruct an image refocused to Z=1, adjacent elemental images are shifted by one pixel and then superimposed. Which pixels in the elemental image group should be superimposed can be easily determined by referring to the PQ array, which shows how the pixels are superimposed. The PQ array is a representation of the row or column numbers of pixels in the elemental image group, folded over for each elemental image. As shown in the figure, for example, the PQ array corresponding to Z=1 is expressed as follows: 1 2 3 4 5 6 7 8 9 10 This PQ array means that the pixel in row 5 (column 5) and the pixel in row 6 (column 6) overlap. By preparing such PQ arrays for each Z value in advance and storing them in storage device 20, it is possible to perform pixel overlap by referencing the PQ array corresponding to the Z value.

[0033] When Z=2, that is, the depth d obj To reconstruct an image refocused to Z = 2, adjacent element images are shifted by two pixels and then superimposed. As shown in the figure, for example, the PQ array corresponding to Z = 2 is expressed as follows: 1 2 3 4 5 6 7 8 9 10 This PQ array means that the pixels in the 4th and 5th rows (4th and 5th columns) and the pixels in the 6th and 7th rows (6th and 7th columns) overlap each other.

[0034] Note that there may be cases where three or more element images overlap in the column and row directions, in which case the PQ array will consist of three or more rows. Also, when the element image matrices and columns have the same number of pixels, as in the above example, a common PQ array can be used for overlapping element images in the row and column directions, but when the element image matrices and columns have different numbers of pixels, a PQ array for the row and column directions must be prepared.

[0035] <Light field image analysis> Next, image analysis by the light field image analysis unit 30 will be described. Fig. 6A is the first half of the flowchart for light field image analysis, showing the image analysis procedure for the first light field image (corresponding to time t=1). Fig. 6B is the second half of the flowchart for light field image analysis, showing the image analysis procedure for the second and subsequent light field images (corresponding to time t=2 and subsequent). Each step is executed by the above-mentioned components of the light field image analysis unit 30.

[0036] The light field image analysis unit 30 reads the initial light field image from the storage device 20 and constructs a group of elemental images. In this embodiment, the light field image captured by the light field camera 10 resembles an optical microscope image, with a dark background and a bright object. A group of microlens images containing pixels with a certain brightness or pixel value is considered to be an image of the object. Therefore, a group of microlens images containing pixels with a certain brightness or pixel value may be detected in the light field image, and a partial image of a certain size containing the group of microlens images may be extracted from the light field image, and a group of elemental images such as that shown in FIG. 3(b) may be constructed from the partial image. For example, if the size of the light field image is 5120 x 5120 pixels and the size of the object captured therein is only a few to a dozen pixels, it is inefficient to perform image analysis on the entire light field image, and therefore it is preferable to narrow down the range of analysis in this way.

[0037] Once the elemental image group is constructed, the image reconstruction unit 31 reconstructs an image from the elemental image group refocused on the depth position of the object (S1). Once the reconstructed image is generated, the bright spot labeling unit 35 labels the representative point of each bright spot, which is a group of pixels corresponding to the object in the reconstructed image, to identify the object (S2). A bright spot here refers to a group of one or more pixels having a certain luminance or pixel value, i.e., a point or small image region with a certain brightness. Once labeling is complete, the three-dimensional coordinate update unit 36 ​​determines the three-dimensional coordinates of the object in the reconstructed image and sets these coordinates as the initial coordinates of the object identified by the label (S3). In addition, in parallel with or asynchronously with steps S1 to S3, the template selection unit 32 selects the elemental image containing the most bright spots from the elemental image group as a template (S4).

[0038] 7 is a diagram illustrating an example of three-dimensionally reconstructing an object from the initial elemental image group, performing bright spot labeling, setting the initial coordinates of the object, and selecting a template. For example, the image reconstruction unit 31 appropriately refers to the PQ array stored in the storage device 20, shifts the elemental images in the elemental image group by a shift amount according to the depth position, and overlaps them to reconstruct images refocused at multiple depth positions. Depth d obj As the value of Δ EI becomes larger, and the size of the reconstructed image becomes smaller accordingly. The image reconstruction unit 31 resizes the reconstructed images, which vary in size depending on the depth position, according to the depth position and stacks them in the Z direction (depth direction) to reconstruct a three-dimensional image of the target object. Note that a high-resolution image can be reconstructed by using the optical sectioning technology disclosed in the application of the present inventors (Patent Application No. 2022-202566) as a method for reconstructing an image refocused at an arbitrary depth position from a group of elemental images.

[0039] To identify pixels in the elemental image group that correspond to the three-dimensional image, for example, the bright spot labeling unit 35 multiplies the X and Y coordinates of each voxel in the three-dimensional image by the resizing ratio of the image at each depth position to convert them into pixel coordinates in the first elemental image group. For example, if the size of an image at a certain Z coordinate in the three-dimensional image is m and the size of that image before resizing is n, the X and Y coordinates of each voxel at that Z coordinate can be converted into corresponding pixel coordinates in the first elemental image group by multiplying them by the resizing ratio n / m. However, since the coordinates of other pixels overlapping with these converted pixel coordinates are unknown, the bright spot labeling unit 35 refers to the PQ array to identify the coordinates of other pixels that will overlap with the pixel at the converted pixel coordinates when reconstructing an image refocused at that depth position, and attaches a label that identifies the object to the representative point of each bright spot consisting of the pixel at the converted pixel coordinates and the group of pixels that will overlap with the pixel at the converted pixel coordinates when reconstructing an image refocused at that depth position. For example, each pixel constituting a bright spot can be binarized to determine its center of gravity, and the center of gravity can be used as the representative point of the bright spot. The same label is assigned to representative points of bright spots corresponding to the same three-dimensional image.

[0040] The correspondence between the labels and the representative points of each bright spot is tabulated and stored as a label table in the storage device 20. The labels are also associated with the three-dimensional coordinates of the object. Specifically, the three-dimensional coordinate update unit 36 ​​determines the coordinates of each voxel of the three-dimensional image, and determines, for example, the center of gravity of the three-dimensional image as a coordinate in three-dimensional space, and sets the center of gravity as the initial coordinate of the object identified by the label.

[0041] In parallel with or asynchronously with the above image reconstruction, bright spot labeling, and initial coordinate setting of the target object, the template selection unit 32 selects the element image containing the most bright spots as the template. For convenience, in the example of FIG. 7, there is one target object, so each element image has one bright spot. In this way, when there are multiple element images that can be candidates for the template, the element image closest to the origin in the element image group, i.e., the element image located in the upper left corner, can be selected as the template. The template is referenced in the bright spot labeling process for the second element image group. This completes the analysis of the first light field image.

[0042] Returning to FIG. 6B, the light field image analysis unit 30 reads the next light field image from the storage device 20 and constructs a group of elemental images. Here, too, the elemental image group may be constructed by narrowing the analysis target range as described above. Once the elemental image group is constructed, the bright spot representative point identification unit 33 determines the representative point of each bright spot in the elemental image group (S5). FIG. 8 is a diagram illustrating the identification of the representative point of each bright spot in the elemental image group. Because the elemental image group is a grayscale image, the bright spot representative point identification unit 33 smoothes the elemental image group using an averaging filter as necessary and then binarizes the elemental image group. The bright spot representative point identification unit 33 then determines the center of gravity of each binarized bright spot and designates the center of gravity as the representative point of that bright spot. Subsequently, the representative point of each bright spot is labeled and tracked.

[0043] The threshold for binarization can be a fixed value or determined based on the characteristics of the elemental images. In the latter case, the approximate percentage of pixels containing the target object is calculated based on the field of view and the size of the target object. For example, if the field of view is 422.4 μm × 422.4 μm and the target object is a circle with a diameter of 4 μm, the percentage of pixels containing the target object in the field of view (elemental images) is approximately 0.0070%. Next, the pixel value and its frequency percentage for each pixel in the elemental images are calculated, and the frequencies are accumulated in descending order until the percentage of pixels containing the target object is reached. The pixel value corresponding to the final sum is set as the threshold. For example, in the above example, the pixel values ​​are accumulated starting from the most frequent ones, and the pixel value when the accumulated value exceeds 99.993% is set as the threshold. This method of determining the threshold is effective for images with high contrast, such as microscopic images, where the target object is clearly separated from the background and the target object occupies a very small proportion of the field of view. If the object can be observed in advance, the pixel values ​​of the background and object can be determined in advance using appropriate image analysis software, and these values ​​can be used as the threshold. Also, Otsu's binarization, a common automatic binarization method, is difficult to apply when the background and object are clearly separated and the object occupies a very small proportion of the entire field of view, but it can be applied depending on the object.

[0044] Returning to FIG. 6B, once the representative points of each bright point in the elemental images are determined, the bright point tracking unit 34 determines the positional change of each bright point in the elemental image group (S6). FIG. 9 is a diagram illustrating the tracking of the representative points of each bright point in the elemental image group. For convenience, FIG. 9 shows two temporally adjacent elemental image groups superimposed on each other. The bright point tracking unit 34 finds the destination of each bright point's representative point between the temporally adjacent elemental image groups by nearest neighbor search. If the frame rate of continuously captured light field images is sufficiently high relative to the moving speed of the target object, as shown in FIG. 9, the deviation in coordinates of each bright point's representative point between the temporally adjacent elemental image groups is very small, and the destination of each bright point's representative point can be found by nearest neighbor search. For example, as shown in the enlarged view, the destinations of five bright point representative points P1 to P5 can be found by nearest neighbor search between the temporally adjacent elemental image groups. When the destination of the representative point of each bright point is found, the bright point tracking unit 34 determines the position change of each bright point from the positional relationship before and after the movement of the representative point of each bright point. For example, for the representative point P1 of the bright point, the XY coordinates at time t-1 are P1 tー1 =(xP1 tー1 ,yP1 t-1 ) and the XY coordinates at time t are P1 t =(xP1 t ,yP1 t ), then the position change of P1 is P1 t -P1 tー1 In the enlarged image, the representative point below P1 and to the right of P3 first appeared at time t, so it cannot be tracked from time t-1 to time t. If this representative point continues to appear after time t, its position will be tracked.

[0045] Returning to FIG. 6B, in parallel with tracking the representative points of each bright point, the bright point labeling unit 35 labels the representative points of each bright point in the template in each element image of the element image group with the same label as the representative points of each bright point in the template (S7). FIG. 10 is a diagram illustrating bright point labeling in element image groups. The left diagram shows the kth element image group, the center diagram shows the k+1th element image group in which a nearest neighbor search for a representative point of a bright point is being performed using the template, and the right diagram shows the k+1th element image group after labeling. For convenience, it is assumed that there is only one representative point of a bright point in each element image, and that the same label is assigned to each representative point of a bright point in the kth element image group in the left diagram. In this case, as shown in the center diagram, the bright point labeling unit 35 uses nearest neighbor search to find the representative point of a bright point in each element image of the k+1th element image group that corresponds to the representative point P0 of each bright point in the template. For convenience, the elemental image group in the center diagram shows a template superimposed on each elemental image in the (k+1)th elemental image group. In this example, each elemental image has only one representative point of a bright spot, so each representative point is found as a representative point corresponding to a representative point of a bright spot in the template. When a corresponding representative point of a bright spot is found in each elemental image, the bright spot labeling unit 35 assigns the same label to the found representative point as the representative point P0 of the corresponding bright spot in the template. As a result, as shown on the right side of the diagram, a label is assigned to each representative point of a bright spot in the (k+1)th elemental image group. In this way, a label is assigned to each representative point of a bright spot in the new elemental image group.

[0046] When there are multiple targets, an element image may contain multiple bright spots resulting from those targets. In such cases, the bright spot labeling unit 35 may label element images containing a predetermined number of bright spots or more. For example, if the number of bright spots in a template is N, element images containing N-1 or more bright spots are labeled. This reduces the number of element images to be labeled, thereby reducing the load associated with the labeling process. Figure 11 illustrates template selection and bright spot labeling for the second and subsequent element image groups when a single element image contains multiple bright spots. The left diagram shows the first element image group, the center diagram shows the mth element image group, and the right diagram shows the nth (n>m)th element image group. The template selection unit 32 selects the element image containing the most bright spots from the first element image group as the template. In this example, an element image containing four bright spots is selected as the template, but there are multiple such element images. In such cases, the template selection unit 32 selects the element image with the greatest degree of separation between the bright spots as the template. If the degree of separation is expressed by SEP, the SEP can be defined, for example, by the following formula: SEP=X max -X min +Y max -Y min However, X max is the maximum X coordinate among the X and Y coordinates of the representative point of multiple bright points included in the element image of the template candidate, and X min is the minimum X coordinate, Y max is the maximum Y coordinate, Y min is the minimum Y coordinate.

[0047] As described above, when an elemental image is extracted from a microlens image, a portion of the microlens image is cut. However, if the target object moves, the bright spots included in the cut area may move within the elemental image, causing the number of bright spots in the elemental image to increase or decrease over time. For example, if we focus on the elemental image enlarged in Figure 11, the number of bright spots in the elemental image is 1 in the first elemental image group (left), increases to 2 in the mth elemental image group (center), and increases to 4 in the nth elemental image group (right). Although the number of bright spots in the elemental image in the mth elemental image group increases to 2, this does not satisfy the condition of N-1 or more, so the elemental image is not subject to labeling. Subsequently, if the number of bright spots in the elemental image in the nth elemental image group increases to 4, the condition of N-1 or more is satisfied, so the elemental image is subject to labeling. For the elemental images to be labeled, the bright spot labeling unit 35 assigns the same labels as the representative points of each bright spot in the template to the representative points of each bright spot that correspond to the representative points of each bright spot in the template.

[0048] Returning to FIG. 6B, once the tracking and labeling of the representative points of each bright spot is complete, the three-dimensional coordinate update unit 36 ​​updates the XY coordinates of the object identified by the label based on the positional changes of the representative points of bright spots with the same label, and updates the Z coordinate of the object based on the spacing between those representative points between adjacent elemental images (S8). For example, the XY coordinates of the object at time t can be calculated by adding the average positional changes of the representative points of bright spots with the same label from time t-1 to time t to the XY coordinates at time t-1. This can be expressed mathematically as follows: (x t ,y t )=(x t-1 ,y t-1 )+〈P t -P t-1 〉 However, (x t ,y t ) is the XY coordinate of the object at time t, (x t-1 ,y t-1 ) are the X and Y coordinates of the object at time t-1, P tare the X and Y coordinates of the representative point of the bright spot with the same label at time t, and P t-1 are the X and Y coordinates of the representative point of the bright points with the same label at time t-1, and the operator 〈〉 represents the average value of each X and Y coordinate.

[0049] The Z coordinate of the target object at time t can be directly calculated based on the interval between the representative points of the bright points with the same label in the element image group at time t between the adjacent element images. Fig. 12 is a diagram for explaining the interval between the representative points of the bright points between the adjacent element images in the element image group. For example, as shown in the enlarged view, the coordinates of the representative points P1 to P5 of the five bright points in the element image group are known in advance, so the interval Δ EI1 , Δ EI2 , Δ EI3 , Δ EI4 The interval Δ EI1 , Δ EI2 , Δ EI3 , Δ EI4 , …, Δ EIn The average of ￣ Δ EIt Then, from equations (1) and (2), the Z coordinate of the object identified by the label at time t is expressed by the following equation: z t =d obj =(( ￣ Δ EIt -EI L )a-MLP*b) / ( ￣ Δ EIt -EI L )M 2

[0050] The Z coordinate of the target object at time t may be calculated using a model formula other than the above formula. This requires that the spacing Δ between elemental images as shown in FIG. 3(b) is previously determined in relation to the characteristics of the light field camera 10 to be actually used. EI and the depth, which is the deviation from the object-side focal plane NOP of the objective lens 70 as shown in FIG. 4, is d objThe relationship between these two is measured and fitted to the model formula. By doing so, the Z coordinate of the object at time t can be calculated using the model formula. ￣ Δ EIt can be found directly from

[0051] Figure 13 is a graph showing tracking performance in the Z direction. Tracking performance in the Z direction was confirmed by capturing images of a fluorescent particle with a radius of 2 μm as it was moved in 1 μm increments in the Z direction from 0 μm to 40 μm on an XYZ motorized stage using the light field camera 10, and analyzing the light field images using the light field image analysis unit 30. (a) shows the input to the motorized stage carrying the target (fluorescent particle), and (b) shows the results of position tracking of the target (fluorescent particle) by the light field image analysis unit 30. Position tracking is not possible near Z = 0 (NOP) in (b) because refocusing is difficult in that area due to the characteristics of the light field camera 10. It can be said that the movement of the target in the Z direction was well tracked at other depth positions where refocusing is possible.

[0052] The light field image analysis unit 30 first (at time t=1) reconstructs the object in three dimensions from the elemental image group to determine its initial coordinates, but thereafter (from time t=2 onward), the light field image analysis unit 30 tracks the three-dimensional spatial position of the object by analyzing only the elemental image group of the two-dimensional image without three-dimensionally reconstructing the object from the elemental image group. Figure 14 is a diagram showing how the light field image analysis unit 30 tracks the object. For example, after determining the initial coordinates (x, y, z) = (553, 362, 44) of particle A, the object, from the first elemental image group, the light field image analysis unit 30 analyzes the elemental image group of the two-dimensional image to track the three-dimensional spatial position of particle A without three-dimensionally reconstructing particle A.

[0053] In this way, the light field image analysis unit 30 can track the three-dimensional spatial position of an object in an image from a light field image with a high frame rate, such as continuous data of a group of elemental images, making it possible to observe high-speed events such as biomolecular and intracellular behavior. For example, the three-dimensional spatial position of an object can be tracked by analyzing a light field image at 100 fps.

[0054] The bright spot group centroid calculation unit 37 analyzes the light field image independently of the image analysis described above. Specifically, the bright spot group centroid calculation unit 37 detects microlens image groups containing pixels with a certain brightness or pixel value in the light field image read from the storage device 20, determines the smallest circle containing the microlens image groups, and calculates the coordinates of its center. For example, if there is one target object, the smallest circle containing the microlens image groups containing the target object is determined and its center is calculated. On the other hand, if there are multiple targets, the microlens image groups containing each target object are scattered. The smallest circle containing the microlens image groups containing each target object is individually determined, and the center of gravity of the centers of these circles is calculated. Alternatively, the smallest circle containing all the scattered microlens image groups may be determined and its center may be calculated. In this way, the bright spot group centroid calculation unit 37 directly and easily calculates the coordinates of the target object from each light field image, without tracking the movement of the target object from consecutive light field images and calculating its coordinates.

[0055] <Motorized stage control> Next, a description will be given of the control of the motorized stage 90 by the stage control unit 40. Fig. 15 is a flowchart of the motorized stage control. Each step is executed by the above-mentioned components of the stage control unit 40.

[0056] The XY control unit 41 acquires the XY coordinates of the object from the light field image analysis unit 40 and determines the error between the acquired XY coordinates and the center of the field of view, i.e., the XY coordinates of the center of the light field image (S11). The acquired XY coordinates of the object may be updated by the three-dimensional coordinate update unit 36 ​​or by the bright spot group centroid calculation unit 37. For example, assuming that the light field image represents a square field of view with length and width L, the upper left corner of the light field image is the origin, and the XY coordinates of the object in the light field image are (x, y), the error of the object from the center of the field of view (L / 2, L / 2) can be determined as a vector expressed as (L / 2-x, L / 2-y).

[0057] Once the error between the X and Y coordinates of the object and the center of the field of view is determined, the XY control unit 41 controls the movement of the motorized stage 90 in the X and Y directions in accordance with the error (S12). Specifically, the XY control unit 41 controls the movement of the motorized stage 90 in the X and Y directions using the error as an X and Y control amount. Note that, depending on the configuration of the light field camera 10, objective lens 70, and other optical systems, the top, bottom, left, and right of the light field image may be reversed from the actual field of view. In such cases, it is necessary to invert the sign of the error to control the movement of the motorized stage 90 in the X and Y directions.

[0058] The XY-directional movement of the motorized stage 90 may be controlled by multiplying the error by a gain between 0 and 1. A large gain allows the object to be quickly moved toward the center of the field of view, but the control amount may be too large, causing the object to move beyond the center of the field of view in a single control, necessitating repeated control to move the object in the opposite direction, potentially resulting in a long time required to bring the object to the center of the field of view. On the other hand, a small gain prevents the object from moving beyond the center of the field of view in a single control, but the amount of object movement in a single control is small, potentially resulting in a long time required to bring the object to the center of the field of view. For this reason, it is desirable to appropriately determine the gain based on various conditions, such as the agility of the object's movement, the performance of the motorized stage 90, and the magnification of the objective lens 70.

[0059] The gain may be varied depending on the error. For example, if the error is large, the gain may be increased to quickly move the object toward the center of the field of view, and as the error becomes smaller, the gain may be decreased to gradually move the object toward the center of the field of view. This allows the object to be brought to the center of the field of view more quickly when it is significantly deviated from the center of the field of view.

[0060] In parallel with or asynchronously with steps S11 and S12, the Z control unit 42 receives from the light field image analysis unit 40 the Z coordinate of the target object or the average value of the intervals between representative points of bright points with the same label between adjacent element images. ￣ Δ EIt The error between the Z coordinate and the specified Z coordinate or the error between the average value and the specified interval value is calculated (S13). ￣ Δ EIt is calculated by the three-dimensional coordinate update unit 36. Then, the Z control unit 42 controls the movement of the motorized stage 90 in the Z direction in accordance with the error (S14). Specifically, the Z control unit 41 controls the movement of the motorized stage 90 in the Z direction using the error as a Z control amount. Alternatively, the Z control unit 41 controls the movement of the motorized stage 90 in the Z direction by using the average ￣ Δ EIt If is smaller than the specified interval value, the average value ￣ Δ EIt The motorized stage 90 is controlled in a direction away from the objective lens 70 until the average value ￣ Δ EIt If is greater than the specified interval value, the average value ￣ Δ EIt Alternatively, the motorized stage 90 may be controlled in a direction approaching the objective lens 70 until the distance between the objective lens 70 and the stage 90 reaches a specified distance value.

[0061] <3D tracking processing parallelization> The processes performed by the components of the light field image analysis unit 30 and the stage control unit 40 can be performed in parallel. For convenience, the process by which the light field image analysis unit 30 reads a light field image from the storage device 20 will be referred to as the "image acquisition thread," the process by which the light field image analysis unit 30 narrows the analysis range of the read light field image to construct a group of element images will be referred to as the "microlens image (ML) detection thread," the process by which the bright spot representative point identification unit 33 determines the representative point of each bright spot will be referred to as the "bright spot detection thread," the process by which the bright spot tracking unit 34 determines the position change of the representative point of each bright spot, the process by which the bright spot labeling unit 35 labels the representative point of each bright spot, and the process by which the three-dimensional coordinate update unit 36 ​​updates the three-dimensional coordinates of the target object will be referred to as the "feature extraction thread," the process by which the XY control unit 41 controls the movement of the motorized stage 90 in the X and Y directions will be referred to as the "XY control thread," the process by which the Z control unit 42 controls the movement of the motorized stage 90 in the Z direction will be referred to as the "Z control thread," and the process by which the bright spot group centroid calculation unit 37 calculates the center coordinates of the smallest circle that contains the microlens image group will be referred to as the "centroid calculation thread." These threads can be operated in parallel so that they receive the processing results of other threads, perform predetermined processing, and then pass the processing results to other threads.

[0062] FIG. 16 is a timing chart of an example of three-dimensional tracking parallel processing. In this parallel processing, the XY control unit 41 controls the XY-directional movement of the motorized stage 90 based on the XY coordinates of the target object updated by the three-dimensional coordinate update unit 36. FIG. 17 is a timing chart of another example of three-dimensional tracking parallel processing. In this parallel processing, the XY control unit 41 controls the XY-directional movement of the motorized stage 90 based on the XY coordinates of the target object updated by the bright spot group centroid calculation unit 37. The time indicated in parentheses following each thread name is the processing time of each thread. In either case, parallel operation of the processes by the light field image analysis unit 30 and the stage control unit 40 enables three-dimensional tracking of the target object more quickly than sequential processing of steps S6 to S14 shown in FIGS. 6B and 15. The XY control thread and the Z control thread are the speed-limiting factors for the parallel processing, but the actual processing times of the XY control thread and the Z control thread may vary depending on the magnitude of the control amount. In particular, once the target object is brought to the center of the field of view and to the specified Z coordinate, it can be tracked with very little control in each of the X, Y, and Z directions, which shortens the processing time of the XY control thread and the Z control thread, and parallel processing of the threads enables faster three-dimensional tracking.

[0063] Effect Figure 18 is a graph comparing the trajectory of a freely moving object (nematode) with the size of the field of view of a high-magnification objective lens. The vertical and horizontal axes of the graph represent positions in the X and Y directions, measured in millimeters. As can be seen from the graph, the object moves freely within an area approximately 700 μm square. The square shown in the graph represents the size of the field of view of a 40x objective lens, measuring 160 μm in length and width, which is much narrower than the range of movement of the object. Therefore, it is difficult to continuously observe this phenomenon with a 40x objective lens, and a low-magnification objective lens of approximately 10x magnification would be necessary. On the other hand, with the light field microscope system 100 according to this embodiment, the motorized stage 90 is controlled in accordance with the object's movement, allowing the object to be constantly captured at the center of the field of view. Moreover, although not shown in the graph of Figure 18, the Z-directional movement of the motorized stage 90 is controlled to follow the object's movement in the Z direction, allowing the object to be constantly captured at a constant depth of field. This allows for continuous observation of a moving object over a wide area using a high magnification objective lens of 40x, 60x or more, capturing the object in greater detail and in the center of the field of view.

[0064] <<Variations>> Instead of reconstructing a three-dimensional image of the object and determining the initial coordinates of the object from that three-dimensional image as explained in Figure 7, the three-dimensional coordinates of the object may be determined directly from an image reconstructed from the initial group of elemental images. Since the Z coordinate of the reconstructed image is known in advance, the initial coordinates of the object in three-dimensional space can be determined by determining the coordinates of the center of gravity of the object in the reconstructed image, i.e., the XY coordinates.

[0065] The template selection unit 32 may update the template by selecting a template not only from the first element image group but also from the second and subsequent element image groups as appropriate. In particular, when there are multiple targets that each rotate or change direction, it is preferable to reselect a template from the second and subsequent element image groups.

[0066] The above explanation is based on the premise that the element images in the element image group are arranged in a square lattice pattern, but when a microlens array with microlenses arranged in a honeycomb pattern is used, the element images may be arranged in a honeycomb pattern. Even in such a case, the above-mentioned identification of representative points of bright spots in the element image group, bright spot labeling, template selection, bright spot tracking, and three-dimensional coordinate updating can be applied.

[0067] As described above, the embodiments have been described as examples of the technology of the present invention. For this purpose, the accompanying drawings and detailed description have been provided. Therefore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the above technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential. Furthermore, because the above-described embodiments are intended to exemplify the technology of the present invention, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Explanation of symbols]

[0068] 100 Light Field Microscope System 110 Light Field 3D Tracking Control Device 10 Light Field Camera 20 Storage device 30 Light field image analysis unit 31 Image reconstruction unit 32 Template Selection Section 33 Bright spot representative point identification part 34 Bright spot tracking unit 35 Bright spot labeling section 36 3D coordinate update section 37 Bright spot group centroid calculation section 40 Stage control section 41 XY control section 42 Z control section 50 Full-field illumination system 70 objective lenses 80 Sample Holder 90 Motorized Stage

Claims

1. A light field three-dimensional tracking control device that controls an electric stage that moves in X, Y, and Z directions, and three-dimensionally tracks an object in a sample holder placed on the electric stage, the object being observed by a light field camera through an objective lens, a light field image analysis unit that analyzes light field images obtained by capturing time-continuous images of the object using the light field camera and tracks a three-dimensional spatial position of the object in the light field images; a stage control unit that controls the movement of the motorized stage in the X, Y, and Z directions so that a relative positional relationship between the objective lens and a three-dimensional spatial position of the target tracked by the light field image analysis unit is kept constant; A light field 3D tracking control device comprising:

2. The light field image analysis unit an image reconstruction unit that reconstructs an image refocused at a depth position of the object from a group of element images of the initial light field image; a template selection unit that selects an element image including the most bright points as a template from a group of element images of the initial light field image; a bright point representative point identifying unit for determining a representative point of each bright point in the element image group of the second and subsequent light field images; a bright point tracking unit for determining a change in the position of a representative point of each bright point in a group of element images of the second and subsequent light field images; a bright point labeling unit that assigns a label for identifying the target to a representative point of each bright point consisting of a group of pixels corresponding to the target in the reconstructed image in the elemental image group of the first light field image, and assigns the same label as the representative point of each bright point in the template to a representative point of each bright point in each elemental image in the elemental image group of the second or subsequent light field images. a three-dimensional coordinate updating unit that obtains the three-dimensional spatial coordinates of the object in the reconstructed image and sets the coordinates as the initial coordinates of the object identified by the label, updates the XY coordinates of the object identified by the label based on a position change of a representative point of a bright point with the same label in a group of elemental images of the second and subsequent light field images, and updates the Z coordinate of the object based on the interval between the representative points of the elemental images. The light field three-dimensional tracking control device according to claim 1 .

3. The stage control unit an XY control unit that acquires XY coordinates of the object from the light field image analysis unit, calculates an error between the XY coordinates and the XY coordinates of a center of a field of view, and controls the movement of the motorized stage in the XY directions in accordance with the error; and a Z control unit that acquires the Z coordinate of the object or an average value of intervals between representative points of bright points with the same label between the adjacent element images from the light field image analysis unit, calculates an error between the Z coordinate and a specified Z coordinate or an error between the average value and a specified interval value, and controls the movement of the motorized stage in the Z direction in accordance with the error. The light field three-dimensional tracking control device according to claim 2 .

4. a thread for reading the light field image temporarily stored in a storage device from the storage device; a thread for determining a representative point of each bright point from the bright point representative point identifying unit; a thread for determining a position change of a representative point of each of the bright points by the bright point tracking unit, labeling the representative point of each of the bright points by the bright point labeling unit, and updating the three-dimensional coordinates of the object by the three-dimensional coordinate updating unit; a sled that controls the movement of the motorized stage in the XY directions by the XY control unit; and The Z control unit processes threads in parallel to control the movement of the electric stage in the Z direction. The light field three-dimensional tracking control device according to claim 3 .

5. the light field image analysis unit has a bright spot group centroid calculation unit that detects a microlens image group including pixels having a certain luminance or pixel value in the light field image, determines the smallest circle that contains the microlens image group, and calculates the center coordinates of the smallest circle; The XY control unit of the stage control unit controls the movement of the electric stage in the XY direction using the coordinates calculated by the bright spot group centroid calculation unit as the XY coordinates of the object. The light field three-dimensional tracking control device according to claim 3 .

6. a thread for reading the light field image temporarily stored in a storage device from the storage device; a thread for calculating the center coordinates of the smallest circle by the bright spot group center of gravity calculation unit; a thread for determining a representative of each bright spot from the bright spot representative part identification unit; a thread for determining a position change of a representative point of each bright point by the origin tracking unit, labeling each bright point by the origin labeling unit, and updating the three-dimensional coordinates of the target object by the three-dimensional coordinate update unit; a sled that controls the movement of the motorized stage in the XY directions by the XY control unit; and The Z control unit processes threads in parallel to control the movement of the electric stage in the Z direction. The light field three-dimensional tracking control device according to claim 5 .

7. The XY control unit controls the movement of the electric stage in the XY directions with a control amount obtained by multiplying the error by a gain corresponding to the error.

7. The light field three-dimensional tracking control device according to claim 3, wherein the light field three-dimensional tracking control device is a light field three-dimensional tracking control device.

8. A light field three-dimensional tracking control method for controlling an electric stage movable in X, Y, and Z directions to three-dimensionally track an object in a sample holder placed on the electric stage and observed by a light field camera through an objective lens, comprising: a step of analyzing light field images obtained by capturing time-sequential images of the object using the light field camera, and tracking a three-dimensional spatial position of the object in the light field images; controlling the movement of the motorized stage in the XYZ directions so that a relative positional relationship between the objective lens and a three-dimensional spatial position of the target tracked by the light field image analysis unit is kept constant; A light field three-dimensional tracking control method comprising:

9. A program that causes a computer to control an electric stage that is movable in X, Y, and Z directions, and three-dimensionally tracks an object in a sample holder placed on the electric stage and observed by a light field camera through an objective lens, a means for analyzing light field images obtained by capturing time-sequential images of the object using the light field camera, and tracking a three-dimensional spatial position of the object in the light field images; and a means for controlling the movement of the motorized stage in the X, Y, and Z directions so that a relative positional relationship between the objective lens and the three-dimensional spatial position of the target object tracked by the light field image analysis unit is kept constant; A program that makes a computer function as a

10. An objective lens, A light field camera and a full-field illumination system that illuminates the full field of view of the light field camera; an electric stage movable in X, Y and Z directions on which a sample holder containing an object to be observed by the light field camera is placed; A light field three-dimensional tracking control device according to any one of claims 1 to 6; A light field microscope system equipped with

11. An objective lens, A light field camera and a full-field illumination system that illuminates the full field of view of the light field camera; an electric stage movable in X, Y and Z directions on which a sample holder containing an object to be observed by the light field camera is placed; The light field three-dimensional tracking control device according to claim 7 ; A light field microscope system equipped with

Images