Selective irradiation control device, selective irradiation control method, program, and selective irradiation system

Abstract

This selective irradiation control device (110) controls a spatial light modulator (64), which spatially modulates incident light to form an arbitrary spatial light pattern, to selectively irradiate an object with the light. The selective irradiation control device (110) comprises: a light field image analysis unit (30) that analyzes a light field image obtained by imaging an object with a light field camera (10) temporally continuously, and tracks the three-dimensional space position of the object in the light field image; and a spatial light modulator control unit (40) that generates a computer hologram for forming a spatial light pattern for selectively irradiating, with light, the three-dimensional space position of the object tracked by the light field image analysis unit (30), and imparts the generated computer hologram to the spatial light modulator (64).

Description

Selective irradiation control device, selective irradiation control method and program, and selective irradiation system

[0001] The present invention relates to a selective illumination control device, a selective illumination control method and program, and a selective illumination system, and in particular to a selective illumination technology that acquires and tracks the three-dimensional spatial position of an object in an image from a light field image that records a light field, and selectively irradiates the object with light.

[0002] High-speed, large-scale three-dimensional imaging is required for delicate and dynamic observation of biomolecules and intracellular behavior. Typically, such three-dimensional images are generated using a confocal microscope by continuously shifting the focal position of the sample in the depth direction (Z direction) at a predetermined interval and capturing images of the sample at each focal position (see, for example, Patent Documents 1 and 2). Another microscope that enables three-dimensional imaging is the spinning disk confocal microscope. The spinning disk confocal method uses a rotating disk with an array of pinholes to irradiate laser light, creating multiple parallel beams of light, which are then used to rapidly scan the sample to form a confocal image.

[0003] JP 2011-044016 A JP 2014-157158 A

[0004] Three-dimensional imaging using confocal microscopes requires scanning in the depth direction, which takes time to measure and makes it unsuitable for observing high-speed events. For example, if a three-dimensional space is imaged by 30 scans in the Z direction at 30 fps, the imaging speed for the entire three-dimensional space is 30 fps / 30 = 1 vps, which means it takes one second to capture the three-dimensional space. Because neural activity is a phenomenon on the order of milliseconds, three-dimensional imaging using confocal microscopes is insufficient in terms of speed for observing biomolecules and intracellular behavior.

[0005] Furthermore, because the data size per unit volume of 3D imaging is extremely large, when such volumetric images are converted into time-series video data, the data size can reach several terabytes. Such large amounts of data take a very long time to transmit even using high-speed wired communication, and may be virtually impossible to transmit using wireless communication, which has even slower transmission speeds.

[0006] Light field microscopy (LFM) is another option for three-dimensional imaging. A light field represents a collection of light rays in three-dimensional space. LFM uses a microlens array, consisting of a two-dimensional array of numerous microlenses, positioned at an intermediate image plane. This allows recording of the light field by determining the positions of light rays passing through the aperture stop of the objective lens and their capture positions on the image sensor. From the light field image recorded by LFM, subaperture images, equivalent to those captured by a small-aperture lens, can be obtained. Subaperture images are partial images of a subject captured from a shifted viewpoint. Each subaperture image has a deep depth of field due to the pinhole effect, and also has parallax between them. Therefore, by shifting and superimposing the subaperture images, a refocused image at any depth can be reconstructed.

[0007] As described above, LFM is suitable for observing high-speed events because it can capture three-dimensional images in a single shot at high speed without scanning in the depth direction. Furthermore, in terms of data size, the light field images themselves are two-dimensional, so the data size is small, and the data size of the video data, which is a series of images over time, is also small.

[0008] Furthermore, when observing biomolecules or intracellular behavior, or in the field of optogenetics, it is important to highlight the target object and irradiate the target object with light precisely at its three-dimensional spatial position.

[0009] Therefore, an object of the present invention is to track the three-dimensional spatial position of extremely small moving objects such as biomolecules and cells at high speed and selectively irradiate the objects with light.

[0010] According to one aspect of the present invention, there is provided a selective illumination control device that controls a spatial light modulator that spatially modulates incident light to form an arbitrary spatial light pattern, thereby selectively irradiating light onto an object, the selective illumination control device comprising: a light field image analysis unit that analyzes light field images of the object captured continuously over time by a light field camera, and tracks the three-dimensional spatial position of the object in the light field images; and a spatial light modulator control unit that generates a computer-generated hologram for forming a spatial light pattern that selectively irradiates light onto the three-dimensional spatial position of the object tracked by the light field image analysis unit, and provides the generated computer-generated hologram to the spatial light modulator. A selective illumination control method and a computer program corresponding thereto are also provided.

[0011] According to another aspect of the present invention, there is provided a selective illumination system including the above-mentioned selective illumination control device, a light field camera that captures a light field image of an object and supplies the image to the selective illumination control device, a full-field illumination system that has a light source and an optical system and illuminates the full field of view of the light field camera, and a selective illumination system that has a light source, an optical system, and a spatial light modulator controlled by the selective illumination control device and selectively irradiates the object with light.

[0012] According to the present invention, the three-dimensional spatial position of an object can be acquired and tracked at high speed based on a light field image of the object, and light can be irradiated onto the object at high speed and with pinpoint accuracy.

[0013] 1 is a schematic diagram of a selective illumination system according to one embodiment of the present invention; FIG. 2 is a block diagram of a selective illumination control device according to one embodiment of the present invention; FIG. 3 is a diagram illustrating (a) a light field image, (b) a group of elemental images, and (c) a reconstructed image according to an example; FIG. 4 is a diagram illustrating the geometric relationship between the parameters of a light field camera according to an example; FIG. 5 is a diagram illustrating the relationship between a PQ array and overlaying elemental images; FIG. 6 is a first half of a flowchart for light field image analysis; FIG. 7 is a second half of a flowchart for light field image analysis; FIG. 8 is a diagram illustrating an example of three-dimensionally reconstructing an object from a first group of elemental images, performing bright spot labeling, setting initial coordinates of the object, and selecting a template; FIG. 9 is a diagram illustrating the identification of a representative point of each bright spot in a group of elemental images; FIG. 10 is a diagram illustrating tracking of a representative point of each bright spot in a group of elemental images; FIG. 11 is a diagram illustrating bright spot labeling in a group of elemental images; FIG. 12 is a diagram illustrating template selection when there are multiple bright spots in one elemental image, and bright spot labeling in second and subsequent group of elemental images; FIG. 13 is a diagram illustrating the spacing between representative points of bright spots between adjacent elemental images in a group of elemental images; FIG. 14 is a graph showing tracking performance in the Z direction. 1 is a diagram showing how an object is tracked by a light field image analysis unit; 2 is a flowchart of spatial light modulator control; 3 is a profile graph of a microscope image with and without selective illumination and pixel values ​​near the object; 4 is another microscope image with and without selective illumination and pixel values ​​near the object;

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present invention, and are not intended to limit the subject matter described in the claims. Furthermore, the dimensions of each component depicted in the drawings, the detailed shapes of the details, and the like may differ from the actual ones.

[0015] <<Embodiment of Selective Illumination System>> Fig. 1 is a schematic diagram of a selective illumination system according to one embodiment of the present invention. The selective illumination system 100 according to this embodiment can pinpoint fluorescence excitation light, activation light, or the like onto minute objects such as biomolecules or cells observed through an objective lens. Generally, the selective illumination system 100 includes a light field camera 10, a full-field illumination system 50, a selective illumination system 60, an objective lens 70, a selective illumination control device 110, and several other optical components. For ease of explanation, it is assumed that a sample containing the object is adjusted to a sample holder 80 such as a glass slide or a petri dish and placed on a stage (not shown), and that the objective lens 70 is further positioned above it.

[0016] The full-field illumination system 50 includes a light source 51 and a collimator lens 52. The light source 51 supplies light for illuminating the full field of view of the light field camera 10. The light emitted from the light source 51 may be natural light, including white light and infrared light, or laser light with a specific wavelength, such as 470 nm or 550 nm, for causing the target to fluoresce. The collimator lens 52 collimates the light emitted from the light source 51. A half mirror 71, a lens 72, a dichroic mirror 73, and an objective lens 70 are sequentially arranged ahead of the collimated light in the traveling direction. Of these, the half mirror 71 and the dichroic mirror 73 are configured to transmit the wavelength range of the light from the light source 51. The light collimated by the collimator lens 52 passes through the half mirror 71, the lens 72, and the dichroic mirror 73 and enters the objective lens 70, thereby uniformly illuminating the sample holder 80 as a whole. In this manner, the full-field illumination system 50 illuminates the full field of view of the light field camera 10.

[0017] The selective irradiation system 60 includes a light source 61, a collimator lens 62, a mirror 63, a spatial light modulator 64, a Fourier transform lens 65, and a DC component removal filter 66. The light source 61 provides light to irradiate the target in the sample in the sample holder 80 to cause the target to fluoresce or photoactivate. The wavelength of the laser light emitted from the light source 61 may be the same as that of the light source 51, or it may be a laser light with a different wavelength, for example, a wavelength of 473 nm. By using a different wavelength from the light source 51, the half mirror 71 can transmit the light from the light source 51 and reflect the light from the spatial light modulator 64. It is desirable to use a light source with as good coherence as possible for the light source 61, such as a solid-state or gas laser light source. The collimator lens 62 collimates the laser light emitted from the light source 61. A mirror 63 and a spatial light modulator 64 are sequentially arranged ahead of the collimated laser light in the traveling direction. The laser light collimated by the collimator lens 62 is reflected by a mirror 63 and enters a spatial light modulator 64 .

[0018] The spatial light modulator 64 is a device that spatially modulates the incident light by modulating the phase or intensity of the incident light to form an arbitrary spatial light pattern; specifically, it is a reflective liquid crystal on silicon spatial light modulator (LCOS-SLM). The spatial light pattern formed by the spatial light modulator 64 is determined by a computer-generated hologram. As will be described later, a computer-generated hologram (CGH) is provided to the spatial light modulator 64 from the selective illumination control device 110.

[0019] Arranged in this order ahead of the light modulated by the spatial light modulator 64 in the direction of travel are a Fourier transform lens 65, a DC component removal filter 66, a half mirror 71, a lens 72, a dichroic mirror 73, and an objective lens 70. The half mirror 71 reflects the wavelength range of the laser light from the light source 61, while the dichroic mirror 73 transmits it. These optical systems allow the pupil plane of the objective lens 70 to be image-transferred to the modulation plane of the spatial light modulator 64. The DC component removal filter 66 is positioned at the focal length of the Fourier transform lens 65 and removes the DC component of the light modulated by the spatial light modulator 64, i.e., light that is not modulated by the spatial light modulator 64 and would otherwise cause unwanted images to form. With this configuration, the selective illumination system 60 can form a desired light pattern on the sample in the sample holder 80 and irradiate targets in the sample with pinpoint light.

[0020] The light field camera 10 includes an imaging lens (tube lens) 1, a microlens array 2, and an image sensor 3. An objective lens 70, a dichroic mirror 73, the imaging lens 1, and the microlens array 2 are sequentially arranged ahead of the sample in the sample holder 80 in the direction of travel of the fluorescence. The dichroic mirror 73 reflects the wavelength range of the fluorescence from the sample. The microlens array 2 is configured by a two-dimensional array of multiple microlenses (lenslets) 2a and is positioned between the imaging lens 1 and the image sensor 3, specifically, at the location of the image formed by the objective lens 70 and the imaging lens 1. The image sensor 3 is a CCD sensor or CMOS sensor, etc., configured by a two-dimensional array of multiple light-receiving elements (not shown) that photoelectrically convert input light and output electrical signals. The electrical signals output from each light-receiving element are converted into digital data by an A / D converter (not shown), and a light field image is output from the light field camera 10. The light field image is supplied to the selective irradiation control device 110.

[0021] <<Embodiment of Selective Irradiation Control Device>> The selective irradiation control device 110 can be configured as a computer device equipped with a storage device and a processor. Fig. 2 is a block diagram of a selective irradiation control device according to one embodiment of the present invention. The selective irradiation control device 110 includes, as its main components, a storage device 20, a light field image analysis unit 30, and a spatial light modulator control unit 40. These components are connected to each other so that they can communicate with each other and exchange data.

[0022] The storage device 20 is a collection of storage devices such as RAM, ROM, SSD, and HDD. The RAM is primarily used as a working memory when the selective irradiation control device 110 performs various calculation processes. It also temporarily stores light field images captured continuously by the light field camera 10 and supplied to the selective irradiation control device 110, reconstructed images generated and used in the image analysis described below, three-dimensional images, and label tables. The ROM, SSD, and HDD primarily store computer programs for operating a computer as the light field image analysis unit 30 and the spatial light modulator control unit 40, or computer programs stored on computer-readable, non-transitory recording media, as well as PQ arrays referenced by the light field image analysis unit 30 and spatial light pattern / computer-generated hologram correspondence sets referenced by the spatial light modulator control unit 40, for long-term storage.

[0023] The light field image analysis unit 30 is a module that analyzes light field images that are captured continuously over time by the light field camera 10 and buffered in the storage device 20, and tracks the three-dimensional spatial position of an object in the light field image. More specifically, the light field image analysis unit 30 includes the following components: an image reconstruction unit 31, a template selection unit 32, a bright point representative point identification unit 33, a bright point tracking unit 34, a bright point labeling unit 35, and a three-dimensional coordinate update unit 36.

[0024] The spatial light modulator control unit 40 is a module that generates a computer-generated hologram to form a spatial light pattern that selectively irradiates light onto the three-dimensional spatial position of an object tracked by the light field image analysis unit 30, and provides the generated computer-generated hologram to the spatial light modulator 64.

[0025] The light field image analysis unit 30 and the spatial light modulator control unit 40 can be realized as hardware such as an ASIC or FPGA, or as software in which a computer program stored in a recording medium (not shown) or storage device 20 is executed by a processor (not shown) such as a CPU or GPU included in the control device 110. They can also be realized by appropriately combining hardware and software.

[0026] The light field image to be analyzed by the image analysis unit 30 includes an array of elemental images, each of which is an arrangement of elemental images captured of the same subject from multiple, mutually offset viewpoints. From such an elemental image group, a refocused image at any depth can be reconstructed using the parallax of each elemental image included therein. FIG. 3 illustrates an example of a light field image, elemental images, and a reconstructed image. (a) The light field image is captured by placing a sheet with the character " / 5" written on it at a position offset from the focal plane of the light field camera 10. The light field image is a group of microlens images MI, in which numerous microlens images MI captured by each microlens 2a of the microlens array 2 of the light field camera 10 are arranged in a square lattice pattern. As indicated by the dashed line, each microlens image MI has an approximately circular shape, reflecting the circular shape of each microlens 2a. Thus, a light field image is a collection of multiple sub-images (microlens images MI in the example of FIG. 3 ) captured of the same subject from multiple, mutually offset viewpoints.

[0027] The area near the boundary of the microlens image MI is dark due to the small amount of light, and since the area is significantly distorted due to capturing light rays that have passed through the edge of the microlens 2 a, the central part of the microlens image MI is used. That is, from each approximately circular microlens image MI, for example, a square area inscribed in the circle is cut out as an element image EI, and these element images EI are arranged in a square lattice pattern to form the (b) element image group.

[0028] As described above, since the sheet on which the character " / 5" is written is located at a position shifted from the focal plane of the light field camera 10, (a) microlens images are recorded in the light field image by shifting the viewpoint of each microlens 2a to partially capture the character " / 5," i.e., microlens images with parallax. For example, if one focuses on the upper left corner of the character "5," the light beam emitted from that part of the subject is recorded with a distance Δ between adjacent microlens images. (b) As shown in the elemental image group, the distance Δ between the microlens images is Δ between the elemental images. ＥＩ Here, the pitch of the microlenses 2a is MLP, and the length of one side of the element image is EI. Ｌ Then, Δ ＥＩ is expressed by the following equation (1): Δ ＥＩ = Δ - (MLP-EI Ｌ ) … (1)

[0029] The reconstructed image (c) is obtained by superimposing adjacent element images in the element image group while shifting them by a shift amount according to the parallax. ＥＩ By shifting the elemental images by an appropriate amount and superimposing them, a reconstructed image (c) is obtained that is refocused at the position of the sheet with the letters " / 5" written on it. In this way, by superimposing the elemental images while appropriately changing the shift amount, it is possible to reconstruct an image refocused at any depth position based on ray tracing in the reverse direction from the acquired image.

[0030] Here, the depth of the object to be refocused, which is the deviation from the object-side focal plane NOP (Native Object Plane) of the objective lens, is defined as d ｏｂｊThen, the distance Δ between rays originating from the same point between adjacent microlens images and the depth d of the point are ｏｂｊ The relationship between the parameters of the light field camera 10 and the objective lens 70 can be expressed using the parameters of the light field camera 10. FIG. 4 is a diagram schematically illustrating the geometric relationship between the parameters of the light field camera 10 according to an example. For convenience, the objective lens 70 and the imaging lens 1 shown in FIG. 1 are depicted as one unit in FIG. 4. O in the diagram indicates the depth d ｏｂｊ A is the image point of the object O formed by the objective lens 70 and the imaging lens 1, B and C are the centers of the adjacent microlenses 2a, and B' and C' are the positions of the light rays from the object O captured by the image sensor 3 via B and C. When we focus on the triangles ABC and AB'C' in the figure, the similarity relationship between triangles gives us BC:B'C'=CA:C'A. This can be expressed in terms of parameters as follows: MLP:Δ=(a+d ｏｂｊ M ２ ): (a + d ｏｂｊ M ２ -b) . By solving this for Δ, the following equation (2) is obtained: Δ = MLP(1-b / (d ｏｂｊ M ２ -a)) ... (2) where MLP is the pitch of the microlenses 2a, M is the magnification of the objective lens 70, a is the distance from the image-side focal plane T-NIP (Native Image Plane) of the imaging lens 1 to the center of the microlens 2a, and b is the distance from the center of the microlens 2a to the image sensor 3. Note that in FIG. 4, a is considered to be a positive value to show the geometric relationship, but if the direction from the object side to the image side is considered to be positive, a needs to be considered to be a negative value. For this reason, in equation (2), a is finally replaced with -a.

[0031] Note that since there is no anisotropy in two mutually perpendicular directions (X direction, Y direction) in a plane perpendicular to the depth direction (Z direction), the same Δ can be applied to both the X direction and the Y direction.

[0032] From equations (1) and (2), ｏｂｊ and Δ ＥＩ Therefore, at any depth position (depth d ｏｂｊ), the element images are shifted by a shift amount Δ ＥＩ The elemental images can be easily superimposed by referring to the PQ array described below.

[0033] FIG. 5 illustrates the relationship between the PQ array and the overlapping of element images. Overlapping pixels are shaded; the left column shows the overlapping state of the element images, and the right column shows the overlapped state of the element images expanded. For convenience, the element image group consists of four element images, D, E, F, and G, arranged two by two, with each element image measuring five pixels vertically and horizontally. The element images are arranged as follows: element image E is to the right of element image D, element image F is below element image D, and element image G is below element image E, i.e., to the right of element image F. To reference each pixel in the element image group, rows and columns are assigned consecutive numbers starting from 1. For example, pixels belonging to element image D are referenced in the range from row 1 to row 5 and column 1 to column 5, while pixels belonging to element image G are referenced in the range from row 6 to row 10 and column 6 to column 10. Hereinafter, pixel [m, n] refers to the pixel located in row m and column n in the element image group.

[0034] Z in the figure is d ｏｂｊ For convenience, d is a depth parameter that represents ｏｂｊ and the shift amount of the element image Δ ＥＩ are expressed by the same Z value. That is, when Z = n, d ｏｂｊ = n, and the shift amount of the element image Δ ＥＩ represents n pixels.

[0035] When Z=0, that is, the depth d ｏｂｊ If .gt.=0, the elemental images are not superimposed on each other, and the elemental images are directly used as a reconstructed image.

[0036] When Z=1, that is, the depth d ｏｂｊTo reconstruct an image refocused to Z=1, adjacent element images are shifted by one pixel and then superimposed. Which pixels in the element image group should be superimposed can be easily determined by referring to a PQ array, which indicates how the pixels are superimposed. The PQ array is a representation of the row or column numbers of pixels in the element image group, folded over for each element image. As shown in the figure, for example, the PQ array corresponding to Z=1 is expressed as follows: 1 2 3 4 5 6 7 8 9 10 This PQ array means that the pixel in the fifth row (fifth column) and the pixel in the sixth row (sixth column) overlap. By preparing such PQ arrays for each Z value in advance and storing them in the storage device 20, pixel superposition can be performed by referencing the PQ array corresponding to the Z value.

[0037] When Z=2, that is, the depth d ｏｂｊ To reconstruct an image refocused to Z=2, adjacent element images are shifted by two pixels and then overlapped. As shown in the figure, for example, the PQ array corresponding to Z=2 is expressed as follows: 1 2 3 4 5 6 7 8 9 10 This PQ array means that the pixels in the 4th and 5th rows (4th and 5th columns) and the pixels in the 6th and 7th rows (6th and 7th columns) overlap each other.

[0038] Note that there may be cases where three or more element images overlap in the column and row directions, in which case the PQ array will be composed of three or more rows. Furthermore, when the matrix of element images has the same number of pixels as in the above example, a common PQ array can be used for overlapping element images in the row direction and the column direction, but when the matrix of element images has a different number of pixels, it is necessary to prepare a PQ array for the row direction and a PQ array for the column direction.

[0039] Next, image analysis by the light field image analysis unit 30 will be described. Fig. 6A is the first half of the flowchart for light field image analysis, showing the image analysis procedure for the first group of elemental images (corresponding to time t = 1). Fig. 6B is the second half of the flowchart for light field image analysis, showing the image analysis procedure for the second and subsequent groups of elemental images (corresponding to time t = 2 and subsequent). Each step is executed by the above-mentioned components of the light field image analysis unit 30.

[0040] When the elemental images of the initial light field image are read from the storage device 20 to the light field image analysis unit 30, the image reconstruction unit 31 reconstructs an image from the elemental images refocused on the depth position of the target (S1). Once the reconstructed image is generated, the bright spot labeling unit 35 labels each representative point of each bright spot, consisting of a group of pixels corresponding to the target in the reconstructed image, to identify the target (S2). A bright spot here refers to a group of one or more pixels with a certain luminance or pixel value, i.e., a point or small image area with a certain brightness. In the selective illumination system 100, the full-field illumination system 50 illuminates the entire field of view of the light field camera 10 with light of a predetermined wavelength, causing the target to fluoresce, making it easier to capture the bright spot. Once labeling is complete, the three-dimensional coordinate update unit 36 ​​determines the three-dimensional spatial coordinates of the target in the reconstructed image and sets these coordinates as the initial coordinates of the target identified by the label (S3). In parallel with or asynchronously with steps S1 to S3, the template selection unit 32 selects, as a template, an element image that includes the most bright spots from the element image group (S4).

[0041] 7 is a diagram illustrating an example of three-dimensionally reconstructing an object from the initial elemental image group, performing bright spot labeling, setting the initial coordinates of the object, and selecting a template. For example, the image reconstruction unit 31 appropriately refers to the PQ array stored in the storage device 20, shifts the elemental images in the elemental image group by a shift amount corresponding to the depth position, and overlaps them to reconstruct images refocused at multiple depth positions. Depth d ｏｂｊ As the value of Δ ＥＩbecomes larger, and the size of the reconstructed image becomes smaller accordingly. The image reconstruction unit 31 resizes the reconstructed images, which vary in size depending on the depth position, according to the depth position and stacks them in the Z direction (depth direction) to reconstruct a three-dimensional image of the target object. Note that, as a method for reconstructing an image refocused at an arbitrary depth position from a group of elemental images, the optical sectioning technology disclosed in the application of the present inventors (Patent Application No. 2022-202566) can be used to reconstruct a high-resolution image.

[0042] To identify pixels in the elemental image group that correspond to the three-dimensional image, for example, the bright spot labeling unit 35 multiplies the X and Y coordinates of each voxel in the three-dimensional image by the resizing ratio of the image at each depth position to convert them into pixel coordinates in the first elemental image group. For example, if the size of an image at a certain Z coordinate in the three-dimensional image is m and the size of that image before resizing is n, the X and Y coordinates of each voxel at that Z coordinate can be converted into corresponding pixel coordinates in the first elemental image group by multiplying them by the resizing ratio n / m. However, since the coordinates of other pixels overlapping with these converted pixel coordinates are unknown, the bright spot labeling unit 35 refers to the PQ array to identify the coordinates of other pixels overlapping with the pixel at the converted pixel coordinates when reconstructing an image refocused at that depth position, and then attaches a label identifying the object to the representative point of each bright spot consisting of the pixel at the converted pixel coordinates and the group of pixels overlapping with the pixel at the converted pixel coordinates when reconstructing an image refocused at that depth position. For example, each pixel constituting a bright spot can be binarized to determine its center of gravity, and the center of gravity can be used as the representative point of the bright spot. The same label is assigned to representative points of bright spots corresponding to the same three-dimensional image.

[0043] The correspondence between the labels and the representative points of each bright point is tabulated and stored as a label table in the storage device 20. The labels are also associated with the three-dimensional coordinates of the object. Specifically, the three-dimensional coordinate update unit 36 ​​determines the coordinates of each voxel of the three-dimensional image, and determines, for example, the center of gravity of the three-dimensional image as a coordinate in three-dimensional space, and sets the center of gravity as the initial coordinate of the object identified by the label.

[0044] In parallel with or asynchronously with the above image reconstruction, bright spot labeling, and initial coordinate setting of the object, the template selection unit 32 selects the element image containing the most bright spots as the template. For convenience, in the example of FIG. 7, there is only one object, so each element image has one bright spot. In this way, when there are multiple element images that can be candidates for the template, the element image closest to the origin in the element image group, i.e., the element image located in the upper left corner, can be selected as the template. The template is referenced in the bright spot labeling process for the second element image group. This completes the analysis process for the first element image group.

[0045] Returning to FIG. 6B , when the elemental image group in the next light field image is read from the storage device 20 to the light field image analysis unit 30, the bright spot representative point identification unit 33 determines the representative point of each bright spot in the elemental image group (S5). FIG. 8 is a diagram illustrating the identification of the representative point of each bright spot in the elemental image group. Because the elemental image group is a grayscale image, the bright spot representative point identification unit 33 smoothes the elemental image group using an averaging filter as necessary and then binarizes the elemental image group. The bright spot representative point identification unit 33 then determines the center of gravity of each binarized bright spot and designates the center of gravity as the representative point of that bright spot. Subsequently, labeling and tracking are performed on the representative point of each bright spot.

[0046] The binarization threshold may be an appropriate fixed value or may be determined according to the characteristics of the elemental image group. In the latter case, the approximate percentage of pixels in which the object appears is first calculated based on the range of the field of view and the size of the object. For example, if the field of view is 422.4 μm × 422.4 μm and the object is a circle with a diameter of 4 μm, the percentage of pixels in which the object appears in the field of view (in the elemental image group) is approximately 0.0070%. Next, the pixel value and its frequency percentage of each pixel in the elemental image group are calculated, and the frequencies are accumulated in descending order until the percentage of pixels in which the object appears is reached. The pixel value corresponding to the final sum is set as the threshold. For example, in the above example, pixel values ​​are accumulated from the most frequent ones, and the pixel value when the accumulated value exceeds 99.993% is set as the threshold. This method of determining the threshold is effective for images with high contrast, such as microscopic images, where the object and background are clearly separated and the object occupies a very small proportion of the entire field of view. If the object can be observed in advance, the pixel values ​​of the background and object can be determined in advance using appropriate image analysis software, and these values ​​can be used as the threshold. Also, Otsu's binarization, a common automatic binarization method, is difficult to apply when the background and object are clearly separated and the object occupies a very small proportion of the entire field of view, but it can be applied depending on the object.

[0047] Returning to FIG. 6B , once the representative points of each bright point in the elemental images are determined, the bright point tracking unit 34 determines the positional change of each bright point in the elemental image group (S6). FIG. 9 is a diagram illustrating the tracking of the representative points of each bright point in the elemental image group. For convenience, FIG. 9 shows two elemental image groups superimposed in time. The bright point tracking unit 34 finds the destination of each bright point's representative point between the elemental image groups that are successively captured in time by nearest neighbor search. If the frame rate of the continuously captured light field images is sufficiently high relative to the moving speed of the target object, as shown in FIG. 9 , the deviation in coordinates of each bright point's representative point between the elemental image groups that are successively captured in time is very small, and the destination of each bright point's representative point can be found by nearest neighbor search. For example, as shown in the enlarged view, the destinations of five bright point representative points P1 to P5 can be found by nearest neighbor search between the elemental image groups that are successively captured in time. When the destination of the representative point of each bright point is found, the bright point tracking unit 34 determines the position change of each bright point from the positional relationship before and after the movement of the representative point of each bright point. For example, for the representative point P1 of the bright point, the XY coordinates at time t-1 are P1 ｔー１ = (x P1 ｔー１ , yP1 ｔ－１ ) and the XY coordinates at time t are P1 ｔ = (x P1 ｔ , yP1 ｔ ), then the position change of P1 is P1 ｔ -P1 ｔー１ In the enlarged view, the representative point below P1 and to the right of P3 first appeared at time t, and therefore cannot be tracked from time t-1 to time t. If this representative point continues to appear after time t, its position will be tracked.

[0048] Returning to FIG. 6B , in parallel with tracking the representative points of each bright point, the bright point labeling unit 35 labels the representative points of each bright point in the template in each element image of the element image group with the same label as the representative points of each bright point in the template (S7). FIG. 10 is a diagram illustrating bright point labeling in element image groups. The left diagram shows the kth element image group, the center diagram shows the k+1th element image group undergoing nearest neighbor search for the representative points of bright points using the template, and the right diagram shows the k+1th element image group after labeling. For convenience, it is assumed that there is only one representative point of a bright point in each element image, and that the same label is assigned to each representative point of a bright point in the kth element image group in the left diagram. In this case, as shown in the center diagram, the bright point labeling unit 35 uses nearest neighbor search to find the representative point of a bright point in each element image in the k+1th element image group that corresponds to the representative point P0 of each bright point in the template. For convenience, the elemental image group in the center diagram shows a template superimposed on each elemental image in the (k+1)th elemental image group. In this example, each elemental image has only one representative point of a bright spot, so each representative point is found as a representative point corresponding to a representative point of a bright spot in the template. When a corresponding representative point of a bright spot is found in each elemental image, the bright spot labeling unit 35 assigns the same label to the found representative point as the representative point P0 of the corresponding bright spot in the template. As a result, as shown on the right side of the diagram, a label is assigned to each representative point of a bright spot in the (k+1)th elemental image group. In this way, a label is assigned to each representative point of a bright spot in the new elemental image group.

[0049] When there are multiple targets, an element image may contain multiple bright spots resulting from those targets. In such cases, the bright spot labeling unit 35 may label element images containing a predetermined number of bright spots or more. For example, if the number of bright spots in a template is N, element images containing N-1 or more bright spots are labeled. This reduces the number of element images to be labeled, thereby reducing the load associated with the labeling process. Figure 11 illustrates template selection and bright spot labeling for the second and subsequent element image groups when a single element image contains multiple bright spots. The left diagram shows the first element image group, the center diagram shows the mth element image group, and the right diagram shows the nth (n > m)th element image group. The template selection unit 32 selects the element image containing the most bright spots as the template from the first element image group. In this example, an element image containing four bright spots is selected as the template, but there are multiple such element images. In such cases, the template selection unit 32 selects the element image with the greatest degree of separation between the bright spots as the template. If the degree of separation is expressed as SEP, then SEP can be defined by the following formula, for example: SEP=X ｍａｘ -X ｍｉｎ +Y ｍａｘ -Y ｍｉｎ However, X ｍａｘ is the maximum X coordinate among the X and Y coordinates of the representative points of the multiple bright points included in the element image of the template candidate, and X ｍｉｎ is the minimum X coordinate, Y ｍａｘ is the maximum Y coordinate, Y ｍｉｎ is the minimum Y coordinate.

[0050] As described above, when an elemental image is extracted from a microlens image, a portion of the microlens image is cut. However, if the target object moves, the bright spots included in the cut area may move within the elemental image, causing the number of bright spots in the elemental image to increase or decrease over time. For example, when focusing on the elemental image enlarged in FIG. 11 , the number of bright spots in the elemental image is 1 in the first elemental image group (left), increases to 2 in the mth elemental image group (center), and increases to 4 in the nth elemental image group (right). Although the number of bright spots in the elemental image in the mth elemental image group increases to 2, this does not satisfy the condition of N-1 or more, so the elemental image is not subject to labeling. When the number of bright spots in the elemental image in the subsequent nth elemental image group increases to 4, this satisfies the condition of N-1 or more, and the elemental image is subject to labeling. For the element images to be labeled, the bright spot labeling unit 35 assigns the same labels as the representative points of each bright spot in the template to the representative points of each bright spot that correspond to the representative points of each bright spot in the template.

[0051] Returning to Fig. 6B, once tracking and labeling of the representative points of each bright spot is complete, the three-dimensional coordinate update unit 36 ​​updates the X and Y coordinates of the object identified by the label based on the positional changes of the representative points of bright spots with the same label, and also updates the Z coordinate of the object based on the distance between those representative points between adjacent element images (S8). For example, the X and Y coordinates of the object at time t can be found by adding the average positional changes of the representative points of bright spots with the same label from time t-1 to time t to the X and Y coordinates at time t-1. This can be expressed mathematically as follows: (x ｔ , y ｔ ) = (x ｔ－１ , y ｔ－１ ) + <P ｔ -P ｔ－１ 〉 However, (x ｔ , y ｔ ) are the XY coordinates of the object at time t, (x ｔ－１ , y ｔ－１ ) are the XY coordinates of the object at time t-1, P ｔ are the X and Y coordinates of the representative point of the bright spot with the same label at time t, and P ｔ－１are the X and Y coordinates of the representative point of the bright points with the same label at time t-1, and the operator < > represents the average value of each of the X and Y coordinates.

[0052] The Z coordinate of the target object at time t can be directly calculated based on the interval between the representative points of the bright points with the same label in the element image group at time t between the adjacent element images. Figure 12 is a diagram for explaining the interval between the representative points of the bright points between the adjacent element images in the element image group. For example, as shown in the enlarged view, the coordinates of the representative points P1 to P5 of the five bright points in the element image group are known in advance, so the interval Δ ＥＩ１ , Δ ＥＩ２ , Δ ＥＩ３ , Δ ＥＩ４ The interval Δ of the representative points of the same labeled bright points between adjacent element images at time t is ＥＩ１ , Δ ＥＩ２ , Δ ＥＩ３ , Δ ＥＩ４ , …, Δ ＥＩｎ The average of ￣ Δ ＥＩｔ Then, from equations (1) and (2), the Z coordinate of the object identified by the label at time t is expressed by the following equation: ｔ = d ｏｂｊ = (( ￣ Δ ＥＩｔ -EI Ｌ )a-MLP*b) / ( ￣ Δ ＥＩｔ -EI Ｌ ) M ２

[0053] FIG. 13 is a graph showing tracking performance in the Z direction. Z-direction tracking performance was confirmed by capturing images of fluorescent particles with a radius of 2 μm moved in 1 μm increments in the Z direction from 0 μm to 40 μm on an XYZ motorized stage using the light field camera 10 and analyzing the light field images using the light field image analysis unit 30. (a) shows the input to the motorized stage carrying the target (fluorescent particle), and (b) shows the results of position tracking of the target (fluorescent particle) by the light field image analysis unit 30. Position tracking is not possible near Z=0 (NOP) in (b) because refocusing is difficult in that area due to the characteristics of the light field camera 10. It can be said that the movement of the target in the Z direction was well tracked at other depth positions where refocusing is possible.

[0054] The light field image analysis unit 30 first (at time t = 1) reconstructs the object in three dimensions from the elemental image group to determine its initial coordinates, but thereafter (from time t = 2 onwards), tracks the three-dimensional spatial position of the object by analyzing only the elemental image group of the two-dimensional image without three-dimensionally reconstructing the object from the elemental image group. Figure 14 is a diagram showing how the light field image analysis unit 30 tracks the object. For example, after determining the initial coordinates (x, y, z) = (553, 362, 44) of particle A, which is the object, from the first elemental image group, the three-dimensional spatial position of particle A is tracked by analyzing the elemental image group of the two-dimensional image without three-dimensionally reconstructing particle A.

[0055] In this way, the light field image analysis unit 30 can track the three-dimensional spatial position of an object in an image from a light field image with a high frame rate, such as continuous data of a group of elemental images, making it possible to observe high-speed events such as biomolecular and intracellular behavior. As an example, the three-dimensional spatial position of an object can be tracked by analyzing a light field image at 100 fps.

[0056] While the light field image analysis unit 30 analyzes the light field image of the object and tracks the three-dimensional spatial position of the object in real time, the spatial light modulator control unit 40 controls the spatial light modulator 64 to irradiate the light supplied from the light source 61 of the selective illumination system 60 onto the three-dimensional spatial position of the object updated by the light field image analysis unit 30.

[0057] 15 is a flowchart of spatial light modulator control. When the light field image analyzer 30 sets the initial coordinates of the target (step S3 in FIG. 6A ) or updates the three-dimensional spatial position of the target (step S8 in FIG. 6B ), the spatial light modulator controller 40 acquires the three-dimensional spatial position from the light field image analyzer 30 directly or via the storage device 20 (S11). The spatial light modulator controller 40 then references a spatial light pattern / computer-generated hologram correspondence set stored in advance in the storage device 20 and selects from among multiple spatial light patterns the one closest to the three-dimensional spatial position of the target (S12). If there are multiple targets, the spatial light pattern closest to the three-dimensional spatial position of each target is selected.

[0058] The spatial light pattern / computer-generated hologram correspondence set is a data set consisting of a combination of multiple spatial light patterns and corresponding computer-generated holograms. The spatial light pattern can be defined, for example, as spot light at each lattice point when a rectangular area in the center of the field of view of the light field camera 10 is divided into a grid. Furthermore, the rectangular area may be expanded in the Z direction to define a spatial light pattern representing a three-dimensional lattice point. Once the spatial light pattern is determined, a computer-generated hologram corresponding to each spatial light pattern can be generated based on various parameters such as the optical characteristics of the objective lens 70 and the wavelength of the laser light emitted from the light source 61 of the selective illumination system 60. The storage device 20 stores the spatial light pattern / computer-generated hologram correspondence set thus prepared. If the spatial light pattern / computer-generated hologram correspondence set is stored on a HDD, it is desirable to write it to RAM and reference the RAM for high-speed search.

[0059] When the spatial light modulator control unit 40 selects a spatial light pattern, it provides a corresponding computer-generated hologram to the spatial light modulator 64 (S13). If there are multiple targets, it provides a computer-generated hologram, which is a combination of computer-generated holograms corresponding to the selected spatial light patterns, to the spatial light modulator 64. As a result, the spatial light modulator 64 performs spatial light modulation according to the provided computer hologram, and a desired spatial light pattern is formed on the sample in the sample holder 80 beyond the objective lens 70, and light is selectively irradiated onto the target in the sample.

[0060] <Effects> The selective illumination system 100 according to this embodiment can quickly acquire and track the three-dimensional spatial position of an object based on a light field image of the object, and can quickly and pinpoint light onto the object. This makes it possible, for example, to control the activity of a target neuron at a single cell level while tracking a freely moving living organism at high speed. Furthermore, since it is possible to illuminate only an arbitrary object while tracking its position, it is possible to reduce unnecessary background light that interferes with observation, and further improve the signal-to-background light ratio (SBR), enabling high-sensitivity observation of only specific biomolecules or cells.

[0061] Figure 16 shows microscopic images with and without selective irradiation and profile graphs of pixel values ​​near the target. The upper left image shows a microscopic image without selective irradiation of the target, and the lower left image shows a microscopic image when light is selectively irradiated onto the target using the selective irradiation system 100. These microscopic images were taken of a nematode. The field of view of these microscopic images is approximately 300 μm in length and width. In the selective irradiation image, a roughly circular light beam with a diameter of 4 to 5 μm is irradiated onto the target at a depth of 56 μm. The right image in each figure is a profile graph plotting the pixel values ​​of pixels on a profile line superimposed on the microscopic image. The target is located approximately in the center of the profile line, and the vertical axis of the profile graph represents the pixel value, while the horizontal axis represents the pixel position on the profile line. The background pixel value without selective irradiation averages 46.25, while the background pixel value with selective irradiation averages 21.13. In this way, when the object is selectively irradiated with light, it is possible to reduce unnecessary background light that impedes observation compared to when selective irradiation is not performed.

[0062] Figure 17 shows other microscope images with and without selective irradiation, as well as profile graphs of pixel values ​​near the target. The upper left image shows a microscope image without selective irradiation of the target, and the lower left image shows a microscope image when light is selectively irradiated onto the target using the selective irradiation system 100. These microscope images were captured of fluorescent beads. The field of view of these microscope images is approximately 300 μm in length and width. In the selective irradiation image, a roughly circular beam of light with a diameter of 4 to 5 μm is irradiated onto the target at a depth of 50 μm. The right image in each figure is a profile graph plotting the pixel values ​​of pixels on a profile line superimposed on the microscope image. The target is located approximately in the center of the profile line, and the vertical axis of the profile graph represents the pixel value, while the horizontal axis represents the pixel position on the profile line. The average signal pixel value without selective irradiation is 20.8, while the average signal pixel value with selective irradiation is 76.5. In this way, when the object is selectively irradiated with light, the unnecessary background light that impedes observation can be reduced and the SBR can be further improved compared to when the object is not selectively irradiated.

[0063] <<Variations>> In the above description, a light-reflecting LCOS-SLM is used as the spatial light modulator 64, but other types of devices such as a light-transmitting LCOS-SLM or a DMD (Digital Micromirror Device) that performs spatial light modulation using micromirrors can also be used.

[0064] In the above description, the spatial light modulator control unit 40 selects a pre-prepared computer-generated hologram and provides it to the spatial light modulator 64 so that computer-generated holograms can be switched quickly in accordance with the movement of the object. However, it is also possible to generate a computer-generated hologram by calculation from the three-dimensional spatial position of the object each time the light field image analysis unit 30 updates the three-dimensional spatial position of the object. Even when there are multiple objects, it is possible to generate computer-generated holograms for selectively irradiating these objects with light. However, since generating a computer-generated hologram by calculation takes more time than selecting from pre-prepared holograms, it is desirable to take measures such as increasing the size of the light spot irradiated on the object to a certain extent so that the object does not deviate from the light irradiation area even if it moves slightly, thereby eliminating the need to frequently regenerate computer-generated holograms.

[0065] Two types of light may be emitted from the light source 51 of the full-field illumination system 50, one of which is used as illumination for tracking the object and the other for fluorescent observation of the object. In this case, the different types of light may be emitted alternately in a time-division manner from the light source 51, or a color filter may be placed in front of the image sensor 3 in the light field camera 10 so that images of the two types of light can be captured simultaneously, or two systems of image sensors 3 may be provided, one for tracking the object and one for fluorescent observation, and the light that has passed through the imaging lens 1 may be split and incident on each image sensor 3.

[0066] Instead of reconstructing a three-dimensional image of the object and determining the initial coordinates of the object from that three-dimensional image as explained in Figure 7, the coordinates of the object in three-dimensional space may be determined directly from an image reconstructed from the initial group of elemental images. Since the Z coordinate of the reconstructed image is known in advance, the initial coordinates of the object in three-dimensional space can be determined by determining the coordinates of the center of gravity of the object in the reconstructed image, i.e., the X and Y coordinates.

[0067] The template selection unit 32 may update the template by selecting a template not only from the first element image group but also from the second and subsequent element image groups as appropriate. In particular, when there are multiple targets that each rotate or change direction, it is preferable to reselect a template from the second and subsequent element image groups.

[0068] The above explanation is based on the premise that the element images in the element image group are arranged in a square lattice pattern, but when a microlens array with microlenses arranged in a honeycomb pattern is used, the element images may be arranged in a honeycomb pattern. Even in such a case, the above-mentioned identification of representative points of bright spots in the element image group, bright spot labeling, template selection, bright spot tracking, and three-dimensional coordinate updating can be applied.

[0069] As described above, the embodiments have been described as examples of the technology of the present invention. For this purpose, the accompanying drawings and detailed description have been provided. Therefore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the above technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential. Furthermore, because the above-described embodiments are intended to exemplify the technology of the present invention, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

[0070] The selective illumination system of the present invention can selectively illuminate an object with light while tracking the object's three-dimensional spatial position, making it useful in the fields of optogenetics and for observing high-speed events such as biomolecular and intracellular behavior.

[0071] 100 Selective illumination system 110 Selective illumination control device 10 Light field camera 30 Light field image analysis unit 31 Image reconstruction unit 32 Template selection unit 33 Bright point representative point identification unit 34 Bright point tracking unit 35 Bright point labeling unit 36 ​​Three-dimensional coordinate update unit 40 Spatial light modulator control unit 50 Full-field illumination system 51 Light source 52 Collimator lens (optical system) 60 Selective illumination system 61 Light source 62 Collimator lens (optical system) 63 Mirror (optical system) 64 Spatial light modulator 65 Fourier transform lens (optical system) 66 DC component removal filter (optical system)

Claims

1. A selective illumination control device that controls a spatial light modulator that spatially modulates incident light to form an arbitrary spatial light pattern and selectively illuminates an object with light, comprising: a light field image analysis unit that analyzes light field images of the object captured continuously over time with a light field camera and tracks the three-dimensional spatial position of the object in the light field image; and a spatial light modulator control unit that generates a computer-generated hologram for forming a spatial light pattern that selectively illuminates light at the three-dimensional spatial position of the object tracked by the light field image analysis unit, and provides the spatial light modulator with the generated computer-generated hologram.

2. The selective illumination control device according to claim 1, characterized in that a combination of a plurality of spatial light patterns and corresponding computer generated holograms is prepared in advance, and the spatial light modulator control unit selects from the plurality of spatial light patterns the one that is closest to the three-dimensional spatial position of the target object, and provides the spatial light modulator with a computer generated hologram corresponding to the selected spatial light pattern.

3. The selective illumination control device according to claim 2, characterized in that there are a plurality of objects, the light field image analysis unit tracks the three-dimensional spatial position of each of the plurality of objects, and the spatial light modulator control unit selects from the plurality of spatial light patterns a spatial light pattern that is closest to the three-dimensional spatial position of each of the plurality of objects, and provides to the spatial light modulator a computer hologram that is a composite of computer holograms corresponding to the selected spatial light patterns.

4. The selective illumination control device according to claim 1, characterized in that the spatial light modulator control unit generates the computer generated hologram by calculation from the three-dimensional spatial position of the object.

5. The selective illumination control device according to claim 4, characterized in that there are a plurality of objects, the light field image analysis unit tracks the three-dimensional spatial position of each of the plurality of objects, and the spatial light modulator control unit generates, by calculation, a computer-generated hologram that forms a spatial light pattern that selectively irradiates light onto the three-dimensional spatial position of each of the plurality of objects.

6. The light field image analysis unit comprises: an image reconstruction unit which reconstructs an image refocused on the depth position of the object from the group of elemental images of the first light field image; a template selection unit which selects an elemental image containing the most bright spots from the group of elemental images of the first light field image as a template; a bright spot representative point identification unit which determines a representative point of each bright spot in the group of elemental images of the second and subsequent light field images; a bright spot tracking unit which determines a positional change of each representative point of each bright spot in the group of elemental images of the second and subsequent light field images; and a bright spot labeling unit which assigns a label for identifying the object to a representative point of each bright spot consisting of a group of pixels corresponding to the object in the reconstructed image in the group of elemental images of the first light field image, and assigns the same label as the representative point of each bright spot in the template to a representative point of each bright spot corresponding to the representative point of each bright spot in the template in each elemental image in the group of elemental images of the second and subsequent light field images. a three-dimensional coordinate updating unit that obtains three-dimensional spatial coordinates of an object in a reconstructed image and sets the coordinates as initial coordinates of the object identified by the label, updates the XY coordinates of the object identified by the label based on a position change of a representative point of a bright point with the same label in a group of elemental images of a second or subsequent light field images, and updates the Z coordinate of the object based on an interval between the representative points of the representative points between adjacent elemental images.

7. A selective illumination control method for selectively irradiating an object with light by controlling a spatial light modulator that spatially modulates incident light to form an arbitrary spatial light pattern, comprising: a step in which a light field image analysis unit analyzes light field images of the object captured continuously over time by a light field camera, and tracks a three-dimensional spatial position of the object in the light field images; and a step in which a spatial light modulator control unit generates a computer-generated hologram for forming a spatial light pattern that selectively irradiates light onto the three-dimensional spatial position of the object tracked by the light field image analysis unit, and provides the generated computer-generated hologram to the spatial light modulator.

8. A program for causing a computer to control a spatial light modulator which spatially modulates incident light to form an arbitrary spatial light pattern, thereby selectively irradiating light onto an object, comprising: a light field image analysis means which analyzes light field images of the object captured continuously over time with a light field camera, and tracks the three-dimensional spatial position of the object in the light field images; and a spatial light modulator control means which generates a computer-generated hologram for forming a spatial light pattern which selectively irradiates light onto the three-dimensional spatial position of the object tracked by the light field image analysis means, and provides the spatial light modulator with the generated computer-generated hologram.

9. A selective illumination system comprising: a selective illumination control device according to any one of claims 1 to 5; a light field camera which captures a light field image of an object and supplies the image to said selective illumination control device; a full-field illumination system having a light source and an optical system and illuminating the full field of view of said light field camera; and a selective illumination system having a light source, an optical system, and a spatial light modulator controlled by said selective illumination control device and selectively irradiating light onto said object.

10. A selective illumination system comprising: a selective illumination control device according to claim 6; a light field camera which captures a light field image of an object and supplies the image to said selective illumination control device; a full-field illumination system having a light source and an optical system and illuminating the full field of view of said light field camera; and a selective illumination system having a light source, an optical system, and a spatial light modulator controlled by said selective illumination control device and selectively irradiating light onto said object.

Images