Image processing device, image processing method, and program
The image processing device automatically generates and reconstructs multi-view image data, which solves the problem of low resolution in the depth direction of light field microscopes, and realizes high-resolution three-dimensional imaging, which is suitable for observing high-speed events in biological molecules and cells.
Patent Information
- Application Number
- CN202380087561.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-12-19
- Filing Date
- 2023-11-22
- Publication Date
- 2025-07-25
AI Technical Summary
When existing light field microscopes reconstruct images, the resolution in the depth direction is low, making it difficult to achieve high-resolution three-dimensional imaging.
The element image group generation unit, the pixel profile generation unit, the pixel depth determination unit and the image reconstruction unit in the image processing device automatically generate and reconstruct an image refocused from the multi-view image data to any depth, thereby improving the resolution in the depth direction.
It has achieved a significant improvement in depth direction resolution, and can perform high-speed and high-resolution three-dimensional imaging, which is suitable for observing high-speed events in biological molecules and cells.
Smart Images
Figure CN120380772A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to an image processing technique suitable for reconstructing an image refocused at an arbitrary depth from a light field image that records a light space (light field). Background Art
[0002] In order to finely and dynamically observe biomolecules and intracellular behavior, high-speed and large-scale three-dimensional imaging is required. As a microscope capable of realizing such three-dimensional imaging, there is a spinning disk confocal microscope. The spinning disk confocal method generates multiple parallel light beams by irradiating a spinning disk with a plurality of pinholes arranged thereon, and rapidly scans a sample with them to form a confocal image. Although this method can obtain high-resolution images, since it is necessary to scan in the depth direction, it takes time in measurement and is not suitable for observing high-speed events. In addition, the device tends to be large in size and high in cost.
[0003] As another option for three-dimensional imaging, there is a light field microscope (LFM: Light Field Microscopy). The light field represents a set of light rays in a three-dimensional space. The LFM can record a light field by disposing a microlens array in which a plurality of microlenses are two-dimensionally arranged at an intermediate image plane and acquiring the passing positions of light rays on the aperture stop of the objective lens and the acquisition positions of light rays on an image sensor. A sub-aperture image equivalent to a small-aperture lens can be obtained from the light field image recorded by the LFM. The sub-aperture image is an image of a subject partially photographed by shifting the viewpoint, and each has a relatively deep depth of field due to the pinhole effect, and at the same time, has a parallax with other sub-aperture images. Therefore, an image refocused at an arbitrary depth can be reconstructed by overlapping the sub-aperture images with an offset.
[0004] As described above, since the LFM can quickly capture a three-dimensional image in one shot without scanning in the depth direction, it is suitable for observing high-speed events. On the other hand, since the three-dimensional space is recorded as a two-dimensional light field image, the resolution is significantly reduced compared to the spinning disk confocal method. Therefore, in the LFM, it is very important to improve the spatial resolution. For this purpose, deconvolution processing is performed on the light field image to seek an improvement in resolution (for example, refer to Non-Patent Document 1), or an incoherent multiscale scattering model is introduced to perform optical sectioning computationally (for example, refer to Non-Patent Document 2).
[0005] [Non-Patent Document 1]Prevedel,R.,Yoon,YG.,Hoffmann,M.et al.Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy.Nat Methods 11,727-730(2014).
[0006] [Non-Patent Document 2]Zhang,Y.,Lu,Z.,Wu,J.et al.Computational optical sectioning with an incoherent multiscale scattering model for light-field microscopy.Nat Commun 12,6391(2021). Summary of the Invention
[0007] -Problems to be Solved by the Invention-
[0008] The inventors of the present application disclosed in a previous patent application (Patent Application No. 2021-185638) an invention that enables the improvement of the resolution in the XY direction for the reconstructed image of LFM. The subject of the present invention is to improve the resolution in the Z direction, i.e., the depth direction, for the reconstructed image of LFM.
[0009] According to an aspect of the present invention, there is provided an image processing apparatus, a corresponding image processing method, a computer program for causing a computer to execute the image processing, or a recording medium recording the computer program. The image processing apparatus reconstructs an image refocused to an arbitrary depth from multi-viewpoint image data, the multi-viewpoint image data being a collection of a plurality of sub-images which are images of the same subject taken from a plurality of viewpoints shifted from each other. The image processing apparatus includes an elemental image group generation unit, a pixel profile generation unit, a pixel depth determination unit, and an image reconstruction unit. The elemental image group generation unit cuts out elemental images which are images taken from mutually different viewpoints from the multi-viewpoint image data and automatically generates an elemental image group. The pixel profile generation unit automatically generates a pixel profile showing the relationship between the offset of the elemental images and the pixel values of each pixel, while changing the offset with respect to the elemental images related to adjacent viewpoints in the elemental image group to partially overlap the elemental images, and taking, as the pixel value of a pixel, a value reflecting the pixel values of those pixels for the pixels overlapping between those elemental images. The pixel depth determination unit automatically associates, with each pixel of each elemental image in the elemental image group, a depth corresponding to the offset of the elemental image when the pixel value of the pixel takes an extreme value, with reference to the pixel profile. The image reconstruction unit offsets and overlaps the elemental images in the elemental image group with an offset corresponding to the specified depth, and substantially sets the pixel values of the pixels not associated with the specified depth to zero, and automatically reconstructs an image refocused to the specified depth.
[0010] - Effects of the Invention -
[0011] According to the present invention, for example, with respect to a reconstructed image of a light field microscope, the resolution in the depth direction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is a schematic diagram of an optical system including an image processing apparatus according to an embodiment of the present invention.
[0013] Figure 2 is a flowchart of an image processing method according to an embodiment of the present invention.
[0014] Figure 3 is a diagram showing (a) multi-viewpoint image data, (b) an elemental image group, and (c) a reconstructed image according to an example.
[0015] Figure 4 is a diagram schematically showing the geometric relationship of various parameters of an optical system according to an example.
[0016] Figure 5This is a diagram for explaining the overlapping of elemental images in pixel profile generation.
[0017] Figure 6 This is a diagram for explaining the association of pixel depth involved in an example.
[0018] Figure 7 This is a diagram for comparing the simple average method, the weighted average method, and the resolution of the reconstructed image obtained by the image processing apparatus according to the present embodiment. Detailed Embodiment
[0019] Hereinafter, embodiments of the present invention will be described in detail with appropriate reference to the accompanying drawings. However, unnecessary detailed descriptions may sometimes be omitted. For example, detailed descriptions of well-known matters and repeated descriptions of substantially the same structures may sometimes be omitted. This is to avoid making the following description unnecessarily lengthy and to make it easier for those skilled in the art to understand. It should be noted that the inventor provides the accompanying drawings and the following description to enable those skilled in the art to fully understand the present invention, but does not intend to limit the subject matter recited in the claims by these. Also, the dimensions, detailed shapes of each component depicted in the drawings may sometimes be different from the actual ones.
[0020] 《Embodiment》
[0021] Figure 1 This is a schematic diagram of an optical system including an image processing apparatus according to an embodiment of the present invention. The optical system 100 includes an optical system 10, a storage device 20, and an image processing device 30. The storage device 20 may also be configured in the cloud. And all or part of the components of the image processing device 30 may also be configured in the cloud.
[0022] The optical system 10 includes a main lens 1, a microlens array 2, and an image sensor 3. The main lens 1 sometimes includes an objective lens and an imaging lens. Here, for convenience, the lens including them is represented as the main lens 1. The microlens array 2 is formed by two-dimensionally arranging a plurality of microlenses (micro lenses) 2a and is disposed between the main lens 1 and the image sensor 3. The image sensor 3 is a CCD sensor, a CMOS sensor, etc., which are formed by two-dimensionally arranging a plurality of light-receiving elements. The plurality of light-receiving elements perform photoelectric conversion on the input light and output an electrical signal, which is not shown. The electrical signal output from each light-receiving element is converted into digital data by an A / D converter (not shown) and output as a light field image of multi-viewpoint image data from the optical system 10.
[0023] The storage device 20 is an aggregate of storage devices such as RAM, ROM, SSD, and HDD. The RAM is mainly used as a working memory when the image processing device 30 performs image processing. The ROM, SSD, and HDD mainly temporarily or permanently store computer programs for operating the image processing device 30, computer programs provided by being stored on a non-transitory recording medium and readable by a computer, multi-viewpoint image data output from the optical system 10, element images generated and used in the subsequent image processing, PQ arrays, pixel profiles, pixel depth tables, reconstructed images, and the like.
[0024] The image processing device 30 includes an element image group generation unit 31, a pixel profile generation unit 32, a pixel depth determination unit 33, and an image reconstruction unit 34. Each of these constituent elements can be implemented as hardware such as an ASIC and an FPGA, or can be implemented as software in which a computer program stored in a recording medium (not shown) and the storage device 20 is executed by a processor (not shown) such as a CPU and a GPU. It can also be implemented by appropriately combining hardware and software. During the execution of image processing, the image processing device 30 appropriately accesses various storage devices constituting the storage device 20 to perform data writing and reading.
[0025] Figure 2 It is a flowchart of an image processing method according to an embodiment of the present invention. Each step is executed by the above-described constituent elements of the image processing device 30.
[0026] The element image group generation unit 31 cuts out captured images from different viewpoints, that is, element images, from the multi-viewpoint image data, and automatically generates an element image group (S1). Figure 3 It is a diagram showing an example of multi-viewpoint image data, an element image group, and a reconstructed image. (a) The multi-viewpoint image data is obtained by photographing a sheet with the character " / " written thereon at a position deviated from the focal plane of the optical system 10. The multi-viewpoint image data is a micro-lens image group in which a plurality of micro-lens images (equivalent to sub-aperture images) MI captured by respective micro-lenses 2a of the micro-lens array 2 of the optical system 10 are arranged in a grid pattern. The shape of each micro-lens image MI is as shown by the dashed line, and becomes a substantially circular shape reflecting the circular shape of each micro-lens 2a. Thus, the multi-viewpoint image data is data in which a plurality of sub-images (in the example of Figure 3 MI) of the same subject captured from a plurality of mutually deviated viewpoints are aggregated.
[0027] Near the boundary of the microlens image MI, it is darker due to less light, and since the light passing through the end portion of the microlens 2a is captured, the distortion is large. Therefore, the central portion of the microlens image MI is used in image processing such as image reconstruction. That is, the feature image group generation unit 31 cuts out, as a feature image EI, for example, a square region inscribed in the circle from each of the substantially circular microlens images MI. The (b) feature image group is formed by arranging those feature images EI in a grid pattern. The cut-out feature images are stored in the storage device 20.
[0028] As described above, since the sheet with the characters " / 5" written on it is located at a position deviated from the focal plane of the optical system 10, the microlens images with parallax, in which the viewpoints are deviated by the respective microlenses 2a and the characters " / 5" are partially photographed, are recorded in the (a) multi-viewpoint image data. For example, when focusing on the upper left corner portion of the character "5", the light rays emitted from this portion of the subject are recorded with an interval Δ between adjacent microlens images. As shown in the (b) feature image group, the interval Δ between the microlens images becomes Δ between the feature images. EI Here, when the pitch of the microlenses 2a is MLP and the length of one side of the feature pixel is EI L , Δ EI is represented by the following formula (1).
[0029] Δ EI = Δ - (MLP - EI L )…(1)
[0030] By shifting and overlapping adjacent feature images in the feature image group by an offset amount corresponding to the parallax, the (c) reconstructed image is obtained. In the Figure 3 example, the (c) reconstructed image focused on the position of the sheet with the characters " / 5" written on it is obtained by shifting and overlapping adjacent feature images by Δ EI each time. In this way, by appropriately changing the offset amount and overlapping the feature images, an image refocused at an arbitrary depth position can be reconstructed according to the ray tracing in the opposite direction from the acquired image.
[0031] Here, regarding the target object to be refocused, when the deviation from the object-side focal plane NOP (Native Object Plane) of the objective lens in the main lens 1, that is, the depth, is d obj , the relationship between the interval Δ of the light rays originating from the same point between adjacent microlens images and the depth d obj of that point can be represented by the respective parameters of the optical system 10. Figure 4 is a diagram schematically showing the geometric relationship of the respective parameters of the optical system 10 involved in one example. O in the figure is located at the depth dobj The target object at the position, A is the imaging point of the main lens 1 for the target object O, B and C are the centers of the adjacent microlenses 2a, and B' and C' are the positions of the light rays from the target object O captured by the image sensor 3 via B and C. When focusing on the triangles ABC and AB'C' in the figure, based on the similarity relationship of the triangles,
[0032] BC:B'C' = CA:C'A
[0033] When expressing it with parameters,
[0034] MLP: Δ=(a + d obj M 2 ):(a + d obj M 2 −b)
[0035] When solving for Δ, the following formula (2) is obtained.
[0036] Δ = MLP(1−b / (d obj M 2 −a))…(2)
[0037] Here, MLP is the pitch of the microlens 2a, M is the magnification of the objective lens of the main lens 1, a is the distance from the image-side focal plane T-NIP (Native Image Plane) of the imaging lens of the main lens 1 to the center of the microlens 2a, and b is the distance from the center of the microlens 2a to the image sensor 3. It should be noted that in Figure 4 , although a is considered as a positive value to represent the geometric relationship, when the direction from the object side to the image side is taken as positive, a needs to be considered as a negative value. Therefore, in formula (2), a is finally replaced with -a.
[0038] It should be noted that since there is no anisotropy in the plane perpendicular to the depth direction (Z direction) with respect to two mutually perpendicular directions (X direction, Y direction), the same Δ can be applied in the X direction and the Y direction respectively.
[0039] That is, according to formulas (1) and (2), if the interval Δ EI of the light rays originating from the same point between adjacent elemental images is known, the depth d obj of that point can be specified. This Δ EI can be estimated from the change characteristics of the pixel values of each pixel when changing the offset so that adjacent elemental image portions overlap. Hereinafter, the specific procedure for estimating Δ EI will be described.
[0040] Return Figure 2, while changing the offset and performing partial overlap on the feature images related to adjacent viewpoints in the feature image group, the pixel profile generation unit 32 automatically generates a pixel profile (S2) representing the relationship between the offset of the feature images and the pixel values of each pixel, using the value reflecting the pixel values of those pixels as the pixel value of that pixel for the pixels overlapping between those feature images. The generated pixel profile is stored in the storage device 20.
[0041] Figure 5 This is a diagram for explaining the overlap of feature images in pixel profile generation. The overlapping pixels are shown in shadow. The left column shows the state where the feature images overlap, and the right column shows the state where the overlap of the feature images is unfolded. For convenience, the feature image group consists of a total of 4 feature images D, E, F, and G, both horizontally and vertically, and the size of each feature image is 5 pixels horizontally and vertically. The configuration relationship of the feature images is that feature image E is arranged on the right side of feature image D, feature image F is arranged below feature image D, and feature image G is arranged on the right side of feature image F, which is below feature image E. To refer to each pixel in the feature image group, consecutive numbers starting from 1 are assigned to the rows and columns. For example, the pixels belonging to feature image D are referred to in the range from the first row to the fifth row and from the first column to the fifth column, and the pixels belonging to feature image G are referred to in the range from the sixth row to the tenth row and from the sixth column to the tenth column. Hereinafter, pixel [m, n] refers to the pixel located in the m-th row and the n-th column in the feature image group.
[0042] Z in the figure represents the depth parameter of d obj . For convenience, the same Z value is used to represent d obj and the offset of the feature image. That is, when Z = n, d obj = n, indicating that the offset of the feature image is as much as n pixels.
[0043] The pixel profile is generated for each pixel in the feature image group. The generation of the pixel profile starts from Z = 0. First, when Z = 0, the pixel profile generation unit 32 records the pixel value of each pixel in the feature image group as the initial pixel value of that pixel in the pixel profile.
[0044] Next, when Z = 1, the pixel profile generation unit 32 offsets the adjacent feature images by one pixel for overlap, calculates the pixel values of the overlapping pixels, and records them in the pixel profile. Which pixels in the feature image group are overlapped can be easily determined by referring to the PQ array indicating the overlapping manner of the pixels. The PQ array represents the row number or column number of the pixels in the feature image group in units of feature images. As shown in the figure, for example, the PQ array corresponding to Z = 1 is represented as follows. 1 2 3 4 5 6 7 8 9 1 0
[0047] The PQ array means that the pixels in the fifth row (fifth column) and the pixels in the sixth row (sixth column) overlap. By preparing such a PQ array for each Z value in advance and storing it in the storage device 20 in advance, the pixel profile generation unit 32 can perform pixel overlap with reference to the PQ array corresponding to the Z value stored in the storage device 20.
[0048] When overlapping by shifting the feature image by one pixel with reference to the PQ array, the pixels in the fifth row that are part of the feature images D and E overlap with the pixels in the sixth row that are part of the feature images F and G. At the same time, the pixels in the fifth column that are part of the feature images D and F overlap with the pixels in the sixth column that are part of the feature images E and G. For these overlapping pixels, values such as the average value and the weighted sum value that reflect the pixel values of those pixels are recorded as the pixel values of those pixels in the pixel profile. For example, the pixel values of the overlapping pixels [5,10] and [6,10] are averaged, and this average value is recorded in the pixel profiles of the two pixels. Also, for the overlapping pixels [5,5], [5,6], [6,5], and [6,6] of the four feature images, the average of the pixel values of these four pixels is recorded in the pixel profiles of these pixels.
[0049] Next, when Z = 2, the pixel profile generation unit 32 overlaps by shifting the adjacent feature images by two pixels, calculates the pixel values of the overlapping pixels, and records them in the pixel profile. As shown in the figure, for example, the PQ array corresponding to Z = 2 is represented as follows. 1 2 3 4 5 6 7 8 9 1 0
[0052] The PQ array means that the pixels in the fourth and fifth rows (the fourth and fifth columns) overlap with the pixels in the sixth and seventh rows (the sixth and seventh columns) respectively. When the feature images are offset by two pixels for overlapping with reference to the PQ array, the pixels in the fourth and fifth rows, which are part of the feature images D and E, overlap with the pixels in the sixth and seventh rows, which are part of the feature images F and G respectively. At the same time, the pixels in the fourth and fifth columns, which are part of the feature images D and F, overlap with the pixels in the sixth and seventh columns, which are part of the feature images E and G respectively. Regarding these overlapping pixels, values such as the average value and the weighted sum value, which reflect the pixel values of these pixels, are recorded as the pixel values of those pixels in the pixel profile. For example, the pixel values of the overlapping pixels [4,10] and [6,10] are averaged, and this average value is recorded in the pixel profiles of the two pixels. And regarding the pixels such as [4,4], [4,6], [6,4], and [6,6] in the overlapping part of the four feature images, the average of the pixel values of these four pixels is recorded in the pixel profiles of these pixels.
[0053] After Z = 3, the pixel profile generation unit 32 is the same as above. Regarding the feature images related to adjacent viewpoints in the feature image group, while changing the offset amount for partial overlapping, it automatically generates the pixel profiles of each pixel. Then, the generation of the pixel profiles continues until the Z value is less than the number of pixels in the row or column of the feature image. In Figure 5 the example of, for example, the generation of the pixel profiles continues until Z = 4. It should be noted that the pixel profiles can also be generated in descending order of the Z value.
[0054] It should be noted that sometimes there are more than three feature images overlapping in the column direction and the row direction. In that case, the PQ array consists of more than three rows. And as shown in the above example, when the rows and columns of the feature image consist of the same number of pixels, the same PQ array can be used for overlapping in the row direction and the column direction of the feature image. However, when the number of pixels in the rows and columns of the feature image is different from each other, it is necessary to prepare a PQ array for the row direction and a PQ array for the column direction.
[0055] Return to Figure 2 , the pixel depth determination unit 33 automatically associates the depth corresponding to the offset amount of the feature image when the pixel value of each pixel of each feature image in the feature image group takes an extreme value with reference to the pixel profile for each pixel of each feature image in the feature image group (S3). This association information is stored in the storage device 20 as a pixel depth table.
[0056] Figure 6This is a diagram for explaining the correlation of pixel depth involved in an example. The graph in the diagram is a graph that visualizes the pixel profile of the object pixel. The vertical axis represents the pixel value, and the horizontal axis represents the depth (Z value). The object pixel is the central pixel of the image of the target object that appears in the four elemental images shown in the diagram. The image of the target object is a roughly circular image that is bright at the center and becomes dimmer towards the periphery. It is assumed that the larger the pixel value, the brighter the pixel. Also, it is assumed that the average value is recorded as the pixel value of the overlapping pixels in the pixel profile. From the graph of the pixel profile, it can be seen that the pixel value of the object pixel remains at the initial value
[130] from Z = 1 to 10, decreases from Z = 10, increases at Z = 23, returns to the initial value at Z = 31, and decreases again after Z = 31. In this way, once the decreasing pixel value increases from Z = 23 to 31, it becomes an extreme value at Z = 31 and decreases again after Z = 31. This is presumably because at Z = 31, the central pixels of the roughly circular target object images in the adjacent elemental images overlap each other. That is, the image of the target object is in focus at Z = 31, or artifacts are generated at, for example, Z = 27 outside, or out of focus at Z = 37. The pixel depth determination unit 33 automatically specifies the depth at which the pixel value takes an extreme value from this pixel profile, and sets Z = 31 as d obj Associate with this pixel.
[0057] It should be noted that Figure 6 the pixel profile shown is merely an example, and all pixel profiles are not limited to becoming Figure 6 the characteristics shown. The pixel profile can vary in various ways in response to the type, state, etc. of the subject. For example, in an image where two target objects with different depths, one bright and the other dim, appear, sometimes the pixel profile is generated such that the pixel value of the dimmer image is averaged with the pixel value of the brighter image, resulting in a pixel profile that exceeds the initial value. At that time, the pixel depth determination unit 33 can also ignore the values greater than the initial pixel value (the pixel value at Z = 0) in the pixel profile, that is, not recognize them as extreme values, and ultimately search for the extreme value below the initial pixel value and associate the depth at that time with this pixel. In this way, the correct extreme value can be identified in the pixel profile of the special mode, and the correct depth can be associated with the pixel.
[0058] Also, when the extreme value is not correctly identified in the pixel profile, or when the extreme value determination is incorrect due to noise, etc. contained in the pixel profile, sometimes an incorrect depth is associated with this pixel. To cope with such a situation, the image depth determination unit 33 can also, for each pixel of each elemental image in the elemental image group, refer to the pixel profile and associate the depth corresponding to the offset of the elemental image when the pixel value of this pixel takes a value within the range specified by the extreme value with this pixel. Using Figure 6An example will be described. If the Z value when the pixel value is taken from the extreme value, for example, within the range of 5%, is still used, then in addition to Z = 31, Z = 29, 30, 32, and 33 are associated with the target pixel. In this way, multiple depths can also be associated with one pixel. In this way, the error in the extreme value determination can be absorbed, and the depth can be associated with the pixel within a range that is considered to be correct to a certain extent.
[0059] Return to Figure 2 , the image reconstruction unit 34 offsets and overlaps the feature images in the feature image group with the offset corresponding to the specified depth, makes the pixel values of the pixels not associated with the specified depth substantially zero, and automatically reconstructs the image refocused on the specified depth (S4). As described with reference to Figure 3 , by offsetting adjacent feature images by Δ in response to the parallax each time and EI overlapping them, a reconstructed image refocused on the depth corresponding to Δ is obtained. At that time, the image reconstruction unit 34 refers to the pixel depth table stored in the storage device 20, uses the pixels corresponding to the specified depth as they are, and for the other pixels, makes the pixel values zero or almost zero, that is, makes them completely black or almost black, to automatically reconstruct the image. Taking the EI example of, when Z = 32 is specified as the depth, the central pixel of the target object is used as it is, and when the other range is specified, the pixel value of this pixel is zero or almost zero, and the image is reconstructed. Figure 6 example for illustration, when Z = 32 is specified as the depth, the central pixel of the target object is used as it is, and when the other range is specified, the pixel value of this pixel is zero or almost zero, and the image is reconstructed.
[0060] 《Effect》
[0061] According to the image processing apparatus 30 according to the present embodiment, optical sectioning can be computationally performed, the light rays from light sources with different depths can be separated from each other in pixel units, and the resolution in the Z direction of the reconstructed image can be improved. And since the image processing can be performed at the frame rate of the optical system 10, three-dimensional imaging of high-speed events can be realized.
[0062] Figure 7This is a diagram for comparing the resolutions of the reconstructed images obtained by the simple averaging method, the weighted averaging method, and the image processing apparatus according to the present embodiment. The simple averaging method means that when overlapping elemental images to reconstruct an image, the average of the pixel values of the overlapping pixels is used as the pixel value of the reconstructed image. The weighted averaging method means that when overlapping elemental images to reconstruct an image, for the overlapping pixels, the weight is relatively larger for the pixels near the center of the region of the elemental image and relatively smaller for the pixels at the edges, and the value after weighted averaging is used as the pixel value of the reconstructed image. Each algorithm sequentially shows, from left to right, the reconstructed image (X-Y diagram) of a certain depth of the light field image data capturing fluorescent particles, the depth image (X-Z diagram) of the dotted line part in the X-Y diagram in the depth direction (Z direction), the graph of the depth versus brightness of the wavy line part in the X-Y diagram, and the graph of the depth versus brightness of the wavy line part in the X-Z diagram. The decimal values written in each graph are the resolutions (full width at half maximum). Compared with the present embodiment, in the reconstructed images obtained by the simple averaging method and the weighted averaging method, the images appear blurred in any direction of the X direction and the Z direction. The resolution of the reconstructed image obtained by the simple averaging method in the X direction is 2.6537 μm, and the resolution in the Z direction is 5.3285 μm. The resolution of the reconstructed image obtained by the weighted averaging method in the X direction is 2.2468 μm, and the resolution in the Z direction is 6.6448 μm. The resolutions of the two are almost the same. In contrast, the resolution of the reconstructed image obtained by the optical sectioning method of the image processing apparatus 30 in the X direction is 0.9074 μm, and the resolution in the Z direction is 1.5050 μm. Compared with the simple averaging method and the weighted averaging method, not only the resolution in the X direction but also the resolution in the Z direction has been greatly improved.
[0063] 《Modification Example》
[0064] The above description is premised on the elemental images being arranged in a grid pattern in the elemental image group. When using a microlens array with microlenses arranged in a honeycomb pattern, the elemental images may be arranged in a honeycomb pattern. Even in that case, by obtaining the interval between the light rays of the same point light source between adjacent elemental images, associating the depth corresponding to that interval with the pixels, and using only the pixels associated with the specified depth when reconstructing the image, a high-resolution image can be obtained.
[0065] In the multi-viewpoint image data, the sub-images do not have to be two-dimensionally arranged and may be arranged in a one-dimensional pattern, either vertically or horizontally.
[0066] The multi-viewpoint image data is not limited to light field images, and any image that is a collection of multiple sub-images of the same subject captured from multiple viewpoints offset from each other is acceptable. For example, it can also be an image captured by a multi-camera with multiple cameras arranged. Moreover, the images captured by each camera in the multi-camera do not have to be aggregated into one image data, and the images captured by each camera can also be saved in their respective files. In that case, the collection of those image files becomes multi-viewpoint image data.
[0067] As described above, as an example of the technology in the present invention, the embodiments have been described. Therefore, the accompanying drawings and detailed descriptions are provided. So, among the components described in the accompanying drawings and detailed descriptions, not only the components necessary to solve the problem are included, but also components that are not necessary to solve the problem may be included for the purpose of exemplifying the technology. Therefore, it should not be immediately considered that those non-essential components are essential just because they are described in the accompanying drawings and detailed descriptions. And, since the above embodiments are for exemplifying the technology in the present invention, various changes, replacements, additions, omissions, etc. can be made within the scope of the claims or its equivalents.
[0068] (Industrial Applicability)
[0069] Since the image processing apparatus according to the present invention can perform high-speed and high-resolution three-dimensional imaging, it is applicable to a light field microscope for observing high-speed events such as biomolecules and intracellular behavior.
[0070] -Symbol Explanation-
[0071] 30: Image processing apparatus
[0072] 31: Element image group generation unit
[0073] 32: Pixel profile generation unit
[0074] 33: Pixel depth determination unit
[0075] 34: Image reconstruction unit
Claims
1. An image processing apparatus that reconstructs an image refocused to an arbitrary depth from multi-viewpoint image data, the multi-viewpoint image data being a set of a plurality of sub-images that are images of the same subject captured from a plurality of viewpoints offset from each other, characterized in that : The image processing apparatus includes an elemental image group generation unit, a pixel profile generation unit, a pixel depth determination unit, and an image reconstruction unit. The elemental image group generation unit automatically generates an elemental image group by cutting out elemental images that are captured images from mutually different viewpoints from the multi-viewpoint image data. The pixel profile generation unit automatically generates a pixel profile representing the relationship between the offset of the elemental images and the pixel values of each pixel, while changing the offset with respect to the elemental images related to adjacent viewpoints in the elemental image group to partially overlap the elemental images, and using the value reflecting the pixel values of those pixels as the pixel value of that pixel for the pixels that overlap between those elemental images. The pixel depth determination unit automatically associates the depth corresponding to the offset of the elemental image when the pixel value of each pixel of each elemental image in the elemental image group takes an extreme value, with respect to each pixel, with reference to the pixel profile. The image reconstruction unit offsets and overlaps the elemental images in the elemental image group with an offset corresponding to the specified depth, and substantially sets the pixel values of the pixels not associated with the specified depth to zero, and automatically reconstructs an image refocused to the specified depth.
2. The image processing apparatus according to claim 1, wherein : The pixel profile generation unit automatically generates the image profile by using the average value of the pixel values of the overlapping pixels as the pixel values of those pixels.
3. The image processing apparatus according to claim 2, wherein : The pixel depth determination unit automatically searches for an extreme value below the initial pixel value in the pixel profile.
4. The image processing apparatus according to claim 1, wherein : The pixel depth determination unit automatically associates the depth corresponding to the offset of the elemental image when the pixel value of each pixel takes a value within a specified range from the extreme value, with respect to each pixel, with reference to the pixel profile.
5. The image processing apparatus according to any one of claims 1 to 4, characterized in that : In the multi-viewpoint image data, the plurality of substantially circular sub-images are arranged in a grid pattern. The elemental image group generation unit automatically cuts out the elemental images of the square regions inscribed in the circle from each of the plurality of sub-images.
6. The image processing apparatus according to any one of claims 1 to 4, characterized in that : The multi-viewpoint image data is a light field image. The sub-images are microlens images.
7. An image processing method is an image processing method executed by a computer for reconstructing an image refocused from multi-viewpoint image data to an arbitrary depth. The multi-viewpoint image data set includes a plurality of sub-images, and the plurality of sub-images are images of the same subject taken from a plurality of viewpoints offset from each other, and is characterized in that : The image processing method includes the step of automatically generating an elemental image group by cutting out elemental images that are captured images from mutually different viewpoints from the multi-viewpoint image data. The step of automatically generating a pixel profile representing the relationship between the offset of the elemental images and the pixel values of each pixel, while changing the offset with respect to the elemental images related to adjacent viewpoints in the elemental image group to partially overlap the elemental images, and using the value reflecting the pixel values of those pixels as the pixel value of that pixel for the pixels that overlap between those elemental images. The step of automatically associating the depth corresponding to the offset of the elemental image when the pixel value of each pixel of each elemental image in the elemental image group takes an extreme value, with respect to each pixel, with reference to the pixel profile, and The step of offsetting and overlapping the elemental images in the elemental image group with an offset corresponding to the specified depth, and substantially setting the pixel values of the pixels not associated with the specified depth to zero, and automatically reconstructing an image refocused to the specified depth.
8. A program that causes a computer to reconstruct an image refocused to an arbitrary depth from multi-viewpoint image data, the multi-viewpoint image data being a collection of a plurality of sub-images that are images of the same subject taken from a plurality of viewpoints offset from each other, characterized in that : The program causes the computer to function as an elemental image group generation unit, a pixel profile generation unit, a pixel depth determination unit, and an image reconstruction unit. The elemental image group generation unit cuts out elemental images that are captured images from mutually different viewpoints from the multi-viewpoint image data, and automatically generates an elemental image group. The pixel profile generation unit changes an offset amount with respect to the elemental images related to adjacent viewpoints in the elemental image group, partially overlaps the elemental images, and, with respect to the pixels that overlap between those elemental images, uses the value reflecting the pixel values of those pixels as the pixel value of that pixel, and automatically generates a pixel profile representing the relationship between the offset amount of the elemental images and the pixel value of each pixel. The pixel depth determination unit, with respect to each pixel of each elemental image in the elemental image group, refers to the pixel profile, and associates the depth corresponding to the offset amount of the elemental image when the pixel value of that pixel takes an extreme value with that pixel. The image reconstruction unit offsets and overlaps the elemental images in the elemental image group with an offset amount corresponding to the specified depth, and makes the pixel values of the pixels not associated with the specified depth substantially zero, and reconstructs an image refocused on the specified depth.