Image processing device, image processing method, and program

Abstract

To reconstruct a clear refocused image from an image captured by a multicore optical fiber with low load and at high speed.SOLUTION: An image processing device 60 comprises: an end-face image processing unit 61 which obtains representative values of pixels in each region of the entire or central part of each core end face in a fiber bundle end-face image, rearranges pixels having the representative values in an xy-coordinate system, and complements other pixels to generate images having different incident angles of light beams; a light field estimation unit 62 which estimates a four-dimensional light field representing an incident angle of each light beam incident on each xy-coordinate by uv vectors from the images having different incident angles of the light beams; and a refocus unit 63 which performs divergence prevention processing by weighting each uv vector in the light field with a coefficient determined according to an arbitrary depth and multiplying the light field, thereby generating a refocused image at the depth.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program, and more particularly to an image processing technique suitable for multi-core optical fiber imaging. [Background technology]

[0002] In recent years, microscopic endoscopes have been widely used as a technique for detailed observation of microstructures in biological and material science. In particular, high-resolution observation is required for non-invasive imaging of living cells and tissues. Against this background, imaging techniques using multi-core optical fibers have attracted attention. Multi-core optical fibers have multiple waveguide cores within a single fiber, and each core can transmit light independently. This enables higher-density information transmission than traditional single-core fibers, and is expected to open up new possibilities in the field of microscopic imaging.

[0003] On the other hand, high-speed, large-scale three-dimensional imaging is required for delicate and dynamic observation of biomolecules and intracellular behavior. In this case, a refocusing technique is required to reconstruct an image refocused at any depth from a captured image. Unlike light field camera images captured through a microlens array, images captured through a multi-core optical fiber do not contain parallax information, which often poses a problem for reconstructing the refocused image. Regarding this issue, a method has been proposed in which a light field is estimated from an image of the end face of a multi-core optical fiber and an image refocused at any depth is generated based on the light field (see, for example, Patent Documents 1 and 2 and Non-Patent Documents 1 and 2). This technique enables the generation of images with multiple focal depths from a single capture in an endoscopic microscope using a multi-core optical fiber by post-processing, thereby realizing more flexible imaging. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2021-511871 [Patent Document 2] Japanese Patent Application Publication No. 2023-134456 [Non-patent literature]

[0005] [Non-Patent Document 1] A. Orth et al., “Extended depth of field imaging through multicore optical fibers,” Opt. Express 26(5), 6407-6419 (2018). [Non-patent document 2] A.Orth et al., “Optical fiber bundles: Ultra-slim light field imaging probes.” Sci. Adv.5, eaav1555(2019). DOI:10.1126 / sciadv.aav1555 Summary of the Invention [Problem to be solved by the invention]

[0006] A 3D image of an object can be reconstructed by shifting and superimposing images with parallax to reconstruct refocused images at multiple depths, then stacking them in the depth direction. However, microscope images are blurred due to the influence of the PSF (Point Spread Function), and refocusing based on such images results in a reconstructed 3D image that is stretched in the depth direction. While deconvolution can remove the influence of the PSF and obtain a clear 3D image, deconvolution has the problem of being computationally intensive and taking a long time to process.

[0007] In view of the above problems, an object of the present invention is to reconstruct a clear refocused image, and further a three-dimensional image, from an image captured by a multi-core optical fiber at high speed with a light load and without deconvolution processing. [Means for solving the problem]

[0008] According to one aspect of the present invention, there is provided an image processing device that reconstructs a refocused image of an arbitrary depth from an end face image of a fiber bundle formed by bundling a large number of optical fibers that are waveguide cores, the image processing device comprising: an end face image processing unit that calculates a representative value of pixels in each region of the entire and central part of each core end face in the end face image, rearranges the pixels having the representative value in an xy coordinate system, and complements the other pixels to generate images with different angles of incidence of light rays; a light field estimation unit that estimates a four-dimensional light field in which the angle of incidence of each light ray incident on each xy coordinate is expressed as a uv vector from the images with different angles of incidence of the light rays; and a refocusing unit that weights each uv vector in the light field by a coefficient determined according to the arbitrary depth, multiplies the light field by the weighted uv vector, and performs divergence prevention processing to generate a refocused image of the depth. A corresponding image processing method and a computer program for causing a computer to execute the above image processing are also provided.

[0009] Furthermore, an image processing device equipped with a three-dimensional image reconstruction unit that reconstructs a three-dimensional image based on the refocused images generated at a plurality of different depths, a corresponding image processing method, and a computer program for causing a computer to execute the above image processing are provided. [Effects of the Invention]

[0010] According to the present invention, a light field is estimated from an image of the end face of a fiber bundle, and by performing shift multiplication on the light field, it is possible to reconstruct clear refocused images and even three-dimensional images quickly and with a low load without performing deconvolution processing. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram of a microscopic endoscope system including an image processing device according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an entire end face of a fiber bundle according to an example. [Figure 3] 1 is a flowchart of an image processing method according to an embodiment of the present invention. [Figure 4] This is an enlarged image of a portion of the fiber bundle end face capturing the fluorescence of multiple fluorescent beads. [Figure 5] 10A and 10B are diagrams illustrating differences in the angle of incidence of light rays onto a core end face due to differences in the depth of the light source. [Figure 6] 10A and 10B are diagrams illustrating differences in luminance distribution at the core end face due to differences in the angle of incidence of light rays onto the core end face. [Figure 7] 10A and 10B are diagrams illustrating how images with different incident angles of light rays are generated from pixels in the entire and central regions of each core end face in a fiber bundle end face image. [Figure 8] FIG. 1 is a diagram illustrating a four-dimensional light field. [Figure 9] FIG. 10 is a diagram illustrating shift multiplication of a light field. [Figure 10] 10A and 10B are diagrams illustrating divergence prevention processing in a local region in shift multiplication. [Figure 11] FIG. 10 is a diagram comparing three-dimensional images reconstructed by shift-add and shift-multiply. [Figure 12] 10 is a graph plotting the depth of an observation object calculated from a three-dimensional image reconstructed by the image processing device according to the embodiment when the distance from the object-side end face of the fiber bundle to the observation object is changed. [Figure 13] This figure compares three-dimensional images reconstructed by processing images of neurons in the hypothalamus of a mouse captured by a fiber bundle using shift-addition and shift-multiplication, respectively. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present invention, and are not intended to limit the subject matter described in the claims. Furthermore, the dimensions of each component depicted in the drawings, the detailed shapes of the details, and the like may differ from the actual ones.

[0013] <Embodiment> 1 is a schematic diagram of a microscopic endoscope system including an image processing device according to one embodiment of the present invention. The microscopic endoscope system 100 according to this embodiment reconstructs a refocused image at any depth, and even a three-dimensional image, from a high-resolution image of an object captured by a fiber bundle consisting of multiple optical fibers. Broadly speaking, the microscopic endoscope system 100 includes a fiber bundle 10, an objective lens 20, a light source 30, an optical system 40, an image sensor 50, and an image processing device 60.

[0014] The fiber bundle 10 is a multi-core optical fiber in which several thousand to several tens of thousands of waveguide cores (hereinafter simply referred to as cores) are bundled. Each core is made of, for example, quartz and is fused and integrated with the cladding. As an example, the fiber bundle 10 is made of approximately 30,000 bundled cores, and an approximately circular image circle with a diameter of approximately 600 μm is formed at its end face. The image circle is coated with a material such as silicone resin or polyimide resin, and the coating diameter is approximately 750 μm. Each core can transmit visible light, and light incident on one end face of the fiber bundle 10 is transmitted to the other end face through each core. The arrangement of each core is matched at both end faces of the fiber bundle 10. Therefore, an image captured by the image circle at one end face of the fiber bundle 10 appears without distortion on the image circle at the other end face.

[0015] Figure 2 is a diagram showing the entire end face of an example fiber bundle. This end face is a view of the other end face when uniform light is incident on one end face of the fiber bundle 10. The granular parts distributed in large numbers on the roughly circular end face of the fiber bundle 10 are the end faces of the individual cores. The black areas where there are no granules are defective cores. Defective cores are black because they cannot transmit light.

[0016] Returning to FIG. 1 , the objective lens 20 is disposed near one end face of the fiber bundle 10. For convenience, the end face of the fiber bundle 10 closest to the object of observation will be referred to as the object-side end face 10a, and the opposite end face closest to the objective lens 20 will be referred to as the camera-side end face 10b. Light from the light source 30 is incident on the objective lens 20 via the optical system 40 and is then irradiated onto the camera-side end face 10b of the fiber bundle 10 through the objective lens 20. The irradiated light is transmitted to the object-side end face 10a of the fiber bundle 10 and illuminates the object of observation. Meanwhile, reflected light from the object of observation, particularly fluorescence from the object, is incident on the object-side end face 10a of the fiber bundle 10, and this light is transmitted to the camera-side end face 10b of the fiber bundle 10. The objective lens 20 collects the light transmitted to the camera-side end face 10b of the fiber bundle 10.

[0017] As described above, the light source 30 supplies light for illuminating an object to be observed at the object-side end surface 10a of the fiber bundle 10. For example, the light emitted from the light source 30 is laser light of a specific wavelength for causing the object to fluoresce.

[0018] The optical system 40 includes various lenses, mirrors, etc. The collimator lens 41 collimates the light emitted from the light source 30. A mirror 42, a collimator lens 43, a dichroic mirror 44, and the objective lens 20 are arranged in this order ahead of the collimated light in the traveling direction. The light collimated by the collimator lens 41 passes through the mirror 42, the collimator lens 43, the dichroic mirror 44, and the objective lens 20 and is incident on the camera-side end face 10b of the fiber bundle 10, and the light is transmitted to the object-side end face 10a of the fiber bundle 10 to illuminate the object being observed.

[0019] Meanwhile, a dichroic mirror 44, a fluorescence filter 45, and a tube lens 46 are arranged ahead in the traveling direction of the light collected by the objective lens 20. The dichroic mirror 44 is configured to transmit light in the wavelength range of the fluorescence of the object being observed. The light collected by the objective lens 20 passes through the dichroic mirror 44, passes through the fluorescence filter 45 and the tube lens 46, and is incident on the image sensor 50. Then, a substantially circular image of the camera-side end face 10b of the fiber bundle 10 is formed on the sensor area of ​​the image sensor 50.

[0020] The image sensor 50 is a CCD sensor, CMOS sensor, or the like configured with a two-dimensional array of multiple light-receiving elements (not shown) that photoelectrically convert incident light and output electrical signals. The electrical signals output from each light-receiving element are converted into digital data by an A / D converter (not shown). That is, the approximately circular image of the camera-side end face 10b of the fiber bundle 10 is converted into image data and supplied to the image processing device 60. Hereinafter, the image data of the image of the camera-side end face 10b of the fiber bundle 10 may be referred to as a fiber bundle end face image.

[0021] The image processing device 60 includes an end face image processing unit 61, a light field estimation unit 62, a refocusing unit 63, and a 3D image reconstruction unit 64. Each of these components can be realized as hardware such as an ASIC or FPGA, or as software in which a computer program stored in a recording medium or storage device (not shown) is executed by a processor (not shown) such as a CPU or GPU. It can also be realized by appropriately combining hardware and software. While performing image processing, the image processing device 60 appropriately accesses various storage devices that make up the storage device to write and read data.

[0022] FIG. 3 is a flowchart of an image processing method according to one embodiment of the present invention. Each step is executed by the above-described components of the image processing device 60. First, as a preliminary step, a uniform light is incident on the object-side end face 10a of the fiber bundle 10, and an image of the camera-side end face 10b is captured to obtain an image of the entire end face of the fiber bundle 10, as shown in FIG. 2. This image is then sent to the image processing device 60, which then identifies the end face of each core from the fiber bundle end face image and determines the x-y coordinates of the center point of each core (S0). As described above, since the core end faces appear bright and approximately circular in the fiber bundle end face image, the core end faces can be easily identified by adjusting the contrast of the fiber bundle end face image and emphasizing the edges.

[0023] It is not necessary to perform this advance preparation every time an observation is made. For example, step S0 may be performed in advance at a time other than the time of observation, and an image of the entire end face of the fiber bundle 10 and the xy coordinates of each core center point obtained from the image may be recorded in a storage device (not shown), and the xy coordinates of each core center point may be read out whenever necessary.

[0024] Next, when a fiber bundle end face image capturing an object to be observed by the fiber bundle 10 is input to the image processing device 60, the end face image processing unit 61 generates images with different angles of incidence of light rays from pixels in the entire area of ​​each core and in each area of ​​the center in the end face image (S1).

[0025] FIG. 4 is an excerpted and enlarged view of an example fiber bundle end face image. This fiber bundle end face image was obtained by capturing a fluorescent image of multiple fluorescent beads in the fiber bundle 10. In the fiber bundle end face image, each core end face appears as a bright, approximately circular granular area, while the cladding between cores and defective cores are dark. Each core end face in the fiber bundle end face image is made up of multiple pixels. The core end faces are not all uniformly bright; each core has a different brightness and pattern. This is largely due to differences in the angle of incidence of light rays on the core end faces at the object-side end face 10a of the fiber bundle 10. Furthermore, differences in the angle of incidence of light rays on the core end faces are caused by differences in the depth of the light source.

[0026] FIG. 5 is a diagram illustrating the difference in the angle of incidence of light rays on the core end face due to differences in the depth of the light source. The up-down direction in FIG. 5 corresponds to the axial direction of the fiber bundle 10, and in FIG. 5, the object-side end face 10a of the fiber bundle 10 is viewed from a direction perpendicular to the axis of the fiber bundle 10. Each core end face 11a of the five horizontally arranged cores 11 corresponds to the object-side end face 10a of the fiber bundle 10. Two point light sources P1 and P2 are located at the end of the object-side end face 10a of the fiber bundle 10 (the lower side in the figure). Point light source P1 is located at a shallow depth, and point light source P2 is located at a deep depth. Light ray L1 from point light source P1 is incident on the central core end face 11a at a small angle of incidence, i.e., an angle close to the axial direction of the core 11. Meanwhile, ray L1 from point light source P2 is also incident on the central core end face 11a at a small angle of incidence, but ray L2 is incident on the core end faces 11a on both sides of the central core end face 11 at a large angle of incidence, and ray L3 is incident on the outer core end face 11a at an even larger angle of incidence. In this way, the light rays from light sources located farther from the object-side end face 10a of the fiber bundle 10, i.e., at a greater depth, are incident on the core end face 11a at larger angles of incidence.

[0027] The position vector of the point light source is r → , the depth of the point light source is z iIf the spread angle of light from a point light source is θ, the PSF of the light emitted from the point light source can be modeled as a two-dimensional Gaussian distribution as expressed by the following equation (1). TIFF2025156220000002.tif14156

[0028] FIG. 6 illustrates the difference in luminance distribution at the core end face due to different incident angles of light rays. When the incident angle of a light ray on the core end face is small, as with the light ray L1 shown in FIG. 5, light propagates through the core in low-order waveguide modes, primarily the fundamental mode, resulting in a luminance distribution D1 with a peak at the core center at the core end face at the camera-side end face 10b of the fiber bundle 10. On the other hand, when the incident angle of a light ray on the core end face is large, as with the light rays L2 and L3 shown in FIG. 5, light propagates through the core in high-order waveguide modes, resulting in luminance distributions D2 and D3 with multiple peaks at symmetrical positions on concentric circles offset from the core center toward the outer edge at the core end face at the camera-side end face 10b of the fiber bundle 10. Note that the locations of the luminance peaks at the core end face for the luminance distributions D2 and D3 are different, due to the difference in the azimuth angle of the light ray incident on the core end face. Therefore, by reconstructing an image based on the pixels at the center of each core end face in the fiber bundle end face image, an image including light rays with small incident angles can be obtained. On the other hand, by reconstructing an image based on all the pixels of each core end face in the fiber bundle end face image, it is possible to obtain an image that includes light rays with large angles of incidence.

[0029] First, the end face image processing unit 61 identifies each core end face in the fiber bundle end face image and determines the x and y coordinates of each core center point. As described above, since each core end face in the fiber bundle end face image has already been identified and the x and y coordinates of each core center point determined in step S0, it is possible to easily identify a circular area with a certain spread from the core center point as a core end face. Note that at this time, dark core end faces below a threshold value may be ignored.

[0030] Once each core end face in the fiber bundle end face image is identified, the end face image processor 61 generates images with different light ray incident angles from each identified core end face. Figure 7 illustrates how images with different light ray incident angles are generated from pixels in the entire and central regions of each core end face in the fiber bundle end face image. The end face image processor 61 crops each identified core end face using a large-aperture circle that covers the entire core and a small-aperture circle that covers only the core center, calculates a representative value for the pixels within each circle, and relocates the pixel with this representative value to the x-y coordinate of the center point of each core. The representative value can be the average or median of the pixel values ​​within each circle. At this time, for x-y coordinates where pixels cannot be relocated due to core defects or misalignment of the cores, the pixel value is interpolated from surrounding pixels. In this way, an image containing light rays with large incident angles is generated from the pixels of the core end face cropped using the large-aperture circle, and an image containing light rays with small incident angles is generated from the pixels of the core end face cropped using the small-aperture circle. The circles of the large and small openings corresponding to each core end face can be easily set as circles having a constant radius from the x and y coordinates of each core center point.

[0031] Returning to FIG. 3, the light field estimation unit 62 estimates a four-dimensional light field in which the angle of incidence of each ray incident on each xy coordinate is expressed as a uv vector from images with different angles of incidence of the ray (S2). FIG. 8 is a diagram illustrating a four-dimensional light field. Rays of light are incident on each xy coordinate from various directions. By expressing the angle of incidence of each ray as a vector in uv space, the light field can be expressed as L(x, y, u, v). u and v represent the inclination angle from the z axis, which is the axial direction of each core, to each of the x and y axes. For example, a ray passing through coordinates (x1, y1) in xy space is expressed as (u1, v1), ..., (u m ,v n ) where m and n are any positive integers. This four-dimensional light field can be obtained by calculating the light field moment vector for each x and y coordinate, i.e., the average light ray direction, from an image containing rays with a large angle of incidence and an image containing rays with a small angle of incidence using the Intensity Transport Equation, which describes the propagation of light intensity.

[0032] When the end face image processing unit 61 generates two types of images, a large aperture image and a small aperture image as shown in Figure 7, which are images with different incident angles of light rays, if the small aperture image is I0 and its maximum convergence angle (aperture) is θ0, and the large aperture image is I1 and its maximum convergence angle (aperture) is θ1, LMI (Light field Moment Imaging) is expressed as the following equation (2). TIFF2025156220000003.tif13166However, ▽ ⊥ =[∂ / ∂x,∂ / ∂y], I=(I0+αI1) / 2, M e → is the effective LM vector.

[0033] From equation (2), the effective LM vector M e → By solving this, the light field is expressed as the following equation (3). TIFF2025156220000004.tif15161 where I is the light intensity of the object, u and v are the inclination angles (angle of the light ray) from the z axis to the x axis and y axis, respectively, and σ is a parameter defined by tan θ', which is empirically given as the incident angle corresponding to each core in tan θ.

[0034] Substitute equation (1) for the light intensity I in equation (3). At this time, the depth z i If there is a point light source at, the light field is expressed by the following equation (4). TIFF2025156220000005.tif16161

[0035] By converting Equation (4) into a light field calculation formula based on the transport of intensity equation through a geometric transformation, we obtain the following Equation (5). TIFF2025156220000006.tif30161

[0036] The end face image processing unit 61 may generate three or more types of images with different incident angles of light rays from each identified core end face. In this case, the light field is determined based on the relationship between these images and the effective LM vector.

[0037] Returning to Figure 3, the refocusing unit 63 weights each uv vector in the light field by a coefficient determined according to a given depth, multiplies the light by the field, and performs divergence prevention processing to generate a refocused image at that depth (S3). The light source of a light ray incident at a large angle to the core end face 11a, such as light rays L2 and L3 shown in Figure 5, is located at a position shifted in the x and y directions when viewed from the core end face 11a. The shift in the center of gravity of such a light source can be determined by multiplying the uv vector representing the angle of incidence of the light ray by a coefficient determined by the depth of the light source and fixed optical parameters (see Patent Documents 1 and 2 and Non-Patent Document 2).

[0038] The shifting of the center of gravity of the light source can be considered similar to parallax. For example, in the case of an image captured by a light field camera, an image refocused at a given depth can be reconstructed by shifting and superimposing multiple elemental images with parallax according to a given depth. When this principle is applied to an image of a fiber bundle end face, an image refocused at a given depth can be reconstructed by weighting each UV vector in the light field with a coefficient determined according to the depth of the light source and superimposing light fields in which the center of gravity of the light source has been shifted. However, the refocusing unit 63 superimposes light fields using shift multiplication rather than shift addition.

[0039] From equation (5), the depth z i Taking the center of gravity of the light field at zi When calculating this coefficient, if an adjustment parameter σ0 is set to the PSF expressed by equation (1) so that the behavior of the PSF is physically correct, the PSF is transformed as shown in equation (6) below. TIFF2025156220000007.tif14159

[0040] Using equation (6), the light field of equation (5) is redefined as the following equation (7). TIFF2025156220000008.tif30160

[0041] The center of gravity for u and v is calculated for the light field of equation (7). The center of gravity of this light field is the u and v vector corresponding to the position of the light ray relative to the depth, i.e., the gradient. The center of gravity is calculated using the definition of equation (8) below. TIFF2025156220000009.tif15157

[0042] In equation (8), ∫L zi dudv∝z i 2 tan 2 θ0+σ0 2 +2ln(2)σ 2 , ∫(u,v)L zi dudv∝z i 2 tan 2 θ0+σ 2 Therefore, equation (8) can be transformed into the following equation (9) through the definition of the center of gravity and the variable transformation h=σ / tanθ0. TIFF2025156220000010.tif14157

[0043] FIG. 9 is a diagram explaining the shift multiplication of light fields. A refocused image at an arbitrary depth is generated by weighting each uv vector in the light field with a coefficient determined according to the arbitrary depth and multiplying the light fields. For example, at a depth z i Refocused image I zi is expressed by the following equation (10), which converts the uv vector in the light field L(x, y, u, v) to the depth z i The coefficient w in the uv space expressed by equation (9) corresponds to zi The light field is obtained by weighting the uv vector with the weighted light field. The operator * denotes multiplication. TIFF2025156220000011.tif31161

[0044] In shift multiplication, the light field is cumulatively multiplied, so the pixel values ​​of the x and y coordinates may become exponentially larger and diverge. Therefore, the refocusing unit 63 performs divergence prevention processing, for example, zi Divergence of pixel values ​​is prevented by normalizing the pixel values ​​of the entire image by the maximum pixel value among them. For example, divergence of pixel values ​​can be prevented by dividing the pixel values ​​of the entire image by the maximum pixel value. In addition to normalization, standardization or normalization within a predetermined value range may also be performed as a divergence prevention process.

[0045] Shift multiplication significantly reduces light rays other than those from the light source at the desired depth during the multiplication process, making the desired light rays clearer and enabling the reconstruction of a 3D image that does not elongate in the depth direction. On the other hand, only the image with the maximum pixel value in the image may be emphasized, while other images may be lost as background noise. This is not a problem when only one image is being observed, but it becomes a problem when multiple images are being observed, as the multiple images will not appear in the refocused image. To solve this problem, divergence prevention processing may be performed for each local region of the xy coordinate system.

[0046] 10 is a diagram illustrating divergence prevention processing in a local region in shift multiplication. The refocusing unit 63 sets a local region in the xy coordinate system of the refocused image, for example, a local region including the origin of the xy coordinate system (the upper left point in the figure), and performs divergence prevention processing of pixel values ​​in the local region, for example, normalizing the pixel values ​​in the local region by the maximum pixel value in the local region. The refocusing unit 63 then sequentially shifts the local region in the refocused image and repeats the above-mentioned divergence prevention processing. This makes it possible to generate a refocused image including multiple images of various brightnesses, without images other than those having the maximum pixel value in the entire image being buried in background noise in shift multiplication.

[0047] 3, the three-dimensional image reconstruction unit 64 reconstructs a three-dimensional image based on the refocused images generated at a plurality of different depths (S4). For example, the refocus unit 63 reconstructs a three-dimensional image based on the refocused images generated at depths z0 to z1. nOnce the refocused images for each depth up to are generated, the three-dimensional image reconstruction unit 64 can reconstruct a three-dimensional image by, for example, stacking these images.

[0048] Effect Figure 11 compares three-dimensional images reconstructed using shift-addition and shift-multiplication. Note that the shift-multiplication technique also includes the anti-divergence processing described above. These figures were reconstructed from end-face images of the fiber bundle 10 capturing the fluorescence of the same multiple fluorescent beads. (a) In the three-dimensional image reconstructed using conventional shift-addition, the images of each fluorescent bead are blurred and elongated in the depth direction. To sharpen these images, deconvolution processing is required, which takes time. (b) In the three-dimensional image reconstructed using shift-multiplication according to this embodiment (without local anti-divergence processing), the images of the fluorescent beads are clearly reconstructed, even though deconvolution processing has not been performed. However, because local anti-divergence processing has not been performed, only the image of the fluorescent bead with the maximum pixel value is emphasized, and the other images have been erased. On the other hand, (c) in the three-dimensional image reconstructed using shift-multiplication according to this embodiment (with local anti-divergence processing), the images of multiple fluorescent beads are clearly reconstructed. In this way, when there are multiple objects to be observed, by performing divergence prevention processing for local regions in combination with shift multiplication, it is possible to reconstruct a clear three-dimensional image of each object to be observed without elongation in the depth direction.

[0049] 12 is a graph plotting the depth of an observed object calculated from a three-dimensional image reconstructed by the image processing device according to the embodiment when the distance from the object-side end face of the fiber bundle to the observed object is changed. Specifically, fluorescent beads with a diameter of 2 μm were used as the observed object, and the distance from the object-side end face 10a of the fiber bundle 10 to the observed object was changed in 5 μm increments from 30 to 75 μm. Images of the end face of the fiber bundle 10 obtained at each change were processed by the image processing device 60, and the depth of the observed object was calculated from the XZ cross section of the reconstructed three-dimensional image. The vertical axis of the graph represents the depth of the observed object [μm] calculated from the reconstructed three-dimensional image, and the horizontal axis represents the distance [μm] of the observed object in the Z direction (depth direction) from the reference position, which is 30 μm, a distance between the object-side end face 10a of the fiber bundle 10 and the observed object. As can be seen from the graph, the error between the depth calculated from the three-dimensional image reconstructed by the image processing device 60 and the ground truth is very small at any depth, regardless of the distance from the object-side end surface 10a of the fiber bundle 10 to the observed object. This means that the image processing device 60 according to this embodiment can accurately obtain the depth of the observed object.

[0050] Figure 13 compares three-dimensional images of neurons in the hypothalamus of a mouse reconstructed by processing images captured by a fiber bundle using shift-addition and shift-multiplication, respectively. While Figure 11 shows a reconstructed image of an artificial object (fluorescent beads), Figure 13 shows a reconstructed image of active neurons in a living organism (mouse). Specifically, labeled neurons whose fluorescence intensity changes in response to neural activity in the hypothalamus, a deep brain region of a mouse, are captured continuously over time using a fiber bundle 10, and the end-face images are reconstructed into three-dimensional images using shift-addition and shift-multiplication, respectively. (a) In the three-dimensional image reconstructed by conventional shift-addition, the images of each neuron are blurred and elongated in the depth direction, and background light is also present. On the other hand, (b) in the three-dimensional image reconstructed by shift-multiplication according to this embodiment (with local region divergence prevention processing), the background light is removed, and the reconstruction is clear enough that each neuron can be observed separately as a granular image.

[0051] As described above, the image processing device 60 according to this embodiment can reconstruct a refocused image and even a three-dimensional image from the end face image of the fiber bundle 10 at high speed and with a low load without deconvolution processing.

[0052] As described above, the embodiments have been described as examples of the technology of the present invention. For this purpose, the accompanying drawings and detailed description have been provided. Therefore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the above technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential. Furthermore, because the above-described embodiments are intended to exemplify the technology of the present invention, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0053] The image processing device according to the present invention can reconstruct clear refocused images and even three-dimensional images from images captured by a multi-core optical fiber at high speed with a low load without deconvolution processing, and is therefore useful for a microscopic endoscope system for observing biological structures and minute structures in materials science in detail. [Explanation of symbols]

[0054] 60 Image processing device 61 End face image processing unit 62 Light field estimation unit 63 Refocusing section 64 3D image reconstruction unit

Claims

1. An image processing device that reconstructs a refocused image at an arbitrary depth from an end face image of a fiber bundle formed by bundling a large number of optical fibers that are waveguide cores, an end face image processing unit that calculates a representative value of pixels in each region of the entire and central part of each core end face in the end face image, rearranges pixels having the representative value in an xy coordinate system, and complements other pixels to generate images with different angles of incidence of light rays; a light field estimation unit that estimates a four-dimensional light field in which the incident angles of each light ray incident on each xy coordinate are expressed as uv vectors from the images having different incident angles of the light ray; a refocusing unit that weights each uv vector in the light field by a coefficient determined according to an arbitrary depth, multiplies the weighted uv vector in the light field, and performs divergence prevention processing to generate a refocused image at the depth.

1. An image processing device comprising:

2. The image processing apparatus according to claim 1 , further comprising a three-dimensional image reconstruction unit that reconstructs a three-dimensional image based on the refocused images generated at a plurality of different depths.

3. The image processing device according to claim 1 , wherein the refocusing unit performs the divergence prevention processing for each local region of an xy coordinate system.

4. An image processing method for reconstructing a refocused image at an arbitrary depth from an end face image of a fiber bundle formed by bundling a large number of optical fibers that are waveguide cores, comprising: a step of automatically generating images with different angles of incidence of light rays by determining a representative value of pixels in each region of the entire and central areas of each core end face in the end face image, and relocating pixels having the representative value in an xy coordinate system and complementing other pixels; A step of automatically estimating a four-dimensional light field in which the incident angles of each ray incident on each xy coordinate are expressed as uv vectors from the images with different incident angles of the ray; and a step of weighting each uv vector in the light field by a coefficient determined according to an arbitrary depth, multiplying the weighted uv vector in the light field, and performing divergence prevention processing to automatically generate a refocused image at the depth. An image processing method comprising:

5. A program for causing a computer to reconstruct a refocused image at an arbitrary depth from an end face image of a fiber bundle formed by bundling a large number of optical fibers that are waveguide cores, an end face image processing means for determining a representative value of pixels in each region of the entire and central part of each core end face in the end face image, and rearranging pixels having the representative value in an xy coordinate system and complementing other pixels to generate images with different angles of incidence of light rays; a light field estimation means for estimating a four-dimensional light field in which the incident angles of each light ray incident on each xy coordinate are expressed as uv vectors from the images having different incident angles of the light ray; a refocusing means for weighting each uv vector in the light field with a coefficient determined according to an arbitrary depth, multiplying the weighted uv vector in the light field with the light field, and performing divergence prevention processing to generate a refocused image at the depth; A program that makes a computer function as a

Images