Incoherent digital hologram signal processing device and imaging device

Abstract

To provide an incoherent digital hologram signal processing device capable of obtaining a depth-of-field expansion image and a depth map from a digital hologram with high accuracy and high efficiency and to provide an imaging device.SOLUTION: The incoherent digital hologram signal processing device includes subject absence domain determination means 34 for obtaining the subject absence domain of a reproduction distance where a subject cannot exist on the basis of the focal distance of each of optical elements including a lens and a spherical phase providing element constituting an optical system for photographing incoherent digital holography and the arrangement interval of each of the optical elements and is configured so as to previously exclude the search processing of the reproduction image of the subject absence domain obtained by the subject absence domain determination means 34 from the search processing of the reproduction image having a local area where sharpness has the highest value in reproduction distance / luminance value extraction means 32.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] The present invention relates to digital holography, and more particularly to an incoherent digital hologram signal processing device that performs signal processing for incoherent digital holography, and an imaging device equipped with the same. [Background technology]

[0002] Incoherent digital holography is a technology that does not require laser light and can capture holograms using incoherent light such as sunlight, LED, and fluorescent light (see Non-Patent Document 1). Using this technology, it is possible to build equipment that could not be realized with conventional digital holography that uses laser light, such as a 3D fluorescent microscope, a 3D camera under sunlight, and a 3D measuring device for radiation temperature distribution (see Non-Patent Document 1).

[0003] In incoherent digital holography, the reflected light from the subject is converted into a hologram by applying the self-interference of incoherent light, and this is then captured as a digital hologram by an imaging element. A signal processing unit applies Fresnel propagation, which allows the light to propagate an arbitrary distance, and angular spectrum calculations to the captured digital hologram, thereby obtaining a two-dimensional reconstructed image of the subject. Three-dimensional information about the subject can be reconstructed based on a group of reconstructed images (group of refocused images) obtained by continuously changing the propagation distance (propagation distance). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Joseph Rosen, Simon Alford, Vijayakumar Anand, Jonathan Art, Petr Bouchal, Zdenek Bouchal, Munkh-Uchral Erdenebat, Lingling Huang, Ayumi Ishii, Saulius Juodkazis, Nam Kim, Peter Kner, Takako Koujin, Yuichi Kozawa, Dong Liang, Jun Liu, Christopher Mann, Abhijit Marar, Atsushi Matsuda, Teruyoshi Nobukawa, Takanori Nomura, Ryutaro Oi, Mariana Potcoava, Tatsuki Tahara, Bang L. Thanh, and Hongqiang Zhou, "Roadmap on Recent Progress in FINCH Technology," Journal of Imaging, (2021), vol. 7, 197. [Non-Patent Document 2] Conor P. McElhinney, Bryan M. Hennelly, and Thomas J. Naughton, "Extended focused imaging for digital holograms of macroscopic three-dimensional objects", Applied Optics, (2008), vol.47, pp.D71-D79. Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, digital holograms contain three-dimensional information about a subject. However, because the three-dimensional information about the subject is reflected as a complex two-dimensional phase distribution, it is not possible to quantitatively obtain information about the subject's depth distance directly from the digital hologram.

[0006] Typically, to obtain depth information of a subject from a digital hologram, the reconstructed image with the highest sharpness is identified from a group of refocused images reconstructed by changing the propagation distance at equal intervals, and the propagation distance at which this reconstructed image is obtained is regarded as the depth distance of the subject (see non-patent document 2). However, in this method, the propagation distance is changed and propagation calculations are performed even in areas where the subject cannot exist, which imposes a large calculation load and requires a long calculation time.

[0007] An object of the present invention is to provide an incoherent digital hologram signal processing device and an imaging device that can obtain an extended depth of field image and a depth map from a digital hologram with high accuracy and high efficiency. [Means for solving the problem]

[0008] In order to achieve the above object, a first incoherent digital hologram signal processing device of the present invention comprises: Incoherent digital hologram information obtained by photographing an object using self-interference of light is input, and the input incoherent digital hologram information is a reconstructed image group acquiring means for successively applying a propagation calculation while changing a reconstruction distance to acquire a reconstructed image group consisting of a plurality of reconstructed images having different reconstruction distances; a reconstruction distance / brightness value extraction means for focusing on any corresponding pixel position in each of the reconstructed images of the obtained reconstructed image group, evaluating the sharpness within a predetermined local area centered on the pixel position, and extracting the reconstruction distance of the predetermined reconstructed image from which the local area having the highest sharpness value was obtained among the reconstructed images of the reconstructed image group, and a brightness value at the pixel position related to the local area having the highest sharpness value; a depth of field extended image / depth map acquisition means for acquiring a depth of field extended image and a depth map of the reproduction distance based on the reproduction distance and the luminance value extracted by the reproduction distance / luminance value extraction means for each of a plurality of pixel positions included in an image area of ​​the reproduction image; and an object-absent region determining means for determining an object-absent region of the reproduction distance in which an object cannot exist, based on the focal length of each of optical elements including a lens and a spherical phase imparting element constituting an optical system for capturing an incoherent digital hologram, and on the arrangement interval of each of the optical elements, The reproduction distance / brightness value extraction means is characterized in that it is configured to preliminarily exclude from the search process of a predetermined reproduction image that searches for the reproduction image relating to the local area having the highest sharpness value the search process of the reproduction image of the object-absent region determined by the object-absent region determination means.

[0009] Here, it is preferable that the plurality of pixels included in the image area of ​​the reconstructed image are all pixels that constitute the reconstructed image.

[0010] Moreover, the above-mentioned "predetermined local region centered on that pixel position" refers to the entire region including the pixel position as the center and consisting of, for example, a total of 9 pixels, with the pixels arranged so as to surround the periphery of the pixel. Of course, the pixels may be arranged so as to further surround the periphery of the region consisting of a total of 9 pixels, for example, to form a region consisting of a total of 25 pixels. Moreover, it is not excluded that the above-mentioned "predetermined local region centered on that pixel position" is composed of each pixel (one pixel). Moreover, the shape of the local region is not limited to a rectangle, and other shapes such as a circle or an ellipse can also be applied.

[0011] A second incoherent digital hologram signal processing device according to the present invention comprises: In the first incoherent digital hologram signal processing device, a depth resolution calculation means for calculating a depth resolution based on information on the focal length and aperture diameter of each of the optical elements including the lens and the spherical phase imparting element constituting the optical system for capturing the incoherent digital hologram, the central wavelength and wavelength width of a wavelength filter, the pixel pitch parameters of an image pickup element, and the arrangement position of each of the optical elements; The reconstructed image group acquisition means is characterized by comprising a propagation distance increment setting means for setting the increment of the propagation distance to each reconstructed image when obtaining a plurality of reconstructed images with different reconstruction distances so that the increment is longer in an area where the depth resolution value calculated by the depth resolution calculation means is lower.

[0012] Here, in the second incoherent digital hologram signal processing device, It is preferable that the plurality of pixels included in the image area of ​​the reconstructed image are all pixels that constitute the reconstructed image.

[0013] A first imaging device of the present invention is characterized by comprising the first incoherent digital hologram signal processing device, an optical system for capturing the incoherent digital hologram, and an imaging element.

[0014] Further, a second imaging device of the present invention is In the first imaging device, The incoherent digital hologram signal processing device is characterized in that it includes a propagation distance increment setting means for setting the propagation distance increment interval so that the interval is longer in an area with a lower depth resolution value, in accordance with the depth resolution value determined based on the focal length and aperture diameter of each optical element including the lens and spherical phase imparting element that constitute the optical system, the central wavelength and wavelength width of the wavelength filter, the pixel pitch parameters of the imaging element, and the positional position of each of the optical elements.

[0015] Furthermore, a third imaging device of the present invention is It is preferable for the present invention to comprise any of the incoherent digital hologram signal processing devices described above, an optical system and an imaging element that utilize self-interference of light to capture three-dimensional image information of a subject input to the incoherent digital hologram signal processing device, and a hologram display device that receives the three-dimensional image information of the subject that has been subjected to signal information processing by the incoherent digital hologram signal processing device and displays a three-dimensional reproduced image of the subject. Effect of the Invention

[0016] According to the incoherent digital hologram signal processing device and imaging device of the present invention, the range of reproduction distances in which a subject may exist is determined by the focal lengths and respective arrangement spacing of the optical elements (lenses, spherical phase imparting elements) that constitute the incoherent digital holography imaging device, and at least reproduction distances that fall outside that range are excluded in advance from the search range for the reproduction distance at which sharpness is highest. This makes it possible to obtain a depth map and an extended depth-of-field image of the subject with high accuracy and high efficiency from a digital hologram captured using incoherent digital holography technology, without changing the optical system. The incoherent digital hologram signal processing device and imaging device of the present invention are significantly different from the prior art in that they focus on a method for generating an extended depth-of-field image. [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic diagram showing a configuration of an imaging device according to an embodiment of the present invention. [Figure 2A] 1 is a block diagram showing a basic configuration of a signal processing device according to an embodiment of the present invention. [Figure 2B] FIG. 2 is a conceptual diagram for explaining the input / output relationship of information and data held in a signal processing device according to an embodiment of the present invention. [Diagram 3] 2B is a conceptual diagram showing the flow ((a) to (i)) of signal processing performed in the signal processing device of FIG. 2A. [Figure 4] 13 is a graph showing the relationship between zs and zr in an optical system (fo=250 mm, fd1=400 mm, fd2=∞ mm, zl=10 mm) of an imaging device according to an embodiment ((a) when zh=200 mm, (b) when zh=600 mm, (c) when zh=800 mm). [Diagram 5]13 is a graph showing the relationship between zs and zr in an optical system (fo=300 mm, fd1=400 mm, fd2=∞ mm, zl=300 mm) of an imaging device according to an embodiment ((a) when zh=100 mm, (b) when zh=300 mm, (c) when zh=600 mm). [Figure 6] 1A and 1B are diagrams showing a reconstructed image of a subject reconstructed by an imaging device according to an embodiment ((a) when photographed with the focus on a subject behind, (b) when photographed with the focus on a subject in front). [Figure 7] 11 is a graph showing the relationship between zs and zr in an imaging device according to an embodiment. [Figure 8] 1A and 1B are diagrams illustrating the results of application of an embodiment ((a) extended depth of field image (b) depth map). [Figure 9] Using different optical systems, two subjects with different depth distances were photographed, and the results of obtaining an extended depth-of-field image and a depth map using the embodiment method are shown ((a) Image of a shogi piece, (b) Image of an animal doll). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an incoherent digital hologram signal processing device and an imaging device according to an embodiment of the present invention will be described with reference to the drawings. First, FIG. 1 shows a conceptual diagram of an imaging device 1 equipped with an incoherent digital hologram signal processing device 26 according to this embodiment. The imaging device 1 according to this embodiment is an optical system based on a two-beam Michelson interferometer, and is composed of an optical system (21, 22, 23A, 23B, 25) for capturing a self-interference hologram, an imaging element 24, and a signal processing device 26 for reproducing the captured hologram. That is, in order to capture a self-interference hologram image of the subject 10, the optical system is composed of a lens (imaging lens) 21, a beam splitter (light combining and splitting element) 22 for splitting the light incident through this lens 21 into two lights and combining the two lights to obtain interference light, a concave mirror 23A and a plane mirror 23B (the concave mirror 23A and the plane mirror 23B are collectively referred to as a spherical phase imparting optical element) for reflecting depth distance information of the subject 10 on the hologram, and further, an imaging element 24 for capturing the hologram is provided.

[0019] 1, the light propagating from the subject 10 is collected by the lens 21 and then split into two beams by the beam splitter 22 to obtain the first and second split beams required for self-interference. Note that the use of the lens 21 is optional and is used to adjust the magnification and improve the light utilization efficiency. In order to reflect the three-dimensional information of the subject 10 in a self-interference hologram, a spherical phase imparting element, which is an optical element that has the role of imparting a spherical phase, is used to impart spherical phases with different radii of curvature to each of the first and second divided beams, and typically, a concave mirror 23A or a plane mirror 23B is used as shown in Fig. 1. Note that the plane mirror 23B is equivalent to imparting a planar phase distribution simply by reflecting light, but it can be regarded as an element that imparts a spherical phase with an infinite radius of curvature, and is treated as one of the spherical phase imparting elements. Thereafter, the two beams of light modulated by the spherical phase imparting element are combined by the beam splitter 22, whereby a self-interference hologram formed on the element surface of the imaging element 24 can be captured.

[0020] Generally, the narrower the wavelength width of the light source, the higher the resolution with which three-dimensional information of the subject 10 can be obtained, so a wavelength filter (bandpass filter) 25 may be installed in the optical system. Also, by arbitrarily changing the central wavelength of the wavelength filter, a hologram of any color component can be obtained. The hologram captured by the image sensor 24 is transferred to a signal processing device 26. The image sensor 24 may be a monochrome one not equipped with a wavelength filter, one equipped with a Bayer array color filter, or even one equipped with a polarizer array filter. Further, with regard to lens 21, although only one is used in FIG. 1, any number of lenses of various types can be disposed in any optical path.

[0021] The optical system used in this embodiment is not limited to the Michelson interferometer type shown in FIG. 1, but may also be a common-path interferometer using polarized light, a Sagnac interferometer, or other equal-path-length interferometers such as a Mach-Zehnder interferometer or a bypass Fizeau type. Furthermore, in order to obtain highly accurate three-dimensional information of the subject, it is better to remove the zero-order light component and the conjugate image component of the captured hologram. To remove these, the phase shift method or the off-axis method can be applied.

[0022] To apply the above-mentioned phase shift method, in the case of the optical system of Figure 1, one of concave mirror 23A and plane mirror 23B is mounted on a piezoelectric element or a moving stage, and this is scanned a distance equivalent to a predetermined phase (e.g., λ / 4) of the wavelength of the light source, and a phase shift amount is imparted between the two split light beams, and a hologram is captured each time. Moreover, when the off-axis method is applied, one of concave mirror 23A and plane mirror 23B is slightly tilted with respect to the other, and a hologram is captured.

[0023] [Signal processing device] Next, the signal processing device 26 constituting the imaging device 1 will be described in detail. The signal processing device 26 according to this embodiment is particularly intended to perform the following two processes. First, <1> , <2> We will briefly explain the following.

[0024] <1> A process to exclude reproduction distances that fall outside the range of variation from the search range for the reproduction distance that maximizes sharpness 1. The propagation calculation is applied successively to the captured digital hologram while changing the reconstruction distance, and a group of reconstructed images with different reconstruction distances is obtained. 2. Focus on a specific pixel position in each reconstructed image in the group of reconstructed images, evaluate the sharpness within a local area centered on that pixel position, and extract the reconstruction distance at which the reconstructed image with the highest sharpness value is obtained, and the brightness value at the focused pixel position of that reconstructed image. 3. By performing the above process on all pixel positions within the plane of the reconstructed image, the following a and b are obtained. a. Obtain the two-dimensional distribution of playback distance as a depth map of playback distance. b. A two-dimensional distribution of brightness values ​​is obtained as an extended depth-of-field image. 4. The range of the reproduction distance where the subject 10 may exist is determined based on the focal length of the optical elements (lens 21 and spherical phase imparting elements (23A, 23B)) constituting the imaging device 1 and the arrangement interval of each of these optical elements, and at least the reproduction distance outside of that range is excluded in advance from the search range of the reproduction distance where the sharpness is highest. This makes it possible to reduce unnecessary calculations in the propagation calculation and sharpness evaluation compared to the conventional technology.

[0025] <2> A process for setting the step size of the propagation distance at uneven intervals based on the depth resolution A process is performed to set the interval width of the propagation distance to unequal intervals based on the focal length and aperture diameter of the lens 21 and spherical phase adding element (23A, 23B) constituting the optical system of the imaging device 1, the central wavelength and wavelength width of the wavelength filter, the parameters of the pixel pitch of the imaging element, and the depth resolution determined by the arrangement positions of each optical element. The unequal intervals here refer to intervals that are changed so that the interval width of the propagation distance becomes longer in areas with lower depth resolution. This makes it possible to reduce unnecessary searches and obtain an extended depth of field image and depth map with high accuracy. A necessary condition for each of the above-mentioned processes to be performed is that the signal processing device 26 stores all of the elements of the optical system (the focal lengths of the optical elements (lens 21 and spherical phase imparting elements (23A, 23B)) and the spacing between each of these elements).

[0026] Next, the basic configuration of the signal processing device 26 will be described with reference to FIG. 2A. First, the signal processing device 26 of this embodiment is equipped with: a reconstructed image group acquisition means 31 that receives incoherent digital hologram information obtained by photographing the subject 10 using self-interference of light, and sequentially applies propagation calculations to the input incoherent digital hologram information while changing the reconstruction distance to obtain a reconstructed image group consisting of a plurality of reconstructed images with different reconstruction distances; a reconstruction distance / luminance value extraction means 32 that focuses on any corresponding pixel positions in each of the reconstructed images of the reconstructed image group obtained by the reconstructed image group acquisition means 31, evaluates the sharpness within a specified local area centered on that pixel position, and extracts the reconstruction distance of the reconstructed image from which the local area with the highest sharpness value was obtained among the reconstructed images of the reconstructed image group and the luminance value at the pixel position related to the local area with the highest sharpness value; and a depth of field extended image / depth map acquisition means 33 that acquires a depth of field extended image and a depth map of the reconstruction distance based on the reconstruction distance and luminance value extracted by the reconstruction distance / luminance value extraction means 32 for each of a plurality of pixel positions included in the image area of ​​the reconstructed image.

[0027] Furthermore, the signal processing device 26 of this embodiment includes a subject-absent domain determining means 34 that determines a subject-absent domain of a reproduction distance in which the subject 10 cannot be present, based on the focal length of each of the optical elements, including the lens 21 and the spherical phase imparting elements (23A, 23B) that constitute the optical system for capturing incoherent digital holography, and the arrangement interval of each of these optical elements, and is configured to preclude the search process for a reconstructed image of the subject-absent domain determined by the subject-absent domain determining means 34 from the reconstructed image search process in the reproduction distance / brightness value extraction means 32 that searches for a reconstructed image relating to a local region having the highest sharpness value.

[0028] Furthermore, the signal processing device 26 of this embodiment is equipped with a depth resolution calculation means 35 that calculates the depth resolution based on the focal length and aperture diameter of each of the optical elements including the lens 21 and the spherical phase imparting element (23A, 23B) that constitute the optical system for capturing incoherent digital holography, the central wavelength and wavelength width of the wavelength filter 25, the pixel pitch parameters of the image sensor 24, and each of the information on the arrangement positions of each of these optical elements, and is equipped with a propagation distance increment setting means 36 that sets the increment of the propagation distance to each reconstructed image when obtaining multiple reconstructed images with different reconstruction distances in the reconstructed image group acquisition means 31 so that the increment is longer in the area where the depth resolution value calculated by the depth resolution calculation means 35 is lower.

[0029] 2B, a description will be given of the input / output relationship of information and data held in the signal processing device 26. As shown in FIG. 2B, the signal processing device 26 includes a pre-calculation processing means 26A and a calculation processing means 26B during shooting. First, the parameters of the optical elements and imaging elements when the imaging device is constructed, the arrangement data of these elements, and information on the central wavelength of the hologram are transferred to and stored in the preliminary calculation processing means 26A. As a result, the preliminary calculation processing means 26A calculates (261) the search range of the depth map (see above) based on at least the focal lengths of all the lenses 21 that constitute the optical system, the focal lengths of the spherical phase imparting elements (23A, 23B), the pixel pitch of the imaging element 24, and information on the arrangement distances of these optical elements (21, 23A, 23B). <1> (corresponding to processing of In this way, by referring to this information, it is possible to know the range of reproduction distances corresponding to the depth distances at which the subject 10 may exist, and unnecessary distance searches can be eliminated in advance when searching for distances to generate a depth map / extended depth-of-field image. This reduces the calculation load and the possibility of erroneous results, enabling higher accuracy.

[0030] Furthermore, in the advance calculation processing means 26A, in addition to the parameters of the optical system described above, information on the effective aperture diameters of all the lenses 21, spherical phase imparting elements (23A, 23B), and image pickup element 24 that constitute the optical system, the central wavelength of the hologram when the hologram was captured, and further, if a wavelength filter 25 is provided, the central wavelength and wavelength width are used to calculate the step width when searching the depth map (261) (see above). <2> (corresponding to processing of Furthermore, if there are elements in the optical system that limit the diameter of light other than the above-mentioned optical elements (21, 23A, 23B) and image sensor 24, information on their effective aperture diameters is also stored in the signal processing unit 26 and is taken into account in the calculation results of the above-mentioned calculation process in the preliminary calculation processing means 26A.

[0031] As will be described later, by referring to this information, the depth resolution of the incoherent digital holography optical system can be determined. Corresponding to this depth resolution, the increment width of the distance search when generating the depth map / extended depth-of-field image can be set to unequal intervals so that the increment width is longer in areas with lower depth resolution values, thereby enabling depth distance information of the subject to be obtained with high accuracy and high efficiency.

[0032] During hologram shooting, the shooting data is transferred to the calculation processing means (26B) during shooting, which extracts complex amplitude distribution information from the hologram data using the phase shift method or the off-axis method (262). When the phase shift method is applied, complex amplitude distribution information can be obtained from multiple holograms by applying the phase shift method algorithm (see Non-Patent Document 1 mentioned above). When the off-axis method is applied, the hologram is Fourier transformed, and only the signal component is extracted using a bandpass filter, and then inverse Fourier transformed to obtain complex amplitude distribution. Between the phase shift method and the off-axis method, the former has better resolution.

[0033] In the calculation processing means 26B during shooting, the depth of field extended image and the depth map can be generated by sequentially performing the following processes based on the extracted complex amplitude distribution (263). At this time, the above search range, step width, and step number may be input by the user from the input means 264 as necessary (264⇒263). Note that the calculation result by the calculation processing means 26B during shooting is output from the signal processing device 26 to the outside as necessary.

[0034] The diffraction propagation calculation method is applied to the extracted complex amplitude distribution u(x,y) and the reproduction distance z r Reconstructed image at U n Get (x,y). When the angular spectrum method is used as a diffraction propagation calculation method, the reconstructed image U n (x, y) is obtained by the following equation (1).

number

[0035] In the optical system of incoherent digital holography, the reconstruction distance z ris the actual depth distance (recording distance) z from the lens 21 (imaging device) to the subject 10 s and has a nonlinear relationship, which is expressed by the following equation (2).

number

[0036] In conventional coherent digital holography, which uses coherent light as illumination, z r =z s On the other hand, the above expressions (2) and (3) of this embodiment, which are expressed nonlinearly, are relationships specific to incoherent digital holography. For example, o = 250 mm, f d1 = 400 mm, f d2 = ∞ mm, z l = 10 mm, z h When z is set to 200 mm, 600 mm, and 800 mm, s and z r The relationships are shown in Figures 4(a) to 4(c), respectively.

number

[0037] As another example, o = 300 mm, f d1 = 400 mm, f d2 = ∞, z l = 300 mm, z h When z is changed to 100 mm, 300 mm, and 600 mm, s and z r The relationships between these are shown in Figures 5(a) to 5(c). As shown in the figures, the graphs showing these relationships have a parabolic shape. As is clear from Figs. 4 and 5, the function z r =f(z s ) is practically a quasi-concave function, z r The domain of is limited to a finite range. Also, since it is a quasi-convex function, sThe value of z r In order to eliminate this ambiguity and use the imaging device 1, r In addition, the distance between the image capture device 1 and the subject 10 is not extremely close. s If we add the information from z r It is possible to limit the range of variation of In this embodiment, this z r By referring to the domain, it is possible to efficiently obtain an extended depth of field image and a depth map of the subject 10.

[0038] Next, referring to FIG. 3, a specific signal processing flow for generating an extended depth-of-field image and a depth map in the present invention will be shown. First, when the phase shift method is used in the imaging device 1, multiple digital holograms are captured, and therefore, by referring to the information on the number of phase steps stored in the memory means in the signal processing device 26, a calculation is performed using the phase shift method to extract information on the complex amplitude distribution (amplitude distribution and phase distribution) (see FIG. 3(a)). In the imaging device 1, when a hologram is captured based on the off-axis method instead of the phase shift method, information on the complex amplitude distribution is extracted by applying band-pass filtering of spatial frequencies.

[0039] The amplitude distribution and phase distribution obtained from the captured hologram are expressed as z r Within the domain of z r is changed N times at arbitrary intervals, and N reconstructed images U1(x,y) to U N (x, y) is obtained (Fig. 3(b)). Each reconstructed image (reconstructed image group) U n The sharpness is evaluated for each arbitrary local area A(i,j) within the (x,y) plane. The shape of the arbitrary local region is not limited to a specific shape and may be a rectangle, a circle, an ellipse, etc. The larger the size of the local region, the more robust the depth information can be extracted against noise, and the smaller the size of the local region, the higher the in-plane resolution of the depth information can be obtained.

[0040] In evaluating the sharpness, it is effective to use, for example, an index for evaluating the variation in luminance within a plane (the following formula (5)).

number

[0041] Alternatively, an index for evaluating the spectral components of a reconstructed image in the spatial frequency plane (the following formula (6)) may be used.

number

[0042] The indices of sharpness applicable to the present invention are not limited to those mentioned above, and various indices can be applied. Representative indices for evaluating sharpness are described in the following non-patent literature. "Elsa SR Fonseca, Paulo T. Fiadeiro, Manuela Pereira, and Antonio Pinheiro, "Comparative analysis of autofocus functions in digital in-line phase-shifting holography", Appl. Opt., (2016), vol. 55, pp. 7663-7674" (hereinafter, this non-patent document will be referred to as Non-Patent Document 3).

[0043] Using one of the evaluation indices, the brightness value of the reconstructed image of the local area with the highest sharpness is identified for each pixel in the plane, and a two-dimensional image is constructed from them, resulting in an extended depth-of-field image (Figure 3(d)). rBy identifying the positions of the pixels in the plane and constructing a two-dimensional image from them, a depth map of the reconstruction distance can be obtained (Figure 3(c)).

[0044] The playback distance z is calculated by the inverse function of the above equations (2) and (3). r Recording distance z s In this case, z r Depending on the value of z s There are two ambiguities in z, so there is no physically plausible z s However, since the depth position map obtained by these processes provides distance information even for areas where no subject exists (unnecessary areas), these unnecessary areas are deleted by referring to the information of the depth of field extended image. This allows the subsequent calculation processes to be performed quickly and efficiently.

[0045] After that, a two-level mask, 0 or 1, is generated from the depth-of-field extended image according to the object shape. That is, a threshold process is applied to the depth-of-field extended image to perform binarization (Fig. 3(e)). Next, if necessary, fine noise smaller than the area of ​​a given pixel is removed (Fig. 3(f)), and morphological transformations of contraction and expansion are applied (Fig. 3(g) and (h)) to obtain a two-tone mask of the object shape. When the subject 10 is captured over the entire field of view, a mask is used in which the entire image is designated as 1. By multiplying the mask generated in this way with the depth position map (see Figure 3(c)), a depth map is obtained (Figure 3(i)).

[0046] Among the above processes, the process with the highest computational load is "z r is changed N times at arbitrary intervals, and N reconstructed images U1(x,y) to U N This is a calculation process to obtain "(x, y)". In this process, all N reconstructed images may be acquired in advance, but in order to save memory usage, rThe depth of field extended image or depth map may be generated by changing the angle once, obtaining a reconstructed image, evaluating the sharpness of the reconstructed image, and using this series of steps to generate a depth of field extended image or depth map using the golden section method. By using the golden section method, the position where the sharpness is at its peak can be quickly found.

[0047] In any of the above methods, in order to obtain a depth-of-field extended image or a depth map with high accuracy, r It is better to set the step size of z to be smaller, while z r In this embodiment, in order to achieve both accuracy and speed, the depth resolution of the imaging device 1 for incoherent digital holography is referred to, and z r The step size is set as follows.

[0048] Depth resolution Δz sX is the central wavelength λ of the hologram to be formed, the effective aperture diameter D of the lens 21, in addition to the parameters of the optical system described above. o , the effective aperture diameter D of the concave mirror 23A d1 , the effective aperture diameter D of the plane mirror 23B d2 , the effective aperture diameter D of the image sensor 24 sensor , the pixel pitch p of the image sensor 24 N By referring to the numerical information, it can be obtained using the following formula (7).

number

[0049]

number

[0050]

number

[0051] In the above equation (9), C is a coefficient for selecting the half-width of the sinc function shape distribution that results in a depth resolution that is set to 1 or less, and is set to 0.5, for example. Referring to the above-mentioned depth resolution, specifically, the value of the above equation (9) is z r By setting the step size as the step size and evaluating the sharpness, it is possible to prevent unnecessary searches and generate a depth-of-field extended image and a depth map with high accuracy and high speed.

[0052] When adding an optical element such as a lens system, it is necessary to change the above-mentioned formulas for the depth resolution accordingly. In addition, since the information on the arrangement distance of the optical element and the information on the focal length of the optical element, which are referred to when calculating the search range and the depth resolution, may contain errors, it is preferable to set the search range and the depth resolution with some margin of performance.

[0053] The depth of field extended image and depth map information generated by the configuration and procedures described above can be output to a display device 27 (see FIG. 1) or a printer 28 (see FIG. 1) so that these images can be visually confirmed. In addition, while the above configuration and procedures focus on generating and outputting an extended depth-of-field image and a depth map, by switching the processing mode, it is also possible to calculate and output a conventional reconstructed image in which the depth of field at the depth position is not extended. EXAMPLES

[0054] Below, we conducted verification of an example in which the numerical values ​​in the imaging device 1 shown in Figure 1 were set as follows, and images were taken of two subjects (a wristwatch placed farther from the imaging device 1 and a dice placed closer) that are positioned on different planes in the depth direction. f o = 750 mm, f d1 = 1000 mm, f d2 = ∞ mm, z l = 100 mm, z h = 200 mm, λ = 633 nm, p N = 6.5 μm, D o = 10 mm, D d1 =D d2 = 12 mm, D sensor=13 mm where f o is the focal length of the lens 21 in FIG. 1, and f d1 , f d2 is the focal length of the concave mirror 23A and the plane mirror 23B, and z h is the distance from the concave mirror 23A or the plane mirror 23B to the image sensor 24. l is the distance from the concave mirror 23A or the plane mirror 23B to the lens 21. λ is the central wavelength of the hologram to be formed, D o is the effective aperture of the lens, D d1 is the effective aperture diameter of the concave mirror 23A, D d2 is the effective aperture diameter of the plane mirror 23B, D sensor is the effective aperture diameter of the image sensor 24, p N is the pixel pitch of the image sensor 24. Figure 6(a) is a reconstructed image focused on the subject in the background (a wristwatch), and Figure 6(b) is a reconstructed image focused on the subject in the foreground (a dice). It is clear that the dice are blurred in Figure 6(a) and the wristwatch is blurred in Figure 6(b).

[0055] In the case of the conventional technique to which this embodiment is not applied, the reproduction distance z r Since the range is unknown, when acquiring an extended depth-of-field image or a depth map, it is necessary to evaluate the sharpness over a wide range of depth distances, resulting in a lot of unnecessary calculation processing. On the other hand, when this embodiment is applied, z r Since the search range can be limited to a necessary area, the amount of wasted calculation processing is reduced. FIG. 7 shows the reproduction distance z r and the actual depth distance (recording distance) z from the lens 21 to the subject 10 s The relationship is shown below.

[0056] In this optical system, z r The search range is from -∞ to -340.85 mm (the upper limit value of -340.85 mm is obtained by the above formula (4)). sis not present in the vicinity of the image capture device 1. For example, s If we assume that subject 10 does not exist at |≦400 mm, z r The search range can be narrowed down to -1219.9 to -340.85 mm. By referring to such advance information, efficient subject search becomes possible.

[0057] In this embodiment, the reproduction distance z r The step size is determined with reference to the depth resolution of the optical system. For example, 400≦z s Playback distance z in the range of ≦2200 mm r In this embodiment, each z is set to one of N=39 values ​​shown in Table 1. r The intervals are set at irregular intervals between 26 mm and 14 mm as shown in the second column of Table 1. r The step size does not decrease monotonically, and there are some localized increases and decreases in the values. This is due to a numerical rounding error when calculating the depth resolution, and since the error is relatively small compared to the size of the step size, it does not pose a problem in practice.

[0058] [Table 1]

[0059] Playback distance z r In the case of conventional technology in which the increment size is set at equal intervals, for example, if the increment size is set uniformly to 26 mm, N becomes small and the calculation load becomes light, but the accuracy of the depth map decreases. Conversely, if the increment size is set uniformly to 14 mm, the accuracy of the depth map improves, but N becomes large and the calculation process takes a long time. In this way, according to the technology of this embodiment, the accuracy of the depth map is improved, and the reconstruction distance z can be calculated in the minimum necessary calculation time. r A search can be carried out.

[0060] FIG. 8(a) shows an extended depth-of-field image acquired by applying this embodiment, and FIG. 8(b) shows a depth map. In the depth-of-field extended image shown in FIG. 8(a), the images of two subjects (a wristwatch and a dice) arranged on different planes in the depth direction are both in focus. Moreover, in the depth map shown in FIG. 8(b), it is clear that the wristwatch and the dice are located at different depth positions (the one with a brighter luminance value indicates that it is located closer to the imaging device 1).

[0061] FIG. 9 shows the results of an extended depth-of-field image and a depth map obtained when photographing a subject different from that in the above embodiment ((a) is a set of shogi pieces (the pawn pieces are positioned close to the depth direction and the knight pieces are positioned farther in the depth direction), and (b) is an animal doll (the leopard doll is positioned close to the depth direction and the zebra doll is positioned farther in the depth direction)) using an embodiment that uses an optical system different from that in the above embodiment. In the depth-of-field extended images shown in Figures 9(a) and (b), the images of two subjects (a pawn piece and a knight piece, and a leopard puppet and a zebra puppet) arranged on different planes in the depth direction are both in focus. In addition, in the depth maps shown in Figs. 9(a) and (b), it is clear that the two subjects are located at different depth positions. r The value of z is visualized in a grayscale image. r Regardless of the actual positional relationship of the subjects, the front-back relationship may differ depending on the shooting device, and in the embodiment of the animal dolls, the relationship between the actual placement position and the playback distance is reversed.

[0062] The incoherent digital hologram signal processing device and the imaging device of the present invention are not limited to the above-described embodiments, and various other modifications are possible. For example, in the optical system of the imaging device according to the above-described embodiment, each member can be appropriately replaced with another member having a similar function. It is not always necessary to measure the sharpness of a local region related to all pixels of the image sensor, and it is also possible to thin out a predetermined number of pixels from among all pixels of the image sensor and measure the sharpness of a local region related to a number of selected pixels. However, it is preferable that the selected number of pixels are uniformly selected from among all pixels of the image sensor. [Explanation of symbols]

[0063] 1. Imaging device 10. Subject 21 Lens 22 Beam splitter (light combining and splitting element) 23A concave mirror 23B plane mirror 24 Image sensor 25 Wavelength Filter 26 Signal Processing Device 26A Pre-computational processing means 26B Calculation processing means during photography 27 Display device 28 Printers 31 Reconstructed image group acquisition means 32. Reconstruction distance / brightness value extraction method 33. Means for acquiring extended depth-of-field images / depth maps 34 Object-absent domain determination means 35 Depth resolution calculation means 36 Propagation distance step size setting means

Claims

1. Incoherent digital hologram information obtained by photographing an object using self-interference of light is input, and the input incoherent digital hologram information is a reconstructed image group acquiring means for successively applying a propagation calculation while changing a reconstruction distance to acquire a reconstructed image group consisting of a plurality of reconstructed images having different reconstruction distances; a reconstruction distance / brightness value extraction means for focusing on any corresponding pixel position in each of the reconstructed images of the obtained reconstructed image group, evaluating the sharpness within a predetermined local area centered on the pixel position, and extracting the reconstruction distance of the predetermined reconstructed image from which the local area having the highest sharpness value was obtained among the reconstructed images of the reconstructed image group, and a brightness value at the pixel position related to the local area having the highest sharpness value; a depth of field extended image / depth map acquisition means for acquiring a depth of field extended image and a depth map of the reproduction distance based on the reproduction distance and the luminance value extracted by the reproduction distance / luminance value extraction means for each of a plurality of pixel positions included in an image area of ​​the reproduction image; and an object-absent region determining means for determining an object-absent region of the reproduction distance in which an object cannot exist, based on the focal length of each of optical elements including a lens and a spherical phase imparting element constituting an optical system for capturing an incoherent digital hologram, and on the arrangement interval of each of the optical elements, An incoherent digital hologram signal processing device characterized in that the reproduction distance / brightness value extraction means is configured to pre-exclude a search process for the reproduction image of the object-absent region determined by the object-absent region determination means from a search process for a predetermined reproduction image that searches for the reproduction image relating to a local region where the sharpness value is the highest.

2. 2. The incoherent digital hologram signal processing apparatus according to claim 1, wherein the plurality of pixels included in the image area of ​​the reconstructed image are all pixels that constitute the reconstructed image.

3. a depth resolution calculation means for calculating a depth resolution based on information on the focal length and aperture diameter of each of the optical elements including the lens and the spherical phase imparting element constituting the optical system for capturing the incoherent digital hologram, the central wavelength and wavelength width of a wavelength filter, the pixel pitch parameters of an image pickup element, and the arrangement position of each of the optical elements; The incoherent digital hologram signal processing device as described in claim 1, characterized in that the reconstructed image group acquisition means is provided with a propagation distance increment setting means for setting the increment of the propagation distance to each reconstructed image when obtaining multiple reconstructed images with different reconstruction distances so that the increment is longer in an area where the depth resolution value calculated by the depth resolution calculation means is lower.

4. 4. The incoherent digital hologram signal processing apparatus according to claim 3, wherein the plurality of pixels contained in the image area of ​​the reconstructed image are all pixels that constitute the reconstructed image.

5. 2. An imaging device comprising: the incoherent digital hologram signal processing device according to claim 1; and an optical system and an imaging element for capturing the incoherent digital hologram.

6. The imaging device described in claim 5, characterized in that the incoherent digital hologram signal processing device is provided with a propagation distance increment setting means that sets the propagation distance increment interval so that the interval is longer in an area with a lower depth resolution value, in accordance with the depth resolution value determined based on the focal length and aperture diameter of each optical element including the lens and spherical phase adding element that constitute the optical system, the center wavelength and wavelength width of the wavelength filter, the pixel pitch parameters of the imaging element, and the arrangement position of each of the optical elements.

7. An imaging device comprising: an incoherent digital hologram signal processing device according to any one of claims 1 to 4; an optical system and an imaging element that capture three-dimensional image information of a subject input to the incoherent digital hologram signal processing device by utilizing self-interference of light; and a hologram display device that receives the three-dimensional image information of the subject that has been subjected to signal information processing by the incoherent digital hologram signal processing device and displays a three-dimensional reproduced image of the subject.

Images