Incoherent digital hologram signal processing device and imaging display system

Abstract

To provide an incoherent digital hologram signal processing device and an imaging display system capable of converting the depth distance information on a subject image acquired by an imaging part based on the incoherent digital holography into the information according to the actual depth distance of the subject and being sent to a display part.SOLUTION: A signal processing device 30 extracts the shape information (silhouette) and depth distance information on a subject 10 from the three-dimensional image information on the digital hologram photographed by an imaging device 20 based on the incoherent digital holography. The configuration is such that this three-dimensional image information is subjected to the correction processing so as to match the information corresponding to the actual depth distance of the subject 10, and then sent to a display device 40.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an incoherent digital hologram signal processing device for performing signal processing of digital holography, and more particularly, incoherent digital holography, and an imaging and display system equipped therewith. [Background technology]

[0002] In digital holography technology, a hologram reflecting the three-dimensional information of a subject is captured by optoelectronic devices such as image sensors and photodetectors using light interference. This hologram is then transferred to a signal processing unit and stored. The signal processing unit performs noise reduction calculations as needed. The data of the digital hologram after this processing is transferred via the signal processing unit to a display device composed of a spatial light modulator, and displayed, allowing viewers to observe the three-dimensional image of the captured subject without wearing special glasses (Non-Patent Literature 1). As described above, in order to capture and display a three-dimensional image of a subject using digital holography technology, it is generally necessary to organically connect three devices: an imaging device, a signal processing unit, and a display device.

[0003] In particular, digital holography imaging devices use a laser light source capable of emitting highly coherent laser light to illuminate the subject. The laser light source must meet eye-safe standards to avoid damaging the human eye, but this requires limiting the intensity of the laser light. This intensity limit makes it difficult to obtain sufficient reflected light intensity when photographing distant subjects, thus hindering hologram formation. Furthermore, when considering outdoor use, the subject is illuminated not only by the laser light emitted from the imaging device's laser light source, but also by incoherent light such as sunlight, which acts as a disturbance.

[0004] Incoherent digital holography is attracting attention as a technology that solves the above-mentioned problems with laser light (Non-Patent Literature 2). This technology applies the phenomenon of self-interference of light to form holograms using a light source with low coherence, i.e., incoherent light, and does not require laser light. It can use light that exists in nature, such as sunlight, incandescent lamps, and LEDs, as the light source. Therefore, the eye-safety problem mentioned above does not exist, and incoherent light such as sunlight, which was a disturbance in digital holography using laser light, can be used as the illumination light for the subject.

[0005] Furthermore, as shown in Patent Documents 1 to 3, digital holography imaging devices do not need to include a special light source capable of outputting highly coherent light, and can capture holograms that reflect the three-dimensional information of the subject (however, as shown in Patent Document 1, in the case of a microscope, it is necessary to incorporate a light source for illuminating the sample into the imaging device). [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Special Publication 2016-533542 [Patent Document 2] Japanese Patent Publication No. 2019-144520 [Patent Document 3] Japanese Patent Publication No. 2021-131457 [Patent Document 4] Japanese Patent Publication No. 2020-190616 [Non-patent literature]

[0007] [Non-Patent Document 1] Georges Nehmetallah and Partha P. Banerjee, "Applications of digital and analog holography in three-dimensional imaging", Advances in Optics and Photonics, (2012), vol.4, pp.472-553. [Non-Patent Document 2] Joseph Rosen and Gary Brooker, "Digital spatially incoherent Fresnel holography", Opt. Lett., (2007), vol.32, pp.912-914. [Non-Patent Document 3] Teruyoshi Nobukawa, Yutaro Katano, Tetsuhiko Muroi, Nobuhiro Kinoshita, and Norihiko Ishii, "Sampling requirements and adaptive spatial averaging for incoherent digital holography", Opt. Express, (2019), vol.27, pp.33634-33651. [Non-Patent Document 4] Victor Arrizon, Guadalupe Mendez and David Sanchez-de-La-Llave, "Accurate encoding of arbitrary complex fields with amplitude-only liquid crystal spatial light modulators", Opt. Express, (2005), vol.13, pp.7913-7927. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] However, in incoherent digital holography, there is a non-linear relationship between the actual depth distance of the subject and the depth distance information of the 3D image of the subject contained in the digital hologram captured by the imaging device. This is due to the fact that incoherent digital holography utilizes the self-interference of light, and is a characteristic that did not exist in digital holography using laser light (in digital holography using laser light, the relationship between the actual depth distance of the subject and the depth distance information of the 3D image of the subject contained in the captured digital hologram is linear and theoretically coincides).

[0009] If a digital hologram captured by an imaging device based on incoherent digital holography is transferred and displayed to a signal processing device and imaging device in the same way as digital holography using laser light, a 3D image that is completely different from the actual depth perception of the subject may be displayed. For example, when multiple subjects are photographed at positions with different depth distances, the front-to-back relationship of these subjects may be reversed in the display. Several incoherent digital holography imaging devices have been proposed so far, but all of them acquire 3D image information of the subject and perform numerical reconstruction using propagation calculations on this 3D image information, and no specific method for displaying a 3D image using a digital hologram has been considered.

[0010] The object of the present invention is to provide an incoherent digital hologram signal processing device and an imaging display system that can convert depth distance information of a subject image acquired by an imaging unit based on incoherent digital holography into information corresponding to the actual depth distance of the subject and send it to a display unit. [Means for solving the problem]

[0011] To achieve the above objective, the incoherent digital hologram signal processing apparatus of the present invention is In an incoherent digital hologram signal processing device that receives incoherent digital hologram information obtained by photographing a subject using the self-interference of light, performs signal information processing on the received incoherent digital hologram information, and sends it to a hologram display unit, The signal information processing is characterized by extracting shape information and depth distance information of the subject from the input incoherent digital hologram information, performing correction processing so that the three-dimensional image information of the subject matches information corresponding to the actual depth distance of the subject, and then sending it to the hologram display unit.

[0012] Furthermore, it is preferable that the incoherent digital hologram signal processing device described above is configured to perform a process to correct the lateral magnification according to the configuration of the optical system and the imaging distance of the subject. Furthermore, it is preferable that the system is configured to create a computer-generated hologram based on the three-dimensional image information of the subject to which the correction processing has been applied.

[0013] Furthermore, the signal information processing preferably involves extracting a complex amplitude distribution from the input incoherent digital hologram information, applying autofocus processing to this complex amplitude distribution to identify the propagation distance at which sharpness is highest across the entire field of view, applying low-pass filtering to the complex amplitude distribution propagated at this propagation distance, performing binarization on the obtained brightness distribution, extracting the shape information of the subject by morphological transformation processing, performing mask processing using this shape information, performing the autofocus processing in the region where the subject exists to create a depth map of the reproduction distance, and transforming this depth map of the reproduction distance based on a relational expression between the reproduction distance and the imaging distance derived from predetermined parameter information of the imaging optical system for capturing the incoherent digital hologram information, thereby extracting the depth distance information of the subject, which is a depth map of the imaging distance. Furthermore, it is preferable that the system is configured to calculate the square root of the brightness distribution and use this square root value as an indicator of the new brightness distribution of the subject.

[0014] Further, based on the resolution of the spatial light modulator constituting the hologram display unit, the brightness distribution of the subject is divided into a plurality of layers stacked in the depth direction, and the propagation calculation of the complex amplitude distribution is performed for each of the divided layers, and the complex amplitude distributions obtained by the propagation calculation for each layer are added to create the computer-generated hologram.

[0015] Further, a parameter information storage unit is provided for storing the parameter information of the optical system related to the imaging unit that captures the subject using self-interference of light and the parameter information of the optical system related to the hologram display unit, and spatial averaging processing is performed according to the difference between the resolution in the imaging unit and the resolution in the hologram display unit, and it is preferable to include first noise removal processing means for removing the noise component added by the imaging unit.

[0016] Further, for the incoherent digital hologram information captured by the imaging unit or the complex amplitude distribution information extracted from the incoherent digital hologram information, data is expanded by any one of nearest neighbor or zero-padding data interpolation methods, and using propagation calculation or Fourier transform, the noise component is diffused into the high-frequency region, and after removing the noise existing in the high-frequency region, it is preferable to include second noise removal processing means for removing the noise component by performing an inverse propagation calculation or an inverse Fourier transform operation.

[0017] Furthermore, the imaging display system of the present invention includes any one of the above-described incoherent digital hologram signal processing devices, an imaging device that captures three-dimensional image information of a subject input to the incoherent digital hologram signal processing device using self-interference of light, and a hologram display device that inputs the three-dimensional image information of the subject that has been subjected to signal information processing by the incoherent digital hologram signal processing device and displays a three-dimensional reproduced image of the subject.

Effect of the Invention

[0018] According to the incoherent digital hologram signal processing device and imaging display system of the present invention, shape information (silhouette) and depth distance information of the subject are extracted from the input incoherent digital hologram information, which is the three-dimensional image information of the subject. This is corrected so that the three-dimensional image information of the subject matches the actual depth distance of the subject. As a result, three-dimensional image information that faithfully reproduces the image of the actual subject can be sent to the display unit. [Brief explanation of the drawing]

[0019] [Figure 1] This is a schematic diagram showing the configuration of an imaging and display system according to an embodiment of the present invention. [Figure 2] This is a schematic diagram showing the configuration of the optical system of the imaging device of the imaging and display system according to an embodiment of the present invention ((a) Michelson interferometer type optical system, (b) optical system of an equipath-length interferometer using a spatial light modulator). [Figure 3] This is a schematic diagram showing the configuration of the optical system of the display device of the imaging display system according to an embodiment of the present invention ((a) arrangement using a general reflective type spatial light modulator, (b) arrangement incorporating a 4f optical system, c) arrangement incorporating an angle selection filter). [Figure 4A] This is a conceptual diagram 1 showing the flow of CGH creation for the imaging and display system according to an embodiment of the present invention. [Figure 4B] This is a conceptual diagram 2 showing the flow of CGH creation for the imaging and display system according to an embodiment of the present invention. [Figure 5] This is a conceptual diagram of the autofocus process used to evaluate sharpness (F). [Figure 6A] This is a conceptual diagram of a noise reduction method using signal processing (a method that uses propagation calculations (zero-padding interpolation)). [Figure 6B] This is a conceptual diagram of a noise reduction method using signal processing (a method that uses a Fourier transform-based method (nearest neighbor interpolation)). [Figure 7]This figure shows images at each stage formed by the imaging and display system according to an embodiment of the present invention ((a) captured hologram group, (b) acquired complex amplitude distribution, (c) brightness distribution after applying autofocus and low-pass filtering, (d) silhouette of the subject acquired by applying binarization and morphological transformation, (e) depth map of the imaging distance of the subject). [Figure 8] This figure shows a phase-type CGH created from imaging data obtained by the imaging and display system according to an embodiment of the present invention. [Figure 9] This figure shows a three-dimensional reconstructed image of a subject reproduced by a spatial light modulator of a display device in an imaging and display system according to an embodiment of the present invention ((a) when the subject in the background is focused on during shooting, (b) when the subject in the foreground is focused on during shooting). [Modes for carrying out the invention]

[0020] Hereinafter, an incoherent digital hologram signal processing device and an imaging and display system according to embodiments of the present invention will be described with reference to the drawings. First, Figure 1 shows a conceptual diagram of the imaging and display system according to this embodiment. As shown in Figure 1, this imaging and display system 1 is organically connected to an imaging device 20 that captures an incoherent digital hologram of a subject 10, a signal processing device (incoherent digital hologram signal processing device) 30 that converts the depth distance information of the digital hologram of the subject acquired by the imaging device 20 into three-dimensional image information that faithfully reproduces the actual subject 10, and a display device 40 that displays the three-dimensional image information converted by the signal processing device 30 on a spatial light modulator so that the viewer can see the three-dimensional reconstructed image 50 of the subject 10.

[0021] These devices do not necessarily need to be connected by wires; the connection method is not limited as long as information and data can be transmitted and received between the imaging device 20 and the signal processing device 30, and between the signal processing device 30 and the display device 40. In addition, while there may be only one of each device 20, 30, and 40, the configuration may also include multiple units of at least one type of device. The following describes each of the above devices 20, 30, and 40 in turn. First, the imaging device 20 and the display device 40 will be briefly described, and then the signal processing device 30, which constitutes the main part of the imaging and display system 1, will be described in detail.

[0022] [Imaging device] The imaging device, which constitutes the imaging and display system 1 and is based on incoherent digital holography, consists of a lens 21A, a photosynthesis splitting element 22A for combining two beams of light to obtain interference light (and for splitting into two beams of light), a spherical phase-imparting optical element 23A for reflecting the depth distance information of the subject in the hologram, and an image sensor 24A for capturing the hologram, for example, as shown in the optical system of Figure 2(a) based on a two-beam Michelson interferometer, in order to capture a self-interfering hologram.

[0023] For example, a beam splitter can be used as the photosynthesis splitting element 22A. As the spherical phase-imparting optical element 23A, a concave mirror 23Aa and a plane mirror 23Ab, as well as lenses (not shown), can be used. Note that a plane mirror can be considered a type of concave mirror that imparts spherical phase with an infinite radius of curvature, and is included in the spherical phase-imparting optical element 23A. Furthermore, the photosynthesis splitting element 22A and the spherical phase-imparting optical element 23A may be realized with the same optical element. For example, as shown in Figure 2(b), a spatial light modulator can be used, or a birefringent lens or metalens (not shown) can be used, and a polarizing optical element can also be arranged in conjunction with it. This allows for the construction of an equipath-length interferometer to induce interference as an incoherent optical system. Furthermore, it is possible to introduce wavelength filters (bandpass filters) 25A for controlling temporal coherence or for controlling and acquiring color, as well as additional spherical phase-imparting optical elements for controlling magnification, light utilization efficiency, and aberrations, into the optical system described above.

[0024] Furthermore, phase shift or off-axis methods may be applied to remove the 0th-order optical component and conjugate image component contained in the hologram. When applying the phase shift method, additional phase shift elements are introduced. These phase shift elements include piezo actuators, polarizer arrays, spatial light modulators, and phase gratings (see Patent Documents 2 and 3).

[0025] Depending on the phase shift method employed, multiple imaging may be required. For example, when applying a time-division phase shift method, a piezo actuator attached to the concave mirror 23Aa or the plane mirror 23Ab is used to scan these concave mirrors 23Aa or the plane mirror 23Ab in the direction of the optical axis, thereby changing the interference conditions of the light by creating a difference in optical path length, i.e., a phase shift, between the two systems of light, and capturing multiple incoherent digital holograms with different positions of brightness and darkness in the interference fringes.

[0026] When using the off-axis method, one of the mirrors is tilted slightly to cause interference, and an incoherent digital hologram is captured. Furthermore, while we will omit details about the operation and characteristics of optical systems using other phase-shift elements, any method can be applied as long as it is possible to extract the complex amplitude distribution.

[0027] Here, parameter information for each element constituting the imaging device 20A, namely the pupil diameter of the spherical phase-imparting optical element 23A and the distance between each element, the distance between the rearmost optical element of the spherical phase-imparting optical element 23A and the image sensor 24A, the number of pixels and pixel pitch of the image sensor 24A, and the number of phase steps when applying the phase shift method, is stored in a predetermined storage unit (parameter information storage unit: not shown) within the signal processing device 30.

[0028] It should be noted that the configuration of the imaging device 20A shown in Figure 2(a) is not the only possible configuration, and various other modifications are possible. Similarly, various other lens arrangement configurations can be adopted. For example, instead of the spherical phase-imparting optical element 23A, which consists of a concave mirror 23Aa and a plane mirror 23Ab, a spherical phase-imparting optical element 23B consisting of a single spatial light modulator 23Ba can be used, as shown in Figure 2(b).

[0029] This spatial light modulator 23Ba is a reflective spatial light modulator that functions as a bifocal variable focus lens system. Thus, when using a reflective spatial light modulator 23Ba as the spherical phase-imparting optical element 23B, a photosynthesis splitting element 22B consisting of a polarizing beam splitter is placed between the lens 21B and the spatial light modulator 23Ba, a polarizer 26Ba is inserted between the lens 21B and the photosynthesis splitting element 22B, and a polarizer 26Bb is inserted between the photosynthesis splitting element 22B and the image sensor 24B to adjust the direction of light propagation. The wavelength filter 25B has the same function as the wavelength filter 25A shown in Figure 2(a).

[0030] The important point here is that, as mentioned above, at least the parameter information of the optical system constituting the imaging device 20 is stored in a predetermined storage unit of the signal processing device 30. Therefore, if the values ​​of the parameter information of the optical system of the imaging device 20 change, the parameter information stored in the predetermined storage unit of the signal processing device 30 will be updated sequentially.

[0031] [Display device] As described above, the display device 40, which constitutes the imaging display system 1, displays the three-dimensional image information converted by the signal processing device 30, allowing the viewer to see the three-dimensional reconstructed image 50 of the subject 10. For example, as shown in Figure 3(a), it includes a laser light source 41A and a spatial light modulator 45A that displays the three-dimensional image information of the subject 10. That is, the laser light output from the laser light source 41A enters the light reflection means (beam splitter) 44A via lenses 42A and 43A, is reflected, and irradiates the reflective spatial light modulator 45A.

[0032] The spatial light modulator 45A displays three-dimensional image information of the subject 10. The light irradiated onto this reflective spatial light modulator 45A carries this three-dimensional image information of the subject 10 and is reflected, so that viewers can see a three-dimensional reconstructed image of the subject 10 based on the three-dimensional image information of the subject 10 carried in this light.

[0033] The display device 40 is not limited to the configuration shown in Figure 3(a), and various other modifications are possible, as is the arrangement of the lenses. For example, as shown in Figure 3(b), the 4f optical system (consisting of lens 46Ba, filter 46Bb, and lens 46Bc) 46B, which removes unwanted diffracted light generated after optical modulation in the spatial light modulator 45B and the 0th-order light and conjugate image generated in the CGH (Computer-Generated Hologram: the same applies hereinafter), can be positioned on the viewer side of the light reflection means 44B (for components corresponding to those shown in Figure 3(a), the symbol A is replaced with B in the numbers). Alternatively, as shown in Figure 3(c), the angle selection filter 47C may be replaced with the 4f optical system 46B to provide a similar function (for components corresponding to those shown in Figure 3(a), the symbol A is replaced with C in the number).

[0034] Furthermore, for example, lenses may be provided to control the magnification, field of view, or depth position of the stereoscopic image displayed on the spatial light modulators 45A, B, and C. Furthermore, although reflective spatial light modulators 45A, B, and C are used in Figures 3(a), (b), and (c), transmissive spatial light modulators may be used instead. Furthermore, if color information is acquired by the imaging device 20, the display device 40 may construct an optical system using light sources and spatial light modulators corresponding to each color, each in parallel.

[0035] Furthermore, it is essential that parameter information such as the number of pixels, pixel pitch, and modulation physical quantities (amplitude, phase, or complex amplitude) of the spatial light modulator in the optical system constituting the display device 40 be stored in a predetermined storage unit of the signal processing device 30. Therefore, if the parameter information of the optical system of the display device 40 changes, the parameter information stored in the predetermined storage unit of the signal processing device 30 will be updated sequentially.

[0036] [Signal processing device] Next, we will explain in detail the signal processing device 30 that constitutes the imaging and display system 1. The signal processing device 30 according to this embodiment is designed to solve problems inherent to incoherent holography by performing two processes: a correction process to obtain three-dimensional image information that is consistent with the depth information of the subject 10, and a noise reduction process for the image sensors 24A and B. Therefore, the features of the signal processing device 30 will be explained below by describing these two processes in order.

[0037] A. Correction process to obtain 3D image information that is consistent with the depth information of the actual subject. To date, no technology or method has been considered for displaying digital holograms captured with an incoherent digital holography imaging device as three-dimensional images using a spatial light modulator. If a digital hologram captured with an incoherent digital holography imaging device were to be directly input into a spatial light modulator and displayed as a three-dimensional image, similar to conventional digital holograms using coherent light, the imaging distance, which is the actual depth distance of the subject, and the reproduction distance included in the digital hologram captured with the incoherent digital holography imaging device would have a non-linear relationship. As a result, the spatial light modulator would display a strange three-dimensional image with a different depth distance and sense of depth from the actual subject 10, making it impossible to obtain an image suitable for actual use. Therefore, in the signal processing device 30 according to this embodiment, in order to solve the problems inherent in the incoherent holography technology, correction processing to obtain three-dimensional image information that is consistent with the depth information of the subject 10 is performed by the following method and procedure.

[0038] First, the signal processing device 30 creates a CGH from the digital hologram obtained by imaging using the procedure shown in Figures 4A and 4B. In other words, when the imaging device 20 uses the phase shift method, multiple digital holograms are captured as shown in Figure 4A(a) (using the digital hologram group acquisition means). Therefore, the phase step number information stored in the storage means within the signal processing device 30 is referenced to perform calculations using the phase shift method and extract information on the complex amplitude distribution (amplitude distribution and phase distribution) (Figure 4A(b): using the complex amplitude distribution extraction means). In the imaging device 20, if the hologram is captured based on the off-axis method instead of the phase-shift method, information on the complex amplitude distribution is extracted by applying spatial frequency bandpass filtering.

[0039] <Generating a depth map> Next, a depth map of the playback distance is generated from the extracted complex amplitude distribution using the propagation calculation-based autofocus procedure shown below. To generate a depth map of the playback distance, first, the silhouette (2D shape) of the subject is obtained from the complex amplitude distribution extracted using the phase shift method or the off-axis method (Figure 4B(f)).

[0040] Specifically, to obtain the silhouette (2D shape) of the subject, the following processes (1) to (4) are performed. (1) Identify the playback distance with the highest sharpness (F) (using playback distance identification means). In other words, the playback distance that yields the highest sharpness is identified by repeating the operations (a) and (b) below. (a) Apply light wave propagation calculations to the complex amplitude distribution and obtain a reconstructed image by changing the reconstruction distance. (i) Evaluate the sharpness of the reconstructed image. As a result, as shown in Figure 5, for example, the propagation distance Z r By varying the frequency from Z1 to Z5 and measuring the sharpness (F) at each propagation distance Z1 to Z5, the propagation distance Z at which the sharpness of the reproduced image is highest can be determined. r Since we can identify (Z3 in the case of Figure 5), this propagation distance Z r This is identified as the playback position of subject 10. The playback image after applying this propagation calculation is shown in Figure 4A(c).

[0041] For calculating the propagation of light waves, angular spectral methods or Fresnel diffraction can be used. As indicators of sharpness, the coefficient of variation of brightness, the derivative, the Laplacian value, the Fourier spectrum of the brightness distribution, and the high-frequency component value of the cosine transform can be used. In this specification, the process described above will be referred to as autofocus. Furthermore, by changing the interval for varying the playback distance according to the depth resolution derived from the optical elements constituting the optical system of the imaging device and the distances between them, it is possible to identify a playback distance with high sharpness (F).

[0042] (2) Obtain a reconstructed image with an extended depth of field (by using depth of field extension means). Next, low-pass filtering is applied to the brightness distribution of the reconstructed image at the reconstructed distance where sharpness is highest. This process extends the depth of field of the complex amplitude distribution and equalizes the blurring of all images positioned at different depth locations. This procedure corresponds to the operation of extending the depth of field by stopping down the aperture in a normal camera, and in this technology, low-pass filtering is implemented by numerically calculating and applying it to the digital hologram's captured data. Figure 4A(d) shows the reconstructed image after extending the depth of field of the complex amplitude distribution. If necessary, non-local averaging may be applied to remove ringing noise that occurs when applying propagation calculations. The noise reduction techniques will be described later.

[0043] (3) The brightness distribution of the reconstructed image is subjected to binarization (using a binarized image acquisition means). Next, the brightness distribution of the reconstructed image shown in Figure 4A(d) is subjected to a binarization process of 0s and 1s based on a predetermined threshold. The threshold for the binarization process may be the median brightness of the reconstructed image, a predetermined percentage of the maximum brightness, or Otsu's binarization algorithm. The binarization method is not limited as long as the shape of the subject 10 can be extracted. The binarized image after such a binarization process is shown in Figure 4B(e). Furthermore, after the binarization process, noise other than the shape of the subject 10 may remain as a value of 1, so it is also possible to remove these using morphological transformation (using morphological transformation means).

[0044] (4) Obtain the silhouette (2D shape) of the subject (using a means for acquiring the subject silhouette). The signal processing described above allows us to obtain the silhouette (2D shape) of subject 10 (Figure 4B(f)). Next, the resulting silhouette (2D shape) of subject 10 is subjected to the following processing steps (5) to (7) to create a CGH.

[0045] (5) Obtain depth distance information for each pixel (using depth distance information acquisition means). (5-1) Obtaining a depth map of the playback distance Next, for each pixel in this silhouette that has a value of 1, the sharpness (F) of the reconstructed image at each reconstruction distance obtained in the process described in (a) above is evaluated in order to apply autofocus processing within an evaluation area of ​​any multiple pixels centered on that pixel. The reconstruction distance with the highest sharpness (F) is identified. The difference between this autofocus procedure and the autofocus procedure described above, which is performed on the complex amplitude distribution in Figure 4A(b), is that the former evaluates sharpness in a predetermined region of the field of view of the reproduced image, while the latter evaluates sharpness over the entire field of view of the reproduced image.

[0046] Furthermore, the in-plane resolution and robustness to noise of the depth map change depending on the size of the evaluation area. Setting a larger evaluation area reduces the influence of noise and allows for relatively high-precision identification of the playback distance, but it reduces the in-plane resolution of the depth map. On the other hand, setting a smaller evaluation area improves the in-plane resolution of the depth map, but the influence of noise increases, potentially reducing the accuracy of playback distance detection. It is preferable to set the size of the evaluation area according to the target in-plane resolution and accuracy of the depth map. The depth map of the playback distance obtained in this way is shown in Figure 4B(g).

[0047] (5-2) Obtaining a depth map of the imaging distance of the subject The regeneration distance identified by the autofocus described above is converted to the imaging distance based on the relationship between the regeneration distance derived from the optical elements constituting the optical system of the imaging device 20 and the distances between them, and the imaging distance (for example, equations (1) and (2) below). In other words, the depth map of the regeneration distance is converted to a depth map of the imaging distance. playback distance z r and imaging distance z sThe relationship is determined by the focal lengths of the spherical phase-imparting optical elements 23A and B that constitute the optical system and the distance between the elements, and in the case of the optical system shown in Figure 2(a) above, it is given by the following equation (1) (see Non-Patent Document 3 disclosed by the present inventors).

[0048]

number

[0049]

number

[0050] In equation (2) above, the ± sign is one of the + and - signs, z s This means selecting the sign that is not negative and is physically consistent. As mentioned above, the playback distance z r and imaging distance z s Although the relationship is nonlinear, by performing a transformation process based on the above equation, the nonlinearity can be corrected and accurate three-dimensional image information of the subject 10 can be obtained.

[0051] Here, equation (2) above is valid when the optical system is composed of the optical elements shown in Figure 2(a) or Figure 2(b). However, when the number of lenses increases, it is possible to derive a theoretical formula corresponding to the optical system configuration by analytically solving the light wave propagation calculation. Therefore, the application of this embodiment is not limited to cases where the optical system is composed of the optical elements shown in Figure 2(a) or Figure 2(b). Through the process described above, depth distance information of the subject 10 can be obtained for each pixel that has a value of 1 in the silhouette of the subject 10.

[0052] By applying this process to all pixels that have a value of 1 in the silhouette, the distribution of depth distance information of subject 10, which is a depth map of the imaging distance, can be obtained, and 3D image information can be obtained by combining the 2D shape of the brightness distribution of subject 10 and the depth map. The depth map of the imaging distance obtained in this way is shown in Figure 4B(h). In the above process, for each pixel representing 1 in the silhouette, the reproduction distance was identified, converted to the imaging distance, and a depth map of the imaging distance was obtained. However, after obtaining the depth map of the reproduction distance, it may be converted collectively based on the above equations (1) and (2) to obtain the depth map of the imaging distance.

[0053] Note that the silhouette (two-dimensional shape) of the subject 10 obtained by the above process changes in the horizontal magnification according to the parameter information of the optical system of the imaging device 20 and the imaging distance. Specifically, the horizontal magnification M in the case of the optical system shown in FIG. 2(a) or FIG. 2(b) t is obtained by the following equation (3).

Equation

[0054] When displaying a three-dimensional image that faithfully reproduces the size perception of the actual subject 10, the image of the subject 10 is resized by a factor of 1 / M. t When prioritizing the field of view over the faithfulness of the size perception, this process is not performed. Here, z l is the distance between the lens arranged behind the subject 10 and the optical element. Note that, similar to the above equations (1) and (2), since the above equation (3) also changes depending on the configuration of the optical system, when the configuration of the optical system changes, the horizontal magnification M t is derived by light wave propagation calculation.

[0055] Furthermore, if necessary, the brightness of each pixel related to the three-dimensional image information of the subject 10 may be square-rooted and used as a new brightness index of the subject 10. When not taking the square root of the brightness, a three-dimensional image robust to noise can be reproduced. However, when taking the square root of the brightness and using this as a new brightness index of the subject 10, although it is slightly more susceptible to noise, it has the advantage of being able to reproduce the brightness of the actual subject 10 more faithfully. It should be noted that the method of calculating the square root of this brightness is a process that is not necessary in coherent digital holography and is specific to incoherent digital holography.

[0056] In incoherent digital holography, analog information of light intensity is obtained as digital information of light amplitude. If this is output directly to the display device 40, the image displayed on the display device 40 will be perceived by the human eye as having the square of the actual light intensity. Therefore, in order to counteract this squaring effect, it is preferable to use the square root of brightness as a new brightness index, as described above.

[0057] (6) Create a CGH (using a calculation means for acquiring CGH). Based on the obtained 3D image information (see Figure 4B(h)), a CGH (Computer-Generated Hologram) for input to the spatial light modulators 45A, B, and C of the display device 40 is created by the following calculation process. Specifically, the depth map (depth map of imaging distance) of the acquired subject 10 is referenced to perform propagation calculations from each pixel of the complex amplitude distribution to an arbitrary hologram display surface, and all of the obtained complex amplitude distributions are added together.

[0058] Furthermore, based on the resolution of the spatial light modulators 45A, B, and C that constitute the display device 40, the three-dimensional distribution of the brightness of the subject 10 is divided into multiple layers stacked in the depth direction, and the propagation calculation of the complex amplitude distribution is performed for each divided layer. By adding the complex amplitude distributions from each of these layers, a CGH can be efficiently created. Specifically, the depth information of the depth map is quantized. Therefore, the 3D image information of the subject 10 can be layered at intervals corresponding to the depth resolution of the display device 40. The depth resolution of the display device 40 is given by equation (4) below.

[0059]

number

[0060] The diameter D of the above hologram is given by the following equation (5), referring to the parameter information of the optical system of the imaging device 20 (see Non-Patent Document 3 mentioned above).

number

[0061]

number

[0062]

number

[0063] As mentioned above, the efficiency of the calculation can be improved by performing propagation calculations of the complex amplitude distribution for each layer and calculating the complex amplitude distribution of the hologram display surface. Finally, from the obtained complex amplitude distribution, calculations based on interference (see, for example, Non-Patent Document 4) or calculations based on diffraction (see, for example, the calculation described in Japanese Patent Application No. 2020-138966 disclosed by the present inventor to the Japan Patent Office) are performed to create an amplitude-type or phase-type CGH according to the modulation physical quantity of the display device 40. The CGH image obtained in this way is shown in Figure 4B(i).

[0064] Furthermore, various conventionally known methods can be used to create amplitude-type or phase-type CGHs. In addition, when an optical system with two spatial light modulators is employed in the display device 40, or when a spatial light modulator capable of modulating a complex amplitude distribution is used, the complex amplitude distribution of the subject 10 can be directly modulated, eliminating the need to perform the above-mentioned interference and diffraction-based calculations and thus eliminating the need to convert to amplitude-type or phase-type.

[0065] B. Noise reduction process generated by the image sensor <Noise reduction processing (for imaging data)> In the signal processing device 30 according to this embodiment, a method and procedure for performing correction processing to obtain three-dimensional image information consistent with the depth information of the subject 10 has been described in order to solve the problems inherent in the incoherent holography technology. Below, we will describe the noise reduction process that occurs in the image sensors 24A and B, which is another problem inherent in the incoherent holography technology. In the imaging device 20 based on incoherent digital holography, random noise originating from the image sensors 24A and B is added when capturing holograms. As a result, it is more susceptible to noise than a normal camera, raising concerns that the quality of the 3D image obtained on the display device 40 may deteriorate.

[0066] Therefore, in the signal processing device 30 according to this embodiment, noise is reduced by comparing the in-plane resolution of the imaging device 20 and the display device 40. Specifically, the pixel pitch s of the image sensors 24A and B constituting the imaging device 20 and the pixel pitch p of the spatial light modulators 45A, B and C constituting the display device 40 are compared, and if p is larger, noise is reduced from the digital hologram, which is the image data, by applying a spatial average calculation to the captured digital hologram or the complex amplitude distribution extracted from the digital hologram.

[0067] For spatial averaging, various methods such as convolution integrals with arbitrary spatial filters, moving averages, weighted averages, and median filters can be used. This allows for the display of high-quality 3D images. In addition, while Patent Document 4 disclosed by the present inventors performs spatial averaging when there is a mismatch between the resolution of the image sensor and the resolution of the optical system constituting the imaging device, the signal processing device 30 of this embodiment differs in that it performs spatial averaging when there is a mismatch between the resolution of the image sensors 24A and B and the resolution of the spatial light modulators 45A, B, and C of the display device 40. According to this embodiment, it is possible to display a high-quality 3D image applicable to the specifications of the display device 40.

[0068] Furthermore, if the pixel pitch s of the image sensors 24A and B constituting the imaging device 20 matches the pixel pitch p of the spatial light modulators 45A, B, and C constituting the display device 40, it is efficient and effective to perform the following noise reduction processing when the image sensors 24A and B are capturing images.

[0069] <Noise reduction processing (during image capture using an image sensor)> In the signal processing device 30, in order to remove noise components generated when the image sensors 24A and 24B of the imaging device 20 capture a hologram, the noise reduction process in propagation calculation shown in Figure 6A, or the noise reduction process by Fourier transform shown in Figure 6B, may be applied.

[0070] In the noise reduction process shown in Figure 6A, the complex amplitude distribution (or hologram) extracted by the phase shift method or off-axis method (Figure 6A(a)) is subjected to zero-padding interpolation to obtain an expanded complex amplitude distribution (or hologram) (Figure 6A(b)). Propagation calculations based on the angular spectrum method or Fresnel diffraction are then applied to this. The propagation distance at this stage is set to a distance long enough to prevent aliasing in the chirp function of the angular spectrum method or Fresnel diffraction. In the complex amplitude distribution after propagation, the region expanded by zero padding extends beyond the field of view or angle of view of the image sensors 24A and B. The components in this region are noise generated during imaging (see Figure 6A(c)). Therefore, all of these values ​​are set to 0, and the backpropagation calculation is performed back to the original hologram plane (see Figure 6A(d)). After this, the expanded region is removed to obtain a complex amplitude distribution (or hologram) with reduced noise components (Figure 6A(e)).

[0071] On the other hand, in the noise reduction process shown in Figure 6B, the complex amplitude distribution (or hologram) extracted by the phase shift method or off-axis method (Figure 6B(a)) is subjected to nearest neighbor interpolation to obtain an expanded complex amplitude distribution (or hologram) (Figure 6B(b)). Next, a Fourier transform is applied to the expanded complex amplitude distribution (or hologram) (Figure 6B(c)). The frequency range above the Nyquist frequency when the hologram was captured is a frequency component that cannot be captured by the image sensors 24A and B, and components in this range can be considered noise. Therefore, components above the Nyquist frequency are cut (trimmed), leaving only the Fourier spectrum in the Nyquist frequency range (Figure 6B(d)). After this, an inverse Fourier transform is applied to obtain a complex amplitude distribution (or hologram) with reduced noise components (Figure 6B(e)).

[0072] The two noise reduction processes described above (noise component reduction based on propagation calculation and Fourier transform) differ only in whether the processing is performed in the real space plane or the Fourier plane (frequency plane). Both reduce noise occurring in the out-of-field region or beyond the frequency component region of the image sensor, and their effects are equivalent. In the above, pixel interpolation processes such as zero padding and nearest neighbor are used for noise reduction, but other pixel interpolation processes, such as bilinear and bicubic interpolation, can also be used for noise reduction in a similar manner.

[0073] Furthermore, since the noise reduction process using pixel interpolation as described above is computationally more demanding than the spatial averaging process mentioned earlier, it is not necessary to perform this process redundantly when noise reduction is performed using the spatial averaging process described above. [Examples]

[0074] The imaging and display system according to this embodiment will be further described below through verification using examples. The imaging and display system 1 according to the embodiment used the optical system shown in Figure 2(b) as the imaging device 20 and the optical system shown in Figure 3(b) as the display device 40 to capture and display three-dimensional images of two shogi pieces (a pawn (close range) and a knight (far range)) which were the subject 10 and were placed at a predetermined distance apart in the depth direction.

[0075] The imaging device 20 used a camera equipped with a lens 21B with a focal length of 500 mm and a diameter of 10 mm, a spatial light modulator 23Ba with a modulation area size of 12.8 mm × 12.8 mm, and a CMOS image sensor 24B with 2048 × 2048 pixels and a pixel pitch of 6.5 μm. The optical path distance between the lens 21B and the spatial light modulator 23Ba was set to 120 mm, and the optical path distance between the spatial light modulator 23Ba and the image sensor 24B was set to 200 mm. In addition, a 4-step phase shift method was used to detect the complex amplitude distribution.

[0076] The display device 40 uses a laser light source 41B with a wavelength of 633 nm and a spatial light modulator 45B with 1024 × 1024 pixels and a pixel pitch of 12.5 μm. The modulation physical quantity of the spatial light modulator 45B is phase. The 4f optical system 46B is constructed using two lenses 46Ba and Bc with a focal length of 100 mm. Furthermore, as the signal processing device 30, a computer equipped with an Intel Core i7-9750H CPU (Intel, Core is a registered trademark) and an NVIDIA GeForce RTX 2060 (NVIDIA, GeForce is a registered trademark) was used, and a signal processing program was created using Python (registered trademark).

[0077] Figure 7(a) shows a group of digital holograms (four images taken with a phase shift of (π / 2) each) captured by the imaging device 20 described above. A 4-step phase shift algorithm was applied to these digital holograms, and the extracted complex amplitude distribution is shown in Figure 7(b). In Figure 7(b), for clarity, only the regions containing optical signals are shown, extracted from each 2048 × 2048 pixel image.

[0078] Furthermore, the method and means used to obtain CGH from this complex amplitude distribution were those described above using Figures 4A and 4B. Figures 7(c), (d), and (e) show the reconstructed image (with extended depth of field), silhouette, and depth map generated from this complex amplitude distribution after applying propagation calculations, respectively. For binarization, 12% of the maximum absolute value of the complex amplitude distribution was used as the threshold, and a closing process with a width of 1% of the field of view was applied to the morphological transformation. In addition, the autofocus evaluation area when creating the depth map was set to a size of 24% of the field of view.

[0079] Figure 8 shows a phase-type CGH created using the signal processing device 30 and the method shown in Figures 4A and 4B (specifically, the method described in the aforementioned Japanese Patent Application No. 2020-138966, which corrects the nonlinearity caused by the carrier phase to enable high-precision modulation of the complex amplitude distribution). The number of pixels in this CGH is 1024 × 1024.

[0080] The pixel pitch of the image sensor 24B of the imaging device 20 is 6.5 μm, and the pixel pitch of the spatial light modulator 45B of the display device 40 is 12.5 μm, resulting in a mismatch of approximately 2x between the two. To remove noise components originating from the image sensor 24B from the captured digital hologram, a 2x2 spatial average is performed, and the effective pixel pitch of the captured digital hologram is set to 13 μm (= 6.5 μm × 2), thereby minimizing the mismatch.

[0081] This phase-type CGH was input to a phase-modulated spatial light modulator 45B, and the generated light was captured and observed using an image sensor. Since the generated light should reproduce the three-dimensional information of the subject, Figure 9 shows images taken by shifting the image sensor along the optical axis so that the image of the subject in the background (knight) and the image of the subject in the foreground (pawn) were in focus. It is clear that when the focus is on the subject in the background (the knight), the image of the subject in the foreground (the pawn) becomes blurred (Figure 9(a)), and conversely, when the focus is on the subject in the foreground (the pawn), the image of the subject in the background (the knight) becomes blurred (Figure 9(b)).

[0082] Thus, verification using this embodiment clearly shows that the front-to-back relationship in the depth direction of the subject 10 is correctly obtained. Therefore, by using the imaging and display system of this embodiment, the depth distance information of the digital hologram of the subject image acquired by the imaging device based on incoherent digital holography is converted into 3D image information that faithfully reproduces the actual subject, making it possible to display a 3D image on the display device 40 that matches the depth distance of the actual subject 10.

[0083] The incoherent digital hologram signal processing device and imaging display system of the present invention are not limited to those of the embodiments described above, and various other modifications are possible. For example, in the imaging and display system of the above-described embodiment, the optical system of the imaging device and display device can be appropriately replaced with other components having similar functions. Furthermore, in the above embodiment, a Michelson-type interference optical system (including a modified Michelson-type equipath-length interference optical system as shown in Figure 2(b)) is used as the optical system of the imaging device, but it is also possible to use other equipath-length interference optical systems such as a Mach-Zehnder type or a bypass-type Fizeau type. [Explanation of symbols]

[0084] 1. Image display system 10 Subjects 20, 20A, 20B Imaging devices 21A, 21B, 42A, 42B, 42C, 43A, 43B, 43C, 46Ba, 46Bc lenses 22A, 22B Photosynthesis Decomposition Element 23A, 23B Spherical phase-imparting optical elements 23Aa concave mirror 23Ab plane mirror 23Ba, 45A, 45B, 45C spatial light modulators 24A, 24B image sensor 25A, 25B wavelength filters 26Ba, 26Bb polarizer 30 Signal Processing Device 40, 40A, 40B, 40C display device 41A, 41B, 41C Laser light source 44A, 44B, 44C Light reflecting means 46Bb filter 47C Angle Selection Filter 50 3D reconstructed image

Claims

1. In an incoherent digital hologram signal processing device that receives incoherent digital hologram information obtained by photographing a subject using the self-interference of light, processes the input incoherent digital hologram information with signal information processing, and sends it to a hologram display unit, The incoherent digital hologram signal processing device is characterized in that the signal information processing is configured to extract shape information and depth distance information of the subject from the input incoherent digital hologram information, and to perform correction processing so that the three-dimensional image information of the subject matches information corresponding to the actual depth distance of the subject, and then send it to the hologram display unit.

2. The incoherent digital hologram signal processing device according to claim 1, characterized in that it is configured to perform a process to correct the lateral magnification according to the configuration of the optical system and the imaging distance of the subject.

3. The incoherent digital hologram signal processing apparatus according to claim 1 or 2, characterized in that it is configured to create a computer-generated hologram based on the three-dimensional image information of the subject to which the correction processing has been performed.

4. The signal information processing is characterized by extracting a complex amplitude distribution from the input incoherent digital hologram information, applying autofocus processing to this complex amplitude distribution to identify the propagation distance at which the sharpness is highest across the entire field of view, applying low-pass filtering to the complex amplitude distribution propagated at this propagation distance, performing binarization processing on the obtained brightness distribution, extracting shape information of the subject by morphological transformation processing, performing mask processing using this shape information, performing the autofocus processing in the region where the subject exists to create a depth map of the reproduction distance, and transforming this depth map of the reproduction distance based on a relational expression between the reproduction distance and the imaging distance derived from predetermined parameter information of the imaging optical system for imaging the incoherent digital hologram information to extract depth distance information of the subject, which is a depth map of the imaging distance. This is the incoherent digital hologram signal processing device according to claim 2, or claim 3, which references claim 2.

5. The incoherent digital hologram signal processing device according to claim 4, characterized in that it is configured to calculate the square root of the brightness distribution and use the value of this square root as an index of a new brightness distribution of the subject.

6. The incoherent digital hologram signal processing apparatus according to claim 4 or 5, referencing claim 3, characterized in that, based on the resolution of the spatial light modulator constituting the hologram display unit, the brightness distribution of the subject is divided into a plurality of layers stacked in the depth direction, the propagation calculation of the complex amplitude distribution is performed for each divided layer, and the computer-generated hologram is created by adding the complex amplitude distributions for each of these layers for which the propagation calculation has been performed.

7. An incoherent digital hologram signal processing apparatus according to any one of claims 1 to 6, comprising a parameter information storage unit that stores parameter information of an optical system relating to an imaging unit that photographs the subject using the self-interference of light, and parameter information of an optical system relating to a hologram display unit, and comprising a first noise reduction processing means that performs spatial averaging processing according to the difference between the resolution of the imaging unit and the resolution of the hologram display unit, and removes noise components added by the imaging unit.

8. The incoherent digital hologram signal processing apparatus according to claim 7, further comprising a second noise reduction processing means that expands the data of the incoherent digital hologram information captured by the imaging unit, or the complex amplitude distribution information extracted from the incoherent digital hologram information, by either a nearest neighbor or zero-padding data interpolation method, diffuses the noise components into the high-frequency region using propagation calculation or Fourier transform, removes the noise present in the high-frequency region, and then removes the noise components by performing backpropagation calculation or inverse Fourier transform.

9. An imaging and display system comprising: an incoherent digital hologram signal processing device according to any one of claims 1 to 8; an imaging device that captures three-dimensional image information of a subject input to the incoherent digital hologram signal processing device using the self-interference of light; and a hologram display device that receives the three-dimensional image information of the subject processed by the incoherent digital hologram signal processing device and displays a three-dimensional reconstructed image of the subject.

Images