Hologram data generation device

Abstract

This invention provides a hologram data generation device that can quickly obtain 3D reproducible hologram data from video data output from existing video equipment. [Solution] The system includes a video data receiving means for receiving video data, a depth estimation means for estimating depth data from the video data, a hologram generation means for generating 3D reproducible hologram data from the video data and depth data, and a hologram data output means for outputting the hologram data.

Description

Technical Field

[0005]

[0001] The present invention relates to a hologram data generation device that generates three-dimensionally reproducible hologram data.

Background Art

[0002] Holography is a technique for recording and reproducing the light waves of a three-dimensional object by utilizing the diffraction and interference phenomena of light. The creation of a hologram in which the light waves of a three-dimensional object are recorded as interference fringes is possible not only by a method using an optical system but also by calculation on a computer. A hologram created by a computer is called a computer-generated hologram (CGH).

[0003] Also, since electroholography can theoretically completely reproduce the wavefront of light and satisfy all the requirements for humans to perceive a three-dimensional object (focus adjustment, convergence, binocular parallax, motion parallax), it is a technique capable of reproducing a natural three-dimensional image for humans. The basic principle of electroholography reproduction is to use an optical system to modulate light (reference light) from a light source such as a laser according to a CGH as a modulation pattern input to a spatial light modulator (SLM), and obtain reproduced light that reproduces the reflected light of a three-dimensional image to reproduce the three-dimensional image.

[0004] For example, as disclosed in Patent Document 1, there is known a device that generates three-dimensionally reproducible hologram data from subject data, which is three-dimensional data, by a hologram data generation device, and inputs the hologram data into an electroholography display device to reproduce a three-dimensional image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] In Patent Document 1, the hologram data generation device uses 3D data from the outset as subject data for generating hologram data, and it was not possible to generate 3D reproducible hologram data from video data obtained from existing televisions, game consoles, computers, etc. Although it is possible to convert 2D video data into 3D data before performing hologram calculations, this increases the amount of data processing required, making hologram calculations time-consuming and thus making real-time generation of hologram data difficult.

[0007] This invention was made in view of these problems, and aims to provide a hologram data generation device that can quickly obtain 3D reproducible hologram data from video data output from an existing video device. [Means for solving the problem]

[0008] To solve the above problems, the hologram data generation apparatus of the present invention is A means for receiving video data, A depth estimation means for estimating depth data from the aforementioned video data, A hologram generation means that generates 3D reproducible hologram data from the aforementioned video data and depth data, The system is characterized by comprising a hologram data output means for outputting the aforementioned hologram data. This feature allows for the generation of 3D reproducible hologram data by combining video data with depth data estimated from the video data and performing hologram calculations. By outputting this hologram data, 3D reproducible hologram data can be obtained at high speed from video data output from existing video equipment.

[0009] The video data receiving means generates a plurality of 2D RGB image data captured from the video data at a predetermined frame rate, The hologram generation means generates the hologram data at the frame rate, The hologram data output means is characterized by outputting the hologram data to the optical system at the frame rate. According to this feature, hologram data generated at a predetermined frame rate is output to the optical system at the predetermined frame rate, enabling the real-time reproduction of a 3D image from video data output from an existing video device.

[0010] The hologram data output means is characterized by outputting the hologram data to a display at the frame rate. This feature allows for real-time verification of the correspondence between the 3D image and the hologram data by displaying the hologram data used to reconstruct the 3D image in the optical system as an image on a display.

[0011] The hologram data output means is characterized by outputting reconstructed image data, simulated on a computer from the hologram data, to a display at the frame rate. This feature allows for real-time verification of the correspondence between the reconstructed image data, which is a simulated image of the 3D image reconstructed in the optical system, and the hologram data, by outputting the reconstructed image data, which is a simulated image of the 3D image reconstructed on a computer, to a display.

[0012] The system is characterized by generating corrected hologram data based on the image output to the aforementioned display. This feature allows for the generation of corrected hologram data output to the optical system based on the hologram data or reconstructed image data displayed on the screen, thereby improving the quality of the three-dimensional image reconstructed in the optical system.

[0013] It is characterized by including stereo estimation means for estimating stereo data from the video data, and using the stereo data as the video data handled by the depth estimation means and the hologram generation means. According to this feature, it becomes possible to generate hologram data for the left eye and hologram data for the right eye respectively from the stereo data, and a three-dimensional image reproduced in the optical system can be viewed with both eyes.

Brief Description of the Drawings

[0014] [Figure 1] It is a diagram showing the configuration of a hologram data generation device in Example 1 according to the present invention. [Figure 2] It is a diagram showing the configuration of a hologram data generation device in Example 2 according to the present invention. [Figure 3] It is a diagram showing the configuration of a hologram data generation device in Example 3 according to the present invention.

Modes for Carrying Out the Invention

[0015] Modes for carrying out the hologram data generation device according to the present invention will be described below based on examples.

Examples

[0016] The hologram data generation device of the present invention generates hologram data that can be three-dimensionally reproduced in real time from video data, which is two-dimensional data output from existing video devices such as cameras, game machines, televisions, computers, etc., rather than shooting data captured by a dedicated device such as a three-dimensional camera.

[0017] As shown in FIG. 1, the hologram data generation device of this embodiment includes video data receiving means for receiving video data a, a receiving unit for receiving two-dimensional RGB image data b1 generated from the video data a by the video data receiving means, depth estimation means for estimating depth data c from the two-dimensional RGB image data b2 input from the receiving unit, hologram generation means for generating hologram data d1 from the two-dimensional RGB image data b2 and the depth data c corresponding to the two-dimensional RGB image data b2, a transmitting unit for inputting the hologram data d1 generated by the hologram generation means and transmitting hologram data d2 to the hologram output means, and hologram data output means for outputting hologram data d3 to an optical system and / or a display.

[0018] Specifically, the hologram data generation device of this embodiment is mainly composed of an FPGA evaluation board including video data receiving means and hologram data output means, and a GPU board including a receiving unit, depth estimation means, hologram generation means, and a transmitting unit.

[0019] Hereinafter, the generation and output of hologram data in the hologram data generation device of this embodiment will be described with reference to FIG. 1.

[0020] First, the video data receiving means in the FPGA evaluation board receives video data a, that is, a video signal, output from an existing video device, and captures it at a predetermined frame rate to generate a plurality of two-dimensional RGB image data b1.

[0021] In this embodiment, NeRF (Neural Radiance Field) is used in the video data receiving means. NeRF is a deep neural network that can input display information of position and direction angle from the video data a and generate two-dimensional RGB image data b1 at high speed. Also, the generation of the two-dimensional RGB image in the video data receiving means is not limited to NeRF, and any method may be used.

[0022] Next, the multiple 2D RGB image data b1 generated in the video data receiving means are sequentially transferred to the GPU board in chronological order and received by the receiving unit.

[0023] Next, multiple 2D RGB image data b2 are sequentially input in time series from the GPU board's receiving unit to the depth estimation means, and depth data c is estimated from each 2D RGB image data b2. The depth data c is the depth data necessary for 3D conversion of the 2D RGB image data b2, and is generated as a depth map corresponding to the 2D RGB image data b2.

[0024] In this embodiment, the depth estimation means uses the depth estimation model MiDaS (Intel). MiDaS is a deep neural network that can generate a depth map corresponding to a single image from that image data at high speed. Furthermore, the depth estimation in the depth estimation means is not limited to MiDaS; any method may be used.

[0025] Next, the hologram generation means performs hologram calculation, i.e., light diffraction calculation, from the two-dimensional RGB image data b2 that is sequentially input from the receiving unit in a time series, and the depth data c corresponding to the two-dimensional RGB image data b2 that is sequentially input from the depth estimation means, and generates hologram data d1 at a predetermined frame rate. The hologram data d1 generated by the hologram generation means is computer-generated hologram (CGH) data.

[0026] In this embodiment, Split Lohmann computer holography is used as the hologram generation means. Split Lohmann computer holography can quickly generate 3D reproducible hologram data d1 from 2D RGB image data b2 and depth data c with a single diffraction calculation. Furthermore, the hologram calculation in the hologram generation means is not limited to Split Lohmann computer holography; any method such as a point cloud data method or a layer-based method may be used.

[0027] Table 1 shows an example of the operating environment for the GPU board. Since MiDaS, as a depth estimation method, utilizes the PyTorch machine learning library, the overall operating environment also uses PyTorch.

[0028] [Table 1]

[0029] Next, the multiple hologram data d1 generated by the hologram generation means are sequentially input in chronological order to the transmission unit of the GPU board.

[0030] Next, the hologram data d2 is sequentially transferred in chronological order from the transmission unit of the GPU board to the hologram data output means on the FPGA evaluation board.

[0031] Finally, the hologram data output means generates hologram data d3 that can be reproduced / displayed by the optical system and / or display from a plurality of hologram data d2, and outputs it sequentially to the optical system and / or display at a predetermined frame rate, thereby reproducing a three-dimensional image, i.e., a reconstructed image, in the optical system, and displaying the hologram data d3, i.e., interference fringes of the hologram, on the display.

[0032] The hologram data output means may be configured to sequentially output the reconstructed image data simulated on a computer from the hologram data d3 to a display at a predetermined frame rate. Alternatively, the hologram data output means may be configured to output both the hologram data d3 and the reconstructed image data to a display. In this case, both the hologram data d3 and the reconstructed image data may be output to a single display, or to separate displays.

[0033] Furthermore, the hologram data output means may be capable of switching the output to either the optical system or the display.

[0034] In this embodiment, the optical system is a holographic display. However, the optical system is not limited to a holographic display; any device capable of reproducing a three-dimensional image from hologram data d3 may be used, such as a near-eye display or a glasses-free display.

[0035] Furthermore, the display in this embodiment is a liquid crystal monitor provided separately from the hologram data generation device. Alternatively, the display may be any device capable of displaying hologram data d3 and reconstructed image data, and may be provided integrally with the hologram data generation device.

[0036] As described above, the hologram data generation device of this embodiment is composed of an FPGA evaluation board equipped with video data receiving means and hologram data output means that performs conversion from 2D data to 3D data, and a GPU board equipped with a receiving unit, depth estimation means, hologram generation means, and transmission unit that performs depth estimation and hologram calculation. By combining a plurality of 2D RGB image data generated from video data with depth data estimated from the 2D RGB image data and performing hologram calculation, it is possible to generate 3D reproducible hologram data. Therefore, by outputting this hologram data, 3D reproducible hologram data d can be obtained at high speed from video data output from an existing video device.

[0037] Furthermore, in this embodiment, the hologram data generation device generates hologram data d3 based on a plurality of 2D RGB image data b1 generated by capturing video data a at a predetermined frame rate in the video data receiving means, and outputs this hologram data d3 to the optical system at a predetermined frame rate by the hologram data output means, thereby enabling the reproduction of a 3D image in real time from video data a output from an existing video device.

[0038] Furthermore, the hologram data output means outputs the hologram data d3, which is used to reproduce the 3D image in the optical system, as an image on a display, allowing the correspondence between the 3D image and the hologram data d3 to be confirmed in real time.

[0039] Furthermore, the hologram data output means outputs the reconstructed image data, which is a simulated image of the three-dimensional image reconstructed in the optical system, as an image to a display by a computer, thereby allowing real-time confirmation of the correspondence between the reconstructed image data and the hologram data d3.

[0040] Here, the hologram data output means does not output the hologram data d3 to the optical system, but outputs the hologram data d3 and / or reconstructed image data only to the display, making it possible to confirm whether the generation of the hologram data d3 is being performed correctly. After confirmation, by switching the output of the hologram data d3 to the optical system, the quality of the three-dimensional image reconstructed in the optical system can be ensured.

[0041] Furthermore, the hologram data generation device of this embodiment uses NeRF as the video data receiving means, MiDaS as the depth estimation means, and Split Lohmann computer holography as the hologram generation means, enabling high-speed generation of hologram data d3 from video data a, thus offering high real-time performance.

[0042] Thus, the hologram data generation device of this embodiment can obtain hologram data d3 that can be reproduced in 3D in real time from video data a, which is 2D data output from an existing video device, without making any changes to the existing video device, and can therefore realize 3D display at low cost.

[0043] Furthermore, the three-dimensional image reconstructed by the optical system from the hologram data d3 generated by the hologram data generation device of this embodiment, and the reconstructed image data output to the display, may be either monochrome or color. [Examples]

[0044] Next, the hologram data generation apparatus according to Example 2 will be described with reference to Figure 2. Note that descriptions of components that are identical to those in Example 1 and therefore redundant will be omitted.

[0045] As shown in Figure 2, the hologram data generation device of this embodiment corrects the hologram data based on the image output to the display.

[0046] More specifically, the hologram data generation device of this embodiment generates correction data f for correcting defects in the 3D image reproduced by the optical system based on the hologram data d3, which is the image output from the hologram data output means to the display, and / or the reproduced image data, and feeds back the correction data f to the hologram generation means on the GPU board.

[0047] Here, a defect in the 3D image refers to, for example, a defect in Split Lohmann computer holography used for hologram generation in hologram generation methods, where the 3D image is spatially shifted depending on the depth due to the refraction of light.

[0048] Furthermore, to generate the correction data f, a deep neural network or the like may be used, which compares imaging data obtained by capturing the image of the hologram data d3 and / or the reconstructed image data output to the display with the image data a or 2D RGB image data b2 to extract defects and generate correction data f to correct those defects.

[0049] Furthermore, the correction data f may be generated in a hologram data generation device, or it may be generated in a hologram correction device provided separately from the hologram data generation device.

[0050] When correction data f is fed back to the hologram generation means, the hologram generation means generates corrected hologram data d1' that reflects the correction based on correction data f to the hologram data d1 generated by performing hologram calculations from the 2D RGB image data b2 that is sequentially input from the receiving unit in a time series and the depth data c that is sequentially input from the depth estimation means.

[0051] Furthermore, the correction based on the correction data f is not limited to reflecting the correction after generating the hologram data d1; the correction may also be reflected at the stage of performing the hologram calculation, directly generating the corrected hologram data d1'.

[0052] The corrected hologram data d1' generated in the hologram generation means is input to the transmission unit, and the corrected hologram data d2' is sequentially transferred in chronological order to the hologram data output means on the FPGA evaluation board.

[0053] The hologram data output means generates corrected hologram data d3' from the corrected hologram data d2' that can be reproduced / displayed by the optical system and / or display, and outputs it sequentially to the optical system and / or display at a predetermined frame rate, thereby reproducing a corrected 3D image in the optical system.

[0054] Furthermore, the display may be configured to output corrected hologram data d3' and / or reconstructed image data simulated on a computer from the corrected hologram data d3', thereby generating further correction data for the corrected hologram data d3' and feeding it back to the hologram generation means.

[0055] Thus, the hologram data generation device of this embodiment can improve the quality of the three-dimensional image reproduced in the optical system by correcting the hologram data output to the optical system in real time based on the hologram data d3 or the image of the reproduced image data output to the display.

[0056] In this embodiment, it has been described that hologram data d3 and corrected hologram data d3' are output from the hologram data output means to the optical system and the display. However, the hologram data output means may be configured to output only the corrected hologram data d3' to the optical system. Specifically, the hologram data output means may be configured to output only the uncorrected hologram data d3 and / or the reconstructed image data simulated on a computer from the hologram data d3 to the display, and to output only the corrected hologram data d3' generated by the feedback of correction data f to the optical system. As a result, although the 3D image reconstructed in the optical system is slightly delayed compared to the image output to the display by the time required for correction, the quality of the 3D image can be improved without significantly impairing real-time performance. [Examples]

[0057] Next, the hologram data generation apparatus according to Example 3 will be described with reference to Figure 3. Note that the description of components that are the same as those in Examples 1 and 2 and therefore overlap will be omitted.

[0058] As shown in Figure 3, the hologram data generation device of this embodiment mainly consists of an FPGA evaluation board equipped with video data receiving means and hologram data output means, and a GPU board equipped with a receiving unit, stereo estimation means, depth estimation means, hologram generation means, and transmission unit, thereby enabling the generation of hologram data for both the left and right eyes.

[0059] In more detail, in this embodiment, the hologram data generation device receives multiple 2D RGB image data b2 sequentially in time series from the receiving unit of the GPU board to the stereo estimation means, thereby estimating stereo data b3L for the left eye and stereo data b3R for the right eye from each 2D RGB image data b2.

[0060] The stereo data b3L for the left eye and the stereo data b3R for the right eye are sequentially input to the depth estimation means, respectively, so that the depth data cL for the left eye and the depth data cR for the right eye are estimated from the stereo data b3L for the left eye and the stereo data b3R for the right eye.

[0061] The hologram generation means performs hologram calculations using stereo data b3L for the left eye and stereo data b3R for the right eye, which are sequentially input from the stereo estimation means, and depth data cL for the left eye and depth data cR for the right eye, which are sequentially input from the depth estimation means, and generates hologram data d1L for the left eye and hologram data d1R for the right eye at a predetermined frame rate.

[0062] Thus, the stereo data b3L for the left eye and the stereo data b3R for the right eye are video data handled by the depth estimation means and the hologram generation means.

[0063] The hologram data d1L for the left eye and hologram data d1R for the right eye, generated in the hologram generation means, are input to the transmission unit, and the hologram data d2L for the left eye and hologram data d2R for the right eye are sequentially transferred in chronological order to the hologram data output means on the FPGA evaluation board.

[0064] The hologram data output means generates hologram data d3L for the left eye and hologram data d3R for the right eye from the hologram data d2L for the left eye and hologram data d2R for the right eye, which can be reproduced / displayed by the optical system and / or display, and outputs them sequentially to the optical system and / or display at a predetermined frame rate, thereby reproducing the 3D image for the left eye and the 3D image for the right eye, respectively, in the optical system.

[0065] Thus, the hologram data generation device of this embodiment is equipped with a stereo estimation means that estimates stereo data b3L and b3R from a plurality of two-dimensional RGB image data b2 generated from video data a. This makes it possible to generate hologram data d3L for the left eye and hologram data d3R for the right eye from the stereo data b3L and b3R, respectively, and enables binocular viewing of the three-dimensional image reproduced by the optical system. Furthermore, in this embodiment, the optical system is preferably a holographic display that shows three-dimensional images corresponding to the left and right eyes, respectively.

[0066] In this embodiment, we have described a method of generating two stereo data sets, d3L for the left eye and d3R for the right eye, from the stereo data sets b3L and b3R estimated by the stereo estimation means. However, the invention is not limited to this, and hologram data sets may be generated for each set viewpoint by performing multi-viewpoint image estimation on the video data.

[0067] Although embodiments of the present invention have been described above with reference to the drawings, the specific configurations are not limited to these embodiments, and any changes or additions that do not depart from the spirit of the present invention are also included. [Industrial applicability]

[0068] The present invention is a hologram data generation device that can reproduce a 3D image in real time from 2D video data output from existing video devices such as cameras, game consoles, televisions, and computers. As such, it does not require any modification of existing video devices or broadcasting equipment, and it can reproduce a 3D image without consuming the bandwidth required for 3D video communication, thus having industrial applicability. Furthermore, the hologram data generation device of the present invention does not require any special technology or equipment to convert the input video data into 3D data, and can utilize existing video devices such as cameras, game consoles, televisions, and computers, making it highly versatile and with a wide range of applications.

Claims

1. A means for receiving video data, A depth estimation means for estimating depth data from the aforementioned video data, A hologram generation means that generates hologram data capable of three-dimensional reproduction from the aforementioned video data and depth data, A hologram data generation apparatus characterized by comprising a hologram data output means for outputting the aforementioned hologram data.

2. The video data receiving means generates a plurality of two-dimensional RGB image data captured from the video data at a predetermined frame rate, The hologram generation means generates the hologram data at the frame rate, The hologram data generation apparatus according to claim 1, characterized in that the hologram data output means outputs the hologram data to the optical system at the frame rate.

3. The hologram data generation apparatus according to claim 2, characterized in that the hologram data output means also outputs the hologram data to a display at the frame rate.

4. The hologram data generation apparatus according to claim 2, characterized in that the hologram data output means outputs reconstructed image data simulated on a computer from the hologram data to a display at the frame rate.

5. The hologram data generation apparatus according to claim 3 or 4, characterized in that it generates corrected hologram data based on the image output to the display.

6. The hologram data generation apparatus according to claim 1, further comprising a stereo estimation means for estimating stereo data from the aforementioned video data, wherein the stereo data is the video data handled by the depth estimation means and the hologram generation means.

Images