Method and apparatus for converting reproduction pixel interval and reproduction wavelength of hologram pattern

Abstract

According to a preferred embodiment of the present invention, a method for converting a reproduction pixel interval and a reproduction wavelength of a hologram pattern comprises the steps of: receiving a first complex hologram pattern; converting the first complex hologram pattern into a second complex hologram pattern through an artificial neural network trained to convert a reproduction pixel interval and a reproduction wavelength; and outputting the second complex hologram pattern.

Description

Method and device for converting pixel spacing and wavelength of reproduction of holographic pattern

[0001] An embodiment of the present invention relates to a method and device for converting a reproduction pixel spacing and a reproduction wavelength of a hologram pattern, and more specifically, to a method and device for converting a reproduction pixel spacing and a reproduction wavelength of a hologram pattern using deep learning technology.

[0002] Holography is a technology that records and reproduces light from an object using the interference phenomenon of light. Recently, the most commonly used computer-generated holograms are created by assuming a target image located a certain distance from the holographic display and calculating the holographic pattern using a numerical light propagation model. Numerical propagation models can be used in a variety of ways, but the most representative method is to assume each pixel of the target image as a point light source and add up all the diffraction patterns emitted by it. The angular spectrum method is the most representative of these.

[0003] When calculating a numerical propagation model, the specifications of the holographic display system, such as the wavelength of the light source and the pixel spacing of the display, must be taken into account. Therefore, to reproduce the same content on different holographic display systems, the same computational process must be performed individually for each system, resulting in the drawback of unnecessary computation. Holographic pattern synthesis suffers from a massive computational bottleneck, proportional to the product of the number of pixels in the target image and the diffusion angle of the holographic display.

[0004] The technical problem to be achieved by an embodiment of the present invention is to provide a method and device for converting a reproduction pixel interval and a reproduction wavelength of a hologram pattern, which can replace repetitive operations using deep learning technology and extract and interpret reproduction image information from an existing hologram pattern to synthesize a hologram pattern reflecting different holographic display specifications.

[0005] As a preferred embodiment of the present invention, a method for converting a reproduction pixel spacing and a reproduction wavelength of a hologram pattern is characterized by including the steps of: receiving a first complex hologram pattern; converting the first complex hologram pattern into a second complex hologram pattern through an artificial neural network trained to convert the reproduction pixel spacing and the reproduction wavelength; and outputting the second complex hologram pattern.

[0006] According to an embodiment of the present invention, not only can individual operations for different holographic displays be effectively reduced, but also it can be used to increase hologram transmission speed using a deep learning algorithm.

[0007] Figure 1 is a drawing showing an example of a numerical propagation model used in the present invention;

[0008] Figure 2 is a drawing showing an example of a hologram reproduction simulation result in which the generation wavelength and reproduction wavelength of the hologram are different.

[0009] FIG. 3 is a drawing illustrating an example of a method for generating a hologram pattern according to an embodiment of the present invention;

[0010] FIG. 4 is a diagram illustrating an example of an artificial neural network according to an embodiment of the present invention;

[0011] FIG. 5 is a diagram illustrating an example of a method for generating training data for an artificial neural network according to an embodiment of the present invention.

[0012] FIG. 6 is a diagram illustrating a learning process when training an artificial neural network using training data according to an embodiment of the present invention.

[0013] FIG. 7 is a diagram illustrating an example of a configuration of a pixel spacing and a wavelength conversion device for reproducing a hologram pattern according to an embodiment of the present invention, and

[0014] FIG. 8 is a drawing showing an experimental example of a reproduction pixel spacing and reproduction wavelength conversion method of another hologram pattern according to an embodiment of the present invention.

[0015] FIG. 9 illustrates a flowchart of a method for reproducing pixel spacing and reproducing wavelength conversion of another hologram pattern according to an embodiment of the present invention.

[0016] As a preferred embodiment of the present invention, the display in which the first complex hologram pattern and the second complex hologram pattern are reproduced is characterized in that the pixel spacing and the wavelength of the light source are different.

[0017] As a preferred embodiment of the present invention, the converting step is characterized by including a step of inputting data obtained by combining the real part and the imaginary part of the first complex hologram pattern in a channel direction into the artificial neural network to generate the second complex hologram pattern.

[0018] As a preferred embodiment of the present invention, the artificial neural network is characterized in that it restores an image in which the second complex hologram pattern is reproduced by depth, and then compares the mask image generated by applying a binary depth mask of the corresponding depth with the original image provided as the correct pattern of the artificial neural network to calculate a loss function. The artificial neural network is characterized in that, after calculating the loss function, it updates the parameters using an automatic gradient calculation technique.

[0019] As another preferred embodiment of the present invention, a device for converting a reproduction pixel interval and a reproduction wavelength of a hologram pattern is characterized by including: an artificial neural network trained to convert a reproduction pixel interval and a reproduction wavelength; and a conversion unit that converts a hologram pattern of the first wavelength into a hologram pattern of the second wavelength through the artificial neural network.

[0020] Hereinafter, a method and device for reproducing a wavelength of a hologram pattern according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0021] Figure 1 is a drawing showing an example of a numerical propagation model used in the present invention.

[0022] Referring to Fig. 1, a holographic display is a technology that can restore three-dimensional stereoscopic information by modulating the wavefront of light. A computer-generated hologram (hereinafter, hologram) is an interference pattern that is reproduced on a holographic display and allows modulation of the wavefront of light on a pixel-by-pixel basis. A hologram can be generated by numerically calculating free-space propagation from a target image plane to a hologram plane. Numerical propagation models that can be used at this time include the angular spectrum method, Fresnel propagation, and Fourier transform. This embodiment uses the angular spectrum method as a hologram synthesis model, but this is only one of the hologram synthesis methods, and other numerical propagation models can be used.

[0023] When generating holograms using a numerical propagation model, the physical elements of the display system are taken into account. For example, the maximum diffraction angle of a holographic pattern is proportional to the ratio of the display pixel spacing and the wavelength of the light source. Therefore, to reproduce the same content on holographic displays with different specifications, the numerical propagation model for the hologram must be repeated for each system, with only different values ​​but the same computational process. If the computation is not repeated and the pixel spacing is , wavelength The hologram generated for pixel spacing , wavelength When playing with , the target depth changes and accurate focus adjustment cannot be provided.

[0024] Figure 2 is a diagram showing an example of a hologram reproduction simulation result in which the generation wavelength and reproduction wavelength of the hologram are different.

[0025] Referring to Fig. 2, the simulation results are shown when the hologram is reproduced in the same reproduction environment as the generation pixel spacing and generation wavelength (hereinafter referred to as the generation environment) and when it is reproduced in a different reproduction environment. The target pixel spacing and wavelength were set to 6.4 um, 660 nm, 532 nm, and 470 nm to generate the hologram, and in both cases (a) and (b), it was reproduced at the same depth (5 mm) on the hologram plane. In the case of (a), RGB was reproduced with wavelengths of 660 nm, 532 nm, and 470 nm for the generation environment of pixel pitch 6.4 um, and in the case of (b), RGB was reproduced with wavelengths of 638 nm, 520 nm, and 450 nm for the pixel pitch 8 um, respectively. Looking at the enlarged images of some parts (210a, 220a), (a) is in focus and clear (210b), and (b) is out of focus and the lines are not clear (220b).

[0026] FIG. 3 illustrates an example of a method for generating a hologram pattern according to one embodiment of the present invention.

[0027] An artificial neural network (300) receives a first complex hologram pattern (310) as input, extracts and interprets playback image information, and outputs a second complex hologram pattern (320) that converts the pixel spacing and wavelength of the first complex hologram pattern (310) so that the playback image can be played back identically even on holographic displays with different pixel spacing and light source wavelengths.

[0028] The artificial neural network (300) utilizes deep learning techniques based on supervised learning strategies. Supervised learning-based deep learning techniques require a large dataset, as they learn inherent statistical distributions from multiple images. In one embodiment of the present invention, a complex hologram dataset generated using a 3,800-pair RGBD dataset can be used as training data.

[0029] The artificial neural network (300) inputs data obtained by concatenating the real part and the imaginary part of the first complex hologram pattern (310) in the channel direction into the artificial neural network (300) to generate a second complex hologram pattern (300). In a holographic display (spatial light modulator), complex modulation is impossible, so encoding in a real-valued pattern is required. However, the artificial neural network (300) provided in the present invention generates the first complex hologram pattern (300), so a user can use an encoding in a desired form.

[0030] The artificial neural network (300) can be implemented using various conventional deep neural networks, such as a convolutional neural network (CNN). An example of the artificial neural network (300) for this embodiment is illustrated in FIG. 4. A method for generating a large dataset for training the artificial neural network (300) will be discussed again in FIGS. 5 and 6.

[0031] FIG. 4 is a diagram illustrating an example of an artificial neural network according to an embodiment of the present invention.

[0032] As a preferred embodiment of the present invention, the network (400) of the artificial neural network receives an RGB full-color complex hologram as input and outputs a converted RGB full-color complex hologram. At this time, a supervised learning strategy that provides an input pattern and a correct pattern (ground truth) is used.

[0033] Referring to FIG. 4, an artificial neural network (400) for converting a reproduction environment of a hologram pattern may include one convolutional layer, ten residual blocks, and one output branch. Each residual block (410) may include two convolutional layers, two batch normalization layers, and an activation function. An example of the activation function is LeakyReLU. The output branch (420) may be configured to include a convolutional layer. This embodiment is merely an example to help understanding, and an artificial neural network for converting a reproduction environment of a hologram pattern may be implemented in various forms and is not limited to this embodiment.

[0034] FIG. 5 is a diagram illustrating an example of a method for generating training data for an artificial neural network according to an embodiment of the present invention.

[0035] The wavelengths of the light sources in the input environment are R in the order of red, green, and blue. 1, The wavelengths are G1 and B1, the pixel size of the spatial light modulator (SLM) is the first pixel size, and the wavelength of the light source of the target environment to be converted is R. 2, The wavelengths are G2 and B2, and the pixel size of the spatial light modulator (SLM) has a second pixel size.

[0036] Referring to Fig. 5, the wavelengths of the light sources of the holographic display system in the input environment are 660 nm, 532 nm, and 470 nm, and the pixel size of the spatial light modulator is 6.4 um. The wavelengths of the light sources of the holographic display system in the target environment are 638 nm, 520 nm, and 450 nm in red, green, and blue, respectively, and the pixel size of the spatial light modulator is 8 um.

[0037] The depth range distribution of the test data assumes a total of five depth planes p1, p2, p3, p4, and p5 from a plane 5 mm away from the spatial light modulator (SLM) to a plane 15 mm away. The pixel spacing and wavelength conversion device for reproducing the hologram pattern can generate a hologram pattern (i.e., a complex hologram pattern) on the spatial light modulator planes (i.e., hologram planes) p1, p2, p3, p4, and p5 by applying a numerical propagation method. At this time, the device generates hologram patterns for various wavelengths and uses them as training data. The device trains an artificial neural network using a supervised learning method using the training data.

[0038] Figure 6 illustrates the learning process when training an artificial neural network using training data according to an embodiment of the present invention.

[0039] Referring further to Fig. 6, the artificial neural network restores the image reproduced by the second complex hologram pattern at depths p1, p2, p3, p4, and p5, and then applies the binary depth mask of the corresponding depths p1, p2, p3, p4, and p5 to generate a masked image.

[0040] The focal stack of the second complex hologram pattern is multiplied by a binary depth mask (element-wise multiplication) and then added to generate a mask image. The learning unit applies the above process to the second complex hologram pattern and the correct hologram pattern, respectively, to generate mask image_out (631) and mask image_target (632), and calculates the L2 norm mean squared error between mask image_out (631) and mask image_target (632) and uses it as a loss function. Afterwards, the parameters of the artificial neural network are updated using PyTorch's automatic gradient operation technology (Autograd).

[0041] FIG. 7 is a diagram illustrating an example configuration of a wavelength conversion device for reproducing a hologram pattern according to an embodiment of the present invention.

[0042] Referring to FIG. 7, the device (700) includes a training data generation unit (710), a learning unit (720), an artificial neural network (730), and a conversion unit (740). In one embodiment, the device (700) may be implemented as a computing device including a memory, a processor, and an input / output device. In this case, each component may be implemented as software, loaded into the memory, and then executed by the processor.

[0043] The training data generation unit (710) generates a dataset for training the artificial neural network (730). For example, the training data generation unit (710) may generate training data using the method described in FIG. 5. In another embodiment, if the training data is predefined, the training data generation unit (710) may be omitted.

[0044] The learning unit (720) trains the artificial neural network (730) using training data. The learning unit (720) can train the artificial neural network (730) using a supervised learning method.

[0045] The conversion unit (740) converts a hologram pattern of a first wavelength into a hologram pattern of a second wavelength using a trained artificial neural network. In other words, the conversion unit (740) inputs a hologram of a first wavelength into an artificial neural network (730) that performs reproducible wavelength conversion between the first wavelength and the second wavelength, and obtains a hologram pattern of a second wavelength from the artificial neural network (730).

[0046] Figure 8 illustrates the results of test inference on a test image using an artificial neural network according to a preferred embodiment of the present invention.

[0047] Since the present invention targets hologram patterns for not only 2D but also 3D objects, test inference was conducted for 3D objects. In Fig. 8, the left side shows the result (800) of reproducing a hologram pattern generated to fit a conversion target environment (pixel spacing 8 um, wavelengths 638 nm, 520 nm, 450 nm) using a conventional numerical propagation model (ASM) according to the target environment. In Fig. 8, the right side shows the result (830) of reproducing a hologram image generated to fit an input environment (pixel spacing 6.4 um, wavelengths 660 nm, 532 nm, 470 nm) according to an artificial neural network according to a preferred embodiment of the present invention according to the target environment.

[0048] Fig. 9 is a flowchart illustrating a method for reproducing pixel spacing and reproducing wavelength of a hologram pattern as a preferred embodiment of the present invention. When comparing the correct reproduction result (800) with the reproduction result (830) after conversion in an artificial neural network according to an embodiment of the present invention, it can be confirmed that the focus is established in a similar depth range and a blur is formed. The enlarged images (810b, 820b) of some images (810a, 820a) of the correct reproduction result (800) and the enlarged images (840b, 850b) of some images (840a, 850a) of the reproduction result (830) after conversion in an artificial neural network according to an embodiment of the present invention.

[0049] A flowchart of a method for reproducing pixel spacing and reproducing wavelength conversion of another hologram pattern according to an embodiment of the present invention is shown.

[0050] As one embodiment of the present invention, a first complex hologram pattern is input into an artificial neural network (S910). The input first complex hologram pattern is converted into a second complex hologram pattern through an artificial neural network trained to convert the reproduction pixel interval and reproduction wavelength (S920).

[0051] The present invention can also be implemented as computer-readable program code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Furthermore, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

[0052] The present invention has been described above, focusing on preferred embodiments thereof. Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from its essential characteristics. Therefore, the disclosed embodiments should be considered illustrative rather than limiting. The scope of the present invention is set forth in the claims, not the foregoing description, and all differences within the scope equivalent thereto should be construed as being encompassed by the present invention.

Claims

1. Step of receiving a first complex hologram pattern; A step of converting the first complex hologram pattern into a second complex hologram pattern through an artificial neural network learned to convert the reproduction pixel interval and the reproduction wavelength; and A method for reproducing pixel spacing and reproducing wavelength conversion of a hologram pattern, characterized by including a step of outputting the second complex hologram pattern.

2. In paragraph 1, A method for reproducing pixel spacing and reproducing wavelength of a hologram pattern, characterized in that a display in which the first complex hologram pattern and the second complex hologram pattern are reproduced has different pixel spacing and wavelengths of light sources.

3. In the first paragraph, the converting step, A method for reproducing pixel spacing and reproducing wavelength of a hologram pattern, characterized by comprising: a step of generating the second complex hologram pattern by inputting data obtained by combining the real part and the imaginary part of the first complex hologram pattern in the channel direction into the artificial neural network.

4. In the first paragraph, the artificial neural network A method for reproducing pixel intervals and wavelengths of a hologram pattern, characterized in that a loss function is calculated by restoring an image in which the second complex hologram pattern is reproduced by depth and then comparing the generated mask image by applying a binary depth mask of the corresponding depth with the original image provided as the correct pattern of the artificial neural network.

5. In paragraph 4, the artificial neural network A method for reproducing pixel spacing and reproducing wavelength conversion of a hologram pattern, characterized in that after calculating the above loss function, parameters are updated using an automatic gradient calculation technique.

6. In the first paragraph, the artificial neural network A method for reproducing pixel spacing and reproducing wavelength conversion of a hologram pattern, characterized by using a deep learning technique based on a supervised learning strategy.

7. An artificial neural network trained to transform the reproducible pixel spacing and reproducible wavelength; and A device for reproducing pixel spacing and wavelength of a hologram pattern, characterized by including a conversion unit that converts a hologram pattern of the first wavelength into a hologram pattern of the second wavelength through the artificial neural network.

8. In paragraph 7, A display in which the first complex hologram pattern and the second complex hologram pattern are reproduced is characterized by having different pixel spacings and wavelengths of light sources, and a device for converting the reproduction pixel spacing and reproduction wavelength of a hologram pattern.

9. In paragraph 1, the conversion unit A device for reproducing pixel spacing and wavelength conversion of a hologram pattern, characterized in that data obtained by combining the real part and the imaginary part of the first complex hologram pattern in the channel direction is input to the artificial neural network to generate the second complex hologram pattern.

10. A device for converting a reproduction pixel interval and a reproduction wavelength of a hologram pattern, characterized in that it further includes a learning unit for training the artificial neural network using a supervised learning method using training data including a dataset of an input hologram pattern of a first wavelength and a correct hologram pattern of a second wavelength in the 7th paragraph.

11. In paragraph 7, A wavelength conversion device for reproducing a hologram pattern, characterized by further including a learning unit for calculating a loss function by restoring an image in which the second complex hologram pattern is reproduced by depth and then comparing the generated mask image by applying a binary depth mask of the corresponding depth with the original image provided as the correct pattern of the artificial neural network.

12. In paragraph 7, the artificial neural network A device for reproducing pixel spacing and reproducing wavelength of a holographic pattern, characterized by using deep learning technology based on a supervised learning strategy.

13. A computer-readable recording medium having recorded thereon a computer program for performing the method described in Article 1.

Images