Method and device for processing three-dimensional holographic images

By dividing the depth image into sub-images and performing Fourier transform in the sub-images, combining zero-filling and ROI processing, the problem of high computational complexity of computer-generated three-dimensional holograms in the prior art is solved, and more efficient three-dimensional hologram processing is achieved .

CN111722514BActive Publication Date: 2024-12-06SAMSUNG ELECTRONICS CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010171085.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-08-02
Filing Date
2020-03-12
Publication Date
2024-12-06
Estimated Expiration
2040-03-12

AI Technical Summary

Technical Problem

The prior art has high computational complexity when processing computer-generated three-dimensional holograms, making it difficult to effectively reduce computational complexity and improve processing efficiency.

Method used

By dividing the depth image into multiple sub-images and performing Fourier transforms in the corresponding sub-images, an interference pattern of computer-generated hologram (CGH) blocks is generated, combining zero-filling and ROI (region of interest) processing, the Fourier transform is optimized. Operation to reduce the amount of calculation.

Benefits of technology

It significantly reduces the computational complexity of CGH processing, improves processing efficiency, and reduces calculation time, and is suitable for high-resolution three-dimensional hologram generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111722514B_ABST
    Figure CN111722514B_ABST
Patent Text Reader

Abstract

A method for processing a three-dimensional holographic image, comprising: obtaining a depth image from depth data of a three-dimensional object; dividing each depth image into a predetermined number of sub-images; obtaining an interference pattern of a computer generated hologram (CGH) block corresponding to each sub-image by performing Fourier transform to calculate an interference pattern of object data included in each sub-image in a CGH plane; and generating a CGH of the three-dimensional object using the obtained interference pattern of the CGH block.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0094536, filed on August 2, 2019, and Korean Patent Application No. 10-2019-0031465, filed on March 19, 2019, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to a method and apparatus for processing a three-dimensional (3D) holographic image, and more particularly, to a method and apparatus for processing a computer-generated hologram (CGH). Background Art

[0004] A hologram is a 3D spatial representation, and since the image of an object is reproduced in a 3D space by controlling the amplitude and phase of light, it has no field of view limitations and almost no stereo fatigue. Therefore, by using a complex spatial light modulator (SLM) that can simultaneously control the amplitude and phase of light, a device that realizes high-resolution holograms in real time has been developed. A hologram can be displayed in a 3D space by using the interference pattern formed when the object wave and the reference wave are incident on the same position. Recently, computer-generated hologram (CGH) technology has been utilized. CGH technology can provide a hologram on a flat panel display by processing the interference pattern used to reproduce a holographic video. A method for generating a digital hologram, such as CGH technology, generates a hologram by approximating an optical signal and calculating the interference pattern generated by mathematical operations. A method for generating a digital hologram, including representing a completed hologram by calculating the data constituting the 3D object based on the fact that the 3D object includes a collection of various data (such as 3D points, polygons, or depth data). Summary of the invention

[0005] A method and apparatus for processing a three-dimensional (3D) holographic image are provided. However, the technical objectives of the present disclosure are not limited thereto, and other technical objectives can be inferred from the following embodiments.

[0006] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of embodiments of the disclosure.

[0007] According to aspects of the present disclosure, a method for processing a three-dimensional holographic image includes: obtaining multiple depth images of a predetermined number of depth layers from depth data of a three-dimensional object, each of the multiple depth images having a predetermined resolution and corresponding to a corresponding depth layer of the predetermined number of depth layers; dividing each of the multiple depth images into a predetermined number of sub-images to be processed by a corresponding processing unit of a plurality of processing units; calculating an interference pattern in a CGH plane corresponding to the corresponding sub-image by performing Fourier transform on object data included in the corresponding sub-image, thereby obtaining multiple interference patterns of multiple CGH blocks in a computer-generated hologram (CGH) plane, wherein each of the multiple CGH blocks corresponds to at least one corresponding sub-image of the predetermined number of sub-images; and generating a CGH of the three-dimensional object using the obtained multiple interference patterns.

[0008] Each sub-image of the predetermined number of sub-images may have an area of ​​at least two pixels.

[0009] The corresponding position of each sub-image of a predetermined number of sub-images in each of the multiple depth images can correspond to the corresponding block position of each CGH block in the CGH plane, and a set of interference patterns calculated from a group of sub-images corresponding to the same block position in each of the multiple depth images can be mapped to a CGH block in the multiple CGH blocks located at the same block position in the CGH plane.

[0010] Generating the CGH may include generating the CGH by merging the obtained plurality of interference patterns to obtain a global interference pattern in the entire region of the CGH plane.

[0011] Obtaining multiple interference patterns may include: for each depth image in a plurality of depth images, determining a sub-image in which object data exists among a predetermined number of sub-images as a region of interest (ROI); and obtaining multiple interference patterns by performing Fourier transform on the determined sub-image and avoiding performing Fourier transform on the remaining sub-images among the predetermined number of sub-images.

[0012] The sub-image determined as the ROI may be determined based on the value of the object data present in the sub-image being greater than a predetermined threshold.

[0013] Obtaining the plurality of interference patterns may further include performing zero padding within a predetermined range of an edge of the sub-image determined as the ROI to form a zero-padded sub-image; and obtaining the plurality of interference patterns by performing Fourier transform on the zero-padded sub-image.

[0014] Obtaining the plurality of interference patterns may further include avoiding performing Fourier transform on a depth image of the plurality of depth images that does not include a sub-image determined as the ROI.

[0015] The Fourier transform may include a double-stage Fourier transform including a primary Fourier transform that calculates the propagation of light waves from the observer's retinal plane to the observer's eye lens plane and a secondary Fourier transform that calculates the propagation of light waves from the eye lens plane to the CGH plane.

[0016] The Fourier transform can calculate the propagation of the light wave from a depth layer among a predetermined number of depth layers to the CGH plane.

[0017] Obtaining multiple interference patterns may include: performing spherical wave convolution on a sub-image corresponding to the ROI among a predetermined number of sub-images; performing zero padding within a certain range on the edge of the sub-image corresponding to the ROI to form a zero-padded sub-image; obtaining an interference pattern by performing Fourier transform on the zero-padded sub-image; and obtaining multiple interference patterns by performing interpolation to match the obtained interference pattern with a sampling grid of a spatial light modulator (SLM).

[0018] The Fourier transform may include a hybrid Fourier transform, which includes: a single-level Fourier transform performed on a first depth image among a plurality of depth images of a first depth layer having a first depth greater than a threshold depth in a predetermined number of depth layers; and a double-level Fourier transform performed on a second depth image among a plurality of depth images of a second depth layer having a second depth less than a threshold depth in a predetermined number of depth layers.

[0019] A non-transitory computer-readable recording medium may have recorded thereon a program for executing the method of the above-mentioned aspect of the present disclosure.

[0020] According to one aspect of the present disclosure, a device for processing a three-dimensional holographic image includes: a memory in which at least one program is stored; and a processor configured to process the three-dimensional holographic image by executing the at least one program, wherein the processor is further configured to: obtain multiple depth images of a predetermined number of depth layers from depth data of a three-dimensional object, each of the multiple depth images having a predetermined resolution and corresponding to a corresponding depth layer of the predetermined number of depth layers, divide each of the multiple depth images into a predetermined number of sub-images to be processed by a corresponding processing unit of a plurality of processing units, obtain multiple interference patterns of multiple CGH blocks in a computer-generated hologram (CGH) plane by performing Fourier transform on object data included in the corresponding sub-image to calculate interference patterns in the CGH plane corresponding to the corresponding sub-image, wherein each of the multiple CGH blocks corresponds to at least one corresponding sub-image of the predetermined number of sub-images, and generate the CGH of the three-dimensional object using the obtained multiple interference patterns.

[0021] Each sub-image of the predetermined number of sub-images may have an area of ​​at least two pixels.

[0022] The corresponding position of each sub-image in a predetermined number of sub-images in each of the multiple depth images can correspond to the corresponding block position of each CGH block in the CGH plane, and a set of interference patterns calculated from a group of sub-images corresponding to the same block position in each of the multiple depth images can be mapped to a CGH block in the multiple CGH blocks located at the same block position in the CGH plane.

[0023] The processor may also be configured to generate the CGH by merging the obtained multiple interference patterns to obtain a global interference pattern in the entire area of ​​the CGH plane.

[0024] The processor can also be configured to: for each depth image among multiple depth images, determine a sub-image in which object data exists among a predetermined number of sub-images as a region of interest (ROI), and obtain multiple interference patterns by performing Fourier transform on the determined sub-image and avoiding performing Fourier transform on the remaining sub-images among the predetermined number of sub-images.

[0025] The sub-image determined as the ROI may be determined based on the value of the object data present in the sub-image being greater than a predetermined threshold.

[0026] The processor may be further configured to: perform zero padding within a predetermined range on an edge of the sub-image determined as the ROI to form a zero-padded sub-image, and obtain a plurality of interference patterns by performing Fourier transform on the zero-padded sub-image.

[0027] The processor may also be configured to avoid performing Fourier transform on a depth image of the plurality of depth images that does not include a sub-image determined to be the ROI.

[0028] The Fourier transform may include a double-stage Fourier transform including a primary Fourier transform that calculates the propagation of light waves from the observer's retinal plane to the observer's eye lens plane and a secondary Fourier transform that calculates the propagation of light waves from the eye lens plane to the CGH plane.

[0029] The Fourier transform can calculate the propagation of the light wave from a depth layer among a predetermined number of depth layers to the CGH plane.

[0030] The processor may also be configured to: perform convolution of a spherical wave on a sub-image corresponding to the ROI among a predetermined number of sub-images, perform zero padding within a predetermined range on an edge of the sub-image corresponding to the ROI to form a zero-padded sub-image, obtain an interference pattern by performing Fourier transform on the zero-padded sub-image, and obtain multiple interference patterns by performing interpolation to match the obtained interference pattern with a sampling grid of a spatial light modulator (SLM).

[0031] The Fourier transform may include a hybrid Fourier transform, which includes: a single-level Fourier transform performed on a first depth image among a plurality of depth images of a first depth layer having a first depth greater than a threshold depth in a predetermined number of depth layers; and a double-level Fourier transform performed on a second depth image among a plurality of depth images of a second depth layer having a second depth less than a threshold depth in a predetermined number of depth layers.

[0032] According to one aspect of the present disclosure, a processor for processing a three-dimensional holographic image includes: a plurality of fast Fourier transform (FFT) cores configured to perform Fourier operations; and a main control unit (MCU) unit), configured to schedule Fourier operations of multiple FFT cores, wherein the MCU is further configured to: obtain multiple depth images of a predetermined number of depth layers from depth data of a three-dimensional object, each of the multiple depth images having a predetermined resolution and corresponding to a corresponding depth layer of the predetermined number of depth layers, divide each of the multiple depth images into a predetermined number of sub-images, and schedule Fourier operations of the multiple FFT cores by assigning an FFT core of the multiple FFT cores to perform Fourier transform on each of the predetermined number of sub-images, wherein each of the multiple FFT cores is configured to perform Fourier transform on object data included in a corresponding sub-image to calculate an interference pattern of a CGH block corresponding to the corresponding sub-image in a CGH plane, and wherein the MCU is further configured to: obtain a plurality of interference patterns by obtaining a corresponding interference pattern corresponding to each of the predetermined number of sub-images based on a result of performing Fourier transform on the multiple FFT cores, and generate a CGH of the three-dimensional object using the multiple interference patterns of the obtained CGH blocks.

[0033] The MCU can also be configured to: for each depth image in the multiple depth images, determine a sub-image in which object data exists in a predetermined number of sub-images as a region of interest (ROI), and assign the corresponding sub-images determined as ROIs in the predetermined number of sub-images to the multiple FFT cores.

[0034] According to one aspect of the present disclosure, a three-dimensional holographic image generating device may include: a memory storing at least one program; and a processor configured to: receive object data about a three-dimensional object, the object data including depth data; divide the object data into multiple depth images according to the depth data, so that each of the multiple depth images has a different depth; for each of the multiple depth images, divide the depth image into multiple sub-images; for each of the multiple sub-images, generate a corresponding interference pattern based on the object data corresponding to the sub-image; combine the generated interference patterns to form a global interference pattern; and generate a computer-generated CGH of the three-dimensional object at the CGH plane based on the global interference pattern.

[0035] The processor may be further configured to perform a first Fourier transform on the object data corresponding to the sub-image based on a distance from the CGH plane to the depth image corresponding to the sub-image being greater than a predetermined distance.

[0036] The processor may be further configured to perform a second Fourier transform on the object data corresponding to the sub-image based on a distance from the CGH plane to the depth image corresponding to the sub-image being less than a predetermined distance.

[0037] The first Fourier transform may include a single-stage Fourier transform.

[0038] The first Fourier transform may include a single-stage Fourier transform, and the second Fourier transform may include a double-stage Fourier transform.

[0039] The processor may be further configured to: for each sub-image of the plurality of sub-images, perform zero padding around the sub-image to form a respective zero-padded sub-image; and generate a respective interference pattern for each zero-padded sub-image.

[0040] The processor can also be configured to: for each of the multiple sub-images, perform spherical wave convolution on the sub-image to form a corresponding convolved sub-image; for each of the multiple convolved sub-images, perform zero padding around the sub-image to form a corresponding zero-padded sub-image; and generate a corresponding interference pattern for each zero-padded sub-image.

[0041] The predetermined distance may be determined based on a sampling interval of a CGH plane and a sampling interval of a CGH block corresponding to a sub-image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0043] Figure 1 is a diagram for describing a principle of processing a computer generated hologram (CGH) according to an embodiment;

[0044] Figure 2A and Figure 2B is a diagram for describing a two-dimensional (2D) image of each depth layer of a three-dimensional (3D) object according to an embodiment, the two-dimensional image being used when a CGH of the 3D object is generated by using a depth map method;

[0045] Figure 3 is a diagram for describing the computational complexity of a CGH process according to an embodiment;

[0046] Figure 4 is a block diagram of a hardware configuration of a 3D holographic image processing apparatus according to an embodiment;

[0047] Figure 5 is a diagram for describing reduction of computational complexity by deep image subdivision according to an embodiment;

[0048] Figure 6is a diagram for describing the computational complexity of CGH processing reduced by m×n subdivision of a depth image according to an embodiment;

[0049] Fig. 7A is a diagram for describing setting a region of interest (ROI) in a sub-image according to an embodiment;

[0050] Figure 7B is a diagram for describing that the number and positions of sub-images determined as ROIs according to an embodiment may be changed according to the number (m×n) of CGH blocks used to divide a depth image;

[0051] Figure 7C is a diagram for describing simulation results for determining an optimal number of blocks according to an embodiment;

[0052] Fig.7D is a block diagram of a detailed hardware configuration of a processor according to an embodiment;

[0053] Fig. 7E is a diagram for describing a method of performing a fast Fourier transform (FFT) operation on a ROI by using an FFT core of an FFT core block according to an embodiment;

[0054] Figure 7F is a diagram for describing performing an FFT operation by assigning an FFT core of an FFT core block to a pixel at a specific position according to another embodiment;

[0055] Fig. 8A and Figure 8B is a diagram for describing zero padding of a sub-image according to an embodiment;

[0056] Fig. 9 is a diagram for comparing and describing interference patterns of CGH blocks generated based on non-zero filling and zero filling according to an embodiment;

[0057] Fig.10 is a diagram for describing simulation results for determining an appropriate range of zero padding according to an embodiment;

[0058] Fig.11 is a diagram for describing a two-stage Fourier transform according to an embodiment;

[0059] Fig.12 is a diagram for describing a hybrid Fourier transform according to an embodiment;

[0060] Fig.13A is a diagram for describing a single-stage Fourier transform according to an embodiment;

[0061] Fig. 13B is a diagram for describing a spherical carrier in a 3D space according to an embodiment;

[0062] Fig. 13C is a diagram for describing simulation results when the influence of omitted spherical carriers is not considered;

[0063] Fig.13D and Fig.13E is a diagram for describing sampling grid matching between a depth layer and a spatial light modulator (SLM) plane (or CGH plane) according to an embodiment;

[0064] Fig.14A and Fig. 14B is a diagram for describing a method of determining a threshold depth when performing a hybrid Fourier transform according to an embodiment;

[0065] Fig.15 is a flowchart of a method of performing CGH processing by dividing a depth image into sub-images according to an embodiment;

[0066] Fig.16A and Fig. 16B is a flowchart of a method for performing CGH processing based on hybrid Fourier transform by dividing a depth image into sub-images according to an embodiment;

[0067] Fig.17 is a flowchart of a method for processing a 3D holographic image according to an embodiment;

[0068] Fig.18 is a diagram for comparing the computational complexity of a case where a non-divided depth image is used (global method) and the computational complexity of a case where a divided depth image is used (local method) according to an embodiment;

[0069] Fig.19 is a graph showing simulation results of 2FFT calculation times of a global method and a local method according to the number of depth layers according to an embodiment;

[0070] Fig. 20 is a graph showing a result of calculating a threshold depth while changing the number of depth layers according to an embodiment; and

[0071] Fig.21 is a graph for comparing FFT calculation times according to the number of depth layers under various Fourier transform schemes according to an embodiment. DETAILED DESCRIPTION

[0072] For the terms used in the embodiments of the present disclosure, current and widely used general terms are selected in consideration of the functions of the present disclosure. However, these terms may be changed according to the intentions of technicians in the relevant fields, the emergence of new technologies, etc. In specific cases, the terms may be arbitrarily selected, and in this case, their definitions will be described in the corresponding disclosed specification. Therefore, the terms used in this specification do not necessarily have to be interpreted as the simple names of the terms, but should be defined based on the meaning of the terms and the overall content of the present disclosure.

[0073] The term "comprise" or "comprising" should not be interpreted or understood to include all of the multiple elements or multiple steps disclosed in this specification without exception. In other words, it should be understood that some of the elements or some of the steps may not be included, or additional elements or steps may also be included.

[0074] The embodiments will be described in detail below with reference to the accompanying drawings. However, the present disclosure can be implemented in various ways and is not limited to the one or more embodiments described herein.

[0075] Figure 1 is a diagram for describing a principle of processing a computer generated hologram (CGH) according to an embodiment. Figure 1 , the principle of processing CGH as 3D image content for holographic display will be briefly described as follows.

[0076] The observer can recognize objects in space through the eyeball. Because the light reflected from the object is refracted through the eye lens on the front of the eyeball and gathered on the retina on the back of the eyeball, the observer can see the object in space. Using this principle, the principle of CGH processing can be implemented.

[0077] When the focus of the observer's eye lens plane W(u,v)14 corresponds to the depth layer L 1 , L M or L N When , we can assume that the depth layer L 1 , L M or L N The image on the retinal plane Q(x 2 ,y 2 )13 has an imaging focus (i.e., is focused on it). Then, the image formed on the retinal plane 13 can be propagated back to the spatial light modulator (SLM) plane (or "CGH plane") P(x 1 ,y 1 )15 is used to calculate the composite light wave field in the SLM plane (or CGH plane) 15, so that the CGH interference pattern used to express the CGH can be obtained.

[0078] According to the generation method, CGH can be divided into a point cloud method, a polygon method, or a depth map (or layer-based) method. In the point cloud method, the surface of the object is represented by multiple points, and the interference pattern at each point is calculated, so the accurate depth can be represented, but the amount of calculation greatly increases according to the number of points. In the polygon method, the surface of the object is expressed as a polygonal mesh, and the interference pattern at each polygonal mesh is calculated, so although the accuracy of the object is reduced, the amount of calculation is small. The depth map method is a layer-based method and a method of generating CGH using a 2D intensity image and depth data, and the amount of calculation can be determined according to the resolution of the image.

[0079] Since the depth map method is a method of calculating CGH after approximating a 3D object as multiple depths and modeling, the efficiency of CGH calculation may be higher than other methods. In addition, CGH can be generated by using only 2D intensity information and depth information (such as a general picture).

[0080] When a CGH is generated according to the depth map method, the Fourier transform operation occupies most of the CGH processing. The Fourier transform in the CGH processing is an operation for obtaining the distribution of a diffraction image obtained by Fresnel diffraction of an image, and corresponds to a generalized Fresnel transform (GFT) or a Fresnel transform. In an embodiment, the Fourier transform may include an inverse Fourier transform, a fast Fourier transform (FFT), a GFT transform, a Fresnel transform, etc., which are operations using Fourier transform. In the FFT operation, the amount of calculation may increase exponentially as the image resolution increases, and therefore, a specific acceleration algorithm for the CGH processing is important.

[0081] According to an embodiment, in order to generate a CGH by using a depth map method, a method of dividing regions of an input image and a method of accelerating a processing algorithm capable of performing an efficient FFT operation on the divided regions (ie, blocks) may be used.

[0082] Figure 2A and Figure 2B is a diagram for describing a 2D image of each depth layer of a 3D object according to an embodiment, the 2D image being used when a CGH of the 3D object is generated by using a depth map method.

[0083] refer to Figure 2A, the 3D object 200 is located in the space between the eye lens plane 14 and the SLM plane (or CGH plane) 15. Hereinafter, the SLM plane 15 will be referred to as the CGH plane. According to the depth map method, the space can be set to be divided into a predetermined number of depth layers. In this case, the number of depth layers can be changed by user settings. For example, the number of depth layers can be 256 or other numbers.

[0084] refer to Figure 2B , the 3D object 200 may be modeled as a depth image 220 corresponding to a predetermined number of depth layers. Figure 2B The illustrated depth image 220 includes object data 221 to 224 of the 3D object 200 at corresponding depths, respectively.

[0085] In the depth image of each depth layer, most areas may have a depth value of 0, and areas with non-zero depth values ​​where object data exists may occupy a relatively small range. Due to the sparsity of the depth image, performing a Fourier transform on the entire area of ​​the depth image may be inefficient. Therefore, it may be desirable to increase the computational efficiency of the CGH processing by taking into account the image sparsity. As described above, areas where object data exists and areas where object data does not exist may be determined based on a depth value of 0, but the present disclosure is not limited thereto, and the reference depth value may be a relatively small random value other than 0.

[0086] Figure 3 is a diagram for describing the computational complexity of a CGH process according to an embodiment.

[0087] refer to Figure 3 , the depth image 310 may be assumed to have a resolution of M×N, with a pixel array consisting of M pixels and N pixels (where M and N are natural numbers). The computational CGH 320 is generated by performing various types of CGH processing (such as Fourier transform) on the depth image 310. In this case, the computational complexity of the CGH processing may be calculated as in Expression 1, and the computational complexity of Expression 1 corresponds to a relative value.

[0088] [Expression 1]

[0089] MNlog(MN)

[0090] That is, the computational complexity of the CGH process may be determined according to the resolution (ie, M×N) of the depth image 310. Therefore, as the image resolution increases, the computational complexity may increase relatively.

[0091] Figure 4 is a block diagram of a hardware configuration of a 3D holographic image processing apparatus 100 according to an embodiment.

[0092] refer to Figure 4 , the 3D holographic image processing apparatus 100 includes a processor 112 and a memory 114. Figure 4 In the 3D holographic image processing device 100 shown in FIG. 1 , only components related to the embodiment are shown. Figure 4 In addition to the components shown in FIG. 1 , the 3D holographic image processing apparatus 100 may further include other components.

[0093] The processor 112 may correspond to a processor provided in various types of computing devices, such as a personal computer (PC), a server device, a television, a mobile device (smartphone, tablet device, etc.), an embedded device, an autonomous vehicle, a wearable device, an augmented reality (AR) device, and an Internet of Things (IoT) device. For example, the processor 112 may correspond to a processor such as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), or a neural processing unit (NPU), but is not limited thereto.

[0094] The processor 112 performs an overall function for controlling the 3D holographic image processing apparatus 100 equipped with the processor 112. The processor 112 may control the 3D holographic image processing apparatus 100 by executing a program stored in the memory 114. For example, when the 3D holographic image processing apparatus 100 is provided in the display device 150, the processor 112 may control image processing performed by the 3D holographic image processing apparatus 100, thereby controlling the display device 150 to display a holographic image.

[0095] The display device 150 may correspond to a device capable of displaying a holographic image in a 3D space based on the CGH generated by the 3D holographic image processing device 100. The display device 150 may include a hardware module for hologram reproduction, such as a spatial light modulator (SLM) 155, and may include various types of display panels, such as LCD and OLED. That is, in addition to the 3D holographic image processing device 100 for generating CGH, the display device 150 may include various hardware modules and hardware configurations for displaying a holographic image. The 3D holographic image processing device 100 may be a separate independent device implemented outside the display device 150. In this case, the display device 150 may receive the CGH data generated by the 3D holographic image processing device 100, and display the holographic image based on the received CGH data. However, the present disclosure is not limited thereto, and the 3D holographic image processing device 100 may be integrated with the display device 150. That is, the implementation of the 3D holographic image processing device 100 and the display device 150 is not limited by any one embodiment.

[0096] The memory 114 is hardware that stores various data processed in the processor 112. For example, the memory 114 may store CGH data that has been processed in the processor 112 and CGH data to be processed. The memory 114 may also store various applications to be driven by the processor 112, such as a holographic playback application, a web browsing application, a game application, and / or a video application.

[0097] The memory 114 may include at least one of a volatile memory or a non-volatile memory. The non-volatile memory includes a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change random access memory (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory includes a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous RAM (SDRAM), a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. In an embodiment, the memory 114 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a compact flash (CF), a secure digital (SD), a micro-SD, a mini-SD, an extreme digital (xD), or a memory stick.

[0098] The processor 112 reads the 2D intensity image and depth data of the 3D object from which the holographic image is to be generated. The processor 112 obtains a plurality of depth images of a predetermined resolution of a predetermined number of depth layers from the depth data of the 3D object. The predetermined resolution can be changed differently according to user settings or input image formats. In an embodiment, it is assumed that the predetermined resolution is Mx N pixels.

[0099] The processor 112 divides each of the depth images into a predetermined number of sub-images corresponding to the processing units for generating the interference pattern of the CGH block (or referred to as "SLM block") in the CGH plane. The predetermined number may be determined and changed by various factors, such as user settings, the resolution of the depth image, and the performance of the 3D holographic image processing device 100.

[0100] The size of one sub-image is larger than the size of one pixel of the resolution of the depth image. That is, the sub-image corresponds to an image of an area where some pixels (i.e., more than one pixel) are gathered together in the depth image. For example, the processor 112 can divide the depth image into m×n sub-images by dividing the entire area of ​​the depth image with an M×N resolution into m×n blocks (where m and n are natural numbers). The size of each of the sub-images may be equal to or different from each other.

[0101] The interference pattern of the CGH block corresponds to a partial CGH interference pattern corresponding to the position of each local area in the CGH plane, and may also be referred to as other terms such as SLM block, local CGH pattern, local SLM pattern, sub-CGH, ​​or sub-SLM. For dividing the depth image into sub-images, the CGH plane can be similarly divided into blocks of a sampling grid. That is, similar to the depth image, the CGH plane can be divided into, for example, m×n sub-CGH areas, and each of the interference patterns of the CGH block can be an interference pattern corresponding to the position of each of the sub-CGH areas on the CGH plane.

[0102] The processor 112 performs a fast Fourier transform FFT for calculating an interference pattern in a CGH plane for object data included in each of the sub-images in individual units having a sub-image size, thereby obtaining an interference pattern of a CGH block corresponding to each of the sub-images.

[0103] As described above, the position of each of the sub-images in the depth image corresponds to the position of each of the interference patterns of the CGH block in the CGH plane. Therefore, the interference patterns calculated from different sub-images in different depth layers located at the same block position can be mapped to the CGH block (or SLM block) located at the same block position in the CGH plane.

[0104] The processor 112 generates the entire CGH of the 3D object by using the obtained interference patterns of the CGH blocks. That is, the processor 112 generates the CGH by combining the obtained interference patterns of the CGH blocks to obtain an interference pattern of the global CGH of the entire area in the CGH plane (ie, a global interference pattern).

[0105] That is, the 3D holographic image processing apparatus 100 performs CGH processing (eg, FFT) by dividing each depth image into sub-images. Figure 5 As shown, as the depth image is divided into sub-images, the computational complexity can be reduced. In addition, the 3D holographic image processing device 100 can set a region of interest (ROI) within the sub-image, or perform a single-stage Fourier transform on the sub-image, so as to perform more efficient CGH processing on the divided sub-images. This will be described in more detail with reference to the accompanying drawings.

[0106] Figure 5 is a diagram for describing that computational complexity is reduced by depth image subdivision according to an embodiment.

[0107] refer to Figure 5, illustrating the computational complexity in the case 501 where the depth image is not divided into sub-images and the computational complexity in the case 502 where the depth image is divided into sub-images. In the case 501 where the depth image having a resolution of M×N is not divided, the computational complexity for generating a CGH process (such as an FFT operation) is as described in Expression 1 above.

[0108] However, in case 502 where a depth image having a resolution of M×N is divided into m×n sub-images and CGH processing is performed on the m×n sub-images to generate m×n CGH blocks (sub-CGHs), the computational complexity can be reduced as in the following Expression 2.

[0109] [Expression 2]

[0110] MN[log(MN)-log(mn)]

[0111] In other words, due to the m×n subdivision, the computational complexity of the CGH process can be relatively reduced compared to the case without subdivision.

[0112] Figure 6 is a diagram for describing that the computational complexity of a CGH process is reduced by m×n subdivision of a depth image according to an embodiment.

[0113] refer to Figure 6 , assuming that the depth image 600 with M×N resolution is divided into m×n CGH blocks 610. For the depth image 600 with M×N resolution, the computational complexity of the CGH process is as described in the above Expression 1. In contrast, the computational complexity calculation process considering m×n subdivision will be described with reference to the following Expression 3.

[0114] [Expression 3]

[0115]

[0116] According to Expression 3, when Fourier transform is performed by localizing the calculation range of M×N resolution into m×n CGH blocks, since the resolution of one block is So the computational complexity of a CGH block can be expressed as When the computational complexity of one CGH block is multiplied by the total number of blocks (m×n) to obtain the computational complexity of the total CGH block, the computational complexity of the total CGH block is MNlog(MN)-MNlog(mn). Therefore, by dividing the depth image 600 into m×n CGH blocks 610 (i.e., sub-images), the computational complexity of CGH processing (such as FFT operations) can be reduced by MNlog(mn). Therefore, as the resolution M×N of the depth image increases and / or the number of CGH blocks (mxn) increases, the computational complexity can be further reduced.

[0117] Fig. 7A is a diagram for describing setting a ROI in a sub-image according to an embodiment.

[0118] refer to Fig. 7A , among the m×n sub-images 701 of the depth image, there may be a sub-image in which the object data 710 exists and a sub-image in which the object data 710 does not exist. In this case, the object data 710 may represent the depth value of the 3D object corresponding thereto in the depth layer. For example, the depth value may be represented as a pixel value. As described above, the sub-image in which the object data 710 exists and the sub-image in which the object data 710 does not exist may be determined based on the depth value 0. However, the present disclosure is not limited thereto, and the reference depth value may be any relatively small value other than 0.

[0119] For each depth image, Figure 4 The processor 112 in the embodiment determines the sub-images in which the object data 710 exists among the m×n sub-images 701 as the ROI 720. That is, the processor 112 may determine the sub-images in which the value of the object data 710 among the m×n sub-images 701 is greater than a predetermined threshold value (for example, 0) as the ROI 720. Fig. 7A In the example of , 14 sub-images at CGH block positions overlapping with the object data 710 among the m×n sub-images 701 may be set as the ROI 720 .

[0120] The processor 112 performs Fourier transform on the sub-image determined as the ROI 720. However, the processor 112 skips the Fourier transform on the remaining sub-images that are not determined as the ROI 720. This is because since there is no object data in the remaining sub-images that are not determined as the ROI 720 (that is, since the depth value or the pixel value is 0), it is not necessary to perform Fourier transform on them. As a result, since the processor 112 only needs to perform Fourier transform on the sub-image of the ROI 720, the amount of calculation can be reduced. The processor 112 obtains the interference pattern of the CGH block corresponding to the ROI sub-image based on the result of Fourier transforming the sub-image determined as the ROI 720.

[0121] The processor 112 may skip Fourier transforming the entire depth image that does not include any sub-image corresponding to the ROI 720 .

[0122] Figure 7B is a diagram for describing that the number and positions of sub-images determined as ROIs according to an embodiment may be changed according to the number (m×n) of CGH blocks used to divide a depth image.

[0123] refer to Figure 7B, an original image having a resolution of about 4000×4000 is divided into 256 depth layers, and object data 731 included in a depth image of a 180th depth layer among the 256 depth layers is shown. The number of pixels occupied by the object data 731 at the full resolution of the depth image is very small.

[0124] Figure 7B ROI 732 when the depth image is divided into, for example, 10×10 blocks and ROI 733 when the depth image is divided into, for example, 40×40 CGH blocks are shown. Figure 4 The processor 112 in each case only needs to perform Fourier transform on the sub-image corresponding to the ROI 732 or 733, so the amount of calculation can be reduced more than when the Fourier transform is performed on the entire depth image. As the number of CGH blocks increases, the range of the entire ROI may decrease, but the number of separate sub-images included in the ROI may increase. Conversely, as the number of CGH blocks decreases, the range of the entire ROI may increase, but the number of separate sub-images included in the ROI may decrease. That is, the amount of calculation of the processor 112 can be adjusted according to the number of CGH blocks.

[0125] Figure 7C is a diagram for describing simulation results for determining an optimal number of blocks according to an embodiment.

[0126] refer to Figure 7C , shows the results of measuring the FFT computation time while increasing the number of CGH blocks under a given set of conditions 740. Figure 7C Under the conditions shown in , when the depth image is divided into 32×32 CGH blocks, the calculation speed is fastest. Theoretically, as the number of CGH blocks increases, the calculation speed also increases, but the calculation load caused by iterative calculation is also generated as the number of CGH blocks increases. Therefore, considering various factors that affect the calculation speed, the appropriate optimal number of CGH blocks can be determined. Figure 7C This is only a simulation result under the given condition 740, and the present disclosure is not limited thereto, and the number of CGH blocks optimized for the use condition may be appropriately determined.

[0127] Fig.7D is a block diagram of a detailed hardware configuration of the processor 112 according to an embodiment.

[0128] Fig.7D The processor 112 corresponds to Figure 4 The processor 112 shown in FIG. Fig.7DAs shown in FIG. 1 , the processor 112 may include a main control unit MCU 1121 and a plurality of FFT cores 1122_1, 1122_2, ..., 1122_j-1 and 1122_j (where j is a natural number). However, Fig.7D The detailed hardware configuration of the processor 112 shown in FIG. 1 is an example, and the processor 112 may further include other hardware configurations, or may be implemented as having the same Fig.7D That is, the detailed hardware configuration of the processor 112 is not limited to Fig.7D The configuration shown in .

[0129] MCU 1121 corresponds to a control unit, which generally controls the CGH processing operation of processor 112. Specifically, MCU 1121 can obtain a plurality of depth images of a predetermined resolution (e.g., M xN) of a predetermined number of depth layers from the depth data of a 3D object, and can divide a given depth image of resolution M x N into a predetermined number (e.g., m x n) of sub-images (i.e., CGH blocks). In addition, MCU 1121 can determine one or more sub-images in which object data exists in the sub-image as ROIs, and can schedule the operations of multiple FFT cores 1122_1, 1122_2, ..., 1122_j-1, and 1122_j by allocating FFT cores to perform Fourier operations on each of the sub-images, so that efficient FFT operations are performed on the ROIs in the divided sub-images (i.e., CGH blocks). That is, MCU 1121 of processor 112 has the function of controlling the CGH processing operation described above and the CGH processing operation to be described below.

[0130] The plurality of FFT cores 1122_1, 1122_2, ..., 1122_j-1, and 1122_j correspond to processing cores that perform Fourier operations (e.g., FFT operations), respectively. Fourier operations, such as FFT operations, correspond to operations that account for most of the CGH processing operations, and the processor 112 can accelerate the CGH processing by parallel processing of the plurality of FFT cores 1122_1, 1122_2, ..., 1122_j-1, and 1122_j, and the plurality of FFT cores 1122_1, 1122_2, ..., 1122_j-1, and 1122_j can be distinguished from the MCU 1121 as an FFT core block 1122 within the processor 112. Each of the plurality of FFT cores 1122_1, 1122_2, ..., 1122_j-1, and 1122_j performs Fourier transform to calculate an interference pattern of object data included in each of the sub-images in the allocated sub-image unit in the CGH plane. In other words, each FFT core can perform Fourier transform on object data included in a single sub-image at a given time. Subsequently, the MCU 1121 obtains an interference pattern of a CGH block corresponding to each of the sub-images based on a result of Fourier transform performed by the FFT cores 1122_1, 1122_2, ..., 1122_j-1, and 1122_j, and generates a CGH of a 3D object by using the interference pattern of the obtained CGH block.

[0131] Each of the plurality of FFT cores 1122_1, 1122_2, ... 1122_j-1, and 1122_j may be implemented with circuit logic in which shift registers, multipliers, and the like for performing FFT operations are combined. The number of FFT cores 1122_1, 1122_2, ... 1122_j-1, and 1122_j in the processor 112 may be variously changed in consideration of various factors such as the performance of the processor 112, the resolution of the display device 150, the resolution of the image, and the like.

[0132] Multiple FFT cores 1122_1, 1122_2, ... 1122_j-1 and 1122_j can perform FFT operations in parallel to perform double-stage Fourier transform (primary Fourier transform and secondary Fourier transform), single-stage Fourier transform or hybrid Fourier transform, which will be described below.

[0133] Fig. 7E is a diagram for describing performing an FFT operation on an ROI by using an FFT core of an FFT core block according to an embodiment.

[0134] As mentioned above Fig.7DAs described above, the MCU 1121 of the processor 112 can determine one or more sub-images containing object data in the sub-image 701 as ROI 720, and can schedule operations of multiple FFT cores 1122_1, 1122_2, ..., 1122_j-1 and 1122_j to perform efficient FFT operations on the ROI 720.

[0135] Specifically, MCU 1121 can assign each sub-image corresponding to ROI 720 to each FFT core, thereby scheduling the operations of FFT cores 1122_1, 1122_2, ..., 1122_j-1 and 1122_j, so that FFT cores 1122_1, 1122_2, ..., 1122_j-1 and 1122_j perform FFT operations on sub-images in ROI 720 in parallel.

[0136] For example, the FFT core 1122_1 may perform an FFT operation on the leftmost sub-image in the first row of the ROI 720, the FFT core 1122_2 may perform an FFT operation on the sub-image located in the second column from the left of the first row of the ROI 720, the FFT core 1122_j-1 may perform an FFT operation on the sub-image located in the second to last column of the last row of the ROI 720, and the FFT core 1122_j may perform an FFT operation on the sub-image located in the last column of the last row of the ROI 720. In this way, the FFT cores 1122_1, 1122_2, ... 1122_j-1, and 1122_j may perform FFT operations on the sub-images of the ROI 720 in parallel, so that efficient FFT operations are performed on the ROI 720.

[0137] For ease of description, Fig. 7E 7 shows that one sub-image in ROI 720 is assigned to one FFT core, but the number of sub-images assigned to one FFT core can be changed in various ways by scheduling MCU 1121. In addition, although Fig. 7E It is not explicitly shown in the figure, but according to the scheduling of MCU 1121, FFT cores 1122_1, 1122_2, ... 1122_j-1 and 1122_j can perform parallel FFT operations on sub-images with the same depth, and can also perform parallel FFT operations on ROIs in depth images with different depths.

[0138] Figure 7F is a diagram for describing performing an FFT operation by assigning an FFT core of an FFT core block to a pixel at a specific position according to another embodiment.

[0139] refer to Figure 7F, each of the FFT cores 1122_1, 1122_2, ..., 1122_j-1 and 1122_j can be mapped to a corresponding pixel array at a certain position in the k-th depth image with a resolution of M×N, and thus can be assigned to perform FFT operations on the mapped pixel array.

[0140] Specifically, for example, when a depth image with a resolution of M×N is grouped into 2×2 pixel arrays, each of the 2×2 pixel arrays may be assigned to one FFT core, and each FFT core may perform an FFT operation on the 2×2 pixel array corresponding thereto.

[0141] All FFT cores in the FFT core block can be assigned to a pixel array in a depth image. However, the present disclosure is not limited to this, and some FFT cores in the FFT core block can be assigned to a pixel array in a depth image of a certain depth, and the remaining FFT cores of the FFT core block can be assigned to a pixel array in a depth image of another depth. That is, the size of the pixel array to be mapped and the number of FFT cores to be mapped to a depth image of one depth can be variously changed according to the number of FFT cores provided in the FFT core block, the processing performance of the FFT core, etc.

[0142] As a result, according to Fig.7D The illustrated embodiment processes FFT operations in parallel using FFT cores 1122_1, 1122_2, ..., 1122_j-1, and 1122_j provided in the processor 112, so that efficient CGH processing can be performed.

[0143] Fig. 8A and Figure 8B is a diagram for describing zero padding of a sub-image according to an embodiment.

[0144] refer to Fig. 8A , when the propagation of a sub-image is calculated by Fourier transform, data may be generated to a wider range than the original sub-image size due to diffraction. That is, when the interference pattern of the CGH block is generated and attached to each other in the original sub-image size, an incorrect diffraction pattern may appear at the sub-image boundary because the data of the portion of the interference pattern that diffracts beyond the edge of the sub-image is lost. Therefore, in order to obtain the correct interference pattern of the CGH block, it is necessary to calculate the propagation using a wider range than the size of the original sub-image.

[0145] therefore, Figure 4 The processor 112 performs zero padding 820 on the edge of the sub-image 810 determined as the ROI within a predetermined range. Then, the processor 112 obtains the interference pattern of the CGH block by performing Fourier transform on the zero-padded sub-image.

[0146] refer to Figure 8B , shows a method for calculating the range of zero filling. When the image is propagated, the range where diffraction occurs can be determined based on the propagation distance, wavelength, and pixel size. For example, the range where diffraction occurs can be determined using Figure 8B The equation shown in is used to calculate L k,min represents the minimum value of the sub-image size according to the distance to the k-th depth layer. Therefore, by k,min Subtracting the original sub-image size, the required zero-filled range can be obtained. When zero-filling is performed on the edge of the sub-image by the diffraction range calculated as described above, and then the propagation is calculated, the propagation can be calculated to include all data diffracted beyond the edge of the original sub-image. Subsequently, when parts of adjacent interference patterns calculated by zero-filling overlap each other, the interference pattern of the correct CGH block can be generated. However, as the zero-filling range increases, the resolution also increases, which may affect the amount of calculation. Therefore, the zero-filling range may be determined to be an appropriate range.

[0147] Fig. 9 is a diagram for comparing and describing interference patterns of CGH blocks generated based on non-zero filling and zero filling according to an embodiment.

[0148] refer to Fig. 9 In the interference pattern 901 based on the CGH block generated without zero filling, the diffraction pattern is missing, so a defect phenomenon occurs. However, on the contrary, in the interference pattern 902 based on the CGH block generated with zero filling, a complete interference pattern can be generated within the zero filling range.

[0149] Fig.10 is a diagram for describing simulation results for determining an appropriate range of zero padding according to an embodiment.

[0150] refer to Fig.10 , the method (Global conv.) that generates CGH without m×n partitioning requires 270.0919 seconds to generate CGH. In addition, the method including m×n partitioning with the zero padding range set to 100% (full) requires 226.4386 seconds to generate CGH, so there is almost no difference in the CGH generation time between the two methods. On the other hand, in the case where the zero padding range is about 30% or more, the deviation is almost no different. Therefore, even if the range of zero padding is set to about 30%, efficient calculation performance can be obtained in terms of calculation time and deviation.

[0151] As reference Figures 5 to 10 As stated, Figure 4The processor 112 in the embodiment may divide the depth image with a resolution of M×N into m×n sub-images (i.e., blocks), and then perform CGH processing (FFT operation, etc.) on each of the sub-images separately. In addition, the processor 112 may select only the ROI sub-image in which the object data exists from the divided sub-images to perform CGH processing. Moreover, the processor 112 may perform CGH processing after first completing zero padding of the ROI sub-image.

[0152] Fig.11 is a diagram for describing a two-stage Fourier transform according to an embodiment.

[0153] refer to Fig.11 The double-stage Fourier transform performs a primary Fourier transform for calculating the propagation of light waves from the observer's retinal plane 13 to the observer's eye lens plane 14 and a secondary Fourier transform for calculating the propagation of light waves from the eye lens plane 14 to the CGH plane 15 for an image focused on the observer's retinal plane 13 (i.e., a depth image).

[0154] As described above, the Fourier transform in the CGH process is an operation for obtaining the distribution of the diffraction image obtained by Fresnel diffraction of the image, and corresponds to GFT or Fresnel transform. In an embodiment, the Fourier transform may include inverse Fourier transform, FFT, GFT, Fresnel transform, etc., which are operations using Fourier transform.

[0155] At different depth layers L N , L m and L 1 The ROIs 1101, 1102 and 1103 determined in the depth image may be processed independently or in parallel.

[0156] Specifically, for the projection onto the retinal plane 13 and the depth layer L N a sub-image corresponding to the ROI 1101 of the depth image (in this case, the sub-image may represent a zero-filled sub-image), Figure 4The processor 112 performs a primary Fourier transform for calculating the propagation of light waves from the retina plane 13 to the eye lens plane 14. The processor 112 applies (e.g., multiplies) a phase profile of the eye lens plane 14 to the result of performing the primary Fourier transform, and then performs a secondary Fourier transform for calculating the propagation of light waves from the eye lens plane 14 to the CGH plane 15. The processor 112 applies (e.g., multiplies) a phase profile of the projection lens to the result of performing the secondary Fourier transform, and then obtains an interference pattern 1111 of a CGH block at a block position on the CGH plane 15 corresponding to the block position of the ROI 1101 of the depth image based on the result of applying the phase profile of the projection lens. The processor 112 performs a phase profile on the CGH plane 15 corresponding to the block position of the ROI 1101 of the depth image. M The sub-image corresponding to the ROI 1102 of the depth image and the depth layer L 1 The same two-stage Fourier transform is performed on the sub-image corresponding to the ROI 1103 of the depth image, thereby obtaining the interference patterns 1112 and 1113 of the CGH block. In this case, as described above, the CGH processing for the ROIs 1101, 1102, and 1103 can be processed independently or in parallel by different processing cores (or processing units) in the processor 112.

[0157] When the interference patterns 1111, 1112, and 1113 of all the CGH blocks in the ROIs 1101, 1102, and 1103 are obtained, the processor 112 merges the obtained interference patterns 1111, 1112, and 1113 of the CGH blocks to obtain the interference pattern of the global CGH (i.e., the global interference pattern) in the entire region of the CGH plane 15, thereby generating the CGH. In this case, in the process of merging the interference patterns 1111, 1112, and 1113 of the CGH blocks, the ranges of the zero filling may overlap with each other.

[0158] Although for ease of description, Fig.11 Only three depth layers L are shown 1 , L m and L N , but the embodiment is not limited thereto, and various numbers of depth layers and various numbers of ROIs may be processed to generate a CGH.

[0159] Fig.12 is a diagram for describing a hybrid Fourier transform according to an embodiment.

[0160] refer to Fig.12 , different from Fig.11 The CGH processing operation using only two-stage Fourier transform, Figure 4 The processor 112 may obtain the interference pattern of the CGH block by using a hybrid Fourier transform.

[0161] The hybrid Fourier transform refers to an operation of selectively performing a double-stage Fourier transform and a single-stage Fourier transform. Specifically, the hybrid Fourier transform refers to performing a single-stage Fourier transform on a depth image (i.e., ROI sub-image) of a depth layer having a depth greater than a threshold depth 1200, and performing a double-stage Fourier transform on a depth image (i.e., ROI sub-image) of a depth layer having a depth less than a threshold depth 1200. In this case, a depth layer greater than the threshold depth 1200 refers to a depth layer at a depth relatively close to the eyeball, and a depth layer less than the threshold depth 1200 refers to a depth layer at a depth relatively close to the CGH plane 15.

[0162] The threshold depth 1200 may be determined and changed by various factors, such as user settings, the number of depth layers, and the performance of the 3D holographic image processing apparatus 100. Optionally, the processor 112 may be based on the following reference Fig.14A The threshold depth condition described above is used to determine the threshold depth 1200.

[0163] A single-stage Fourier transform will be described with reference to the accompanying drawings.

[0164] Fig.13A is a diagram for describing a single-stage Fourier transform according to an embodiment.

[0165] refer to Fig.13A , which is different from the double-stage Fourier transform that performs two Fourier transforms, first from the retinal plane F(x 2 ,y 2 )1301 to the eye lens plane G(u,v)1302, and then from the eye lens plane G(u,v)1302 to the CGH plane H(x 1 ,y 1 )1303, a single-stage Fourier transform refers to an operation of obtaining the interference pattern of the CGH block by only one Fourier transform from the depth layer Lt(ut,vt)1300 directly to the CGH plane 1303.

[0166] Since the CGH interference pattern is generated based on Fresnel transform (Fourier transform), the CGH interference pattern can be calculated only for the portion where the diffraction ranges of the image overlap each other. Figure 3 (or Figure 5 The size of the depth image without m×n subdivision described in the case 501 is large, so the propagation from the depth layer to the CGH plane 1303 can be calculated only when the depth layer is significantly away from the CGH plane 1303 so that the diffraction overlap occurs sufficiently. However, for small-sized sub-images, such as the sub-images of the depth image with m×n subdivision described above (e.g., Figure 5502), the CGH interference pattern can be directly calculated even at a short distance from the CGH plane 1303. In an embodiment, the operation using this principle is defined as a single-stage Fourier transform.

[0167] References Fig.12 The described hybrid Fourier transform refers to an operation using a single-stage Fourier transform and a double-stage Fourier transform based on a threshold depth.

[0168] When the ROI is determined, Figure 4 The processor 112 may perform CGH processing on the sub-image of the ROI by using at least one of CGH processing using double-stage Fourier transform, CGH processing using hybrid Fourier transform, and CGH processing using single-stage Fourier transform.

[0169] Fig. 13B is a diagram for describing a spherical carrier in a 3D space according to an embodiment.

[0170] refer to Fig. 13B , spherical carrier refers to the change in magnitude and position of the phase during the Fourier transform (i.e., FFT) process used to calculate the propagation. Fig. 13B The area FD shown k The information of the propagating spherical carrier is omitted in a single-stage Fourier transform, so the image can be made smaller without being appropriately enlarged according to the distance from the eyeball. Therefore, it is necessary to calculate the convolution about the influence of the spherical carrier.

[0171] Fig. 13C is a diagram for describing simulation results when the influence of omitted spherical carriers is not considered.

[0172] refer to Fig. 13C , shows that without considering the area FD k The result 1330 of generating a CGH interference pattern in the case of a propagating spherical carrier. Since the image is not magnified to the maximum size, the size of the CGH block (SLM block) is smaller than the size of the sub-image. Therefore, even if the interference pattern of the CGH block (interference pattern of the SLM block) is stamped, the interval between the CGH blocks (SLM blocks) will become wider because the size of the CGH block (SLM block) is relatively small compared to the size of the sub-image.

[0173] Since each of the depth images is divided into sub-images, and in the sub-images (or ROI sub-images), the position of the CGH block (SLM block) is determined by marking by the position of the corresponding sub-image rather than by the linear carrier, the phase of the central part of the spherical carrier is cut by the sub-image size and multiplied by all the sub-images. Through this calculation, the size of the corresponding image can be accurately enlarged using only the spherical carrier element in addition to the linear carrier element that generates the image offset.

[0174] Fig.13D and Fig.13E is a diagram for describing sampling grid matching between a depth layer and a CGH plane according to an embodiment.

[0175] refer to Fig.13D , when a CGH block is obtained by propagating an image from a depth layer to a CGH plane through a single-stage Fourier transform, the size of the CGH block increases due to diffraction. In this case, when the CGH block with the increased size is marked on the CGH plane as it is, the CGH block with the increased size does not match the CGH plane, so sampling grid matching between the depth layer and the CGH plane is required.

[0176] refer to Fig.13D , the block size of the CGH plane can be expressed as Expression 4, and the total size of the CGH plane can be expressed as Expression 5. In addition, the total size of the depth image of the depth layer can be expressed as Expression 6.

[0177] [Expression 4]

[0178]

[0179] (L: block size, λ: wavelength, d k : distance between planes, and Δ: pixel size)

[0180] [Expression 5]

[0181] (T x1 , T y1 )=(N x Δx 1 , N y Δy 1 )

[0182] (T: total size, N: resolution)

[0183] [Expression 6]

[0184]

[0185] (F: focal length, and D eye : The distance between the lens and the retina of the eye)

[0186] refer to Fig.13E Since the sampling grid of the generated CGH block has different diffraction overlap ranges depending on the distance to the depth layer and is different for each depth layer and also different from the size of the CGH plane, it is necessary to match the sampling grid with the CGH plane. Therefore, the processor 112 interpolates the generated CGH block to the size (that is, resolution) of the sampling grid suitable for the CGH plane, and then performs the process of cutting the edge according to the size of the CGH plane.

[0187] That is, the processor 112 obtains an interference pattern by performing convolution of a spherical wave on a sub-image corresponding to the ROI, performing zero padding on an edge of the sub-image corresponding to the ROI to a predetermined range, and performing a single-stage Fourier transform on the zero-padding sub-image, and obtains the interference pattern of the CGH block by performing interpolation for matching the interference pattern to a sampling grid of a spatial light modulator (SLM).

[0188] Fig.14A and Fig. 14B is a diagram for describing a method of determining a threshold depth when performing hybrid Fourier transform according to an embodiment.

[0189] refer to Fig.14A , when the diffraction overlap range is larger than the size of the SLM block (CGH block) of the CGH plane, a single-level Fourier transform can be applied. That is, since the diffraction overlap range may only increase when the depth layer is a little away from the CGH plane, a single-level Fourier transform can be applied. Therefore, when Figure 4 When the processor 112 in the embodiment performs the hybrid Fourier transform, the processor 112 performs a double-stage Fourier transform on a depth layer having a short distance between the CGH plane and the depth layer, and performs a single-stage Fourier transform on a depth layer having a long distance between the CGH plane and the depth layer. In order to determine whether the distance between the CGH plane and the depth layer is short or long, a threshold depth may be determined.

[0190] If the diffraction overlap range is larger than the size of the CGH plane, it means that the size of the sampling interval of the SLM block (CGH block) before interpolation is larger than the size of the sampling interval of the CGH plane. Fig.14A From the equation shown in , it can be seen how the size of the sampling interval of the SLM block changes depending on the distance from the CGH plane to the depth layer. Fig. 14B As shown, when the equation is represented as a curve graph, the sampling interval of the SLM block (CGH block) increases as the distance from the depth layer increases, and a single-stage Fourier transform can be performed starting at a point where the sampling interval of the SLM block (CGH block) is greater than the sampling interval of the CGH plane. Therefore, the threshold depth can be based on Fig.14AThe threshold depth condition 1350 shown in FIG.

[0191] Fig.15 is a flowchart of a method for performing CGH processing by dividing a depth image into sub-images according to an embodiment. Fig.15 Since the method of performing the CGH process is related to the embodiment described with reference to the above drawings, even if omitted hereinafter, the above description given with reference to the above drawings can also be applied to the Fig.15 method.

[0192] In operation 1501, Figure 4 The processor 112 in reads the 2D intensity image and depth data of the 3D object from which the holographic image is to be generated.

[0193] In operation 1502 , the processor 112 sets the number of blocks (eg, m×n) into which a depth image of a depth layer obtained from depth data of a 3D object may be divided.

[0194] In operation 1503 , the processor 112 obtains a sub-image corresponding to a block by dividing the depth image into a set number (m×n) of CGH blocks.

[0195] In operation 1504, the processor 112 determines one or more sub-images corresponding to ROIs among the obtained sub-images for each depth image. For example, ROI 1 to ROI k (where k is a natural number) may be determined.

[0196] In operation 1505 , the processor 112 performs zero padding on the sub-image corresponding to the ROI 1 .

[0197] In operation 1506 , the processor 112 performs a double-stage Fourier transform (eg, FFT) on the zero-padded sub-image, and obtains an interference pattern of the CGH block corresponding to the ROI 1 based on a result of performing the double-stage Fourier transform.

[0198] In operation 1507 , the processor 112 labels the obtained interference pattern of the CGH block to the CGH plane.

[0199] In operations 1508 to 1510 , the processor 112 performs the CGH process corresponding to operations 1505 to 1507 on the sub-image corresponding to ROI 2 , thereby marking the interference pattern of the CGH block corresponding to ROI 2 on the CGH plane.

[0200] In operations 1511 to 1513 , the processor 112 performs the CGH process corresponding to operations 1505 to 1507 on the sub-image corresponding to the ROI k, thereby marking an interference pattern of the CGH block corresponding to the ROI k on the CGH plane.

[0201] The processor 112 may perform the processing of operations 1505 to 1507 , the processing of operations 1508 to 1510 , and the processing of operations 1511 to 1513 independently or in parallel.

[0202] In operation 1514 , the processor 112 generates a CGH by merging the interference patterns of the marked CGH blocks and obtaining an interference pattern of the global CGH in the entire region of the CGH plane (ie, a global interference pattern).

[0203] Fig.16A and Fig. 16B is a flowchart of a method for performing CGH processing based on hybrid Fourier transform by dividing a depth image into sub-images according to an embodiment. Fig.16A and Fig. 16B Since the method of performing the CGH process is related to the embodiment described with reference to the above drawings, even if omitted hereinafter, the above description given with reference to the above drawings can also be applied to the Fig.16A and Fig. 16B method.

[0204] In operation 1601, Figure 4 The processor 112 in reads the 2D intensity image and depth data of the 3D object from which the holographic image is to be generated.

[0205] In operation 1602 , the processor 112 sets the number of blocks (eg, m×n) into which a depth image of a depth layer obtained from depth data of a 3D object may be divided.

[0206] In operation 1603 , the processor 112 obtains a sub-image corresponding to a block by dividing the depth image into a set number (m×n) of CGH blocks.

[0207] In operation 1604 , the processor 112 determines, for each depth image, one or more sub-images among the obtained sub-images corresponding to the ROI.

[0208] In operation 1605, the processor 112 determines whether the depth layer of each depth image is far from or close to the CGH plane based on the threshold depth. If it is determined that the depth layer of each depth image is far from the CGH plane, the processor 112 performs operations 1606 and 1611. However, if it is determined that the depth layer of each depth image is close to the CGH plane, the processor 112 performs Fig. 16B Operations 1616 and 1619 are shown in FIG.

[0209] In operation 1606 , the processor 112 performs spherical wave convolution on the sub-image corresponding to the ROI 1 .

[0210] In operation 1607 , the processor 112 performs zero padding on the convolved sub-image.

[0211] In operation 1608, the processor 112 performs a single-stage Fourier transform (eg, FFT) on the zero-padded sub-image.

[0212] In operation 1609 , the processor 112 matches the sampling grid by performing interpolation on the result of performing a single-stage Fourier transform (eg, FFT).

[0213] In operation 1610 , the processor 112 marks the interference pattern of the grid-matched CGH block to the CGH plane.

[0214] In operations 1611 to 1615 , the processor 112 performs CGH processing corresponding to operations 1606 to 1610 on the sub-image corresponding to ROI j, thereby marking an interference pattern of a CGH block corresponding to ROI j on a CGH plane (where j is a natural number).

[0215] In operation 1616 , the processor 112 performs zero padding on the sub-image corresponding to ROI j+1.

[0216] In operation 1617, the processor 112 performs a double-stage Fourier transform (eg, FFT) on the zero-padded sub-image, and obtains an interference pattern of a CGH block corresponding to ROI j+1 based on a result of performing the double-stage Fourier transform.

[0217] In operation 1618 , the processor 112 labels the obtained interference pattern of the CGH block to the CGH plane.

[0218] In operations 1619 to 1621 , the processor 112 performs the CGH process corresponding to operations 1616 to 1618 on the sub-image corresponding to the ROI k, thereby marking the interference pattern of the CGH block corresponding to the ROI k on the CGH plane.

[0219] The processor 112 may perform the processing of operations 1606 to 1610 , the processing of operations 1611 to 1615 , the processing of operations 1616 to 1618 , and the processing of operations 1619 to 1621 independently or in parallel.

[0220] In operation 1622 , the processor 112 merges the interference patterns of the marks of the CGH blocks to obtain an interference pattern of the global CGH in the entire region of the CGH plane (ie, a global interference pattern), thereby generating a CGH.

[0221] Fig.17 is a flow chart of a method for processing a 3D holographic image according to an embodiment. Fig.17Since the 3D holographic image processing method is related to the embodiment described with reference to the above drawings, even if omitted hereinafter, the above description given with reference to the above drawings may also be applied to the Fig.17 method.

[0222] In operation 1701 , the processor 112 obtains a plurality of depth images of a predetermined resolution at a predetermined number of depth layers from depth data of a 3D object.

[0223] In operation 1702 , the processor 112 divides each of the depth images into a predetermined number of sub-images, each sub-image corresponding to a processing unit for generating an interference pattern of a CGH block in a CGH plane.

[0224] In operation 1703, the processor 112 performs Fourier transform for calculating interference patterns of object data included in each of the sub-images in a CGH plane in units of sub-images (ie, individually), thereby obtaining interference patterns of CGH blocks corresponding to each of the sub-images.

[0225] In operation 1704 , the processor 112 generates a CGH of the 3D object by using the obtained interference pattern of the CGH block.

[0226] Fig.18 is a graph for comparing the computational complexity of a case where a non-divided depth image is used (global method) and the computational complexity of a case where a divided depth image is used (local method) according to an embodiment. Fig.18 A graph obtained by simulating the computational complexity of the global method and the computational complexity of the local method while increasing the image resolution is shown. The graph shows the results when the number of depth layers (i.e., depth levels) is set differently. The results are shown when the number of depth layers is 1, 4, 16, and 256. The more the number of depth layers and the higher the image resolution, the higher the computational efficiency of the local method. Fig.18 In the present invention, 2FFT refers to a two-stage Fourier transform, and 1FFT refers to a single-stage Fourier transform.

[0227] Fig.19 2 is a diagram showing simulation results of 2FFT calculation time of a global method and a local method according to the number of depth layers according to an embodiment. Fig.19 ,In the global method, the computation time increases greatly as the number of depth layers increases, but in the case of the local method, relatively large changes do not occur.,In addition, the computation time of the local method is significantly,reduced compared to the global method.

[0228] Fig. 20 is a graph showing the result of calculating the threshold depth while changing the number of depth layers according to an embodiment. Fig. 20, line 2000 represents the threshold depth. As shown in the figure, the threshold depth depends on the number of depth layers. Based on line 2000, the depth layer on which the double-stage Fourier transform is to be performed and the depth layer on which the single-stage Fourier transform is to be performed can be distinguished from each other.

[0229] Fig.21 is a diagram for comparing FFT calculation times according to the number of depth layers under various Fourier transform schemes according to an embodiment. Fig.21 , the local method is much faster than the global method, and the local method that performs mixed Fourier transform ("local mixing") is faster than the local method that only performs two-stage Fourier transform ("local 2FFT"). When the number of depth layers is 256 under the given condition, the local method ("local 2FFT") is about 30 times faster than the global method, and the local method ("local mixing") is about 36 times faster than the global method.

[0230] The above-described embodiments can be implemented in a general-purpose digital computer to be written as a program that can be executed on a computer, and a computer-readable recording medium is used to operate the program. In addition, the data structure used in the embodiment can be recorded on a computer-readable recording medium via various units. Examples of computer-readable recording media include magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.), optical recording media (e.g., CD-ROM or DVD), etc.

[0231] It should be understood that the embodiments described herein should be considered only as illustrative and not restrictive. The description of features or aspects in each embodiment should generally be considered as applicable to other similar features or aspects in other embodiments. Although one or more embodiments have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made without departing from the spirit and scope defined by the appended claims.

Claims

1. A method for processing a three-dimensional holographic image, the method include: obtaining a plurality of depth images of a predetermined number of depth layers from the depth data of the three-dimensional object, each depth image of the plurality of depth images having a predetermined resolution and corresponding to a respective depth layer of the predetermined number of depth layers; dividing each of the plurality of depth images into a predetermined number of sub-images for processing by a corresponding processing unit of the plurality of processing units; obtaining a plurality of interference patterns of a plurality of CGH blocks in a CGH plane by performing Fourier transform on object data included in the corresponding sub-image to calculate interference patterns in a CGH plane of a computer-generated hologram corresponding to the corresponding sub-image, each CGH block in the plurality of CGH blocks corresponding to at least one corresponding sub-image of a predetermined number of sub-images; and Use the obtained multiple interference patterns to generate the CGH of the three-dimensional object, Wherein, obtaining a plurality of interference patterns comprises: For each of the plurality of depth images, determining a sub-image in which object data exists among a predetermined number of sub-images as a region of interest (ROI); and A plurality of interference patterns are obtained by performing Fourier transform on the determined sub-images and refraining from performing Fourier transform on the remaining sub-images among the predetermined number of sub-images.

2. The method according to claim 1, in, Each sub-image of the predetermined number of sub-images has an area of ​​at least two pixels.

3. The method according to claim 2, in, a corresponding position of each of the predetermined number of sub-images in each of the plurality of depth images corresponding to a corresponding block position of each of the plurality of CGH blocks in the CGH plane, and Wherein, a set of interference patterns calculated from a set of sub-images corresponding to the same block position in each of the plurality of depth images are mapped to CGH blocks located at the same block position in the CGH plane among the plurality of CGH blocks.

4. The method according to claim 1, in, Generating the CGH includes generating the CGH by merging the obtained plurality of interference patterns to obtain a global interference pattern in the entire region of the CGH plane.

5. The method according to claim 1, in, The sub-image determined as the ROI is determined based on the value of the object data present in the sub-image being greater than a predetermined threshold.

6. The method according to claim 1, in, Obtaining multiple interference patterns also includes: Performing zero padding within a predetermined range on the edge of the sub-image determined as the ROI to form a zero-padded sub-image; and Multiple interference patterns are obtained by performing Fourier transform on the zero-filled sub-images.

7. The method according to claim 1, in, Obtaining the plurality of interference patterns further includes avoiding performing Fourier transform on a depth image of the plurality of depth images that does not include a sub-image determined as the ROI.

8. The method according to claim 1, in, The Fourier transform comprises a double-stage Fourier transform comprising a primary Fourier transform for calculating the propagation of light waves from the observer's retinal plane to the observer's eye lens plane and a secondary Fourier transform for calculating the propagation of light waves from the eye lens plane to the CGH plane.

9. The method according to claim 1, in, The Fourier transform calculates propagation of light waves from a depth layer in a predetermined number of depth layers to the CGH plane.

10. The method according to claim 9, wherein Obtaining multiple interference patterns include: performing a spherical wave convolution on a sub-image corresponding to the ROI among the predetermined number of sub-images; performing zero padding within a predetermined range on an edge of the sub-image corresponding to the ROI to form a zero-padded sub-image; The interference pattern is obtained by performing Fourier transform on the zero-filled sub-image; as well as Multiple interference patterns are obtained by performing interpolation to match the obtained interference pattern with a sampling grid of a spatial light modulator (SLM).

11. The method according to claim 9, in, The Fourier transform comprises a hybrid Fourier transform, and the hybrid Fourier transform comprises: a single-level Fourier transform of a first depth image of a plurality of depth images of a first depth layer of a predetermined number of depth layers having a first depth greater than a threshold depth; and A two-level Fourier transform is performed on a second depth image of a plurality of depth images of a second depth layer having a second depth less than a threshold depth in a predetermined number of depth layers.

12. A non-transitory computer-readable recording medium having recorded thereon a program for executing the method of claim 1 on a computer.

13. A device for processing a three-dimensional holographic image, the device include: a memory storing at least one program; as well as a processor configured to process a three-dimensional holographic image by executing the at least one program, The processor is further configured to: obtaining a plurality of depth images of a predetermined number of depth layers from the depth data of the three-dimensional object, each of the plurality of depth images having a predetermined resolution and corresponding to a respective depth layer of the predetermined number of depth layers, dividing each of the plurality of depth images into a predetermined number of sub-images for processing by a corresponding processing unit of the plurality of processing units, obtaining a plurality of interference patterns of a plurality of CGH blocks in a CGH plane by performing Fourier transform on object data included in the corresponding sub-image to calculate interference patterns in the CGH plane of the computer generated hologram corresponding to the corresponding sub-image, each CGH block in the plurality of CGH blocks corresponding to at least one corresponding sub-image of a predetermined number of sub-images, and Use the obtained multiple interference patterns to generate the CGH of the three-dimensional object, Wherein, the processor is further configured to: For each of the plurality of depth images, determining a sub-image having object data among a predetermined number of sub-images as a region of interest ROI, and A plurality of interference patterns are obtained by performing Fourier transform on the determined sub-images and refraining from performing Fourier transform on the remaining sub-images among the predetermined number of sub-images.

14. The device according to claim 13, in, Each sub-image of the predetermined number of sub-images has an area of ​​at least two pixels.

15. The device according to claim 14, in, a corresponding position of each of a predetermined number of sub-images in each of the plurality of depth images corresponding to a corresponding block position of each of the plurality of CGH blocks in the CGH plane, and Wherein, a set of interference patterns calculated from a set of sub-images corresponding to the same block position in each of the plurality of depth images are mapped to CGH blocks in the plurality of CGH blocks located at the same block position in the CGH plane.

16. The device according to claim 13, in, The processor is further configured to generate the CGH by merging the obtained plurality of interference patterns to obtain a global interference pattern in the entire region of the CGH plane.

17. The device according to claim 13, in, The sub-image determined as the ROI is determined based on the value of the object data present in the sub-image being greater than a predetermined threshold.

18. The device according to claim 13, in, The processor is further configured to: performing zero padding within a predetermined range on the edge of the sub-image determined as the ROI to form a zero-padded sub-image, and Multiple interference patterns are obtained by performing Fourier transform on the zero-filled sub-images.

19. The device according to claim 13, in, The processor is further configured to avoid performing Fourier transform on a depth image of the plurality of depth images that does not include a sub-image determined to be the ROI.

20. The device according to claim 13, in, The Fourier transform comprises a double-stage Fourier transform comprising a primary Fourier transform for calculating the propagation of light waves from the observer's retinal plane to the observer's eye lens plane and a secondary Fourier transform for calculating the propagation of light waves from the eye lens plane to the CGH plane.

21. The device according to claim 13, in, The Fourier transform calculates propagation of light waves from a depth layer in a predetermined number of depth layers to the CGH plane.

22. The device according to claim 21, in, The processor is further configured to: performing a spherical wave convolution on a sub-image corresponding to the ROI among the predetermined number of sub-images, performing zero padding within a predetermined range on an edge of the sub-image corresponding to the ROI to form a zero-padded sub-image, The interference pattern is obtained by performing a Fourier transform on the zero-padded sub-image, and A plurality of interference patterns are obtained by performing interpolation to match the obtained interference pattern with the sampling grid of the spatial light modulator SLM.

23. The device according to claim 21, in, The Fourier transform comprises a hybrid Fourier transform, and the hybrid Fourier transform comprises: a single-level Fourier transform of a first depth image of a plurality of depth images of a first depth layer of a predetermined number of depth layers having a first depth greater than a threshold depth; and A two-level Fourier transform is performed on a second depth image of a plurality of depth images of a second depth layer having a second depth less than a threshold depth in a predetermined number of depth layers.

24. A processor for processing a three-dimensional holographic image, the processor include: A plurality of fast Fourier transform (FFT) cores configured to perform Fourier operations; as well as The main control unit MCU is configured to schedule the Fourier operations of multiple FFT cores, Wherein, the MCU is further configured as: obtaining a plurality of depth images of a predetermined number of depth layers from the depth data of the three-dimensional object, each of the plurality of depth images having a predetermined resolution and corresponding to a respective depth layer of the predetermined number of depth layers, dividing each of the plurality of depth images into a predetermined number of sub-images, and scheduling Fourier operations of the plurality of FFT cores by assigning an FFT core of the plurality of FFT cores to perform Fourier operations on each of a predetermined number of sub-images, wherein each of the plurality of FFT cores is configured to perform Fourier transform on the object data included in the corresponding sub-image to calculate an interference pattern of a CGH block corresponding to the corresponding sub-image in a CGH plane, and Wherein, the MCU is further configured as: obtaining a plurality of interference patterns by obtaining a corresponding interference pattern corresponding to each of a predetermined number of sub-images based on results of performing Fourier transforms on the plurality of FFT cores, and Use the multiple interference patterns of the obtained CGH blocks to generate a CGH of a three-dimensional object, Wherein, the MCU is further configured as: For each of the plurality of depth images, determining a sub-image having object data among a predetermined number of sub-images as a region of interest ROI, and A plurality of interference patterns are obtained by performing Fourier transform on the determined sub-images and refraining from performing Fourier transform on the remaining sub-images among the predetermined number of sub-images.

25. The processor according to claim 24, in, The MCU is also configured to: For each of the plurality of depth images, determining a sub-image in which object data exists among a predetermined number of sub-images as a region of interest ROI, and Corresponding sub-images determined as ROIs among the predetermined number of sub-images are allocated to the plurality of FFT cores.