Multi-depth 3D computer-generated hologram synthesis method based on interlayer edge processing

Abstract

A multi-depth 3D computer-generated hologram synthesis method based on interlayer edge processing comprises the following steps: for a slice obtained based on depth layering, firstly, carrying out information supplement on a shielded edge part of a picture slice, carrying out smooth supplement processing on a phase shielded part, then combining processed amplitudes and phases into a complex amplitude field, and propagating by using an angular spectrum method; and similar processing is carried out when reaching the next layer until all depth layers are processed to obtain a complex amplitude hologram. According to the method, a clear three-dimensional scene with a correct shielding relationship can be reconstructed, the generated hologram has a clear front-back relationship, and the edges of a focusing surface and a non-focusing surface are clearer and closer to a real scene; according to the method, the interlayer edge artifacts of the hologram are greatly reduced, the non-focusing surface of the generated hologram has a natural optical diffraction phenomenon, and the method is expected to be practically applied to holographic display, VA / AR and other scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer-generated holography (CGH), and in particular relates to a multi-depth 3D computer-generated hologram synthesis method based on inter-layer edge processing. Background Art

[0002] Holography can encode the spatial information of a three-dimensional scene into a hologram. By diffracting it, it can reconstruct the original scene and provide all the natural visual depth cues required by the human visual system, including occlusion, eye accommodation, and stereoscopic vision. However, it has always been a challenge to reconstruct large and complex three-dimensional scenes, especially virtual three-dimensional scenes, using optical holography due to the limitations of experimental device complexity and performance flexibility. Since Dennis Gabor invented the holographic principle in the late 1940s, holographic technology has made great progress. With the emergence of stable laser sources, computers, and spatial light modulators (Spatial Light Modulator, SLM), it has gradually become possible to generate high-quality phase holograms (Phase-only Hologram, POH) based on computers. In essence, computational holography is to simulate optical diffraction propagation by using computers to establish an accurate propagation model, and then establish the connection between the phase hologram and the target real light field through the diffraction propagation model, and finally solve the target phase map in reverse. Computational holography can provide multiple layers of accurate depth information for the generated images. Since it simulates optical diffraction propagation, it is less likely to cause visual fatigue than other display technologies, and the images have a stronger sense of reality. Therefore, computational holography has broad application scenarios in the display field.

[0003] The method of generating computer-generated holograms based on deep layered slicing uses the depth map to process the RGB image in layers, then propagates the farthest image layer to the previous layer and uses the mask to achieve the occlusion effect, and finally obtains a 3D hologram. However, in real scenes, light propagation is not point-to-point propagation, but point-to-surface diffraction propagation. There is a lack of optical diffraction supplementary effect of the occluded part of the edge of the previous layer of image. Due to the lack of information about the occluded object, the boundary between slices is equivalent to the edge of the light field. This causes the hologram obtained by the above method to produce coherent superposition at the edge of the current layer, resulting in black stripes at the edge of the diffraction field, and the occlusion relationship between layers cannot be properly handled. Summary of the invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a multi-depth 3D computational hologram synthesis method based on inter-layer edge processing, which can synthesize multi-depth 3D computational holograms with low computational cost, have a clear three-dimensional scene with correct occlusion relationship, and greatly reduce the edge artifacts between hologram layers.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A multi-depth 3D computer-generated hologram synthesis method based on inter-layer edge processing comprises the following steps:

[0007] Step 1: First, load the RGB-D image, generate five binary masks and five binary inverse masks with opposite binary values ​​according to the depth range of the depth map, and use the masks to divide the image into five layers;

[0008] Step 2: Supplement the information of the edge of the slice corresponding to the deepest depth, and then combine the supplemented slice with the initial zero phase into a complex amplitude field;

[0009] Step 3: Use the angular spectrum method to propagate the complex amplitude field to the next depth plane, and the obtained complex amplitude field is also obtained;

[0010] Step 4: Use an inverse mask to mask the slice position corresponding to the depth of the layer of complex amplitude field obtained by propagation and replace it with the slice of the depth layer;

[0011] Step 5: supplement the information of the obscured edge part of the amplitude in the complex amplitude obtained in step 4, and perform smoothing and supplementation processing on the obscured part of the phase, and then recombine the processed amplitude and phase into a complex amplitude;

[0012] Step 6: Repeat steps 3, 4 and 5 until step 5 is completed on the last depth plane to obtain the final complex amplitude image and complete the synthesis of the hologram.

[0013] Furthermore, in the step 1, the RGB-D image is two images: one is a common RGB three-channel color image, and the other is a Depth image. The Depth image is similar to a grayscale image, and each pixel value thereof is the relative distance between the sensor and the object.

[0014] Furthermore, in step 2, the complex amplitude field refers to an image composed of complex numbers, which can be split into an amplitude image and a phase image through transformation. The complex amplitude field is usually used for optical diffraction calculation.

[0015] Furthermore, in step 1, the binary mask is a binary four-dimensional tensor with the same size as the input image tensor, which is generated by dividing the normalized depth map into intervals at equal intervals according to depth values. The binary inverse mask is numerically opposite to the binary mask.

[0016] In the step five, the information of the obscured edge portion of the amplitude is supplemented by using the information of the pixels around the edge to supplement the edge area of ​​the obscured portion, so that a part of the supplemented image edge is extended to compensate for the optical diffraction effect of the obscured portion of the image edge, and the supplemented image edge extension is as similar as possible to the existing edge information.

[0017] In the step five, the phase-occluded part is smoothed and supplemented by first using the information of the pixels around the edge to supplement the edge area of ​​the occluded part, and then using the information of the supplemented part to smoothly fill all the internal occluded phases, so that the obtained phase map ensures that the phase of the slice area is the same and the slice phase edge is smoothly in contact, eliminating the edge phase mutation of the phase map as much as possible.

[0018] The technical concept of the present invention is as follows: for slices obtained based on depth stratification, firstly, the information of the obscured edge part of the image slice is supplemented, and the phase obscured part is smoothed and supplemented, and then the processed amplitude and phase are combined into a complex amplitude field and propagated using the angular spectrum method. Similar processing is performed when it reaches the next layer until all depth layers are processed to obtain a complex amplitude hologram. Compared with the original method, this method can reconstruct a clear three-dimensional scene with a correct occlusion relationship by supplementing the edge information between layers and smoothly filling the phase information to generate a multi-depth 3D hologram. The generated hologram has a clear front and back relationship, and the edges of the focused and non-focused surfaces are clearer and closer to the real scene. In addition, this method greatly reduces the edge artifacts between holographic layers, and the non-focused surface of the generated hologram has a natural optical diffraction phenomenon, which is expected to be practically applied in holographic display, VA / AR and other scenarios.

[0019] The beneficial effects of the present invention are mainly manifested in:

[0020] 1. Low computational cost multi-depth 3D computational hologram synthesis: The present invention uses a multi-depth 3D computational hologram synthesis method based on inter-layer edge processing. Compared with other existing methods, it can generate holograms at a lower computational cost and obtain comparable hologram quality.

[0021] 2. Clear three-dimensional scene with correct occlusion relationship: The present invention compensates for the optical diffraction effect of the occluded part of the image edge by supplementing edge pixels, correctly handles the occlusion effect in the discontinuous edges of the segmented depth map objects, and the generated hologram has a clear front-to-back relationship. The edges of the focused and non-focused surfaces are clearer and closer to the real scene.

[0022] 3. Greatly reduce edge artifacts between hologram layers: The present invention eliminates the edge phase mutation of the phase diagram as much as possible by smoothing and supplementing the phase-blocked part. The complex amplitude hologram obtained by the phase replacement method greatly reduces the edge artifacts between hologram layers, has an effect closer to real optical propagation, and the propagated image is more in line with the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the flow chart of the present invention;

[0024] Figure 2 It is a schematic diagram of generating a binary mask and an inverse mask and corresponding depth layer slices according to the present invention;

[0025] Figure 3 are the generation results of different depths by the present invention and other methods, wherein (a) represents the generation result of the traditional layer-based multi-depth 3D hologram, (b)(c)(d)(e) are the results of amplitude and phase processing by the method of the present invention during the generation process, and are the images obtained by focusing on four different square parts at the bottom, left, right, and top respectively;

[0026] Figure 4 The following is a detailed comparison of the real scene results generated by the present invention and other methods, wherein (a) is the hologram obtained by the traditional method, and (b) is the result obtained by the method of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings.

[0028] Reference Figure 1 to Figure 4 , a multi-depth 3D computer generated hologram synthesis method based on interlayer edge processing, the flow chart of the present invention is as follows Figure 1 As shown, the following steps are included:

[0029] A multi-depth 3D computer-generated hologram synthesis method based on inter-layer edge processing comprises the following steps:

[0030] Step 1: First, load the RGB-D image, generate five binary masks and five binary inverse masks with opposite binary values ​​according to the depth range of the depth map, and use the masks to divide the image into five layers;

[0031] The RGB-D image is two images: one is a common RGB three-channel color image, and the other is a Depth image. The Depth image is similar to a grayscale image, and each pixel value thereof is the relative distance between the sensor and the object.

[0032] The binary mask is a binary four-dimensional tensor with the same size as the input image tensor, which is generated by dividing the normalized depth map into intervals at equal intervals according to the depth values. The binary inverse mask is numerically opposite to the binary mask.

[0033] In step 1, the RGB-D image is first loaded, and five binary masks and five binary inverse masks with opposite binary values ​​are generated according to the depth range of the depth map ( Figure 2 Schematic diagram, M n (x, y) is the normal mask, is the reverse mask), and the image is divided into five layers using the mask. The slicing effect is shown in Figure 2 The formula for the binary mask is:

[0034]

[0035]

[0036] Among them, depth(x, y) is the depth distribution of the 3D scene, x, y are the horizontal and vertical coordinates of the depth image matrix, z n-1 With z n In order to artificially define the boundaries of depth stratification, the entire depth range "0-1" is divided into five intervals to obtain the boundary values ​​of each interval.

[0037] Step 2: Supplement the information of the edge of the slice corresponding to the deepest depth, and then combine the supplemented slice with the initial zero phase into a complex amplitude field;

[0038] The complex amplitude field refers to an image composed of complex numbers, which can be split into an amplitude image and a phase image through transformation. The complex amplitude field is usually used for optical diffraction calculation;

[0039] Step 3: Use the angular spectrum method to propagate the complex amplitude field to the next depth plane, and the obtained complex amplitude field is also obtained;

[0040] The complex amplitude field u0(x, y) is propagated to the next depth plane using the angular spectrum method, and the obtained complex amplitude field is also obtained. The formula of the angular spectrum method is:

[0041] The angular spectrum method (ASM) is used as a diffraction propagation method to calculate the reconstructed amplitude and phase in the target plane.

[0042] The expression of ASM is given by the following equation:

[0043]

[0044] Among them, f ASM (φ) is the propagation operator, i represents the imaginary unit, φ(x, y) is the phase value, u0(x, y) is the complex amplitude field distribution on the holographic plane generated by the incident coherent light source, represents the Fourier transform, is the transfer function in ASM, The expression is written as:

[0045]

[0046] Where λ is the wavelength, f x 、f y are the spatial frequencies in the x and y directions respectively, and z is the distance between the hologram and the target plane.

[0047] Step 4: Use an inverse mask to mask the slice position corresponding to the depth of the layer of complex amplitude field obtained by propagation and replace it with the slice of the depth layer;

[0048] Step 5: supplement the information of the obscured edge part of the amplitude in the complex amplitude obtained in step 4, and perform smoothing and supplementation processing on the obscured part of the phase, and then recombine the processed amplitude and phase into a complex amplitude;

[0049] The information supplement of the amplitude blocked edge portion is performed by using the information of pixels around the edge to supplement the edge area of ​​the blocked portion, so that a portion of the supplemented image edge is extended to compensate for the optical diffraction effect of the blocked portion of the image edge, and the supplemented image edge extension is as similar as possible to the existing edge information.

[0050] The smoothing and supplementing processing of the phase occluded part is firstly to supplement the edge area of ​​the occluded part by using the information of the pixels around the edge, and then to smoothly fill all the internal occluded phases with the information of the supplemented part, so that the obtained phase map not only ensures that the phase of the slice area of ​​the layer is the same, but also ensures that the edge of the slice phase is smoothly in contact, and eliminates the edge phase mutation of the phase map as much as possible.

[0051] The obscured edge of the amplitude in the obtained complex amplitude is supplemented with information, and the obscured part of the phase is smoothed and supplemented, and then the processed amplitude and phase are recombined into a complex amplitude; the expression of the above process is given by the following equation:

[0052]

[0053]

[0054]

[0055] Among them, C′ n yes The complex amplitude field propagated by the angular spectrum method (ASM), C′ n .a is the complex amplitude field C′ n The amplitude component, C′ n.p is the complex amplitude field C′ n The phase component of , Inpaint(·) represents image occluded edge supplementation (IOES), PsInpaint(·) represents phase occlusion smoothing supplementation (POSS), This is the amplitude graph processed by IOES. is the phase image processed by POSS;

[0056] Step 6: Repeat steps 3, 4 and 5 until step 5 is completed on the last depth plane to obtain the final complex amplitude image and complete the synthesis of the hologram.

[0057] In an embodiment of the present invention, using Figure 1 The method shown, Figure 3 Figure 4 The resolution of the original image is set to 1024×1024 and 1920×1080 respectively, and the wavelengths of the three colors are 638nm, 520nm, and 450nm respectively. The RGB image is divided into five layers according to the depth map, and the interval Δd of the five layers is set to 0.4mm. The simulation results are shown in Figure 3 , 4 As shown, Figure 3 (a) is the result of traditional layer-based multi-depth 3D hologram generation. Figure 3 (b)-(e) are the results of amplitude and phase processing during the generation process of the method of the present invention, and the images are obtained by focusing on four different blocks at the bottom, left, right, and top, respectively. Figure 4 (a) is the hologram obtained by the traditional method. Figure 4 (b) is the result obtained by the method of the present invention, Figure 4 Both holograms are focused on the front character model. It can be seen that compared with the traditional method, the present invention can reconstruct a clear three-dimensional scene with a correct occlusion relationship. The generated hologram has a clear front-to-back relationship. The edges of the focused and non-focused surfaces are clearer and closer to the real scene. In addition, the results also show that this method greatly reduces the edge artifacts between hologram layers, and the non-focused surface of the generated hologram has a natural optical diffraction phenomenon.

[0058] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A multi-depth 3D computer generated hologram synthesis method based on inter-layer edge processing, characterized in that: The method comprises the following steps: Step 1: First, load the RGB-D image, generate five binary masks and five binary inverse masks with opposite binary values ​​according to the depth range of the depth map, and use the masks to divide the image into five layers; Step 2: Supplement the information of the edge of the slice corresponding to the deepest depth, and then combine the supplemented slice with the initial zero phase into a complex amplitude field; Step 3: Use the angular spectrum method to propagate the complex amplitude field to the next depth plane, and the obtained complex amplitude field is also obtained; Step 4: Use an inverse mask to mask the slice position corresponding to the depth of the layer of complex amplitude field obtained by propagation and replace it with the slice of the depth layer; Step 5: supplement the information of the obscured edge part of the amplitude in the complex amplitude obtained in step 4, and perform smoothing and supplementation processing on the obscured part of the phase, and then recombine the processed amplitude and phase into a complex amplitude; Step 6: Repeat steps 3, 4 and 5 until step 5 is completed on the last depth plane to obtain the final complex amplitude image and complete the synthesis of the hologram.

2. The multi-depth 3D computer generated hologram synthesis method based on inter-layer edge processing according to claim 1, characterized in that: In the step 1, the RGB-D image is two images: one is a common RGB three-channel color image, and the other is a Depth image, and each pixel value of the Depth image is the relative distance between the sensor and the object.

3. A multi-depth 3D computer generated hologram synthesis method based on interlayer edge processing as claimed in claim 1 or 2, characterized in that: In the step 2, the complex amplitude field refers to an image composed of complex numbers, which can be split into an amplitude image and a phase image through transformation. The complex amplitude field is usually used for optical diffraction calculation.

4. A multi-depth 3D computer generated hologram synthesis method based on inter-layer edge processing as claimed in claim 1 or 2, characterized in that: In step 1, the binary mask is a binary four-dimensional tensor with the same size as the input image tensor, which is generated by dividing the normalized depth map into intervals at equal intervals according to the depth values. The binary inverse mask is numerically opposite to the binary mask.

5. A multi-depth 3D computer generated hologram synthesis method based on inter-layer edge processing as claimed in claim 1 or 2, characterized in that: In the step five, the information of the obscured edge portion of the amplitude is supplemented by using the information of the pixels around the edge to supplement the edge area of ​​the obscured portion, so that a part of the supplemented image edge is extended to compensate for the optical diffraction effect of the obscured portion of the image edge, and the supplemented image edge extension is as similar as possible to the existing edge information.

6. A multi-depth 3D computer generated hologram synthesis method based on inter-layer edge processing as claimed in claim 1 or 2, characterized in that: In the step five, the phase-occluded part is smoothed and supplemented by first using the information of the pixels around the edge to supplement the edge area of ​​the occluded part, and then using the information of the supplemented part to smoothly fill all the internal occluded phases, so that the obtained phase map ensures that the phase of the slice area is the same and the slice phase edge is smoothly in contact, eliminating the edge phase mutation of the phase map as much as possible.

Images