A high-fidelity hologram generation method based on a focus stack network

Abstract

The application provides a high-fidelity hologram generation method based on a focus stack network, which comprises six steps: in the first step, a full-focus image and a depth image of a 3D object are processed into a focus stack by using a focus stack renderer; in the second step, a focus stack network is used to output a complex amplitude distribution of a hologram of the focus stack of the 3D object; in the third step, a holographic reconstruction image is generated by using an angular spectrum diffraction propagation model; in the fourth step, a target in-focus and out-of-focus image and a reconstructed in-focus and out-of-focus image are calculated; in the fifth step, a loss function is calculated, and a focus stack network and a learnable Zernike phase are optimized; the above five steps are repeated, and when the focus stack network is iteratively trained for a fixed number of rounds, the training of the focus stack network is stopped, and the output of the second step is the required complex amplitude distribution; in the sixth step, the complex amplitude distribution of the focus stack is compensated by using the Zernike phase, and then a high-fidelity hologram of the 3D object is generated by using a two-phase encoding method.

Description

I. TECHNICAL FIELD

[0001] The present application relates to holographic display technology, more particularly, the present application relates to a high-fidelity hologram generation method based on a focal stack network. II. BACKGROUND

[0002] Holographic 3D display technology can reconstruct a realistic 3D object, and is therefore considered as one of the most promising augmented reality and virtual reality display technologies. However, due to the use of coherent light illumination in the reconstruction of a hologram, the existing holographic 3D display technology has the problem of inaccurate defocus blur of the holographic reconstruction, which restricts the development of holographic 3D display technology. In order to realize the calculation of high-fidelity holograms, researchers have proposed hologram calculation methods based on data supervision and model supervision combined with deep learning technology. Some researchers have also proposed a hologram training strategy based on hardware-in-the-loop optimization. However, these methods usually only consider the case where the input of the deep neural network is the full-focus image and the depth map of the 3D object, which leads to the network being unable to learn accurate focus and defocus information. In addition, some researchers have used an incoherent diffraction propagation model to reconstruct a multi-layer 3D object, however, this method usually requires the use of time-division multiplexing technology, which puts a demand on the refresh rate of the modulating device. At present, how to calculate the high-fidelity hologram of a 3D object and realize the accurate reconstruction of the defocus blur of the holographic reconstruction is still a difficult problem that has not been solved in the field of holographic 3D display. III. SUMMARY

[0003] The present application proposes a high-fidelity hologram generation method based on a focal stack network. As shown in the accompanying drawings, Figure 1 the method includes six steps: first, using a focal stack renderer to process the full-focus image and the depth map of the 3D object into a focal stack; second, inputting the focal stack of the 3D object into a focal stack network, the focal stack network being used to output the complex amplitude distribution of the hologram of the focal stack of the 3D object; third, using an angular spectrum diffraction propagation model to reversely propagate the complex amplitude distribution to obtain the hologram of the 3D object; fourth, using a focal stack renderer to process the hologram of the 3D object into a full-focus image; fifth, using a focal stack renderer to process the hologram of the 3D object into a depth map; and sixth, using a focal stack renderer to process the hologram of the 3D object into a full-focus image and a depth map. iThe method uses a distance of (i=1,2,...,M) to obtain M layers of holographic reconstruction images. Fourth, based on the depth map and the number of layers M, M binary masks are obtained. The inner products of the M binary masks and the focal stack of the 3D object are then calculated to obtain the target focused and defocused images. Simultaneously, the inner products of the M binary masks and the M layers of holographic reconstruction images are calculated to obtain reconstructed focused and defocused images. Fifth, based on the target focused and defocused images obtained in step 4 and the reconstructed focused and defocused images, a loss function is calculated to optimize the focal stack network and the learnable Zernike phase. These five steps are repeated. After the focal stack network has been iteratively trained for a fixed number of rounds, training of the focal stack network is terminated. The output of step 2 is the complex amplitude distribution of the high-fidelity hologram of the focal stack of the 3D object. Sixth, the complex amplitude distribution of the high-fidelity hologram of the focal stack is compensated using the Zernike phase, and then encoded using a dual-phase encoding method to obtain a high-fidelity hologram of the 3D object.

[0004] In the first step, the focus stack renderer uses a defocus blur generation technique based on the circle of confusion and the Gaussian blur kernel. Figure 2 As shown in the figure, the focal length of the thin lens is f, the aperture of the thin lens is D, and there is a point light source A and a point light source B on the optical axis of the thin lens. After passing through the thin lens, the light emitted by point A forms an image at point A', and the light emitted by point B forms an image at point B'. Point A' is in front of the CCD target surface, while point B' is exactly on the CCD target surface. According to the principles of geometric optics, the imaging relationships between points A and A', and between points B and B' are:

[0005]

[0006] Among them, u A is the object distance at point A, u B is the object distance at point B, v A is the image distance of point A', v B is the image distance of point B'. According to the relationship of similar triangles, the diffusion radius σ of the diffusion circle is expressed as:

[0007]

[0008] Where F represents the aperture number, F = f / D, || represents the absolute value of the symbol, and p represents the pixel pitch of the CCD target surface. The diffusion radius σ is used as the standard deviation of the Gaussian blur kernel, and the focal stack J is obtained by convolving the fully focused image with the Gaussian blur kernel. The focal stack J is expressed as:

[0009]

[0010] Where ⊙ represents the convolution operation, I represents the fully focused image of the 3D object, and x and y represent the pixel values ​​in the x and y directions, respectively.

[0011] In step two, the structural diagram of the focal stack network is shown in Fig. 2. The focal stack network contains three skip-connection layers, four down-sampling blocks, five receptive field blocks, five up-sampling blocks and one hyperbolic tangent activation function layer. The skip-connection layers are used to transfer the shallow feature information extracted by the focal stack network to the deep layers of the focal stack network directly. The down-sampling blocks are used to increase the channel number of the image tensor and reduce the height and width of the image tensor. The receptive field blocks do not change the channel number, height and width of the image tensor. The up-sampling blocks are used to reduce the channel number of the image tensor and increase the height and width of the image tensor. The hyperbolic tangent activation function layer is used to constrain the pixel value of the output tensor in the range of -π~π. Figure 3

[0012] As shown in Fig. 3, each down-sampling block is composed of sequential network 1, sequential network 2, sequential network 3, two 1x1 convolution layers and two pixel un-reconstruction layers. As shown in Fig. 4, each up-sampling block is composed of sequential network 4, sequential network 5, sequential network 6, two 1x1 convolution layers and two pixel reconstruction layers. As shown in Fig. 5, each receptive field block is composed of branch network 1, branch network 2, branch network 3, branch network 4, two 1x1 convolution layers and a channel connection operation. Figure 4 Figure 4 Figure 4

[0013] The specific structures of the sequential network 1, the sequential network 2 and the sequential network 3 are shown in Figs. 3(a)-3(c) respectively. The sequential network 1, the sequential network 2 and the sequential network 3 contain 3x3 convolution layers with a step of 1, 3x3 convolution layers with a step of 2 and parameter linear rectifier function layers. The specific structures of the sequential network 4, the sequential network 5 and the sequential network 6 are shown in Figs. 4(d)-4(f) respectively. The sequential network 4, the sequential network 5 and the sequential network 6 contain 3x3 convolution layers with a step of 1, pixel reconstruction layers and parameter linear rectifier function layers. The specific structures of the branch network 1, the branch network 2, the branch network 3 and the branch network 4 are shown in Figs. 5(g)-5(j) respectively. The branch network 1, the branch network 2, the branch network 3 and the branch network 4 contain 1x1 convolution layers with a step of 1, 3x3 convolution layers with a step of 1, 3x3 convolution layers with a step of 1 and a dilation rate of 1, 3x3 convolution layers with a step of 1 and a dilation rate of 3, 3x3 convolution layers with a step of 1 and a dilation rate of 5 and parameter linear rectifier function layers. Figure 5 Figure 5 Figure 5

[0014] Preferably, the complex amplitude distribution of the hologram of the focal stack of the 3D object is compensated by using a Zernike phase composed of a fifth-order Zernike polynomial, which is expressed as:

[0015] ​​​​​​​

[0016] Among them, a3~a7 represent the coefficients of the Zernike polynomials, ρ represents the polar radius, θ represents the azimuth angle, Z3(ρ,θ) controls the defocus of the image, Z4(ρ,θ) and Z5(ρ,θ) control the astigmatism of the image in the vertical and horizontal directions, Z6(ρ,θ) and Z7(ρ,θ) control the coma of the image in the vertical and horizontal directions, a3~a7 are learning parameters, and Z3~Z7 are the non-differentiable Zernike polynomial basis. After the coordinate transformation, the complex amplitude distribution U of the hologram of the focal stack of the compensated 3D object is ’ H Expressed as:

[0017]

[0018] Among them, j represents an imaginary number, U H Complex amplitude distribution of a hologram representing a focal stack of an uncompensated 3D object.

[0019] Preferably, the loss function is selected by the perceptual loss function Loss PE , multi-scale structural similarity loss function Loss MS , total variance loss function Loss TV , mean square error loss function Loss MSE And the mean absolute error loss function Loss MAE The composite loss function is formed, and the expression of the composite loss function Loss is:

[0020]

[0021] Among them, ɑ, β, η and ε are the coefficients of the perceptual loss function, multi-scale structural similarity loss function, total variation loss function, mean square root error loss function and mean absolute error loss function respectively. rf represents the reconstructed all-focus holographic image, I rdf represents the reconstructed defocused holographic image, I tdf Represents the out-of-focus blurred image of the target in the focus stack. IV. Description of the Figures

[0022] Attachment Figure 1 This is a flow chart of a high-fidelity hologram generation method based on a focal stacking network of the present invention.

[0023] Attachment Figure 2 Schematic diagram of thin lens imaging in the focal stack renderer of the present invention.

[0024] Attachment Figure 3 Schematic diagram of the overall structure of the focus stacking network of the present invention.

[0025] Attachment Figure 4 FIG. 1 is a schematic diagram of the structure of the components of the focus stack network of the present invention. Figure 4 (a) is a schematic diagram of the structure of the downsampling block. Figure 4 (b) is a schematic diagram of the structure of the upsampling block. Figure 4 (c) is a schematic diagram of the structure of the receptive field block.

[0026] Attachment Figure 5 It is a structural diagram of the sequential network and branch network of the present invention. Figure 5 (a) is a schematic diagram of the structure of the sequential network 1. Figure 5 (b) is a schematic diagram of the structure of the sequential network 2. Figure 5 (c) is a schematic diagram of the structure of the sequence network 3. Figure 5 (d) is a schematic diagram of the structure of the sequential network 4. Figure 5 (e) is a schematic diagram of the structure of the sequential network 5. Figure 5 (f) is a schematic diagram of the structure of the sequential network 6, Figure 5 (g) is a schematic diagram of the structure of the branch network 1, Figure 5 (h) is a schematic diagram of the structure of the branch network 2, Figure 5 (i) is a schematic diagram of the structure of the branch network 3, Figure 5 (j) is a schematic structural diagram of the branch network 4.

[0027] Attachment Figure 6 This is an experimental effect diagram of an embodiment of a high-fidelity hologram generation method based on a focal stacking network of the present invention. Figure 6 (a) and attached Figure 6 (b) is a focus stack of two focal planes of a 3D object generated using a focus stack renderer. Figure 6 (c) and attached Figure 6 (d) is the holographic reconstruction effect of a high-fidelity hologram of a 3D object generated using the method of the present invention.

[0028] The diagram numbers in the above drawings are:

[0029] (1) Thin lens, (2) CCD target surface, (3) Circle of confusion.

[0030] It should be understood that the above drawings are merely schematic and not drawn to scale. V. Specific Implementation Methods

[0031] The following detailed description of an embodiment of a high-fidelity hologram generation method based on a focus stack network is provided to further describe the present application. It is necessary to point out here that the following embodiment is only used to further illustrate the present application and cannot be understood as a limitation on the protection scope of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application according to the above description, which still falls within the protection scope of the present application.

[0032] In an embodiment of the present application, when generating a focus stack of a 3D object using a focus stack renderer, the number of focal planes of the focus stack renderer is set to 2, the depth of each focal plane is 0.05 m and 0.10 m respectively, the size of the Gaussian blur kernel is set to 51, the number of apertures F of the focus stack renderer is 0.8, the focal length f of the thin lens is 8 mm, the aperture D of the thin focus stack is 10 mm, and the pixel pitch p of the CCD target surface is 3.5 μm. In the training process of the focus stack network, the laser wavelengths used to train the blue, green and red holograms are 450 nm, 520 nm and 638 nm respectively. The recorded 3D object is a full-focus image and a depth map of a "toy on a sofa" with a resolution of 1356 x 2040. For the Zernike polynomials of the red channel, the values of a3-a7 are -0.44, -0.53, 0.0, 0.0 and 0.03 respectively; for the Zernike polynomials of the green channel, the values of a3-a7 are -0.38, -0.25, 0.0, 0.0 and 0.04 respectively; for the Zernike polynomials of the blue channel, the values of a3-a7 are -0.09, -0.16, 0.04, -0.03 and 0.06 respectively. In the composite loss function, the values of ɑ, β, η and ε are 0.03, 1, 1 x 10 -6 and 0.5 respectively. In the optical reconstruction experiment, the resolution of the spatial light modulator used is 1080 x 1920, the pixel pitch is 6.4 μm, and the refresh rate is 60 Hz. After training the focus stack network, the training loss value of the red channel is 0.022, and the validation loss value is 0.042. The training loss value of the green channel is 0.024, and the validation loss value is 0.034. The training loss value of the blue channel is 0.026, and the validation loss value is 0.03. The resolution of the output hologram is 1072 x 1920.

[0033] The focus stacks of the "toy on a sofa" at two focal planes generated by the focus stack renderer are shown in Figs. 6(a) and 6(b) respectively. In order to verify that the high-fidelity hologram generation method proposed in the present application can accurately reconstruct the out-of-focus blur of a 3D object at different depths, the output hologram is loaded onto a spatial light modulator, and the holographic reconstruction effect of the 3D object is captured after irradiation by a color laser. The holographic reconstruction images of the "toy on a sofa" when focusing on two focal planes are shown in Figs. 7(a) and 7(b) respectively. Figure 6 It can be seen from Figs. 7(a) and 7(b) that the focus stack network can accurately reconstruct the out-of-focus blur of the "toy on a sofa" at different depths, and the holographic reconstruction effect is good.Figure 6 (c) and 6(d) are shown. By comparing the input focal stack and the reconstructed holographic reconstruction, it can be found that the high-fidelity hologram generation method based on focal stack proposed in the present application can restore the out-of-focus blur of the 3D object with high fidelity.

Claims

1. A high-fidelity hologram generation method based on a focal stacking network, characterized in that: The method includes six steps: the first step is to use a focus stack renderer to process the full focus map and depth map of the 3D object into a focus stack; the second step is to input the focus stack of the 3D object into a focus stack network, and the focus stack network is used to output the complex amplitude distribution of the hologram of the focus stack of the 3D object; the third step is to use the angular spectrum diffraction propagation model to reversely propagate the complex amplitude distribution to obtain a holographic reconstruction image of M layers; the fourth step is to obtain M binary masks according to the depth map and the number of layers M, and then calculate the inner product of the M binary masks and the focus stack of the 3D object to obtain the target focus and defocus images, and at the same time, calculate the M binary masks and the M-layer holographic reconstruction In the fifth step, the loss function is calculated based on the target focus and defocus images obtained in the fourth step and the reconstructed focus and defocus images, and the focal stack network and the learnable Zernike phase are optimized; the above five steps are repeated. When the focal stack network is iteratively trained for a fixed number of rounds, the training of the focal stack network is stopped. At this time, the output of step two is the complex amplitude distribution of the high-fidelity hologram of the focal stack of the 3D object; in the sixth step, the complex amplitude distribution of the high-fidelity hologram of the focal stack is compensated using the Zernike phase, and the high-fidelity hologram of the 3D object is encoded using the dual-phase encoding method.

2. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, wherein: In step 1, the focus stack renderer uses a defocus blur generation technique based on the circle of confusion and Gaussian blur kernel. The focal length of the thin lens is f, and the aperture of the thin lens is D. There is a point light source A and a point light source B on the optical axis of the thin lens. After passing through the thin lens, the light emitted by point A forms an image at point A', and the light emitted by point B forms an image at point B'. Point A' is located in front of the CCD target surface, while point B' is exactly on the CCD target surface. According to the principles of geometric optics, the imaging relationships between points A and A', and between points B and B' are: Among them, u A is the object distance at point A, u B is the object distance at point B, v A is the image distance of point A', v B is the image distance of point B'. According to the relationship of similar triangles, the diffusion radius σ of the diffusion circle is expressed as: Where F represents the aperture number, F = f / D, || represents the absolute value of the symbol, and p represents the pixel pitch on the CCD target surface. The diffusion radius σ is used as the standard deviation of the Gaussian blur kernel, and the focal stack J is obtained by convolving the fully focused image with the Gaussian blur kernel: Where ⊙ represents the convolution operation, I represents the fully focused image of the 3D object, and x and y represent the pixel values ​​in the x and y directions, respectively.

3. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, wherein: In step 2, the focal stack network contains three skip connection layers, four downsampling blocks, five receptive field blocks, five upsampling blocks and a hyperbolic tangent activation function layer; the skip connection layer is used to directly transfer the shallow feature information extracted by the focal stack network to the deep layer of the focal stack network; the downsampling block is used to increase the number of channels of the image tensor and reduce the height and width of the image tensor; the receptive field block does not change the number of channels, height and width of the image tensor; the upsampling block is used to reduce the number of channels of the image tensor and increase the height and width of the image tensor; the hyperbolic tangent activation function layer is used to constrain the pixel values ​​of the output tensor to the range of -π to π; Each downsampling block consists of sequential network 1, sequential network 2, sequential network 3, two 1×1 convolutional layers and two pixel decomposition and reconstruction layers; each upsampling block consists of sequential network 4, sequential network 5, sequential network 6, two 1×1 convolutional layers and two pixel reconstruction layers; each receptive field block consists of branch network 1, branch network 2, branch network 3, branch network 4, two 1×1 convolutional layers and a channel connection operation; Sequential network 1, sequential network 2 and sequential network 3 contain a 3×3 convolution layer with a stride of 1, a 3×3 convolution layer with a stride of 2 and a parametric linear rectification function layer; sequential network 4, sequential network 5 and sequential network 6 contain a 3×3 convolution layer with a stride of 1, a pixel reorganization layer and a parametric linear rectification function layer; branch network 1, branch network 2, branch network 3 and branch network 4 contain a 1×1 convolution layer with a stride of 1, a 3×3 convolution layer with a stride of 1, a 3×3 convolution layer with a dilation rate of 1 and a stride of 1, a 3×3 convolution layer with a dilation rate of 3 and a stride of 1, a 3×3 convolution layer with a dilation rate of 5 and a stride of 1 and a parametric linear rectification function layer.

4. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, wherein: The complex amplitude distribution of the hologram of the focal stack of the 3D object is compensated using a Zernike phase consisting of a fifth-order Zernike polynomial, which is expressed as: Among them, a3~a7 represent the coefficients of the Zernike polynomials, ρ represents the polar radius, θ represents the azimuth angle, Z3(ρ,θ) controls the defocus of the image, Z4(ρ,θ) and Z5(ρ,θ) control the astigmatism of the image in the vertical and horizontal directions, Z6(ρ,θ) and Z7(ρ,θ) control the coma of the image in the vertical and horizontal directions, a3~a7 are learning parameters, and Z3~Z7 are the non-differentiable Zernike polynomial basis; after coordinate transformation, the complex amplitude distribution U' of the hologram of the focal stack of the compensated 3D object H Expressed as: Among them, j represents an imaginary number, U H Complex amplitude distribution of a hologram representing a focal stack of an uncompensated 3D object.

5. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, wherein: The loss function is selected by the perceptual loss function Loss PE , multi-scale structural similarity loss function Loss MS , total variance loss function Loss TV , mean square error loss function Loss MSE And the mean absolute error loss function Loss MAE The composite loss function is formed, and the expression of the composite loss function Loss is: Loss=αLoss PE (I rf ,I)+βLoss MS (I rdf ,I tdf )+ηLoss TV (I rf ) +ε[(Loss MSE (I rf ,I)+0.1×Loss MAE (I rf ,I) +Loss MSE (I rdf ,I tdf )+0.1×Loss MAE (I rdf ,I tdf )] Among them, ɑ, β, η and ε are the coefficients of the perceptual loss function, multi-scale structural similarity loss function, total variation loss function, mean square root error loss function and mean absolute error loss function respectively. rf represents the reconstructed all-focus holographic image, I rdf represents the reconstructed defocused holographic image, I tdf Represents the out-of-focus blurred image of the target in the focus stack.

Images