High-fidelity hologram generation method based on focus stack network

Through the method based on the focus stack network, the full focus map and depth map of 3D objects are processed, combined with the angular spectrum diffraction propagation model and Zenik phase, the defocusing inaccuracy problem of holographic 3D display in the prior art is solved, and high insurance is achieved Reconstruction and clarity improvement of true holograms.

CN118963090AActive Publication Date: 2024-11-15BEIHANG UNIV
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202411312167.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-11-15
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

Existing holographic 3D display technology has the problem of inaccurate defocus blur when reconstructing high-fidelity holograms, and existing methods usually only consider that the input of the deep neural network is the full focus map and depth map of 3D objects, and cannot effectively learn accurate. focus and defocus information.

Method used

Using a high-fidelity hologram generation method based on the focus stack network, the full focus map and depth map of 3D objects are processed through the focus stack renderer, and the complex amplitude distribution is output into the focus stack network, combining the angular spectrum diffraction propagation model and Zenik phase Perform encoding, calculate loss functions and optimize the network, and realize the reconstruction of high-fidelity holograms.

Benefits of technology

The accurate reconstruction of high-fidelity holograms of 3D objects is achieved, especially high-fidelity restoration of defocused blur at different depths, improving the clarity of holographic reproduction images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963090A_ABST
    Figure CN118963090A_ABST
Patent Text Reader

Abstract

The invention provides a high-fidelity hologram generation method based on a focus stack network. The method comprises the following six steps of: 1, processing a full-focus image and a depth image of a 3D (three-dimensional) object into a focus stack by using a focus stack renderer; a second step of outputting a complex amplitude distribution of a hologram of a focus stack of the 3D object using a focus stack network; 3, generating a holographic reproductive image by using an angular spectrum diffraction propagation model; 4, calculating a target focusing image, a target defocusing image, a reconstructed focusing image and a reconstructed defocusing image; 5, calculating a loss function, and optimizing the focus stack network and a learnable Zernike phase; the five steps are repeated, after the focus stack network is iteratively trained for a fixed round, training of the focus stack network is stopped, and at the moment, output in the second step is needed complex amplitude distribution; and 6, compensating the complex amplitude distribution of the focus stack by using a Zernike phase, and then generating a high-fidelity hologram of the 3D object by using a double-phase coding method.
Need to check novelty before this filing date? Find Prior Art

Description

1. Technical Field

[0001] The present invention relates to holographic display technology, and more specifically, to a method for generating high-fidelity holograms based on a focal stack network. 2. Background Technology

[0002] Holographic 3D display technology can reconstruct realistic 3D objects, and is therefore regarded as one of the most promising augmented reality and virtual reality display technologies. However, since coherent light illumination is usually used in the reconstruction of holograms, the existing holographic 3D display technology has the problem of inaccurate defocus blur of the holographic reconstruction image, which restricts the development of holographic 3D display technology. In order to realize the calculation of high-fidelity holograms, researchers have proposed hologram calculation methods based on data supervision and model supervision in combination with deep learning technology. Other researchers have proposed hologram training strategies based on hardware-in-the-loop optimization. However, these methods usually only consider the situation where the input of the deep neural network is the full focus map and depth map of the 3D object, resulting in the network being unable to learn accurate focus and defocus information. In addition, some researchers have used incoherent diffraction propagation models to reconstruct multi-layer 3D objects. However, this method usually requires the use of time-division multiplexing technology, which puts forward requirements on the refresh rate of the modulation device. At present, how to calculate high-fidelity holograms of 3D objects and realize accurate reconstruction of the defocus blur of the holographic reconstruction image is still an unresolved problem in the field of holographic 3D display. III. Summary of the invention

[0003] The present invention proposes a method for generating a high-fidelity hologram based on a focal stack network. Figure 1 As shown in FIG. 1 , the method includes six steps: the first step is to process the full focus map and depth map of the 3D object into a focus stack using a focus stack renderer; the second step is to input the focus stack of the 3D object into a focus stack network, and the focus stack network is used to output the complex amplitude distribution of the hologram of the focus stack of the 3D object; the third step is to back-propagate the complex amplitude distribution using an angular spectrum diffraction propagation model i(i=1,2,...,M) to obtain the M-layer holographic reconstruction image; the fourth step, according to the depth map and the number of layers M, M binary masks are obtained, and then the inner product of the M binary masks and the focal stack of the 3D object is calculated to obtain the target focus and defocus images. At the same time, the inner product between the M binary masks and the M-layer holographic reconstruction image is calculated to obtain the reconstructed focus and defocus images; the fifth step, based on the target focus and defocus images obtained in the fourth step and the reconstructed focus and defocus images, the loss function is calculated to optimize the focal stack network and the learnable Zernike phase. Repeat the above five steps. When the focal stack network is iteratively trained for a fixed number of rounds, the training of the focal stack network stops. At this time, the output of step 2 is the complex amplitude distribution of the high-fidelity hologram of the focal stack of the 3D object. The sixth step, the complex amplitude distribution of the high-fidelity hologram of the focal stack is compensated by the Zernike phase, and then the high-fidelity hologram of the 3D object is encoded using the dual-phase encoding method.

[0004] In step 1, the focus stack renderer uses an out-of-focus blur generation technique based on diffusion circles and Gaussian blur kernels. Figure 2 As shown, the focal length of the thin lens is f, the aperture of the thin lens is D, and there is a point light source A and a point light source B on the optical axis of the thin lens. After passing through the thin lens, the light emitted by point A forms an image at point A', and the light emitted by point B forms an image at point B'. Point A' is located in front of the CCD target surface, and point B' is exactly on the CCD target surface. According to the principle of geometric optics, the imaging relationship between point A and point A', and between point B and point B' are:

[0005]

[0006] Among them, u A is the object distance at point A, u B is the object distance at point B, v A is the image distance of point A', v B is the image distance of point B'. According to the relationship of similar triangles, the diffusion radius σ of the diffusion circle is expressed as:

[0007]

[0008] Where F represents the aperture number, F = f / D, || represents the absolute value of the symbol, and p represents the pixel spacing of the CCD target surface. The diffusion radius σ is used as the standard deviation of the Gaussian blur kernel, and the focus stack J is obtained by convolving the full focus image with the Gaussian blur kernel. The focus stack J is expressed as:

[0009]

[0010] Among them, ⊙ represents the convolution operation, I represents the fully focused image of the 3D object, and x and y represent the pixel values ​​in the x and y directions, respectively.

[0011] In step 2, the structure diagram of the focus stack network is shown in the attached figure. Figure 3 As shown. The focal stack network contains three skip connection layers, four downsampling blocks, five receptive field blocks, five upsampling blocks, and a hyperbolic tangent activation function layer. The skip connection layer is used to directly pass the shallow feature information extracted by the focal stack network to the deep layer of the focal stack network. The downsampling block is used to increase the number of channels of the image tensor and reduce the height and width of the image tensor. The receptive field block does not change the number of channels, height, and width of the image tensor. The upsampling block is used to reduce the number of channels of the image tensor and increase the height and width of the image tensor. The hyperbolic tangent activation function layer is used to constrain the pixel values ​​of the output tensor to the range of -π to π.

[0012] As attached Figure 4 As shown in (a), each downsampling block consists of sequential network 1, sequential network 2, sequential network 3, two 1×1 convolutional layers and two pixel decomposition and reorganization layers. Figure 4 As shown in (b), each upsampling block consists of sequential network 4, sequential network 5, sequential network 6, two 1×1 convolutional layers and two pixel reorganization layers. Figure 4 As shown in (c), each receptive field block consists of branch network 1, branch network 2, branch network 3, branch network 4, two 1×1 convolutional layers and a channel connection operation.

[0013] The specific structures of sequential network 1, sequential network 2 and sequential network 3 are shown in the attached figure. Figure 5 As shown in (a)-(c), sequential network 1, sequential network 2 and sequential network 3 contain a 3×3 convolution layer with a step size of 1, a 3×3 convolution layer with a step size of 2 and a parametric linear rectifier function layer. The specific structures of sequential network 4, sequential network 5 and sequential network 6 are shown in the attached figure. Figure 5 As shown in (d)-(f), sequential network 4, sequential network 5 and sequential network 6 contain a 3×3 convolution layer with a step size of 1, a pixel reorganization layer and a parameter linear rectification function layer. The specific structures of branch network 1, branch network 2, branch network 3 and branch network 4 are shown in the attached figure. Figure 5 As shown in (g)-(j), branch network 1, branch network 2, branch network 3 and branch network 4 contain a 1×1 convolution layer with a stride of 1, a 3×3 convolution layer with a stride of 1, a 3×3 convolution layer with a dilation rate of 1 and a stride of 1, a 3×3 convolution layer with a dilation rate of 3 and a stride of 1, a 3×3 convolution layer with a dilation rate of 5 and a stride of 1, and a parametric linear rectification function layer.

[0014] Preferably, the complex amplitude distribution of the hologram of the focal stack of the 3D object is compensated using a Zernike phase consisting of a fifth-order Zernike polynomial, which is expressed as:

[0015]

[0016] Among them, a3~a7 represent the coefficients of Zernike polynomials, ρ represents the polar radius, θ represents the azimuth, Z3(ρ,θ) controls the defocus of the image, Z4(ρ,θ) and Z5(ρ,θ) control the astigmatism of the image in the vertical and horizontal directions, Z6(ρ,θ) and Z7(ρ,θ) control the coma of the image in the vertical and horizontal directions, a3~a7 are learning parameters, and Z3~Z7 are non-differentiable Zernike polynomial bases. After coordinate transformation, the complex amplitude distribution U of the hologram of the focal stack of the compensated 3D object is ’ H It is expressed as:

[0017]

[0018] Among them, j represents an imaginary number, U H Complex amplitude distribution of a hologram representing a focal stack of an uncompensated 3D object.

[0019] Preferably, the loss function is selected by the perceptual loss function Loss PE , multi-scale structural similarity loss function Loss MS , total variance loss function Loss TV , mean square root error loss function Loss MSE And the mean absolute error loss function Loss MAE The composite loss function is formed, and the expression of the composite loss function Loss is:

[0020]

[0021] Among them, ɑ, β, η and ε are the coefficients of the perceptual loss function, multi-scale structural similarity loss function, total variation loss function, mean square root error loss function and mean absolute error loss function, respectively. rf represents the reconstructed fully focused holographic image, I rdf represents the reconstructed defocused holographic image, I tdf Represents the out-of-focus blurred image of the target in the focus stack. IV. Description of the drawings

[0022] Attached Figure 1 It is a flow chart of a high-fidelity hologram generation method based on a focal stack network of the present invention.

[0023] Attached Figure 2 Schematic diagram of thin lens imaging in the focal stack renderer of the present invention.

[0024] Attached Figure 3 Schematic diagram of the overall structure of the focus stacking network of the present invention.

[0025] Attached Figure 4 Schematic diagram of the structure of the components of the focus stack network of the present invention. Figure 4 (a) is a schematic diagram of the structure of the downsampling block. Figure 4 (b) is a schematic diagram of the structure of the upsampling block. Figure 4 (c) is a schematic diagram of the structure of the receptive field block.

[0026] Attached Figure 5 It is a schematic diagram of the structure of the sequential network and the branch network of the present invention. Figure 5 (a) is a schematic diagram of the structure of the sequential network 1. Figure 5 (b) is a schematic diagram of the structure of the sequential network 2. Figure 5 (c) is a schematic diagram of the structure of sequence network 3. Figure 5 (d) is a schematic diagram of the structure of the sequential network 4. Figure 5 (e) is a schematic diagram of the structure of the sequential network 5. Figure 5 (f) is a schematic diagram of the structure of the sequential network 6. Figure 5 (g) is a schematic diagram of the structure of the branch network 1, Figure 5 (h) is a schematic diagram of the structure of the branch network 2. Figure 5 (i) is a schematic diagram of the structure of the branch network 3, Figure 5 (j) is a schematic diagram of the structure of branch network 4.

[0027] Attached Figure 6 Attached is an experimental effect diagram of an embodiment of a high-fidelity hologram generation method based on a focal stacking network of the present invention. Figure 6 (a) and Figure 6 (b) is a focus stack of two focal planes of a 3D object generated using a focus stack renderer. Figure 6 (c) and Figure 6 (d) is the holographic reconstruction effect of a high-fidelity hologram of a 3D object generated using the method of the present invention.

[0028] The diagram numbers in the above-mentioned drawings are:

[0029] (1) Thin lens, (2) CCD target surface, (3) Circle of confusion.

[0030] It should be understood that the above drawings are only schematic and are not drawn to scale. V. Specific implementation methods

[0031] The following is a detailed description of an embodiment of a high-fidelity hologram generation method based on a focal stack network proposed by the present invention, and further describes the present invention. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Personnel skilled in the art may make some non-essential improvements and adjustments to the present invention based on the above invention content, which still fall within the scope of protection of the present invention.

[0032] The embodiment of the present invention is as follows: when a focal stack of a 3D object is generated using a focal stack renderer, the number of focal planes of the focal stack renderer is set to 2, the depth of each focal plane is 0.05m and 0.10m respectively, the size of the Gaussian blur kernel is set to 51, the aperture number F of the focal stack renderer is 0.8, the focal length f of the thin lens is 8mm, the aperture D of the thin focal stack is 10mm, and the pixel pitch p of the CCD target surface is 3.5μm. During the training process of the focal stack network, the laser wavelengths used to train the blue, green and red holograms are 450nm, 520nm and 638nm respectively. The recorded 3D object is a fully focused image and depth map of a "toy on the sofa" with a resolution of 1356×2040. For the Zernike polynomials of the red channel, the values ​​of a3 to a7 are -0.44, -0.53, 0.0, 0.0, and 0.03, respectively; for the Zernike polynomials of the green channel, the values ​​of a3 to a7 are -0.38, -0.25, 0.0, 0.0, and 0.04, respectively; for the Zernike polynomials of the blue channel, the values ​​of a3 to a7 are -0.09, -0.16, 0.04, -0.03, and 0.06, respectively. In the composite loss function, the values ​​of ɑ, β, η, and ε are 0.03, 1, 1×10 -6 and 0.5. In the optical reconstruction experiment, the resolution of the spatial light modulator used was 1080×1920, the pixel pitch was 6.4μm, and the refresh rate was 60Hz. After training the focal stack network, the training loss value of the red channel was 0.022 and the validation loss value was 0.042. The training loss value of the green channel was 0.024 and the validation loss value was 0.034. The training loss value of the blue channel was 0.026 and the validation loss value was 0.03. The resolution of the output hologram is 1072×1920.

[0033] The focus stacks of the two focal planes of the “Toys on the Sofa” generated by the Focus Stack Renderer are shown in the attached figure. Figure 6 (a) and 6(b). In order to verify that the high-fidelity hologram generation method proposed in the present invention can accurately reconstruct the defocus blur of 3D objects at different depths, the output hologram is loaded onto the spatial light modulator and the holographic reconstruction effect of the 3D object is captured after being irradiated by the color laser. The holographic reconstruction image of the "toy on the sofa" when the two focal planes are focused separately is shown in the attached figure. Figure 6 By comparing the input focus stack and the reconstructed holographic reproduction image, it can be found that the high-fidelity hologram generation method based on focus stack proposed in the present invention can restore the out-of-focus blur of 3D objects with high fidelity.

Claims

1. A high-fidelity hologram generation method based on a focal stacking network, characterized in that: The method includes six steps: the first step is to use a focus stack renderer to process the full focus map and depth map of the 3D object into a focus stack; the second step is to input the focus stack of the 3D object into a focus stack network, and the focus stack network is used to output the complex amplitude distribution of the hologram of the focus stack of the 3D object; the third step is to use the angular spectrum diffraction propagation model to reversely propagate the complex amplitude distribution to obtain a holographic reconstruction image of M layers; the fourth step is to obtain M binary masks according to the depth map and the number of layers M, and then calculate the inner product of the M binary masks and the focus stack of the 3D object to obtain the target focus and defocus images, and at the same time, calculate the M binary masks and the M-layer holographic reconstruction In the fifth step, based on the target focus and defocus images obtained in the fourth step and the reconstructed focus and defocus images, the loss function is calculated to optimize the focal stack network and the learnable Zernike phase; the above five steps are repeated. When the focal stack network is iteratively trained for a fixed number of rounds, the training of the focal stack network is stopped. At this time, the output of step two is the complex amplitude distribution of the high-fidelity hologram of the focal stack of the 3D object; in the sixth step, the complex amplitude distribution of the high-fidelity hologram of the focal stack is compensated by the Zernike phase, and the high-fidelity hologram of the 3D object is encoded by the dual-phase encoding method.

2. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, characterized in that: In step 1, the focus stack renderer uses a defocus blur generation technology based on the circle of confusion and the Gaussian blur kernel. The focal length of the thin lens is f, the aperture of the thin lens is D, and there is a point light source A and a point light source B on the optical axis of the thin lens. After passing through the thin lens, the light emitted by point A forms an image at point A', and the light emitted by point B forms an image at point B'; point A' is located in front of the CCD target surface, and point B' is exactly on the CCD target surface; according to the principle of geometric optics, the imaging relationships between point A and point A', and between point B and point B' are: Among them, u A is the object distance at point A, u B is the object distance at point B, v A is the image distance of point A', v B is the image distance of point B'. According to the relationship of similar triangles, the diffusion radius σ of the diffusion circle is expressed as: Where F represents the aperture number, F = f / D, || represents the absolute value of the symbol, and p represents the pixel spacing of the CCD target surface; the diffusion radius σ is used as the standard deviation of the Gaussian blur kernel, and the focal stack J is obtained by convolving the fully focused image with the Gaussian blur kernel: Among them, ⊙ represents the convolution operation, I represents the fully focused image of the 3D object, and x and y represent the pixel values ​​in the x and y directions, respectively.

3. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, characterized in that: In step 2, the focal stack network includes three skip connection layers, four downsampling blocks, five receptive field blocks, five upsampling blocks and a hyperbolic tangent activation function layer; the skip connection layer is used to directly transfer the shallow feature information extracted by the focal stack network to the deep layer of the focal stack network; the downsampling block is used to increase the number of channels of the image tensor and reduce the height and width of the image tensor; the receptive field block does not change the number of channels, height and width of the image tensor; the upsampling block is used to reduce the number of channels of the image tensor and increase the height and width of the image tensor; the hyperbolic tangent activation function layer is used to constrain the pixel value of the output tensor to the range of -π to π; Each downsampling block consists of sequential network 1, sequential network 2, sequential network 3, two 1×1 convolutional layers and two pixel decomposition and reorganization layers; each upsampling block consists of sequential network 4, sequential network 5, sequential network 6, two 1×1 convolutional layers and two pixel reorganization layers; each receptive field block consists of branch network 1, branch network 2, branch network 3, branch network 4, two 1×1 convolutional layers and a channel connection operation; Sequential network 1, sequential network 2 and sequential network 3 contain 3×3 convolution layers with a stride of 1, 3×3 convolution layers with a stride of 2 and parametric linear rectification function layers; sequential network 4, sequential network 5 and sequential network 6 contain 3×3 convolution layers with a stride of 1, pixel reorganization layers and parametric linear rectification function layers; branch network 1, branch network 2, branch network 3 and branch network 4 contain 1×1 convolution layers with a stride of 1, 3×3 convolution layers with a stride of 1, 3×3 convolution layers with a dilation rate of 1 and a stride of 1, 3×3 convolution layers with a dilation rate of 3 and a stride of 1, 3×3 convolution layers with a dilation rate of 5 and a stride of 1 and a parametric linear rectification function layer.

4. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, characterized in that: The complex amplitude distribution of the hologram of the focal stack of the 3D object is compensated using a Zernike phase consisting of a fifth-order Zernike polynomial, which is expressed as: Among them, a3~a7 represent the coefficients of Zernike polynomials, ρ represents the polar radius, θ represents the azimuth, Z3(ρ,θ) controls the defocus of the image, Z4(ρ,θ) and Z5(ρ,θ) control the astigmatism of the image in the vertical and horizontal directions, Z6(ρ,θ) and Z7(ρ,θ) control the coma of the image in the vertical and horizontal directions, a3~a7 are learning parameters, and Z3~Z7 are non-differentiable Zernike polynomial bases; after coordinate transformation, the complex amplitude distribution U' of the hologram of the focal stack of the compensated 3D object H It is expressed as: Among them, j represents an imaginary number, U H Complex amplitude distribution of a hologram representing a focal stack of an uncompensated 3D object.

5. The method for generating a high-fidelity hologram based on a focal stacking network according to claim 1, characterized in that: The loss function is selected by the perceptual loss function Loss PE , multi-scale structural similarity loss function Loss MS , total variance loss function Loss TV , mean square root error loss function Loss MSE And the mean absolute error loss function Loss MAE The composite loss function is formed, and the expression of the composite loss function Loss is: Loss=αLoss PE (I rf ,I)+βLoss MS (I rdf ,I tdf )+ηLoss TV (I rf )+ε[(Loss MSE (I rf ,I)+0.1×Loss MAE (I rf ,I)+Loss MSE (I rdf ,I tdf )+0.1×Loss MAE (I rdf ,I tdf )] Among them, ɑ, β, η and ε are the coefficients of the perceptual loss function, multi-scale structural similarity loss function, total variation loss function, mean square root error loss function and mean absolute error loss function, respectively. rf represents the reconstructed fully focused holographic image, I rdf represents the reconstructed defocused holographic image, I tdf Represents the out-of-focus blurred image of the target in the focus stack.

Citation Information

Patent Citations

  • Method and camera for the real-time acquisition of visual information from three-dimensional scenes

    CN101952855A

  • Method and apparatus to combine biometric sensing and other functionality

    CN102378983A

  • A full-focus scanning imaging method based on Wigner distribution function

    CN109257524A

  • LED array light source, lensless microscope and image processing method

    CN109270670A

  • Light intensity transmission equation-based phase contrast and differential interference contrast micro-imaging method

    CN109581645A