Real-time hologram generation method based on Fourier-inspired neural network

Abstract

A real-time hologram generation method based on a Fourier-inspired neural network uses a modified U-Net network model with skip connections in the encoding phase to fuse spatial features from the spatial branch with Fourier features from the Fourier branch, converting the image into a phase-only hologram. In the decoding phase, the reconstruction result is obtained using the angular spectrum method. This method can be used for real-time and high-fidelity phase hologram generation, and the reconstructed image is free of speckle noise.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a real-time high-quality hologram generation method based on an unsupervised Fourier-inspired neural network. Background Art

[0002] Holographic displays offer pixel-level focus control and aberration correction, making them a promising technology for next-generation virtual reality (VR) and augmented reality (AR) optical devices. Computer-generated holography (CGH) is a method for generating holographic patterns by numerically simulating the diffraction and interference of light. Traditional CGH algorithms can be categorized as iterative or non-iterative. To date, these algorithms have always traded off computation time for display image quality. Summary of the Invention

[0003] In response to the deficiency of existing holography, which is that the speed and quality of holography are mutually restricted, resulting in the inability to generate high-fidelity holograms in real time, the present invention proposes a real-time hologram generation method based on a Fourier-inspired neural network, which can be used for real-time and high-fidelity position hologram generation, and the reconstructed image is free of speckle noise.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a real-time hologram generation method based on a Fourier-inspired neural network. In the encoding stage, an improved U-Net network model with jump connections is used to fuse spatial features from the spatial branch and Fourier features from the Fourier branch to convert the image into a pure phase hologram; in the decoding stage, the reconstruction result is obtained through the angular spectrum method.

[0006] The improved U-Net network model includes: two convolutional layers, four Fourier modules and HardTanh activation layer, wherein: the two convolutional layers extract spatial features from the input image, namely local feature maps; each Fourier module converts the local feature maps into frequency domain by using two-dimensional fast Fourier transform (2D-FFT) and then passes two 1 1. The convolution layer extracts the global Fourier features and converts them to the original domain through inverse Fourier transform (IFFT). Then, a new feature map is obtained by adding the local feature map and the global feature map. The HardTanh activation layer constrains the output hologram to Phase of the range.

[0007] The improved U-Net network model is trained using mean square error (MSE), perceptual loss function, and total variation (TV) regularizer as loss functions to avoid checkerboard holograms. Specifically: is the target image, is the pure phase hologram output by the network, For pre-training VGG-19 - the output of the layer, is the corresponding weight, is the penalty coefficient of the TV regularizer.

[0008] The angular spectrum method refers to: the propagation distance of the pure phase hologram The obtained complex amplitude image , ,in: is the pure phase hologram output by the network, is the wavelength, and is the spatial frequency, is the propagation distance between the SLM and the display plane, is the Fourier transform, is the propagation factor, through The absolute value of the reconstructed image .

[0009] The present invention relates to a system for implementing the above-mentioned method, comprising: a training module and a reconstruction module, wherein: the training module learns global information in the frequency domain using Fourier transform on the input image and performs global feature extraction; the reconstruction module performs angular spectrum propagation on the obtained pure phase hologram to obtain a complex amplitude image, and moduloses the complex amplitude image to obtain a reconstructed image, thereby completing the propagation from the hologram to the image.

[0010] Technical Effects

[0011] This invention addresses the issues of poor reconstruction quality and generalization of holograms in existing neural network-based hologram generation methods. Compared to existing technologies, this invention performs a Fourier transform on the input image, converting the spatial domain into the frequency domain to obtain global information. It then learns global features in the frequency domain for use in the generation of pure phase holograms, achieving better visual effects and generalization results through holographic reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flowchart of the present invention;

[0013] Figure 2 Schematic diagram of the real-world optical reconstruction system;

[0014] Figure 3 It is a schematic diagram of the effect of the simulation embodiment;

[0015] Figure 4 This is a schematic diagram of the effect of a real-world embodiment. DETAILED DESCRIPTION

[0016] like Figure 1 As shown, this embodiment relates to a real-time hologram generation method based on a Fourier-inspired neural network, which specifically includes:

[0017] In the first step, a neural network model is used to fuse the spatial features from the spatial branch and the Fourier features from the Fourier branch to convert the image into a pure phase hologram. Specifically:

[0018] 1.1) For each input image, resize it to 1080p by zero padding or cropping;

[0019] 1.2) Pass the image through 3 3 convolution, LeakyReLU activation function and batch normalization operation to extract features;

[0020] 1.3) The features extracted in 1.2) are sequentially passed through four downsampling Fourier modules, including:

[0021] 1.3.1) For the features extracted in 1.2), the spatial domain is passed through 3 3 convolution, LeakyReLU activation function and batch normalization operation to extract local features;

[0022] 1.3.2) For the features extracted in 1.2), global features are extracted by two Fourier sub-block operations, and fast Fourier transform, 1 1 convolution, LeakyReLU activation function, batch normalization operation and inverse fast Fourier transform;

[0023] 1.3.3) Performing a pixel-based addition operation on the local features extracted in 1.3.1) and the global features extracted in 1.3.2) to obtain an output having global features;

[0024] 1.4) For the output with global features obtained in 1.3), finally pass through 3 3. Convolution and HandTanh activation function to obtain pure phase hologram output;

[0025] In the second step, the pure phase hologram obtained in the first step is subjected to angular spectrum propagation to obtain a complex amplitude image, and then the complex amplitude image is modulo-ed to obtain the final reconstructed image.

[0026] In the first step, for the Fourier-inspired network training, the 800-image DIV2K dataset is selected as the training set. The training data is augmented by horizontal flipping and rotation of the images. The batch size used is 1 and the initial learning rate is The network is trained using the AdamW optimizer with a momentum of (0.9, 0.999). The training period is 40. The cosine decay strategy is then used to reduce the learning rate. 5, the weight of the perceptual loss for , TV regularizer The weight is In the second step, for the angular spectrum propagation of the hologram, the wavelength of the light source is 543 nm and the propagation distance is 7 cm. The SLM resolution is set to 1080 1920, the corresponding pixel pitch is 3.74 m. All experiments were performed using NVIDIA GeForce RTX 3090 GPU cards for training and testing.

[0027] Example 1

[0028] This embodiment uses the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) to evaluate the performance of the present invention and other methods. As shown in Table 1, in the numerical simulation reconstruction, the PSNR and SSIM of the present invention are 29.16 dB and 0.935, respectively, which are 4.97 dB and 0.17 higher than the U-Net method. Although the performance is slightly worse than the 500-iteration SGD method, the present invention improves the hologram generation time by 1585 times compared to the SGD method. The visual effect is as follows: Figure 2 The reconstructed image obtained by this method has higher resolution and can see more details, which is better than other reconstruction methods.

[0029] Example 2

[0030] Compared with Example 1, this embodiment performs optical reconstruction in a real-world optical system to obtain a pure phase hologram. Figure 2 The figure shows the schematic diagram of the optical reconstruction device of this embodiment. The SLM in the figure is HOLOEYE GAEA-2-VIS-036 with a resolution of 3840 2160, pixel pitch 3.74 To match the resolution of the SLM, the hologram was resized to 4K using nearest-neighbor interpolation. Other experimental parameters were consistent with the simulation results. An optional 4F system was provided to filter out high-order diffraction artifacts. The final reconstructed image was captured with a Canon EOS M10 camera.

[0031] This example uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to evaluate the performance of the present invention and other methods. Figure 4As shown in the figure. While the GS method provides high contrast, it suffers from the most severe speckle noise. The DPAC method exhibits low contrast because its checkerboard pattern produces ghost copies and causes intensity loss in the central area. SGD achieves a good balance between image contrast and speckle noise, but is still affected by speckle noise in some scenes and cannot generate high-quality holograms in real time. The deep learning method U-Net can only provide low-contrast, low-resolution reconstructed images. The optical reconstructed images obtained using this method have better contrast and less speckle noise, which is better than other reconstruction methods.

[0032] The following prior art is used for comparison with the method of the present invention:

[0033] GS: "Gerchberg R W. A practical algorithm for the determination of plane from image and diffraction pictures[J]. Optik, 1972, 35(2): 237-246."

[0034] DPAC: "Maimone A, Georgiou A, Kollin J S. Holographic near-eyedisplays for virtual and augmented reality[J]. ACM Transactions on Graphics(Tog), 2017, 36(4): 1-16."

[0035] SGD: "Peng Y, Choi S, Padmanaban N, et al. Neural holography with camera-in-the-loop training[J]. ACM Transactions on Graphics (TOG), 2020, 39(6): 1-14."

[0036] UNet《Wu J, Liu K, Sui X, et al. High-speed computer-generatedholography using an autoencoder-based deep neural network[J]. Optics Letters, 2021, 46(12): 2908-2911.》

[0037] Table 1

[0038] algorithm PSNR(dB) SSIM Time(s) GS 23.41 0.661 3.412 DPAC 26.73 0.852 0.001 SGD 32.03 0.941 23.78 U-Net 24.19 0.765 0.007 The present invention 29.16 0.935 0.015

[0039] Compared with the existing technology, the simulation results of the present method after obtaining the hologram are shown in Table 1 and Figure 3 As shown in the figure, it can be seen that this discovery can achieve higher image resolution and image fidelity than the convolution-based U-Net method. Compared with the iterative method, the generation speed of 1080p hologram only takes 0.015s, which is 1585 times that of the iterative method SGD, truly achieving real-time generation of high-fidelity pure phase holograms. The results of real-world experiments are shown in the figure. Figure 4 As shown, similar to the simulation results, the hologram generated by the present invention can reconstruct a clearer image effect.

[0040] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A real-time hologram generation method based on Fourier-inspired neural network, characterized in that: In the encoding stage, an improved U-Net network model with skip connections is used to fuse the spatial features from the spatial branch and the Fourier features from the Fourier branch to convert the image into a pure phase hologram. In the decoding stage, the reconstruction result is obtained by the angular spectrum method; The improved U-Net network model includes: two convolutional layers, four Fourier modules and HardTanh activation layer, wherein: the two convolutional layers extract spatial features from the input image, namely local feature maps; each Fourier module converts the local feature maps into frequency domain by using two-dimensional fast Fourier transform (2D-FFT) and then passes two 1 1. The convolution layer extracts the global Fourier features and converts them to the original domain through inverse Fourier transform (IFFT). Then, a new feature map is obtained by adding the local feature map and the global feature map. The HardTanh activation layer constrains the output hologram to Phase of the range.

2. The real-time hologram generation method based on Fourier-inspired neural network according to claim 1 is characterized in that: The improved U-Net network model is trained using mean square error, perceptual loss function and total variation regularizer as loss functions to avoid checkerboard holograms, specifically: is the target image, is the pure phase hologram output by the network, For pre-training VGG-19 The output of the layer, is the corresponding weight, is the penalty coefficient of the total variation regularizer; The angular spectrum method refers to: the propagation distance of the pure phase hologram The obtained complex amplitude image , ,in: is the pure phase hologram output by the network, is the wavelength, and is the spatial frequency, is the propagation distance between the SLM and the display plane, is the Fourier transform, is the propagation factor, through The absolute value of the reconstructed image .

3. The real-time hologram generation method based on Fourier-inspired neural network according to claim 1 or 2 is characterized in that: include: In the first step, a neural network model is used to fuse the spatial features from the spatial branch and the Fourier features from the Fourier branch to convert the image into a pure phase hologram. Specifically: 1.1) For each input image, resize it to 1080p by zero padding or cropping; 1.2) Pass the image through 3 3 convolution, LeakyReLU activation function and batch normalization operation to extract features; 1.3) The features extracted in 1.2) are sequentially passed through four downsampling Fourier modules, including: 1.3.1) For the features extracted in 1.2), the spatial domain is passed through 3 3 convolution, LeakyReLU activation function and batch normalization operation to extract local features; 1.3.2) For the features extracted in 1.2), global features are extracted by two Fourier sub-block operations, and fast Fourier transform, 1 1 convolution, LeakyReLU activation function, batch normalization operation and inverse fast Fourier transform; 1.3.3) Performing a pixel-based addition operation on the local features extracted in 1.3.1) and the global features extracted in 1.3.2) to obtain an output having global features; 1.4) For the output with global features obtained in 1.3), finally pass through 3 3. Convolution and HandTanh activation function to obtain pure phase hologram output; In the second step, the pure phase hologram obtained in the first step is subjected to angular spectrum propagation to obtain a complex amplitude image, and then the complex amplitude image is modulo-ed to obtain the final reconstructed image.

4. A real-time hologram generation system based on a Fourier-inspired neural network for implementing the method of any one of claims 1 to 3, characterized in that: include: The training module and the reconstruction module include: the training module learns global information in the frequency domain by using Fourier transform on the input image and performs global feature extraction; The reconstruction module performs angular spectrum propagation on the obtained pure phase hologram to obtain a complex amplitude image, and modulosizes the complex amplitude image to obtain a reconstructed image, thereby completing the propagation from the hologram to the image.

Images