A neural network method, system and device for generating three-dimensional holograms

Abstract

A neural network method, system and device for generating a three-dimensional hologram, the method of which is based on an improved Holo-U2Net network to construct a two-layer nested U-shaped structure, the outer layer U-shaped structure comprising 11 residual U-shaped modules RSU, and the inner layer U-shaped structure residual U-shaped module RSU, for extracting multi-scale features and high-frequency features; the outer layer U-shaped structure allows the network to be deeper and to fuse the feature maps output from the bottleneck layer residual U-shaped module RSU and each residual U-shaped module RSU in the encoding stage; the network extracts multi-scale features within the block and aggregates multi-level features between the blocks, having stronger fitting capability and the ability to generate high-quality holograms; a lightweight Fourier convolution module is implemented in the residual U-shaped module RSU, comprising convolution, batch normalization, a linear rectifier function, and the addition of feature information obtained in the Fourier domain, further improving the ability of the Holo-U2Net network to generate high-quality holograms; the system and device can generate high-quality three-dimensional holograms based on the neural network method for generating a three-dimensional hologram, and have the advantages of high image quality, short time consumption and low memory consumption.

Description

Technical Field

[0001] This invention belongs to the field of hologram generation technology, specifically relating to a neural network method, system, and device for generating three-dimensional holograms. Background Technology

[0002] Common methods for generating 3D object holograms mainly include point-based methods (PBM) [Maimone, A. Georgiou, and JSKollin, “Holographic near-eye displays for virtual and augmented reality,” ACM Trans. on Graph. (Tog) 36, 1–16 (2017).], polygon-based methods, and layer-based methods. In PBM, object points are treated as light sources emitting spherical waves, illuminating the hologram plane. The target hologram is obtained by superimposing the stripe pattern of all object points onto the hologram plane. However, for complex 3D objects or high-resolution holograms, computation time and memory requirements increase significantly. In polygon-based methods, the 3D object is described using polygons, which can be viewed as polygonal apertures. The target hologram is formed by adding diffraction patterns from all polygonal apertures. In layer-based methods, several layers parallel to the hologram plane represent the 3D object, and sub-holograms calculated by Fresnel diffraction from each layer are superimposed to form the target hologram. However, generating complex 3D objects with consistent depth to achieve accurate focus on holograms, especially for close-up display applications, remains challenging.

[0003] In the process of generating holograms using a computer based on the point-source method through physical simulation, three-dimensional objects are modeled as points, each point being treated as a light source emitting spherical waves. These waves illuminate the hologram plane, producing interference fringes, which are then superimposed on the plane to generate a hologram. Therefore, researchers can use rendering, ray tracing, and Fresnel diffraction techniques combined with red-green-blue (RGB) three-channel images and depth maps recording depth information to generate a holographic dataset containing RGB images, depth images, and their corresponding amplitude and phase maps. On this dataset, a deep learning-based neural network is used to obtain a network model that derives holograms from RGB-D images. Thus, the network model possesses the ability to infer holograms from RGB-D images.

[0004] Leveraging deep learning techniques and the powerful parallel computing capabilities of graphics processing units (GPUs), several neural networks have been proposed and used for real-time hologram generation. Shi et al. [L. Shi, B. Li, C. Kim, et al., “Towards real-time photorealistic 3d holography with deep neural networks,” Nature 591, 234–239 (2021)] synthesized the large-scale hologram dataset MIT-CGH-4K using a point-based method that considers occlusion, and proposed TensorHolo to generate holograms. They obtained pure phase holograms by applying an improved biphase encoding method, and the miniaturized TensorHolo, accelerated by TensorRT, can achieve 60 FPS hologram generation. Liu et al. [K. Liu, J. Wu, Z. He, and L. Cao, “4k-dmdnet: diffraction model-driven network for 4k computer-generated holography,” Opto-Electronic Adv. 6, 220135–1(2023)] proposed a diffraction model-driven neural network, named DMDNet, to synthesize pure phase holograms with a resolution of 3840x2160 in 0.26 seconds. DMDNet mainly consists of a UNet-based network, combining residual methods and subpixel convolution methods to generate pure phase holograms. Chen et al. [H. Chen, L. Huang, T. Liu, and A. Ozcan, “Fourier imager network (fin): A deep neural network for hologram reconstruction with superior external generalization,” Light. Sci. & Appl. 11, 254 (2022)] proposed a Fourier imaging network that utilizes residual connections and applies convolutional extraction capabilities in both the spatial and Fourier domains to achieve superior external generalization. Dong et al. proposed a Fourier-inspired neural network module and demonstrated that this module can improve the quality of generated holograms.Dong et al. [Z. Dong, C. Xu, Y. Ling, et al., “Fourier-inspired neural module for real-time and high-fidelity computer-generated holography,” Opt. Lett. 48, 759–762 (2023)] proposed a hologram generation method based on Vision Transformer, utilizing a U-shaped Transformer (UFormer) to generate pure phase holograms, demonstrating the potential of attention mechanisms in hologram generation. In summary, we found that U-shaped networks can solve problems such as high-frequency detail preservation and sub-hologram superposition involving contextual information. However, these works have not yet further explored the potential of U-shaped networks, and the network's performance in generating high-quality learning can still be improved.

[0005] Current technologies use computational simulations to obtain holograms, which is time-consuming, and the image quality varies significantly depending on the specific method used. Obtaining high-quality holograms is time-consuming and requires a large amount of memory for computation. Summary of the Invention

[0006] To overcome the problems existing in the prior art, the present invention discloses a neural network method, system, and device for generating three-dimensional holograms; it aggregates features at various network depths through a residual U-shaped module (RSU) and expands the receptive field by combining ordinary convolution and dilated convolution; it effectively extracts image edges and high-frequency features; and it improves structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) by adding a lightweight Fourier convolution module; it obtains high-quality holograms while reducing processing time and memory consumption.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A neural network method for generating three-dimensional holograms, characterized by comprising the following steps:

[0009] Step 1: Composition and division of holographic data dataset: Perform data standardization on the holographic dataset of red, green and blue three-channel images - depth images RGB-D and amplitude and phase maps, and divide it into training set, validation set and test set;

[0010] Step 2: Based on the U2Net network, a nested U-shaped network Holo-U2Net network including an outer U-shaped network structure and an inner U-shaped network structure is obtained. The standardized red, green and blue three-channel image-depth image RGB-D is input into the Holo-U2Net network.

[0011] Step 3, Predict the hologram: Perform the first convolution on the output feature maps of the bottleneck layer RSU-4F and the residual U-shaped modules RSU in the decoding stage, and connect them with the red, green and blue three-channel images and the depth image RGB-D. Then, perform the second convolution and hyperbolic tangent activation function (Tanh) to obtain the predicted hologram.

[0012] Step 4: Obtain optical device parameters; set batch size, Adam optimizer parameters, and learning rate; define loss function; train Holo-U2Net network on the training set of the holographic dataset, validate the model on the validation set to determine the final parameters, and test the final model on the test set of the holographic dataset.

[0013] Step 5: Replace the three operations of convolution, batch normalization (BN), and rectified linear function (ReLU) in each internal residual U-shaped module RSU with convolution, batch normalization (BN), and rectified linear function (ReLU), and add a lightweight Fourier convolution module to obtain feature information in the Fourier domain; repeat steps 1 to 4, and train, validate, and test the model again on the holographic dataset.

[0014] The specific method for step 1 is as follows: Set the RGB-D data range of the red, green, and blue three-channel image-depth image to [-0.5, 0.5], and the amplitude data range to... The phase data range is [0,1]; the dataset is reasonably divided into training set, validation set, and test set according to the size of the dataset.

[0015] The outer U-shaped structure described in step 2 is divided into an encoding stage, a bottleneck layer, and a decoding stage. The encoding stage consists of residual U-shaped modules (RSU), including residual U-shaped modules RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. The bottleneck layer consists of a single residual U-shaped module RSU-4F. The decoding stage consists of residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 in sequence. The encoding stage and the decoding stage are related. In the coding stage, the residual U-shaped modules (RSUs) exhibit symmetry at the same height. Sampling operations are canceled between the outer U-shaped residual U-shaped modules (RSUs), ensuring that the feature map sizes among them remain consistent. During the coding stage, the output feature maps of residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 are respectively connected to the residual U-shaped modules RSU-4F, RSU-4, RSU-5, and RSU-7 in the decoding stage. Before SU-6 and Residual U-shaped Module RSU-7; that is, the input of the Residual U-shaped Module RSU-4F in the decoding stage comes from the skip connection between the output feature maps of the bottleneck layer Residual U-shaped Module RSU-4F and the encoding stage Residual U-shaped Module RSU-4F; the input of the Residual U-shaped Module RSU-4 in the decoding stage comes from the skip connection between the output feature map of the encoding stage Residual U-shaped Module RSU-4F and the output feature map of the decoding stage Residual U-shaped Module RSU-4F; the input of the Residual U-shaped Module RSU-5 in the decoding stage comes from the output feature map of the encoding stage Residual U-shaped Module RSU-5 and the Residual U-shaped Module RSU-7 in the decoding stage. The input of the residual U-shaped module RSU-4 is a skip connection of the output feature map of the residual U-shaped module RSU-6 in the decoding stage and the output feature map of the residual U-shaped module RSU-5 in the decoding stage. The input of the residual U-shaped module RSU-7 in the decoding stage is a skip connection of the output feature map of the residual U-shaped module RSU-7 in the encoding stage and the output feature map of the residual U-shaped module RSU-6 in the decoding stage. The input feature map of the residual U-shaped module RSU-7 in the encoding stage comes from the normalized red-green-blue three-channel image-depth image RGB-D.

[0016] Step 2 describes the inner U-shaped structure as the internal network structure of residual U-shaped modules RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. Residual U-shaped modules RSU-7, RSU-6, RSU-5, and RSU-4 are denoted as Residual U-shaped module RSU-L, where L is an integer from 7 to 4. The inner U-shaped structure, like the outer U-shaped structure, is divided into an encoding stage, a bottleneck layer, and a decoding stage. For the input feature map received by the inner U-shaped structure of residual U-shaped module RSU-L, in the encoding stage, two... The process involves performing convolution with the feature map size unchanged, batch normalization (BN), and rectified linear LU (ReLU). The first convolution with the feature map size unchanged, BN, and ReLU are executed, changing the number of channels from Cin to Cout. The second convolution with the feature map size unchanged, BN, and ReLU are executed again, changing the number of channels from Cout to Cmid. Then, average pooling downsampling is performed to halve the feature map size, and convolution with the feature map size unchanged and the number of output channels Cmid, BN, and ReLU are executed. The bottleneck layer's input feature map is obtained by performing convolution, batch normalization (BN), and rectified linear LU (ReLU) a total of n (n = L-2) times. In the bottleneck layer, the input feature map is subjected to dilated convolution with a dilation factor of 2 and Cmid output channels, while maintaining the feature map size. This dilation is then output to the decoding stage. In the decoding stage, the output feature map of the bottleneck layer is first connected to the final output feature map of the decoding stage via a skip connection to form a new feature map. Then, convolution, batch normalization (BN), and rectified linear LU (ReLU) are performed with the feature map size remaining constant and Cmid output channels. Finally, bilinear interpolation upsampling doubles the feature map size. The doubled feature map is skip-connected to the feature map before the nth average pooling downsampling in the encoding stage. Convolution, batch normalization (BN), and rectified linear function (ReLU) are performed with the feature map size unchanged and the number of output channels Cmid. Then, n bilinear interpolation upsampling, skip connections, convolution, batch normalization (BN), and rectified linear function (ReLU) are performed. Each subsequent upsampling is skip-connected to the feature map before the previous average pooling downsampling in the encoding stage. After the final upsampling, skip connections, convolution, batch normalization (BN), and rectified linear function (ReLU) are completed, the feature map size is now the same as the feature map size of the initial input residual U-shaped module RSU.The feature map at this point is skipped and connected to the feature map obtained after the second execution of convolution, batch normalization (BN), and rectified linear function (ReLU) with the same feature map size in the symmetric encoding stage. Then, convolution, batch normalization (BN), and rectified linear function (ReLU) with the same feature map size and output channel number Cout are performed. Then, the feature map is added to the feature map obtained after the first execution of convolution, batch normalization (BN), and rectified linear function (ReLU) with the same feature map size in the encoding stage to obtain the final output feature map. The residual U-shaped module RSU-4F is obtained by modifying the residual U-shaped module RSU-4. That is, the original downsampling, convolution, batch normalization (BN), and rectified linear function (ReLU) of the residual U-shaped module RSU-4 are replaced with dilated convolution, batch normalization (BN), and rectified linear function (ReLU) with a dilation factor of 2, and dilated convolution, batch normalization (BN), and rectified linear function (ReLU) with a dilation factor of 4, and convolution and batch normalization of the bottleneck layer.

[0017] The Batch Normalization (BN) and Rectified Linear Function (ReLU) functions are replaced with dilated convolution, batch normalization (BN), and Rectified Linear Function (ReLU) with a dilation factor of 8. In the RSU-4 decoding stage of the residual U-shaped module, the upsampling, convolution, batch normalization (BN), and Rectified Linear Function (ReLU) functions are replaced with dilated convolution, batch normalization (BN), and Rectified Linear Function (ReLU) with a dilation factor of 4, and dilated convolution, batch normalization (BN), and Rectified Linear Function (ReLU) with a dilation factor of 2. The rest of the structure and parameters remain unchanged.

[0018] The specific method of step 3 is as follows: the output feature maps of the bottleneck layer residual U-shaped module RSU-4F module of the outer U-shaped structure, the output feature maps of the residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 modules in the decoding stage are convolved with the feature map size unchanged and the number of output channels increased to 6, resulting in a total of 6 feature maps; the 6 feature maps are then skip-connected with the standardized red-green-blue three-channel image-depth image RGB-D feature map, and then convolved with the feature map size unchanged and the number of output channels increased to 6, and Tanh is applied to obtain the final amplitude and phase hologram output.

[0019] The method for defining the loss function in step 4 is as follows:

[0020] The difference between the predicted hologram and the target hologram is calculated using the loss function L. hologram Let L represent the difference between the image obtained from the propagation of the predicted hologram and the target hologram, expressed by the loss function L. propagaionThe total loss function can be expressed as:

[0021] L = L hologram +L propagation (1)

[0022] Use L hologram Calculate the mean squared error loss (L2 loss) between the real hologram and the predicted hologram.

[0023] As shown in formula (2);

[0024] A gt A true hologram representing the amplitude. The true hologram represents the phase, while A represents the predicted amplitude and φ represents the predicted phase, forming a complex-form predicted hologram; Phase Difference Corrected is expressed by the expression The calculation yields the corrected phase difference between the predicted and true phases. In formula (2), α and β are weighting parameters.

[0025]

[0026] L propagation =σL amp +εL tv (3)

[0027] Loss function L propagtion Including the total variation loss and mean square error loss (L2 loss) at the focal depth of the hologram calculated by the angle spectral method (ASM), the calculation is performed by simulating free space wave propagation and combining formulas (4)(5)(6)(7); parameters σ and ε are weighting parameters; H is the amplitude and phase hologram provided by the dataset, H gt The hologram is obtained through network inference; H is discretized into a pixel grid, where m corresponds to the horizontal coordinate in the pixel grid and n corresponds to the vertical coordinate in the pixel grid; h represents the height of the physical hologram, w represents the width of the physical hologram; d is the focusing depth used to propagate the predicted hologram and the real hologram. in It is the depth of the input depth map in the red, green and blue three-channel image-depth image RGB-D; the EXP exponent term and parameter μ are used to adjust the attention weights of the network between the focal and defocus regions, D is the distance between the near and far shear planes of the view frustum, the predicted amplitude map with attention weights in the focal stack and the real amplitude map are obtained by propagating the predicted hologram and the real hologram to depth d through ASM propagation, i.e., formula (5); L tv The total variation loss is calculated by comparing the focal depth stacks of midpoint holograms propagated to different focal depths with real holograms propagated to the same depth. The operator TV represents the calculation of the total variation.

[0028] L amp =||AMP focal (H,D,d,d in )-AMP focal (H gt ,D,d,d in )||2(4)

[0029]

[0030]

[0031] L tv =||TV(AMP) focal (H,D,d,d in ))-TV(AMP focal (H gt ,D,d,d in ))||1 (7)

[0032] The specific method of step 4 is as follows: Optical parameters of the optical equipment are set, including SLM pixel size and RGB laser emitter wavelength; the attenuation rate of the first moment estimate, the attenuation rate of the second moment estimate, and the learning rate of the Adam optimizer are set; the batch size is set according to the number of graphics cards and their memory; the loss function is calculated, and training ends when the loss converges stably; the model is validated on the validation set, and the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) indices of the predicted hologram and the real hologram are calculated; the model with the best quality evaluation index between the predicted hologram and the real hologram is selected as the final model; the model is then tested on the test set to obtain the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) indices of the final model.

[0033] The specific method for step 5 is as follows:

[0034] The three operations of convolution, batch normalization (BN), and rectified linear function (ReLU) in each residual U-shaped module RSU are replaced with a lightweight Fourier convolution module that includes convolution, batch normalization (BN), and rectified linear function (ReLU) and adds feature information in the Fourier domain. The parameter settings of the convolution in the lightweight Fourier convolution module are consistent with the parameter settings of each convolution in the residual U-shaped module RSU. The lightweight Fourier convolution module first performs a fast Fourier transform (FFT) on the input feature map to obtain a feature map with complex values. Convolution is then performed on the real and imaginary parts of the complex feature map, and the results are merged into a complex tensor. Then, the complex tensor is processed by inverse fast Fourier transform (IFFT) to obtain a feature map. The feature map obtained by convolving the feature map obtained by inverse Fourier transform (IFFT) with the input feature map of the lightweight Fourier convolution module is then added, batch normalized (BN), and rectified linear function (ReLU) are performed to obtain the output feature map of the lightweight Fourier convolution module.

[0035] A neural network system for generating 3D holograms includes: an outer U-shaped structure, an inner U-shaped structure, and replaceable lightweight Fourier convolutional modules, wherein:

[0036] The outer U-shaped structure allows the network to be deeper and integrates the RSU output feature maps from the bottleneck layer residual U-shaped module and the RSU output feature maps from each residual U-shaped module in the encoding stage; the network extracts multi-scale features within blocks and aggregates multi-level features between blocks, which has a stronger fitting ability and the ability to generate high-quality holograms;

[0037] The inner U-shaped structure utilizes the Residual U-shaped module (RSU) to extract multi-scale and high-frequency features;

[0038] The lightweight Fourier convolution module, in the implementation of the residual U-shaped module RSU, performs convolution, batch normalization (BN), and rectified linear function (ReLU) operations in sequence. Therefore, the lightweight Fourier convolution module is designed to include convolution, batch normalization (BN), and rectified linear function (ReLU), and adds the acquisition of Fourier domain feature information to further improve the Holo-U2Net network's ability to generate high-quality holograms.

[0039] A neural network device for generating three-dimensional holograms, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor is used to generate a high-quality three-dimensional hologram according to the neural network method in steps 1 to 5 when executing the computer program.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1. The Holo-U2Net network incorporates a Residual U-shaped module (RSU) to explore features at various network depths, and combines ordinary convolution and dilated convolution to expand the receptive field; it effectively extracts image edges and high-frequency features, some of which significantly impact the quality of hologram generation. This invention was tested on the MIT-CGH-4K dataset to assess the quality of generated holograms. According to the structural similarity (SSIM) between predicted and real amplitude images in Table 1, our method performs best, with an average of 0.9985, exceeding the second-best model, TensorHolo, by 0.0015. In terms of peak signal-to-noise ratio (PSNR), our method averages 46.15, exceeding TensorHolo by 2.73 dB, with a structural similarity improvement of 0.0016. In the evaluation of focus stack-based holograms, high-quality holograms can be obtained from the input feature map derived from a standardized RGB-D image of the red, green, and blue channels. In end-to-end holographic imaging systems, the holograms generated by this invention also exhibit the best quality.

[0044] 2. The lightweight Fourier convolution module of this invention replaces the processes of convolution, batch normalization (BN), and Rectified Linear Function (ReLU), which can further improve the Holo-U2Net network's ability to generate high-quality holograms. The lightweight Fourier convolution module only includes three convolution parameters that need to be updated during training, so it does not significantly increase the number of model parameters. The convolution parameter settings in the lightweight Fourier convolution module are consistent with those in Holo-U2Net without the lightweight Fourier convolution module, thus ensuring that adding this module does not require extensive modifications to the source code and maintains compatibility and consistency with the model framework. We conducted ablation experiments on the Holo-U2Net network with and without the addition of the lightweight Fourier convolution module, as shown in Table 2. In terms of structural similarity (SSIM) and peak signal-to-noise ratio (PSNR), adding the lightweight Fourier convolution module can improve the Holo-U2Net network's structural similarity (SSIM) by 0.0014 and peak signal-to-noise ratio (PSNR) by 0.32dB.

[0045] In summary, compared with existing technologies, this invention aggregates features from various network depths through the Residual U-shaped module RSU, and expands the receptive field by combining ordinary convolution and dilated convolution; it effectively extracts image edges and high-frequency features; by adding a lightweight Fourier convolution module, structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) are improved; and it obtains high-quality holograms while reducing processing time and memory consumption. Attached Figure Description

[0046] Figure 1 This is the overall architecture diagram of the Holo-U2Net network of this invention.

[0047] Figure 2 This is a structural diagram of the internal composition of the residual U-shaped module RSU-7 of the present invention.

[0048] Figure 3 This is a structural diagram of the internal composition of the residual U-shaped module RSU-4F of the present invention.

[0049] Figure 4 This invention relates to a lightweight Fourier convolution module.

[0050] Figure 5 This invention verifies the focusing and defocusing of three-dimensional holograms of real-world images.

[0051] Figure 6 These are samples from the MIT-CGH-4K test set of this invention and multiple network inference results. Detailed Implementation

[0052] The present invention will now be described in further detail with reference to the accompanying drawings.

[0053] A neural network method for generating 3D holograms specifically includes the following steps:

[0054] Step 1: Composition and division of holographic data dataset: Perform data standardization on the holographic dataset of red-green-blue three-channel images-depth images (RGB-D) and amplitude and phase images, and divide it into training set, validation set and test set;

[0055] Step 2: Based on the U2Net network, a nested U-shaped network Holo-U2Net network including an outer U-shaped network structure and an inner U-shaped network structure is obtained. The standardized input feature map is taken from the standardized red-green-blue three-channel image-depth image RGB-D image and input into the Holo-U2Net network.

[0056] Step 3, Predict the hologram: Perform the first convolution on the output feature maps of the bottleneck layer residual U-shaped module RSU-4F and the residual U-shaped modules RSU in the decoding stage, and connect them with the input feature maps from the standardized red-green-blue three-channel image-depth image RGB-D. Then, perform the second convolution and hyperbolic tangent activation function (Tanh) to obtain the predicted hologram.

[0057] Step 4: Obtain optical device parameters; set batch size, Adam optimizer parameters, and learning rate; define loss function; train Holo-U2Net network on the training set of the holographic dataset, validate the model on the validation set to determine the final parameters, and test the final model on the test set of the holographic dataset.

[0058] Step 5: Replace the three operations of convolution, batch normalization (BN), and rectified linear function (ReLU) in each internal U-shaped residual U-shaped module RSU with convolution, batch normalization (BN), and rectified linear function (ReLU), and add a lightweight Fourier convolution module to obtain the feature information of the Fourier domain; repeat steps 1 to 4, and train, validate, and test the model again on the holographic dataset.

[0059] The specific method for step 1 is as follows: the input feature map is set from the standardized red-green-blue three-channel image-depth image RGB-D data range of [-0.5, 0.5], and the amplitude data range is... The phase data range is [0,1]; the dataset is reasonably divided into training set, validation set, and test set according to the size of the dataset.

[0060] The outer U-shaped structure described in step 2 is divided into an encoding stage, a bottleneck layer, and a decoding stage. The encoding stage consists of residual U-shaped modules (RSU), including residual U-shaped modules RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. The bottleneck layer consists of a single residual U-shaped module RSU-4F. The decoding stage consists of residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 in sequence. The encoding stage and the decoding stage... The residual U-shaped modules (RSUs) exhibit symmetry at the same height. Sampling operations are canceled between the residual U-shaped modules (RSUs) in the outer U-shaped structure, ensuring that the feature map sizes among them remain consistent. During the encoding stage, the output feature maps of residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 are respectively connected to the residual U-shaped modules RSU-4F, RSU-4, RSU-5, and RSU-6 in the decoding stage. Before the RSU-7 residual U-module; that is, the input of the RSU-4F residual U-module in the decoding stage comes from the skip connection between the output feature maps of the bottleneck layer residual U-module RSU-4F and the encoding stage residual U-module RSU-4F; the input of the RSU-4F residual U-module in the decoding stage comes from the skip connection between the output feature maps of the encoding stage residual U-module RSU-4F and the decoding stage residual U-module RSU-4F; the input of the RSU-5 residual U-module in the decoding stage comes from the skip connection between the output feature maps of the encoding stage residual U-module RSU-5 and the output feature maps of the decoding stage residual U-module RSU-4. The input to the residual U-module RSU-6 in the decoding stage is a skip connection between the output feature map of the residual U-module RSU-6 in the encoding stage and the output feature map of the residual U-module RSU-5 in the decoding stage; the input to the residual U-module RSU-7 in the decoding stage is a skip connection between the output feature map of the residual U-module RSU-7 in the encoding stage and the output feature map of the residual U-module RSU-6 in the decoding stage; the input feature map of the residual U-module RSU-7 in the encoding stage comes from the standardized input feature map, which is a standardized RGB-D image of the red, green, and blue three-channel image-depth image.

[0061] Step 2 describes the inner U-shaped structure as the internal network structure of residual U-shaped modules RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. Residual U-shaped modules RSU-7, RSU-6, RSU-5, and RSU-4 are denoted as Residual U-shaped module RSU-L, where L is an integer from 7 to 4. The inner U-shaped structure, like the outer U-shaped structure, is divided into an encoding stage, a bottleneck layer, and a decoding stage. For the input feature map received by the inner U-shaped structure of residual U-shaped module RSU-L, in the encoding stage, two... The process involves performing convolution with the feature map size unchanged, batch normalization (BN), and rectified linear LU (ReLU). The first convolution with the feature map size unchanged, BN, and ReLU are executed, changing the number of channels from Cin to Cout. The second convolution with the feature map size unchanged, BN, and ReLU are executed again, changing the number of channels from Cout to Cmid. Then, average pooling downsampling is performed to halve the feature map size, and convolution with the feature map size unchanged and the number of output channels Cmid, BN, and ReLU are executed. The bottleneck layer's input feature map is obtained by performing convolution, batch normalization (BN), and rectified linear LU (ReLU) a total of n (n = L-2) times. In the bottleneck layer, the input feature map is subjected to dilated convolution with a dilation factor of 2 and Cmid output channels, while maintaining the feature map size. This dilation is then output to the decoding stage. In the decoding stage, the output feature map of the bottleneck layer is first connected to the final output feature map of the decoding stage via a skip connection to form a new feature map. Then, convolution, batch normalization (BN), and rectified linear LU (ReLU) are performed with the feature map size remaining constant and Cmid output channels. Finally, bilinear interpolation upsampling doubles the feature map size. The doubled feature map is skip-connected to the feature map before the nth average pooling downsampling in the encoding stage. Convolution, batch normalization (BN), and rectified linear function (ReLU) are performed with the feature map size unchanged and the number of output channels Cmid. Then, n bilinear interpolation upsampling, skip connections, convolution, batch normalization (BN), and rectified linear function (ReLU) are performed. Each subsequent upsampling is skip-connected to the feature map before the previous average pooling downsampling in the encoding stage. After the final upsampling, skip connections, convolution, batch normalization (BN), and rectified linear function (ReLU) are completed, the feature map size is now the same as the feature map size of the initial input residual U-shaped module RSU.The feature map at this point is skip-connected with the feature map obtained from the second execution of convolution, batch normalization (BN), and rectified linear function (ReLU) in the symmetric encoding stage (with the feature map size unchanged), and then convolution, batch normalization (BN), and rectified linear function (ReLU) are performed with the feature map size unchanged and the number of output channels Cout. Then, the feature map is added to the feature map obtained from the first execution of convolution, batch normalization (BN), and rectified linear function (ReLU) based on the feature map size unchanged, to obtain the final output feature map. The residual U-shaped module RSU-4F is obtained by modifying the residual U-shaped module RSU-4, that is, replacing the original downsampling, convolution, batch normalization (BN), and rectified linear function (ReLU) in the residual U-shaped module RSU-4 with... The dilation coefficient is 2, with batch normalization (BN) and rectified linear LU; dilated convolution with a dilation coefficient of 4, with batch normalization (BN) and rectified linear LU; the convolution, batch normalization (BN), and rectified linear LU in the bottleneck layer are replaced with dilated convolution with a dilation coefficient of 8, with batch normalization (BN) and rectified linear LU; the upsampling, convolution, batch normalization (BN), and rectified linear LU in the RSU-4 decoding stage of the residual U-shaped module are replaced with dilated convolution with a dilation coefficient of 4, with batch normalization (BN) and rectified linear LU; and dilated convolution with a dilation coefficient of 2, with batch normalization (BN) and rectified linear LU. All other structures and parameters remain unchanged.

[0062] The specific method of step 3 is as follows: the output feature map of the bottleneck layer residual U-shaped module RSU-4F module of the outer U-shaped structure, the output feature map of the residual U-shaped module RSU-4F module, residual U-shaped module RSU-4 module, residual U-shaped module RSU-5 module, residual U-shaped module RSU-6 module, and residual U-shaped module RSU-7 module of the decoding stage are subjected to convolution with the feature map size unchanged and the number of output channels 6, resulting in a total of 6 feature maps; the input feature map of the 6 features comes from the skip connection of the standardized red-green-blue three-channel image-depth image RGB-D feature map, and then undergoes convolution with the feature map size unchanged and the number of output channels 6, and Tanh to obtain the final amplitude and phase hologram output.

[0063] The method for defining the loss function in step 4 is as follows:

[0064] The difference between the predicted hologram and the target hologram is calculated using the loss function L. hologram Let L represent the difference between the image obtained from the propagation of the predicted hologram and the target hologram, expressed by the loss function L. propagaion The total loss function is expressed as:

[0065] L = L hologram +L propagation (1)

[0066] Use L hologram Calculate the mean squared error loss (L2 loss) between the real hologram and the predicted hologram.

[0067] As shown in formula (2);

[0068] A gt A true hologram representing the amplitude. The true hologram represents the phase, while A represents the predicted amplitude and φ represents the predicted phase, forming a complex-form predicted hologram; Phase Difference Corrected is expressed by the expression The calculation yields the corrected phase difference between the predicted and true phases. In formula (2), α and β are weighting parameters.

[0069]

[0070] L propagation =σL amp +εL tv (3)

[0071] Loss function L propagtion Including the total variation loss and mean square error loss (L2 loss) at the focal depth of the hologram calculated by the angle spectral method (ASM), the calculation is performed by simulating free space wave propagation and combining formulas (4)(5)(6)(7); parameters σ and ε are weighting parameters; H is the amplitude and phase hologram provided by the dataset, H gt The hologram is obtained through network inference; H is discretized into a pixel grid, where m corresponds to the horizontal coordinate in the pixel grid and n corresponds to the vertical coordinate in the pixel grid; h represents the height of the physical hologram, w represents the width of the physical hologram; d is the focusing depth used to propagate the predicted hologram and the real hologram. in The input feature map is derived from the depth map of the input depth map in the standardized red-green-blue three-channel image-depth image RGB-D; the EXP exponent term and parameter μ are used to adjust the attention weights of the network between the focal and defocus regions, D is the distance between the near and far shear planes of the view frustum, and the predicted amplitude map with attention weights in the focal stack and the real amplitude map are obtained by propagating the predicted hologram and the real hologram to depth d through ASM propagation, i.e., formula (5); L tv The total variation loss is calculated by comparing the focal depth stacks of midpoint holograms propagated to different focal depths with real holograms propagated to the same depth. The operator TV represents the calculation of the total variation.

[0072] L amp =||AMP focal (H,D,d,d in )-AMP focal (H gt ,D,d,d in )||2 (4)

[0073]

[0074]

[0075] L tv =||TV(AMP) focal (H,D,d,d in ))-TV(AMP focal (H gt ,D,d,d in ))||1 (7)

[0076] The specific method of step 4 is as follows: Optical parameters of the optical equipment are set, including SLM pixel size and RGB laser emitter wavelength; the attenuation rate of the first moment estimate, the attenuation rate of the second moment estimate, and the learning rate of the Adam optimizer are set; the batch size is set according to the number of graphics cards and their memory; the loss function is calculated, and training ends when the loss converges stably; the model is validated on the validation set, and the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) indices of the predicted hologram and the real hologram are calculated; the model with the best quality evaluation index between the predicted hologram and the real hologram is selected as the final model; the model is then tested on the test set to obtain the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) indices of the final model.

[0077] The specific method for step 5 is as follows:

[0078] The three operations of convolution, batch normalization (BN), and rectified linear function (ReLU) in each residual U-shaped module RSU are replaced with a lightweight Fourier convolution module that includes convolution, batch normalization (BN), and rectified linear function (ReLU) and adds feature information in the Fourier domain. The parameter settings of the convolution in the lightweight Fourier convolution module are consistent with the parameter settings of each convolution in the residual U-shaped module RSU. The lightweight Fourier convolution module first performs a fast Fourier transform (FFT) on the input feature map to obtain a feature map with complex values. Convolution is then performed on the real and imaginary parts of the complex feature map, and the results are merged into a complex tensor. Then, the complex tensor is processed by inverse fast Fourier transform (IFFT) to obtain a feature map. The feature map obtained by convolving the feature map obtained by inverse Fourier transform (IFFT) with the input feature map of the lightweight Fourier convolution module is then added, batch normalized (BN), and rectified linear function (ReLU) are performed to obtain the output feature map of the lightweight Fourier convolution module.

[0079] Example

[0080] We improve upon U2Net and propose the Holo-U2Net network. The overall architecture of the Holo-U2Net network is shown below. Figure 1 In this architecture, the Holo-U2Net network is a two-layer nested U-structure. The outer layer consists of a large U-structure containing 11 residual U-shaped modules, and the inner layer represents the U-shaped structure inside each residual U-shaped module's RSU.

[0081] In the Residual U-shaped module RSU-L, L represents different heights of the module, including 7, 6, 5, 4, and 4F, enabling the module to extract multi-scale features from the input feature map. Figure 2 and Figure 3 In this paper, we provide a detailed representation of the internal structure of the Residual U-Module RSU-7 and Residual U-Module RSU-4F. This nested U-structure efficiently extracts multi-scale features within blocks and aggregates multi-level features between blocks. There is no downsampling or upsampling process between the Residual U-Module RSUs in the encoding and decoding parts to preserve the repetitive structure and the same feature map size. H and W represent the height and width of the current feature map. The parameters k, s, and d in Conv represent the kernel size, stride, and dilation coefficient, respectively. “BN” represents batch normalization. “Discard” indicates that layers of this height will be discarded sequentially as L decreases, corresponding to Residual U-Module RSU-6, Residual U-Module RSU-5, and Residual U-Module RSU-4. Cin, Cmid, and Cout all represent the current number of channels in the feature map. The Residual U-Module RSU-4F utilizes dilated convolutions without any upsampling or pooling processes, thus its feature maps have the same resolution as the input feature maps.

[0082] The Holo-U2Net network input feature maps are derived from standardized RGB-D images (red, green, and blue channels) with a depth range of [-0.5, 0.5]. This image serves as the input to the Residual U-module RSU-7 in the encoding stage, with a channel count Cin of 4. The Residual U-modules RSU-6 through RSU-4 have Cin of 64, Cmid of 16, and Cout of 64. The feature maps generated by the Residual U-modules RSU-4F through RSU-7 in the encoding stage are concatenated with the feature maps generated by the Residual U-module RSU-4F in the bottleneck layer and the Residual U-modules RSUs in the decoding stage, respectively, to form the input feature maps for the Residual U-modules in the decoding stage. Therefore, the Residual U-module RSU-4F has a Cin of 128, a Cmid of 16, and a Cout of 64. The convolutional kernels used within the Residual U-modules RSU are all 3x3 in size with a stride of 1. The fill grid is set to 1. When the dilation parameter of the convolution is greater than 1, the fill grid setting is consistent with the dilation parameter.

[0083] Starting with the residual U-shaped module RSU-7 in the encoding stage, the input feature map of the Holo-U2Net network comes from a standardized RGB-D image (red, green, and blue channels) and depth image. After convolution, the number of channels increases from 4 to 64. After further convolution, the number of channels is reduced to 16. This can be represented as the change in channel number from Cin to Cout to Cmid. Five average pooling operations are then performed for downsampling, followed by five convolution operations, maintaining the number of channels at 16. The feature map size is reduced to one-thirty-second of its original size. A dilated convolution operation with a dilation factor of 2 is performed on the bottleneck layer of the residual U-shaped module RSU-7. In the decoding stage, skip connections are used to connect the feature maps from the same layer as the encoding stage. These maps, with 32 channels, are used as input to the convolution operations, resulting in an output of 16 channels. Five convolution operations and five average pooling operations are performed to restore the feature map size, making it consistent with the input image size in the dataset. After the convolution operation, the output channel is 64. It is added to the feature map of the first convolution operation performed inside the residual U-shaped module RSU to obtain the output of the residual U-shaped module RSU-7.

[0084] Next, the data flows to residual U-shaped modules RSU-6, RSU-5, and RSU-4. The difference between these modules and residual U-shaped module RSU-7 is that after the second convolution operation in the encoding stage, the number of downsampling pooling and convolution operations performed are 4, 3, and 2 times respectively. The decoding stage corresponds to this.

[0085] When passing through the Residual U-shaped module RSU-4F, the module does not contain pooling processing. This module is considered equivalent to the Residual U-shaped module RSU-4. In the encoding stage, the convolution after the first pooling is replaced with a dilated convolution with a dilation factor of 2, the convolution operation after the second pooling is replaced with a dilated convolution with a dilation factor of 4, the bottleneck layer is replaced with a dilated convolution with a dilation factor of 8, and the decoding stage uses dilated convolutions with a dilation factor of 4 and a dilation convolution with a dilation factor of 2. The rest is the same as the Residual U-shaped module RSU-4F.

[0086] The type module RSU-4 is consistent.

[0087] When using residual U-shaped modules RSU4F, RSU4, RSU-5, RSU-6, and RSU-7 in the decoding stage, the input feature maps of each module originate from the output feature maps of the same layer in the encoding stage and are connected in a skip connection to the output feature maps of the lower-level modules in the encoding stage. The bottleneck layer residual U-shaped module RSU-4F, the decoding stage residual U-shaped module RSU-4F, and the residual U-shaped module RSU-4 to the residual U-shaped module RSU-7...

[0088] The RSU-7 module connects the results of convolution (3x3 kernel, stride 1, grid fill 1), batch normalization (BN), and Rectified Linear Array (ReLU) to RGB-D, and then again through convolution (3x3 kernel, stride 1, grid fill 1), batch normalization (BN), and Rectified Linear Array (ReLU). This 6-channel data result is activated using the Tanh activation function, limiting the data range to [-1, 1]. The first three channels are then selected, and their numerical range is further limited. Save the amplitude hologram. Take the last three channels, limit the value range to [0,1], subtract the average phase, add 0.5, and save the phase hologram.

[0089] During the training of Holo-U2Net, the loss function is designed based on two perspectives:

[0090] The difference between the predicted hologram and the target hologram, and the difference between the image propagated from the predicted hologram and the target hologram. Therefore, the total loss function is expressed as Equation (1):

[0091] L = L hologram +L propagation (1)

[0092] Loss function L hologram The L2 loss between the real hologram and the predicted hologram is calculated using Equation (2). A gt and The true hologram represents the amplitude and phase, while A and φ represent the predicted amplitude and phase, forming a predicted hologram in complex form. PhaseDiffé renceCorrected is expressed by the expression... The calculation yields the corrected phase difference between the predicted and true phases. In formula (2), the parameters α and β are set to 1 and 3, respectively.

[0093]

[0094] L propagation =σL amp +εL tv (3)

[0095] Loss function L propagtion The total variation loss and L2 loss at the focal depth of the hologram, calculated using the angle spectral method (ASM), are simulated to represent free-space wave propagation and can be calculated using formulas (4)(5)(6)(7). Parameters σ and ε are set to 15 and 15, respectively. H is the hologram provided by the dataset. gt This is a hologram obtained through network inference. H is discretized as a pixel grid, where m and n correspond to the horizontal and vertical coordinates within the pixel grid, respectively. h and w represent the height and width of the physical hologram. d is the focusing depth used to propagate the predicted and real holograms. in This is the depth of the input depth map in RGB-D. The depth map data range is [0,1]. This interval is divided into 20 parts on average. A depth histogram is calculated on the depth map (using torch.histc). The depth histogram index is sorted in descending order. Noise obtained by standard normal sampling is added to each element of the index array. The array elements are normalized. The first 15 array elements are taken, and 5 array elements are randomly taken from the index array. Through these steps, 20 values ​​of focus depth d can be obtained. The EXP exponent term and parameter μ are used to adjust the attention weight of the network between the focus and defocus regions. In the experiment, it is set to 0.35. D is the distance between the near and far shear planes of the view frustum. In the experiment, it is set to 6mm. The predicted amplitude map and the real amplitude map with attention weights in the focus stack can be obtained by propagating the predicted hologram and the real hologram to depth d through ASM propagation, i.e., formula (5).

[0096] L amp =||AMP focal (H,D,d,d in )-AMP focal (H gt ,D,d,d in )||2 (4)

[0097]

[0098]

[0099] L tv =||TV(AMP) focal (H,D,d,d in ))-TV(AMP focal (H gt ,D,d,d in ))||1 (7)

[0100] L tv The total variation loss is calculated by comparing the focal depth stacks of midpoint holograms propagated to different focal depths with real holograms propagated to the same depth. The operator TV denotes the calculation of the total variation.

[0101] During the training of Holo-U2Net, we designed a lightweight module based on Fourier transform and convolution to replace the processes of convolution, batch normalization (BN), and Rectified Lune Function (ReLU), which can further improve the performance of our network. This module is designed as follows: Figure 4 As shown. This module only includes three convolution parameters that need to be updated during training, so it does not significantly increase the number of model parameters. Its implementation involves performing two convolution operations on the real and imaginary parts of the input feature map after FFT processing. The results of the real and imaginary parts are merged into a complex tensor. Then, the complex tensor is processed by IFFT to obtain the feature map. Figure 2 For features Figure 2 Features obtained through convolution Figure 1 Perform addition, batch normalization (BN), and ReLU operations. The convolution parameter settings in this module are consistent with the Holo-U2Net network without the lightweight Fourier convolution module. This ensures that adding this module does not require extensive modifications to the source code, maintaining compatibility and consistency with the model framework.

[0102] The Holo-U2Net network structure can effectively extract image edges and high-frequency features. These features have a significant impact on the quality of hologram generation, which stems from the deeper network structure and the fusion of multiple modules to predict holograms.

[0103] To evaluate the superior performance of the Holo-U2Net network compared to other network architectures, we used the MIT-CGH-4K test set with a resolution of 192×192 for comparison. Optical parameters included a 16 mm SLM pixel size and laser emitter wavelengths of 450 nm, 520 nm, and 638 nm. Holo-U2Net was trained for 300 epochs using the Adam optimizer with a learning rate of 1e-3. The decay rate for the first-order moment estimate of the Adam optimizer was set to 0.9, and the decay rate for the second-order moment estimate was set to 0.999. We reproduced and retrained several networks that perform well in holographic production, including UNet, U2Net, UFormer, and TensorHolo. The quality of the holograms generated by these networks was evaluated using metrics including structural similarity (SSIM) and peak signal-to-noise ratio (PSNR).

[0104]

[0105] Table 1. Network performance metrics on the MIT-CGH-4K test set

[0106]

[0107] Table 2. Experimental ablation results: Evaluation of Holo-U2Net on the MIT-CGH-4K test set.

[0108] like Figure 5 It can be seen that the 3D hologram reconstructs a depth image with sharp focus and blurred defocus. The sofa image on the left is an RGB image and its depth image, with the area marked by the orange box being the region of interest. As the focal distance increases with depth, the focus stack on the right focuses on the bear's eyes, the calendar, and the purple toy's head, respectively. Other areas become blurred due to the defocus effect. The ROI area on the right, marked with an orange pentagram, is the region of focus.

[0109] like Figure 6 It can be seen that the amplitude and phase holograms obtained by each network inference in the comparative experiment are compared with the actual situation of the three samples in the rightmost dataset. The structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) values ​​are displayed below the amplitude image. By visually magnifying the edge regions of the image and the high-frequency regions of the phase hologram, it can be found that the holograms obtained by the method of this invention are very similar to the holograms provided in the dataset. The feedback from the metrics of the three samples also corroborates that the method of this invention can generate high-quality three-dimensional holograms.

[0110] As can be seen from the above embodiments, compared with the prior art, the present invention has the ability to generate high-quality three-dimensional holograms, and the added lightweight Fourier convolution module can further improve the quality of the holograms generated by the network. Based on the structural similarity (SSIM) of the predicted and actual amplitude images in Table 1, the method proposed in this invention performs best, with an average SSIM of 0.9985, exceeding the existing TensorHolo by 0.0015. In terms of peak signal-to-noise ratio (PSNR), the proposed method achieves an average PSNR of 46.15 dB, exceeding TensorHolo by 2.73 dB. In the evaluation of holograms based on focal depth stacks, the holograms generated by this invention also achieve the best performance. Based on whether or not a lightweight Fourier convolution module is added, the ablation experiment results (Table 2) show that, in terms of both SSIM and PSNR, adding the lightweight Fourier convolution module improves Holo-U2Net by 0.0014 and 0.32 dB respectively.

Claims

1. A neural network method for generating three-dimensional holograms, characterized in that, Specifically, the following steps are included: Step 1: Composition and division of holographic data dataset: Perform data standardization on the holographic dataset of red, green and blue three-channel images - depth images RGB-D and amplitude and phase maps, and divide it into training set, validation set and test set; Step 2: Based on the U2Net network, a nested U-shaped network Holo-U2Net network including an outer U-shaped network structure and an inner U-shaped network structure is obtained. The standardized red, green and blue three-channel image-depth image RGB-D is input into the Holo-U2Net network. The outer U-shaped structure, the inner U-shaped structure, and the replaceable lightweight Fourier convolution module, wherein: The outer U-shaped structure allows the network to be deeper and integrates the RSU output feature maps from the bottleneck layer residual U-shaped module and the RSU output feature maps from each residual U-shaped module in the encoding stage; the network extracts multi-scale features within blocks and aggregates multi-level features between blocks, which has a stronger fitting ability and the ability to generate high-quality holograms; The inner U-shaped structure utilizes the Residual U-shaped module (RSU) to extract multi-scale and high-frequency features; In the implementation of the Residual U-shaped module RSU, the three operations of convolution, batch normalization (BN), and rectified linear function (ReLU) are executed consecutively. Therefore, the lightweight Fourier convolution module is designed to include convolution, batch normalization (BN), and rectified linear function (ReLU), and also incorporates the acquisition of feature information in the Fourier domain, further enhancing the Holo-U2Net network's ability to generate high-quality holograms. Step 3, Predict the hologram: Perform the first convolution on the output feature maps of the bottleneck layer RSU-4F and the residual U-shaped modules RSU in the decoding stage, and connect them with the red, green and blue three-channel images and the depth image RGB-D. Then, perform the second convolution and the hyperbolic tangent activation function Tanh to obtain the predicted hologram. Step 4: Obtain optical device parameters; set batch size, Adam optimizer parameters, and learning rate; define loss function; train the Holo-U2Net network on the training set of the holographic dataset, validate the model on the validation set to determine the final parameters, and test the final model on the test set of the holographic dataset. Step 5: Replace the three operations (convolution, batch normalization (BN), and rectified linear function (ReLU)) in each internal residual U-shaped module RSU with operations that include convolution, batch normalization (BN), and rectified linear function (ReLU), and add a lightweight Fourier convolution module to obtain feature information in the Fourier domain; repeat steps 1 to 4 to train, validate, and test the model again on the holographic dataset.

2. The neural network method for generating three-dimensional holograms according to claim 1, characterized in that, The specific method for step 1 is as follows: Set the RGB-D data range of the red, green, and blue three-channel image-depth image to [-0.5, 0.5], and the amplitude data range to [0, ...]. The phase data range is [0,1]; the dataset is reasonably divided into training set, validation set and test set according to the size of the dataset.

3. The neural network method for generating three-dimensional holograms according to claim 1, characterized in that, The outer U-shaped structure described in step 2 is divided into an encoding stage, a bottleneck layer, and a decoding stage. The encoding stage consists of residual U-shaped modules (RSU), including residual U-shaped modules RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. The bottleneck layer consists of a single residual U-shaped module RSU-4F. The decoding stage consists of residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 in sequence. The encoding stage and the decoding stage are related. In the coding stage, the residual U-shaped modules (RSUs) exhibit symmetry at the same height. Sampling operations are canceled between the outer U-shaped residual U-shaped modules (RSUs), ensuring that the feature map sizes among them remain consistent. During the coding stage, the output feature maps of residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 are respectively connected to the residual U-shaped modules RSU-4F, RSU-4, RSU-5, and RSU-7 in the decoding stage. Before SU-6 and Residual U-shaped Module RSU-7; that is, the input of the Residual U-shaped Module RSU-4F in the decoding stage comes from the skip connection between the output feature maps of the bottleneck layer Residual U-shaped Module RSU-4F and the encoding stage Residual U-shaped Module RSU-4F; the input of the Residual U-shaped Module RSU-4 in the decoding stage comes from the skip connection between the output feature map of the encoding stage Residual U-shaped Module RSU-4F and the output feature map of the decoding stage Residual U-shaped Module RSU-4F; the input of the Residual U-shaped Module RSU-5 in the decoding stage comes from the output feature map of the encoding stage Residual U-shaped Module RSU-5 and the Residual U-shaped Module RSU-7 in the decoding stage. The input of the residual U-shaped module RSU-4 is a skip connection of the output feature map of the residual U-shaped module RSU-6 in the decoding stage and the output feature map of the residual U-shaped module RSU-5 in the decoding stage. The input of the residual U-shaped module RSU-7 in the decoding stage is a skip connection of the output feature map of the residual U-shaped module RSU-7 in the encoding stage and the output feature map of the residual U-shaped module RSU-6 in the decoding stage. The input feature map of the residual U-shaped module RSU-7 in the encoding stage comes from the normalized red-green-blue three-channel image-depth image RGB-D.

4. The neural network method for generating three-dimensional holograms according to claim 1, characterized in that, The inner U-shaped structure described in step 2 is the internal network structure of residual U-shaped modules RSU-7, RSU-6, RSU-5, RSU-4, and RSU-4F. Residual U-shaped modules RSU-7, RSU-6, RSU-5, and RSU-4 are denoted as residual U-shaped module RSU-L, where L is an integer from 7 to 4. The inner U-shaped structure, like the outer U-shaped structure, is divided into an encoding stage, a bottleneck layer, and a decoding stage. For the input feature map received by the inner U-shaped structure of residual U-shaped module RSU-L, in the encoding stage, two convolutions with invariant feature map size, batch normalization (BN), and ReLU are performed first. That is, the first execution of convolutions with invariant feature map size, batch normalization (BN), and ReLU changes the number of channels from Cin to Cout. The second execution builds upon the first, performing convolution with the same feature map size, batch normalization (BN), and ReLU, changing the number of channels from Cout to Cmid. Then, average pooling downsampling halves the feature map size, followed by convolution with the same feature map size and Cmid output channels, batch normalization (BN), and ReLU. This process is repeated n times (n=L-2) to obtain the input feature map for the bottleneck layer. At the bottleneck layer, a dilated convolution with the same feature map size, a dilation factor of 2, and Cmid output channels is performed on the input feature map, and then output to the decoding stage. In the decoding stage, the output feature map of the bottleneck layer is first connected to the final output feature map of the decoding stage via a skip connection to form a new feature map. Then, convolution with the same feature map size and Cmid output channels, batch normalization (BN), and ReLU are performed. Finally, bilinear interpolation upsampling doubles the feature map size, and the doubled feature map is then combined with the nth average pooling operation from the encoding stage. Before pooling downsampling, the feature map is skipped and connected, performing convolution, batch normalization (BN), and ReLU with the output channel count Cmid while keeping the feature map size constant. Then, n iterations of bilinear interpolation upsampling, skip connections, convolution, BN, and ReLU are performed. Each subsequent upsampling iteration is skipped and connected to the feature map before the previous average pooling downsampling in the encoding stage. After the final upsampling, skip connections, convolution, BN, and ReLU, the feature map size is now the same as the initial input residual U-shaped module RSU. This feature map is then skipped and connected to the feature map obtained after the second iteration of convolution, BN, and ReLU with the feature map size constant in the symmetric encoding stage, and performing convolution, BN, and ReLU with the output channel count Cout while keeping the feature map size constant. Finally, the feature map is added to the feature map obtained after the first iteration of convolution, BN, and ReLU with the feature map size constant in the encoding stage to obtain the final output feature map.The Residual U-shaped Module RSU-4F is obtained by modifying the Residual U-shaped Module RSU-4. Specifically, the original downsampling, convolution, batch normalization (BN), and ReLU functions of the Residual U-shaped Module RSU-4 are replaced with dilated convolution with a dilation factor of 2, batch normalization (BN), ReLU, and dilated convolution with a dilation factor of 4. The convolution, batch normalization (BN), and ReLU functions of the bottleneck layer are replaced with dilated convolution with a dilation factor of 8. In the decoding stage of the Residual U-shaped Module RSU-4, the upsampling, convolution, batch normalization (BN), and ReLU functions are replaced with dilated convolution with a dilation factor of 4, batch normalization (BN), and ReLU, and dilated convolution with a dilation factor of 2. The remaining structure and parameters remain unchanged. ; 5. The neural network method for generating three-dimensional holograms according to claim 1, characterized in that, The specific method of step 3 is as follows: the output feature maps of the bottleneck layer residual U-shaped module RSU-4F module of the outer U-shaped structure, the output feature maps of the residual U-shaped modules RSU-4F, RSU-4, RSU-5, RSU-6, and RSU-7 modules in the decoding stage are convolved with the feature map size unchanged and the number of output channels increased to 6, resulting in a total of 6 feature maps; the 6 feature maps are then skip-connected with the standardized red-green-blue three-channel image-depth image RGB-D feature map, and then convolved with the feature map size unchanged and the number of output channels increased to 6, and Tanh is applied to obtain the final amplitude and phase hologram output.

6. The neural network method for generating three-dimensional holograms according to claim 1, characterized in that, The method for defining the loss function in step 4 is as follows: The difference between the predicted hologram and the target hologram is predicted using a loss function. Let the difference between the image obtained from the propagation of the predicted hologram and the target hologram be represented by the loss function. The total loss function is expressed as: (1) use The mean squared error loss (L2 loss) between the real hologram and the predicted hologram is calculated as shown in Equation (2); A true hologram representing the amplitude. A true hologram representing the phase, while The predicted result represents the amplitude. The predicted phase results are used to form a predicted hologram in complex form; Through expression The calculation yields the corrected phase difference between the predicted and true phases. In formula (2), α and β are weighting parameters. (2) (3) loss function The total variation loss and mean square error loss—L2 loss at the focal depth of the hologram are calculated using the angle spectrum method (ASM), simulating free space wave propagation, and are obtained by combining formulas (4)(5)(6)(7); parameters σ and ε are weighting parameters; It is an amplitude and phase hologram provided by the dataset. The hologram is obtained through network inference; The discretization is represented as a pixel grid, where m corresponds to the horizontal coordinate in the pixel grid and n corresponds to the vertical coordinate in the pixel grid; meanwhile, h represents the height of the physical hologram and w represents the width of the physical hologram. It is the focusing depth used to propagate the predictive hologram and the real hologram. It is the depth of the input depth map in the red, green and blue three-channel image-depth image RGB-D; the EXP exponent term and parameter μ are used to adjust the attention weight of the network between the focal and defocus areas, D is the distance between the near and far shear planes of the view cone, the predicted amplitude map with attention weights in the focal stack and the real amplitude map are obtained by propagating the predicted hologram and the real hologram to the depth d through ASM propagation, i.e. formula (5); The total variation loss is calculated by comparing the focal depth stacks of midpoint holograms propagated to different focal depths with the real holograms propagated to the same depth. The operator TV represents the calculation of the total variation. (4) (5) (6) (7)。 7. A neural network method for generating three-dimensional holograms according to claim 1 or 6, characterized in that, The specific method of step 4 is as follows: setting the optical parameters of the optical device, including the SLM pixel size and the RGB laser emitter wavelength; Set the decay rate for the first-order moment estimate and the decay rate for the second-order moment estimate of the Adam optimizer, as well as the learning rate; set the batch size according to the number of GPUs and GPU memory; calculate the loss function, and end training when the loss converges stably; validate the model on the validation set, calculate the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) of the predicted hologram and the real hologram, and select the model with the best quality evaluation index between the predicted hologram and the real hologram as the final model; use this model to test on the test set and obtain the final model's structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR).

8. The neural network method for generating three-dimensional holograms according to claim 1, characterized in that, The specific method for step 5 is as follows: The three operations (convolution, batch normalization (BN), and rectified linear LU) in each residual U-shaped module RSU are replaced with a lightweight Fourier convolution module that includes convolution, batch normalization (BN), and rectified linear LU, and incorporates features from the Fourier domain. The parameter settings of the convolution in the lightweight Fourier convolution module are consistent with those of the convolutions in the residual U-shaped module RSU. The lightweight Fourier convolution module first performs a fast Fourier transform on the input feature map to obtain a feature map with complex values. Convolution is then performed on the real and imaginary parts of the complex feature map, and the results are merged into a complex tensor. Then, the complex tensor undergoes an inverse fast Fourier transform to obtain a feature map. The feature map obtained by convolving the inverse Fourier transform with the input feature map of the lightweight Fourier convolution module is then summed, batch normalized (BN), and rectified linear LU is applied to obtain the output feature map of the lightweight Fourier convolution module.

9. A neural network device for generating three-dimensional holograms, characterized in that, include: Memory, used to store computer programs; A processor is used to generate a high-quality three-dimensional hologram according to the neural network method in steps 1 to 5 when executing the computer program.

Images