Realistic blurred 3D hologram reconstruction method based on joint training of multiple neural networks

Abstract

A method for reconstructing a real fuzzy three-dimensional hologram by joint training of multiple neural networks, for each training step, the RGB-D image loaded in the training set is divided into three layers according to the depth map, one layer is randomly selected as the focus surface, and the other two layers are non-focus surfaces, and the three layers are respectively input into multiple neural networks as complex amplitude generators to synthesize complex amplitude fields, which are encoded as phase-only holograms and then propagated to the near field for image reconstruction, and the loss function is obtained for the focus surface and non-focus surface and the corresponding Gaussian blurred target image after masking with a binary mask template, and the gradient is back-propagated to optimize the weight of the neural network. The hologram reconstructed by the present invention has the advantages of high image quality, obvious depth distinction, and real image blur, and is more similar to the natural image observed by the human eye, and is expected to be practically applied in scenes such as holographic display, VA / AR, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of computer generated holography (CGH), and in particular to a method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks. Background Art

[0002] Holography is a technology that records and reproduces the interference pattern of light waves, which can generate images with a sense of depth and three-dimensional effects. Unlike photographic or television images, holographic images record the phase and amplitude information of light waves, which allows holographic images to more accurately capture the three-dimensional structure and shape of objects. Computational holography is a method of using computers to process and generate holographic images. It combines computer science and optical technology, eliminating the complex optical recording process in traditional holographic technology. Holographic display can be achieved using only computer-generated holographic images and spatial light modulators (SLM), which improves the quality, flexibility and application range of holographic images. The development of this technology will help promote the further application and research of holographic technology in various fields such as games, medical care, education, and transportation.

[0003] A neural network is a computer network model composed of neurons and hierarchical structures that simulates the biological nervous system. Deep learning technology learns abstract representations of data through multi-level neural networks to achieve learning of complex patterns and features. At present, deep learning has achieved remarkable results in many fields such as computer vision and natural language processing. The method based on deep learning has become a very promising research for generating computer holograms. It overcomes the trade-off between image quality and running time in the simulation / optimization-based methods of traditional computer holography technology, and can achieve end-to-end high-quality real-time holographic display. At present, deep learning is mainly used in layer-based computer holography algorithms. The layer-based method divides the volume of the object into different layers, records holograms for each layer separately, and finally synthesizes a hologram containing information of all layers. The three-dimensional hologram is reconstructed at different propagation distances to obtain reconstructed images focused on different layers. However, since only the target image is set on the focus plane after the image is reconstructed, unnatural artifacts will be generated for the non-focused layers, resulting in a blur effect different from the real scene. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention proposes a method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks. For each training step, the RGB-D image loaded in the training set is divided into three layers according to the depth map, one layer is randomly selected as the focus surface, and the other two layers are non-focus surfaces. The three layers are respectively input into multiple neural networks as complex amplitude generators to synthesize the complex amplitude field, which is encoded as a phase-only hologram and then propagated to the near field for image reconstruction. After masking with a binary mask template, the loss function is obtained for the focus surface and non-focus surface and the corresponding Gaussian blurred target image, and the gradient is back-propagated to optimize the weight of the neural network. Compared with the original method, the hologram reconstructed by the present invention has the advantages of high image quality, obvious depth distinction, and real image blur. It is more similar to the natural image observed by the human eye and is expected to be practically applied in scenes such as holographic display, VA / AR, etc.

[0005] The technical solution of the present invention is as follows:

[0006] A method for reconstructing a real blurred three-dimensional hologram by jointly training multiple neural networks comprises the following steps:

[0007] Step 1: For each training step, load the RGB-D image, generate three binary masks according to the depth map, and divide the image into three layers; randomly select one layer as the focus plane and input it to the first complex amplitude generator, and the other two layers are input to the second and third complex amplitude generators respectively according to the distance from the focus plane; the complex amplitude fields of the three layers are output as one complex amplitude field;

[0008] Step 2: Use the angular spectrum method to propagate the complex amplitude field to the SLM plane, and perform phase-only encoding on the complex amplitude hologram to obtain a phase-only hologram;

[0009] Step 3: According to the distance between the focal plane and the SLM plane in the three layers, the phase-only hologram is back-propagated by the angular spectrum method to obtain the reconstructed image;

[0010] Step 4: Use Gaussian convolution kernels of different sizes to blur the input image, and then multiply it by three binary masks to obtain three target images, corresponding to the focused surface and two non-focused surfaces; multiply the reconstructed image by the same mask, and substitute each part and the corresponding target image into the loss function to calculate the loss value;

[0011] Step 5: Send the total loss value back to the network model to update the network weights through back propagation, and continuously optimize the hologram. When the set number of training rounds is reached, the training ends;

[0012] Step 6: Input the RGB-D image into the trained model and load the generated computational hologram onto the spatial light modulator to obtain the reconstruction of the stereoscopic image.

[0013] Furthermore, in the step 1, the RGB-D image is two images: one is a common RGB three-channel color image, and the other is a Depth image. The Depth image is similar to a grayscale image, and each pixel value thereof is the actual distance between the sensor and the object.

[0014] Preferably, the RGB-D dataset used is MIT-CGH-4K.

[0015] More preferably, the first, second and third complex amplitude generators refer to three neural networks with the same structure, which are used for training and generating complex amplitude fields of the focusing plane and the non-focusing plane respectively.

[0016] Furthermore, in the step 4, the binary mask template is a binary four-dimensional tensor with the same size as the input image tensor, which is generated by dividing the normalized depth map into intervals with equal intervals according to pixel values.

[0017] Furthermore, in the step 1, the three complex amplitude fields are synthesized by splicing the amplitudes and phases of the three complex amplitude fields along the channel dimension and inputting them into a small neural network that only contains one downsampling and one upsampling. Finally, the amplitudes and phases output by the network are spliced ​​along the channel dimension to obtain a complex amplitude field containing three layers of information.

[0018] Furthermore, in the step 2, only phase encoding is performed because existing commercial spatial light modulators can only modulate amplitude or phase individually, and phase-modulated spatial light modulators have higher diffraction efficiency; the phase-only encoding method is to input the complex amplitude hologram into a neural network serving as a holographic encoder.

[0019] Furthermore, in the step three, the reconstructed image refers to the amplitude of the reconstructed complex amplitude light field.

[0020] In the step 4, the input image is blurred using Gaussian convolution kernels of different sizes, and then multiplied by three binary masks to obtain three target images, in order to add constraints to the focused surface and the non-focused surface of the reconstructed image at the same time.

[0021] In the step 4, the loss function refers to a loss function that combines perceptual loss and mean square loss (MSE loss).

[0022] In step six, the trained model refers to a model that has been trained for a specified number of times, at which point the loss function of the training set converges, and the quality of the reconstructed image on the test set meets the structural similarity index requirements, and continuing training will not significantly improve the results.

[0023] In the step six, obtaining the reconstruction of the stereoscopic image refers to receiving the reconstructed holographic image at the corresponding diffraction distance.

[0024] The beneficial effects of the present invention are:

[0025] (1) Obtaining high-quality and high-speed image reconstruction: The present invention replaces the iterative method with a deep learning-based method. Compared with the iterative method, the hologram can be generated more quickly and the reconstruction quality can be comparable.

[0026] (2) Clear distinction between different depths: The present invention divides the target object into three layers of different depths and randomly reconstructs one layer in each training iteration. Therefore, it is possible to reconstruct high-quality images of 3D objects focused on different depths, with clear distinction between different depths.

[0027] (3) Obtaining a realistic blur effect: The present invention adds a corresponding blurred target image to the non-focused surface and uses multiple neural networks to jointly train the blur states of different layers. Therefore, compared with the existing deep learning-based methods, the non-focused surface will not produce unnatural artifacts, but a more realistic blur effect, and the image quality of the focused surface will not be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the present invention;

[0029] Figure 2 It is a demonstration diagram of the binary mask template used in step 1 and step 4 of the present invention for image masking and the Gaussian kernel for image blurring;

[0030] Figure 3 It is the framework of the neural network used as the complex amplitude generator and holographic encoder in the present invention, wherein Netdown is a downsampling block, Netup is an upsampling block, GN is a group normalization, ReLU is a ReLU activation function, Conv2d is a two-dimensional convolution, Delated Conv2d is a two-dimensional dilated convolution, and ConvTranspose2d is a two-dimensional transposed convolution;

[0031] Figure 4 1 is a light path diagram used in an embodiment of the present invention, wherein 1 is a laser light source, 2 is a filter, 3 is a polarizer, 4 is a beam expander, 5 is a collimating lens, 6 is a beam splitter, 7 is a spatial light modulator, 8 is a 4f system, 9 is an aperture stop, and 10 is a camera;

[0032] Figure 5 It is the simulation reconstruction result of different depths by the present invention and other methods;

[0033] Figure 6 It is a detailed comparison of the reconstruction results of the present invention and other methods. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with the accompanying drawings and examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent changes of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0035] Reference Figure 1 to Figure 6 , a method for reconstructing a real blurred three-dimensional hologram by jointly training multiple neural networks, comprising the following steps:

[0036] Step 1: For each training step, a random image is loaded from the 3800 RGB-D images in the MIT-CGH-4K training dataset and the pixel values ​​are normalized as the input of the neural network model. target , the target image is divided into three layers according to the uniform quantization of the depth map into three binary masks, as shown in Figure 2 As shown, the formula for the binary mask is:

[0037]

[0038] Among them, D(x,y) is the pixel value of the depth map, interval {j} is the quantization interval;

[0039] Randomly select one of the three layers As the focus plane input first neural network Gomplex_Generator1, ikk = 0, 1, 2, the remaining two layers As the non-focusing surface, ikkk=0, 1, 2∩ikkk≠ikk, according to the interval Δd or 2*Δd between the non-focusing surface and the focusing surface, the second or third neural network Gomplex_Generator2, Gomplex_Generator3 is input. The three neural network structures used are as follows: Figure 3 As shown in the figure, the neural network contains four downsampling blocks and four upsampling blocks. The downsampling block consists of two groups of group normalization, activation function, convolution kernel and a jump connection, and the upsampling block consists of two groups of group normalization, activation function and transposed convolution and a jump connection. The amplitude and phase distribution of the three-layer output are synthesized into complex amplitude fields and superimposed into a complex amplitude field.

[0040] Step 2: Convert the complex amplitude field The complex amplitude field of the SLM plane is obtained by propagating to the spatial light modulator plane using the angular spectrum method. The formula of the angular spectrum method is:

[0041] f ASM (Φ(x,y),z)=IFFT{FFT{e iΦ(x,y) u src (x,y)}·H(fx ,f y ,z)}

[0042]

[0043] where Φ(x,y) is the phase-only hologram, H(f x ,f y ,z) is the transfer function, λ is the wavelength, z is the distance between the target plane and the SLM plane, and f x ,f y is the spatial frequency, IFFT and FFT are fast Fourier transform and inverse fast Fourier transform respectively;

[0044] The complex amplitude field of the SLM plane is input into the fourth neural network Holo_Encoder. The structure of the holographic encoder Holo_Encoder is as follows: Figure 3 As shown, the output is only the phase hologram H o ;

[0045] Step 3: According to the distance between the focus plane and the SLM plane at the target plane (d+ikk*Δd), the phase-only hologram is back-propagated to the target plane to obtain the reconstructed amplitude and phase distribution;

[0046] Step 4: Blur the input image to generate the target image of the non-focused surface. The process is as follows Figure 2 As shown: for the input image a target The Gaussian kernels with different Gaussian radii σ are used for blurring, corresponding to the focus plane, the non-focus plane with a spacing of Δd from the focus plane, and the non-focus plane with a spacing of 2*Δd from the focus plane. The formula for Gaussian blurring is:

[0047]

[0048]

[0049] Among them, X, Y are the input and output images, u, v are the ranks of the Gaussian kernel, i, j are the ranks of the image, S is the normalization constant,

[0050] Load the blurred images into binary masks to obtain new three-layer target images The reconstructed image obtained by applying the reconstructed amplitude to three binary masks Respectively with the target image Bring in the mixed loss function calculation, the mixed loss function is expressed as:

[0051]

[0052] Here j represents the jth layer of the 3D object. represents the target amplitude of the j-th layer as a label, represents the reconstructed amplitude, P(·) represents the transformation to the perceptual feature space, represents the average amplitude of the complex amplitude field of the SLM plane, H represents the amplitude of the complex amplitude field of the SLM plane, s is the set scale factor, τ and γ are the set relative weights of the loss function;

[0053] Step 5: Get the total loss value loss val The network weights are updated through back propagation back to the network model to complete a training step. The training process is repeated continuously. When the data of the training set is traversed once, a training round ends. When the training round reaches the specified training round hyperparameter epoch, the loss function converges, and the structural similarity index SSIM of the reconstructed image no longer improves significantly. The training ends and the network model weights are saved.

[0054] Step 6: The reconstruction light path of the hologram is as follows Figure 4 As shown, the hologram calculated by the trained model is loaded onto the spatial light modulator, and the reconstructed image of the hologram is received at the corresponding diffraction distance through the 4f system.

[0055] In an embodiment of the present invention, using Figure 1 In the method shown, the target image resolution is set to 1072x1072, the spatial light modulator resolution is 1920x1080, and the wavelength is set to 520nm. The RGB image is divided into three layers according to the depth map, the interval Δd between the three layers is set to 5mm, and the distance between the target plane and the SLM plane is set to 150mm. The reconstruction distances of the three layers are 150mm, 155mm, and 160mm respectively. The simulation results are shown in Figure 5 As shown, the first row (ori) is the reconstruction result without constraining the non-focused surface, the second row (V1) is the reconstruction result using a neural network as a complex amplitude generator and constraining the non-focused surface, and the third row (V2) is the reconstruction result of the method of the present invention. The lower right corner of the image is the structural similarity SSIM index of the focused surface. It can be seen that the method proposed by the present invention can reconstruct images of focused surfaces at different depths with high quality. Figure 6 Yes Figure 5 The enlarged details of the reconstruction result at a medium depth of 160 mm show the result. It can be seen that for the non-focused surface, the present invention has no unnatural artifacts compared with other methods, has a more realistic blur effect, and does not reduce the imaging quality of the focused surface.

[0056] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks, characterized in that: The following steps are involved: Step 1: For each training step, load the RGB-D image, generate three binary masks according to the depth map, and divide the image into three layers; randomly select one layer as the focus plane and input it to the first complex amplitude generator, and the other two layers are input to the second and third complex amplitude generators respectively according to the distance from the focus plane; the complex amplitude fields of the three layers are output as one complex amplitude field; Step 2: Use the angular spectrum method to propagate the complex amplitude field to the SLM plane, and perform phase-only encoding on the complex amplitude hologram to obtain a phase-only hologram; Step 3: According to the distance between the focal plane and the SLM plane in the three layers, the phase-only hologram is back-propagated by the angular spectrum method to obtain the reconstructed image; Step 4: Use Gaussian convolution kernels of different sizes to blur the input image, and then multiply it by three binary masks to obtain three target images, corresponding to the focused surface and two non-focused surfaces; multiply the reconstructed image by the same mask, and substitute each part and the corresponding target image into the loss function to calculate the loss value; In step 4, the loss function is a combination of perceptual loss and mean square loss. The blurred images are loaded with binary masks to obtain new target images of three layers. The reconstructed image obtained by applying the reconstructed amplitude to three binary masks Respectively with the target image Bring in the mixed loss function calculation, the mixed loss function is expressed as: Here j represents the jth layer of the 3D object. represents the target amplitude of the j-th layer as a label, represents the reconstructed amplitude, P(·) represents the transformation to the perceptual feature space, represents the average amplitude of the complex amplitude field of the SLM plane, H represents the amplitude of the complex amplitude field of the SLM plane, s is the set scale factor, τ and γ are the set relative weights of the loss function; Step 5: Send the total loss value back to the network model to update the network weights through back propagation, and continuously optimize the hologram. When the set number of training rounds is reached, the training ends; Step 6: Input the RGB-D image into the trained model and load the generated computational hologram onto the spatial light modulator to obtain the reconstruction of the stereoscopic image.

2. The method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks as claimed in claim 1, characterized in that: In the step 1, the RGB-D image is two images: one is a common RGB three-channel color image, and the other is a Depth image, where each pixel value of the Depth is the actual distance between the sensor and the object.

3. The method for reconstructing a real fuzzy 3D hologram by jointly training multiple neural networks as claimed in claim 2, characterized in that: The RGB-D dataset used is MIT-CGH-4K.

4. The method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks according to any one of claims 1 to 3, characterized in that: The first, second and third complex amplitude generators refer to three neural networks with the same structure, which are used to train and generate complex amplitude fields of the focused plane and the non-focused plane respectively.

5. The method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks according to any one of claims 1 to 3, characterized in that: In the step 4, the binary mask is a binary four-dimensional tensor having the same size as the input image tensor, which is generated by dividing the normalized depth map into intervals at equal intervals according to pixel values.

6. The method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks according to any one of claims 1 to 3, characterized in that: In the step 1, the three complex amplitude fields are synthesized by splicing the amplitudes and phases of the three complex amplitude fields along the channel dimension and inputting them into a small neural network that only contains one downsampling and one upsampling. Finally, the amplitudes and phases output by the network are spliced ​​along the channel dimension to obtain a complex amplitude field containing three layers of information.

7. The method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks according to any one of claims 1 to 3, characterized in that: In the step 2, the phase-only encoding method is to input the complex amplitude hologram into the neural network serving as a holographic encoder.

8. The method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks according to any one of claims 1 to 3, characterized in that: In the step three, the reconstructed image refers to the amplitude of the reconstructed complex amplitude light field.

9. The method for reconstructing a real fuzzy three-dimensional hologram by jointly training multiple neural networks according to any one of claims 1 to 3, characterized in that: In step six, the trained model refers to a model that has been trained for a specified number of times, at which point the loss function of the training set converges, and the quality of the reconstructed image on the test set meets the requirements of the structural similarity index, and continued training has no significant improvement on the results; the reconstruction of the stereoscopic image refers to receiving the reconstructed holographic image at the corresponding diffraction distance.

Images