Controllable tilt hologram reconstruction method based on nufft and unet

Abstract

The application discloses a controllable tilt hologram reconstruction method based on NUFFT and Unet, and the technical idea is as follows: regarding the process of generating a hologram by the Unet network as an optimization process, firstly, an initial hologram is directly generated by the Unet network, then a desired tilt image is obtained by a preset propagation model, and then the hologram and the Unet network are optimized by using a gradient descent algorithm of a loss function; in the optimization process, the tilt degree of the generated image can be controlled by using the preset propagation model. Compared with a traditional method for obtaining a hologram, the hologram reconstructed by the application has the advantages of high image quality, continuous depth clues, no user perspective limitation and the like, can give users a better experience, and is expected to obtain practical application in AR / XR scenes and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of holographic display, and particularly relates to a controllable tilt hologram reconstruction method combining NUFFT and Unet. BACKGROUND

[0002] With the development of deep learning and the improvement of terminal device computing power, deep learning has been applied to many scenarios. Holographic display can reconstruct real three-dimensional light field, providing real depth cues for users, and is considered to be the most promising technology in the next generation of AR, VR or XR, which will revolutionize the fields of industry, education, gaming, and medicine. However, the traditional algorithm still needs multiple iterations when generating holograms, which takes a long time to calculate and is difficult to apply to real-life scenarios. In the currently commercialized AR, VR or XR products, different images are often provided to the left and right eyes to achieve binocular visual differences, or the background image is blurred to provide psychological hints to provide users with the experience of viewing stereoscopic images. The biggest problem of these technologies based on planar light display is that they cannot provide monocular depth cues, which will cause visual fatigue, dizziness, and nausea caused by vergence conflict after long-term use of the product. Holographic three-dimensional display is a display method that uses wavefront information to reconstruct three-dimensional scenes, which can accurately reconstruct the intensity and depth information of three-dimensional targets. In recent years, the development of computational holography technology can eliminate the complex interference recording process in traditional holographic technology, and the holographic display process can be realized by using a spatial light modulator and a computer numerically calculated hologram. Therefore, combining deep learning with holographic display and eliminating the iteration process when generating holograms will become the preferred solution for future holographic display development.

[0003] In computational holography, in order to obtain the new light field distribution after the light field propagates a certain distance, a diffraction algorithm is needed. Common diffraction algorithms include point source method and angular spectrum method. The point source method calculates the diffraction of each point in the target field and superimposes it to obtain the diffraction of the plane light field distribution. The advantage is that it can calculate the diffraction of three-dimensional targets, but the disadvantage is that the calculation speed is slow. The angular spectrum method calculates the change of the angular spectrum of the plane light field after propagating in a certain direction by performing Fourier analysis on the complex field distribution. The advantage is that the calculation speed is fast, but the disadvantage is that it can only calculate the diffraction of parallel light, and it is difficult to provide depth cues for continuous propagation of light field in space, which also greatly limits the observation angle of users in use. SUMMARY

[0004] The application aims to provide a controllable slant hologram reconstruction method based on NUFFT and Unet, and the technical idea is: regarding the process of generating hologram by Unet network as an optimization process, first generating an initial hologram by Unet network, then obtaining a desired slant image by a preset propagation model, and then optimizing the hologram and Unet network by gradient descent algorithm of loss function; in the optimization process, the slant degree of the generated image can be controlled by using the preset propagation model. Compared with the traditional method of obtaining hologram, the hologram reconstructed by the application has the advantages of high image quality, continuous depth clues, no user perspective limitation, etc., can give users a better experience, and is expected to be applied in AR / XR scenes.

[0005] The technical scheme of the application is as follows:

[0006] The controllable slant hologram reconstruction method based on NUFFT and Unet includes the following four steps:

[0007] Step 1: generating the phase value of the initial hologram by Unet network according to the input target image, and then obtaining the initial hologram by complex operation. For the phase value of the initial hologram, an alternative solution is to use a random phase in (-pi, pi) as the phase value of the initial hologram.

[0008] Step 2: reconstructing the holographic plane image from the initial hologram by using the traditional diffraction algorithm (angular spectrum method) to obtain the reconstructed complex amplitude light field, then obtaining the slant light field after rotating the holographic plane image by a certain angle by NUFFT (non-uniform Fourier transform) method, and superimposing multiple reconstructed plane images to obtain the reconstructed stereoscopic image. Finally, the total amplitude loss value of the stereoscopic image is obtained by substituting the reconstructed amplitude and the target stereoscopic image into the loss function.

[0009] Step 3: calculating the gradient of the loss value to the hologram Ho, and optimizing the initial hologram according to the back propagation algorithm. If Unet is to be trained, the gradient is propagated to Unet network again, and Unet network is continuously optimized. When the number of optimization times meets the set number of times, or the quality of the reconstructed image meets the requirements, the optimization process is stopped.

[0010] Step 4: loading the optimized hologram or the hologram generated by using Unet network to a spatial light modulator to obtain the reconstruction of the stereoscopic image.

[0011] Further, the Unet network refers to a pre-set image generation structure that can be optimized, and the parameters thereof are determined by actual scene requirements. The input and output sizes should be multiples of 2 raised to the power of n, and n represents the number of network layers.

[0012] Further, the complex operation refers to the complex light field obtained by superimposing the light field with a constant amplitude and the generated phase value.

[0013] Further, the random phase within (-pi, pi) refers to any value within its range.

[0014] Further, the reconstructed complex amplitude light field obtained by reconstructing the holographic image from the initial hologram using the diffraction algorithm refers to the use of the angular spectrum method or other differentiable diffraction algorithm to calculate the complex amplitude light field distribution of the hologram reconstructed in the target plane.

[0015] Further, the NUFFT, i.e. non-uniform Fourier transform, refers to a method that can achieve the effect of fast Fourier transform by inputting non-uniform sampling frequency when the spatial frequency of the frequency domain sampling is no longer uniform.

[0016] Further, the tilt degree of the reconstructed image refers to the angle between the plane of the reconstructed image and the plane of the spatial light modulator. Since in the rotation of the light field along the central axis at any position of the image, the change of the frequency domain is reflected in the change of the sampling frequency, different sampling frequencies can control the tilt degree of the image. Therefore, the tilt degree of the reconstructed image can be controlled by changing the sampling frequency.

[0017] Further, the reconstructed volume image refers to the superposition of the reconstructed holographic images received at the corresponding diffraction planes, while considering and only considering the part that needs to be optimized in the superposition process, and avoiding repeated calculation of image content in the form of zero for other parts.

[0018] Further, the initial hologram or the hologram generated by Unet refers to a pure phase hologram. Since the existing commercial spatial light modulator is only of pure phase type or pure amplitude type, and the pure phase type spatial light modulator has higher diffraction efficiency, the pure phase hologram is calculated in the present application.

[0019] Further, the loss function refers to the mean square loss function.

[0020] Further, the total loss value refers to the loss value obtained by substituting the amplitude of each layer of the reconstructed image in the stereoscopic image into the loss function, and then weightedly summing the total loss value according to the area of the image.

[0021] Further, the gradient calculation of the loss value on the hologram Ho and the optimization of the initial hologram according to the back propagation algorithm refer to the gradient calculation of the total loss function loss on the pure phase hologram Ho in the initial plane to obtain the gradient value According to the back propagation formula Hi = Ho-lr*Grad, the updated pure phase hologram Hi is obtained, wherein lr is the learning rate.

[0022] Further, the continuous optimization refers to the process of repeatedly reconstructing the hologram, calculating the total loss value, performing gradient calculation, and finally updating the hologram according to the back propagation algorithm.

[0023] Further, the stopping of the optimization process when the number of optimizations meets the set number or the quality of the reconstructed image meets the requirement refers to stopping the optimization when the number of optimizations reaches the set upper limit or the score of the reconstructed image calculated by the image quality evaluation function reaches the set threshold, and retaining the hologram obtained by the last iteration as the optimized hologram.

[0024] Further, the spatial light modulator refers to a pure phase spatial light modulator. When the first-order diffraction image and the second-order diffraction image are superimposed on the reconstruction plane, a 4f system filter is also needed to eliminate the second-order diffraction and higher-order diffraction images.

[0025] The present application has the following beneficial effects:

[0026] 1) Obtain an adjustable tilted image: The present application is an iterative optimization method, which can continuously optimize the reconstructed image of the calculated hologram to the set target by adjusting the tilt of the reconstructed image in the reconstruction process. Since the tilt of the reconstructed image is determined by the sampling frequency of the image in the propagation process, the tilt of the image can be adjusted by changing the sampling frequency.

[0027] 2) Obtain high-quality image reconstruction: The present application is an iterative optimization method, and only considers the amplitude factor in the loss function. Therefore, the method has better quality than the reconstructed hologram obtained by the traditional Gerchberg Saxton (GS) algorithm. At the same time, the GS algorithm is very easy to produce phase aliasing in the iteration process, while the optimization method based on gradient descent does not.

[0028] 3) Obtain real display effect: The present application uses non-uniform Fourier transform to reconstruct the image to obtain the tilted display effect, which can give the user continuous depth cues, and can also compensate for the tilted viewing angle of the user due to environmental factors and other problems when watching, giving the user a better viewing experience.

[0029] 4) Obtain faster generation speed: The present application can train a preset hologram model through the Unet network, and use the trained model to quickly generate a hologram according to the input image to meet the real-time interaction demand in real life. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a flowchart of a controllable tilted hologram reconstruction method based on NUFFT and Unet according to the present application;

[0031] Figure 2 This is a flowchart of the Unet network generating phase in step 1 of the present invention;

[0032] Figure 3 This is the optical path diagram used in the embodiments of the present invention;

[0033] Figure 4 This is a comparison chart of the simulation reconstruction results of this invention and the simulation reconstruction effects of existing methods;

[0034] Figure 5 This is a simulation reconstruction result of a single-layer image using the Unet network, as described in this invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that these embodiments are only for illustrating the present invention and are not intended to limit the scope of the present invention. After reading the present invention, those skilled in the art will find that various equivalent modifications of the present invention fall within the scope defined by the appended claims.

[0036] This invention proposes a controllable tilt hologram reconstruction method based on NUFFT and Unet. The flowchart of this invention is attached. Figure 1 As shown, it includes the following steps:

[0037] Step 1: Generate the initial phase using the Unet network based on the target image to be displayed. The Unet network is structured as follows: Figure 2 As shown, firstly, features from n layers of the image are extracted using a convolutional kernel with a stride of 1 and a size of 3×3, based on an initial value of n. During convolution, reflection padding is used to prevent the loss of edge information, followed by activation using the ReLU function. Then, ten layers of downsampling and ten layers of upsampling are performed. For each downsampling layer, features are first extracted using a convolutional kernel with a stride of 1 and a size of 3×3, followed by downsampling using a convolutional kernel with a stride of 2 and a size of 4×4. Reflection padding is used in each convolution process, followed by normalization and activation using the ReLU function. The ReLU function is expressed as: f(x,k)=max(0,x)+k*min(0,x), where k represents the slope of the negative portion when the input is negative. The upsampling process is the reverse of the downsampling process: first, a convolutional kernel with a stride of 2 and a size of 4×4 is used, followed by deconvolution using a convolutional kernel with a stride of 1 and a size of 3×3. Finally, the initial phase is obtained by deconvolution using a convolution kernel with a stride of 1 and a size of 3×3.

[0038] If random phases are used directly for iteration, a random phase within (-π, π) is directly generated as the initial hologram.

[0039] Step 2: The complex amplitude light field distribution u0(x, y) of the spatial light modulator plane is synthesized by the phase distribution of the initial hologram Ho and the amplitude distribution with all amplitude values equal to 1, and the complex amplitude light field distribution u0(x, y) is obtained by the angular spectrum method

[0040]

[0041] The complex amplitude light field distribution u of each reconstruction plane is calculated i (i = 1, 2,..., N), wherein

[0042]

[0043] represents the transfer function in the angular spectrum method, λ represents the wavelength, f x ,f y represents the spatial frequency, and z represents the light field transmission distance.

[0044] The expression of the spatial frequency f' of the tilted plane is as follows: x ,f′ y

[0045]

[0046]

[0047] wherein λ represents the wavelength, f x ,f y represents the spatial frequency, θ x represents the tilt angle generated by the rotation of the image along the y axis, and θ y represents the tilt angle generated by the rotation of the image along the x axis. The frequency domain image of the tilted plane is obtained by the method of frequency domain interpolation reconstruction of the frequency domain image of the original image. Zero padding is added when performing non-uniform Fourier transform to prevent phase aliasing problems in the calculation process. The light field u title of the tilted plane is obtained by inverse Fourier transform of the frequency domain image.

[0048] Since the contents to be displayed on each reconstruction plane are different, the complex amplitude light field distribution u i (i = 1, 2,..., N) of each reconstruction plane is multiplied by Mask i , and each Mask is determined by the content to be displayed.

[0049] The amplitude loss value loss i is obtained by substituting the reconstructed amplitude and the target stereoscopic image into the mean square loss function MSELoss; the total loss value loss sum is obtained by summing the amplitude loss value of each reconstruction plane, and the expression of loss sum is as follows:

[0050] ​

[0051] S i is a scale factor proportional to the size of each layer image, and s0 is a learnable scale factor.

[0052] Step 3: the total loss function loss calculated in step 2 is used to update the hologram Ho. sum Gradient calculation is performed on the pure phase hologram Ho to obtain the gradient value According to the back propagation formula Hi = Ho - lr*Grad, the updated pure phase hologram Hi is obtained, wherein lr is the learning rate. The reconstructed hologram, the total loss value calculation, the gradient calculation, and finally the update of the hologram by the back propagation algorithm are regarded as an optimization process of the hologram, and the process of optimizing the hologram is repeatedly optimized until the optimization times reach the set upper limit, and the hologram obtained in the last iteration is retained as the optimized hologram.

[0053] The reconstruction light path of the hologram is shown in Figure 3 The optimized hologram is loaded onto the spatial light modulator, and the reconstructed image of the hologram is received at the corresponding diffraction distance through the 4f system.

[0054] In the embodiment of the present application, the method flow shown in the accompanying Figure 1 The target image resolution is set to 1360x880, the resolution of the spatial light modulator is 1920x1080, and the wavelength is set to 520nm. Three reconstruction planes are used for layered stereoscopic images, and the numbers and bicycle patterns are at two fixed depths, the first plane imaging depth is set to 60mm, and the second plane imaging depth is set to 62mm. If the tilted propagation model is used, the arrow is in a tilted plane, and if only the angular spectrum method is used, the arrow imaging depth is set to 64mm. The simulation results are shown in the accompanying Figure 4 The simulation reconstruction images at 60mm, 62mm and 64mm of the prior art and the present application are shown. The first row is the simulation result using the traditional GS encoding method; the second row is the simulation result obtained by using the traditional stochastic gradient descent method; and the third row is the simulation result obtained by using the method of the present application. It can be seen that the method of the present application can provide better depth cues compared with the traditional GS algorithm, and the image reconstruction quality is much better. Compared with the traditional stochastic gradient descent method, the image reconstruction ability is similar, but the method can better provide depth cues. It can provide better and more realistic viewing experience for users.

[0055] At the same time, the Unet network shown in the accompanying Figure 2 is used to train the DIV2K dataset. The accompanying Figure 5For the reconstruction images of hologram obtained by using Unet network at 8mm, 9mm, 10mm, 11mm and 12mm, the demonstration images are not included in the training set. The results show that the reconstruction quality of hologram is good, and at the same time, with the change of depth, the clear area also changes accordingly, and the reconstructed image provides obvious depth cue. At the same time, using Unet network, the hologram generation time is greatly reduced, and the hologram generation only needs 1 second. With the optimization of subsequent network and the improvement of equipment performance, the hologram generation time will be greatly shortened, and it is expected to be applied in practical scenarios.

Claims

1. A controllable tilt hologram reconstruction method based on NUFFT and Unet, characterized in that, The method comprises the following steps: Step 1: generating phase values of an initial hologram by a Unet network according to an input target image, and then obtaining the initial hologram by complex operation; Step 2: reconstructing holographic plane images from the initial hologram by an angular spectrum method to obtain reconstructed complex amplitude light fields, and then obtaining a tilted light field after rotation of the holographic plane images by a NUFFT (non-uniform Fourier transform) method, superimposing a plurality of reconstructed plane images to obtain a reconstructed stereoscopic image, and substituting the reconstructed amplitude and the target stereoscopic image into a loss function to obtain a total amplitude loss value of the stereoscopic image; Step 3: calculating a gradient of the loss value with respect to the hologram Ho, and optimizing the initial hologram according to a back propagation algorithm, if the Unet network is to be trained, propagating the gradient to the Unet network again, continuously optimizing the Unet network, and stopping the optimization process when the number of optimizations meets a set number or the quality of the reconstructed image meets a requirement; Step 4: loading the optimized hologram or the hologram generated by using the Unet network to a spatial light modulator to obtain a reconstruction of the stereoscopic image.

2. The controllable tilt hologram reconstruction method based on NUFFT and Unet according to claim 1, wherein, In step 1, the Unet network is used to generate a hologram representing only a phase, and the hologram generated by the Unet network is a pure phase hologram; for the phase value of the initial hologram, a random phase in [-π, π] is used as the phase value of the initial hologram.

3. The controllable tilt hologram reconstruction method based on NUFFT and Unet according to claim 1, wherein, In step 2, the NUFFT (non-uniform Fourier transform) performs a frequency domain to space domain transformation according to a calculated sampling frequency, and adds zero padding in the calculation process to prevent phase aliasing.

4. The controllable tilt hologram reconstruction method based on NUFFT and Unet according to claim 3, characterized in that, The sampling frequency in step 2 is related to an angle of the reconstructed light field relative to the spatial light modulator, i.e., an angle of tilting.

5. The controllable tilt hologram reconstruction method based on NUFFT and Unet according to claim 4, characterized in that, The sampling frequency in step 2 is calculated from an original spatial frequency and the angle of tilting.

6. The controllable tilt hologram reconstruction method based on NUFFT and Unet according to claim 1, wherein, The total amplitude loss value of the reconstructed stereoscopic image in step 2 is a weighted superposition of loss values of parts to be optimized in the received reconstructed holographic image at corresponding diffraction planes, and non-optimized parts are avoided from repeated calculation of image content by being set to zero.

7. The controllable tilt hologram reconstruction method based on NUFFT and Unet according to claim 1, wherein, In step 4, the spatial light modulator is a pure phase spatial light modulator, and a 4f system filter is needed to eliminate secondary diffraction and higher order diffraction images when a first order diffraction image and a second order diffraction image are superimposed on a reconstruction plane.

Images