A hologram generation method based on deep learning and angular spectrum method

Abstract

The application discloses a hologram generation method based on deep learning and angular spectrum method, and belongs to the field of deep learning and digital holography.The application solves the problems of long time consumption, high cost, and difficulty in guaranteeing the quality of obtained data in the existing training data set acquisition method, and the problem of poor quality of a reconstructed image obtained by using a denoised hologram output based on the existing deep learning method.The application proposes an HG-Net network structure, uses multiple input intensity distributions and holograms obtained by ASM propagation as training data sets by randomly adjusting the density and size of the speckle, so as to improve the complexity of the training data set.The HG-Net network is trained by using the training data set, so as to improve the generalization of the model.The real part hologram and the imaginary part hologram output by the HG-Net network are combined to obtain a generated hologram, and numerical and optical reconstruction is carried out based on the generated hologram.The method can be applied to hologram generation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning and digital holography, and particularly relates to a hologram generation method based on deep learning and angular spectrum method (ASM). BACKGROUND

[0002] In digital holography, when a laser source is used to digitally record a hologram, due to the high coherence of the laser source and the rough surface of the object, coherent superposition is caused when reflection occurs, so that speckle noise is generated in the obtained digital hologram, resulting in a decrease in the image quality obtained by numerical reconstruction based on the digital hologram. Therefore, how to effectively remove the speckle noise while preserving the image information of the digital hologram as much as possible to improve the quality of the reconstructed image is a current research hotspot.

[0003] At present, there are three common methods to eliminate speckle noise in digital hologram: the first method is to use a partially coherent light source or a low coherence light source, which can reduce speckle noise by replacing the laser light source with a partially coherent light source. However, this requires a high-brightness partially coherent light source, otherwise the quality of the reconstructed image will be reduced. The second method is to use digital image processing technology, which can effectively remove speckle noise through traditional filters, resampling masks, and BM3D methods. However, this method will damage the details of the image and reduce the quality of the reconstructed image. The third method is to use deep learning to remove speckle noise. K Zhang et al. proposed a denoising convolutional neural network based on residual learning and batch normalization to reduce general noise. WONSEOK JEON et al. used a multi-scale convolutional neural network to realize speckle noise reduction in digital holographic images. Z j Shen et al. used a dual-path parallel convolutional neural network to remove speckle noise in different types of OCT images. Q Fang et al. used a conditional generative adversarial network based on deep learning to remove speckle noise. Compared with the above two traditional methods, the deep learning-based method does not require complex and strict experiments. Instead, it trains the neural network based on a pre-acquired data set and directly maps the denoised hologram. Moreover, with the breakthrough in GPU computing speed, the learning speed and performance are improved, realizing real-time application. Juyeon Seo et al. proposed a hologram compression network based on deep learning, which greatly improved the quality of full-complex hologram numerical reconstruction. Qingwei Liu et al. proposed a dual-channel parallel neural network (DCPNet) for generating phase-only holograms, which were encoded into two real-valued phase units through a double-phase amplitude encoding method. Experimental data showed that DCPNet could generate 2k resolution phase-only holograms within 36ms. Yun Liu et al. proposed ANLVENet based on FFDNet, combined with an asymmetric pyramid non-local block and an edge extraction module to suppress speckle noise in holographic phase fringe patterns. Experimental data showed that the ANLVENet method not only had better advantages in speckle noise suppression, but also preserved more details of the image edges.

[0004] In recent years, the application of deep learning in the field of digital holography has made significant progress, mainly due to the advancement of deep learning theory and the breakthrough of graphics processing unit (GPU) technology. In holographic display technology, deep learning models such as convolutional neural networks (CNN), generative adversarial networks (GAN), and recurrent neural networks (RNN) have shown excellent performance. However, the training of deep learning models relies on the input of a large amount of training data, which poses a challenge in the field of digital holography. Currently, there is a lack of widely available open-source data sets in digital holography, resulting in most researchers having to collect training data through time-consuming and costly optical experiments. This method of obtaining training data not only is inefficient, but also often makes it difficult to ensure the quality of the data. Moreover, the quality of the reconstructed image obtained from the denoised hologram output by the existing deep learning method is still poor. Therefore, the development and sharing of high-quality open-source data sets are crucial for promoting the development of digital holography, and proposing a new hologram generation method is essential for improving the quality of the reconstructed image. SUMMARY

[0005] The purpose of the present application is to solve the problems of long time-consuming, high cost, and difficulty in ensuring the quality of the obtained data in the existing training data set acquisition method, and the poor quality of the reconstructed image obtained from the denoised hologram output by the existing deep learning method. Therefore, a hologram generation method based on deep learning and angular spectrum method is proposed.

[0006] The technical scheme adopted by the present application to solve the above technical problems is: a hologram generation method based on deep learning and angular spectrum method, the method specifically comprises the following steps:

[0007] Step one, randomly generate each speckle image with different speckle density and speckle size;

[0008] Step two, take each speckle image generated in step one as the input of the angular spectrum method, simulate the propagation process of the angular spectrum method to obtain each complex amplitude hologram; extract the real part and the imaginary part of each complex amplitude hologram to obtain the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram;

[0009] Take the real part hologram corresponding to each complex amplitude hologram as the input of the real part hologram training data set, and take the imaginary part hologram corresponding to each complex amplitude hologram as the input of the imaginary part hologram training data set;

[0010] Step three, obtain each complex amplitude hologram without speckle noise from each complex amplitude hologram in step two and each speckle image in step one, extract the real part and the imaginary part of each complex amplitude hologram without speckle noise to obtain the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram without speckle noise;

[0011]

[0011] The real part hologram corresponding to each complex amplitude hologram after removing speckle noise is taken as a label of a real part hologram training data set, and the imaginary part hologram corresponding to each complex amplitude hologram after removing speckle noise is taken as a label of an imaginary part hologram training data set;

[0012] Step four, constructing a first HG-Net network and a second HG-Net network, training the first HG-Net network by using the input and label of the real part hologram training data set, and training the second HG-Net network by using the input and label of the imaginary part hologram training data set;

[0013] Step five, extracting the real part and the imaginary part of a digital hologram to be removed of speckle noise to obtain an extracted real part hologram I1 and an extracted imaginary part hologram I2;

[0014] Taking the real part hologram I1 as the input of the trained first HG-Net network, taking the real part hologram output by the first HG-Net network as I3, taking the imaginary part hologram I2 as the input of the trained second HG-Net network, and taking the imaginary part hologram output by the second HG-Net network as I4;

[0015] Step six, combining the real part hologram I3 and the imaginary part hologram I4 to obtain a combined complex amplitude hologram.

[0016] Further, in the step one, the size of each generated speckle image is 512*512.

[0017] Further, the first HG-Net network comprises two parts of encoding and decoding; wherein:

[0018] The encoding part comprises a first ResBlock module, a first downsampling unit, a second downsampling unit, a third downsampling unit and a fourth downsampling unit;

[0019] The decoding part comprises a first upsampling unit, a second upsampling unit, a third upsampling unit, a fourth upsampling unit and a second ResBlock module.

[0020] Further, the working process of the first HG-Net network is as follows:

[0021] In the first HG-Net network, the input image first passes through the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit and the fourth downsampling unit in sequence, and then the output of the fourth downsampling unit is taken as the input of the first upsampling unit;

[0022] The output of the first upsampling unit and the output of the third downsampling unit are taken as the input of the second upsampling unit;

[0023] Taking the output of the second upsampling unit and the output of the second downsampling unit as the input of the third upsampling unit;

[0024] Taking the output of the third upsampling unit and the output of the first downsampling unit as the input of the fourth upsampling unit;

[0025] The output of the fourth upsampling unit and the output of the first ResBlock module are used as the input of the second ResBlock module;

[0026] The output of the second ResBlock module is used as the output of the first HG-Net network.

[0027] Furthermore, the first ResBlock module sequentially includes a first convolutional layer, a first BN layer, a first ReLU layer, a second convolutional layer, a second BN layer, and a second ReLU layer;

[0028] The input of the first ResBlock module passes through the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence. The input of the first ResBlock module and the output of the second ReLU activation function layer are then spliced ​​together, and the splicing result is used as the output of the first ResBlock module.

[0029] Furthermore, the first downsampling unit sequentially includes a maximum pooling layer, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer;

[0030] The input of the first downsampling unit passes through the maximum pooling layer, the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence. The input of the first downsampling unit and the output of the second ReLU activation function layer are then spliced ​​together, and the spliced ​​result is used as the output of the first downsampling unit;

[0031] The second down-sampling unit, the third down-sampling unit, and the fourth down-sampling unit have the same structure and working process as the first down-sampling unit.

[0032] Furthermore, the first upsampling unit sequentially includes a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer;

[0033] The input of the first up-sampling unit sequentially passes through a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer and a second ReLU activation function layer, and then the input of the first up-sampling unit and the output of the second ReLU activation function layer are spliced, and the spliced result is taken as the output of the first up-sampling unit;

[0034] The second up-sampling unit, the third up-sampling unit and the fourth up-sampling unit have the same structure and working process as the first up-sampling unit.

[0035] Preferably, in the fourth step, the optimizer used in the training is RMSProp.

[0036] Preferably, in the fourth step, the loss function used in the training is MSEloss.

[0037] Preferably, in the fourth step, the learning rate is set to 0.001, the batch_size is set to 8, and the epoch is set to 100.

[0038] The present application has the following beneficial effects:

[0039] The present application proposes an HG-Net network structure, which improves the complexity of the data set by randomly adjusting the density and size of the spots, using multiple input intensity distributions and holograms obtained by ASM propagation as training data sets, and shortens the time required to obtain the training data set, reduces the cost and ensures the quality of the obtained data. Training the HG-Net network using the data set can improve the generalization of the model. The real part hologram and the imaginary part hologram output by the HG-Net network are combined to obtain the generated hologram. Numerical and optical reconstruction are performed based on the generated hologram, and the PSNR, SSI, SSIM and SMPI of the reconstructed image are calculated. The results show that, compared with the traditional method, the present application can simply and efficiently generate high-quality holograms, thereby improving the quality of the reconstructed image. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of a hologram generation method based on deep learning and angular spectrum method according to the present application is shown in the figure;

[0041] Figure 2 A schematic diagram of the HG-Net network structure is shown in the figure;

[0042] In the figure, N represents the number of channels;

[0043] Figure 3 A schematic diagram of the loss change in the training process is shown in the figure;

[0044] Figure 4(a) is a real part hologram generated by the first HG-Net network Figure 1 ;

[0045] Figure 4(b) is a real part hologram generated by the first HG-Net network Figure 2 ;

[0046] Figure 4(c) is an imaginary part hologram generated by the second HG-Net network Figure 1 ;

[0047] Figure 4(d) is an imaginary part hologram generated by the second HG-Net network Figure 2 ;

[0048] Figure 4(e) is a real part hologram generated by the angular spectrum method Figure 1 ;

[0049] Figure 4(f) is a real part hologram generated by the angular spectrum method Figure 2 ;

[0050] Figure 4(g) is an imaginary part hologram generated by the angular spectrum method Figure 1 ;

[0051] Figure 4(h) is an imaginary part hologram generated by the angular spectrum method Figure 2 ;

[0052] Figure 5(a) is the original hologram;

[0053] Figure 5(b) is the reconstructed image one;

[0054] Figure 5(c) is the reconstructed image two;

[0055] Figure 5(d) is the reconstructed image three;

[0056] Figure 5(e) is the reconstructed image four;

[0057] Figure 6(a) is a 3D view of Figure 5(b);

[0058] Figure 6(b) is a 3D view of Figure 5(c);

[0059] Figure 6(c) is a 3D view of Figure 5(d);

[0060] Figure 6(d) is a 3D view of Figure 5(e);

[0061] Figure 7(a) is an image histogram of Figure 5(b);

[0062] Figure 7(b) is an image histogram of Figure 5(c);

[0063] Figure 7(c) is an image histogram of Figure 5(d);

[0064] Figure 7(d) is an image histogram of Figure 5(e). DETAILED DESCRIPTION

[0065] DETAILED DESCRIPTION Figure 1 The present embodiment is described. The hologram generation method based on deep learning and angular spectrum method described in the present embodiment specifically includes the following steps:

[0066] Step one, randomly generate each speckle image with different speckle density and speckle size using matlab;

[0067] By randomly adjusting the density and size of the speckles and combining the speckle density and size in different ways in the data set, the diversity of the data set is increased, which can improve the generalization ability of the model when facing different situations;

[0068] Step two, each speckle image generated in step one is used as the input of the angular spectrum method, and the process of angular spectrum method propagation is simulated using matlab to obtain each complex amplitude hologram; the real part and the imaginary part of each complex amplitude hologram are extracted to obtain the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram;

[0069] The real part hologram corresponding to each complex amplitude hologram is used as the input of the real part hologram training data set, and the imaginary part hologram corresponding to each complex amplitude hologram is used as the input of the imaginary part hologram training data set;

[0070] Step three, according to each complex amplitude hologram in step two and each speckle image in step one, each complex amplitude hologram without speckle noise is obtained, and the real part and the imaginary part of each complex amplitude hologram without speckle noise are extracted to obtain the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram without speckle noise;

[0071] The real part hologram corresponding to each complex amplitude hologram without speckle noise is used as the label of the real part hologram training data set, and the imaginary part hologram corresponding to each complex amplitude hologram without speckle noise is used as the label of the imaginary part hologram training data set;

[0072] Step four, construct the first HG-Net network and the second HG-Net network, train the first HG-Net network using the input and label of the real part hologram training data set, and train the second HG-Net network using the input and label of the imaginary part hologram training data set;

[0073] Step five, extract the real part and the imaginary part of the digital hologram to be removed from the speckle noise to obtain the extracted real part hologram I1 and the imaginary part hologram I2;

[0074] The real part hologram I1 is taken as the input of the trained first HG-Net network, the real part hologram output by the first HG-Net network is recorded as I3, the imaginary part hologram I2 is taken as the input of the trained second HG-Net network, and the imaginary part hologram output by the second HG-Net network is recorded as I4; wherein I3 and I4 are the holograms after denoising.

[0075] Step six, the real part hologram I3 is combined with the imaginary part hologram I4 to obtain the combined complex amplitude hologram.

[0076] The method only needs to train the first HG-Net network and the second HG-Net network, and can generate a hologram with higher quality compared with the traditional algorithm, while significantly reducing the calculation cost, and can obtain better hologram generation effect with less resource investment.

[0077] Specific implementation method two: different from the specific implementation method one, in the step one, the size of each generated speckle image is 512*512.

[0078] The other steps and parameters are the same as those in the specific implementation method one.

[0079] Specific implementation method three: in combination with Figure 2 The present embodiment is described. The first HG-Net network includes an encoding (Encoder) and a decoding (Decoder) in the present embodiment, which is different from the specific implementation method one or two; wherein:

[0080] The encoding part includes a first ResBlock module, a first down-sampling unit, a second down-sampling unit, a third down-sampling unit and a fourth down-sampling unit.

[0081] The decoding part includes a first up-sampling unit, a second up-sampling unit, a third up-sampling unit, a fourth up-sampling unit and a second ResBlock module.

[0082] The other steps and parameters are the same as those in the specific implementation method one or two.

[0083] The first Resblock module is used in the encoding part of the present application instead of the original convolution module, which improves the feature extraction capability of the network, the bilinear interpolation method is used in the decoding part to restore the resolution of the original image, and the number of channels is reduced. Under the premise of not reducing the network performance, the parameter amount is reduced by about 2 times by reducing the number of channels, so that the network is lightweight.

[0084] The application adopts four down-sampling units to obtain high-level semantic information. In order to reduce the spatial information loss caused by the down-sampling process, the output of the down-sampling unit is taken as the input of the up-sampling unit through the skip-layer connection. Through the fusion of deep features and shallow features, the feature map recovered by the up-sampling process contains more low-level semantic information, thereby ensuring the quality of the generated hologram. The HG-Net network of the application can be applied to remove speckle noise of the hologram, and can retain the edge part of the image without losing the resolution of the image.

[0085] Specific implementation four: different from one of the specific implementations one to three, the working process of the first HG-Net network is:

[0086] In the first HG-Net network, the input image first passes through the first ResBlock module, the first down-sampling unit, the second down-sampling unit, the third down-sampling unit and the fourth down-sampling unit in turn, and then the output of the fourth down-sampling unit is taken as the input of the first up-sampling unit;

[0087] The output of the first up-sampling unit and the output of the third down-sampling unit are taken as the input of the second up-sampling unit;

[0088] The output of the second up-sampling unit and the output of the second down-sampling unit are taken as the input of the third up-sampling unit;

[0089] The output of the third up-sampling unit and the output of the first down-sampling unit are taken as the input of the fourth up-sampling unit;

[0090] The output of the fourth up-sampling unit and the output of the first ResBlock module are taken as the input of the second ResBlock module;

[0091] The output of the second ResBlock module is taken as the output of the first HG-Net network.

[0092] The other steps and parameters are the same as one of the specific implementations one to three.

[0093] The second HG-Net network has the same structure and working process as the first HG-Net network. The application improves the accuracy by introducing residual connection, i.e. directly connecting the encoder and the decoder, which reduces the processing time to a certain extent. In this way, the information lost in different layers of the encoding part can be retained, and additional parameters and operations are not increased when relearning the lost information.

[0094] Specific implementation five: different from one of the specific implementations one to four, the first ResBlock module comprises a first convolutional layer, a first BN layer, a first ReLU layer, a second convolutional layer, a second BN layer and a second ReLU layer in turn.

[0095] The input of the first ResBlock module sequentially passes through a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer and a second ReLU activation function layer, and then the input of the first ResBlock module and the output of the second ReLU activation function layer are spliced, and the spliced result is taken as the output of the first ResBlock module.

[0096] The other steps and parameters are the same as one of the first to fourth embodiments.

[0097] Embodiment six: different from one of the first to fifth embodiments, the first down-sampling unit sequentially comprises a max-pooling layer, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer and a second ReLU activation function layer.

[0098] The input of the first down-sampling unit sequentially passes through a max-pooling layer, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer and a second ReLU activation function layer, and then the input of the first down-sampling unit and the output of the second ReLU activation function layer are spliced, and the spliced result is taken as the output of the first down-sampling unit.

[0099] The second down-sampling unit, the third down-sampling unit and the fourth down-sampling unit have the same structure and working process as the first down-sampling unit.

[0100] The other steps and parameters are the same as one of the first to fifth embodiments.

[0101] The HG-Net network of the application adopts a residual structure when down-sampling, so that the feature extraction capability of the HG-Net network can be improved.

[0102] Embodiment seven: different from one of the first to sixth embodiments, the first up-sampling unit sequentially comprises a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer and a second ReLU activation function layer.

[0103] The input of the first up-sampling unit sequentially passes through a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer and a second ReLU activation function layer, and then the input of the first up-sampling unit and the output of the second ReLU activation function layer are spliced, and the spliced result is taken as the output of the first up-sampling unit.

[0104] The second up-sampling unit, the third up-sampling unit and the fourth up-sampling unit have the same structure and working process as the first up-sampling unit.

[0105] The other steps and parameters are the same as one of the first to sixth embodiments.

[0106] The HG-Net network of the present application adopts a residual structure when up-sampling, which can improve the feature extraction capability of the HG-Net network.

[0107] It should be noted that the numbering of each layer in the first ResBlock module, the numbering of each layer in the first down-sampling unit, and the numbering of each layer in the first up-sampling unit are performed separately, for example, the first convolutional layer in the first ResBlock module, the first convolutional layer in the first down-sampling unit, and the first convolutional layer in the first up-sampling unit are not the same convolutional layer, but represent the first convolutional layer in the module or unit where they are located.

[0108] Embodiment eight: different from one of the first to seventh embodiments, in step four, the optimizer used during training is RMSProp.

[0109] The other steps and parameters are the same as one of the first to seventh embodiments.

[0110] Embodiment nine: different from one of the first to eighth embodiments, in step four, the loss function used during training is MSEloss.

[0111] The other steps and parameters are the same as one of the first to eighth embodiments.

[0112] Embodiment ten: different from one of the first to ninth embodiments, in step four, the learning rate during training is set to 0.001, batch_size is set to 8, and epoch is set to 100.

[0113] The other steps and parameters are the same as one of the first to ninth embodiments.

[0114] All the codes of the present application are implemented using Python and pytorch, and are trained on an Intel(R) Xeon(R) Platinum 8255C CPU and Nvidia GeForce GTX 3070 GPU of a remote server, which takes about 8 hours.

[0115] Embodiment

[0116] The embodiment provides a hologram generation method based on deep learning and angular spectrum method (ASM), and the method specifically comprises the following steps:

[0117] Step one, randomly generate each speckle image with different speckle density and speckle size by using matlab, and the size of each image is 512*512;

[0118] Step two, take each speckle image generated in step one as the input of the angular spectrum method, and simulate the propagation process of the angular spectrum method by using matlab to obtain each complex amplitude hologram; extract the real part and the imaginary part of each complex amplitude hologram respectively to obtain the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram;

[0119] Take the real part hologram corresponding to each complex amplitude hologram as the input of the real part hologram training data set, and take the imaginary part hologram corresponding to each complex amplitude hologram as the input of the imaginary part hologram training data set;

[0120] Step three, obtain each complex amplitude hologram without speckle noise from each complex amplitude hologram and each speckle image, and extract the real part and the imaginary part of each complex amplitude hologram without speckle noise respectively to obtain the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram without speckle noise;

[0121] Take the real part hologram corresponding to each complex amplitude hologram without speckle noise as the label of the real part hologram training data set, and take the imaginary part hologram corresponding to each complex amplitude hologram without speckle noise as the label of the imaginary part hologram training data set;

[0122] Step four, construct a first HG-Net network and a second HG-Net network, taking the first HG-Net network as an example:

[0123] The first HG-Net network includes an encoding part and a decoding part; wherein: the encoding part includes a first ResBlock module, a first downsampling unit, a second downsampling unit, a third downsampling unit and a fourth downsampling unit; the decoding part includes a first upsampling unit, a second upsampling unit, a third upsampling unit, a fourth upsampling unit and a second ResBlock module;

[0124] In the first HG-Net network, the input image first passes through the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit and the fourth downsampling unit in turn, and then the output of the fourth downsampling unit is taken as the input of the first upsampling unit;

[0125] The output of the first upsampling unit and the output of the third downsampling unit are taken as the input of the second upsampling unit;

[0126] The output of the second upsampling unit and the output of the second downsampling unit are taken as the input of the third upsampling unit;

[0127] the output of the third up-sampling unit and the output of the first down-sampling unit as the input of a fourth up-sampling unit;

[0128] the output of the fourth up-sampling unit and the output of the first ResBlock module as the input of a second ResBlock module;

[0129] the output of the second ResBlock module as the output of the first HG-Net network.

[0130] The specific working process of the first HG-Net network is as follows: the input image first passes through a pair of 3*3 convolution layers, and each convolution layer is connected with a BN layer and a ReLU activation function layer, deepens the depth of the image, and changes the dimension of the image into (32, 512, 512), wherein kernel_size=3, stride=1, and padding=1.

[0131] Next, the image passes through a first down-sampling unit, and in the first down-sampling unit, the image first passes through a maximum pooling layer, the stride of the maximum pooling layer is 2, the kernel_size is 2, and the maximum pooling layer changes the size of the image to half of the original size; then two 3*3 convolution layers are connected, and the output dimension is a feature map of 64, and each convolution layer is also connected with a BN layer and a ReLU activation function layer, a total of four down-sampling units are needed, each down-sampling unit will halve the size of the image and double the number of channels of the image, and the last down-sampling unit outputs an image with a dimension of 512. In the decoding part, four up-sampling units are correspondingly arranged, each up-sampling unit will halve the number of channels of the image and change the size of the image to twice the original size, until the dimension of the image is restored to (32, 512, 512), and finally a kernel_size=1 convolution layer is used for output.

[0132] The first HG-Net network is trained by using the input and label of the real part hologram training data set, and the second HG-Net network is trained by using the input and label of the virtual part hologram training data set, and the loss function changes during the training process as shown in Figure 3 The training is stopped when the loss function value converges.

[0133] Step five, extracting the real part and the virtual part of the digital hologram to be removed from the speckle noise to obtain the extracted real part hologram I1 and the virtual part hologram I2.

[0134] The real part hologram I1 is taken as the input of the trained first HG-Net network, the real part hologram output by the first HG-Net network is denoted as I3, the imaginary part hologram I2 is taken as the input of the trained second HG-Net network, and the imaginary part hologram output by the second HG-Net network is denoted as I4; wherein I3 and I4 are the holograms after denoising.

[0135] Step six, the real part hologram I3 and the imaginary part hologram I4 are combined to obtain the combined complex amplitude hologram, that is, the generated hologram, and then the reconstruction is performed based on the generated hologram to obtain the reconstructed image.

[0136] Result analysis

[0137] The real part hologram generated by the first HG-Net network is shown in FIG. 4(a) and FIG. 4(b), the imaginary part hologram generated by the second HG-Net network is shown in FIG. 4(c) and FIG. 4(d), the real part hologram generated by the angular spectrum method is shown in FIG. 4(e) and FIG. 4(f), and the imaginary part hologram generated by the angular spectrum method is shown in FIG. 4(g) and FIG. 4(h). In order to verify the effect of the present application, the real part hologram generated by the first HG-Net network and the imaginary part hologram generated by the second HG-Net network are combined with the real part hologram and the imaginary part hologram generated by the ASM, and the four different combination modes are named as Net, Asm, Net_r_Asm_i and Net_i_Asm_r respectively.

[0138] wherein Asm represents the hologram obtained by combining the real part hologram generated by the ASM and the imaginary part hologram generated by the ASM; Net represents the hologram obtained by combining the real part hologram generated by the first HG-Net network and the imaginary part hologram generated by the second HG-Net network; Net_r_Asm_i represents the hologram obtained by combining the real part hologram generated by the first HG-Net network and the imaginary part hologram generated by the ASM; and Net_i_Asm_r represents the hologram obtained by combining the imaginary part hologram generated by the second HG-Net network and the real part hologram generated by the ASM.

[0139] The original hologram without speckle noise is shown in FIG. 5(a), the reconstructed image obtained by the hologram according to the combination mode Asm is shown in FIG. 5(b), the reconstructed image obtained by the hologram according to the combination mode Net is shown in FIG. 5(c), the reconstructed image obtained by the hologram according to the combination mode Net_r_Asm_i is shown in FIG. 5(d), and the reconstructed image obtained by the hologram according to the combination mode Net_i_Asm_r is shown in FIG. 5(e).

[0140] For the sake of clear comparison of the details, the right small graph is the magnified part in the red box, compared with the reconstructed image of the hologram based on the combination mode Asm, it can be seen that the details of the reconstructed image generated by the hologram using the combination mode Net are restored more, the details information is saved more, and the noise around the letter B and the letter C is also less.

[0141] In order to more intuitively see the edge information of the reconstructed image, as shown in FIG. 6(a), FIG. 6(b), FIG. 6(c) and FIG. 6(d) are 3D visual graphs of the reconstructed images corresponding to the combination mode Asm, the combination mode Net, the combination mode Net_r_Asm_i and the combination mode Net_i_Asm_r respectively, x and y represent the coordinate positions of the pixel points in the image, and z represents the gray value of the pixel point. The details of the reconstructed images corresponding to the combination mode Asm, Net, Net_r_Asm_i are relatively complete, while the reconstructed image corresponding to the combination mode Net_i_Asm_r loses more details in the original image and has more noise. The horizontal coordinate represents the size of the pixel, and the vertical coordinate represents the number of the same pixel value. FIG. 7(a), FIG. 7(b), FIG. 7(c) and FIG. 7(d) are image histograms of the reconstructed images corresponding to the combination mode Asm, Net, Net_r_Asm_i and Net_i_Asm_r respectively, it can be seen that the pixel distribution of the reconstructed images corresponding to the combination mode Asm and Net_r_Asm_i is relatively uniform, while the pixel distribution of the reconstructed images corresponding to the combination mode Net and Net_i_Asm_r is relatively concentrated.

[0142] The reconstructed image is evaluated by using the index peak signal-to-noise ratio (Peak Signal-to-noise Ratio, PSNR), speckle suppression index (Speckle Suppression Index, SSI), structural similarity (Structural Similarity, SSIM), speckle suppression and mean preservation index (Speckle Suppression and Mean Preservation Index, SMPI) to verify the advantages of the method of the application.

[0143] The speckle suppression index (SSI) is usually used to measure the suppression effect of speckle, and the specific calculation formula is as follows:

[0144]

[0145] Wherein, S i and M i respectively represent the standard deviation and the average value of the original reconstructed image. Similarly, S o and M oThe standard deviation and the mean value of the filtered image, respectively. The smaller the SSI, the better the speckle noise reduction performance.

[0146] The specific calculation formula of PSNR is as follows:

[0147]

[0148] Wherein, MSE is the mean square error between the original image and the processed image;

[0149]

[0150] MAX1 represents the maximum value of the image pixel, and the unit of PSNR is dB, so the larger the PSNR value, the less the distortion of the image, and the higher the similarity between the reproduced image and the original image.

[0151] The specific calculation formula of SSIM is as follows:

[0152]

[0153] The image similarity is measured from three aspects of brightness, contrast and structure, wherein, μ x ,μ y represent the average values of the two images, σ x ,σ y represent the variances of the two images. The value of SSIM ranges from 0 to 1, and the larger the value of SSIM, the smaller the image distortion.

[0154] The specific calculation formula of SMPI is as follows:

[0155]

[0156] Compared with SSI, SMPI considers the mean value difference between the original image and the reproduced image. When the mean value of the reproduced image deviates too much from the mean value of the original image, the credibility of the value of SMPI is higher than that of SSI. The smaller the value of SMPI, the better the performance in mean value preservation and noise reduction, μ f and μ o represent the mean values of the original image and the reproduced image, σ f and σ o represent the standard deviations of the original image and the reproduced image.

[0157] As shown in Table 1, the evaluation results after numerical reconstruction of the holograms obtained based on four combinations are as follows:

[0158] Table 1

[0159]

[0160] It can be seen that the quality of the reconstructed image of Net_r_Asm_i combining the real hologram generated by HG-Net and the imaginary hologram generated by ASM is higher than that of Asm, and compared with the Net combining the real hologram and the imaginary hologram generated by HG-Net, the SSIM index is improved by 0.0329, the SSI is reduced by 0.2628, and the SMPI is reduced by 0.0644. From the analysis of the three evaluation indexes, the hologram quality combining the real hologram and the imaginary hologram generated by HG-Net is higher, and only the PSNR is reduced by 0.0031 dB. The experimental data show that the combination of deep learning and traditional method can not only save computing resources, but also generate higher quality hologram.

[0161] As shown in Table 2, the results of four algorithms are compared.

[0162] Table 2

[0163]

[0164] It can be seen that the quality of the reconstructed image of Net_r_Asm_i combining the real hologram generated by HG-Net and the imaginary hologram generated by ASM is higher than that of three traditional algorithms.

[0165] The application proposes an algorithm for efficiently generating holograms based on deep learning and ASM, proposes an HG-Net network structure, and makes a large sample data set to effectively train the network. The data set is composed of multiple input intensity distributions and holograms obtained by ASM propagation, not simply training random spot images and their propagated holograms, but by randomly adjusting the density and size of the spots, combining basic shapes into the data set to generate the training data set, so as to improve the complexity of the data set and improve the generalization of the model. Then these random spot images generate real holograms and imaginary holograms by ASM propagation, and two different models are trained using two data sets, and then the real and imaginary holograms generated by ASM are combined to obtain four different types of holograms. In order to verify the effectiveness of the method of the application, numerical and optical reconstruction are carried out, and the PSNR, SSI, SSIM and SMPI of the reconstructed image are calculated, and the performance of the algorithm is quantitatively evaluated. The results show that compared with the traditional computer generated hologram algorithm, the algorithm for generating high quality hologram based on HG-Net and ASM can simply and efficiently generate high quality hologram, thereby improving the quality of the reconstructed image.

[0166] The above calculation examples of the present application are only used to illustrate the calculation model and calculation process of the present application, and are not used to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art, and all the embodiments cannot be exhausted here. Any obvious changes or variations derived from the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A hologram generation method based on deep learning and angular spectrum method, characterized in that: The method specifically comprises the following steps: Step 1: randomly generate spot images with different spot densities and spot sizes; Step 2: Using the speckle images generated in step 1 as input to the angular spectrum method, simulating the propagation process of the angular spectrum method to obtain each complex amplitude hologram; extracting the real part and imaginary part of each complex amplitude hologram to obtain the real part hologram and imaginary part hologram corresponding to each complex amplitude hologram; The real hologram corresponding to each complex amplitude hologram is used as the input of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram is used as the input of the imaginary hologram training data set; Step 3: Obtaining each complex amplitude hologram with speckle noise removed based on each complex amplitude hologram in Step 2 and each speckle image in Step 1, extracting the real part and imaginary part of each complex amplitude hologram with speckle noise removed, to obtain a real part hologram and an imaginary part hologram corresponding to each complex amplitude hologram with speckle noise removed; The real part hologram corresponding to each complex amplitude hologram with speckle noise removed is used as the label of the real part hologram training data set, and the imaginary part hologram corresponding to each complex amplitude hologram with speckle noise removed is used as the label of the imaginary part hologram training data set; Step 4: Construct the first HG-Net network and the second HG-Net network, use the input and labels of the real hologram training data set to train the first HG-Net network, and use the input and labels of the imaginary hologram training data set to train the second HG-Net network; Step 5: extracting the real part and imaginary part of the digital hologram from which speckle noise is removed, and obtaining the extracted real part hologram I1 and imaginary part hologram I2; The real hologram I1 is used as the input of the trained first HG-Net network, and the real hologram output by the first HG-Net network is recorded as I3. The imaginary hologram I2 is used as the input of the trained second HG-Net network, and the imaginary hologram output by the second HG-Net network is recorded as I4. Step 6: Combine the real hologram I3 with the imaginary hologram I4, combine the real hologram I3 with the ASM imaginary hologram, combine the imaginary hologram I4 with the ASM real hologram, and combine the ASM real hologram with the ASM imaginary hologram to obtain the complex amplitude holograms after each combination.

2. The hologram generation method based on deep learning and angular spectrum method according to claim 1, characterized in that: In the step 1, the size of each generated spot image is 512×512.

3. The hologram generation method based on deep learning and angular spectrum method according to claim 1, characterized in that: The first HG-Net network includes two parts: encoding and decoding; wherein: The encoding part includes a first ResBlock module, a first downsampling unit, a second downsampling unit, a third downsampling unit and a fourth downsampling unit; The decoding part includes a first upsampling unit, a second upsampling unit, a third upsampling unit, a fourth upsampling unit and a second ResBlock module.

4. The hologram generation method based on deep learning and angular spectrum method according to claim 3, characterized in that: The working process of the first HG-Net network is as follows: In the first HG-Net network, the input image first passes through the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit, and the fourth downsampling unit in sequence, and then the output of the fourth downsampling unit is used as the input of the first upsampling unit; Taking the output of the first upsampling unit and the output of the third downsampling unit as the input of the second upsampling unit; Taking the output of the second upsampling unit and the output of the second downsampling unit as the input of the third upsampling unit; using the output of the third upsampling unit and the output of the first downsampling unit as the input of the fourth upsampling unit; The output of the fourth upsampling unit and the output of the first ResBlock module are used as the input of the second ResBlock module; The output of the second ResBlock module is used as the output of the first HG-Net network.

5. The hologram generation method based on deep learning and angular spectrum method according to claim 4, characterized in that: The first ResBlock module includes a first convolutional layer, a first BN layer, a first ReLU layer, a second convolutional layer, a second BN layer, and a second ReLU layer in sequence; The input of the first ResBlock module passes through the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence. The input of the first ResBlock module and the output of the second ReLU activation function layer are then spliced ​​together, and the splicing result is used as the output of the first ResBlock module.

6. The hologram generation method based on deep learning and angular spectrum method according to claim 5, characterized in that: The first downsampling unit includes a maximum pooling layer, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer in sequence; The input of the first downsampling unit passes through the maximum pooling layer, the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence. The input of the first downsampling unit and the output of the second ReLU activation function layer are then spliced ​​together, and the spliced ​​result is used as the output of the first downsampling unit; The second down-sampling unit, the third down-sampling unit, and the fourth down-sampling unit have the same structure and working process as the first down-sampling unit.

7. The hologram generation method based on deep learning and angular spectrum method according to claim 6, characterized in that: The first upsampling unit includes, in sequence, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer; The input of the first upsampling unit passes through the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence. The input of the first upsampling unit and the output of the second ReLU activation function layer are then concatenated, and the concatenated result is used as the output of the first upsampling unit; The second upsampling unit, the third upsampling unit, and the fourth upsampling unit have the same structure and working process as the first upsampling unit.

8. The hologram generation method based on deep learning and angular spectrum method according to claim 7, characterized in that: In step 4, the optimizer used during training is RMSProp.

9. The hologram generation method based on deep learning and angular spectrum method according to claim 8, characterized in that: In step 4, the loss function used in training is MSEloss.

10. The hologram generation method based on deep learning and angular spectrum method according to claim 9, characterized in that: In step 4, the learning rate is set to 0.001, the batch_size is set to 8, and the epoch is set to 100 during training.

Images