Hologram generation method based on deep learning and angular spectrum method

Through the combination of deep learning and angular spectroscopy, high-quality holograms are generated, which solves the efficiency and quality problems of digital hologram training data acquisition, improves the quality of reproducible images, and realizes efficient hologram generation.

CN118982598AActive Publication Date: 2024-11-19JILIN UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411128393.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-19
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

In the prior art, the training data acquisition of digital holograms takes time, is high cost and is difficult to guarantee. The quality of the after-noise denoising holograms output based on deep learning methods is poor, and there is a lack of high-quality open source data sets and effective holograms. Generate method.

Method used

Using the hologram generation method based on deep learning and angular spectroscopy method, randomly generates spot images, simulates the propagation process of angle spectroscopy, extracts the real and imaginary parts of complex amplitude holograms, builds the HG-Net network, and uses the training data set to perform Network training, combining real and imaginary holograms to generate high-quality holograms.

Benefits of technology

It significantly reduces the acquisition time and cost of training data sets, improves the quality of the data set, enhances the generalization of the model, and the quality of the generated holograms is better than that of the traditional methods, and improves the PSNR, SSI, SSIM and SMPI indicators of the reproduced images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982598A_ABST
    Figure CN118982598A_ABST
Patent Text Reader

Abstract

The invention discloses a hologram generation method based on deep learning and an angular spectrum method, and belongs to the technical field of deep learning and digital holography. The problems that an existing training data set obtaining method is long in data obtaining time consumption and high in cost, the quality of obtained data is difficult to guarantee, and the quality of a reproductive image obtained by a denoised hologram output based on an existing deep learning method is poor are solved. According to the method, an HG-Net network structure is provided, the density and size of spots are randomly adjusted, and a plurality of input intensity distributions and a hologram obtained through ASM propagation serve as a training data set, so that the complexity of the training data set is improved. The training data set is used for training the HG-Net network, so that the generalization of the model can be improved. And combining the real part hologram and the imaginary part hologram output by the HG-Net network to obtain a generated hologram, and performing numerical and optical reconstruction based on the generated hologram. The method can be applied to hologram generation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and digital holography, and specifically relates to a hologram generation method based on deep learning and angular spectrum method (ASM). Background Art

[0002] In digital holography, when a laser source is used to digitally record a hologram, due to the high coherence of the laser light source and the rough surface of the object, coherent superposition will occur during reflection, thus generating speckle noise in the obtained digital hologram, resulting in reduced image quality when numerically reconstructing the digital hologram. Therefore, how to effectively remove speckle noise while preserving the digital holographic image information as much as possible to improve the quality of the reconstructed image is a current research hotspot.

[0003] At present, there are three common methods to eliminate speckle noise in digital holograms: the first method is to use a partially coherent light source or a low-coherence light source. Using a partially coherent light source instead of a laser light source can achieve the purpose of reducing speckle noise, but this requires relying on a high-brightness partially coherent light source, otherwise the quality of the reconstructed image will be reduced; the second method is to use digital image processing technology. Through traditional filters, resampling masks, BM3D and other methods, speckle noise can also be effectively removed, but it will destroy the detailed information in the image, resulting in a decrease in the quality of the reconstructed image; the third method is to remove speckle noise based on deep learning. K Zhang et al. proposed a denoising convolutional neural network using residual learning and batch normalization to reduce general noise. WONSEOK JEON et al. used a multi-scale convolutional neural network to achieve speckle denoising of digital holographic images. Z j Shen et al. used a dual-path parallel convolutional neural network to remove speckle noise from different types of OCT images. Q Fang et al. used a conditional generative adversarial network based on deep learning to remove speckle noise. Compared with the above two traditional methods, the deep learning-based method does not require complex and rigorous experiments. Instead, it trains the neural network according to the pre-acquired data set and directly maps the denoised hologram. Moreover, with the breakthrough of GPU computing speed, the learning speed and performance are improved, and real-time application is realized. Juyeon Seo et al. proposed a hologram compression network based on deep learning, which greatly improved the quality of numerical reconstruction of full complex holograms. Qingwei Liu et al. proposed a dual-channel parallel neural network (DCPNet) for generating phase-only holograms, and encoded it into two real-valued phase units through a dual-phase amplitude encoding method. Experimental data show that DCPNet can generate 2k resolution phase-only holograms within 36ms. Yun Liu et al. proposed ANLVENet based on FFDNet and combined asymmetric pyramid non-local blocks with edge extraction modules to suppress speckle noise in holographic phase fringe patterns. Experimental data show that the ANLVENet method not only has better advantages in speckle noise suppression, but also retains more image edge details.

[0004] In recent years, the application of deep learning in the field of digital holography has achieved remarkable development, mainly due to the progress of deep learning theory and breakthroughs in graphics processing unit (GPU) technology. In holographic display technology, deep learning models such as convolutional neural network (CNN), generative adversarial network (GAN) and recurrent neural network (RNN) have shown excellent performance. However, the training of deep learning models depends on the input of a large amount of training data, which is challenging in the field of digital holography. At present, there is a lack of widely available open source datasets for digital holography, which has forced most researchers to collect training data through time-consuming and costly optical experiments. This way of obtaining training data is not only inefficient, but also often difficult to ensure the high quality of data. Moreover, the quality of the reconstructed image obtained from the denoised hologram output by the existing deep learning method is still poor. Therefore, the development and sharing of high-quality open source datasets is crucial to promote the development of digital holography and to propose a new hologram generation method to improve the quality of the reconstructed image. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of long data acquisition time, high cost, difficulty in ensuring the quality of acquired data in existing training data set acquisition methods, and poor quality of reconstructed images obtained from denoised holograms output by existing deep learning methods, and propose a hologram generation method based on deep learning and angular spectrum method.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a hologram generation method based on deep learning and angular spectrum method, the method specifically comprises the following steps:

[0007] Step 1, randomly generate spot images with different spot densities and spot sizes;

[0008] Step 2: Use the spot images generated in step 1 as inputs of the angular spectrum method, simulate the propagation process of the angular spectrum method to obtain the complex amplitude holograms; extract the real part and imaginary part of each complex amplitude hologram, and obtain the real part hologram and imaginary part hologram corresponding to each complex amplitude hologram;

[0009] The real hologram corresponding to each complex amplitude hologram is used as the input of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram is used as the input of the imaginary hologram training data set;

[0010] Step 3: obtaining each complex amplitude hologram with speckle noise removed according to each complex amplitude hologram in step 2 and each speckle image in step 1, extracting the real part and the imaginary part of each complex amplitude hologram with speckle noise removed, and obtaining the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram with speckle noise removed;

[0011] The real hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the imaginary hologram training data set;

[0012] Step 4: construct the first HG-Net network and the second HG-Net network, train the first HG-Net network using the input and labels of the real hologram training data set, and train the second HG-Net network using the input and labels of the imaginary hologram training data set;

[0013] Step 5: extracting the real part and imaginary part of the digital hologram from which speckle noise is to be removed, and obtaining an extracted real part hologram I1 and an imaginary part hologram I2;

[0014] The real hologram I1 is used as the input of the trained first HG-Net network, and the real hologram output by the first HG-Net network is recorded as I3. The imaginary hologram I2 is used as the input of the trained second HG-Net network, and the imaginary hologram output by the second HG-Net network is recorded as I4.

[0015] Step six: Combine the real hologram I3 and the imaginary hologram I4 to obtain a combined complex amplitude hologram.

[0016] Furthermore, in the step 1, the size of each spot image generated is 512×512.

[0017] Furthermore, the first HG-Net network includes two parts: encoding and decoding; wherein:

[0018] The encoding part includes a first ResBlock module, a first downsampling unit, a second downsampling unit, a third downsampling unit and a fourth downsampling unit;

[0019] The decoding part includes a first upsampling unit, a second upsampling unit, a third upsampling unit, a fourth upsampling unit and a second ResBlock module.

[0020] Furthermore, the working process of the first HG-Net network is:

[0021] In the first HG-Net network, the input image first passes through the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit and the fourth downsampling unit in sequence, and then the output of the fourth downsampling unit is used as the input of the first upsampling unit;

[0022] Using the output of the first up-sampling unit and the output of the third down-sampling unit as the input of the second up-sampling unit;

[0023] Using the output of the second up-sampling unit and the output of the second down-sampling unit as the input of the third up-sampling unit;

[0024] Using the output of the third up-sampling unit and the output of the first down-sampling unit as inputs of a fourth up-sampling unit;

[0025] Using the output of the fourth upsampling unit and the output of the first ResBlock module as the input of the second ResBlock module;

[0026] The output of the second ResBlock module is used as the output of the first HG-Net network.

[0027] Furthermore, the first ResBlock module includes a first convolutional layer, a first BN layer, a first ReLU layer, a second convolutional layer, a second BN layer, and a second ReLU layer in sequence;

[0028] The input of the first ResBlock module passes through the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first ResBlock module and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first ResBlock module.

[0029] Furthermore, the first downsampling unit sequentially includes a maximum pooling layer, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer;

[0030] The input of the first downsampling unit passes through the maximum pooling layer, the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first downsampling unit and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first downsampling unit;

[0031] The second down-sampling unit, the third down-sampling unit, and the fourth down-sampling unit have the same structure and working process as the first down-sampling unit.

[0032] Furthermore, the first upsampling unit sequentially includes a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer;

[0033] The input of the first upsampling unit passes through the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first upsampling unit and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first upsampling unit;

[0034] The second up-sampling unit, the third up-sampling unit, and the fourth up-sampling unit have the same structure and working process as the first up-sampling unit.

[0035] Preferably, in step 4, the optimizer used during training is RMSProp.

[0036] Preferably, in step 4, the loss function used in training is MSEloss.

[0037] Preferably, in step 4, during training, the learning rate is set to 0.001, the batch_size is set to 8, and the epoch is set to 100.

[0038] The beneficial effects of the present invention are:

[0039] The present invention proposes an HG-Net network structure, which randomly adjusts the density and size of spots, and uses multiple input intensity distributions and holograms obtained through ASM propagation as training data sets, thereby increasing the complexity of the data set, and the time required to obtain the training data set is short, the cost is low, and the quality of the acquired data can be guaranteed. Using the data set to train the HG-Net network can improve the generalization of the model. The real hologram and the imaginary hologram output by the HG-Net network are combined to obtain the generated hologram. Numerical and optical reconstruction is performed based on the generated hologram, and the PSNR, SSI, SSIM and SMPI of the reproduced image are calculated. The results show that compared with the traditional method, the method of the present invention can simply and efficiently generate high-quality holograms, thereby improving the quality of the reproduced image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of a hologram generation method based on deep learning and angular spectrum method of the present invention;

[0041] Figure 2 It is a schematic diagram of the HG-Net network structure;

[0042] In the figure, N represents the number of channels;

[0043] Figure 3 Schematic diagram of loss changes during training;

[0044] Figure 4(a) shows the real part hologram generated by the first HG-Net network Figure 1 ;

[0045] Figure 4(b) shows the real part hologram generated by the first HG-Net network Figure 2 ;

[0046] Figure 4(c) shows the imaginary hologram generated by the second HG-Net network Figure 1 ;

[0047] Figure 4(d) shows the imaginary hologram generated by the second HG-Net network Figure 2 ;

[0048] Figure 4(e) shows the real part hologram generated by the angular spectrum method Figure 1 ;

[0049] Figure 4(f) shows the real part hologram generated by the angular spectrum method Figure 2 ;

[0050] Figure 4(g) shows the imaginary hologram generated by the angular spectrum method Figure 1 ;

[0051] Figure 4(h) shows the imaginary hologram generated by the angular spectrum method Figure 2 ;

[0052] Figure 5(a) is the original hologram;

[0053] Figure 5(b) is the reconstructed image 1;

[0054] Figure 5(c) is the reconstructed image 2;

[0055] Figure 5(d) is the reconstructed image 3;

[0056] Figure 5(e) is the reconstructed image 4;

[0057] FIG6( a ) is a 3D visual diagram of FIG5( b );

[0058] FIG6( b ) is a 3D visual diagram of FIG5( c );

[0059] FIG6( c ) is a 3D visual image of FIG5( d );

[0060] FIG6( d ) is a 3D visual image of FIG5( e );

[0061] Figure 7(a) is the image histogram of Figure 5(b);

[0062] Figure 7(b) is the image histogram of Figure 5(c);

[0063] Figure 7(c) is the image histogram of Figure 5(d);

[0064] Figure 7(d) is the image histogram of Figure 5(e). DETAILED DESCRIPTION

[0065] Specific implementation method 1: Combination Figure 1 The present embodiment describes a method for generating a hologram based on deep learning and angular spectrum method, and the method specifically comprises the following steps:

[0066] Step 1: Use Matlab to randomly generate spot images with different spot densities and spot sizes;

[0067] By randomly adjusting the density and size of the spots and incorporating them into the dataset in different combinations, increasing the diversity of the dataset in this way can improve the generalization ability of the model in different situations;

[0068] Step 2: Use the spot images generated in step 1 as the input of the angular spectrum method, and use Matlab to simulate the propagation process of the angular spectrum method to obtain each complex amplitude hologram; extract the real part and imaginary part of each complex amplitude hologram to obtain the real part hologram and imaginary part hologram corresponding to each complex amplitude hologram;

[0069] The real hologram corresponding to each complex amplitude hologram is used as the input of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram is used as the input of the imaginary hologram training data set;

[0070] Step 3: obtaining each complex amplitude hologram with speckle noise removed according to each complex amplitude hologram in step 2 and each speckle image in step 1, extracting the real part and the imaginary part of each complex amplitude hologram with speckle noise removed, and obtaining the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram with speckle noise removed;

[0071] The real hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the imaginary hologram training data set;

[0072] Step 4: construct the first HG-Net network and the second HG-Net network, train the first HG-Net network using the input and labels of the real hologram training data set, and train the second HG-Net network using the input and labels of the imaginary hologram training data set;

[0073] Step 5: extracting the real part and imaginary part of the digital hologram from which speckle noise is to be removed, and obtaining an extracted real part hologram I1 and an imaginary part hologram I2;

[0074] The real hologram I1 is used as the input of the trained first HG-Net network, and the real hologram output by the first HG-Net network is recorded as I3. The imaginary hologram I2 is used as the input of the trained second HG-Net network, and the imaginary hologram output by the second HG-Net network is recorded as I4; where I3 and I4 are the denoised holograms;

[0075] Step six: Combine the real hologram I3 and the imaginary hologram I4 to obtain a combined complex amplitude hologram.

[0076] The method of the present invention only needs to train the first HG-Net network and the second HG-Net network to generate a hologram with higher quality than the traditional algorithm, while significantly reducing the calculation cost, and can obtain a better hologram generation effect with less resource investment.

[0077] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that in step 1, the size of each spot image generated is 512×512.

[0078] The other steps and parameters are the same as those in the first embodiment.

[0079] Specific implementation method three: Combination Figure 2 This embodiment is different from the first or second embodiment in that the first HG-Net network includes two parts: an encoder and a decoder; wherein:

[0080] The encoding part includes a first ResBlock module, a first downsampling unit, a second downsampling unit, a third downsampling unit and a fourth downsampling unit;

[0081] The decoding part includes a first upsampling unit, a second upsampling unit, a third upsampling unit, a fourth upsampling unit and a second ResBlock module.

[0082] The other steps and parameters are the same as those in the first or second embodiment.

[0083] The encoding part of the present invention uses the first Resblock module to replace the original convolution module, which improves the feature extraction ability of the network. The decoding part uses a bilinear interpolation method to restore the resolution of the original image and reduce the number of channels. Without reducing the performance of the network, the number of channels can reduce the number of parameters by about 2 times, thereby achieving a lightweight network.

[0084] The present invention uses four downsampling units to obtain high-level semantic information. In order to reduce the spatial information loss caused by the downsampling process, the output of the downsampling unit is used as the input of the upsampling unit through a skip-layer connection. By fusing deep features with shallow features, the feature map restored by the upsampling process contains more low-level semantic information, ensuring the quality of the generated hologram. The HG-Net network of the present invention is applied to remove the speckle noise of the hologram, which can retain the edge part of the image without losing the resolution of the image.

[0085] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the working process of the first HG-Net network is:

[0086] In the first HG-Net network, the input image first passes through the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit and the fourth downsampling unit in sequence, and then the output of the fourth downsampling unit is used as the input of the first upsampling unit;

[0087] Using the output of the first up-sampling unit and the output of the third down-sampling unit as the input of the second up-sampling unit;

[0088] Using the output of the second up-sampling unit and the output of the second down-sampling unit as the input of the third up-sampling unit;

[0089] Using the output of the third up-sampling unit and the output of the first down-sampling unit as inputs of a fourth up-sampling unit;

[0090] Using the output of the fourth upsampling unit and the output of the first ResBlock module as the input of the second ResBlock module;

[0091] The output of the second ResBlock module is used as the output of the first HG-Net network.

[0092] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0093] The second HG-Net network has the same structure and working process as the first HG-Net network. The present invention improves the accuracy by introducing residual connection, that is, directly connecting the encoder and the decoder, and reduces the processing time to a certain extent. In this way, the information lost in different layers in the encoding part can be retained, and no additional parameters and operations are added when relearning the lost information.

[0094] Specific implementation mode 5: This implementation mode is different from any one of specific implementation modes 1 to 4 in that the first ResBlock module includes a first convolutional layer, a first BN layer, a first ReLU layer, a second convolutional layer, a second BN layer, and a second ReLU layer in sequence;

[0095] The input of the first ResBlock module passes through the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first ResBlock module and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first ResBlock module.

[0096] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0097] Specific implementation method six: This implementation method is different from any one of specific implementation methods one to five in that the first downsampling unit includes a maximum pooling layer, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer in sequence;

[0098] The input of the first downsampling unit passes through the maximum pooling layer, the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first downsampling unit and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first downsampling unit;

[0099] The second down-sampling unit, the third down-sampling unit, and the fourth down-sampling unit have the same structure and working process as the first down-sampling unit.

[0100] The other steps and parameters are the same as those in Specific Implementations 1 to 5.

[0101] The HG-Net network of the present invention adopts a residual structure during downsampling, which can improve the feature extraction capability of the HG-Net network.

[0102] Specific implementation manner seven: This implementation manner is different from any one of specific implementation manners one to six in that the first upsampling unit sequentially includes a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer;

[0103] The input of the first upsampling unit passes through the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first upsampling unit and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first upsampling unit;

[0104] The second up-sampling unit, the third up-sampling unit, and the fourth up-sampling unit have the same structure and working process as the first up-sampling unit.

[0105] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0106] The HG-Net network of the present invention adopts a residual structure during upsampling, which can improve the feature extraction capability of the HG-Net network.

[0107] It should be noted that the present invention numbering each layer in the first ResBlock module, each layer in the first downsampling unit, and each layer in the first upsampling unit are performed separately. For example, the first convolutional layer in the first ResBlock module, the first convolutional layer in the first downsampling unit, and the first convolutional layer in the first upsampling unit are not the same convolutional layer, but only represent the first convolutional layer of the module or unit in which they are located.

[0108] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that in step four, the optimizer used in training is RMSProp.

[0109] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0110] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that in step 4, the loss function used in training is MSEloss.

[0111] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0112] Specific implementation method ten: This implementation method is different from any one of specific implementation methods one to nine in that, in step four, the learning rate is set to 0.001, the batch_size is set to 8, and the epoch is set to 100 during training.

[0113] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0114] All codes of the present invention are implemented using Python and pytorch, and training is performed on a remote server with an Intel(R) Xeon(R) Platinum 8255C CPU and an Nvidia GeForce GTX 3070 GPU, which takes about 8 hours.

[0115] Example

[0116] This embodiment proposes a hologram generation method based on deep learning and angular spectrum method (ASM), and the method is specifically as follows:

[0117] Step 1: Use MATLAB to randomly generate spot images with different spot densities and spot sizes, and the size of each image is 512×512;

[0118] Step 2: Use the spot images generated in step 1 as the input of the angular spectrum method, and use Matlab to simulate the propagation process of the angular spectrum method to obtain each complex amplitude hologram; extract the real part and imaginary part of each complex amplitude hologram to obtain the real part hologram and imaginary part hologram corresponding to each complex amplitude hologram;

[0119] The real hologram corresponding to each complex amplitude hologram is used as the input of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram is used as the input of the imaginary hologram training data set;

[0120] Step 3: obtaining each complex amplitude hologram with speckle noise removed according to each complex amplitude hologram and each speckle image, extracting the real part and the imaginary part of each complex amplitude hologram with speckle noise removed, and obtaining the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram with speckle noise removed;

[0121] The real hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the imaginary hologram training data set;

[0122] Step 4: Build the first HG-Net network and the second HG-Net network. Take the first HG-Net network as an example:

[0123] The first HG-Net network includes two parts: encoding and decoding; wherein: the encoding part includes the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit and the fourth downsampling unit; the decoding part includes the first upsampling unit, the second upsampling unit, the third upsampling unit, the fourth upsampling unit and the second ResBlock module;

[0124] In the first HG-Net network, the input image first passes through the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit and the fourth downsampling unit in sequence, and then the output of the fourth downsampling unit is used as the input of the first upsampling unit;

[0125] Using the output of the first up-sampling unit and the output of the third down-sampling unit as the input of the second up-sampling unit;

[0126] Using the output of the second up-sampling unit and the output of the second down-sampling unit as the input of the third up-sampling unit;

[0127] Using the output of the third up-sampling unit and the output of the first down-sampling unit as inputs of a fourth up-sampling unit;

[0128] Using the output of the fourth upsampling unit and the output of the first ResBlock module as the input of the second ResBlock module;

[0129] The output of the second ResBlock module is used as the output of the first HG-Net network.

[0130] The specific working process of the first HG-Net network is as follows: the input image first passes through a pair of 3*3 convolutional layers, and each convolutional layer is connected to a BN layer and a ReLU activation function layer to deepen the depth of the image and change the dimension of the image to (32, 512, 512), where kernel_size = 3, stride = 1, and padding = 1;

[0131] Next, it passes through the first downsampling unit, and then passes through a maximum pooling layer in the first downsampling unit. The maximum pooling layer has stride = 2 and kernel_size = 2. The maximum pooling layer reduces the size of the image to half of the original size; then it is connected to two 3*3 convolutional layers, and the output dimension is a feature map of 64. Similarly, each convolutional layer is connected to a BN layer and a ReLU activation function layer. A total of four downsampling units are required. Each downsampling unit will halve the size of the image and double the number of channels of the image. The last downsampling unit outputs an image with a dimension of 512. In the decoding part, four upsampling units are set up accordingly. Each upsampling unit will halve the number of channels of the image and double the image size until the image dimension is restored to (32, 512, 512). Finally, it passes through a convolutional layer with kernel_size = 1 for output.

[0132] The first HG-Net network is trained using the input and labels of the real hologram training data set, and the second HG-Net network is trained using the input and labels of the imaginary hologram training data set. The change of the loss function during the training process is shown in Figure 3 As shown, training is stopped when the loss function value converges;

[0133] Step 5: extracting the real part and imaginary part of the digital hologram from which speckle noise is to be removed, and obtaining an extracted real part hologram I1 and an imaginary part hologram I2;

[0134] The real hologram I1 is used as the input of the trained first HG-Net network, and the real hologram output by the first HG-Net network is recorded as I3. The imaginary hologram I2 is used as the input of the trained second HG-Net network, and the imaginary hologram output by the second HG-Net network is recorded as I4; where I3 and I4 are the denoised holograms;

[0135] Step six: combine the real hologram I3 and the imaginary hologram I4 to obtain a combined complex amplitude hologram, that is, obtain a generated hologram, and then reconstruct based on the generated hologram to obtain a reproduced image.

[0136] Results Analysis

[0137] The real hologram generated by the first HG-Net network is shown in Figures 4(a) and 4(b), the imaginary hologram generated by the second HG-Net network is shown in Figures 4(c) and 4(d), the real hologram generated by the angular spectrum method is shown in Figures 4(e) and 4(f), and the imaginary hologram generated by the angular spectrum method is shown in Figures 4(g) and 4(h). In order to verify the effect of the present invention, the real hologram generated by the first HG-Net network and the imaginary hologram generated by the second HG-Net network are combined with the real hologram and imaginary hologram generated by ASM, and the four different combination modes are named Net, Asm, Net_r_Asm_i and Net_i_Asm_r, respectively.

[0138] Among them, Asm represents the hologram obtained by combining the real hologram generated by ASM and the imaginary hologram generated by ASM; Net represents the hologram obtained by combining the real hologram generated by the first HG-Net network and the imaginary hologram generated by the second HG-Net network; Net_r_Asm_i represents the hologram obtained by combining the real hologram generated by the first HG-Net network and the imaginary hologram generated by ASM; Net_i_Asm_r represents the hologram obtained by combining the imaginary hologram generated by the second HG-Net network and the real hologram generated by ASM;

[0139] The original hologram without speckle noise is shown in Figure 5(a), Figure 5(b) is the reproduced image reconstructed from the hologram obtained according to the combination mode Asm, Figure 5(c) is the reproduced image reconstructed from the hologram obtained according to the combination mode Net, Figure 5(d) is the reproduced image reconstructed from the hologram obtained according to the combination mode Net_r_Asm_i, and Figure 5(e) is the reproduced image reconstructed from the hologram obtained according to the combination mode Net_i_Asm_r.

[0140] In order to clearly contrast the details, the small picture on the right is the enlarged part of the red box in the reconstructed image. Compared with the reconstructed image of the hologram obtained based on the combined mode Asm, it can be clearly seen that the reconstructed image generated by using the combined mode Net to generate the hologram has more restored details, more detailed information is preserved, and there is less noise around the letters B and C.

[0141] In order to see the edge information of the reproduced image more intuitively, Figures 6(a), 6(b), 6(c) and 6(d) are 3D visual diagrams of the reproduced images corresponding to the combination mode Asm, the combination mode Net, the combination mode Net_r_Asm_i and the combination mode Net_i_Asm_r, respectively. x and y represent the coordinate position of the pixel in the image, and z represents the grayscale value of the pixel. The details of the reproduced images corresponding to the combination modes Asm, Net and Net_r_Asm_i are relatively complete, while the reproduced image corresponding to the combination mode Net_i_Asm_r loses more details in the original image and has more noise. The horizontal coordinate represents the size of the pixel, and the vertical coordinate represents the number of the same pixel value. Figures 7(a), 7(b), 7(c), and 7(d) are the image histograms of the reproduced images corresponding to the combination modes Asm, Net, Net_r_Asm_i, and Net_i_Asm_r, respectively. It can be seen that the pixel distribution of the reproduced images corresponding to the combination modes Asm and Net_r_Asm_i is relatively uniform, while the pixel distribution of the reproduced images corresponding to the combination modes Net and Net_i_Asm_r is relatively concentrated.

[0142] The indicators Peak Signal-to-noise Ratio (PSNR), Speckle Suppression Index (SSI), Structural Similarity (SSIM), and Speckle Suppression and Mean Preservation Index (SMPI) are used to evaluate the reconstructed image to verify the advantages of the method of the present invention.

[0143] The speckle suppression index (SSI) is usually used to measure the speckle suppression effect. The specific calculation formula is as follows:

[0144]

[0145] Among them, S i and M i Represent the standard deviation and mean of the original reconstructed image. o and M oare the standard deviation and mean of the filtered image respectively. The smaller the SSI, the better the speckle noise reduction performance.

[0146] The specific calculation formula of PSNR is as follows:

[0147]

[0148] Among them, MSE is the mean square error between the original image and the processed image;

[0149]

[0150] MAX1 represents the maximum value of the image pixel. The unit of PSNR is dB, so the larger the PSNR value, the less distortion of the image, and the higher the similarity between the reproduced image and the original image.

[0151] The specific calculation formula of SSIM is as follows:

[0152]

[0153] Image similarity is measured from three aspects: brightness, contrast and structure, where μ x ,μ y represents the average value of two images, σ x ,σ y Represents the variance of two images. The SSIM value range is [0,1]. The larger the SSIM value, the smaller the image distortion value.

[0154] The specific calculation formula of SMPI is as follows:

[0155]

[0156] Compared with SSI, SMPI takes into account the mean difference between the original image and the reconstructed image. When the mean of the reconstructed image deviates too much from the mean of the original image, the reliability of the SMPI value is higher than that of SSI. The smaller the SMPI value, the better the performance in mean preservation and noise reduction. f and μ o represents the mean of the original image and the reconstructed image, σ f and σ o Indicates the standard deviation of the original image and the reproduced image.

[0157] As shown in Table 1, the evaluation results after numerical reconstruction of the holograms obtained based on the four combinations are as follows:

[0158] Table 1

[0159]

[0160] It can be seen that the quality of the reconstructed image of Net_r_Asm_i, which combines the real hologram generated by HG-Net with the imaginary hologram generated by ASM, is higher than that of Asm, and compared with Net, which combines the real hologram and imaginary hologram generated by HG-Net, the SSIM index is improved by 0.0329, the SSI is 0.2628 lower, and the SMPI is 0.0644 lower. From the analysis of these three evaluation indicators, the hologram that combines the real hologram and the imaginary hologram generated by HG-Net has higher quality, and the PSNR is only 0.0031dB lower. The experimental data shows that the combination of deep learning and traditional methods can not only save computing resources, but also generate higher quality holograms.

[0161] As shown in Table 2, the experimental results of the four algorithms are compared:

[0162] Table 2

[0163]

[0164] It can be seen that the quality of the reconstructed image of Net_r_Asm_i, which combines the real hologram generated by HG-Net with the imaginary hologram generated by ASM, is higher than that of the three traditional algorithms.

[0165] The present invention proposes an algorithm for efficiently generating holograms based on deep learning and ASM, proposes an HG-Net network structure, and produces a large sample data set to effectively train the network. The data set consists of multiple input intensity distributions and the holograms obtained by propagation through ASM. Instead of simply training random spot images and their propagated holograms, the training data set is generated by randomly adjusting the density and size of the spots and combining basic shapes into the data set, thereby increasing the complexity of the data set and improving the generalization of the model. Then these random spot images are propagated through ASM to generate real holograms and imaginary holograms, and two different models are trained using the two data sets, which are then combined with the real and imaginary holograms generated by ASM to obtain four different types of holograms. In order to verify the effectiveness of the method of the present invention, numerical and optical reconstructions are performed, and the PSNR, SSI, SSIM and SMPI of the reconstructed image are calculated, and the performance of the algorithm is quantitatively evaluated. The results show that compared with the traditional computer-generated hologram algorithm, the algorithm for generating high-quality holograms based on HG-Net and ASM can simply and efficiently generate high-quality holograms, thereby improving the quality of the reconstructed image.

[0166] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A hologram generation method based on deep learning and angular spectrum method, characterized in that: The method specifically comprises the following steps: Step 1, randomly generate spot images with different spot densities and spot sizes; Step 2: Use the spot images generated in step 1 as inputs of the angular spectrum method, simulate the propagation process of the angular spectrum method to obtain the complex amplitude holograms; extract the real part and imaginary part of each complex amplitude hologram, and obtain the real part hologram and imaginary part hologram corresponding to each complex amplitude hologram; The real hologram corresponding to each complex amplitude hologram is used as the input of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram is used as the input of the imaginary hologram training data set; Step 3: obtaining each complex amplitude hologram with speckle noise removed according to each complex amplitude hologram in step 2 and each speckle image in step 1, extracting the real part and the imaginary part of each complex amplitude hologram with speckle noise removed, and obtaining the real part hologram and the imaginary part hologram corresponding to each complex amplitude hologram with speckle noise removed; The real hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the real hologram training data set, and the imaginary hologram corresponding to each complex amplitude hologram with speckle noise removed is used as a label of the imaginary hologram training data set; Step 4: construct the first HG-Net network and the second HG-Net network, train the first HG-Net network using the input and labels of the real hologram training data set, and train the second HG-Net network using the input and labels of the imaginary hologram training data set; Step 5: extracting the real part and imaginary part of the digital hologram from which speckle noise is to be removed, and obtaining an extracted real part hologram I1 and an imaginary part hologram I2; The real hologram I1 is used as the input of the trained first HG-Net network, and the real hologram output by the first HG-Net network is recorded as I3. The imaginary hologram I2 is used as the input of the trained second HG-Net network, and the imaginary hologram output by the second HG-Net network is recorded as I4. Step six: Combine the real hologram I3 and the imaginary hologram I4 to obtain a combined complex amplitude hologram.

2. The hologram generation method based on deep learning and angular spectrum method according to claim 1, characterized in that: In the step 1, the size of each spot image generated is 512×512.

3. The hologram generation method based on deep learning and angular spectrum method according to claim 1, characterized in that: The first HG-Net network includes two parts: encoding and decoding; wherein: The encoding part includes a first ResBlock module, a first downsampling unit, a second downsampling unit, a third downsampling unit and a fourth downsampling unit; The decoding part includes a first upsampling unit, a second upsampling unit, a third upsampling unit, a fourth upsampling unit and a second ResBlock module.

4. The hologram generation method based on deep learning and angular spectrum method according to claim 3, characterized in that: The working process of the first HG-Net network is: In the first HG-Net network, the input image first passes through the first ResBlock module, the first downsampling unit, the second downsampling unit, the third downsampling unit and the fourth downsampling unit in sequence, and then the output of the fourth downsampling unit is used as the input of the first upsampling unit; Using the output of the first up-sampling unit and the output of the third down-sampling unit as the input of the second up-sampling unit; Using the output of the second up-sampling unit and the output of the second down-sampling unit as the input of the third up-sampling unit; Using the output of the third up-sampling unit and the output of the first down-sampling unit as inputs of a fourth up-sampling unit; Using the output of the fourth upsampling unit and the output of the first ResBlock module as the input of the second ResBlock module; The output of the second ResBlock module is used as the output of the first HG-Net network.

5. The method for generating a hologram based on deep learning and angular spectrum method according to claim 4, characterized in that: The first ResBlock module includes a first convolutional layer, a first BN layer, a first ReLU layer, a second convolutional layer, a second BN layer, and a second ReLU layer in sequence; The input of the first ResBlock module passes through the first convolutional layer, the first BN layer, the first ReLU activation function layer, the second convolutional layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first ResBlock module and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first ResBlock module.

6. The method for generating a hologram based on deep learning and angular spectrum method according to claim 5, characterized in that: The first downsampling unit includes a maximum pooling layer, a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer in sequence; The input of the first downsampling unit passes through the maximum pooling layer, the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first downsampling unit and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first downsampling unit; The second down-sampling unit, the third down-sampling unit, and the fourth down-sampling unit have the same structure and working process as the first down-sampling unit.

7. The method for generating a hologram based on deep learning and angular spectrum method according to claim 6, characterized in that: The first upsampling unit includes a first convolutional layer, a first BN layer, a first ReLU activation function layer, a second convolutional layer, a second BN layer, and a second ReLU activation function layer in sequence; The input of the first upsampling unit passes through the first convolution layer, the first BN layer, the first ReLU activation function layer, the second convolution layer, the second BN layer, and the second ReLU activation function layer in sequence, and then the input of the first upsampling unit and the output of the second ReLU activation function layer are spliced, and the splicing result is used as the output of the first upsampling unit; The second up-sampling unit, the third up-sampling unit, and the fourth up-sampling unit have the same structure and working process as the first up-sampling unit.

8. The method for generating a hologram based on deep learning and angular spectrum method according to claim 7, characterized in that: In step 4, the optimizer used in training is RMSProp.

9. The method for generating a hologram based on deep learning and angular spectrum method according to claim 8, characterized in that: In the step 4, the loss function used in training is MSEloss.

10. The method for generating a hologram based on deep learning and angular spectrum method according to claim 9, characterized in that: In step 4, the learning rate is set to 0.001, the batch_size is set to 8, and the epoch is set to 100 during training.

Citation Information

Patent Citations

  • Dense sample lensless microscopic imaging device and method based on deep learning

    CN110308547A

  • Method for improving digital holographic reproductive image quality

    CN116362997A

  • Real fuzzy three-dimensional hologram reconstruction method based on joint training of multiple neural networks

    CN117876591A

  • Harmonic trap circuit using shunt stub and class-f power amplifier including the same

    KR1020230026200A