Phase retrieval method for lensless holographic microscopy based on decoupling-fusion network
By adopting the combination of decoupling-fusion network and Fresnel diffraction model in lensless holographic microscopy, the problems of poor interpretability and high time cost of existing phase recovery methods are solved, achieving high accuracy and low cost phase recovery effect.
Patent Information
- Application Number
- CN202210177683.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-02-25
AI Technical Summary
The existing phase recovery methods have problems such as poor interpretability, high time cost and low image reconstruction accuracy.
Using the lensless holographic microscopy imaging phase recovery method based on the decoupling-fusion network, using multi-frame holograms acquired at different defocus distances as input, combined with the Fresnel diffraction model, the network parameters are updated through the gradient descent method to realize the self-important Supervised learning, reduce reconstruction errors, and improve generalization performance and interpretability.
It improves the generalization performance and interpretability of the phase recovery network, reduces the time cost, improves the reconstruction accuracy and visual effect, reduces the number of gradient descent rounds, and achieves rapid convergence.
Smart Images

Figure CN114529476B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of computer vision and microscopic imaging, and more specifically, to a lensless holographic microscopic imaging phase recovery method based on a decoupling-fusion network, which can be used for amplitude and phase contrast microscopic imaging of a coaxial holographic lensless microscopic imaging system. Background Art
[0002] In medical and biological research, microscopes are a commonly used research tool, which plays an extremely important role in observing pathological sections and microbial structures. However, due to its structural characteristics and imaging principles, traditional microscopes will bring problems such as incompatibility between field of view and magnification, lens aberration and chromatic distortion, and cannot record sample depth information, and have poor imaging for translucent phase-type objects. Lensless holographic microscopy technology is a technology based on the principle of optical holography, which uses photoelectric sensors such as CMOS or CCD to approach the sample plane, collect sample brightness images in coherent light fields or partially coherent light fields, and use algorithms to restore the complete wavefront information of the sample plane.
[0003] In the lensless holographic microscopy system, the sample plane is brought close to the CMOS or CCD photoelectric sensor plane. The axial distance between the two planes is called the defocus distance, which is usually between a few hundred micrometers and a few millimeters. The system uses a light source that emits completely coherent light or partially coherent light to illuminate the sample plane so that it is projected onto the CMOS or CCD photoelectric sensor plane. The imaging process approximates the Fresnel diffraction model in the near field region. The brightness image recorded by the CMOS or CCD photoelectric sensor is the interference hologram of the sample in the light field. From the collected single-frame or multi-frame hologram, the phase recovery algorithm can be used to solve the complex amplitude information of the wavefront containing amplitude and phase on the sample plane, thereby realizing amplitude or phase difference imaging of the sample plane. Compared with traditional optical microscopes, the lensless microscopy system has the characteristics of a large field of view, a simple system structure, no lens distortion, aberration, etc., and the depth information can be calculated from the recovered phase information to achieve three-dimensional imaging.
[0004] In lensless holographic microscopy imaging systems, the common methods for solving the phase using the collected hologram brightness image can be divided into two categories: phase recovery methods based on traditional iterations and phase recovery methods based on neural networks. Among them, the phase recovery methods based on traditional iterations, for example, the GS phase iteration algorithm proposed by Gerchberg and Saxton in the article entitled "A practical algorithm for the determination of the phase from image and diffraction plane pictures" in 1972, randomly generates the initial phase, synthesizes the complex amplitude with the brightness image collected from the sample plane, iteratively projects the complex amplitude between the sample plane and the imaging plane, and replaces the amplitude with the known brightness images on the two planes. The phase information is corrected during the iteration process until the termination condition is met and the iteration is stopped. This method can be used to obtain phase information that meets certain error requirements. Some subsequent studies proposed phase recovery algorithms based on iterative methods, which are mostly improved on the GS iterative phase recovery algorithm. This type of iterative algorithm will fall into a local optimal solution after several rounds of iterations, resulting in a slow decrease in error, and inevitably requires a large amount of calculation, which is time-consuming and costly.
[0005] As deep learning has made remarkable progress in image restoration, image reconstruction and other image problems, many scholars have begun to explore the application of neural networks in phase recovery to solve many problems faced by traditional methods. At present, typical phase recovery methods based on deep learning are as follows: Yichen Wu et al. published an article titled "Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery" in the journal "Optica" in 2018, disclosing a phase recovery method based on deep learning, which uses a U-shaped network structure to achieve end-to-end phase recovery. This method is trained on a large number of data sets with real labels to learn the mapping relationship between the brightness image of a single-frame hologram collected at a fixed defocus distance and the complex amplitude of the sample plane. The advantage of this method is that in the application, the accuracy requirement for accurately obtaining the defocus distance from the sample plane to the imaging plane is not high, because this method is robust within a certain defocus distance range. However, this method requires a large amount of data with real labels for training, so the workload of obtaining the dataset is huge, and the network has poor interpretability. In addition, the reconstruction performance on experimental data that is out of the style of the dataset is poor, and it has limitations in generalization performance.
[0006] In 2020, Fei Wang et al. published an article titled "Phase imaging with an untrained neural network" in the journal Light: Science & Applications, disclosing a phase recovery method that combines physical models with deep neural networks. The method uses a codec network structure. The encoding end uses a U-type network to generate a prediction of the required complex amplitude. The decoding end uses the known Fresnel diffraction physical model to forward propagate the complex amplitude predicted by the encoding end to the imaging plane and calculate the brightness information. The pixel value error is calculated with the actual brightness image collected on the imaging plane, and the gradient descent method is used to optimize the network parameters to achieve self-supervised learning, so there is no need to obtain the true label of the data. In addition, this deep learning method does not require training. The network structure is designed with manual prior knowledge, and there is no need to obtain learning prior knowledge from the training set. When used, a single-frame hologram is input, and the model parameters are trained in the self-supervised learning process to reduce the loss function. Finally, the result of the encoding end network output is used as the prediction of the complex amplitude. This non-end-to-end network structure also has a large number of iterative processes when solving the phase, and the time cost is high. Moreover, this method and the aforementioned method of Yichen Wu et al. only use one defocused hologram as a constraint, so the reconstruction result has a high error and the visual effect is poor.
[0007] In summary, existing phase retrieval methods have the problems of poor interpretability, high time cost, and low image reconstruction accuracy. Summary of the invention
[0008] The purpose of the present invention is to provide a lensless holographic microscopy phase recovery method based on a decoupling-fusion network to address the deficiencies in the above-mentioned prior art, so as to solve the problems of poor interpretability, high time cost and low image reconstruction accuracy existing in the existing phase recovery methods.
[0009] The core technical idea of the phase recovery method of the present invention is: use multiple frames of holograms collected at different defocus distances as the input of the phase recovery network, use the output of the decoupling-fusion network as the predicted complex amplitude, and forward propagate it to each defocus plane in combination with the known Fresnel diffraction model, calculate the mean absolute error with the brightness image actually collected at each defocus plane, sum the mean absolute errors of each defocus plane as the total loss function, and use the gradient descent method to update the phase recovery network parameters. The training process of the phase recovery network is self-supervised, so there is no need to obtain the real complex amplitude label of the data. The present invention saves the parameters as good initial values after pre-training on a small data set collected or simulated. When solving the actual phase recovery problem, there is no need to start a large number of gradient descent processes from random parameters. Its convergence speed is faster than that of an untrained generative network. The method of the present invention requires less time and has low time cost. Since the convolutional neural network has a clear task in the decoupling-fusion network proposed by the present invention, it learns a simple nonlinear mapping rather than a complex image reconstruction inverse process, thereby improving the generalization performance and interpretability of the phase recovery network. In addition, the method of the present invention uses multi-frame constraints instead of single-frame constraints, which reduces reconstruction errors and improves the visual effects of amplitude or phase contrast imaging. Therefore, the reconstruction accuracy of the method of the present invention is high and the visual effect of the reconstructed image is better.
[0010] Specifically, the technical solution adopted by the present invention is as follows:
[0011] The present application provides a lensless holographic microscopy phase recovery method based on a decoupling-fusion network, the method comprising the following steps: S1, measuring the parameters of the lensless holographic microscopy system to be recovered; S2, acquiring samples and constructing a training sample set and a test sample set; S3, constructing a phase recovery network; S4, training the constructed phase recovery network; S5, performing phase recovery to solve the complex amplitude.
[0012] Furthermore, the parameters in step S1 include the central wavelength of the coherent light source, the pixels of the photoelectric sensor, and the defocus distance of the defocus plane from the sample plane.
[0013] Furthermore, the samples in step S2 are brightness images with a resolution of N×N of M samples collected in the lensless holographic microscopy imaging system to be restored at s defocus planes, and M groups of brightness image data are obtained, with a total of M×s brightness images, where M≥300, s≥2, and N=768.
[0014] Furthermore, the training sample set and the test sample set in step S2 are obtained by dividing the M groups of brightness image data in a ratio of 9:1.
[0015] Furthermore, the phase recovery network in step S3 includes a decoupling network, a reverse propagation Fresnel diffraction layer, a fusion network, a forward propagation Fresnel diffraction layer, and a brightness extraction layer.
[0016] Furthermore, the decoupled network includes four downsampling modules, four upsampling modules, and two ordinary convolution modules.
[0017] Furthermore, the downsampling module includes three convolutional layers and a maximum pooling layer for downsampling and uses a ReLU activation function, and the upsampling module includes two convolutional layers and a deconvolutional layer for upsampling and uses a ReLU activation function.
[0018] Furthermore, the fusion network includes four downsampling modules, four upsampling modules, and two ordinary convolution modules.
[0019] Furthermore, the back-propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse transform operator of the discrete Fourier transform.
[0020] Furthermore, the forward propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse transform operator of the discrete Fourier transform.
[0021] Compared with the prior art, the present invention has the following beneficial effects:
[0022] (1) The present invention uses a decoupling network to decouple dual-channel complex matrix information from a single-channel holographic brightness image, uses a fusion network to fuse multi-frame acquisition information, and combines it with the Fresnel diffraction physical model. The network learning goal is clear and the interpretability is strong.
[0023] (2) The present invention uses multiple frames of brightness images collected at different defocus distances to extract information and uses them as loss function constraints. Compared with the single-frame recovery method, the phase recovery value is closer to the true value, the reconstruction accuracy is higher, and the visual effect of the reconstructed phase difference and amplitude images is good;
[0024] (3) The phase recovery network used in the present invention is trained on a small data set and the parameters are saved as the initial values of the gradient descent. Compared with the traditional deep learning method without training, the number of gradient descent rounds required is fewer, and the phase recovery speed is improved by learning prior knowledge. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 A schematic diagram of a lensless holographic microscopy imaging phase recovery method based on a decoupling-fusion network provided by the present invention;
[0026] Figure 2A schematic diagram of a phase recovery network constructed in step S3 of the lensless holographic microscopy imaging phase recovery method based on a decoupling-fusion network provided by the present invention;
[0027] Figure 3 To collect the brightness images of the USAF1951 resolution plate at different defocus distances, Figure 3 (a) Figure 3 (b) Figure 3 (c) Figure 3 (d) The corresponding defocus distances are 0.710mm, 1.185mm, 1.685mm, and 2.178mm respectively;
[0028] Figure 4 To collect brightness images of small intestinal epithelial cell sample slices at different defocus distances, Figure 4 (a) Figure 4 (b) Figure 4 (c) Figure 4 (d) The corresponding defocus distances are 0.865mm, 1.305mm, 1.804mm, and 2.304mm respectively;
[0029] Figure 5 is the result of phase recovery using the GS iterative algorithm, where Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) The amplitude imaging results of the USAF1951 resolution plate, the phase contrast imaging results of the USAF1951 resolution plate, the amplitude imaging results of the small intestinal epithelial cell sample slice, and the phase contrast imaging results of the small intestinal epithelial cell sample slice after 300 iterations of the GS iterative algorithm;
[0030] Figure 6 This is the result of phase recovery using the deep learning algorithm proposed by Fei Wang et al. Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) The amplitude imaging results of the USAF1951 resolution plate, the phase contrast imaging results of the USAF1951 resolution plate, the amplitude imaging results of the small intestinal epithelial cell sample slice, and the phase contrast imaging results of the small intestinal epithelial cell sample slice after 300 rounds of gradient descent using the deep learning algorithm proposed by Fei Wang et al.;
[0031] Figure 7 is the result of phase recovery performed by the method of the present invention, wherein Figure 7 (a), Figure 7(b), Figure 7 (c) Figure 7(d) are respectively the amplitude imaging results of the USAF1951 resolution plate, the phase contrast imaging results of the USAF1951 resolution plate, the amplitude imaging results of the small intestinal epithelial cell sample slices, and the phase contrast imaging results of the small intestinal epithelial cell sample slices after 100 rounds of gradient descent by the method of the present invention. DETAILED DESCRIPTION
[0032] In order to make the implementation process of the present invention clearer, it will be described in detail below with reference to the accompanying drawings.
[0033] The present invention provides a lensless holographic microscopy imaging phase recovery method based on a decoupling-fusion network, such as Figure 1 As shown, the specific steps are as follows:
[0034] S1, measuring the parameters of the lensless holographic microscopy system to be restored;
[0035] Lensless holographic microscopy technology is a technology based on the principle of coaxial holography. It uses photoelectric sensors such as CMOS / CCD to approach the sample plane, collect the brightness image of the sample in a coherent light field or a partially coherent light field, and the brightness image is the obtained holographic image. The algorithm is used to restore the complete wavefront information of the sample plane. Specifically, the central wavelength of the coherent light source, the pixel size of the photoelectric sensor, and the defocus distance of the defocus plane from the sample plane are all parameters in the transfer function. In order to obtain the parameters in the transfer function, the central wavelength λ of the coherent light source is measured, and the pixel size p of the photoelectric sensor is obtained. The sensor in this embodiment takes a CMOS sensor as an example, and measures the s defocus distances {L r |r∈1,2,…,s}, specifically, since Fresnel diffraction occurs in the near field, the defocus distance is 0.1mm-3mm, so the Fresnel diffraction formula can be used to approximate the optical propagation process. The sample can be fixed, that is, the sample plane is fixed, and the photoelectric sensor is moved, that is, the defocus plane is moved, or the photoelectric sensor can be fixed and the sample is moved; the present invention is described by taking a system of a fixed sample and a mobile photoelectric sensor as an example, which is conducive to maintaining the stability of the sample.
[0036] S2, obtain samples and construct training sample sets and test sample sets;
[0037] The samples of this embodiment are brightness images with a resolution of N×N of M samples collected in a lensless holographic microscopy imaging system at s defocused planes, and M groups of brightness image data are obtained, with a total of M×s brightness images, where M≥300, so that the sufficiency of the training data set can be guaranteed; s≥2, at least two frames, to ensure that multi-frame reconstruction can be performed; N=768, which can adapt to the video memory size of this embodiment. This embodiment uses bmp format images, and the network structure is relatively large. For a graphics card with 12GB video memory, it is more suitable to use images with a resolution of 768×768, while for other graphics card devices with smaller or larger video memory, the value of N needs to be adjusted; in addition, the parameters trained on the training sample set with a resolution of 768×768 can also complete the recovery work for inputs of other resolutions. When implemented, the trained phase recovery network can be recovered using images with a resolution of 512×512 or 1024×1024. Because the convolution operation of the neural network is a 3×3 convolution kernel sliding operation, there is no mandatory requirement for the input resolution. Randomly, the M groups of brightness image data are divided into a training sample set and a test sample set in a ratio of 9:1, that is, 9M / 10 groups of brightness image data in the M groups of brightness image data constitute the training sample set, and the remaining 1M / 10 groups of brightness image data constitute the test sample set.
[0038] S3, build phase recovery network;
[0039] like Figure 2 As shown, the present invention replaces the nonlinear mapping part in the inverse problem of image reconstruction with a phase recovery network. The phase recovery network uses multiple frames of brightness images collected at different defocus distances as input. The decoupling network is used to map the single-channel brightness image collected in each frame into a dual-channel complex matrix, and back-propagates to the sample plane through the Fresnel diffraction model. The information of multiple complex matrices is fused through the fusion network, and the predicted complex amplitude is output. The loss function end forward propagates the predicted complex amplitude to the collection plane at each defocus distance through the known Fresnel diffraction model, calculates the mean absolute error with the brightness image actually collected on the corresponding plane, and optimizes the parameters of the phase recovery network through the back propagation of the gradient using the gradient descent method. Figure 2 There are s brightness extraction layers, reconstruction losses, decoupling networks, back-propagation Fresnel diffraction layers, and forward-propagation Fresnel diffraction layers in the training, corresponding to samples collected at s defocus distances respectively. The s reconstruction losses are combined into a total loss function. As the training progresses, the parameters of the s decoupling networks are different due to different mapping relationships.
[0040] S31, build a phase recovery network;
[0041] like Figure 2As shown in Figure 1, the phase recovery network includes a decoupling network, a back-propagation Fresnel diffraction layer, a fusion network, a forward-propagation Fresnel diffraction layer, and a brightness extraction layer. Both the decoupling network and the fusion network use a common U-shaped network structure, but the number of channels in each convolutional layer is different.
[0042] The decoupled network includes four downsampling modules, four upsampling modules, and two ordinary convolution modules, which are used to transform the defocus plane brightness image brightness matrix {I r |r∈1,2,…,s} decoupled into a complex matrix of two-channel tensors The first channel tensor is a real matrix, the second channel tensor is an imaginary matrix, and the light wave itself is a complex matrix, which can better reflect the actual imaging process. Specifically, each downsampling module includes three convolutional layers and a maximum pooling layer used for downsampling and uses the ReLU function as the activation function of the first two convolutional layers. The convolutional layer is used to extract features, so that the number of channels gradually increases, and the pooling operation will reduce the resolution. Each upsampling module includes a deconvolution layer used for upsampling and two convolutional layers and uses the ReLU activation function, where the deconvolution layer is used to increase the resolution. The residual connection is used to add the output tensor of the second convolutional layer in the downsampling module and the tensor of the same size output by the third convolutional layer. The residual structure network is easy to optimize, which can alleviate the problem of gradient vanishing caused by the increase in depth of the deep neural network. At the same time, it can also improve the training speed of the decoupled network and reduce the time cost; the skip layer connection is used to connect the tensor output by the maximum pooling layer of the downsampling module and the tensor of the same size output by the deconvolution layer in the corresponding upsampling module along the channel direction as the input of the first convolutional layer in the upsampling module. In this way, the shallower convolutional layer features can be brought out, rich low-level information can be retained, and the low-level image information loss and resolution reduction caused by the pooling operation can be compensated, thereby improving the accuracy of image reconstruction, and reducing the problem of gradient vanishing and network degradation. The specific parameter settings related to the downsampling module, upsampling module, and ordinary convolution module are detailed in Table 1. More specifically, the settings of the decoupling network are as follows: first downsampling module → second downsampling module → third downsampling module → fourth downsampling module → first ordinary convolution module → first upsampling module → second upsampling module → third upsampling module → fourth upsampling module → second ordinary convolution module, wherein the first ordinary convolution module is used to connect the upsampling and downsampling paths of the decoupling network, and the second ordinary convolution module is used to enable the decoupling network to output tensor information of the required size.
[0043] The back propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse transform operator of the discrete Fourier transform. It is used to convert the complex matrix output by the s decoupled networks The linear operation of back-propagating Fresnel diffraction from the defocused plane to the sample plane is denoted as {F-1 r |r∈1,2,…,s}:
[0044]
[0045] Where F represents the discrete Fourier transform operator; F -1 represents the inverse transform operator of discrete Fourier transform; f x and f y is the frequency domain unit, H r (f x ,f y ) represents the transfer function matrix, which is expressed as:
[0046]
[0047] Where j is an imaginary unit; L r is the defocus distance from the rth defocus plane to the sample plane; λ is the central wavelength of the light source; the wave number k = 2π / λ; the transfer functions generated at different defocus distances are different.
[0048] The fusion network includes four downsampling modules, four upsampling modules, and two ordinary convolution modules, which are used to fuse the information of the hologram brightness image collected from multiple frames and generate the complex amplitude predicted by the sample plane. Specifically, the input layer sums the s complex matrices output by the back-propagation Fresnel diffraction layer. Each downsampling module includes three convolutional layers and a maximum pooling layer used for downsampling and uses the ReLU activation function. The convolutional layer is used to extract features so that the number of channels gradually increases, and the pooling operation reduces the resolution. Each upsampling module includes two convolutional layers and a deconvolutional layer used for upsampling and uses the ReLU activation function. The deconvolutional layer is used to increase the resolution. The specific parameter settings are shown in Table 1. The residual connection is used to add the output tensor of the second convolutional layer in the downsampling module and the tensor of the same size output by the third convolutional layer. The residual structure network is easy to optimize, which can alleviate the problem of gradient vanishing caused by the increase in depth of the deep neural network. At the same time, it can also improve the training speed of the decoupled network and reduce the time cost; the skip layer connection is used to connect the tensor output by the maximum pooling layer of the downsampling module and the tensor of the same size output by the deconvolution layer in the corresponding upsampling module along the channel direction as the input of the first convolutional layer in the upsampling module. In this way, the shallower convolutional layer features can be brought out, rich low-level information can be retained, and the low-level image information loss and resolution reduction caused by the pooling operation can be compensated, thereby improving the accuracy of image reconstruction, and reducing the problem of gradient vanishing and network degradation. The specific parameter settings related to the downsampling module, upsampling module, and ordinary convolution module are detailed in Table 1. More specifically, the settings of the fusion network are as follows: the fifth downsampling module → the sixth downsampling module → the seventh downsampling module → the eighth downsampling module → the third ordinary convolution module → the fifth upsampling module → the sixth upsampling module → the seventh upsampling module → the eighth upsampling module → the fourth ordinary convolution module, wherein the third ordinary convolution module is used to connect the upsampling and downsampling paths on the fusion network, and the fourth ordinary convolution module is used to enable the fusion network to output tensor information of the required size.
[0049] The forward propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse transform operator of the discrete Fourier transform. Forward propagation from the sample plane to s defocused planes, outputting the complex amplitude prediction matrix of s defocused planes
[0050]
[0051] Where F represents the discrete Fourier transform operator; F -1 H stands for the inverse transform operator of discrete Fourier transform; r (f x ,f y ) represents the transfer function matrix, f x and f y is the frequency domain unit, is the complex amplitude predicted by the fusion network in the sample plane.
[0052] The brightness extraction layer includes a calculation module for predicting the complex amplitude matrices of the s defocus planes output from the forward propagation Fresnel diffraction layer Extract the brightness image matrix from This makes it easier to compare the actual brightness image detected by the photoelectric sensor and calculate the loss. The expression for extracting brightness information is:
[0053]
[0054] in, represent The first channel of the tensor represents the real part of the complex amplitude; represent The second channel of the tensor represents the imaginary part of the complex amplitude.
[0055] The present invention uses a decoupling network to decouple dual-channel complex matrix information from a single-channel holographic brightness image, uses a fusion network to fuse multi-frame acquisition information, and combines it with the Fresnel diffraction physical model. The network learning goal is clear and the interpretability is strong.
[0056] Table 1: Parameter settings of the decoupling network and fusion network in the phase recovery network.
[0057]
[0058]
[0059]
[0060] S32, define the total loss function L of the phase recovery network.
[0061] The expression of the total loss function L of the phase recovery network is as follows:
[0062]
[0063] Among them, mean(*) is a pixel-by-pixel mean operator. is the hologram brightness matrix predicted by the rth defocus plane output by the brightness extraction layer, I r is the hologram brightness matrix actually collected at the rth defocus plane. The 1-norm is adopted, which is friendly to detail information and can effectively improve the resolution. The expression of the total loss function L can obtain the difference between the collected brightness and the brightness obtained by brightness extraction according to the predicted complex amplitude. The smaller the difference, the more accurate the predicted complex amplitude is, and the larger the difference, the less accurate the predicted complex amplitude is, which can reflect the accuracy of phase recovery of the method of the present invention.
[0064] The present invention utilizes multi-frame holograms collected at different defocus distances to extract information and uses them as loss function constraints. Compared with a single-frame recovery method, a phase recovery value is closer to the true value, the reconstruction accuracy is higher, and the visual effects of the reconstructed phase difference and amplitude images are good.
[0065] S4, training the constructed phase recovery network;
[0066] The brightness images {I r |r∈1,2,…,s} is used as the input of the phase recovery network of the present invention, and the phase recovery network is iteratively trained J times, where J is greater than 100, so as to ensure that the network is fully trained. Specifically, it is 300 times in this embodiment. Each round needs to traverse all training sample sets and test sample sets. First, all samples in the training sample set are sequentially input into the phase recovery network to train the network and optimize the network parameters; then all samples in the test sample set are sequentially input into the phase recovery network to test whether the trained network needs to continue training. After the test sample set meets certain loss function requirements, the parameters of the phase recovery network are saved; in this embodiment, the Adam optimizer provided by the PyTorch framework is selected to optimize the parameters in the phase recovery network, and the learning rate is set to 0.0001. The average loss on the test sample set is required to be less than or equal to 3, so as to ensure the accuracy of the prediction. The average loss refers to the average of the total loss L of all test samples.
[0067] The phase recovery network used in the present invention only needs to be trained on a small data set and does not require a large amount of data. It can effectively reduce the training time and time cost, save parameters as initial values of gradient descent, and require fewer gradient descent rounds than traditional deep learning methods without training. The phase recovery speed is improved by learning prior knowledge.
[0068] S5, perform phase recovery to solve the complex amplitude;
[0069] The parameters saved in step S4 are loaded into the phase recovery network. Under the same system parameters as those used for phase recovery network training, the brightness image data of the sample under test at s defocused planes are collected and input into the phase recovery network. The total loss function is calculated. The Adam optimizer is used to perform gradient descent optimization parameters K rounds again until the total loss function on the input data satisfies L≤α, where α is a manually selected loss function threshold. Specifically, the loss function threshold α is equal to 0.5 to ensure the phase recovery accuracy of the phase recovery network. The prediction result of the complex amplitude of the sample plane output by the fusion network is This is the required complex amplitude, that is, obtaining the complex amplitude completes the phase recovery.
[0070] Implementation conditions and result analysis of the embodiments of the present invention:
[0071] 1. Implementation conditions;
[0072] The method of the present invention is applicable to any mutually compatible hardware and software platforms. The hardware test platform used in this embodiment is: Intel Core i7 CPU with a main frequency of 3.60GHz and 16GB of memory; GPU is: NVIDIA TITAN XP, 12GB of video memory; software simulation platform is: Windows 10 64-bit operating system; software simulation language: Python; deep learning framework: PyTorch.
[0073] 2. Result analysis.
[0074] The present invention, the GS iterative algorithm and the method of Fei Wang et al. are used to perform phase recovery on the experimental data collected by the same system. The collected images are as follows: Figure 3 As shown, the phase difference and amplitude imaging results are as follows Figure 4 shown.
[0075] Figure 3 To collect the brightness images of the USAF1951 resolution plate at different defocus distances, Figure 3 (a) Figure 3 (b) Figure 3 (c) Figure 3 (d) The corresponding defocus distances are 0.710mm, 1.185mm, 1.685mm, and 2.178mm, respectively. The present invention uses multiple frames of brightness images collected at different defocus distances to extract information and as a loss function constraint. The phase recovery value is closer to the true value than the single-frame recovery method, the reconstruction accuracy is higher, and the visual effect of the reconstructed phase difference and amplitude images is good.
[0076] Figure 4 To collect brightness images of small intestinal epithelial cell sample slices at different defocus distances, Figure 4 (a) Figure 4 (b) Figure 4 (c) Figure 4 (d) The corresponding defocus distances are 0.865mm, 1.305mm, 1.804mm, and 2.304mm, respectively. The present invention uses multiple frames of brightness images collected at different defocus distances to extract information and as a loss function constraint. The phase recovery value is closer to the true value than the single-frame recovery method, the reconstruction accuracy is higher, and the visual effect of the reconstructed phase difference and amplitude images is good.
[0077] Figure 5 is the result of phase recovery using the GS iterative algorithm, where Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) The amplitude imaging results of the USAF1951 resolution plate, the phase contrast imaging results of the USAF1951 resolution plate, the amplitude imaging results of the small intestinal epithelial cell sample slice, and the phase contrast imaging results of the small intestinal epithelial cell sample slice after 300 rounds of gradient descent using the GS iterative algorithm. Specifically, the method disclosed by Gerchberg and Saxton in the article entitled "A practical algorithm for the determination of the phase from image and diffraction plane pictures" in 1972 was used to obtain Figure 5 result.
[0078] Figure 6 This is the result of phase recovery using the deep learning algorithm proposed by Fei Wang et al. Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) The amplitude imaging results of the USAF1951 resolution plate, the phase contrast imaging results of the USAF1951 resolution plate, the amplitude imaging results of the small intestinal epithelial cell sample slice, and the phase contrast imaging results of the small intestinal epithelial cell sample slice after 300 rounds of gradient descent using the deep learning algorithm proposed by Fei Wang et al. Specifically, the method disclosed in an article titled "Phase imaging with an untrained neural network" published by Fei Wang et al. in the journal "Light: Science & Applications" in 2020 was used to obtain Figure 6 The results in .
[0079] Figure 7 is the result of phase recovery performed by the method of the present invention, wherein Figure 7 (a), Figure 7(b), Figure 7 (c) Figure 7 (d) are the amplitude imaging results of the USAF1951 resolution plate, the phase contrast imaging results of the USAF1951 resolution plate, the amplitude imaging results of the small intestinal epithelial cell sample slice, and the phase contrast imaging results of the small intestinal epithelial cell sample slice after 100 rounds of gradient descent of the method of the present invention. On the one hand, the method of the present invention is trained on a small data set and the parameters are saved as the initial values of the gradient descent. Compared with the traditional deep learning method without training, the number of gradient descent rounds required is less. The number of iterations of the method of the present invention, 100, is less than 300 of the two comparison methods. The phase recovery speed of the method of the present invention is faster. On the other hand, by comparing Figure 5 , Figure 6 , Figure 7 It can be seen that the results of phase recovery of the GS iterative algorithm and the existing deep learning method have obvious noise and shadows for the sample amplitude imaging, indicating that the GS iterative algorithm will quickly fall into the local optimal solution, resulting in the phase recovery value not being smooth enough. The method of the present invention has learned a large amount of prior knowledge of data, so the image is smoother, with less noise and less shadows; the existing deep learning method only uses a single image, so the constraints are not enough, resulting in inaccurate phase recovery values. The method of the present invention uses multiple frames, has more constraints, better recovery effect, and more accurate values. The amplitude recovery results obtained by the method of the present invention have almost no noise and shadows; there are different degrees of overexposure for sample phase difference imaging, and the phase recovery results obtained by the method of the present invention have no obvious overexposure phenomenon and have better visual effects. Since the present invention uses multiple frames of brightness images collected at different defocus distances to extract information and as a loss function constraint, the phase recovery value is closer to the true value than the single-frame recovery method, the reconstruction accuracy is higher, and the visual effect of the reconstructed phase difference and amplitude images is good.
[0080] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A lensless holographic microscopy phase recovery method based on a decoupling-fusion network, characterized in that: The method comprises the following steps: S1, measuring the parameters of the lensless holographic microscopy system to be restored; S2, obtain samples and construct training sample sets and test sample sets; S3, build a phase recovery network; S4, training the constructed phase recovery network; S5, perform phase recovery to solve the complex amplitude; Among them, the phase recovery network in step S3 includes a decoupling network, a back-propagation Fresnel diffraction layer, a fusion network, a forward propagation Fresnel diffraction layer, and a brightness extraction layer; the decoupling network includes four down-sampling modules, four up-sampling modules, and two ordinary convolution modules; the down-sampling module includes three convolution layers and a maximum pooling layer used for down-sampling and uses a ReLU activation function, and the up-sampling module includes two convolution layers and a deconvolution layer used for up-sampling and uses a ReLU activation function; the fusion network includes four down-sampling modules, four up-sampling modules, and two ordinary convolution modules; the back-propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse transform operator of a discrete Fourier transform; the forward propagation Fresnel diffraction layer includes a discrete Fourier transform operator, a transfer function product module, and an inverse transform operator of a discrete Fourier transform.
2. The lensless holographic microscopy imaging phase recovery method based on a decoupling-fusion network according to claim 1 is characterized in that: The parameters in step S1 include the central wavelength of the coherent light source, the pixels of the photoelectric sensor, and the defocus distance of the defocus plane from the sample plane.
3. The lensless holographic microscopy imaging phase recovery method based on a decoupling-fusion network according to claim 2 is characterized in that: The sample in step S2 is collected in the lensless holographic microscopy imaging system to be restored. M Samples in s The resolution of the out-of-focus plane is N × N The brightness image is M Group brightness image data, total M × s brightness images, where .
4. The lensless holographic microscopy imaging phase recovery method based on a decoupling-fusion network according to claim 3 is characterized in that: The training sample set and the test sample set in step S2 are M The group brightness image data is obtained by dividing it in a ratio of 9:1.
