Dual-Modal Digital Holographic Imaging Method, System and Memory Based on Deep Learning

Abstract

The present invention relates to the field of holographic imaging technology, and in particular to a dual-modal digital holographic imaging method, system and memory based on deep learning. A dual-modal digital holographic imaging method based on deep learning proposed by the present invention includes constructing an image restoration network model with a reconstructed image as the input and a restored image as the output, and the image restoration network model corresponds to the light source one by one; performing frequency selection processing on the double-beam interference holographic image, and outputting frequency selection images corresponding to the two light sources respectively. After angular spectrum reconstruction, the two frequency selection images are input into the corresponding image restoration models; the restored images output by the two image restoration models are pixel-overlaid and normalized to obtain a fused restored image. By combining frequency selection processing and deep learning, the present invention effectively eliminates interlayer crosstalk and pseudo-noise, and intuitively demonstrates high-fidelity imaging performance and achieves higher image resolution.

Description

Technical Field

[0001] The present invention relates to the field of holographic imaging technology, and in particular to a dual-modal digital holographic imaging method, system and memory based on deep learning. Background Art

[0002] Multimodal 3D imaging has become one of the research hotspots in the scientific and industrial communities. Digital holography provides technical support for 3D reconstruction by simultaneously recording multi-dimensional information (including the amplitude and phase of an object) on a 2D recording medium. Digital holography can be divided into incoherent digital holography and coherent digital holography.

[0003] Incoherent digital holography generates a self-interference hologram through phase encoding, and uses phase-shifting interferometry (PSI) to decode and reconstruct target waves with different wavelengths and polarization directions. Incoherent holographic technology has demonstrated that the interference imaging principle violates the Lagrange invariant, showing the potential of super-resolution imaging.

[0004] Compared with the off-axis configuration, the self-interference method combined with PSI can generate an incoherent hologram with a relatively large spatial bandwidth product (SBWP), and is capable of holographically sensing a three-dimensional field through a frequently changing point spread function (psf).

[0005] Coherent holography usually uses an off-axis method to separate the target wavefront from crosstalk terms (DC and conjugate terms). Due to the excellent coherence of the laser, long-distance imaging does not cause extensive diffusion of the wavefront. In addition, the phase-shifting interferometry is also applicable to the reconstruction of incoherent holograms, but it will introduce scattering noise.

[0006] When incoherent and coherent wavefronts are mixed and recorded holographically, it is difficult for conventional methods to effectively separate and reconstruct the superimposed holograms of these two modes. Summary of the Invention

[0007] In order to overcome the above-mentioned dual-modal holographic reconstruction problem in the prior art, the present invention proposes a dual-modal digital holographic imaging method based on deep learning, which realizes holographic reconstruction with dual fields of view, high fidelity and high resolution.

[0008] A dual-modal digital holographic imaging method based on deep learning proposed by the present invention includes the following steps:

[0009] S1. Construct an image restoration network model with the reconstructed image as the input and the restored image as the output. The image restoration network model corresponds to the light source one by one;

[0010] S2. Perform frequency selection processing on the dual-beam interference holographic image, and output frequency selection images corresponding to the two light sources respectively. The two frequency selection images are input into the corresponding image restoration models;

[0011] S3. Superimpose and normalize the restored images output by the two image restoration models to obtain a fused restored image.

[0012] Preferably, the image restoration network model includes a first convolutional unit, a second convolutional unit, a third convolutional unit, a fourth convolutional unit, a fifth convolutional unit, a sixth convolutional unit, a first deconvolutional unit, a second deconvolutional unit, a third deconvolutional unit, a fourth deconvolutional unit, and a fully connected layer connected in sequence; in the first to sixth convolutional units, the output of the previous convolutional unit is input to the subsequent convolutional unit after being processed by max pooling; the output of the Nth convolutional unit is also connected to the input of the (5 - N)th deconvolutional unit, where 1 ≤ N < 5.

[0013] Preferably, the first convolutional unit, the second convolutional unit, the third convolutional unit, the fourth convolutional unit, the fifth convolutional unit, the sixth convolutional unit, the first deconvolutional unit, the second deconvolutional unit, the third deconvolutional unit, and the fourth deconvolutional unit are all composed of a convolutional layer, a BatchNorm2d function, and an activation function stacked in sequence.

[0014] Preferably, in step S2, a double - beam interference holographic image is collected through an optical system.

[0015] The optical system includes: a first optical path, a second optical path, and an interference optical path.

[0016] The first optical path includes, arranged along the light propagation direction: an electric light source, a collimating lens, a lens, and a first microscopic objective lens.

[0017] The second optical path includes, arranged along the light propagation direction: a laser, a beam diffuser, a second microscopic objective lens, and a phase - shift module.

[0018] The intersection position of the first optical path and the second optical path is the detection position, which is used to place the detection target; on the first optical path, the detection position is between the lens and the first microscopic objective lens; on the second optical path, the detection position is between the beam diffuser and the second microscopic objective lens.

[0019] The interference optical path includes: a beam splitter, a spatial light modulator, and an electrical coupling device.

[0020] The outgoing light of the first microscopic objective lens and the outgoing light of the phase - shift module are combined by the beam splitter and then transmitted to the spatial light modulator, and the outgoing light of the spatial light modulator is collected by the electrical coupling device to generate a hologram.

[0021] When both the electric light source and the laser are started, the hologram collected by the electrical coupling device is the double - beam interference holographic image of S2.

[0022] Preferably, the training method of the two - path image restoration network model includes the following steps:

[0023] S11. Place the detection target at the detection position to obtain the training sets corresponding to the two light sources. The method for obtaining the training set of one light source is as follows: Turn on this light source and turn off the other light source. When the SLM loads the mask, the image after angular spectrum diffraction of the hologram collected by the CCD is used as the reconstructed image. When the SLM is replaced by a mirror, the image collected by the CCD is used as the real image. Adjust the light output angle of the electric light source to generate multiple groups of samples {reconstructed image, real image} to form the training set of this light source.

[0024] S12. Train the two image restoration network models on the corresponding training sets respectively.

[0025] S13. When the number of iterations of the two image restoration network models both reaches an integer multiple of the set threshold, calculate the overall loss function L total ;

[0026] If the overall loss function L total converges, fix the two image restoration network models; otherwise, return to step S12.

[0027] Preferably, the overall loss function is:

[0028] L total =( L '1 + L '2) / ( L '1 / L '2)

[0029] where L '1 represents the loss function of the image restoration network model corresponding to the electric light source, L '2 represents the loss function of the image restoration network model corresponding to the laser;

[0030] Or L '2 represents the loss function of the image restoration network model corresponding to the electric light source, L '1 represents the loss function of the image restoration network model corresponding to the laser;

[0031] The loss function of the image restoration network model is the sum of the mean square error loss and the spectral norm regularization term; during the training process of the image restoration network model, minimizing the expectation of the loss function is used as the optimization goal to iterate the image restoration network model.

[0032] Preferably, the method for training the image restoration network model on the training set in S12 includes the following steps:

[0033] Extract training samples from the training set and substitute them into the image restoration network model to enable the image restoration network model to perform autonomous learning;

[0034] Extract verification samples from the training set and input the images into the image restoration network model, and the image restoration network model predicts the restored images of the verification samples;

[0035] Calculate the mean square error between the restored image and the real image on the verification samples, and update the parameters of the image restoration network model according to the optimization objective; the optimization objective is the expectation of the sum of the mean square error and the spectral norm regularization term;

[0036] Repeat the above steps until the number of updates of the image restoration network model reaches an integer multiple of the set threshold.

[0037] Preferably, a reflecting member is provided at the end of the first optical path; a reflecting member is provided on the second optical path between the second microscope objective and the phase shift module.

[0038] A system adopting the dual-modal digital holographic imaging method based on deep learning as described above proposed by the present invention includes an optical system, a frequency selection module, a first image restoration network model, a second image restoration network model and an image superposition module;

[0039] The optical system is used to generate a two-beam interference holographic image, the two-beam interference holographic image is frequency-selected by the frequency selection module to be two frequency-selected images, the two frequency-selected images are respectively processed by the first image restoration network model and the second image restoration network model, and the output images of the first image restoration network model and the second image restoration network model are output after pixel superposition by the image superposition module.

[0040] A memory proposed by the present invention stores a computer program, a first image restoration network model and a second image restoration network model, and the computer program is used to implement the dual-modal digital holographic imaging method based on deep learning when executed.

[0041] The advantages of the present invention are as follows:

[0042] (1) For the dual-modal digital holographic imaging method based on deep learning proposed by the present invention, first, a digital holographic recording is performed by mixing coherent light and incoherent light, then the target wave spectrum is separated by frequency selection processing, and then a neural network is used to convert a low-resolution reconstructed image into a high-resolution reconstructed image, realizing the fusion reconstruction of multi-modal holography. By combining frequency selection processing and deep learning, the present invention effectively eliminates interlayer crosstalk and pseudo-noise, and intuitively demonstrates high-fidelity imaging performance and achieves higher image resolution.

[0043] (2) The present invention adopts phase aperture coding imaging to achieve parallel reconstruction after multi-target wave overlap, obtains a dual FOV (field of view) from the original hologram, has a larger bandwidth compared with traditional holography, and realizes large FOV (field of view) and super-resolution reconstruction. The present invention is applicable to fields such as multi-dimensional information compression, industrial transient recording, and super-resolution imaging.

[0044] (3) The present invention is more friendly to incoherent holography and has less background noise. The dual-modal holographic fusion technology can strip and reconstruct incoherent holographic information with high-fidelity quality in complex environments. This has important value for holographic detection of multiple light sources and interference in harsh environments.

[0045] (4) The present invention simplifies the architecture complexity of the optical system through frequency selection processing, reduces the difficulty of dual-modal holographic fitting, and realizes end-to-end feature extraction and direct mapping.

[0046] (5) The present invention adopts a hybrid loss function during model training, which simultaneously includes root mean square error (RMSE) and spectral norm (SPN-R). Compared with a single loss function that only includes RMSE, the hybrid loss function not only makes the training images closer to the real images in terms of pixels, but also accelerates network convergence and reduces overfitting.

[0047] (6) The present invention includes a frequency selection module for stripping aliased spectra, trains using two different configurations of convolutional neural networks (CNNs), and uses root mean square error (RMSE) and spectral norm (SPN-R) as loss functions to optimize the bias through backpropagation. Obtain the optimal weights under limited training conditions to achieve dual-field-of-view, high-fidelity, and high-resolution holographic reconstruction. Description of the Drawings

[0048] Figure 1 It is a flowchart of a dual-modal digital holographic imaging method based on deep learning.

[0049] Figure 2 It is a topological schematic diagram of a dual-modal digital holographic imaging method based on deep learning.

[0050] Figure 3 It is a structural diagram of an image restoration network model.

[0051] Figure 4 It is a schematic diagram of an optical system;

[0052] 11. Electric light source; 12. Collimating lens; 13. Lens; 14. First microscope objective; 15. First reflector; 21. Laser; 22. Beam expander (BE); 23. Second microscope objective; 24. Phase shift module (PSM); 25. Second reflector; 30. Beam splitter (BS); 40. Spatial light modulator (SLM); 50. Charge-coupled device (CCD); 60. Detection position.

[0053] FIG. 5(a) is a double-beam interference holographic image of the sample in Example 1;

[0054] FIG. 5(b) is a reconstructed image obtained by angular spectrum reconstruction of a selected-frequency image obtained after frequency selection processing of FIG. 5(a);

[0055] FIG. 5(c) is a reconstructed image obtained by angular spectrum reconstruction of another selected-frequency image obtained after frequency selection processing of FIG. 5(a);

[0056] FIG. 5(d) is a reconstructed image obtained by mapping a selected-frequency image obtained after frequency selection processing of FIG. 5(a) through a corresponding image restoration network model;

[0057] FIG. 5(e) is a reconstructed image obtained by mapping another selected-frequency image obtained after frequency selection processing of FIG. 5(a) through a corresponding image restoration network model.

[0058] Figure 6 It is a comparison of the intensity distributions of the reconstructed images of the first selected-frequency image in Example 1 by two methods.

[0059] Figure 7 It is a comparison of the intensity distributions of the reconstructed images of the second selected-frequency image in Example 1 by two methods.

[0060] FIG. 8(a) shows the angular spectrum reconstruction at axial distances equal to 0, 0.25 mm, and 0.5 mm;

[0061] a1 is the conventional imaging of the resolution test chart when Δz = 0, a2 is the conventional imaging of the resolution test chart when Δz = 0.25 mm, and a3 is the conventional imaging of the resolution test chart when Δz = 0.5 mm; b1 is the double-beam interference holographic image of the resolution test chart when Δz = 0, b2 is the double-beam interference holographic image of the resolution test chart when Δz = 0.25 mm, and b3 is the double-beam interference holographic image of the resolution test chart when Δz = 0.5 mm; c1 is the hologram obtained by angular spectrum reconstruction of one selected-frequency image of b1 when Δz = 0, c2 is the hologram obtained by angular spectrum reconstruction of one selected-frequency image of b2 when Δz = 0.25 mm, and c3 is the hologram obtained by angular spectrum reconstruction of one selected-frequency image of b3 when Δz = 0.5 mm; d1 is the hologram obtained by angular spectrum reconstruction of the other selected-frequency image of b1 when Δz = 0, d2 is the hologram obtained by angular spectrum reconstruction of the other selected-frequency image of b2 when Δz = 0.25 mm, and d3 is the hologram obtained by angular spectrum reconstruction of the other selected-frequency image of b3 when Δz = 0.5 mm;

[0062] Figure 8(b) shows the angular spectrum reconstruction at axial distances equal to 0.75 mm, 1 mm, and 1.25 mm;

[0063] a4 is the conventional imaging of the resolution test chart when Δz = 0.75 mm, a5 is the conventional imaging of the resolution test chart when Δz = 1 mm, and a6 is the conventional imaging of the resolution test chart when Δz = 1.25 mm; b4 is the double-beam interference holographic image of the resolution test chart when Δz = 0.75 mm, b5 is the double-beam interference holographic image of the resolution test chart when Δz = 1 mm, and b6 is the double-beam interference holographic image of the resolution test chart when Δz = 1.25 mm; c4 is the hologram obtained by angular spectrum reconstruction of one selected-frequency image of b4 when Δz = 0.75 mm, c5 is the hologram obtained by angular spectrum reconstruction of one selected-frequency image of b5 when Δz = 1 mm, and c6 is the hologram obtained by angular spectrum reconstruction of one selected-frequency image of b6 when Δz = 1.25 mm; d4 is the hologram obtained by angular spectrum reconstruction of the other selected-frequency image of b4 when Δz = 0.75 mm, d5 is the hologram obtained by angular spectrum reconstruction of the other selected-frequency image of b5 when Δz = 1 mm, and d6 is the hologram obtained by angular spectrum reconstruction of the other selected-frequency image of b6 when Δz = 1.25 mm;

[0064] Figure 8(c) shows the model reconstruction at axial distances equal to 0, 0.25 mm, and 0.5 mm;

[0065] e1 is the hologram obtained by mapping a frequency - selected image of b1 when Δz = 0 through the corresponding image restoration network model, e2 is the hologram obtained by mapping a frequency - selected image of b2 when Δz = 0.25 mm through the corresponding image restoration network model, and e3 is the hologram obtained by mapping a frequency - selected image of b3 when Δz = 0.5 mm through the corresponding image restoration network model; f1 is the hologram obtained by mapping another frequency - selected image of b1 when Δz = 0 through the corresponding image restoration network model, f2 is the hologram obtained by mapping another frequency - selected image of b2 when Δz = 0.25 mm through the corresponding image restoration network model, and f3 is the hologram obtained by mapping another frequency - selected image of b3 when Δz = 0.5 mm through the corresponding image restoration network model;

[0066] Figure 8(d) shows the model reconstruction when the axial distance is equal to 0.75 mm, 1 mm, and 1.25 mm;

[0067] e4 is the hologram obtained by mapping a frequency - selected image of b4 when Δz = 0.75 mm through the corresponding image restoration network model, e5 is the hologram obtained by mapping a frequency - selected image of b5 when Δz = 1 mm through the corresponding image restoration network model, and e6 is the hologram obtained by mapping a frequency - selected image of b6 when Δz = 1.25 mm through the corresponding image restoration network model; f4 is the hologram obtained by mapping another frequency - selected image of b4 when Δz = 0.75 mm through the corresponding image restoration network model, f5 is the hologram obtained by mapping another frequency - selected image of b5 when Δz = 1 mm through the corresponding image restoration network model, and f6 is the hologram obtained by mapping another frequency - selected image of b6 when Δz = 1.25 mm through the corresponding image restoration network model.

[0068] Figure 9(a) shows the peak signal - to - noise ratio of different holograms;

[0069] Figure 9(b) shows the structural similarity of different holograms.

[0070] Figure 10(a) shows the holographic reconstruction display of the USAF1951 resolution target in Example 2;

[0071] Figure 10(b) shows the holographic reconstruction display of Paramecium in Example 2;

[0072] Figure 10(c) shows the holographic reconstruction display of fruit pulp hair in Example 2;

[0073] Figure 10(d) shows the holographic reconstruction display of Ascaris eggs in Example 2;

[0074] Figure 10(e) shows the holographic reconstruction display of yeast in Example 2;

[0075] Figures 10(a) to 10(e)Among them, G represents the double-beam interference holographic image of the sample, H represents the holographic fusion image obtained by pixel superposition after angular spectrum reconstruction of the frequency-selected image after frequency selection processing of G, and I represents the holographic fusion image obtained by pixel superposition after mapping the frequency-selected image after frequency selection processing through the image restoration network model; J is a comparative display of the pixel intensity change curves of the same line on the three images G, H, and I. Specific Embodiment

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0077] Refer to Figure 1 、 Figure 2 A dual-modal digital holographic imaging method based on deep learning proposed in this embodiment includes the following steps:

[0078] S1. Construct an image restoration network model with the reconstructed image as the input and the restored image as the output. The image restoration network model corresponds to the light source one by one;

[0079] The reconstructed image refers to the holographic image generated by collecting the object light after spatial modulation by a charge-coupled device (CCD); or the holographic image after frequency selection processing.

[0080] S2. Perform frequency selection processing on the double-beam interference holographic image, output frequency-selected images corresponding to the two light sources respectively, and input the two frequency-selected images into the corresponding image restoration models after angular spectrum reconstruction.

[0081] S3. Superpose and normalize the restored images output by the two image restoration models to obtain a fused restored image (FSDL-FH).

[0082] Specifically, during the frequency selection processing, the two frequency ranges respectively correspond to the two light sources, and then the frequency-selected image selects the corresponding image restoration network model according to its frequency range.

[0083] Refer to Figure 3 The image restoration network model proposed in this embodiment includes a first convolutional unit, a second convolutional unit, a third convolutional unit, a fourth convolutional unit, a fifth convolutional unit, a sixth convolutional unit, a first deconvolutional unit, a second deconvolutional unit, a third deconvolutional unit, a fourth deconvolutional unit, and a fully connected layer connected in sequence. The input of the first convolutional unit is the input of the image restoration network model, and the output of the fully connected layer is the output of the image restoration network model.

[0084] The output of the first convolutional unit is input into the second convolutional unit after max-pooling processing, the output of the second convolutional unit is input into the third convolutional unit after max-pooling processing, the output of the third convolutional unit is input into the fourth convolutional unit after max-pooling processing, the output of the fourth convolutional unit is input into the fifth convolutional unit after max-pooling processing, and the output of the fifth convolutional unit is input into the sixth convolutional unit after max-pooling processing;

[0085] The output of the fourth convolutional unit is also connected to the input of the first deconvolutional unit, the output of the third convolutional unit is also connected to the input of the second deconvolutional unit, the output of the second convolutional unit is also connected to the input of the third deconvolutional unit, and the output of the first convolutional unit is also connected to the input of the fourth deconvolutional unit.

[0086] The first convolutional unit, the second convolutional unit, the third convolutional unit, the fourth convolutional unit, the fifth convolutional unit, the sixth convolutional unit, the first deconvolutional unit, the second deconvolutional unit, the third deconvolutional unit, and the fourth deconvolutional unit are all composed of sequentially stacked convolutional layers, BatchNorm2d functions, and activation functions. The activation function specifically uses the leaky ReLU function.

[0087] The double-beam interference holographic image is collected through Figure 4 the optical system shown. The optical system includes: a first optical path, a second optical path, and an interference optical path.

[0088] The first optical path includes, arranged along the light propagation direction: an electric light source 11, a collimating lens 12, a lens 13, and a first microscope objective 14; the electric light source 11 can specifically use a xenon lamp.

[0089] The second optical path includes, arranged along the light propagation direction: a laser 21, a beam diffuser 22, a second microscope objective 23, and a phase shift module 24.

[0090] The intersection position of the first optical path and the second optical path is the detection position 60, and the detection position 60 is used to place the detection target; on the first optical path, the detection position 60 is located between the lens 13 and the first microscope objective 14; on the second optical path, the detection position 60 is located between the beam diffuser 22 and the second microscope objective 23;

[0091] The interference optical path includes: a beam splitter 30, a spatial light modulator 40, and an electrical coupling device 50;

[0092] The light emitted from the electric light source 11 sequentially passes through the collimating lens 12, the lens 13, and the first microscope objective 14 and is incident on the incident surface of the beam splitter 30. The light emitted from the laser 21 sequentially passes through the beam diffuser 22, the second microscope objective 23, and the phase shift module 24 and is incident on the incident surface of the beam splitter 30; the beam splitter 30 combines the two incident light beams and transmits them to the spatial light modulator 40, and the light emitted from the spatial light modulator 40 is collected by the electrical coupling device 50 to generate a hologram.

[0093] In specific implementation, reflectors may also be provided on the first optical path and the second optical path to adjust the light propagation direction. The first reflector 15 on the first optical path is arranged at the end of the first optical path, and the second reflector 25 on the second optical path is arranged between the second microscope objective and the phase shift module.

[0094] When both the light source and the laser are started, the hologram collected by the electrical coupling device is the double-beam interference holographic image in the above step S2.

[0095] The training method of the two-channel image restoration network model includes the following steps:

[0096] S11. Place a detection target at the detection position 60; when only the light source is started and the SLM loads a mask (CPM), the image after angular spectrum diffraction of the hologram collected by the CCD is used as the first reconstructed image; when the SLM is replaced with a mirror, the image collected by the CCD is used as the first real image; adjust the light output angle of the light source 11 to generate multiple groups of first samples {first reconstructed image, first real image} to form a first training set;

[0097] When only the laser 21 is started and the SLM loads a mask (CPM), the image after angular spectrum diffraction of the hologram collected by the CCD is used as the second reconstructed image; when the SLM is replaced with a mirror, the image collected by the CCD is used as the second real image; adjust the light output angle of the laser 21 to generate multiple groups of second samples {second reconstructed image, second real image} to form a second training set;

[0098] S12. Train the first image restoration network model on the first training set and train the second image restoration network model on the second training set; during the training process, the expectation of minimizing the sum of the mean square error loss and the spectral norm regularization term is used as the optimization objective to iterate the image restoration network model;

[0099] S13. When the number of iterations of the first image restoration network model and the second image restoration network model reaches an integer multiple of the set threshold, calculate the overall loss function L total ; The set threshold can specifically be an integer greater than 10. In subsequent embodiments, the set threshold can specifically be set to 100 times.

[0100] If the overall loss function L total converges, fix the first image restoration network model and the second image restoration network model; otherwise, return to step S12.

[0101] The overall loss function is: L total =( L '1+ L'2) / ( L '1 / L '2)

[0102] Among them, L '1 represents the loss function of the first image restoration network model, L '2 represents the loss function of the second image restoration network model.

[0103] The loss function of the image restoration network model is the sum of the mean square error loss and the spectral norm regularization term, and is expressed by the formula as follows:

[0104] L '1 = L 1( x 1 o , x 1 t ) + λ 1|| x 1 o || SPN-R

[0105] L '2 = L 2( x 2 o , x 2 t ) + λ 2|| x 2 o || SPN-R

[0106] Among them, L 1( x 1 o , x 1 t ) represents the mean square error of the first image restoration network model, x 1 o represents the restored image output by the first image restoration network model, x 1 t represents the first real image; λ 1 represents the first trade-off parameter, and its value range is (0,1]; || x 1 o || SPN-R represents x 1 o 's spectral norm regularization term (Spectral Norm Regularization);

[0107] L 2( x 2 o , x 2 t ) represents the mean square error of the second image restoration network model,x 2 o represents the restored image output by the second image restoration network model x 2 t represents the second real image; λ 2 represents the second trade-off parameter, and its value range is (0, 1]; || x 2 o || SPN-R represents x 2 o the spectral norm regularization term of;

[0108] ;

[0109] ;

[0110] wherein, x 1 o ( x, y ) represents the pixel value at the coordinate ( x 1 o ) on the restored image x, y ), x 1 t ( x, y ) represents the pixel value at the coordinate ( x 1 t ) on the first real image; x, y ) on the first real image; x 2 o ( x, y ) represents the pixel value at the coordinate ( x 2 o ) on the restored image x, y ), x 2 t ( x, y ) represents the pixel value at the coordinate ( x 2 t ) on the second real image x, y ); C is the image length and H is the image width.

[0111] Specifically, the method for training the first image restoration network model on the first training set includes the following steps:

[0112] S121. Extract the first training samples from the first training set and substitute them into the first image restoration network model, and let the first image restoration network model perform autonomous learning;

[0113] S122. Extract the first verification samples from the first training set and input them into the first image restoration network model, and the first image restoration network model predicts the restored images of the first verification samples;

[0114] S123. Calculate the mean square error between the first restored image and the real image on the first verification sample L 1( x 1 o , x 1 t ), and update the parameters of the first image restoration network model according to the first optimization objective;

[0115] The first optimization objective is the expectation of the sum of the mean square error and the spectral norm regularization term, and the formula is expressed as:

[0116] ;

[0117] That is, the first optimization objective is to minimize the loss function L '1 of the first image restoration network model;

[0118] S124. Determine whether the update times of the first image restoration network model reach an integer multiple of the set threshold; if not, return to step S121; if so, execute step S13.

[0119] One update of the first image restoration network model includes steps S121 - S123.

[0120] Specifically, the method for training the second image restoration network model on the second training set includes the following steps:

[0121] S125. Extract a second training sample from the second training set and substitute it into the second image restoration network model to enable the second image restoration network model to perform autonomous learning;

[0122] S126. Extract a second verification sample from the second training set and input it into the second image restoration network model, and the second image restoration network model predicts the restored image of the second verification sample;

[0123] S127. Calculate the mean square error between the second restored image and the real image on the second verification sample L 2( x 2 o , x 2 t ), and update the parameters of the second image restoration network model according to the second optimization objective;

[0124] The second optimization objective is the expectation of the sum of the mean square error and the spectral norm regularization term, and the formula is expressed as:

[0125] ;

[0126] That is, the second optimization objective is to minimize the loss function L '2 of the second image restoration network model;

[0127] S128. Determine whether the update count of the second image restoration network model reaches an integer multiple of the set threshold. If not, return to step S125. If yes, execute step S13.

[0128] One update of the second image restoration network model includes steps S125 - S127.

[0129] In this embodiment, the trade - off coefficients λ 1 and λ 2 are both set values, and are specifically set to 0.1 in subsequent embodiments.

[0130] The following elaborates on the above - mentioned dual - mode digital holographic imaging method based on deep learning in combination with specific embodiments.

[0131] Embodiment 1

[0132] This embodiment uses the Figure 4 shown optical system, where the electric light source 11 uses a xenon lamp with a central wavelength of 633 nm and a bandwidth of 20 nm, and the laser 21 uses a helium - neon gas laser.

[0133] On the first optical path, the incoherent light emitted by the xenon lamp is collimated by the collimating lens 12, and then focused on the sample surface through the lens 13. The incoherent light carrying the object wave information is imaged by the first microscope objective 14, and then reaches the surface of the spatial light modulator 40 through the beam splitter 30. The spatial light modulator 40 is composed of a polarizer and a half - wave plate (HWP).

[0134] On the second optical path, the beam diffuser 22 irradiates the pump laser onto the sample surface at the detection position 60. Then, the coherent light carrying the sample information is imaged by the second microscope objective 23, and reaches the surface of the spatial light modulator 40 through the phase - shift module 24 and the beam splitter 30.

[0135] The beam splitter 30 combines the incident light into one beam and projects it onto the surface of the spatial light modulator 40, and then projects it onto the charge - coupled device 50.

[0136] In this embodiment, the sample uses a USAF1951 resolution test target. The reconstruction results are shown in FIGS. 5(a), 5(b), 5(c), 5(d), and 5(e). Among them, FIG. 5(a) is the hybrid hologram collected by the charge - coupled device 50 when both the xenon lamp and the laser 21 are turned on, that is, the double - beam interference holographic image.

[0137] Two selected-frequency images obtained after frequency selection processing of the double-beam interference holographic image in Fig. 5(a). The two selected-frequency images are respectively subjected to angular spectrum reconstruction to obtain the reconstructed images in Fig. 5(b) and Fig. 5(c), showing the dual-mode holographic separation of incoherent light and coherent light; the two selected-frequency images are respectively mapped through the corresponding image restoration network model to obtain the reconstructed images in Fig. 5(d) and Fig. 5(e).

[0138] From Figures 5(a) to 5(e) It can be seen that frequency selection processing can strip and reconstruct the wavefront in the corresponding channels, eliminating the interference of conjugate terms and zero-level terms.

[0139] In this embodiment, the same column of pixels shown by the yellow line is selected on Fig. 5(b) and Fig. 5(d) for intensity comparison, and the result is as Figure 6 shown; the same column of pixels shown by the yellow line is selected on Fig. 5(c) and Fig. 5(e) for intensity comparison, and the result is as Figure 7 shown. By comparison, the resolution of Fig. 5(b) and Fig. 5(c) is 128.0 lp / mm; the resolution of Fig. 5(d) and Fig. 5(e) is 203.0 lp / mm. It can be seen that the resolution of the holographic reconstructed image after deep learning processing has increased by 58.59%, achieving super-resolution reconstruction.

[0140] In the optical system, the focal point of the first microscope objective lens 14 coincides with the focal point of the second microscope objective lens 23, and the distance between the sample and the focal point is denoted as the axial distance Δz. In this embodiment, the axial distance Δz is moved in steps of 0.25 mm until the axial distance Δz reaches 1.25 mm, and the holograms at each position are recorded in sequence, as shown in Fig. 8(a), Fig. 8(b), Fig. 8(c), and Fig. 8(d) specifically, where a1-a6 are conventional images at different axial distances Δz, that is, in the optical system, after replacing the spatial light modulator (SLM) with a mirror, the holograms collected by the charge-coupled device 50; b1-b6 are double-beam interference holographic images at different axial distances Δz;

[0141] The double-beam interference holographic images at different axial distances Δz in b1-b6 are subjected to frequency selection processing to obtain two selected-frequency images. The reconstructed images obtained by respectively subjecting the two corresponding selected-frequency images at different axial distances Δz in b1-b6 to angular spectrum reconstruction are shown as c1-c6 and d1-d6; the reconstructed images obtained by respectively mapping the two corresponding selected-frequency images at different axial distances Δz in b1-b6 through the corresponding image restoration network model are shown as e1-e6 and f1-f6.

[0142] In this embodiment, the images of each column with the same numbers but different letters in FIGS. 8(a), 8(b), 8(c), and 8(d) are compared to obtain the peak signal-to-noise ratio and structural similarity of the reconstructed images obtained by two methods on two optical paths of the optical system at different axial distances Δz. Specifically, as shown in FIGS. 9(a) and 9(b), where FSUnet(Chanel 1) represents the reconstructed image after the frequency-selected image corresponding to the first optical path is mapped through the image restoration network model, FSUnet(Chanel2) represents the reconstructed image after the frequency-selected image corresponding to the second optical path is mapped through the image restoration network model, FS(Chanel 1) represents the reconstructed image after the frequency-selected image corresponding to the first optical path is reconstructed by the angular spectrum, and FS(Chanel 2) represents the reconstructed image after the frequency-selected image corresponding to the second optical path is reconstructed by the angular spectrum. It can be seen that both the peak signal-to-noise ratio and the structural similarity decrease as the axial distance Δz decreases, and the descending gradient of the reconstruction method through the image restoration network model is more significant, proving that the reconstruction through the image restoration network model has a higher axial resolution, and proving that the proposed dual-modal digital holographic imaging method based on deep learning has the potential to output the target slice.

[0143] Example 2

[0144] In this embodiment, the optical system of Example 1 is adopted, and the USAF1951 resolution target, paramecium, fruit pulp hair, Ascaris egg, and yeast are used as samples respectively.

[0145] Referring to FIG. 10(a), for the USAF1951 resolution target, the Figure 4 optical system is used to obtain the double-beam interference holographic image G, and then the double-beam interference holographic image is subjected to frequency selection processing; after the two-way frequency-selected images are reconstructed by the angular spectrum, pixel superposition is performed to obtain the holographic fusion image H after frequency-selected reconstruction; after the two-way frequency-selected images are mapped through the image restoration network model, pixel superposition is performed to obtain the holographic fusion image I after frequency-selected reconstruction; the intensity change trends of the double-beam interference holographic image and the two holographic fusion images on the same intensity cross-section are shown in FIG. J.

[0146] In this embodiment, the double-beam interference holographic images G, the holographic fusion images H reconstructed by the angular spectrum, the holographic fusion images I after model mapping, and the intensity change trend comparison diagrams J corresponding to the paramecium, fruit pulp hair, Ascaris egg, and yeast are shown in FIGS. 10(b), 10(c), 10(d), and 10(e) respectively.

[0147] By comparing the two holographic fusion images of the same sample, it can be seen that the holographic fusion image I obtained by the proposed dual-modal digital holographic imaging method based on deep learning is clearer and has less background noise.

[0148] From the comparison of the intensity change trends of the double-beam interference holographic images and two holographic fusion images of the same sample on the same intensity cross-section, it can be seen that for the holographic fusion image I of each sample, compared with H, the curve characteristics are more matched with the cross-section at the corresponding position of the target.

[0149] Of course, for those skilled in the art, the present invention is not limited to the details of the above exemplary embodiments, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be construed as limiting the claimed rights.

[0150] In addition, it should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0151] The technologies, shapes, and structures not detailed in the present invention are all well-known technologies.

Claims

1. A dual-modal digital holographic imaging method based on deep learning, characterized in that: The following steps are involved: S1. Construct an image restoration network model whose input is the reconstructed image and whose output is the restored image. The image restoration network model corresponds to the light source one by one. S2, performing frequency selection processing on the dual-beam interference holographic image, outputting frequency-selected images with frequencies corresponding to the two light sources, and inputting the two frequency-selected images into the corresponding image restoration model; S3, superimposing and normalizing the restored images output by the two image restoration models to obtain a fused restored image; In step S2, a double-beam interference holographic image is collected by an optical system; the optical system includes: a first optical path, a second optical path and an interference optical path; The first optical path includes: an electric light source, a collimating lens, a lens and a first microscope objective lens arranged along the light propagation direction; The second optical path includes: a laser, a beam diffuser, a second microscope objective lens and a phase shift module arranged along the light propagation direction; The intersection of the first optical path and the second optical path is a detection position, and the detection position is used to place the detection target; on the first optical path, the detection position is located between the lens and the first microscope objective lens; on the second optical path, the detection position is located between the beam diffuser and the second microscope objective lens; The interference optical path includes: a beam splitter, a spatial light modulator and an electric coupling device; The outgoing light of the first microscope objective lens and the outgoing light of the phase shift module are combined by a beam splitter and then transmitted to the spatial light modulator, and the outgoing light of the spatial light modulator is collected by an electrical coupling device to generate a hologram; When both the electric light source and the laser are started, the hologram generated by the electric coupling device is the double-beam interference holographic image of S2; The training method of the two-way image restoration network model includes the following steps: S11, placing a test target at a test position, and obtaining training sets corresponding to two light sources; the training set of one light source is obtained by: starting the light source and turning off the other light source, and when the SLM is loaded with a mask, the image of the hologram collected by the CCD after angular spectrum diffraction is used as the reconstructed image; when the SLM is replaced with a reflector, the image collected by the CCD is used as the real image; adjusting the light output angle of the electric light source, generating multiple groups of samples {reconstructed image, real image} to constitute the training set of the light source; S12, respectively training two image restoration network models on the corresponding training sets; S13. When the number of iterations of the two image restoration network models reaches an integer multiple of the set threshold, the overall loss function is calculated. L total ; If the overall loss function L total If convergence occurs, the two-way image restoration network model is fixed; otherwise, the process returns to step S12; The overall loss function is: L total =( L '1+ L '2) / ( L '1 / L '2)； in, L '1 represents the loss function of the image restoration network model corresponding to the electric light source, L '2 represents the loss function of the image restoration network model corresponding to the laser; or, L '2 represents the loss function of the image restoration network model corresponding to the electric light source, L '1 represents the loss function of the image restoration network model corresponding to the laser; The loss function of the image restoration network model is the sum of the mean square error loss and the spectral norm regularization term. During the training process of the image restoration network model, the optimization goal is to minimize the expectation of the loss function to iterate the image restoration network model.

2. The dual-modal digital holographic imaging method based on deep learning according to claim 1, characterized in that: The image restoration network model includes a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit, a fifth convolution unit, a sixth convolution unit, a first deconvolution unit, a second deconvolution unit, a third deconvolution unit, a fourth deconvolution unit and a fully connected layer which are sequentially connected; in the first to sixth convolution units, the output of the previous convolution unit is input into the next convolution unit after maximum pooling processing; the output of the Nth convolution unit is also connected to the input of the 5th to Nth deconvolution units, 1≤N<5.

3. The dual-modal digital holographic imaging method based on deep learning according to claim 2, characterized in that: The first convolution unit, the second convolution unit, the third convolution unit, the fourth convolution unit, the fifth convolution unit, the sixth convolution unit, the first deconvolution unit, the second deconvolution unit, the third deconvolution unit and the fourth deconvolution unit are all composed of sequentially stacked convolution layers, BatchNorm2d functions and activation functions.

4. The dual-modal digital holographic imaging method based on deep learning according to claim 1, characterized in that: The method for training the image restoration network model on the training set in S12 includes the following steps: Extract training samples from the training set and substitute them into the image restoration network model, so that the image restoration network model can learn autonomously; A verification sample is extracted from the training set and input into the image restoration network model, and the image restoration network model predicts the restored image of the verification sample; The mean square error between the restored image and the real image is calculated on the verification sample, and the image restoration network model parameters are updated according to the optimization target; the optimization target is the expectation of the sum of the mean square error and the spectral norm regularization term; Repeat the above steps until the number of updates of the image restoration network model reaches an integer multiple of the set threshold.

5. The dual-modal digital holographic imaging method based on deep learning according to claim 1, characterized in that: A reflector is provided at the end of the first optical path; and a reflector is provided between the second microscope objective lens and the phase shift module on the second optical path.

6. A system using the dual-modal digital holographic imaging method based on deep learning as described in any one of claims 1 to 5, characterized in that: It includes an optical system, a frequency selection module, a first image restoration network model, a second image restoration network model and an image superposition module; The optical system is used to generate a double-beam interference holographic image. The double-beam interference holographic image is frequency-selected by a frequency selection module to be two-channel frequency-selected images. The two-channel frequency-selected images are processed by a first image restoration network model and a second image restoration network model respectively. The output image of the first image restoration network model and the output image of the second image restoration network model are output after pixel superposition by an image superposition module.

7. A memory, characterized in that: A computer program, a first image restoration network model and a second image restoration network model are stored, and when the computer program is executed, it is used to implement the deep learning-based dual-modal digital holographic imaging method as described in any one of claims 1-5.

Images