Bimodal digital holographic imaging method and system based on deep learning and memory

By employing a deep learning-based dual-modal digital holographic imaging method, utilizing an image reconstruction network model and frequency selection processing, the problem of separating and reconstructing mixed coherent and incoherent holograms is solved, achieving high-resolution and high-fidelity holographic reconstruction. This method is suitable for multi-dimensional information compression and holographic detection in complex environments.

CN118981152AActive Publication Date: 2024-11-19ZHEJIANG DINGLI IND

Patent Information

Application Number
CN202411471803.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-22
Publication Date
2024-11-19
Estimated Expiration
2044-10-22

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively separate and reconstruct hybrid coherent and incoherent holograms, resulting in low holographic reconstruction quality. In particular, it is difficult to achieve high-fidelity and high-resolution multimodal holographic reconstruction in complex environments.

Method used

A deep learning-based dual-modal digital holographic imaging method is adopted. By constructing an image reconstruction network model and combining frequency selection processing and convolutional neural network, the dual-beam interference holographic image is separated and reconstructed. The network parameters are optimized using a hybrid loss function to achieve high-resolution and high-fidelity holographic reconstruction.

Benefits of technology

It achieves high-resolution and high-fidelity holographic reconstruction, eliminates interlayer crosstalk and pseudo-noise, and is suitable for multi-dimensional information compression, industrial transient recording and super-resolution imaging, adapting to holographic detection in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981152A_ABST
    Figure CN118981152A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of holographic imaging, in particular to a bimodal digital holographic imaging method and system based on deep learning and a memory. The invention provides a deep learning-based bimodal digital holographic imaging method, which comprises the following steps of: constructing image restoration network models of which the input is a reconstructed image and the output is a restored image, and enabling the image restoration network models to be in one-to-one correspondence with light sources; performing frequency selection processing on the double-beam interference holographic image, outputting frequency selection images with frequencies respectively corresponding to the two paths of light sources, performing angular spectrum reconstruction on the two paths of frequency selection images, and inputting the two paths of frequency selection images into corresponding image restoration models; and carrying out pixel superposition and normalization on the restored images output by the two image restoration models to obtain a fused restored image. According to the method, through combination of frequency selection processing and deep learning, interlayer crosstalk and pseudo noise are effectively eliminated, high-fidelity imaging performance is visually displayed, and higher image resolution is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of holographic imaging technology, and in particular to a dual-modal digital holographic imaging method, system and memory based on deep learning. Background Art

[0002] Multimodal 3D imaging has become one of the research hotspots in the scientific and industrial communities. Digital holography provides technical support for 3D reconstruction by simultaneously recording multidimensional information (including the amplitude and phase of the object) on a 2D recording medium. Digital holography can be divided into incoherent digital holography and coherent digital holography.

[0003] Incoherent digital holography generates self-interference holograms through phase encoding and uses phase-shift interferometry (PSI) to decode and reconstruct target waves of different wavelengths and polarization directions. Incoherent holography has demonstrated that the interference imaging principle violates the Lagrangian invariant and shows the potential for super-resolution imaging.

[0004] Compared with the off-axis configuration, the self-interference method combined with PSI can produce incoherent holograms with a considerable spatial bandwidth product (SBWP) and enable holographic perception of 3D fields with frequently changing point spread function (psf).

[0005] Coherent holography usually uses an off-axis method to separate the target wavefront from the crosstalk terms (DC and conjugate terms). Due to the excellent coherence of lasers, long-distance imaging does not lead to extensive diffusion of the wavefront. In addition, phase-shift interferometry is also applicable to the reconstruction of incoherent holography, but it will introduce scattering noise.

[0006] When incoherent and coherent wavefronts are mixed and recorded holographically, it is difficult for conventional methods to effectively separate and reconstruct the superimposed hologram of these two modes. Summary of the invention

[0007] In order to overcome the dual-modal holographic reconstruction problem in the above-mentioned prior art, the present invention proposes a dual-modal digital holographic imaging method based on deep learning, which realizes dual-field-of-view, high-fidelity, and high-resolution holographic reconstruction.

[0008] The present invention proposes a dual-modal digital holographic imaging method based on deep learning, comprising the following steps: S1. Construct an image restoration network model whose input is the reconstructed image and whose output is the restored image. The image restoration network model corresponds to the light source one by one. S2, performing frequency selection processing on the dual-beam interference holographic image, outputting frequency-selected images with frequencies corresponding to the two light sources, and inputting the two frequency-selected images into the corresponding image restoration model; S3. Superimpose and normalize the restored images output by the two image restoration models to obtain a fused restored image.

[0009] Preferably, the image restoration network model includes a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit, a fifth convolution unit, a sixth convolution unit, a first deconvolution unit, a second deconvolution unit, a third deconvolution unit, a fourth deconvolution unit and a fully connected layer which are sequentially connected; in the first to sixth convolution units, the output of the previous convolution unit is input into the next convolution unit after maximum pooling processing; the output of the Nth convolution unit is also connected to the input of the 5th to Nth deconvolution units, 1≤N<5.

[0010] Preferably, the first convolution unit, the second convolution unit, the third convolution unit, the fourth convolution unit, the fifth convolution unit, the sixth convolution unit, the first deconvolution unit, the second deconvolution unit, the third deconvolution unit and the fourth deconvolution unit are all composed of sequentially stacked convolution layers, BatchNorm2d functions and activation functions.

[0011] Preferably, in step S2, a double-beam interference holographic image is collected by an optical system; The optical system comprises: a first optical path, a second optical path and an interference optical path; The first optical path includes: an electric light source, a collimating lens, a lens and a first microscope objective lens arranged along the light propagation direction; The second optical path includes: a laser, a beam diffuser, a second microscope objective lens and a phase shift module arranged along the light propagation direction; The intersection of the first optical path and the second optical path is a detection position, and the detection position is used to place the detection target; on the first optical path, the detection position is located between the lens and the first microscope objective lens; on the second optical path, the detection position is located between the beam diffuser and the second microscope objective lens; The interference optical path includes: a beam splitter, a spatial light modulator and an electric coupling device; The outgoing light of the first microscope objective lens and the outgoing light of the phase shift module are combined by a beam splitter and then transmitted to the spatial light modulator, and the outgoing light of the spatial light modulator is collected by an electrical coupling device to generate a hologram; When both the electric light source and the laser are started, the hologram generated by the electric coupling device is the double-beam interference holographic image of S2.

[0012] Preferably, the training method of the two-path image restoration network model comprises the following steps: S11, placing a test target at a test position, and obtaining training sets corresponding to two light sources; the training set of one light source is obtained by: starting the light source and turning off the other light source, and when the SLM is loaded with a mask, the image of the hologram collected by the CCD after angular spectrum diffraction is used as the reconstructed image; when the SLM is replaced with a reflector, the image collected by the CCD is used as the real image; adjusting the light output angle of the electric light source, generating multiple groups of samples {reconstructed image, real image} to constitute the training set of the light source; S12, respectively training two image restoration network models on the corresponding training sets; S13. When the number of iterations of the two image restoration network models reaches an integer multiple of the set threshold, the overall loss function is calculated. L total ; If the overall loss function L total If convergence occurs, the two-way image restoration network model is fixed; otherwise, the process returns to step S12.

[0013] Preferably, the overall loss function is: L total =( L '1+ L '2) / ( L '1 / L '2) in, L '1 represents the loss function of the image restoration network model corresponding to the electric light source, L '2 represents the loss function of the image restoration network model corresponding to the laser; or, L '2 represents the loss function of the image restoration network model corresponding to the electric light source, L '1 represents the loss function of the image restoration network model corresponding to the laser; The loss function of the image restoration network model is the sum of the mean square error loss and the spectral norm regularization term. During the training process of the image restoration network model, the optimization goal is to minimize the expectation of the loss function to iterate the image restoration network model.

[0014] Preferably, the method of training the image restoration network model on the training set in S12 comprises the following steps: Extract training samples from the training set and substitute them into the image restoration network model, so that the image restoration network model can learn autonomously; A verification sample is extracted from the training set and input into the image restoration network model, and the image restoration network model predicts the restored image of the verification sample; The mean square error between the restored image and the real image is calculated on the verification sample, and the image restoration network model parameters are updated according to the optimization target; the optimization target is the expectation of the sum of the mean square error and the spectral norm regularization term; Repeat the above steps until the number of updates of the image restoration network model reaches an integer multiple of the set threshold.

[0015] Preferably, a reflective element is provided at the end of the first optical path; and a reflective element is provided between the second microscope objective lens and the phase shift module on the second optical path.

[0016] The present invention proposes a system using the dual-modal digital holographic imaging method based on deep learning, comprising an optical system, a frequency selection module, a first image restoration network model, a second image restoration network model and an image superposition module; The optical system is used to generate a double-beam interference holographic image. The double-beam interference holographic image is frequency-selected by a frequency selection module to be two-channel frequency-selected images. The two-channel frequency-selected images are processed by a first image restoration network model and a second image restoration network model respectively. The output image of the first image restoration network model and the output image of the second image restoration network model are output after pixel superposition by an image superposition module.

[0017] The present invention proposes a memory storing a computer program, a first image restoration network model and a second image restoration network model. When the computer program is executed, it is used to implement the dual-modal digital holographic imaging method based on deep learning.

[0018] The advantages of the present invention are: (1) The present invention proposes a dual-modal digital holographic imaging method based on deep learning. First, coherent light and incoherent light are mixed for digital holographic recording. Then, the target wave spectrum is separated by frequency selection processing. Then, a neural network is used to convert the low-resolution reconstructed image into a high-resolution reconstructed image, thereby realizing the fusion reconstruction of multi-modal holography. The present invention effectively eliminates inter-layer crosstalk and pseudo-noise by combining frequency selection processing and deep learning, and intuitively demonstrates high-fidelity imaging performance and achieves higher image resolution.

[0019] (2) The present invention adopts phase aperture coded imaging to achieve parallel reconstruction after overlapping of multiple target waves, and obtains dual FOV (field of view) from the original hologram. Compared with traditional holography, it has a larger bandwidth and achieves large FOV (field of view) and super-resolution reconstruction. The present invention is suitable for multi-dimensional information compression, industrial transient recording, super-resolution imaging and other fields.

[0020] (3) The present invention is more friendly to incoherent holography, with less background noise. The dual-modal holographic fusion technology can strip and reconstruct incoherent holographic information with high fidelity quality in complex environments. This is of great value for holographic detection with multiple light sources and interference in harsh environments.

[0021] (4) The present invention simplifies the architectural complexity of the optical system through frequency selection processing, reduces the difficulty of dual-modal holographic fitting, and realizes end-to-end feature extraction and direct mapping.

[0022] (5) The present invention adopts a hybrid loss function in the model training process, which includes both the mean square error (RMSE) and the spectral norm (SPN-R). Compared with a single loss function that only includes RMSE, the hybrid loss function not only makes the training image closer to the real image in terms of pixels, but also accelerates the network convergence and reduces overfitting.

[0023] (6) The present invention includes a frequency selection module for stripping aliased spectra, uses two different configurations of convolutional neural networks (CNNs) for training, uses the mean square error (RMSE) and the spectral norm (SPN-R) as loss functions, and optimizes the bias through back propagation. The optimal weights are obtained under limited training conditions to achieve dual-field, high-fidelity, and high-resolution holographic reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of a dual-modal digital holographic imaging method based on deep learning.

[0025] Figure 2 A topological diagram of a dual-modal digital holographic imaging method based on deep learning.

[0026] Figure 3 Restore the network model structure diagram for the image.

[0027] Figure 4 is a schematic diagram of the optical system; 11. Electric light source; 12. Collimating lens; 13. Lens; 14. First microscope objective lens; 15. First reflector; 21. Laser; 22. Beam diffuser (BE); 23. Second microscope objective lens; 24. Phase shift module (PSM); 25. Second reflector; 30. Beam splitter (BS); 40. Spatial light modulator (SLM); 50. Charge coupled device (CCD); 60. Detection position.

[0028] FIG5( a ) is a double-beam interference holographic image of the sample in Example 1; FIG5(b) is a reconstructed image obtained by angular spectrum reconstruction of a frequency-selective image obtained after frequency-selective processing in FIG5(a); FIG5(c) is a reconstructed image obtained by angular spectrum reconstruction of another frequency-selective image obtained after frequency-selective processing in FIG5(a); FIG5(d) is a reconstructed image obtained by mapping a frequency-selective image obtained after frequency-selective processing in FIG5(a) through a corresponding image restoration network model; FIG5(e) is another frequency-selective image obtained after frequency-selective processing in FIG5(a), and a reconstructed image is obtained by mapping the corresponding image restoration network model.

[0029] Figure 6This is a comparison of the intensity distribution of the first frequency-selective image in Example 1 reconstructed using two methods.

[0030] Figure 7 This is a comparison of the intensity distribution of the second frequency-selective image in Example 1 reconstructed in two ways.

[0031] Figure 8(a) shows the angular spectrum reconstruction when the axial distance is equal to 0, 0.25 mm, and 0.5 mm; a1 is the conventional imaging of the resolution plate when Δz=0, a2 is the conventional imaging of the resolution plate when Δz=0.25mm, and a3 is the conventional imaging of the resolution plate when Δz=0.5mm; b1 is the double-beam interference holographic image of the resolution plate when Δz=0, b2 is the double-beam interference holographic image of the resolution plate when Δz=0.25mm, and b3 is the double-beam interference holographic image of the resolution plate when Δz=0.5mm; c1 is the hologram obtained by angular spectrum reconstruction of one-channel frequency-selective images of b1 when Δz=0, c 2 is a hologram obtained by angular spectrum reconstruction of one frequency-selective image of b2 when Δz=0.25mm, c3 is a hologram obtained by angular spectrum reconstruction of one frequency-selective image of b3 when Δz=0.5mm; d1 is a hologram obtained by angular spectrum reconstruction of another frequency-selective image of b1 when Δz=0, d2 is a hologram obtained by angular spectrum reconstruction of another frequency-selective image of b2 when Δz=0.25mm, d3 is a hologram obtained by angular spectrum reconstruction of another frequency-selective image of b3 when Δz=0.5mm; Figure 8(b) shows the angular spectrum reconstruction when the axial distance is equal to 0.75 mm, 1 mm, and 1.25 mm; a4 is the conventional imaging of the resolution plate when Δz=0.75mm, a5 is the conventional imaging of the resolution plate when Δz=1mm, and a6 is the conventional imaging of the resolution plate when Δz=1.25mm; b4 is the double-beam interference holographic image of the resolution plate when Δz=0.75mm, b5 is the double-beam interference holographic image of the resolution plate when Δz=1mm, and b6 is the double-beam interference holographic image of the resolution plate when Δz=1.25mm; c4 is a frequency-selective image of b4 when Δz=0.75mm obtained by angular spectrum reconstruction. , c5 is the hologram obtained by angular spectrum reconstruction of one channel of frequency-selective images of b5 when Δz=1mm, c6 is the hologram obtained by angular spectrum reconstruction of one channel of frequency-selective images of b6 when Δz=1.25mm; d4 is the hologram obtained by angular spectrum reconstruction of another channel of frequency-selective images of b4 when Δz=0.75mm, d5 is the hologram obtained by angular spectrum reconstruction of another channel of frequency-selective images of b5 when Δz=1mm, d6 is the hologram obtained by angular spectrum reconstruction of another channel of frequency-selective images of b6 when Δz=1.25mm; Figure 8(c) shows the model reconstruction when the axial distance is equal to 0, 0.25 mm, and 0.5 mm; e1 is the hologram obtained by mapping one channel of frequency-selective images of b1 through the corresponding image restoration network model when Δz=0, e2 is the hologram obtained by mapping one channel of frequency-selective images of b2 through the corresponding image restoration network model when Δz=0.25mm, e3 is the hologram obtained by mapping one channel of frequency-selective images of b3 through the corresponding image restoration network model when Δz=0.5mm; f1 is the hologram obtained by mapping another channel of frequency-selective images of b1 through the corresponding image restoration network model when Δz=0, f2 is the hologram obtained by mapping another channel of frequency-selective images of b2 through the corresponding image restoration network model when Δz=0.25mm, f3 is the hologram obtained by mapping another channel of frequency-selective images of b3 through the corresponding image restoration network model when Δz=0.5mm; Figure 8(d) shows the model reconstruction when the axial distance is equal to 0.75 mm, 1 mm, and 1.25 mm; e4 is the hologram obtained by mapping one frequency-selective image of b4 through the corresponding image restoration network model when Δz=0.75mm, e5 is the hologram obtained by mapping one frequency-selective image of b5 through the corresponding image restoration network model when Δz=1mm, e6 is the hologram obtained by mapping one frequency-selective image of b6 through the corresponding image restoration network model when Δz=1.25mm; f4 is the hologram obtained by mapping another frequency-selective image of b4 through the corresponding image restoration network model when Δz=0.75mm, f5 is the hologram obtained by mapping another frequency-selective image of b5 through the corresponding image restoration network model when Δz=1mm, f6 is the hologram obtained by mapping another frequency-selective image of b6 through the corresponding image restoration network model when Δz=1.25mm.

[0032] Figure 9(a) shows the peak signal-to-noise ratio of different holograms; Figure 9(b) shows the structural similarity of different holograms.

[0033] FIG10( a ) is a holographic reconstruction display of the USAF1951 resolution plate in Example 2; FIG10( b ) is a holographic reconstruction of the Paramecium in Example 2; FIG10( c ) is a holographic reconstruction of the pulp hairs in Example 2; FIG10( d ) is a holographic reconstruction of the Ascaris egg in Example 2; FIG10( e ) is a holographic reconstruction of the yeast in Example 2; Figure 10(a) to Figure 10(e)In the figure, G represents the dual-beam interference holographic image of the sample, H represents the holographic fusion image obtained by pixel superposition after angular spectrum reconstruction of the frequency-selective image after frequency-selective processing by G, and I represents the holographic fusion image obtained by pixel superposition after mapping the frequency-selective image after frequency-selective processing by the image restoration network model; J is a comparative display of the pixel intensity change curves of the same line in the three figures G, H, and I. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Reference Figure 1 , Figure 2 , a dual-modal digital holographic imaging method based on deep learning proposed in this embodiment includes the following steps: S1. Construct an image restoration network model whose input is the reconstructed image and whose output is the restored image. The image restoration network model corresponds to the light source one by one. The reconstructed image refers to the holographic image generated by collecting the object light with a charge coupled device (CCD) after spatial modulation; or the holographic image after frequency selection processing.

[0036] S2. Perform frequency selection processing on the dual-beam interference holographic image, and output frequency-selected images with frequencies corresponding to the two light sources. The two frequency-selected images are input into the corresponding image restoration model after angular spectrum reconstruction.

[0037] S3. Superimpose and normalize the restored images output by the two image restoration models to obtain a fused restored image (FSDL-FH).

[0038] Specifically, in the frequency selection process, the two frequency intervals correspond to the two light sources respectively, and then the frequency selected image selects the corresponding image restoration network model according to its frequency interval.

[0039] Reference Figure 3 The image restoration network model proposed in this embodiment includes a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit, a fifth convolution unit, a sixth convolution unit, a first deconvolution unit, a second deconvolution unit, a third deconvolution unit, a fourth deconvolution unit and a fully connected layer connected in sequence. The input of the first convolution unit is the input of the image restoration network model, and the output of the fully connected layer is the output of the image restoration network model.

[0040] The output of the first convolution unit is input into the second convolution unit after the maximum pooling process, the output of the second convolution unit is input into the third convolution unit after the maximum pooling process, the output of the third convolution unit is input into the fourth convolution unit after the maximum pooling process, the output of the fourth convolution unit is input into the fifth convolution unit after the maximum pooling process, and the output of the fifth convolution unit is input into the sixth convolution unit after the maximum pooling process; The output of the fourth convolution unit is also connected to the input of the first deconvolution unit, the output of the third convolution unit is also connected to the input of the second deconvolution unit, the output of the second convolution unit is also connected to the input of the third deconvolution unit, and the output of the first convolution unit is also connected to the input of the fourth deconvolution unit.

[0041] The first convolution unit, the second convolution unit, the third convolution unit, the fourth convolution unit, the fifth convolution unit, the sixth convolution unit, the first deconvolution unit, the second deconvolution unit, the third deconvolution unit and the fourth deconvolution unit are all composed of sequentially stacked convolution layers, BatchNorm2d functions and activation functions, and the activation function specifically adopts the leaky ReLU function.

[0042] Double beam interference holographic image Figure 4 The optical system shown in the figure is collected, and the optical system includes: a first optical path, a second optical path and an interference optical path.

[0043] The first optical path includes: an electric light source 11, a collimating lens 12, a lens 13 and a first microscope objective lens 14 arranged along the light propagation direction; the electric light source 11 can specifically be a xenon lamp.

[0044] The second optical path includes: a laser 21, a beam diffuser 22, a second microscope objective lens 23 and a phase shift module 24 arranged along the light propagation direction.

[0045] The intersection of the first optical path and the second optical path is a detection position 60, and the detection position 60 is used to place the detection target; on the first optical path, the detection position 60 is located between the lens 13 and the first microscope objective lens 14; on the second optical path, the detection position 60 is located between the beam diffuser 22 and the second microscope objective lens 23; The interference optical path includes: a beam splitter 30, a spatial light modulator 40 and an electrical coupling device 50; The outgoing light of the electric light source 11 passes through the collimating lens 12, the lens 13 and the first microscope objective lens 14 in sequence and is incident on the incident surface of the beam splitter 30. The outgoing light of the laser 21 passes through the beam diffuser 22, the second microscope objective lens 23 and the phase shift module 24 in sequence and is incident on the incident surface of the beam splitter 30. The beam splitter 30 combines the two incident light beams and transmits them to the spatial light modulator 40. The outgoing light of the spatial light modulator 40 is collected by the electric coupling device 50 to generate a hologram.

[0046] In specific implementation, reflectors may be provided on the first optical path and the second optical path to adjust the light propagation direction. The first reflector 15 on the first optical path is provided at the end of the first optical path, and the second reflector 25 on the second optical path is provided between the second microscope objective lens and the phase shift module.

[0047] When both the electric light source and the laser are started, the hologram generated by the electric coupling device is the double-beam interference holographic image of step S2 above.

[0048] The training method of the two-way image restoration network model includes the following steps: S11, placing a detection target at the detection position 60; starting only the electric light source, and when the SLM is loaded with a mask (CPM), the image of the hologram collected by the CCD after angular spectrum diffraction is used as the first reconstructed image; when the SLM is replaced with a reflector, the image collected by the CCD is used as the first real image; adjusting the light output angle of the electric light source 11, generating multiple groups of first samples {first reconstructed image, first real image} to form a first training set; Only the laser 21 is started, and when the SLM is loaded with a mask (CPM), the image of the hologram collected by the CCD after angular spectrum diffraction is used as the second reconstructed image; when the SLM is replaced with a reflector, the image collected by the CCD is used as the second real image; the light output angle of the laser 21 is adjusted to generate multiple groups of second samples {second reconstructed image, second real image} to form a second training set; S12, training a first image restoration network model on a first training set, and training a second image restoration network model on a second training set; during the training process, minimizing the expectation of the sum of the mean square error loss and the spectral norm regularization term is used as an optimization goal to iterate the image restoration network model; S13, when the number of iterations of the first image restoration network model and the second image restoration network model reaches an integer multiple of the set threshold, the overall loss function is calculated L total ; The threshold value may be set to an integer greater than 10. In subsequent embodiments, the threshold value may be set to 100 times.

[0049] If the overall loss function L total If convergence occurs, the first image restoration network model and the second image restoration network model are fixed; otherwise, the process returns to step S12.

[0050] The overall loss function is: L total =( L '1+ L '2) / ( L '1 / L '2) in, L'1 represents the loss function of the first image restoration network model, L '2 represents the loss function of the second image restoration network model.

[0051] The loss function of the image restoration network model is the sum of the mean square error loss and the spectral norm regularization term, and the formula is as follows: L '1= L 1( x 1 o , x 1 t )+ λ 1|| x 1 o || SPN-R L '2= L 2( x 2 o , x 2 t )+ λ 2|| x 2 o || SPN-R in, L 1( x 1 o , x 1 t ) represents the mean square error of the first image restoration network model, x 1 o represents the restored image output by the first image restoration network model, x 1 t represents a first real image; λ 1 represents the first trade-off parameter, and its value range is (0,1]; || x 1 o || SPN-R express x 1 o Spectral Norm Regularization; L 2( x 2 o , x 2 t ) represents the mean square error of the second image restoration network model, x 2 o represents the restored image output by the second image restoration network model, x 2 t represents a second real image; λ 2 represents the second trade-off parameter, whose value range is (0,1]; || x2 o || SPN-R express x 2 o Spectral Norm Regularization; ; ; in, x 1 o ( x,y ) indicates restoring the image x 1 o Upper coordinate( x,y ), x 1 t ( x,y ) represents the first real image x 1 t Upper coordinate( x,y ) at the pixel value; x 2 o ( x,y ) indicates restoring the image x 2 o Upper coordinate( x,y ), x 2 t ( x,y ) represents the second real image x 2 t Upper coordinate( x,y ) at the pixel value; C is the image length and H is the image width.

[0052] Specifically, the method for training a first image restoration network model on a first training set includes the following steps: S121, extracting a first training sample from the first training set and substituting it into the first image restoration network model, so that the first image restoration network model performs autonomous learning; S122, extracting a first verification sample from the first training set and inputting it into the first image restoration network model, and the first image restoration network model predicting a restored image of the first verification sample; S123, calculating the mean square error between the first restored image and the real image on the first verification sample L 1( x 1 o , x 1 t ), and updating the first image restoration network model parameters according to the first optimization objective; The first optimization objective is the expectation of the sum of the mean square error and the spectral norm regularization term, which can be expressed as: ; That is, the first optimization goal is to minimize the loss function of the first image restoration network model L '1's expectations; S124, determining whether the number of updates of the first image restoration network model reaches an integer multiple of a set threshold; if not, returning to step S121; if yes, executing step S13.

[0053] An update of the first image restoration network model includes steps S121-S123.

[0054] Specifically, the method for training the second image restoration network model on the second training set includes the following steps: S125, extracting a second training sample from the second training set and substituting it into the second image restoration network model, so that the second image restoration network model performs autonomous learning; S126, extracting a second verification sample from the second training set and inputting it into the second image restoration network model, and the second image restoration network model predicting a restored image of the second verification sample; S127, calculating the mean square error between the second restored image and the true image on the second verification sample L 2( x 2 o , x 2 t ), and updating the second image restoration network model parameters according to the second optimization objective; The second optimization objective is the expectation of the sum of the mean square error and the spectral norm regularization term, which is expressed as: ; That is, the second optimization goal is to minimize the loss function of the second image restoration network model L '2 expectations; S128, determine whether the update times of the second image restoration network model reaches an integer multiple of the set threshold; if not, return to step S125; if yes, execute step S13.

[0055] One update of the second image restoration network model includes steps S125-S127.

[0056] In this embodiment, the trade-off coefficient λ 1 and λ 2 are all set values, and are specifically set to 0.1 in the subsequent embodiments.

[0057] The above-mentioned dual-modal digital holographic imaging method based on deep learning is described below in conjunction with specific embodiments.

[0058] Example 1 This embodiment adopts Figure 4In the optical system shown, the electric light source 11 adopts a xenon lamp with a central wavelength of 633nm and a bandwidth of 20nm, and the laser 21 adopts a helium-neon gas laser.

[0059] In the first optical path, the incoherent light emitted by the xenon lamp is collimated by the collimating lens 12, and then focused to the sample surface by the lens 13. The incoherent light carrying the object wave information is imaged by the first microscope objective lens 14, and then passes through the beam splitter 30 to reach the surface of the spatial light modulator 40. The spatial light modulator 40 is composed of a polarizer and a half-wave plate (HWP).

[0060] In the second optical path, the beam diffuser 22 irradiates the pump laser onto the sample surface at the detection position 60; then, the coherent light carrying the sample information is imaged through the second microscope objective 23 and reaches the surface of the spatial light modulator 40 through the phase shift module 24 and the beam splitter 30.

[0061] The beam splitter 30 combines the incident light into one beam and projects it onto the surface of the spatial light modulator 40 , and then onto the electrical coupling device 50 .

[0062] In this embodiment, the sample uses a USAF1951 resolution plate, and the reconstruction results are shown in Figures 5(a), 5(b), 5(c), 5(d), and 5(e), wherein Figure 5(a) is a hybrid hologram collected by the electric coupling device 50 when both the xenon lamp and the laser 21 are started, that is, a double-beam interference holographic image; The dual-beam interference holographic image Figure 5(a) is subjected to frequency selective processing to obtain two frequency selected images. The two frequency selected images are respectively subjected to angular spectrum reconstruction to obtain reconstructed images Figures 5(b) and 5(c), showing the dual-modal holographic separation of incoherent light and coherent light; the two frequency selected images are respectively mapped by the corresponding image restoration network model to obtain reconstructed images Figures 5(d) and 5(e).

[0063] from Figure 5(a) to Figure 5(e) It can be seen that the frequency selection processing can strip off and reconstruct the wavefront in the corresponding channel and eliminate the interference of the conjugate term and the zero-level term.

[0064] In this embodiment, the same column of pixels indicated by the yellow line is selected in FIG. 5( b ) and FIG. 5( d ) for intensity comparison, and the results are as follows: Figure 6 As shown; in Figure 5(c) and Figure 5(e), the same column of pixels shown by the yellow line is selected for intensity comparison, and the results are as follows Figure 7 As shown in the figure, the resolution of Figure 5(b) and Figure 5(c) is 128.0lp / mm; the resolution of Figure 5(d) and Figure 5(e) is 203.0lp / mm. It can be seen that the resolution of the holographic reconstructed image after deep learning processing is improved by 58.59%, achieving super-resolution reconstruction.

[0065] In the optical system, the focus of the first microscope objective lens 14 coincides with the focus of the second microscope objective lens 23, and the distance between the sample and the focus is recorded as the axial distance Δz. In this embodiment, the axial distance Δz is moved in steps of 0.25 mm until the axial distance Δz reaches 1.25 mm, and the hologram at each position is recorded in sequence, as shown in Figures 8(a), 8(b), 8(c), and 8(d), where a1-a6 are conventional imaging at different axial distances Δz, that is, in the optical system, after the spatial modulator (SLM) is replaced with a reflector, the hologram collected by the electric coupling device 50; b1-b6 are double-beam interference holographic images at different axial distances Δz; The dual-beam interference holographic images at different axial distances Δz in b1-b6 are processed by frequency selection to obtain two frequency-selective images. The two frequency-selective images corresponding to different axial distances Δz in b1-b6 are reconstructed by angular spectrum reconstruction as shown in c1-c6 and d1-d6; the two frequency-selective images corresponding to different axial distances Δz in b1-b6 are mapped by the corresponding image restoration network model to obtain reconstructed images as shown in e1-e6 and f1-f6.

[0066] In this embodiment, the columns of images with different letters and the same numbers in Figures 8(a), 8(b), 8(c), and 8(d) are compared to obtain the peak signal-to-noise ratio and structural similarity of the reconstructed images obtained by the two methods on the two optical paths of the optical system at different axial distances Δz, as shown in Figures 9(a) and 9(b), wherein FSUnet(Chanel 1) represents the reconstructed image of the frequency-selective image corresponding to the first optical path after being mapped by the image restoration network model, FSUnet(Chanel2) represents the reconstructed image of the frequency-selective image corresponding to the second optical path after being mapped by the image restoration network model, FS(Chanel 1) represents the reconstructed image of the frequency-selective image corresponding to the first optical path after angular spectrum reconstruction, and FS(Chanel 2) represents the reconstructed image of the frequency-selective image corresponding to the second optical path after angular spectrum reconstruction. It can be seen that both the peak signal-to-noise ratio and the structural similarity decrease with the decrease of the axial distance Δz, while the decrease gradient after the image restoration network model reconstruction method is more significant, proving that the image restoration network model reconstruction has a higher axial resolution, proving that the dual-modal digital holographic imaging method based on deep learning proposed in the present invention has the potential to output target slices.

[0067] Example 2 This embodiment adopts the optical system of embodiment 1, and takes USAF1951 resolution plate, Paramecium, fruit pulp hairs, Ascaris eggs and yeast as samples.

[0068] Referring to Figure 10(a), for the USAF1951 resolution plate, Figure 4An optical system is used to obtain a double-beam interference holographic image G, and then the double-beam interference holographic image is subjected to frequency selective processing; after the two frequency-selective images are reconstructed through angular spectrum, pixel superposition is performed to obtain a holographic fusion image H after frequency-selective reconstruction; after the two frequency-selective images are mapped through an image restoration network model, pixel superposition is performed to obtain a holographic fusion image I after frequency-selective reconstruction; the intensity variation trends of the double-beam interference holographic image and the two holographic fusion images on the same intensity cross section are shown in Figure J.

[0069] In this embodiment, the dual-beam interference holographic image G corresponding to Paramecium, fruit pulp hairs, Ascaris eggs and yeast, the holographic fusion image H reconstructed by angular spectrum, the holographic fusion image I after model mapping and the intensity change trend comparison diagram J are shown in Figure 10(b), Figure 10(c), Figure 10(d) and Figure 10(e), respectively.

[0070] By comparing the two holographic fusion images of the same sample, it can be seen that the holographic fusion image I obtained by the dual-modal digital holographic imaging method based on deep learning proposed in the present invention is clearer and has less background noise.

[0071] Comparison of the intensity variation trends of the dual-beam interference holographic image and the two holographic fusion images of the same sample on the same intensity cross section shows that the curve characteristics of the holographic fusion image I of each sample are more consistent with the cross section of the corresponding position of the target than H.

[0072] Of course, it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any reference numerals in the claims should not be regarded as limiting the claims involved.

[0073] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0074] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.

Claims

1. A dual-modal digital holographic imaging method based on deep learning, characterized in that: The following steps are involved: S1. Construct an image restoration network model whose input is the reconstructed image and whose output is the restored image. The image restoration network model corresponds to the light source one by one. S2, performing frequency selection processing on the dual-beam interference holographic image, outputting frequency-selected images with frequencies corresponding to the two light sources, and inputting the two frequency-selected images into the corresponding image restoration model; S3. Superimpose and normalize the restored images output by the two image restoration models to obtain a fused restored image.

2. The dual-modal digital holographic imaging method based on deep learning according to claim 1, characterized in that: The image restoration network model includes a first convolution unit, a second convolution unit, a third convolution unit, a fourth convolution unit, a fifth convolution unit, a sixth convolution unit, a first deconvolution unit, a second deconvolution unit, a third deconvolution unit, a fourth deconvolution unit and a fully connected layer which are sequentially connected; in the first to sixth convolution units, the output of the previous convolution unit is input into the next convolution unit after maximum pooling processing; the output of the Nth convolution unit is also connected to the input of the 5th to Nth deconvolution units, 1≤N<5.

3. The dual-modal digital holographic imaging method based on deep learning according to claim 2, characterized in that: The first convolution unit, the second convolution unit, the third convolution unit, the fourth convolution unit, the fifth convolution unit, the sixth convolution unit, the first deconvolution unit, the second deconvolution unit, the third deconvolution unit and the fourth deconvolution unit are all composed of sequentially stacked convolution layers, BatchNorm2d functions and activation functions.

4. The dual-modal digital holographic imaging method based on deep learning according to claim 1, characterized in that: In step S2, a double-beam interference holographic image is collected by an optical system; The optical system comprises: a first optical path, a second optical path and an interference optical path; The first optical path includes: an electric light source, a collimating lens, a lens and a first microscope objective lens arranged along the light propagation direction; The second optical path includes: a laser, a beam diffuser, a second microscope objective lens and a phase shift module arranged along the light propagation direction; The intersection of the first optical path and the second optical path is a detection position, and the detection position is used to place the detection target; on the first optical path, the detection position is located between the lens and the first microscope objective lens; on the second optical path, the detection position is located between the beam diffuser and the second microscope objective lens; The interference optical path includes: a beam splitter, a spatial light modulator and an electric coupling device; The outgoing light of the first microscope objective lens and the outgoing light of the phase shift module are combined by a beam splitter and then transmitted to the spatial light modulator, and the outgoing light of the spatial light modulator is collected by an electrical coupling device to generate a hologram; When both the electric light source and the laser are started, the hologram generated by the electric coupling device is the double-beam interference holographic image of S2.

5. The dual-modal digital holographic imaging method based on deep learning according to claim 4, characterized in that: The training method of the two-way image restoration network model includes the following steps: S11, placing a test target at a test position, and obtaining training sets corresponding to two light sources; the training set of one light source is obtained by: starting the light source and turning off the other light source, and when the SLM is loaded with a mask, the image of the hologram collected by the CCD after angular spectrum diffraction is used as the reconstructed image; when the SLM is replaced with a reflector, the image collected by the CCD is used as the real image; adjusting the light output angle of the electric light source, generating multiple groups of samples {reconstructed image, real image} to constitute the training set of the light source; S12, respectively training two image restoration network models on the corresponding training sets; S13. When the number of iterations of the two image restoration network models reaches an integer multiple of the set threshold, the overall loss function is calculated. L total ; If the overall loss function L total If convergence occurs, the two-way image restoration network model is fixed; otherwise, the process returns to step S12.

6. The dual-modal digital holographic imaging method based on deep learning according to claim 5, characterized in that: The overall loss function is: L total =( L '1+ L '2) / ( L '1 / L '2) in, L '1 represents the loss function of the image restoration network model corresponding to the electric light source, L '2 represents the loss function of the image restoration network model corresponding to the laser; or, L '2 represents the loss function of the image restoration network model corresponding to the electric light source, L '1 represents the loss function of the image restoration network model corresponding to the laser; The loss function of the image restoration network model is the sum of the mean square error loss and the spectral norm regularization term. During the training process of the image restoration network model, the optimization goal is to minimize the expectation of the loss function to iterate the image restoration network model.

7. The dual-modal digital holographic imaging method based on deep learning according to claim 5, characterized in that: The method for training the image restoration network model on the training set in S12 includes the following steps: Extract training samples from the training set and substitute them into the image restoration network model, so that the image restoration network model can learn autonomously; A verification sample is extracted from the training set and input into the image restoration network model, and the image restoration network model predicts the restored image of the verification sample; The mean square error between the restored image and the real image is calculated on the verification sample, and the image restoration network model parameters are updated according to the optimization target; the optimization target is the expectation of the sum of the mean square error and the spectral norm regularization term; Repeat the above steps until the number of updates of the image restoration network model reaches an integer multiple of the set threshold.

8. The dual-modal digital holographic imaging method based on deep learning according to claim 4, characterized in that: A reflector is provided at the end of the first optical path; and a reflector is provided between the second microscope objective lens and the phase shift module on the second optical path.

9. A system using the dual-modal digital holographic imaging method based on deep learning as described in any one of claims 1 to 8, characterized in that: It includes an optical system, a frequency selection module, a first image restoration network model, a second image restoration network model and an image superposition module; The optical system is used to generate a double-beam interference holographic image. The double-beam interference holographic image is frequency-selected by a frequency selection module to be two-channel frequency-selected images. The two-channel frequency-selected images are processed by a first image restoration network model and a second image restoration network model respectively. The output image of the first image restoration network model and the output image of the second image restoration network model are output after pixel superposition by an image superposition module.

10. A memory, characterized in that: A computer program, a first image restoration network model and a second image restoration network model are stored, and when the computer program is executed, it is used to implement the deep learning-based dual-modal digital holographic imaging method as described in any one of claims 1-8.

Citation Information

Patent Citations

  • Dual-mode digital holographic microscope

    CN107966110A

  • Optical self-interference digital holographic reconstruction method and system based on deep learning

    CN116147531A

  • High-speed multi-mode depth-of-field continuation microscopic imaging system and method based on electronic control zoom lens

    CN117111283A

  • Three-dimensional detection method for apparent defects of polaroid

    CN117368228A

  • Three-dimensional reconstruction system and method

    CN117422821A

Cited By

  • Digital holographic reconstruction optimization model and method based on cycle consistency adaptive optimization

    CN120509450A

  • System for realizing dual-channel self-interference digital holographic real-time edge enhancement through T-Net

    CN120782652A

  • A system for realizing double-channel self-interference digital holographic real-time edge enhancement by T-Net

    CN120782652B