Holographic reconstruction method based on frequency domain enhancement and physical driving depth expansion network

Abstract

The invention discloses a holographic reconstruction method based on frequency domain enhancement and a physical driving deep expansion network, which is used for solving the problems of twin image interference and phase information loss in digital holographic reconstruction and constructing a multi-stage deep expansion network architecture. A physical forward propagation model based on an angular spectrum method is fused into a residual error back projection mechanism, a data fidelity module is constructed to ensure that a reconstruction result meets physical diffraction constraints, a multi-scale gating frequency domain sensing mechanism is introduced into a priori module, global features are extracted in a frequency domain through an annular gating strategy, and a reconstruction result is obtained. In combination with mixed down-sampling and frequency spectrum feature fusion technologies, effective interaction of spatial domain and frequency domain features is realized, a self-supervision strategy based on a physical model is adopted in a training process, and network optimization is constrained by using strength consistency of a measurement domain. According to the method, twin image noise can be effectively suppressed, the amplitude and phase information of the hologram can be recovered in a high-fidelity manner, and the method has excellent generalization capability and robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computational imaging and signal processing, and relates to a method for digital holographic wavefront reconstruction by combining deep learning with a physical imaging model. The present application is particularly suitable for lensless holographic microscopic imaging, quantitative phase imaging, and biomedical microscopic observation, etc. BACKGROUND

[0002] Digital holography is an important means for obtaining the three-dimensional morphology and refractive index distribution of microscopic objects by recording the interference pattern of the object's diffracted light wave and using numerical reconstruction algorithms to recover the amplitude and phase distribution of the object. However, existing image sensors (such as CMOS or CCD) can only respond to intensity information of the light wave and lose the phase information, so the recovery of the light field from the intensity hologram is essentially a phase recovery problem with ill-posedness.

[0003] The hologram records the interference fringes of the object light and the reference light. In the reverse propagation reconstruction, the twin image will overlap with the original image, seriously affecting the imaging quality. Although the traditional physical iterative algorithm such as the Gerchberg-Saxton algorithm has a clear physical meaning, it has slow convergence speed and is easily trapped in local minimum under low signal-to-noise ratio or under-sampling conditions, making it difficult to completely eliminate the twin image. In recent years, the end-to-end convolutional neural network has made a breakthrough in reconstruction speed, but most of the networks are pure data-driven "black box" models, lacking physical constraints. This means that the network has poor generalization ability on data outside the training set distribution, and is difficult to adapt to changes in imaging parameters such as diffraction distance and wavelength. Existing deep learning methods usually split the light field into real and imaginary parts as two independent real channels for processing. This approach breaks the inherent algebraic coupling characteristics of the complex domain, making it difficult to unwrap the phase or causing phase truncation errors. The essence of hologram is the diffraction field of the object, and the information of a point on the object will spread to the entire surface of the hologram. Ordinary convolutional neural networks are limited by local receptive fields and are difficult to capture this global long-distance dependence, resulting in loss of high-frequency details in the reconstruction result.

[0004] Therefore, there is an urgent need for a high-fidelity holographic reconstruction method that can combine the rigor of the physical model with the expressive ability of deep learning, and effectively handle the global and local features of the complex domain. SUMMARY

[0005] In view of the defects of the prior art, the present application provides a complex-valued holographic reconstruction method based on frequency domain enhancement and physical-driven deep unfolding network. The present application adopts a deep unfolding strategy to unfold the physical iterative optimization algorithm into a neural network layer, and designs a complex-valued frequency domain enhancement prior module to realize the organic combination of physical constraints and data-driven.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] S1, control the holographic imaging system to collect the intensity hologram of the measured object at M different diffraction distances in the optical axis direction, and stack the M intensity holograms to construct an input holographic tensor Y;

[0008] S2, perform convolution feature extraction on Y by using an initialization network Net1, and expand the channel dimension to a multi-channel feature of a preset dimension to generate an initial multi-channel complex feature distribution Z (0) ;

[0009] S3, input Z (0) to a deep unfolding network comprising K cascaded stages for iterative optimization, compress the input high-dimensional feature to a single-channel light field G (1) through a dimension reduction projection layer (1) , input G (2) to a residual back-projection data fidelity module based on an angular spectrum method, calculate the diffraction residual and perform back propagation for correction to obtain the light field G (2) , remap G H back to a high-dimensional feature through a dimension increasing projection layer, remove artifacts and restore high-frequency information by using a complex-valued frequency domain enhancement prior module, and output the high-dimensional complex feature Z optimized in this stage;

[0010] S4, input the high-dimensional complex feature output by the last cascaded stage in the deep unfolding network to a reconstruction network Net2 to generate normalized amplitude images and phase images;

[0011] S5, based on the ASM forward physical propagation constraint, optimize the parameters of the deep unfolding network without the true value.

[0012] Preferably, in step S2, the initialization network Net1 uses a large-scale convolution kernel not less than 7x7 for spatial filtering of the input holographic tensor, then performs feature correction through a batch normalization layer and a nonlinear activation layer, and finally expands the feature channel number from the input dimension to 2N dimensions through a point convolution layer to complete the mapping from the measurement domain to the high-dimensional complex feature domain.

[0013] Preferably, in step S3, the residual back-projection data fidelity module uses an angular spectrum method to construct a forward propagation operator A and an adjoint propagation operator A H , wherein the two operators are embedded with a transfer function stack generated according to the physical size of the hologram, wavelength and diffraction distance, calculate the predicted intensity field of the current light field after propagation by the A operator, calculate the residual between the predicted intensity field and the real measured hologram, and perform back propagation of the residual to the object plane through the A H operator, and update Y by gradient descent with a learnable step parameter a to enhance the physical consistency of the light field.

[0014] Preferably, in step S3, the complex-valued frequency domain enhancement prior module adopts a complex-valued neural network architecture, the network body is a U-shaped structure comprising an encoder, a decoder and a skip connection, and all convolutional layers, normalization layers and activation function layers are defined in the complex number domain, at each level of the encoder and the bottleneck layer, a multi-scale gated frequency domain perception unit is introduced in parallel, which is used to capture long-distance global context information in the frequency domain, wherein the complex-valued frequency domain enhancement prior module injects the frequency domain features into the spatial domain feature stream through a residual connection.

[0015] Preferably, the multi-scale gated frequency domain perception unit comprises a frequency domain ring gating mechanism, performs two-dimensional fast Fourier transform (FFT) and frequency spectrum centering on the input complex feature map, converts the feature from the spatial domain to the frequency domain, constructs multiple sets of concentric ring masks with different radius thresholds, divides the frequency spectrum into low-frequency, medium-frequency and high-frequency regions, respectively weights and gates the frequency spectrum energy in different ring regions through learnable parameters, uses a complex multi-layer perception to perform nonlinear mapping on the modulated frequency spectrum features, and finally restores the processed features to the spatial domain through inverse Fourier transform.

[0016] Preferably, the down-sampling operation of the complex-valued frequency domain enhancement prior module adopts a hybrid space-frequency strategy, constructs spatial and frequency domain branches working in parallel, the spatial domain branch extracts local spatial features through complex convolution, the frequency domain branch is converted to the frequency domain through FFT, uses a radial Butterworth low-pass filter with a learnable cutoff frequency for anti-aliasing processing, inversely transforms back to the spatial domain after cutting the high frequency, and adaptively weights and merges the spatial and frequency domain branches through a gating fusion layer to obtain output features.

[0017] Preferably, the complex-valued frequency domain enhancement prior module introduces an FFT feature fusion module in the skip connection between the encoder and the decoder, performs FFT on the encoder features and the decoder features respectively, extracts the phase spectrum of the decoder features, and fuses the amplitude spectrum of the encoder features and the phase spectrum of the decoder features in the frequency domain to generate context features, calculates the spatial root mean square (RMS) of the context features RMS1 and the spatial RMS of the decoder features RMS2 respectively, rescales the context features by calculating the ratio of RMS2 and RMS1, and injects the rescaled features into the decoder path.

[0018] Preferably, in step S5, the ASM forward physical propagation constraint propagates the predicted object light field output by the network to the holographic recording plane through ASM forward physical propagation during the training process, obtains a predicted holographic intensity map, and optimizes the parameters of the deep unfolding network based on the intensity consistency loss between the predicted holographic intensity map and the original input holographic tensor in step S1 and the total variation regularization loss of the predicted phase without object true value.

[0019] The present application has the advantages that:

[0020] 1、The method ingeniously constructs an iterative network architecture based on deep unfolding, explicitly embeds the angular spectrum diffraction physical model into the data fidelity module of the network, and the network is no longer an uninterpretable model for simply fitting the data distribution, but converts the holographic reconstruction into an optimization solving process under the constraint of the physical diffraction law. Through the alternating projection between the feature manifold and the physical measurement space, the network output is forced to be constrained in the solution space that satisfies the wave equation, thereby suppressing the non-physical artifacts of pure deep learning methods on low signal-to-noise ratio or out-of-distribution data. This physical constraint mechanism not only guarantees the authenticity of the reconstructed light field in the propagation characteristics, but also significantly enhances the robustness of the algorithm in the face of environmental noise and system errors.

[0021] 2、The application designs a multi-scale gated frequency domain perception unit, uses the global attribute of Fourier transform and the ring gating mechanism, so that the network can perceive the widely distributed diffraction fringe information in the hologram, and effectively solves the problem of insufficient local receptive field of the traditional convolutional neural network.

[0022] 3、The application supports a self-supervised learning strategy based on physical forward propagation, which can complete network training only by relying on the actually collected holographic intensity diagram, without the need to register to obtain the corresponding amplitude and phase true value data, thereby significantly reducing the dependence on high-cost experimental true value data and reducing the calibration difficulty and experimental complexity. The method has good adaptability to the number of input holograms. Under the condition of single input, relying on the complex-valued frequency domain enhancement prior, noise and artifacts can be effectively suppressed, and lost high-frequency details can be recovered, and still a clear structure and high contrast reconstruction result is obtained. When multiple inputs are used, the axial diversity information brought by different diffraction distances or focal planes can be fully utilized, and through physical consistency constraint and multi-view fusion, interference components such as twin images can be effectively weakened and eliminated, and the resolution, signal-to-noise ratio and stability of the reconstructed image are improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The flowchart of the holographic reconstruction method based on the frequency domain enhancement and physically driven deep unfolding network of the application

[0024] Figure 2 The embodiment diagram of the holographic reconstruction method based on the frequency domain enhancement and physically driven deep unfolding network of the application DETAILED DESCRIPTION

[0025] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto.

[0026] Reference Figure 1 The holographic reconstruction method based on the frequency domain enhancement and physically driven deep unfolding network of the application includes the following steps:

[0027] S1, a coaxial holographic imaging system is built, a precision displacement stage sensor is controlled to move along the optical axis, and three intensity holograms are collected at three equidistant axial positions. The three collected original holograms are preprocessed, cropped to a 256x256 pixel region to match the network input, then normalized to map the pixel intensity values to the [0, 1] interval. The three processed images are stacked in the channel dimension to construct a shaped input hologram tensor Y;

[0028] S2, Y is input to the initialization network Net1. First, a two-dimensional convolutional layer is used, which selects a large-scale convolution kernel to effectively capture widely distributed interference fringe features in the shallow layer of the network, and maps and expands the channel dimension to high-dimensional features to generate an initial high-dimensional complex feature distribution Z (0) ;

[0029] S3, Z (0) is input into a deep unfolding network containing K=5 cascaded stages for iterative optimization. The input high-dimensional features are compressed to a single-channel light field G (1) through a dimension reduction projection layer (1) . G (2) is input into a residual back-projection data fidelity module based on the angular spectrum method to calculate the diffraction residual and correct it by back propagation to obtain the light field G (2) . G (2) is remapped back to high-dimensional features through a dimension increasing projection layer, and a complex-valued frequency domain enhancement prior module is used to remove artifacts and restore high-frequency information through a multi-scale gated frequency domain perception mechanism to output the optimized high-dimensional complex features of this stage.

[0030] S4, the high-dimensional features Z (5) output by the fifth stage are input into Net2, which compresses the channels through a complex convolutional layer and generates normalized amplitude images and phase images respectively using nonlinear activation functions to decouple.

[0031] S5, based on the ASM forward physical propagation constraint, the parameters of the deep unfolding network are optimized without the true value.

[0032] Referring to Figure 2 , in step S1, a semiconductor laser with a wavelength of 532 nm is used as a light source, and a laser beam passes through a collimating beam expander to form a plane wave with flat wavefront to illuminate the sample to be measured. A Basler acA2040-90μm type CCD camera is used to collect holograms, with a resolution of 2048x2048 pixels and a pixel size of 5.5μm. Three coaxial holograms are recorded at a diffraction distance of 4mm, 8mm and 12mm.

[0033] Referring to Figure 2In step S2, the first layer of the initialization network Net1 uses a large-scale convolution kernel with a size of 7*7, a step of 1, and padding of 3 to perform spatial filtering on the input holographic tensor to capture long-distance correlation features of the holographic interference fringes, then performs feature correction through a batch normalization layer and a ReLU nonlinear activation layer, and finally expands the feature channel number from the input dimension to 64 dimensions through a point convolution layer with a convolution kernel size of 1*1, completes the mapping from the measurement domain to the high-dimensional complex feature domain, and generates an initial high-dimensional complex feature distribution Z (0) .

[0034] Referring to Figure 2 In step S3, the residual back-projection data fidelity module uses an angular spectrum method to construct a forward propagation operator A and an adjoint propagation operator A H The two operators are embedded with a transfer function stack H generated according to the physical size 256*256 of the hologram, the wavelength 532 nm, the pixel size 5.5 μm, and the diffraction distance 4 mm, 8 mm, and 12 mm f , calculates G (1) , and the predicted intensity field I pred after propagation through the A operator pred , calculates the difference residual r between the predicted intensity field I meas and the real measured hologram I H , reverses the propagation of the residual r back to the object plane through the A (2) operator, and updates Y through gradient descent with a learnable step size parameter with an initial value of 0.08 to enhance the physical consistency of the light field, and uses a 1*1 convolution to map the corrected light field G i back to the 64-channel high-dimensional features.

[0035] Referring to Figure 2 In step S3, the complex-valued frequency domain enhancement prior module uses a complex-valued convolutional neural network architecture, the network main body is an UNet containing an encoder, a decoder, and a jump connection with 3 down-sampling stages, and all convolution layers, normalization layers, and activation function layers are defined in the complex domain, at each level and the bottleneck layer of the encoder, a multi-scale gated frequency domain perception unit is introduced in parallel to capture long-distance global context information in the frequency domain, and the prior module injects frequency domain features into the spatial domain feature stream through a residual connection.

[0036] Referring to Figure 2 The multi-scale gated frequency domain perception unit includes a frequency domain ring gating mechanism, performs two-dimensional fast Fourier transform and frequency spectrum centering on the input complex feature map, converts the features from the spatial domain to the frequency domain, constructs three sets of concentric ring masks M i with different radius thresholds of 0 to 0.2, 0.2 to 0.6, and 0.6 to 1 i , divides the frequency spectrum into low, medium, and high frequency regions, and uses learnable parameters si The spectral energy in different annular regions is weighted and gated modulated respectively:

[0037]

[0038] wherein, is a Sigmoid activation function, which uses a complex multilayer perceptron to perform nonlinear mapping on the modulated spectral features, and finally restores the processed features to the spatial domain through inverse Fourier transform.

[0039] Referring to Figure 2 , the downsampling operation in the prior module adopts a hybrid spectral-spatial downsampling strategy, constructing spatial and spectral branches that work in parallel. The spatial branch uses a complex convolution with a step size of 2 to extract local spatial features Y spatial The spectral branch first converts the input to the frequency domain, applies a radial Butterworth low-pass filter with a cutoff frequency of 0.4 for anti-aliasing, then crops the high-frequency part in the center of the frequency domain to reduce the resolution, and finally inversely transforms it back to the spatial domain:

[0040]

[0041] wherein Crop represents a center cropping operation, and the output features of the spatial and spectral branches are adaptively weighted and merged by the gating fusion layer:

[0042]

[0043] wherein is a gating coefficient generated by a convolution layer and a Sigmoid activation function, used to dynamically adjust the fusion ratio of spatial and spectral features.

[0044] Referring to Figure 2 , the prior module introduces an FFT feature fusion module in the skip connection between the encoder and the decoder, performs Fourier transform on the encoder features and the decoder features respectively, extracts the phase spectrum of the decoder features:

[0045]

[0046] and fuses it with the amplitude spectrum of the encoder features in the frequency domain to generate context features, which can be represented as:

[0047]

[0048] wherein, and are the feature maps of the encoder and the decoder respectively, and are their corresponding spectrums, is the preliminary fused feature, denotes the phase factor of the decoder feature.

[0049] respectively calculate the spatial root mean square (RMS1) energy of the context feature and the spatial root mean square (RMS2) energy of the decoder feature, calculate the ratio of the numerator RMS2 and the denominator RMS1, and rescale the context feature using the scaling coefficient:

[0050]

[0051] wherein, is the final fused feature injected into the decoder, denotes calculating the root mean square energy of the feature map and injecting the rescaled feature into the decoder path.

[0052] Referring to Figure 2 , an ASM forward physical propagation layer is constructed, and in the training process, the predicted object light field output by the network is propagated to the holographic recording plane through the ASM forward physical propagation layer to obtain a predicted holographic intensity map, the intensity consistency loss between the predicted holographic intensity map and the original input holographic tensor collected in step S1 is calculated, and the total variation regularization loss of the predicted phase is combined, and the weight is set to 5x10 -3 , without the object true value, the parameters of the depth unfolding network are optimized. The hardware platform is configured as NVIDIARTX 4090D, the batch size is set to 8, the optimizer is selected as AdamW, and the initial learning rate is 1x10 -4 .

[0053] The above is the preferred embodiment of the present application, but the embodiments of the present application are not limited by the above, any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application, should be equivalent replacement method, all included in the protection scope of the present application.

Claims

1. A holographic reconstruction method based on frequency domain enhancement and physically driven deep unfolding networks, characterized in that, Includes the following steps: S1. Control the holographic imaging system to acquire intensity holograms of the object under test at M different diffraction distances along the optical axis, and stack the acquired intensity holograms to construct the input holographic tensor Y; S2. Use the initialization network Net1 to perform convolutional feature extraction on Y, and expand the channel dimension to a preset dimension of multi-channel features to generate an initial multi-channel complex feature distribution Z. (0) ; S3, Z (0) The input is fed into a deep unfolded network containing K cascaded stages for iterative optimization. The high-dimensional features of the input are compressed through a dimensionality reduction projection layer to obtain the single-channel light field G. (1) , will G (1) Input the residual back projection data fidelity module based on the angular spectrum method, calculate the diffraction residual, and correct it by backpropagation to obtain the light field G. (2) G is projected through an updimensional projection layer (2) Remap back to high-dimensional features, use the complex-valued frequency domain enhancement prior module to remove artifacts and recover high-frequency information, and output the optimized high-dimensional complex feature Z at this stage; S4. Input the high-dimensional complex features output from the last cascade stage in the deep unfolded network into the reconstructed network Net2 to generate normalized amplitude and phase images. S5. Based on the ASM forward physics propagation constraint, optimize the parameters of the deep unfolded network without the need for truth supervision.

2. The holographic reconstruction method based on frequency domain enhancement and physically driven depth unfolding network according to claim 1, characterized in that, In step S2, the initialization network Net1 uses a large-scale convolutional kernel of no less than 7×7 to perform spatial filtering on the input holographic tensor in its first layer. Then, it performs feature correction through a batch normalization layer and a nonlinear activation layer. Finally, it expands the number of feature channels to 2N dimensions through a point convolutional layer, thus completing the mapping from the measurement domain to the high-dimensional complex feature domain.

3. The holographic reconstruction method based on frequency domain enhancement and physically driven deep unfolding network according to claim 1, characterized in that, In step S3, the residual back projection data fidelity module uses the angle spectrum method (ASM) to construct the forward propagation operator A and the adjoint propagation operator A. H Two of the operators embed transfer function stacks generated based on the physical size, wavelength, and diffraction distance of the hologram. They calculate the predicted intensity field after the current light field propagates through operator A, and then calculate the residual between the predicted intensity field and the actual measured hologram. The residual is then passed through operator A. H The operator is backpropagated back to the object plane and updated with gradient descent on Y using a learnable step size parameter α, thereby enhancing the physical consistency of the light field.

4. The holographic reconstruction method based on frequency domain enhancement and physically driven deep unfolding network according to claim 1, characterized in that, In step S3, the complex-valued frequency domain enhancement prior module adopts a complex-valued neural network architecture. The main body of the network is a U-shaped structure containing an encoder, a decoder, and skip connections. All convolutional layers, normalization layers, and activation function layers are defined in the complex domain. At each level of the encoder and the bottleneck layer, multi-scale gated frequency domain sensing units are introduced in parallel to capture long-distance global context information in the frequency domain. The complex-valued frequency domain enhancement prior module injects frequency domain features into the spatial domain feature stream through residual connections.

5. The holographic reconstruction method based on frequency domain enhancement and physically driven depth unfolding network according to claim 4, characterized in that, The multi-scale gated frequency domain sensing unit includes a frequency domain ring gate mechanism. It performs a two-dimensional fast Fourier transform (FFT) and spectral centering on the input complex feature map to transform the features from the spatial domain to the frequency domain. It constructs multiple sets of concentric ring masks with different radius thresholds to divide the spectrum into low-frequency, mid-frequency, and high-frequency regions. It uses learnable parameters to perform weighted gate modulation on the spectral energy in different ring regions. It uses a complex multilayer perceptron to perform nonlinear mapping on the modulated spectral features. Finally, it restores the processed features to the spatial domain through inverse Fourier transform (IFFT).

6. The holographic reconstruction method based on frequency domain enhancement and physically driven depth unfolding network according to claim 4, characterized in that, The downsampling operation in the complex-valued frequency domain enhancement prior module adopts a hybrid spatial-frequency strategy, constructing parallel spatial and frequency domain branches. The spatial branch uses complex convolution to extract local spatial features, while the frequency branch is transformed to the frequency domain through FFT and anti-aliasing is performed using a radial Butterworth low-pass filter with a learnable cutoff frequency. After pruning the high-frequency part, it is inverse Fourier transformed back to the spatial domain. The output features are obtained by adaptively weighting and merging the spatial and frequency domain branches through a gated fusion layer.

7. The holographic reconstruction method based on frequency domain enhancement and physically driven deep unfolding network according to claim 4, characterized in that, The complex-valued frequency domain enhancement prior module introduces an FFT feature fusion module in the skip connection between the encoder and decoder. It performs FFT on the encoder features and decoder features respectively, extracts the phase spectrum of the decoder features, and fuses it with the amplitude spectrum of the encoder features in the frequency domain to generate context features. It calculates the spatial root mean square (RMS1) of the context features and the spatial root mean square (RMS2) of the decoder features respectively, and rescales the context features by calculating the ratio of RMS2 to RMS1. The rescaled context features are then injected into the decoder path.

8. The holographic reconstruction method based on frequency domain enhancement and physically driven depth unfolding network according to claim 1, characterized in that, In step S5, the ASM forward physical propagation constraint propagates the predicted object light field output by the network to the holographic recording plane via the ASM forward physical propagation during the training process to obtain a predicted holographic intensity map. Based on the intensity consistency loss between the predicted holographic intensity map and the original input holographic tensor in step S1, as well as the total variation regularization loss of the predicted phase, the parameters of the depth unfolding network are optimized without requiring the object ground truth.

Images