A Depth-of-Field Extension Method for Digital Holographic Microscopy Based on Spatial Adaptive Complex Generative Adversarial Networks

By constructing a complex generative adversarial network model, the technical challenge of depth-of-field extension in digital holographic microscopy was solved, achieving high-resolution and high-fidelity large-depth-of-field imaging, which is suitable for full-field imaging of complex samples.

CN123113267APending Publication Date: 2026-11-17ZHEJIANG SCI-TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611252975.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

When extending the depth of field, existing digital holographic microscopy methods are sensitive to speckle noise and diffraction artifacts. Multi-peak response makes focal plane positioning difficult. Real-valued depth networks disrupt the physical coupling of complex amplitudes. Fixed-parameter convolution kernels cannot adaptively compensate for spatial defocusing effects, thus limiting reconstruction accuracy and generalization ability.

Method used

A spatially adaptive complex generative adversarial network (GAN) approach is adopted. Hologram sequences are acquired through an off-axis digital holographic microscope system, and a complex GAN model is constructed, including a full complex value generator and a block probability discriminator. The multi-objective joint loss function is optimized to achieve adaptive compensation of the defocus effect by complex convolutional layers and dynamic kernels, and to reconstruct the complex amplitude.

Benefits of technology

While maintaining high spatial resolution, it significantly expands the imaging depth of field, achieving high-fidelity quantitative imaging. It is suitable for samples with tilted surfaces or non-uniform depth distribution, eliminating the need for axial mechanical scanning and improving reconstruction accuracy and generalization ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123113267A_ABST
    Figure CN123113267A_ABST
Patent Text Reader

Abstract

This invention discloses a depth-of-field extension method for digital holographic microscopy based on a spatially adaptive complex generative adversarial network (GAN). The method includes acquiring a sequence of defocused holograms, performing numerical reconstruction preprocessing to obtain defocused and focused complex amplitudes, and segmenting them to construct a dataset; constructing a complex GAN containing a full complex value generator and a block probability discriminator, wherein the generator consists of a Unet feature extraction backbone and a complex dynamic residual reconstruction branch; optimizing the multi-objective joint loss function through alternating adversarial training to obtain a depth-of-field extension model; and inputting the preprocessed defocused hologram to be reconstructed into the depth-of-field extension model to output the predicted focused complex amplitude. This invention significantly extends the imaging depth of field with almost no sacrifice in lateral resolution, achieving end-to-end reconstruction of complex amplitudes, and is suitable for samples with tilted surfaces or non-uniform depth distributions, without requiring axial mechanical scanning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of digital holographic reconstruction and deep learning technology, and specifically relates to a digital holographic microscopy depth-of-field extension method based on spatial adaptive complex generative adversarial networks. Background Technology

[0002] Digital holographic microscopy (DHM), a crucial technique in computational imaging, reconstructs the complex amplitude distribution of the object beam wavefront by recording the interference hologram of the object beam and reference beam, combined with diffraction propagation theories (such as angular spectral analysis or Fresnel diffraction). While DHM can reconstruct images at different depths through numerical propagation, it still faces depth-of-field (DOF) limitations in large-scale three-dimensional quantitative imaging. This leads to intensity blurring and phase distortion in sample regions outside the depth of field after holographic reconstruction. This limitation is particularly pronounced in high-resolution imaging when high numerical aperture (NA) microscope objectives are used to improve spatial resolution. Therefore, achieving high-fidelity quantitative imaging with large depth of field while maintaining high spatial resolution is the core challenge currently facing DHM.

[0003] To extend the depth of field in DHM imaging, existing methods primarily rely on numerical autofocus to determine the optimal reconstruction distance. This involves numerical diffraction propagation along the axial direction, using evaluation functions such as variance, gradient magnitude, Laplacian energy, and information entropy to quantify the reconstruction sharpness of each plane, and then using extreme values ​​to locate the optimal focal plane. However, these focusing evaluation metrics based on intensity or phase characteristics are highly sensitive to the inherent speckle noise and diffraction artifacts in coherent imaging, easily introducing spurious fluctuations into the evaluation curve. When the sample's three-dimensional structure spans multiple focal planes, the coherent superposition of reconstructed light fields at different depths causes the focusing contributions to compete and produce a multi-peak response, making precise localization extremely difficult.

[0004] In recent years, deep learning has provided an efficient data-driven paradigm for holographic imaging. However, while current methods achieve efficient reconstruction, their underlying real-valued network architecture struggles to maintain the physical consistency of complex amplitudes. Existing frameworks decompose complex-valued holographic data into two-channel real-valued tensors with real and imaginary parts (or amplitude and phase). Real-valued convolution operators cannot follow the rules of complex algebra operations, thus severing the inherent physical coupling of complex amplitudes. Research in the field of complex signal processing shows that complex-valued neural networks can directly follow complex algebra operations, maintaining the inherent coupling between real and imaginary parts, and are superior to real-valued equivalent architectures in terms of representational capability. In holographic imaging applications, end-to-end methods based on complex-valued networks also demonstrate the advantage of directly modeling complex amplitude fields. Furthermore, in extended depth-of-field imaging tasks, increased defocus distance causes the object wavefront diffraction to degrade, exhibiting highly non-stationary behavior, with significant differences in point spread function and phase transfer characteristics at different depths. Although physical perception complex attention and light field data-driven methods have improved the ability to model defocus degradation, the fixed-parameter globally shared convolutional kernels lack content-adaptive perception and are difficult to adaptively compensate for distance-dependent spatial changes in defocus effects at different axial positions, which limits the reconstruction accuracy and generalization ability of the network in a wide depth of field. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention provides a digital holographic microscopy depth-of-field extension method based on a spatially adaptive complex generative adversarial network. This method solves the technical problems of existing digital holographic microscopy when extending depth of field, such as the sensitivity of numerical autofocus methods to speckle noise and diffraction artifacts, the difficulty in focal plane positioning due to multi-peak response, the fragmentation of complex amplitude physical coupling by real depth networks, and the inability of fixed-parameter convolution kernels to adaptively compensate for spatial defocusing effects, which limits reconstruction accuracy and generalization ability. This method achieves high-fidelity quantitative imaging with large depth of field while maintaining high spatial resolution.

[0006] The technical solution adopted by the digital holographic microscopy depth-of-field extension method of the present invention is as follows:

[0007] S1. The off-axis digital holographic microscopy system is used to acquire the focused hologram of the sample under test at the reference focusing position and the defocused hologram at different axial defocus distances to form a hologram sequence.

[0008] S2. Perform numerical reconstruction preprocessing on each defocused hologram in the hologram sequence to obtain the corresponding defocused complex amplitude.

[0009] S3. Numerical reconstruction preprocessing is performed on the focused holograms in the hologram sequence. The optimal reconstruction distance is accurately calibrated by the automatic focusing algorithm. The wrapped phase is reconstructed and unfolded at the optimal reconstruction distance by the angular spectrum method. The residual aberration is then corrected by the alternating direction method to obtain the focused complex amplitude.

[0010] S4. Divide each defocused complex amplitude into image blocks to construct the input dataset; divide the focused complex amplitude into image blocks in the same way as the true value labels.

[0011] S5. Construct a complex generative adversarial network model, which includes a full complex value generator and a block probability discriminator.

[0012] S6. Input the input dataset into the full complex value generator. Optimize the multi-objective joint loss function through alternating adversarial training between the full complex value generator and the block probability discriminator until the block probability discriminator can no longer distinguish between the output of the full complex value generator and the true value label. Obtain the trained full complex value generator and use it as a depth extension model.

[0013] S7. Obtain the defocused hologram to be reconstructed, then perform numerical reconstruction preprocessing, extract and construct complex tensors, and divide them into image blocks. Input the images into the depth extension model for processing and output the corresponding predicted focus complex amplitude.

[0014] The acquisition of the hologram sequence is carried out in equal steps within the depth range of the positive and negative axes, with a preset reference focusing position as the center. The hologram sequence includes a focused hologram acquired at the reference focusing position and defocused holograms at multiple defocus distances.

[0015] The numerical reconstruction preprocessing in steps S1 and S2 involves performing a fast Fourier transform on the hologram, extracting the +1 level spectrum through spatial filtering and shifting it to the center of the frequency domain, separating and compensating for background phase aberrations using principal component analysis, and obtaining the aberration-precorrected complex amplitude through inverse Fourier transform.

[0016] Step S4 specifically involves: extracting the real and imaginary parts of each defocused complex amplitude, and using the real and imaginary parts as two channels to form a complex tensor corresponding to each defocused complex amplitude, which includes two independent channels; then spatially dividing each complex tensor into non-overlapping image blocks according to a preset fixed size, and constructing an input dataset from all the segmented image blocks; and extracting the real and imaginary parts of the focused complex amplitude to form a complex tensor, and segmenting it into image blocks of the same size as ground truth labels.

[0017] The full complex value generator comprises a Unet feature extraction backbone and a complex dynamic residual reconstruction branch. The Unet feature extraction backbone adopts an encoder-decoder architecture, with a bottleneck layer between the encoder and decoder. The encoder consists of multiple downsampling blocks connected in series, and the decoder consists of multiple upsampling blocks connected in series. Image patches in the input dataset serve as the input to the first downsampling block in the encoder, and the input to each subsequent downsampling block comes from the output of the previous downsampling block. The output of the last downsampling block is processed by the bottleneck layer and then serves as the input to the first upsampling block in the decoder. The input to each subsequent upsampling block comes from the feature map obtained by concatenating the output of the previous upsampling block and the output of the downsampling block at the corresponding scale. The bottleneck layer and each... Each upsampling block outputs a guided complex feature map at the corresponding scale; the complex dynamic residual reconstruction branch includes multiple complex dynamic residual blocks; the image blocks in the input dataset are downsampled multiple times and used as the first input of the first complex dynamic residual block. The first complex dynamic residual block also receives the guided complex feature map output by the bottleneck layer as the second input, and the output of the first complex dynamic residual block is sent to the next complex dynamic residual block; each subsequent complex dynamic residual block receives the output of the previous complex dynamic residual block as the first input, and also receives the guided complex feature map output by the upsampling block at the corresponding resolution scale as the second input, and the output of each subsequent complex dynamic residual block is sent to the next level complex dynamic residual block; the last complex dynamic residual block outputs the reconstructed complex amplitude.

[0018] Each complex dynamic residual block is processed according to the following steps: the first input of each complex dynamic residual block is processed by multiple cascaded complex-valued convolutional blocks and then concatenated with the corresponding second input along the channel to obtain a first residual feature map. The first residual feature map is processed by multiple cascaded complex-valued convolutional blocks and a complex-valued convolutional layer in the first branch to obtain a complex residual. The first residual feature map is further processed by multiple cascaded complex-valued convolutional blocks and a complex-valued convolutional layer in the second branch to obtain a dynamic kernel. The dynamic kernel is multiplied by the first input to obtain the dynamic residual. The dynamic residual, the complex residual, and the first input are added element-wise along the channel dimension to output a complex amplitude, which is used as the output of the current complex dynamic residual block. The complex-valued convolutional block is composed of a complex-valued convolutional layer and an activation layer connected in series.

[0019] The block probability discriminator consists of multiple feature extraction convolutional blocks connected in series with an output convolutional layer; each feature extraction convolutional block consists of an embedding spectrum normalized convolutional layer and an activation layer connected in series; the output convolutional layer contains only an embedding spectrum normalized convolutional layer; the first feature extraction convolutional block of the block probability discriminator receives the output of the full complex value generator and the true value label respectively, and the output convolutional layer finally outputs the spatial block probability matrix.

[0020] The multi-objective joint loss function includes generator loss and discriminator loss; the generator loss includes pixel-level average absolute error loss, multi-scale structural similarity loss, feature matching loss and generator adversarial loss; the discriminator loss is used to distinguish between the ground truth label and the output of the full complex value generator.

[0021] The generator loss is set according to the following formula:

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] The discriminator loss is set according to the following formula:

[0028] ;

[0029] in, For generator loss; , , and These are pixel-level mean absolute error loss, multi-scale structural similarity loss, feature matching loss, and generator adversarial loss, respectively. , , , and All are weighting coefficients; and All are indexes; This represents the total number of pixels in the feature map. and These are the imaginary and real parts of the focused complex amplitude, respectively; and These are the focused complex amplitudes corresponding to the first The imaginary and real parts of each pixel; and These represent the imaginary and real parts of the reconstructed complex amplitude, respectively. and These are the reconstructed complex amplitudes corresponding to the first... The imaginary and real parts of each pixel; It is a multi-scale structural similarity index; For the first The total number of elements in the output feature map of each feature extraction convolutional block; This represents the number of feature extraction convolutional blocks in the block probability discriminator; It is an L1 norm; and These are the outputs of the true value label and the reconstructed complex amplitude after being processed by the block probability discriminator, respectively. For the true value label, after the first Feature maps output by each feature extraction convolutional block; For discriminator loss; To reconstruct the complex amplitude via the first Feature maps output by each feature extraction convolutional block; To calculate the expectation of the reconstructed complex amplitude output by the full complex value generator over the generated sample distribution; To calculate the expectation of the focused complex amplitude on the real data distribution.

[0030] Step S7 specifically involves: acquiring the defocus hologram to be reconstructed at any axial position within the positive and negative axial depth range of the test sample at the reference focusing position; performing numerical reconstruction preprocessing on the defocus hologram to be reconstructed to obtain the defocus complex amplitude to be reconstructed; extracting the real and imaginary parts of the defocus complex amplitude to be reconstructed to form complex tensors, and dividing them into image blocks for input into the depth-of-field extension model for processing to output the corresponding predicted focus complex amplitude.

[0031] The beneficial effects of this invention are:

[0032] 1) The full complex value generator of the present invention directly processes the defocus complex amplitude and maintains the algebraic structure of the complex components through complex value convolutional layers and activation layers, thereby improving both reconstruction fidelity and depth-of-field extension capabilities.

[0033] 2) To address the spatial nonstationarity of defocus degradation characteristics, this invention designs a complex dynamic residual block. By assigning an independent complex local kernel to each pixel position through a dynamic kernel, the ability to characterize and compensate for distance-dependent spatial changes in defocus effects is enhanced.

[0034] 3) This invention significantly expands the imaging depth of field with almost no sacrifice in lateral resolution, realizes end-to-end reconstruction of complex amplitude, and is suitable for samples with inclined surfaces or non-uniform depth distribution without the need for axial mechanical scanning.

[0035] 4) Through systematic experimental verification, this invention extends the theoretical depth of field from about 10.6 μm to about 600 μm (an extension factor of about 56 times) in an off-axis digital holographic microscopy system using an NA 0.25 microscope objective, while maintaining high-fidelity positional recovery and structural resolution within an axial range of ±300 μm. For samples with tilted surfaces or non-uniform depth distribution, clear full-field imaging can be achieved in a single acquisition without axial mechanical scanning. Attached Figure Description

[0036] Figure 1 This is an overall architectural diagram of the present invention; wherein, Figure 1 (a) in the figure is a schematic diagram of the optical path of an off-axis digital holographic microscope system; Figure 1 (b) in the figure is a flowchart of the method of the present invention; Figure 1 (c) is a schematic diagram of the complex generative adversarial network model of the present invention.

[0037] Figure 2 This is a diagram illustrating the overall architecture of a complex generative adversarial network model; where, Figure 2 (a) in the diagram is the overall structure of the full complex value generator; Figure 2 (b) in the diagram is the overall structure of the block probability discriminator; Figure 2 (c) in the diagram is a detailed structural diagram of the complex residual block; Figure 2 (d) in the diagram is a detailed structural diagram of the complex dynamic residual block; Figure 2 (e) in the diagram is a schematic diagram of dynamic kernel generation and local dot product operation in complex dynamic filtering.

[0038] Figure 3 This is a reconstruction result of a female Ascaris cell section with an angled view, as shown in the example. Figure 3 (a) in the image is a defocused hologram; Figure 3 (b) and Figure 3 In the diagram, (c) represents the real and imaginary parts of the complex amplitude input to the network, respectively. Figure 3 In this context, (d) represents the true value label. Figure 3 In the diagram, (e) represents the reconstructed complex amplitude output by the network reconstruction. Figure 3 (f) in the figure represents the quantitative phase curves of three different defocus regions. Detailed Implementation

[0039] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

[0040] like Figure 1 As shown in (b) of this embodiment, the digital holographic microscopy depth-of-field extension method includes the following steps:

[0041] S1. The off-axis digital holographic microscopy system is used to acquire the focused hologram of the sample under test at the preset reference focusing position and the defocused hologram at different axial defocus distances to form a hologram sequence.

[0042] Specifically, the reference focusing position is the initial focusing position predetermined by manual or automatic focusing, which serves as the axial reference for acquiring the hologram sequence.

[0043] The acquisition of the hologram sequence is carried out in equal steps within the depth range of the positive and negative axes, with the preset reference focusing position as the center. The hologram sequence includes the focused hologram acquired at the reference focusing position and the defocused holograms at multiple defocus distances.

[0044] Specifically, such as Figure 1 As shown in (a), the off-axis digital holographic microscopy system includes a He-Ne laser, neutral density filters (NDF1 / NDF2), collimator and beam expander (BE), beam splitter (BS1 / BS2), mirror (M1 / M2), pinhole filter (PF1 / PF2), microscope objective (MO), tube lens (TL), and CMOS camera. The sample is fixed on a nanoscale linear piezo motor (LPM) stage. With the coarse focusing position as the reference focusing position, a sequence of holograms (including holograms at the coarse focusing position and off-focus holograms) is acquired in equal steps within the depth range of the positive and negative axes. The recorded hologram size is 2560×2560 pixels.

[0045] S2. Perform numerical reconstruction preprocessing on each defocused hologram in the hologram sequence to obtain the corresponding defocused complex amplitude.

[0046] S3. Numerical reconstruction preprocessing is performed on the focused holograms in the hologram sequence. The optimal reconstruction distance is accurately calibrated by the automatic focusing algorithm. The wrapped phase is reconstructed and unfolded at the optimal reconstruction distance by the angular spectrum method. The residual aberration is then corrected by the alternating direction method to obtain the focused complex amplitude.

[0047] Specifically, the complex amplitude after PCA correction is first obtained through numerical reconstruction preprocessing. Then, an autofocus algorithm based on the Tamura coefficient is used to accurately calibrate the optimal reconstruction distance. The complex amplitude is reconstructed at this precise distance using the angular spectrum method, and the wrapped phase is extracted and unfolded. Subsequently, the residual aberration is corrected using the alternating direction method, and the corrected sample phase is converted into a focused complex amplitude through complex exponential operations. This process generates a complete 2560×2560 pixel focused complex amplitude field, which is then cropped into 256×256 pixel or 512×512 pixel image blocks according to their spatial location corresponding to the input block in subsequent dataset construction, and decomposed into dual channels representing the real and imaginary parts of the network labels.

[0048] The numerical reconstruction preprocessing in steps S1 and S2 involves performing a Fast Fourier Transform (FFT) on the hologram, extracting the +1 level spectrum through spatial filtering and shifting it to the center of the frequency domain, separating and compensating for background phase aberrations using Principal Component Analysis (PCA), and obtaining the aberration-precorrected complex amplitude through an Inverse Fourier Transform (IFFT).

[0049] S4. Divide each defocused complex amplitude into image blocks to construct the input dataset; divide the focused complex amplitude into image blocks in the same way as the true value labels.

[0050] Specifically, the real and imaginary parts of each defocused complex amplitude are extracted separately, and the real and imaginary parts are used as two independent channels to form a complex tensor with two independent channels for each defocused complex amplitude. Each complex tensor is then spatially divided into non-overlapping image blocks according to a preset fixed size, and the input dataset is constructed from all the image blocks obtained by the segmentation. The real and imaginary parts of the focused complex amplitude are extracted to form a complex tensor, and then divided into image blocks of the same size as the ground truth labels.

[0051] like Figure 1 As shown in (c), S5, construct a complex generative adversarial network model, which includes a full complex value generator and a block probability discriminator based on the PatchGAN architecture.

[0052] like Figure 2 As shown in (a), the full complex value generator includes a Unet feature extraction backbone and a complex dynamic residual reconstruction branch.

[0053] The Unet feature extraction backbone adopts an encoder-decoder architecture with a bottleneck layer between the encoder and the decoder. The encoder is composed of multiple downsampling blocks connected in series, and the decoder is composed of multiple upsampling blocks connected in series. Specifically, the encoder gradually reduces the spatial resolution and extracts multi-scale features through multiple downsampling blocks, and the decoder restores the spatial resolution layer by layer through the corresponding upsampling blocks.

[0054] Image patches in the input dataset serve as the input to the first downsampled block in the encoder. The input to each subsequent downsampled block is derived from the output of the previous downsampled block. The output of the last downsampled block is processed by the bottleneck layer and then serves as the input to the first upsampled block in the decoder. The input to each subsequent upsampled block is derived from the feature map obtained by concatenating the output of the previous upsampled block and the output of the downsampled block at the corresponding scale along the channel. The bottleneck layer and the output of each upsampled block are guided complex feature maps at the corresponding scale.

[0055] like Figure 2As shown in (a) of the figure, in this embodiment, the encoder includes three cascaded downsampling blocks, the decoder includes three cascaded upsampling blocks, and a bottleneck layer is provided between the encoder and the decoder; the first downsampling block receives image blocks from the input dataset as input, the second downsampling block receives the output of the first downsampling block as input, the third downsampling block receives the output of the second downsampling block as input, and the output of the third downsampling block is processed by the bottleneck layer to obtain the guided complex feature map. The first upsampling block receives the guided complex feature map. As input.

[0056] The guided complex feature map is obtained by concatenating the outputs of the first upsampling block and the third downsampling block along the channel. The output of the second upsampling block and the output of the second downsampling block are concatenated along the channel to obtain the guided complex feature map. The input to the third upsampling block; the output of the third upsampling block and the output of the first downsampling block are concatenated along the channel to obtain the guided complex feature map. As the fourth complex dynamic residual block The second input; the bottleneck layer, the first upsampling block, the second upsampling block and the third upsampling block output guided complex feature maps of the corresponding scales respectively.

[0057] The first, second, and third downsampling blocks in the encoder are each composed of three complex-valued convolutional blocks connected in series. The stride of the first complex-valued convolutional layer in each downsampling block is set to 2, and the stride of the other two layers is 1, so as to achieve a gradual reduction of spatial resolution (H→H / 2→H / 4→H / 8).

[0058] The bottleneck layer is located after the third downsampling block and consists of the first complex-valued convolutional block, the first complex-valued residual block, the second complex-valued residual block, and the second complex-valued convolutional block connected in series, while maintaining the H / 8 resolution.

[0059] The decoder consists of three upsampling blocks, corresponding to the recovery from H / 8 to H / 4, H / 4 to H / 2, and H / 2 to H, respectively. Each upsampling block is composed of one complex-valued transposed convolutional layer (stride 2, achieving doubled resolution) and two complex-valued residual blocks concatenated. After upsampling, the final result is output through one complex-valued convolutional layer (stride 1).

[0060] Each complex-valued convolutional block consists of a complex-valued convolutional layer and a LeakyReLU activation layer connected in series. The entire network passes the features of each layer of the encoder to the corresponding decoder layer through skip connections.

[0061] like Figure 2As shown in (c), each complex-valued residual block includes two complex-valued convolutional layers and one LeakyReLU activation layer. The input of the complex-valued residual block is processed sequentially through the first complex-valued convolutional layer, the LeakyReLU activation layer, and the second complex-valued convolutional layer. The result is then added element-wise to the input of the complex-valued residual block via a skip connection to obtain the final output of the current complex-valued residual block.

[0062] The complex dynamic residual reconstruction branch consists of multiple complex dynamic residual blocks (CDRBs). Image patches in the input dataset are downsampled multiple times and used as the first input to the first CDRB. The first CDRB also receives a guiding complex feature map output from the bottleneck layer as its second input. The output of the first CDRB is fed into the next CDRB. Each subsequent CDRB receives the output of the previous CDRB as its first input and a guiding complex feature map output from the upsampled block at the corresponding resolution scale as its second input. The output of each subsequent CDRB is fed into the next level CDRB. The last CDRB finally outputs the reconstructed complex amplitude.

[0063] like Figure 2 As shown in (a) of the figure, in this embodiment, the complex dynamic residual reconstruction branch includes four complex dynamic residual blocks. The image blocks in the input dataset are downsampled by 8 times. As the first complex dynamic residual block The first input, the first complex dynamic residual block It also receives the guided complex feature map output from the bottleneck layer. As the second input; the second complex dynamic residual block Receive the first complex dynamic residual block Output As the first input, it also receives the guided complex feature map output from the first upsampled block. As the second input; the third complex dynamic residual block Receive the second complex dynamic residual block Output As the first input, it also receives the guided complex feature map output from the second upsampled block. As the second input; the fourth complex dynamic residual block Receive the third complex dynamic residual block Output As the first input, it also receives the guided complex feature map output from the third upsampled block. As the second input, the fourth complex dynamic residual block Final output focused complex amplitude .

[0064] Each complex dynamic residual block is processed according to the following steps:

[0065] The first input of each complex dynamic residual block is processed by multiple cascaded complex-valued convolutional blocks and then concatenated with the corresponding second input along the channel to obtain the first residual feature map. The first residual feature map is processed by multiple cascaded complex-valued convolutional blocks and a complex-valued convolutional layer in the first branch to obtain the complex residual. The first residual feature map is further processed by multiple cascaded complex-valued convolutional blocks and a complex-valued convolutional layer in the second branch to obtain the dynamic kernel. The dynamic kernel is multiplied by the first input to obtain the dynamic residual. The dynamic residual, the complex residual, and the first input are added element-wise along the channel dimension to output the complex amplitude, which is used as the output of the current complex dynamic residual block. Each complex-valued convolutional block is composed of a complex-valued convolutional layer and an activation layer connected in series.

[0066] like Figure 2 As shown in (d) in this embodiment, each complex dynamic residual block First input After processing by three concatenated complex-valued convolutional blocks, it is compared with the corresponding second input. After concatenation along the channel, the first residual feature map is obtained. The first residual feature map is then processed by three complex-valued convolutional blocks and one complex-valued convolutional layer connected in series in the first branch to obtain the complex residual. The first residual feature map is further processed by two complex-valued convolutional blocks and one complex-valued convolutional layer connected in series through the second branch to obtain the dynamic kernel. Dynamic kernel With the first input The dynamic residual is obtained after performing a dot product. Dynamic residuals Complex residuals and the first input The complex amplitude is output after element-wise addition along the channel dimension. As the current complex dynamic residual block The output of .

[0067] Specifically, such as Figure 2 As shown in (d) in the diagram, each complex dynamic residual block The overall operation can be represented as:

[0068]

[0069] Dynamic residual High-frequency features lost during defocusing degradation were extracted, while complex residuals... This focuses on representing low-frequency features of the basic content obscured by defocus degradation. Both of these, along with the first input... Together, they recover the potential focused complex amplitude.

[0070] The second branch first transforms the input features through two complex convolutional blocks and one complex convolutional layer, generating a dynamic kernel that matches the dimension of the input space. (like Figure 2 (as shown in block d in the diagram).

[0071] Dynamic kernel With the first input The dot product is essentially complex dynamic filtering (CDF), such as... Figure 2 As shown in (e) in the diagram. Complex dynamic filtering adaptively generates spatially varying pixel-by-pixel kernels based on the input content, achieving position-dependent defocus compensation. For pixel positions... From dynamic kernel Extract from the pixel position The complex local kernel centered on (like Figure 2 As shown in (e), Finally, the local kernel of this complex number... Compared with the input feature map by position Centered Perform local dot product operations (weighted aggregation of positional inner products in the complex field) on the neighborhood to generate positions. The dynamic residual response; after traversing all spatial locations, the complete dynamic residual is obtained. The formula for complex dynamic filtering is: .

[0072] The complex dynamic residual block of the present invention assigns an independent complex local kernel to each pixel position through a dynamic kernel, thereby enhancing the ability to characterize and compensate for the defocus effect of spatial variation dependent on distance.

[0073] like Figure 2 (b) and Figure 1 As shown in (c), the block probability discriminator consists of multiple feature extraction convolutional blocks (ConvBlock) and an output convolutional layer (SN-Conv) concatenated. Each feature extraction convolutional block consists of a concatenated embedding spectral normalization (SN) convolutional layer and an activation layer. The number of output channels of each feature extraction convolutional block increases with each layer, and the output convolutional layer contains only the embedding spectral normalization convolutional layer. The first feature extraction convolutional block of the block probability discriminator receives the output of the full complex value generator. The output of the convolutional layer is the patch score, which is used to quantify the difference between the output of the full complex value generator and the distribution of the ground truth (GT) in local regions.

[0074] Specifically, the number of output channels in each feature extraction convolutional block increases progressively with each layer; each element in the spatial block probability matrix corresponds to a local region of the input image, and its value represents the probability that the local region is classified as a real sample. When the values ​​of all elements in the spatial block probability matrix approach 0.5, it indicates that the block probability discriminator cannot distinguish between the output of the full complex value generator and the real value label, that is, the reconstructed complex amplitude of the full complex value generator is highly consistent with the focused complex amplitude.

[0075] like Figure 2 As shown in (b) in this embodiment, the block probability discriminator is composed of four feature extraction convolutional blocks (ConvBlock) and one output convolutional layer (SN-Conv) connected in series, and finally outputs the spatial block probability matrix Patchscore.

[0076] S6. Input the input dataset into the full complex value generator. Optimize the multi-objective joint loss function through alternating adversarial training between the full complex value generator and the block probability discriminator until the block probability discriminator can no longer distinguish between the output of the full complex value generator and the true value label. Obtain the trained full complex value generator and use it as a depth extension model.

[0077] Specifically, the alternating adversarial training includes fixing the parameters of the full complex value generator, using the true value label and the output of the full complex value generator as inputs to the block probability discriminator, calculating the discriminator loss and updating the parameters of the block probability discriminator; fixing the parameters of the block probability discriminator, using the output of the full complex value generator to calculate the generator loss and update the parameters of the full complex value generator.

[0078] The multi-objective joint loss function includes generator loss and discriminator loss; generator loss includes pixel-level mean absolute error loss, multi-scale structural similarity loss, feature matching loss, and generator adversarial loss; discriminator loss is used to distinguish between the ground truth label and the output of the full complex value generator.

[0079] The generator loss is set according to the following formula:

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] The discriminator loss is set according to the following formula:

[0086] ;

[0087] in, For generator loss; , , and These are pixel-level mean absolute error loss, multi-scale structural similarity loss, feature matching loss, and generator adversarial loss, respectively. , , , and All are weighting coefficients; and All are indexes; This represents the total number of pixels in the feature map. and These are the imaginary and real parts of the focused complex amplitude, respectively; and These are the focused complex amplitudes corresponding to the first The imaginary and real parts of each pixel; and These represent the imaginary and real parts of the reconstructed complex amplitude, respectively. and These are the reconstructed complex amplitudes corresponding to the first... The imaginary and real parts of each pixel; It is a multi-scale structural similarity index; For the first The total number of elements in the output feature map of each feature extraction convolutional block; This represents the number of feature extraction convolutional blocks in the block probability discriminator; It is an L1 norm; and These are the outputs of the true value label and the reconstructed complex amplitude after being processed by the block probability discriminator, respectively. For the true value label, after the first Feature maps output by each feature extraction convolutional block; For discriminator loss; To reconstruct the complex amplitude via the first Feature maps output by each feature extraction convolutional block; To calculate the expectation of the reconstructed complex amplitude output by the full complex value generator over the generated sample distribution; To calculate the expectation of the focused complex amplitude over the real data distribution, in this embodiment, , , and The values ​​are 1, 0.3, 0.1 and 0.3 respectively.

[0088] S7. Obtain the defocused hologram to be reconstructed, then perform numerical reconstruction preprocessing, extract and construct complex tensors, and divide them into image blocks. Input the images into the depth extension model for processing and output the corresponding predicted focus complex amplitude.

[0089] Specifically, the defocus hologram to be reconstructed is obtained at any axial position within the positive and negative axial depth range of the reference focusing position of the sample to be tested. The defocus hologram to be reconstructed is preprocessed with numerical reconstruction to obtain the defocus complex amplitude to be reconstructed. The real and imaginary parts of the defocus complex amplitude to be reconstructed are extracted to form a complex tensor, which is input into the depth-of-field extension model for processing and outputting the corresponding predicted focusing complex amplitude.

[0090] Specifically, the depth-of-field extension model is the fully complex value generator trained in step six. During the inference phase, a block probability discriminator is not required. The defocused hologram to be reconstructed can be a defocused hologram acquired at any axial defocus distance within the positive and negative axial depth range. The depth-of-field extension model can perform end-to-end focused complex amplitude reconstruction on it, thereby achieving a significant extension of the imaging depth of field.

[0091] To further verify the beneficial effects of the present invention, this embodiment conducted an experiment on a slice of female Ascaris cells at an angle. The reconstruction results obtained from the experiment are shown in the figure below. Figure 3 As shown (the axial span reaches approximately 300 μm, and the defocusing amount gradually increases from right to left). Among them, Figure 3 (a) in the image is a defocused hologram; Figure 3 (b) and Figure 3 In the diagram, (c) represents the real and imaginary parts of the complex amplitude input to the network, respectively. Figure 3 In this context, (d) represents the true value label. Figure 3 In the diagram, (e) represents the reconstructed complex amplitude output by the network reconstruction. Figure 3 (f) in the figure represents the quantitative phase curves of three different defocus regions.

[0092] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for extending the depth of field in digital holographic microscopy based on spatially adaptive complex generative adversarial networks, characterized in that, Includes the following steps: S1. Acquire focused holograms of the sample under test at the reference focusing position and defocused holograms at different axial defocusing distances using an off-axis digital holographic microscopy system to form a hologram sequence; S2. Perform numerical reconstruction preprocessing on each defocused hologram in the hologram sequence to obtain the corresponding defocused complex amplitude; S3. Numerical reconstruction preprocessing is performed on the focused holograms in the hologram sequence. The optimal reconstruction distance is accurately calibrated by the automatic focusing algorithm. The wrapped phase is reconstructed and unfolded at the optimal reconstruction distance by the angular spectrum method. The residual aberration is then corrected by the alternating direction method to obtain the focused complex amplitude. S4. Divide each defocused complex amplitude into image blocks to construct the input dataset; The focused complex amplitude is divided into image blocks in the same way and used as ground value labels; S5. Construct a complex generative adversarial network model, which includes a full complex value generator and a block probability discriminator. S6. Input the input dataset into the full complex value generator. Optimize the multi-objective joint loss function through alternating adversarial training between the full complex value generator and the block probability discriminator until the block probability discriminator can no longer distinguish between the output of the full complex value generator and the true value label. Obtain the trained full complex value generator and use it as a depth extension model. S7. Obtain the defocused hologram to be reconstructed, then perform numerical reconstruction preprocessing, extract and construct complex tensors, and divide them into image blocks. Input the images into the depth extension model for processing and output the corresponding predicted focus complex amplitude.

2. The digital holographic microscopy depth-of-field extension method according to claim 1, characterized in that: The acquisition of the hologram sequence is carried out in equal steps within the depth range of the positive and negative axes, with a preset reference focusing position as the center. The hologram sequence includes a focused hologram acquired at the reference focusing position and defocused holograms at multiple defocus distances.

3. The digital holographic microscopy depth-of-field extension method according to claim 1, characterized in that: The numerical reconstruction preprocessing in steps S1 and S2 involves performing a fast Fourier transform on the hologram, extracting the +1 level spectrum through spatial filtering and shifting it to the center of the frequency domain, separating and compensating for background phase aberrations using principal component analysis, and obtaining the aberration-precorrected complex amplitude through inverse Fourier transform.

4. The digital holographic microscopy depth-of-field extension method according to claim 1, characterized in that, Step S4 specifically involves: The real and imaginary parts of each defocused complex amplitude are extracted separately, and the real and imaginary parts are used as two channels to form a complex tensor with two independent channels for each defocused complex amplitude. Each complex tensor is then spatially divided into non-overlapping image blocks according to a preset fixed size. The input dataset is constructed from all the image blocks obtained by the segmentation. The real and imaginary parts of the focused complex amplitude are extracted to form a complex tensor, which is then divided into image blocks of the same size as the real value labels.

5. The digital holographic microscopy depth-of-field extension method according to claim 1, characterized in that: The full complex value generator includes a Unet feature extraction backbone and a complex dynamic residual reconstruction branch; The Unet feature extraction backbone adopts an encoder-decoder architecture with a bottleneck layer between the encoder and decoder. The encoder consists of multiple downsampling blocks connected in series, and the decoder consists of multiple upsampling blocks connected in series. Image patches in the input dataset serve as the input to the first downsampling block in the encoder, and the input to each subsequent downsampling block comes from the output of the previous downsampling block. The output of the last downsampling block is processed by the bottleneck layer and then serves as the input to the first upsampling block in the decoder. The input to each subsequent upsampling block comes from the feature map obtained by concatenating the output of the previous upsampling block and the output of the downsampling block at the corresponding scale. The bottleneck layer and each upsampling block output guided complex feature maps at the corresponding scale. The complex dynamic residual reconstruction branch includes multiple complex dynamic residual blocks. Image blocks in the input dataset are downsampled multiple times and used as the first input of the first complex dynamic residual block. The first complex dynamic residual block also receives a guiding complex feature map output from the bottleneck layer as the second input. The output of the first complex dynamic residual block is sent to the next complex dynamic residual block. Each subsequent complex dynamic residual block receives the output of the previous complex dynamic residual block as the first input and also receives a guiding complex feature map output from the upsampled block at the corresponding resolution scale as the second input. The output of each subsequent complex dynamic residual block is sent to the next level complex dynamic residual block. The last complex dynamic residual block outputs the reconstructed complex amplitude.

6. The digital holographic microscopy depth-of-field extension method according to claim 5, characterized in that, Each of the aforementioned complex dynamic residual blocks is processed according to the following steps: The first input of each complex dynamic residual block is processed by multiple cascaded complex-valued convolutional blocks and then concatenated with the corresponding second input along the channel to obtain a first residual feature map. The first residual feature map is processed by multiple cascaded complex-valued convolutional blocks and a complex-valued convolutional layer in the first branch to obtain a complex residual. The first residual feature map is further processed by multiple cascaded complex-valued convolutional blocks and a complex-valued convolutional layer in the second branch to obtain a dynamic kernel. The dynamic kernel is multiplied by the first input to obtain the dynamic residual. The dynamic residual, the complex residual, and the first input are added element-wise along the channel dimension to output a complex amplitude, which is used as the output of the current complex dynamic residual block. The complex-valued convolutional block is composed of complex-valued convolutional layers and activation layers connected in series.

7. The digital holographic microscopy depth-of-field extension method according to claim 1, characterized in that: The block probability discriminator consists of multiple feature extraction convolutional blocks connected in series with an output convolutional layer; each feature extraction convolutional block consists of an embedding spectrum normalized convolutional layer and an activation layer connected in series; the output convolutional layer contains only an embedding spectrum normalized convolutional layer; the first feature extraction convolutional block of the block probability discriminator receives the output of the full complex value generator and the true value label respectively, and the output convolutional layer finally outputs the spatial block probability matrix.

8. The digital holographic microscopy depth-of-field extension method according to claim 1, characterized in that: The multi-objective joint loss function includes generator loss and discriminator loss; the generator loss includes pixel-level average absolute error loss, multi-scale structural similarity loss, feature matching loss and generator adversarial loss; the discriminator loss is used to distinguish between the ground truth label and the output of the full complex value generator.

9. The digital holographic microscopy depth-of-field extension method according to claim 8, characterized in that: The generator loss is set according to the following formula: ; ; ; ; ; The discriminator loss is set according to the following formula: ; in, For generator loss; , , and These are pixel-level mean absolute error loss, multi-scale structural similarity loss, feature matching loss, and generator adversarial loss, respectively. , , , and All are weighting coefficients; and All are indexes; This represents the total number of pixels in the feature map. and These are the imaginary and real parts of the focused complex amplitude, respectively; and These are the focused complex amplitudes corresponding to the first The imaginary and real parts of each pixel; and These represent the imaginary and real parts of the reconstructed complex amplitude, respectively. and These are the reconstructed complex amplitudes corresponding to the first... The imaginary and real parts of each pixel; It is a multi-scale structural similarity index; For the first The total number of elements in the output feature map of each feature extraction convolutional block; This represents the number of feature extraction convolutional blocks in the block probability discriminator; It is an L1 norm; and These are the outputs of the true value label and the reconstructed complex amplitude after being processed by the block probability discriminator, respectively. For the true value label, after the first Feature maps output by each feature extraction convolutional block; For discriminator loss; To reconstruct the complex amplitude via the first Feature maps output by each feature extraction convolutional block; To calculate the expectation of the reconstructed complex amplitude output by the full complex value generator over the generated sample distribution; To calculate the expectation of the focused complex amplitude on the real data distribution.

10. The digital holographic microscopy depth-of-field extension method according to claim 1, characterized in that, Step S7 specifically involves: Obtain the defocus hologram to be reconstructed at any axial position within the positive and negative axial depth range of the reference focusing position of the sample to be tested. Perform numerical reconstruction preprocessing on the defocus hologram to be reconstructed to obtain the defocus complex amplitude to be reconstructed. The real and imaginary parts of the defocus complex amplitude to be reconstructed are extracted to form complex tensors, which are then divided into image blocks and input into the depth-of-field extension model for processing to output the corresponding predicted focus complex amplitude.