Phase difference wavefront-image estimation method based on depth image prior

Abstract

The invention discloses a phase difference wavefront-image estimation method based on depth image prior. The method comprises the following steps: firstly, constructing a forward physical model of an incoherent imaging system, and generating an observation image with out-of-focus aberration by introducing phase difference; and then using a depth image prior network to jointly optimize image domain loss and wavefront domain loss under the condition of no external training data by taking the consistency of an observation image and a model generation image as a constraint so as to simultaneously obtain a high-quality recovered image and accurate wavefront estimation. The method provided by the invention can realize accurate wavefront estimation and high-quality image recovery under the condition of large aberration under the condition of only needing a small number of acquired images, and has the advantages of strong anti-noise capability, no need of an additional wavefront sensor and low system complexity. The method can be widely applied to fluorescence microscopic imaging and other high-resolution microscopic imaging technologies, and the imaging resolution and the image quality are improved.

Description

Technical Field

[0001] This invention belongs to the field of fluorescence microscopy imaging technology, specifically relating to a phase difference wavefront-image estimation method based on depth image prior. Background Technology

[0002] In modern fluorescence microscopy systems, optical aberrations are a common and unavoidable problem. Fluorescence microscopes use objectives, lenses, and other optical elements to focus the fluorescence signal of a sample onto a detector, thereby achieving high-resolution imaging. However, during light propagation, imperfections in any optical element, system alignment errors, or inhomogeneities in the refractive index within the sample can cause wavefront distortion. This distortion directly leads to broadening of the point spread function (PSF) and attenuation of the optical transfer function (OTF), resulting in image blurring and reduced resolution.

[0003] Traditional aberration correction methods include hardware compensation based on adaptive optics and indirect estimation based on algorithms. Adaptive optics methods rely on wavefront detectors and correction devices, and can eliminate aberrations to a certain extent, but they suffer from problems such as hardware complexity, high cost, and limited imaging speed. Algorithm-based wavefront estimation methods often do not require additional hardware, but they are prone to failure in noisy environments and have limitations in convergence and accuracy for scenes with large aberrations.

[0004] Therefore, existing methods still have significant shortcomings in terms of wavefront estimation accuracy, image restoration quality, robustness, and computational efficiency. A new method is needed that can achieve high-precision aberration correction and high-quality image restoration with reduced hardware dependence.

[0005] Technical problems with existing technologies:

[0006] 1) High hardware cost and high system complexity: Traditional adaptive optics methods rely on dedicated hardware devices such as wavefront detectors and deformable mirrors, resulting in high system cost and complex hardware integration, which limits the widespread application of the technology.

[0007] 2) Strong dependence on training data: Existing deep learning-based aberration correction methods require a large number of labeled training datasets, which are costly to collect and label, and are difficult to cover all application scenarios, thus limiting their generalization ability.

[0008] 3) Insufficient wavefront estimation accuracy: Traditional blind estimation methods have multiple solutions and limited estimation accuracy of wavefront distortion, especially in scenarios with large aberrations, where convergence is poor and estimation errors are large, affecting the aberration correction effect.

[0009] 4) Insufficient robustness and stability: Algorithm-based wavefront estimation methods (such as the phase difference method) are prone to failure in noisy environments, have poor adaptability to different types and intensities of aberrations, and lack algorithm stability.

[0010] 5) Lack of physical constraints: Existing deep learning methods lack physical model constraints, and the estimation results may violate the laws of optical physics, resulting in insufficient algorithm reliability and interpretability.

[0011] 6) Insufficient computational efficiency and real-time performance: Existing methods have limitations in computational efficiency and convergence speed, making it difficult to meet the requirements of real-time or near-real-time imaging processing, which affects the temporal resolution and practicality of imaging. Summary of the Invention

[0012] In view of the above, this invention provides a method for wavefront-image estimation based on depth image priors, which can achieve high-precision wavefront-image estimation without increasing the cost and complexity of instruments and equipment or significantly increasing processing time. This invention is achieved through the following technical solutions:

[0013] This invention discloses a phase difference wavefront-image estimation method based on depth image priors, comprising the following steps:

[0014] 1) Acquire at least two target microscopic image sequences and additional phases;

[0015] 2) A depth image prior network is used to generate the initial phase by calculating the phase ordinal of the encoded data;

[0016] 3) By using the physical model of the coherent imaging system and the target microscopic image sequence, initial phase and additional phase, the initial target image and the reconstructed microscopic image sequence are obtained, and image estimation is achieved;

[0017] 4) Calculate the loss function using the initial phase, the target microscopic image sequence, and the reconstructed microscopic image sequence, iteratively update the network parameters, update the initial phase, and achieve wavefront estimation.

[0018] As a further improvement, the target microscopic image sequence described in this invention is a group of images containing aberrations, obtained by adding an additional phase to the first image in the sequence.

[0019] As a further improvement, the depth image prior network described in this invention is a convolutional neural network, with the input being a calculated and encoded phase ordinal and the output being a two-dimensional initial phase, enabling training and inference without external training data.

[0020] As a further improvement, the physical model of the coherent imaging system described in this invention includes two steps: deconvolution and image reconstruction. The input to deconvolution is the target microscopic image sequence, the initial phase, and the additional phase, and the output is the initial target image. The input to image reconstruction is the initial target image, the initial phase, and the additional phase, and the output is the reconstructed microscopic image sequence.

[0021] As a further improvement, the loss function described in this invention includes image domain loss and wavefront domain loss. The image domain loss is used to constrain the consistency between the estimated image and the real degraded image, and the wavefront domain loss is used to constrain the rationality of the wavefront generated by the physical model.

[0022] The beneficial effects of this invention are as follows:

[0023] 1) Significantly reduces hardware costs and system complexity: This invention does not require expensive hardware devices such as wavefront sensors and deformable mirrors. High-precision aberration correction can be achieved through software algorithms or simple phase modulators. The algorithm can be directly integrated into existing microscopic imaging systems as a software module without hardware modification. It is easy to deploy, has a short upgrade cycle, and low implementation risk, which greatly reduces the technical application threshold and cost.

[0024] 2) Completely eliminates dependence on training data: The self-supervised learning framework is built using an implicit neural network architecture, which can complete training and inference using only the input degraded image sequence itself, without any external labeled datasets; the network structure itself serves as a prior regularization constraint for deep images, avoiding overfitting problems, significantly improving the robustness and generalization ability of the algorithm, and adapting to different application scenarios.

[0025] 3) Significantly improves wavefront estimation accuracy: By introducing a controllable additional phase, an image sequence with known phase differences is constructed, providing additional constraint information for wavefront estimation, which significantly enhances the uniqueness and accuracy of wavefront estimation and effectively solves the multi-solution problem of traditional blind estimation methods; it can still maintain good convergence and estimation accuracy in large aberration scenarios, breaking through the accuracy bottleneck of existing methods.

[0026] 4) Enhanced robustness and stability: A dual loss function mechanism is adopted, with image domain loss constraining the consistency between the reconstructed image and the observed image, and wavefront domain loss constraining the physical rationality of the wavefront. The two complement each other to form a stable constraint, enabling the algorithm to obtain reliable results in noisy environments and under various aberration types and intensities, which significantly improves the robustness and applicability of the method.

[0027] 5) Physical model constraints ensure reliability: The accurate physical model of the coherent imaging system (including diffraction propagation, deconvolution and image reconstruction processes) is embedded into the neural network optimization process to ensure that the estimation results strictly conform to the laws of optical physics; the differentiable implementation of the physical model enables the entire framework to be trained end-to-end, realizing the organic combination of data-driven optimization and physical constraints, and improving the interpretability of the algorithm and the reliability of the results.

[0028] 6) Improved computational efficiency and real-time performance: Physical model constraints significantly accelerate algorithm convergence and reduce the number of iterations; optimized network architecture and computational process improve the efficiency of a single iteration, enabling the algorithm to meet the requirements of real-time or near-real-time imaging processing, and improving the temporal resolution and practicality of imaging. Attached Figure Description

[0029] Figure 1 This is a schematic flowchart of the method of the present invention;

[0030] Figure 2 A flowchart illustrating the direct and indirect adaptive optics measurement and calibration process;

[0031] In the figure, (a) traditional adaptive optics achieves direct measurement of aberrations through wavefront sensors and direct correction through deformable mirrors or spatial light modulators; (b) computational adaptive optics can achieve indirect measurement of aberrations through algorithms and directly recover the image.

[0032] Figure 3 Comparison of results from fluorescent microbead imaging experiments;

[0033] In the figure, (a) is the original image with aberrations, (b) is the estimated image of the proposed method, (c) is the true reference image of the sample, (d) is the true reference aberration wavefront of the original image, (e) is the estimated aberration wavefront of the proposed method, and (f), (d), and (e) are the wavefront differences of the proposed method (in micrometers). Detailed Implementation

[0034] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Figure 1 This is a flowchart illustrating the method of the present invention; the present invention discloses a phase difference wavefront-image estimation method based on depth image priors, comprising the following steps:

[0036] 1) Obtain at least two target microscopic image sequences and additional phases; the target microscopic image sequence is a group of images containing aberrations, obtained by adding additional phases to the first image in the sequence.

[0037] 2) A deep image prior network is used to generate the initial phase by calculating the encoded phase ordinal. The deep image prior network is a convolutional neural network. The input is the calculated and encoded phase ordinal, and the output is the two-dimensional initial phase. It can perform training and inference without external training data.

[0038] 3) Using the physical model of the coherent imaging system, the initial target image and the reconstructed microscopic image sequence are obtained by utilizing the target microscopic image sequence, the initial phase and the additional phase, thus realizing image estimation. The physical model of the coherent imaging system includes two steps: deconvolution and image reconstruction. The input of deconvolution is the target microscopic image sequence, the initial phase and the additional phase, and the output is the initial target image. The input of image reconstruction is the initial target image, the initial phase and the additional phase, and the output is the reconstructed microscopic image sequence.

[0039] 4) Calculate the loss function using the initial phase, the target microscopic image sequence, and the reconstructed microscopic image sequence. Iteratively update the network parameters and the initial phase to achieve wavefront estimation. The loss function includes image domain loss and wavefront domain loss. Image domain loss is used to constrain the consistency between the estimated image and the real degraded image, while wavefront domain loss is used to constrain the rationality of the wavefront generated by the physical model.

[0040] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0041] I. Technical Background and Data Acquisition Process

[0042] In modern fluorescence microscopy, optical aberrations are almost unavoidable. While the aberration effects may be subtle in shallow imaging, their cumulative effect can significantly degrade the signal-to-noise ratio and resolution in deep samples or under super-resolution imaging conditions. For example, in deep tissue imaging, cellular structures gradually become blurred and their brightness decreases, severely impacting quantitative analysis. Adaptive optics is an important approach to addressing this problem. As an advanced optical technology for real-time dynamic correction of wavefront distortion, adaptive optics works by using hardware or algorithms to replace wavefront measurement and correction elements to achieve real-time wavefront monitoring and aberration compensation. Generally, there are two methods: direct and indirect. Figure 2 As shown. Figure 2The diagram illustrates the measurement and correction processes for direct and indirect adaptive optics. (a) Traditional adaptive optics directly measures aberrations using wavefront sensors and directly corrects them using deformable mirrors or spatial light modulators. (b) Computational adaptive optics can indirectly measure aberrations using algorithms and directly reconstruct the image. However, direct wavefront measurement in adaptive optics typically relies on Shaker-Hartmann or pyramid sensors, which require complex optical paths and bright "guide stars," increasing hardware costs and potentially leading to additional light loss and increased sample illumination dose. The limitations of this method are particularly pronounced in live-cell or fast-motion imaging scenarios.

[0043] To reduce hardware dependence, researchers proposed the Phase Diversity (PD) method in indirect wavefront detection, as described in the literature [Johnson C, Guo M, et al. Phase-diversity-based wavefront sensing for fluorescence microscopy[J]. Optica, 2024, 11: 806-820.]. By acquiring multiple images with different defocus amounts, wavefront information is retrieved using the differences between them. The PD method avoids external sensors, but it still faces challenges in practical applications: when the imaging signal-to-noise ratio is low, the traditional numerical optimization process is prone to getting trapped in local extrema, leading to unstable estimations; for cases with large aberrations, the high computational complexity and slow convergence speed also limit its application in real-time imaging. In recent years, the rise of deep learning methods has brought new ideas. The Deep Image Prior (DIP) method, as described in the literature [Ulyanov D, Vedaldi A, Lempitsky V. Deep Image Prior [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018: 9446-9454.], demonstrates that convolutional neural networks inherently possess prior knowledge of images, enabling them to complete image restoration tasks without requiring a large amount of training data. Based on this, this invention innovatively proposes combining DIP with phase difference physical modeling, utilizing limited degraded image data to simultaneously achieve wavefront estimation and image restoration. This not only avoids the introduction of additional hardware but also improves robustness in noisy environments.

[0044] The data acquisition process of this invention includes five key steps. First, system calibration is performed using standard samples such as fluorescent microbeads or a resolution plate to determine system parameters such as pixel size, operating wavelength, and numerical aperture, and to measure and correct camera dark current and inhomogeneity. Next, fluorescently labeled samples are prepared, placed on the microscope stage, and adjusted to the region of interest. Then, the image is acquired by manually or automatically focusing to the target depth, setting an appropriate exposure time to fully utilize the image's dynamic range while avoiding saturation. Subsequently, defocused images are acquired, which can be achieved using different methods: either by moving the objective or sample along the optical axis by a distance Δz using a motorized translation stage, or by applying a defocus phase using a spatial light modulator or deformable mirror; multiple images can be selectively acquired to provide more constraints. Finally, data preprocessing is performed, including dark field correction by subtracting dark current from the original image, and normalizing the image intensity to the range of 0 to 1, providing high-quality input data for subsequent phase retrieval algorithms. The complete flowchart of this method is shown below. Figure 1 As shown. Figure 1 This is a flowchart illustrating the method of the present invention; it includes steps such as image acquisition, depth image prior network, physical model of coherent imaging system, loss function calculation and output results.

[0045] II. Depth Image Prior Optimization Framework

[0046] 2.1 Principle of Phase Difference Measurement

[0047] The core idea of ​​the phase difference (PD) method is to introduce a known additional phase into the imaging optical path. Acquire multiple images with different defocus amounts. Let K images be acquired, and the total phase corresponding to the k-th image be:

[0048]

[0049] The corresponding observation images are:

[0050]

[0051] in The additional phase is the additional phase of the k-th image. The introduction of the additional phase provides additional constraint information for wavefront estimation, significantly enhancing the uniqueness and accuracy of wavefront estimation and avoiding the multiple solution problem of traditional blind estimation methods. The additional phase can be implemented in the following three ways: (1) moving the sample or objective lens to change the focal plane position (mechanical focusing); (2) inserting a variable focus lens or zoom element in the optical path; (3) applying programmable phase modulation using a spatial light modulator or deformable mirror.

[0052] 2.2 Deep Image Prior Network Architecture

[0053] This invention introduces a depth image prior network. As an image generator, θ represents the network parameters, and H×W represents the Zernic phase ordinal and the size of the phase to be predicted. The network input is the Zernic phase ordinal Zern(C), and the output is the estimated initial phase.

[0054]

[0055] The network architecture adopts a U-Net or encoder-decoder structure, specifically including:

[0056] (1) Encoder: Contains multiple downsampling modules, each consisting of a convolutional layer, a batch normalization layer, and an activation function. (2) Decoder: Contains multiple upsampling modules, each consisting of a transposed convolutional layer or a bilinear interpolation upsampling layer, a batch normalization layer, and an activation function. (3) Skip connections: Skip connections are established between corresponding layers of the encoder and decoder to fuse shallow features with deep features, preserving detailed information.

[0057] (4) Output layer: The last convolutional layer (1×1 kernel) maps the features to the output image space and uses an activation function to limit the output range.

[0058] III. Establishment of Physical Model

[0059] 3.1 Forward Propagation Model of Coherent Imaging System

[0060] Let I be the true intensity distribution on the plane of the object. obj (r), where r = (x, y) are spatial coordinates. In coherent or partially coherent imaging systems, the imaging process can be represented as a convolution model:

[0061]

[0062] Where * denotes a two-dimensional convolution operation. For phase aberration The point spread function is affected, where n(r) is the noise term (including photon shot noise, readout noise, etc.).

[0063] In the frequency domain, the above convolution relationship can be expressed as a simple product. This can be obtained through a two-dimensional Fourier transform:

[0064]

[0065] Where f = (f_x, f_y) are spatial frequency coordinates. The symbol represents the optical transfer function (OTF), and the tilde indicates the Fourier transform. The relationship between OTF and PSF is as follows:

[0066] For an ideal diffraction-limited imaging system At this point, the OTF reaches its maximum value and is limited only by the numerical aperture of the optical system. When aberrations introduce additional phase modulation, they cause a decrease in OTF amplitude and phase distortion, which manifests as image blurring, reduced contrast, and structural distortion.

[0067] 3.2 Calculation of the point spread function

[0068] For a given wavefront phase The point spread function can be calculated using diffraction integrals. Within the framework of scalar diffraction theory, the complex amplitude at the exit pupil is:

[0069]

[0070] Where P(r) is the pupil function, describing the aperture shape of the optical system (usually circular). After Fourier transform, the complex amplitude distribution of the image plane is obtained, and its modulus square is the PSF:

[0071]

[0072] This calculation process can be efficiently implemented using the Fast Fourier Transform (FFT), with a computational complexity of O(N). 2 logN), where N is the image size.

[0073] IV. Loss Function Calculation

[0074] Unlike traditional deep learning methods, the network of this invention does not require pre-training data, but is trained directly by optimizing the following objective function:

[0075]

[0076] This includes three losses:

[0077] (1) Image domain loss The consistency between the constrained network-generated image and the observed image after passing through the forward physics model:

[0078]

[0079] This loss is directly based on the imaging physics model, ensuring that the estimation results conform to the laws of optical propagation. In practical implementation, the convolution operation is efficiently computed in the frequency domain using FFT.

[0080] (2) Wavefront regularization loss Constraints on the physical plausibility and smoothness of wavefront estimation:

[0081]

[0082] The first term is a gradient penalty term, which encourages smooth wavefront phase; the second term is an amplitude penalty term, which prevents the wavefront coefficient from becoming too large.

[0083] (3) Network regularization loss Preventing network overfitting and observation noise:

[0084]

[0085] This is the standard L2 weight decay term. The weights λ of each loss term... wf f,λ reg It needs to be adjusted according to the specific application scenario.

[0086] The effectiveness of this invention is demonstrated through an experiment. In this experiment, wide-field fluorescence microscopy images of standard fluorescent microbeads with a diameter of 0.5 μm were processed, as shown below. Figure 3 As shown. Figure 3 The results of the fluorescent microbead imaging experiment are shown in the comparison diagram. (a) is the original image with aberrations, (b) is the estimated image using the proposed method, (c) is the true reference image of the sample, (d) is the true reference aberration wavefront of the original image, (e) is the estimated aberration wavefront using the proposed method, and (f), (d), and (e) are the wavefront differences (in micrometers) of the proposed method. Due to severe aberrations such as spherical aberration and coma in the system, the original image exhibits widened dot spread, blurred edges, and decreased contrast. Two images with different defocus amounts were acquired and processed using the phase difference wavefront-image estimation method based on depth image prior proposed in this invention. The results show that the edges of the fluorescent microbeads in the restored image are clear and sharp, the dot spread size is significantly reduced, and both brightness and contrast are improved. Simultaneously, the estimated wavefront is highly consistent with the true reference wavefront, and the difference between the two is controlled within the sub-micrometer range, demonstrating that this invention can achieve high-precision wavefront estimation and high-quality image restoration under conditions of limited observation images.

[0087] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A phase difference wavefront-image estimation method based on depth image priors, comprising the following steps: 1) Acquire at least two target microscopic image sequences and additional phases; 2) A depth image prior network is used to generate the initial phase by calculating the phase ordinal of the encoded data; 3) By using the physical model of the coherent imaging system and the target microscopic image sequence, initial phase and additional phase, the initial target image and the reconstructed microscopic image sequence are obtained, and image estimation is achieved; 4) Calculate the loss function using the initial phase, the target microscopic image sequence, and the reconstructed microscopic image sequence, iteratively update the network parameters, update the initial phase, and achieve wavefront estimation.

2. The phase difference wavefront-image estimation method according to claim 1, characterized in that: The target microscopic image sequence is a group of images containing aberrations, obtained by adding an additional phase to the first image in the sequence.

3. The phase difference wavefront-image estimation method according to claim 1, characterized in that: The depth image prior network is a convolutional neural network. Its input is a calculated and encoded phase ordinal number, and its output is a two-dimensional initial phase. It can perform training and inference without external training data.

4. The phase difference wavefront-image estimation method according to claim 1, 2, or 3, characterized in that: The physical model of the coherent imaging system includes two steps: deconvolution and image reconstruction. The input to deconvolution is the target microscopic image sequence, the initial phase, and the additional phase, and the output is the initial target image. The input to image reconstruction is the initial target image, the initial phase, and the additional phase, and the output is the reconstructed microscopic image sequence.

5. The phase difference wavefront-image estimation method according to claim 4, characterized in that: The loss function includes image domain loss and wavefront domain loss. The image domain loss is used to constrain the consistency between the estimated image and the real degraded image, and the wavefront domain loss is used to constrain the rationality of the wavefront generated by the physical model.

Images