Fluorescent microscopic image restoration method based on aberration and noise hybrid network

Abstract

The invention discloses a fluorescence microscopic image recovery method based on an aberration and noise hybrid network, which comprises the following steps: synthesizing training data through an aberration adding module based on a Zernike aberration mode and a noise adding module based on variance stabilization transformation and Gaussian noise estimation, and training a denoising-aberration-removing hybrid network model to recover deep data. The invention aims to solve challenges caused by optical aberration and noise through data synthesis and deep learning at the same time, so that the quality of fluorescence microscopic imaging is improved; therefore, aberration correction and noise elimination can be realized under the condition that additional hardware conditions are not needed, so that the application range is further expanded, and high-quality imaging can be realized in a wider range of settings.

Description

Technical Field

[0001] The invention belongs to the technical field of fluorescence microscopic imaging, and in particular relates to a fluorescence microscopic image restoration method based on an aberration and noise hybrid network. Background Art

[0002] The imaging quality of fluorescence microscopy systems is often affected by optical aberrations. Aberration-induced image degradation becomes more pronounced with increasing imaging depth, complicating the visualization of structures within thick biological specimens.

[0003] Adaptive optics systems restore image quality by measuring the distorted wavefront and applying a corrective wavefront to mitigate aberrations. However, adaptive optics systems require special and expensive components, such as deformable mirrors and wavefront sensors, which make them difficult to integrate into the system and require careful maintenance. In addition, the wavefront measurement process slows down imaging speed and introduces more illumination dose, limiting the practicality of adaptive optics in dynamic biological research.

[0004] In 2023, Hu Qi et al. proposed a general method MLAO in the literature [Hu Q, Hailstone M, Wang J. Universal adaptive optics for microscopes controlled by embedded neural networks [J]. Light, Science and Applications, 2023, 12(1): 270]. This method divides a series of images with specific aberrations by the image to be corrected through Fourier transformation and then performs inverse Fourier transformation to obtain a pseudo PSF (point spread function). The neural network is used to extract the required aberration information from the pseudo PSF, and finally the aberration correction amount for the deformable mirror is estimated. Although this method can estimate the aberration without using a wavefront sensor, its correction still depends on wavefront correction devices such as deformable mirrors, and its implementation is still difficult.

[0005] To address these challenges, researchers proposed the DeAbe method [Guo M, Wu Y, Hobson C M. Deep learning-based aberration compensation technology improves the contrast and resolution of fluorescence microscopy [J]. Nature Communications 2025, 16(1): 313], which uses deep learning to improve image degradation caused by aberrations. Since the shallow part of the acquired image is less affected by aberrations than the deep part, DeAbe introduces synthetic aberrations in the shallow part of the image so that the synthesized image simulates the degradation of the deep part of the image, and then trains the neural network to reverse the effects of these aberrations; once the network is trained, it can directly act on the real acquired deep aberration data. Through this carefully designed computational strategy, DeAbe eliminates the need for complex wavefront measurements and significantly reduces aberration blur in deep images when acquiring images of thick samples; DeAbe provides significant enhancements in image quality and subsequent analysis, such as improved assessment of vascular directionality in mouse tissues and better membrane / nuclear segmentation in Caenorhabditis elegans embryos; DeAbe is also applicable to a variety of different microscopy techniques, including confocal, light sheet, multiphoton, and super-resolution microscopy.

[0006] In the field of fluorescence microscopy, researchers face the dilemma of a "performance pyramid" where they must make trade-offs between phototoxicity / photobleaching, observation time, and resolution. This trade-off is particularly critical when considering live cell imaging, as reducing the illumination light dose may be critical and the resulting decrease in signal-to-noise ratio may hinder image recovery. However, despite its many advantages, the DeAbe method still has certain limitations when dealing with low signal-to-noise conditions. Summary of the invention

[0007] In view of the above, the present invention provides a fluorescence microscopy image restoration method based on an aberration and noise hybrid network, which can correct the aberration blur of the fluorescence microscopy image without sacrificing time resolution and without increasing the irradiation dose.

[0008] A fluorescence microscopy image restoration method based on an aberration and noise hybrid network comprises the following steps: (1) For shallow clear image data in the fluorescence microscopy image, aberration is added to it to generate synthetic aberration image data; (2) adding noise to the synthetic aberration image data to generate synthetic degraded shallow image data; (3) combining shallow clear image data and its corresponding synthetic aberration image data and synthetic degraded shallow image data as training samples for model training; (4) Constructing a denoising-de-aberration hybrid network model for fluorescence microscopy image restoration, and training the model using training samples; the denoising-de-aberration hybrid network model includes: A denoising module is used to perform denoising on the input synthetic degraded shallow image data to obtain intermediate image data that still has aberration blur; A de-aberration module is used to remove the aberration blur in the intermediate image data, so as to obtain clear image data as the final output result; (5) By inputting the deep blurred image data in the fluorescence microscopy image into the trained model, the corresponding clear deep image can be restored.

[0009] Furthermore, the specific implementation of step (1) is expressed as follows:

[0010] in: i is the synthetic aberration image data, i 0 is the shallow clear image data without aberration, represents the Fourier transform, represents the inverse Fourier transform, is the aberration term.

[0011] Furthermore, the aberration term The expression is as follows:

[0012] in: h 0 is the ideal point spread function, is the incoherent point spread function, α To set a constant (usually set to 10 -6 ).

[0013] Furthermore, the incoherent point spread function When the signal is limited to the pupil aperture, its expression is as follows:

[0014] in: is the aberration phase term expressed in the pupil coordinate system, u represents the pupil coordinates, j is an imaginary unit and e is a natural constant.

[0015] Furthermore, the aberration phase term It reflects the size of the added aberration, which is represented by the Zernike basis function and its corresponding Zernike aberration coefficient (polynomial coefficient). By adjusting the Zernike aberration coefficient, different degrees of aberration can be added to the clear image.

[0016] Furthermore, the specific implementation method of step (2) is as follows: first, the Poisson noise in the fluorescence microscopy image is converted into Gaussian noise by a variance-stabilized transformation method, then an estimated value of the noise is calculated by a Gaussian noise estimation method based on weak texture region selection, and finally, an equal amount of noise is added to the synthetic aberration image data according to the estimated value, thereby generating synthetic degraded shallow image data.

[0017] Furthermore, the denoising module is composed of a convolutional layer D1, three cascaded residual groups, and a convolutional layer D2, which are connected in sequence, and the input of the convolutional layer D1 is added to the output of the convolutional layer D2 as the output result of the denoising module; the de-aberration module is composed of a convolutional layer D3, five cascaded residual groups, and a convolutional layer D4, which are connected in sequence, and the input of the convolutional layer D3 is added to the output of the convolutional layer D4 as the output result of the de-aberration module; the residual group is composed of three cascaded residual channel attention blocks and a convolutional layer D5, which are connected in sequence, and the input of the first residual channel attention block is added to the output of the convolutional layer D5 as the output result of the residual group.

[0018] Furthermore, the residual channel attention block is composed of a convolution layer D6, a Relu activation function layer R1, a convolution layer D7, a global average pooling layer, a convolution layer D8, a Relu activation function layer R2, a convolution layer D9, and a Sigmoid non-linear activation function layer connected in sequence, and the input of the global average pooling layer is multiplied by the output of the Sigmoid non-linear activation function layer and then added to the input of the convolution layer D6, which is the output result of the residual channel attention block.

[0019] Furthermore, in the step (4), the synthetic degraded shallow image data in the training sample is used as input and the following total loss function is used to train the model;

[0020]

[0021]

[0022] in: L Total is the total loss function, L Denoise is the denoising loss function, L DeAbe is the de-aberration loss function, d is the synthetic aberration image data in the training sample, x ' is the intermediate image data output by the denoising module in the model, y is the image data output by the de-aberration module in the model, d 0 is the shallow clear image data in the training sample, To set a constant (usually set to 10 -3 ).

[0023] The fluorescence microscopy image restoration method based on the hybrid aberration and noise network of the present invention aims to solve the challenges brought by optical aberration and noise through data synthesis and deep learning, thereby improving the quality of fluorescence microscopy imaging. In order to improve the performance of DeAbe in low signal-to-noise ratio conditions, the present invention proposes a denoising-deraberration hybrid network model, which integrates noise estimation into the existing DeAbe framework using a self-supervised deep learning method, and realizes the simultaneous restoration of noise and aberration.

[0024] The research results show that compared with existing methods, especially under low signal-to-noise ratio conditions, the present invention significantly improves the image restoration quality and can achieve aberration correction without the need for expensive adaptive optics hardware, which further enhances its scope of use and enables high-quality imaging in a wider range of settings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is a schematic diagram of the generation process of simulated degradation training data of the present invention.

[0026] Figure 2 It is a schematic diagram of the execution flow of the aberration module in the present invention.

[0027] Figure 3 Schematic diagram of the execution flow of the noise module in the present invention.

[0028] Figure 4 Schematic diagram of the overall structure and training process of the denoising-de-aberration hybrid network model of the present invention.

[0029] Figure 5 This is a comparison diagram of experimental results of performing microscopic image restoration on GFP membrane-labeled nematode embryos using the present invention. DETAILED DESCRIPTION

[0030] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] The technical solution of the present invention is mainly based on two key viewpoints: (1) The "shallow" or "near-side" images closer to the imaging end of the three-dimensional fluorescent volume are usually close to the diffraction limit and have no significant aberrations; (2) As the imaging depth increases, the signal will gradually weaken due to the distortion and absorption of light, which makes the aberration and noise effects of the deeper layers more significant than those of the shallow parts. Based on these two viewpoints, the present invention uses high-quality "shallow" images as reference labels to generate data that simulates the degradation of the deep part of the sample as input, trains the model, and thus achieves the de-aberration effect.

[0032] In order to achieve denoising and aberration suppression simultaneously, the overall work of this embodiment includes the generation of simulated degradation training data, network construction and training, and network aberration removal inference.

[0033] Before generating training data, we first need to extract shallow and deep sub-blocks from the experimentally acquired 3D data. Shallow sub-blocks are determined by areas closer to the detection lens, which have no or only slight aberrations and high signal-to-noise ratios. For simulation data, shallow sub-blocks are defined by simulation data without aberrations; deep sub-blocks are defined by areas where the signal and resolution are significantly degraded. In order to better simulate the degradation process, this embodiment uses an aberration module and a noise module to gradually introduce aberrations and noise into shallow sub-blocks, such as Figure 1 As shown, this degradation process is inspired by the intrinsic image formation process:

[0034] in: o Represents an object, i represents the observed, degraded data, is the incoherent point spread function including aberrations, represents the convolution operation; and N represents Poisson noise, which is the main noise source in low-light-intensity fluorescence microscopy.

[0035] The aberration module introduces aberrations through convolution operations to generate noise-free synthetic aberration data. The specific process is as follows: Figure 2 As shown: Since the convolution operation in the spatial domain can be transformed into a product operation in the frequency domain, the imaging formula can be written in the frequency domain using Fourier transform:

[0036] in: represents the Fourier transform, is the optical transfer function.

[0037] When ideal imaging is performed, the aberration terms can be ignored, so , then we can get:

[0038] in: i 0 is the aberration-free data, h 0 is the ideal point spread function.

[0039] In order to obtain aberration-free data without knowing the real sample i 0The synthetic aberration data is obtained by simulation i , substituting the above formula into the imaging formula in the frequency domain, the aberration term is It can be expressed as:

[0040] in: α is a constant used in division operations where the denominator is zero; as a rule of thumb, α ≤10 -2 There will be good results, in this embodiment α Set to 10 -6 .

[0041] Therefore, the synthesized aberration data i It can be expressed as:

[0042] Incoherent Point Spread Function When the signal is confined within the pupil aperture, it can be expressed as:

[0043] Aberration phase term The Zernike basis function and its corresponding coefficient c m describe:

[0044] in: Different aberration modes are described, c m Describes the magnitude of the aberration, by adjusting and c m The degree of aberration can be varied. In indexing the coefficients of the Zernike polynomials and adding aberrations, the American National Standards Institute convention is used in this embodiment, and the aberration added does not exceed the fourth-order Zernike polynomial ( m max = 14), except for the piston and tilt components ( m =0,1,2).

[0045] The workflow of the noise module is as follows Figure 3 As shown, in order to determine the amount of noise that needs to be added, the noise module needs to estimate the noise level in the deep low-quality sub-blocks.

[0046] Under low light conditions, the noise distribution in fluorescence microscopy mainly follows the Poisson distribution, which is a signal-dependent noise. First, the variance stability transformation can convert the signal-dependent noise into signal-independent noise. A commonly used variance stability transformation method is the Anscombe transformation:

[0047] in: z represents the original observation data, z ' represents the transformed data.

[0048] At this point, the imaging model can be expressed as:

[0049] in: x is noise-free data, n ' is the transformed noise (obeying Gaussian distribution).

[0050] There are many methods for the distribution of Gaussian noise. The method used in this implementation is the one in the literature [Liu Xin, Tanaka Makoto, Okutsu Tomi. Noise level estimation using weak texture blocks of a single noisy image [C] / / 2012 IEEE 19th International Image Processing Conference. IEEE, 2012: 665-668], which can achieve noise estimation of a single image by selecting weak texture areas.

[0051] By performing principal component analysis on the above formula, we can obtain:

[0052] in: It is noisy data z 'The covariance matrix, It is noise-free data x The covariance matrix of is the covariance matrix The minimum eigenvalue of is the standard deviation of Gaussian noise obtained after VST transformation, which characterizes the degree of noise. Since the minimum eigenvalue of the covariance matrix of noise-free data is is unknown, so it is difficult to estimate the standard deviation of Gaussian noise. In the Gaussian noise estimation method, they estimate the noise by iteratively selecting weak texture regions from the noisy data; the minimum eigenvalue of the covariance matrix of the noise-free weak texture region is close to zero, and the standard deviation of the noise can be simplified to:

[0053] in: It is the minimum eigenvalue of the covariance matrix of the weak texture region selected in the noise data.

[0054] After obtaining the standard deviation that characterizes the noise level Afterwards, this embodiment can specifically add noise to the synthetic aberration data.

[0055] The overall structure and training process of the denoising-de-aberration hybrid network model designed by the present invention are as follows: Figure 4 As shown in Figure 1, it includes a denoising module and a de-aberration module. The two modules are connected sequentially. The whole process can be described as:

[0056]

[0057] in: x is the synthetic degraded shallow data, x ' is the denoising result, x ' contains no noise but has aberrations, y is the final output result, y There is neither noise nor aberration in the image; DN represents the denoise module, DA Denotes a deabe module.

[0058] This implementation uses a three-dimensional residual channel attention network (3D-RCAN) to construct a denoising module and a de-aberration module. The basic structure of 3D-RCAN is a residual group RG (Residual Group), each RG contains three residual channel attention blocks RCAB (Residual Channel Attention Block), and the structure of RG can be described as:

[0059]

[0060]

[0061] in: CA is the channel attention module, Conv represents the convolutional layer, GAP is the global average pooling, ReLu and Sigmoid is a nonlinear activation function, is the multiplication operation in the channel dimension, Summation operation in the pixel dimension.

[0062] Since the de-aberration process is more complicated than the denoising process, this embodiment adds more residual groups to the de-aberration module. The denoising module uses 3 residual groups, while the de-aberration module uses 5 residual groups. x 'and y It is expressed as:

[0063]

[0064] In this embodiment, feature extraction is performed Conv The number of channels is set to 32. CA The reduction ratio is set to 8. During the network training process, noise-free synthetic aberration data is used to supervise the denoising module, and high-quality shallow sub-blocks are used to supervise the entire network. This two-step separation supervision method can introduce more image details than the single-step supervision method using only high-quality shallow sub-blocks, thereby improving the denoising and de-aberration performance of the network. The Charbonnier loss function is used in the training of both the denoising and de-aberration modules:

[0065]

[0066]

[0067] in: d is the synthetic aberration data, d 0 It is a high-quality shallow sub-block. is a constant, which is set to 10 in this embodiment. -3 .

[0068] Hereinafter, we verify the effectiveness of the present invention through an experiment, in which we apply the method of the present invention to nematode embryos expressing GFP (Green Fluorescent Protein) membrane marker, labeling head neurons and intestinal cells, and imaging using a dual-view light sheet microscope. Figure 5 Compared with the experimental results, the original data showed a high degree of noise and blurred the membrane boundary; compared with the original data, the present invention well restored the blurred features, thereby enhancing the resolution and contrast.

[0069] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A fluorescence microscopy image restoration method based on a hybrid network of aberration and noise, characterized in that: The steps include: (1) For shallow clear image data in the fluorescence microscopy image, aberration is added to it to generate synthetic aberration image data; (2) adding noise to the synthetic aberration image data to generate synthetic degraded shallow image data; (3) combining shallow clear image data and its corresponding synthetic aberration image data and synthetic degraded shallow image data as training samples for model training; (4) Constructing a denoising-de-aberration hybrid network model for fluorescence microscopy image restoration, and training the model using training samples; the denoising-de-aberration hybrid network model includes: A denoising module is used to perform denoising on the input synthetic degraded shallow image data to obtain intermediate image data that still has aberration blur; A de-aberration module is used to remove the aberration blur in the intermediate image data, so as to obtain clear image data as the final output result; (5) By inputting the deep blurred image data in the fluorescence microscopy image into the trained model, the corresponding clear deep image can be restored.

2. The fluorescence microscopic image restoration method based on a hybrid aberration and noise network according to claim 1, characterized in that: The specific implementation of step (1) is expressed as follows: ; in: i is the synthetic aberration image data, i 0 is the shallow clear image data without aberration, represents the Fourier transform, represents the inverse Fourier transform, is the aberration term.

3. The fluorescence microscopic image restoration method based on a hybrid aberration and noise network according to claim 2, characterized in that: The aberration term The expression is as follows: ; in: h 0 is the ideal point spread function, is the incoherent point spread function, α To set a constant.

4. The fluorescence microscopic image restoration method based on a hybrid aberration and noise network according to claim 3, characterized in that: The incoherent point spread function When the signal is limited to the pupil aperture, its expression is as follows: ; in: is the aberration phase term expressed in the pupil coordinate system, u represents the pupil coordinates, j is the imaginary unit and e is a natural constant.

5. The fluorescence microscopic image restoration method based on a hybrid aberration and noise network according to claim 4, characterized in that: The aberration phase term It reflects the size of the added aberration, which is represented by the Zernike basis function and its corresponding Zernike aberration coefficient. By adjusting the Zernike aberration coefficient, different degrees of aberration can be added to the clear image.

6. The fluorescence microscopic image restoration method based on a hybrid aberration and noise network according to claim 1, characterized in that: The specific implementation method of step (2) is as follows: first, the Poisson noise in the fluorescence microscopy image is converted into Gaussian noise by using a variance stabilization transformation method, then an estimated value of the noise is calculated by using a Gaussian noise estimation method based on weak texture region selection, and finally, an equal amount of noise is added to the synthetic aberration image data according to the estimated value, thereby generating synthetic degraded shallow image data.

7. The fluorescence microscopic image restoration method based on a hybrid aberration and noise network according to claim 1, characterized in that: The denoising module is composed of a convolutional layer D1, three cascaded residual groups, and a convolutional layer D2 connected in sequence, and the input of the convolutional layer D1 is added to the output of the convolutional layer D2 as the output result of the denoising module; the de-aberration module is composed of a convolutional layer D3, five cascaded residual groups, and a convolutional layer D4 connected in sequence, and the input of the convolutional layer D3 is added to the output of the convolutional layer D4 as the output result of the de-aberration module; the residual group is composed of three cascaded residual channel attention blocks and a convolutional layer D5 connected in sequence, and the input of the first residual channel attention block is added to the output of the convolutional layer D5 as the output result of the residual group.

8. The fluorescence microscopic image restoration method based on a mixed aberration and noise network according to claim 7, characterized in that: The residual channel attention block is composed of a convolution layer D6, a Relu activation function layer R1, a convolution layer D7, a global average pooling layer, a convolution layer D8, a Relu activation function layer R2, a convolution layer D9, and a Sigmoid non-linear activation function layer connected in sequence, and the input of the global average pooling layer is multiplied by the output of the Sigmoid non-linear activation function layer and then added to the input of the convolution layer D6. The result is used as the output result of the residual channel attention block.

9. The fluorescence microscopic image restoration method based on a hybrid aberration and noise network according to claim 1, characterized in that: In the step (4), the synthetic degraded shallow image data in the training sample is used as input and the model is trained using the following total loss function; ; ; ; in: L Total is the total loss function, L Denoise is the denoising loss function, L DeAbe is the de-aberration loss function, d is the synthetic aberration image data in the training sample, x ' is the intermediate image data output by the denoising module in the model, y is the image data output by the de-aberration module in the model, d 0 is the shallow clear image data in the training sample, To set a constant.

Images