A multi-segment fluorescence microscopy signal enhancement system and its training method

Abstract

The present invention discloses a multi-stage fluorescence microscopy signal enhancement system and a training method thereof. The system includes a plurality of signal quality restoration modules connected in sequence for the fluorescence signal reduction link in reverse order of the fluorescence signal quality reduction link; the fluorescence signal to be enhanced passes through the plurality of signal quality restoration modules in sequence and is enhanced into a high-quality fluorescence signal in reverse order of the fluorescence signal quality reduction link. The training method includes the following steps: obtaining training samples: preparing training sample images and supervised sample images for each signal quality restoration module; the model training process is specifically as follows: performing separate training on each signal quality restoration module and / or performing end-to-end training on a plurality of signal quality restoration modules connected in sequence for the fluorescence signal reduction link. The present invention realizes high-fidelity fluorescence signal enhancement and avoids distortion and smoothing phenomena.

Description

Technical Field

[0001] The present invention belongs to the field of microscopic imaging, and more specifically, relates to a multi-segment fluorescence microscopic signal enhancement system and a training method thereof. Background Art

[0002] Fluorescence microscope is an optical microscope that uses fluorescent groups or fluorescent proteins for microscopic imaging. Compared with conventional optical microscopes, since it uses the spontaneous fluorescence of organisms for imaging, it has less interference from illumination light and therefore has higher contrast and signal-to-noise ratio. At the same time, fluorescence imaging is closely related to biotechnology such as genetic engineering. Specific proteins or molecules of biological tissues or cells are specifically labeled through fluorescent labeling technology to obtain images with corresponding labeled substances, from which the morphology, characteristics or other biological information of the sample can be briefly obtained. However, since the fluorophores in the organism are limited by the maximum photon exposure, excessive light intensity will cause the fluorophores in the organism to bleach or inactivate. Therefore, the fluorescence signal must be observed under limited light intensity, which will greatly reduce the signal-to-noise ratio of the acquired fluorescence image, which is extremely unfavorable for the analysis of biological information. At the same time, due to the optical diffraction limit, the resolution of the acquired image is also extremely limited.

[0003] How to obtain high signal-to-noise ratio and high-resolution fluorescence microscopy images from low signal-to-noise ratio and low-resolution fluorescence microscopy images obtained by short exposure is a very meaningful thing for biologists. Using deep learning to improve the signal-to-noise ratio and resolution of a single microscopic image, network training is a one-time job. The trained network can be used on any computer, and the deduction and calculation are extremely fast compared to ordinary super-resolution algorithms. This work will promote the application of image acquisition methods that combine software and hardware in the field of biomedical imaging. Existing methods for achieving fluorescence microscopy signal enhancement through network training generally directly pair low signal-to-noise ratio, low-resolution fluorescence data with high signal-to-noise ratio, high-resolution data for training, thereby improving the image signal-to-noise ratio or improving the signal-to-noise ratio and resolution at the same time.

[0004] However, the current fluorescence signal method based on deep learning has a decrease in the fidelity of the enhanced image and has a very obvious smoothing phenomenon. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a multi-stage fluorescence microscopy signal enhancement system and a training method thereof, the purpose of which is to restore the collected fluorescence microscopy signals step by step in the reverse order of the fluorescence signal quality reduction stages to improve the fidelity, thereby solving the technical problem that the prior art uses an overall network to enhance the fluorescence signal, resulting in a decrease in image fidelity and obvious smoothing.

[0006] To achieve the above object, according to one aspect of the present invention, there is provided a multi-segment fluorescence microscopy signal enhancement system, which, in the reverse order of the fluorescence signal quality degradation links, includes a plurality of signal quality restoration modules connected in sequence for the fluorescence signal degradation links; the fluorescence signal to be enhanced passes through the plurality of signal quality restoration modules in sequence and is enhanced into a high-quality fluorescence signal in the reverse order of the fluorescence signal quality degradation links.

[0007] Preferably, in the multi-segment fluorescence microscopy signal enhancement system, there is also a residual connection between the input end and the output end of the signal quality restoration module for backpropagating the gradient.

[0008] Preferably, in the multi-segment fluorescence microscopy signal enhancement system, the signal-to-noise ratio improvement module is used to improve the signal-to-noise ratio of the input fluorescence image and is a convolutional neural network with a U-shaped structure for pixel-to-pixel translation tasks; preferably:

[0009] The deblurring module is used to deblur the input blurred high-signal-to-noise ratio fluorescence image and is a deep neural network composed of a convolutional network with frequency domain information interaction; preferably:

[0010] The super-resolution module is used to improve the resolution of the input fluorescence image and is a deep neural network with strong modeling or inverse solution ability;

[0011] The isotropic module is used to improve the axial resolution of the input fluorescence image to be equivalent to the lateral resolution, and is preferably a two-dimensional network using the same architecture as the deblurring module.

[0012] Preferably, in the multi-segment fluorescence microscopy signal enhancement system, the convolutional neural network with a U-shaped structure adopts a convolutional neural network with a U-shaped backbone downsampling layer simplified to three layers or less; preferably, a convolutional neural network with a U-shaped structure containing simplified residual blocks and subspace attention modules is adopted; preferably, its activation function adopts a linear activation function to adapt to grayscale images.

[0013] According to another aspect of the present invention, there is provided a training method for the multi-segment fluorescence microscopy signal enhancement system, which includes the following steps:

[0014] Obtain training samples: For each signal quality restoration module, prepare its training sample image and supervision sample image;

[0015] The model training process is specifically as follows: Each signal quality restoration module is trained separately and / or the signal quality restoration modules connected in sequence for the fluorescence signal degradation links are trained end-to-end.

[0016] Preferably, in the training method, the obtaining of the training samples is specifically:

[0017] For the super-resolution module: Use the collected high-resolution and high signal-to-noise ratio microscopic fluorescence images as the supervised sample images, downsample them by 2 to 8 times to obtain the training sample images of the super-resolution module, and pair the super-resolution module supervised sample images and training sample images to obtain the training dataset of the super-resolution module; or

[0018] For the deblurring module: Use the training sample images obtained from the latter module as the supervised sample images of the deblurring module, convolve with the point spread function simulated according to the system parameters to obtain the training sample images of the deblurring module, and pair the deblurring module supervised sample images and training sample images to obtain the training dataset of the deblurring module; or

[0019] For the signal-to-noise ratio improvement module: Use the training sample images obtained from the latter module as the supervised sample images of the signal-to-noise ratio improvement module, add Gaussian noise and Poisson noise, etc. to simulate various signal-to-noise ratios of real-shot live cells to obtain the training sample images of the denoising module, and pair the signal-to-noise ratio improvement module supervised sample images and training sample images to obtain the training dataset of the signal-to-noise ratio improvement module.

[0020] Preferably, in the training method, each signal quality restoration module is trained separately, specifically:

[0021] Use the input and output sample images of each signal quality restoration module to perform supervised learning on the signal quality restoration module, respectively minimize the loss function for each signal quality restoration module, obtain each trained signal quality restoration module, and connect them in reverse order of the fluorescence signal quality reduction link to obtain multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link.

[0022] Preferably, in the training method, the loss function loss1 used by the signal-to-noise ratio improvement module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as:

[0023]

[0024]

[0025] where N is the number of voxels of the image, and respectively refer to the true intensity spatial distribution of the supervised sample MR1 of the signal-to-noise ratio improvement module and the predicted value of the network output of the signal-to-noise ratio improvement module, is the two-norm of the two, used to characterize the intensity difference between the network output and the supervised sample; and respectively represent the high-level features extracted by the neural network at the l-th layer of x and x0, Represents the calculation of cosine distance; α and β represent preset weighting coefficients.

[0026] The loss function loss2 adopted by the deblurring module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as:

[0027]

[0028] where N is the number of voxels in the image, and respectively refer to the true intensity spatial distribution of the deblurring module's supervised sample MR2 and the predicted value of the deblurring module network output, is the Euclidean norm of the two, used to characterize the intensity difference between the network output and the supervised sample; γ and η represent preset weighting coefficients.

[0029] The loss function loss3 adopted by the super-resolution module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as:

[0030]

[0031] where N is the number of voxels in the image, HR and SR respectively refer to the true intensity spatial distribution of the super-resolution module's supervised sample and the predicted value of the super-resolution module network output, is the Euclidean norm of the two, used to characterize the intensity difference between the network output and the supervised sample; λ and μ represent preset weighting coefficients.

[0032] The loss function loss4 adopted by the isotropic module is the weighted error of the mean squared error and the absolute error; it can be specifically expressed as:

[0033]

[0034] where N is the number of voxels in the image, GT XY and Pred XY respectively refer to the true intensity spatial distribution of the isotropic module's supervised sample and the predicted value of the isotropic module network output, is the Euclidean norm of the two, ||GT XY -Pred XY ||1 is the Manhattan norm of the two, and the two are combined to characterize the intensity difference between the network output and the supervised sample; λ and μ represent preset weighting coefficients.

[0035] Preferably, in the training method, when separately training each signal quality restoration module and end-to-end training multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link, first separately train each signal quality restoration module, and then perform end-to-end training on multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link;

[0036] Jointly train multiple signal quality restoration modules by successively superimposing them in the connection order. In particular, jointly train the signal-to-noise ratio improvement module, the deblurring module, and the super-resolution module by successively superimposing them. Specifically: First train the signal-to-noise ratio improvement module, then connect the trained signal-to-noise ratio improvement module and the untrained deblurring module for end-to-end joint training, and finally connect the trained signal-to-noise ratio improvement module, deblurring module, and the untrained super-resolution module for end-to-end joint training to obtain the trained signal-to-noise ratio improvement module, deblurring module, and super-resolution module.

[0037] Preferably, in the training method, when jointly training multiple signal quality restoration modules by successively superimposing them in the connection order, the loss function representing the comprehensive loss value of each signal quality restoration module used in each superimposed training is the weighted sum of the loss functions of each superimposed module. Generally speaking, compared with the prior art through the above technical solutions conceived by the present invention, the following beneficial effects can be achieved:

[0038] The multi-stage fluorescence microscopy signal enhancement system adopted by the present invention restores the fluorescence signal one by one in the reverse order of the fluorescence signal quality reduction link, thereby realizing high-fidelity fluorescence signal enhancement and avoiding distortion and flattening phenomena caused by directly training and modeling the same network from the collected fluorescence data to high-quality fluorescence data.

[0039] In a preferred solution, add a residual connection for backpropagating gradients, thereby facilitating network training for multi-module connection.

[0040] In a preferred solution, adopt a targeted model and loss function for each restoration link respectively, further improving the restoration effect of each link of the fluorescence signal, thereby ensuring good overall signal enhancement.

[0041] The training method of the multi-stage fluorescence microscopy signal enhancement system provided by the present invention ensures the signal quality restoration effect of each module by preparing training sample images and supervised sample images suitable for each module and performing supervised training on each signal quality restoration module, and finally improves the fidelity of overall fluorescence signal enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic structural diagram of a multi-stage fluorescence microscopy signal enhancement system provided by an embodiment of the present invention;

[0043] Figure 2 It is a schematic structural diagram of a multi - segment fluorescence microscopy signal enhancement system with residual connection provided by an embodiment of the present invention. Detailed implementation manners

[0044] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] The multi - segment fluorescence microscopy signal enhancement system provided by the present invention includes a plurality of signal quality restoration modules connected in sequence for the fluorescence signal reduction link in reverse order of the fluorescence signal quality reduction link; the fluorescence signal to be enhanced passes through the plurality of signal quality restoration modules in sequence and is enhanced into a high - quality fluorescence signal in reverse order of the fluorescence signal quality reduction link; specifically, the plurality of signal quality restoration modules are, in sequence: a signal - to - noise ratio improvement module, a de - blurring module, a super - resolution module, and / or an isotropic module;

[0046] There is also a residual connection between the input end and the output end of the signal quality restoration module for back - propagating gradients.

[0047] The signal - to - noise ratio improvement module is used to improve the signal - to - noise ratio of the input fluorescence image and is a convolutional neural network with a U - shaped structure for pixel - to - pixel translation tasks. Specifically, DRUNet, NBNet, and other dedicated neural networks for signal - to - noise ratio improvement can be used; preferably, a convolutional neural network with a U - shaped backbone downsampling layer simplified to three layers or less is used; preferably, a U - shaped convolutional neural network containing simplified residual blocks and subspace attention modules is used; preferably, its activation function uses a linear activation function to adapt to grayscale images.

[0048] The de - blurring module is used to de - blur the input high - signal - to - noise - ratio fluorescence image with blur and is a deep neural network composed of a convolutional network with frequency - domain information interaction. Specifically, DeepRFT, MIMO - UNet can be used; preferably, a convolutional neural network with a U - shaped backbone downsampling layer simplified to three layers or less is used; preferably, a U - shaped convolutional neural network containing simplified residual blocks and subspace attention modules is used; preferably, its activation function uses a linear activation function to adapt to grayscale images.

[0049] The super-resolution module is used to improve the resolution of the input fluorescence image. For a deep neural network with strong modeling or inversion capabilities, RCAN or IRN can be specifically adopted. The isotropic module is used to improve the axial resolution of the input fluorescence image to be equivalent to the lateral resolution. Preferably, a two-dimensional network with the same architecture as the deblurring module is used, and specifically, two-dimensional DeepRFT or MIMO-UNet can be adopted. Preferably, a convolutional neural network with a U-shaped backbone downsampling layer simplified to three layers is adopted. Preferably, a convolutional neural network with a U-shaped structure containing five residual Fourier convolutional blocks is adopted. Preferably, a linear activation function is used as the activation function to adapt to grayscale images.

[0050] We found that the quality of fluorescence images of biological samples, especially living samples, is not high because the fluorophores in the biological body are limited by the maximum photon exposure. Excessive light intensity will cause the fluorophores in the biological body to bleach or inactivate. Therefore, fluorescence signal observation must be carried out under limited light intensity, which will greatly reduce the signal-to-noise ratio of the acquired fluorescence image. At the same time, due to the blurring function inherent in the optical system and the optical diffraction limit, the resolution of the acquired image is also extremely limited. Therefore, it is extremely important to improve the signal-to-noise ratio and resolution of fluorescence microscopic images. When using the same neural network for fluorescence signal enhancement, while taking into account signal-to-noise ratio improvement, deblurring, and super-resolution, and even including compensating for axial resolution loss, since they are all a process of inverse solution, it is impossible to obtain relatively accurate restoration results at the same time, and the distortion and flattening phenomena of the enhanced fluorescence microscopic image are obvious.

[0051] The present invention performs restoration and reconstruction in the reverse order of the fluorescence signal quality degradation links. For each fluorescence signal quality degradation link: accurately collect the input and output training samples, select a suitable neural network for modeling, and can effectively calculate each inverse solution well. Finally, it can successfully achieve parsing a high-fidelity, high-signal-to-noise ratio, and high-resolution fluorescence microscopic image from a low-signal-to-noise ratio fluorescence microscopic image.

[0052] At the same time, migrating these well-performing networks to the scenario tasks processed by the present invention is by no means simply splicing after changing to a three-dimensional network. Because after splicing the three networks, the depth of the network will inevitably be very deep. Therefore, after connecting the three networks other than the isotropic network, residual connections should be considered to be added between them to ensure that the gradient can be effectively backpropagated, and at the same time, it also provides effective help for the joint parameter tuning mentioned later. See the connection diagram in Figure 2 。

[0053] The training method of the multi-segment fluorescence microscopic signal enhancement system provided by the present invention includes the following steps:

[0054] Obtaining training samples: For each signal quality restoration module, prepare its training sample images and supervision sample images; specifically:

[0055] For the super-resolution module: Use the collected high-resolution and high-signal-to-noise microscopic fluorescence images as the supervision sample images, and obtain the training sample images of the super-resolution module by downsampling by n times (n is an integer greater than or equal to 2 and less than or equal to 8). Pair the supervision sample images and training sample images of the super-resolution module to obtain the training dataset of the super-resolution module.

[0056] For the deblurring module: Use the training sample images obtained from the subsequent module as the supervision sample images of the deblurring module, and convolve with the point spread function simulated according to the system parameters to obtain the training sample images of the deblurring module. Pair the supervision sample images and training sample images of the deblurring module to obtain the training dataset of the deblurring module.

[0057] For the signal-to-noise ratio improvement module: Use the training sample images obtained from the subsequent module as the supervision sample images of the signal-to-noise ratio improvement module, and add Gaussian noise and Poisson noise, etc., to simulate various signal-to-noise ratios of real-shot living cells to obtain the training sample images of the denoising module. Pair the supervision sample images and training sample images of the signal-to-noise ratio improvement module to obtain the training dataset of the signal-to-noise ratio improvement module.

[0058] The model training process is as follows: Train each signal quality restoration module separately and / or perform end-to-end training on multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link; when training each signal quality restoration module separately and performing end-to-end training on multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link, first train each signal quality restoration module separately, and then perform end-to-end training on multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link; preferably, perform joint training on multiple signal quality restoration modules by successive stacking according to the connection order, especially perform joint training on the signal-to-noise ratio improvement module, deblurring module, and super-resolution module by successive stacking. Specifically: First train the signal-to-noise ratio improvement module, then connect the trained signal-to-noise ratio improvement module and the untrained deblurring module for end-to-end joint training, and finally connect the trained signal-to-noise ratio improvement module, deblurring module, and untrained super-resolution module for end-to-end joint training to obtain the trained signal-to-noise ratio improvement module, deblurring module, and super-resolution module.

[0059] The specific process of training each signal quality restoration module separately is as follows:

[0060] Supervised learning is performed on the input and output sample images of each signal quality restoration module to minimize the loss function for each signal quality restoration module respectively, obtaining each trained signal quality restoration module, which are connected in reverse order of the fluorescence signal quality reduction link to obtain multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link. Among them:

[0061] The loss function loss1 adopted by the signal-to-noise ratio improvement module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as:

[0062]

[0063]

[0064] Among them, N is the number of voxels of the image, and respectively refer to the true intensity spatial distribution of the supervised sample MR1 of the signal-to-noise ratio improvement module and the predicted value of the network output of the signal-to-noise ratio improvement module, is the two-norm of the two, which is used to characterize the intensity difference between the network output and the supervised sample; and respectively represent the high-level features extracted by the neural network at the l-th layer of x and x0, represents the calculation of the cosine distance; α and β represent the preset weighting coefficients.

[0065] The loss function loss2 adopted by the deblurring module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as:

[0066]

[0067] Among them, N is the number of voxels of the image, and respectively refer to the true intensity spatial distribution of the supervised sample MR2 of the deblurring module and the predicted value of the network output of the deblurring module, is the two-norm of the two, which is used to characterize the intensity difference between the network output and the supervised sample; γ and η represent the preset weighting coefficients.

[0068] The loss function loss3 adopted by the super-resolution module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as:

[0069]

[0070] Wherein, N is the number of voxels of the image, HR and SR respectively refer to the true intensity spatial distribution of the supervised samples of the super-resolution module and the predicted value of the network output of the super-resolution module, is the second norm of the two, used to characterize the intensity difference between the network output and the supervised samples; λ and μ represent preset weighting coefficients.

[0071] The loss function loss4 adopted by the isotropic module is the weighted error of the mean square error and the absolute value error; it can be specifically expressed as:

[0072]

[0073] Wherein, N is the number of voxels of the image, GT XY and Pred XY respectively refer to the true intensity spatial distribution of the supervised samples of the isotropic module and the predicted value of the network output of the isotropic module, is the second norm of the two, ||GT XY -Pred XY ||1 is the first norm of the two, and the two are combined to characterize the intensity difference between the network output and the supervised samples; λ and μ represent preset weighting coefficients.

[0074] In the existing end-to-end fluorescence signal enhancement method, since no loss function is designed for each reason for the reduction of signal quality, and an overall loss function such as the mean square error based on each pixel is used, the restored signal has an obvious smoothing phenomenon. The present invention extracts more potential information at the deep feature level by introducing the perceptual loss LPIPS. At the same time, this loss combines the judgment of the human eye on the signal quality, and can more effectively retain the texture information of the signal, so that the finally restored signal does not have a smoothing effect.

[0075] When jointly training multiple signal quality restoration modules in sequence by successive superposition according to the connection order, the loss function representing the comprehensive loss value of each signal quality restoration module used in each superposition training is the weighted sum of the loss functions of each superposed module. For example, when superposing the signal-to-noise ratio improvement module and the deblurring module, the loss function loss12 is written as:

[0076] loss12 = σ·loss1 + τ·loss2

[0077] Wherein, σ and τ are the weighting coefficients of the loss function loss1 and the loss function loss2.

[0078] The end-to-end training of multiple signal quality restoration modules for the fluorescence signal reduction link connected in sequence is specifically as follows:

[0079] Supervised learning is performed on the input and output sample images of each signal quality restoration module, so that the loss function representing the comprehensive loss value of each signal quality restoration module is minimized, and multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link are obtained.

[0080] The loss function loss123 representing the comprehensive loss value of each signal quality restoration module is the weighted sum of the loss functions loss1, loss2, and loss3, and can be specifically expressed as:

[0081] loss123 = σ·loss1 + τ·loss2 + ω·loss3 where σ, τ, and ω are the weighting coefficients of the loss function loss1, the loss function loss2, and the loss function loss3.

[0082] However, in actual training, these three modules may be trained separately. When each module reaches the best value, the network converges. But in practice, it may be necessary to consider that training each module separately may generate some false signals, and some signals may be lost during calculation. Therefore, joint training may be more effective and can restore more details. However, in the case of direct joint training, the network may not be able to find the best convergence point, resulting in a decline in overall performance. So a better strategy is to pre-train the first signal-to-noise ratio improvement module, then connect to the deblurring module for the second pre-training, and finally connect to the super-resolution module for joint training. While ensuring that each module can achieve good results, joint parameter tuning is performed, and the error of the subsequent loss function can also be backpropagated to the previous ones, supervising the effects of the previous modules, so that the final multi-stage fluorescence microscopy signal enhancement method based on deep learning can achieve the best restoration effect, with richer details and higher fidelity in the final image.

[0083] And since the isotropic module only performs a function mainly for improving the axial resolution and is a two-dimensional network, it does not need to be jointly tuned with the previous three modules. Given that the XY plane of the optical system should be isotropic, the idea of this method is to convolve the slices of the XY plane with the PSF of the XZ plane of the system to obtain the degraded slices of the plane, and then use the degraded to pair with the original high-quality XY for learning, thus obtaining a neural network similar to deconvolution. In X, it is equivalent to the horizontal direction, in Y, it is equivalent to the axial direction, is equivalent to the horizontal direction after being blurred by the PSF, and It corresponds to the axial direction after blurring by PSF. Then, the original image powder to be restored is sliced along XZ and YZ respectively, sent into the trained deconvolution neural network for restoration, and then spliced back respectively to obtain two restored three-dimensional images. The average of the two can be taken to obtain the final isotropic three-dimensional image.

[0084] The following are examples:

[0085] The multi-segment fluorescence microscopy signal enhancement system provided by the present invention includes a plurality of sequentially connected signal quality restoration modules for the fluorescence signal reduction link in reverse order of the fluorescence signal quality reduction link; the fluorescence signal to be enhanced passes through the plurality of signal quality restoration modules in sequence and is enhanced into a high-quality fluorescence signal in reverse order of the fluorescence signal quality reduction link; specifically, the plurality of signal quality restoration modules are: a signal-to-noise ratio improvement module, a deblurring module, a super-resolution module, and / or an isotropic module;

[0086] The signal-to-noise ratio improvement module is used to improve the signal-to-noise ratio of the input fluorescence image and is a deep neural network composed of a convolutional network. Specifically, dedicated neural networks for signal-to-noise ratio improvement such as DRUNet and NBNet can be used. The design of the network mainly needs to consider the randomness of noise and the stability of the signal. Therefore, how to effectively introduce this prior information into the network is extremely crucial. DRUnet uses the efficient end-to-end image translation ability of UNet and the powerful modeling ability of residual blocks to build the main framework of the network; while NBNet reduces noise through image adaptive projection, learns a set of reconstruction bases in the feature space to train a network that can separate signals and noise, and can achieve image denoising by selecting the corresponding bases in the signal subspace and projecting the input into the signal subspace. The projection method can naturally preserve the local structure of the input signal, especially for regions with weak light or weak texture;

[0087] The deblurring module is used to perform deblurring processing on the input high signal-to-noise ratio fluorescent image with blur, and it is a deep neural network composed of convolutional networks. Specifically, DeepRFT and MIMO-UNet can be used. Since it is necessary to parse a clear image from a blurred image, which is different from separating noise and signal in the denoising task, it learns the difference between blurred and sharpened image pairs, and reconstructing a clear image from a blurred image requires changing low-frequency and high-frequency information. Therefore, when designing the network, the interaction of image frequency domain information should be considered. DeepRFT proposes a residual fast Fourier transform with convolutional blocks (Res FFT-Conv Block), which can capture both long-term and short-term interactions and integrate low-frequency and high-frequency residual information, making it very suitable for designing deblurring networks. MIMO-UNet performs multi-scale input and output processing for the deblurring task, uses a single UNet to simulate multiple cascaded UNets, and considers deconvolution at multiple scales, making the designed network more robust and achieving good results at the same time.

[0088] The super-resolution module is used to improve the resolution of the input fluorescent image, and it is a deep neural network. Specifically, RCAN and IRN can be used. The work done by the super-resolution module mainly focuses on how to recover the information lost during the image downsampling process. This is a relatively difficult problem, mainly manifested in that there are many high-resolution (HR) images that can be downsampled to obtain a low-resolution (LR) image, which makes the super-resolution problem a multi-solution problem and is very difficult to implement, requiring high modeling ability for the network. RCAN greatly improves the network's modeling ability by adding a channel attention mechanism during the design of the convolutional network, thus greatly enhancing the performance of the super-resolution network. As an explorer in the new field of super-resolution networks, IRN abandons the use of convolutional neural networks and adopts reversible neural networks to retain the information lost during image downsampling in a solution space, effectively alleviating the multi-solution problem.

[0089] The isotropic module is used to improve the axial resolution of the input fluorescent image to be equivalent to the lateral resolution, and it is a deep neural network composed of convolutional networks.

[0090] Specifically, two-dimensional DeepRFT can be used. The network function of this module is similar to that of the deblurring module, both for deblurring. However, the difference is that the deblurring module performs deblurring operations on each dimension of two-dimensional or three-dimensional signals, while the isotropic module is specifically used to remove axial blur of three-dimensional data and is a two-dimensional network. Combining the previous analysis, it is reasonable to use two-dimensional DeepRFT.

[0091] However, it should be noted that we do not simply use existing networks such as DRUNet or DeepRFT directly. Since the above-mentioned networks are all for the single task of restoring two-dimensional color images, while what we need to do is a task of jointly denoising, deblurring, and super-resolving two-dimensional or three-dimensional grayscale images. The computational complexity of the network is the first issue to be considered. In order to perform multi-task joint and three-dimensional adaptation, it is necessary to inherit and simplify some of the network structures. For example, for the U-shaped network structures used in the denoising module, deblurring module, and isotropic module, our optimization improvement is to reduce the number of downsampling layers of the U-shaped backbone to three layers. At the same time, the residual blocks, subspace attention modules, and residual Fourier convolution modules that enhance the performance of the corresponding U-shaped network need to be reduced to five layers. At the same time, the activation of the last layer of the neural network is also changed from the original Relu activation to a linear activation. The purpose of doing this is to adapt to the situation when we process grayscale images. Negative activation of the output value is not what we focus on, and we can directly suppress it, which can make the background of the calculated fluorescence image cleaner. At the same time, when processing three-dimensional data, the corresponding convolutions and poolings also need to be changed to three-dimensional. The Fourier transforms that appear in the Res FFT-Conv Block module included in the corresponding DeepRFT network also need to be changed to three-dimensional Fourier transform and three-dimensional inverse Fourier transform.

[0092] The training method of the multi-segment fluorescence microscopy signal enhancement system provided in this embodiment includes the following steps:

[0093] Obtain training samples: For each signal quality restoration module, prepare its training sample images and supervised sample images; specifically:

[0094] For the super-resolution module: The high-resolution, high-signal-to-noise microscopy fluorescence image HR collected, that is, the supervised sample image, is downsampled by 8 times to obtain the training sample image MR2 of the super-resolution module. Pairing MR2 and HR can obtain the training data set of the super-resolution module.

[0095] For the deblurring module: Convolve the MR2 obtained from the previous module, that is, the supervised sample image, with the point spread function simulated according to the system parameters to obtain the training sample image MR1 of the deblurring module. Pairing MR1 and MR2 can obtain the training data set of the deblurring module.

[0096] For the signal-to-noise ratio improvement module: Add Gaussian noise and Poisson noise, etc., to MR1 obtained from the previous module to simulate various signal-to-noise ratios of actual live cells to obtain the training sample image LR of the denoising module. Pairing LR and MR1 can obtain the training data set of the denoising module.

[0097] The model training process is as follows: Multiple signal quality restoration modules are jointly trained by successive superposition according to the connection order. In particular, the signal-to-noise ratio improvement module, the deblurring module, and the super-resolution module are jointly trained by successive superposition. Specifically, it is in turn:

[0098] (1) Train the signal-to-noise ratio improvement module. The loss function loss1 adopted by the signal-to-noise ratio improvement module is the weighted error of the mean squared error and the deep network feature error LPIPS. It can be specifically expressed as:

[0099]

[0100]

[0101] where N is the number of voxels of the image, and respectively refer to the true intensity spatial distribution of the supervised sample MR1 of the signal-to-noise ratio improvement module and the predicted value of the network output of the signal-to-noise ratio improvement module. is the Euclidean norm of the two, which is used to characterize the intensity difference between the network output and the supervised sample; and respectively represent the high-level features extracted by the neural network at the l-th layer of x and x0. represents the calculation of the cosine distance; α and β represent the preset weighting coefficients.

[0102] (2) Connect the trained signal-to-noise ratio improvement module and the untrained deblurring module for end-to-end joint training. The adopted loss function loss12 is written as:

[0103] loss12 = σ·loss1 + τ·loss2

[0104] where σ and τ are the weighting coefficients of the loss function loss1 and the loss function loss2.

[0105] The loss function loss2 adopted by the deblurring module is the weighted error of the mean squared error and the deep network feature error LPIPS. It can be specifically expressed as:

[0106]

[0107] where N is the number of voxels of the image, and respectively refer to the true intensity spatial distribution of the supervised sample MR2 of the deblurring module and the predicted value of the network output of the deblurring module. is the Euclidean norm of the two, which is used to characterize the intensity difference between the network output and the supervised sample; γ and η represent the preset weighting coefficients.

[0108] (3) Connect the trained signal-to-noise ratio improvement module, deblurring module, and untrained super-resolution module for end-to-end joint training to obtain the trained signal-to-noise ratio improvement module, deblurring module, and super-resolution module; the loss function loss123 is the weighted sum of the loss functions loss1, loss2, and loss3, and can be specifically expressed as:

[0109] loss123 = σ·loss1 + τ·loss2 + ω·loss3

[0110] Among them, σ, τ, and ω are the weighting coefficients of the loss function loss1, the loss function loss2, and the loss function loss3.

[0111] The loss function loss3 adopted by the super-resolution module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as:

[0112]

[0113] Among them, N is the number of voxels of the image, HR and SR respectively refer to the true intensity spatial distribution of the super-resolution module's supervised samples and the predicted value of the super-resolution module's network output, is the two-norm of the two, used to characterize the intensity difference between the network output and the supervised sample; λ and μ represent the preset weighting coefficients.

[0114] And since the isotropic module only performs a function mainly for improving the axial resolution and is a two-dimensional network. Therefore, it does not need to be jointly tuned with the previous three modules. Separate training is adopted. The loss function loss4 adopted by the isotropic module is the weighted error of the mean squared error and the absolute value error; it can be specifically expressed as:

[0115]

[0116] Among them, N is the number of voxels of the image, GT XY and Pred XY respectively refer to the true intensity spatial distribution of the isotropic module's supervised samples and the predicted value of the isotropic module's network output, is the two-norm of the two, ||GT XY -Pred XY ||1 is the one-norm of the two, and the two are combined to characterize the intensity difference between the network output and the supervised sample; λ and μ represent the preset weighting coefficients.

[0117] In view of the fact that the XY plane of the optical system should be isotropic, the idea of this method is to select the PSF of the XZ plane of the system to convolve the slices of the XY plane to obtain the degraded Slices of the surface, and then using the degraded Pair with the original high-quality XY for learning, thus obtaining a neural network similar to deconvolution. In the X direction, it is equivalent to the horizontal direction, and in the Y direction, it is equivalent to the axial direction. It is equivalent to the horizontal direction after being blurred by the PSF, while It is equivalent to the axial direction after being blurred by the PSF. Then slice the original image to be restored along XZ and YZ respectively, send it into the trained deconvolution neural network for restoration, and then splice them back respectively to obtain two restored three-dimensional images. Averaging the two can obtain the final isotropic three-dimensional image.

[0118] Those skilled in the art can easily understand that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multi-stage fluorescence microscopy signal enhancement system, characterized in that In the reverse order of the fluorescence signal quality reduction links, it includes a plurality of sequentially connected signal quality restoration modules for the fluorescence signal reduction links; the fluorescence signal to be enhanced sequentially passes through the plurality of signal quality restoration modules and is enhanced into a high-quality fluorescence signal in the reverse order of the fluorescence signal quality reduction links; Specifically, the plurality of signal quality restoration modules are, in sequence: a signal-to-noise ratio improvement module, a deblurring module, a super-resolution module, and an isotropic module; The signal-to-noise ratio improvement module is used to improve the signal-to-noise ratio of the input fluorescence image; The deblurring module is used to perform deblurring processing on the input blurred high signal-to-noise ratio fluorescence image; The super-resolution module is used to improve the resolution of the input fluorescence image; The isotropic module is used to improve the axial resolution of the input fluorescence image to be equivalent to the lateral resolution; The loss function loss4 used for training the isotropic module is the weighted error of the mean squared error and the absolute error; it can be specifically expressed as: where N is the number of voxels of the image, GT XY and Pred XY respectively refer to the true intensity spatial distribution of the supervised samples of the isotropic module and the predicted values of the network output of the isotropic module. is the two-norm of the two, ||GT XY -Pred XY ||1 is the one-norm of the two, and the two are combined to characterize the intensity difference between the network output and the supervised samples; λ and μ represent preset weighting coefficients.

2. The multi-stage fluorescence microscopic signal enhancement system according to claim 1, wherein There is also a residual connection between its input end and the output end of the signal quality restoration module, which is used to backpropagate the gradient.

3. The multi-stage fluorescence microscopic signal enhancement system according to claim 1, wherein The signal-to-noise ratio improvement module is a convolutional neural network with a U-shaped structure for pixel-to-pixel translation tasks; The deblurring module is a deep neural network composed of a convolutional network with frequency domain information interaction; The super-resolution module is a modeling or inverse depth neural network; The isotropic module is a two-dimensional network with the same architecture as the deblurring module.

4. The multi-segment fluorescence microscopy signal enhancement system according to claim 3, wherein, The convolutional neural network with a U-shaped structure uses a convolutional neural network with a U-shaped backbone downsampling layer simplified to three layers or less; it uses a convolutional neural network with a U-shaped structure containing simplified residual blocks and subspace attention modules; its activation function uses a linear activation function to adapt to grayscale images.

5. The training method of the multi-segment fluorescence microscopy signal enhancement system according to any one of claims 1 to 4, characterized in that It includes the following steps: Obtain training samples: For each signal quality restoration module, prepare its training sample images and supervised sample images; The model training process is specifically as follows: Each signal quality restoration module is trained separately, and / or the signal quality restoration modules for multiple sequentially connected fluorescence signal reduction links are trained end-to-end.

6. The training method according to claim 5, wherein Specifically, the obtaining of the training samples: For the super-resolution module: The collected high-resolution and high signal-to-noise ratio microscopic fluorescence images are used as supervised sample images, and are downsampled by 2 to 8 times to obtain the training sample images of the super-resolution module. The super-resolution module supervised sample images and training sample images are paired to obtain the training dataset of the super-resolution module; Or For the deblurring module: The training sample images obtained by the latter module are used as the supervised sample images of the deblurring module, and the training sample images of the deblurring module are obtained by convolving the point spread function simulated according to the system parameters. The deblurring module supervised sample images and training sample images are paired to obtain the training dataset of the deblurring module; or For the signal-to-noise ratio improvement module: The training sample images obtained by the latter module are used as the supervised sample images of the signal-to-noise ratio improvement module, and the training sample images of the denoising module are obtained by simulating various signal-to-noise ratios of real-shot live cells. The signal-to-noise ratio improvement module supervised sample images and training sample images are paired to obtain the training dataset of the signal-to-noise ratio improvement module.

7. The training method according to claim 5, wherein The separate training of each signal quality restoration module is specifically as follows: Using the input and output sample images for each signal quality restoration module, supervised learning is performed on the signal quality restoration module to minimize the loss function for each signal quality restoration module respectively, obtaining each trained signal quality restoration module, which are connected in reverse order of the fluorescence signal quality reduction link to obtain multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link.

8. The training method according to claim 7, wherein The loss function loss1 adopted by the signal-to-noise ratio improvement module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as: where N is the number of voxels of the image, and respectively refer to the true intensity spatial distribution of the supervised sample MR1 of the SNR improvement module and the predicted value of the network output of the SNR improvement module, is the L2 norm of the two, which is used to characterize the intensity difference between the network output and the supervised sample; and respectively represent the high-level features extracted by the neural network at the l-th layer of x and x0, represents the calculation of the cosine distance; α and β represent preset weighting coefficients; The loss function loss2 adopted by the deblurring module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as: where N is the number of voxels of the image, and respectively refer to the true intensity spatial distribution of the deblurring module's supervised sample MR2 and the predicted value of the deblurring module network output, is the L2 norm of the two, used to characterize the intensity difference between the network output and the supervised sample; γ and η represent preset weighting coefficients; The loss function loss3 adopted by the super-resolution module is the weighted error of the mean squared error and the deep network feature error LPIPS; it can be specifically expressed as: Where N is the number of voxels of the image, HR and SR respectively refer to the true intensity spatial distribution of the super-resolution module's supervised samples and the predicted values output by the super-resolution module network, and is the L2 norm of the two, which is used to characterize the intensity difference between the network output and the supervised samples; λ and μ represent preset weighting coefficients.

9. The training method according to claim 5, wherein When separately training each signal quality restoration module and performing end-to-end training on multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link, first, each signal quality restoration module is separately trained, and then end-to-end training is performed on multiple sequentially connected signal quality restoration modules for the fluorescence signal reduction link. The joint training of multiple signal quality restoration modules is performed by successive stacking according to the connection order, that is, the signal-to-noise ratio improvement module, the deblurring module, and the super-resolution module are successively stacked for joint training. Specifically: first, the signal-to-noise ratio improvement module is trained, then the trained signal-to-noise ratio improvement module and the untrained deblurring module are connected for end-to-end joint training, and finally, the trained signal-to-noise ratio improvement module, deblurring module, and untrained super-resolution module are connected for end-to-end joint training to obtain the trained signal-to-noise ratio improvement module, deblurring module, and super-resolution module.

10. The training method according to claim 9, characterized in that, When performing joint training on multiple signal quality restoration modules by successive stacking according to the connection order, the loss function representing the comprehensive loss value of each signal quality restoration module used in each stacking training is the weighted sum of the loss functions of each module.

Images