A super-resolution fluorescence image reconstruction method based on a diffusion model and related equipment

Abstract

The application provides a super-resolution fluorescence image reconstruction method based on a diffusion model and related equipment, and belongs to the technical field of super-resolution imaging of structured light illumination microscopy. The method comprises the following steps: obtaining a micro image to be reconstructed; inputting the micro image to be reconstructed into a pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result; and inputting the first preprocessing result into a reverse stage of a pre-trained diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the micro image to be reconstructed. The scheme uses the progressive generation mechanism of the diffusion model and the preprocessing result of the residual channel attention network to perform image reconstruction, effectively overcomes the problem of neural network spectral bias, exhibits better detail restoration capability in image reconstruction, and improves the image reconstruction effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of structured light illumination microscopy super-resolution imaging, and in particular to a super-resolution fluorescence image reconstruction method based on a diffusion model and related equipment. BACKGROUND

[0002] Super-resolution fluorescence microscopy imaging breaks through the constraint of the optical diffraction limit, and provides a powerful observation tool for exploring the physiological activities of nanoscale subcellular structures. However, super-resolution imaging is not a "what you see is what you get", but is severely dependent on the reconstruction processing of the post-processing algorithm, which puts high requirements on the performance of the algorithm and the quality of the original imaging results.

[0003] Most of the traditional physical-based image reconstruction algorithms need accurate optical parameters and time-consuming iterative operations, and are greatly affected by noise, aberration and other adverse factors in actual application, which greatly limits the observation ability of the super-resolution microscope under the demand of low light toxicity and photobleaching, and the image reconstruction effect is poor. SUMMARY

[0004] Therefore, the embodiments of the present application provide a super-resolution fluorescence image reconstruction method based on a diffusion model and related equipment to solve the problem of poor image reconstruction effect of the traditional physical-based image reconstruction algorithm.

[0005] To achieve the above object, the embodiments of the present application provide the following technical scheme:

[0006] The first aspect of the embodiments of the present application discloses a super-resolution fluorescence image reconstruction method based on a diffusion model, which comprises:

[0007] Obtaining a micro image to be reconstructed;

[0008] Inputting the micro image to be reconstructed into a pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result;

[0009] Inputting the first preprocessing result into a reverse stage of a pre-trained diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the micro image to be reconstructed;

[0010] Wherein, the residual channel attention network and the diffusion model are trained based on a pre-set semi-synthetic data set.

[0011] Preferably, inputting the micro image to be reconstructed into a pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result comprises:

[0012] input the micro image to be reconstructed into a pre-trained residual channel attention network for denoising and prediction to obtain a first preprocessing result, the first preprocessing result at least including a first denoised image obtained by denoising and a first point spread function obtained by prediction.

[0013] Preferably, the process of training the residual channel attention network and the diffusion model based on the preset semi-synthetic data set comprises:

[0014] Based on the optical degradation model of the microscopic imaging system, a semi-synthetic data set covering subcellular structures is constructed, the semi-synthetic data set at least including original images of different signal-to-noise ratios and resolutions;

[0015] The residual channel attention network is trained using the semi-synthetic data set and a loss function until convergence is achieved.

[0016] A second preprocessing result is obtained, the second preprocessing result being obtained by preprocessing the original images in the semi-synthetic data set using the converged residual channel attention network, the second preprocessing result at least including a second denoised image obtained by denoising and a second point spread function obtained by prediction;

[0017] The diffusion model is trained using the second preprocessing result until convergence is achieved.

[0018] Preferably, based on the optical degradation model of the microscopic imaging system, a semi-synthetic data set covering subcellular structures is constructed, comprising:

[0019] Based on the optical degradation model of the microscopic imaging system, different Gaussian blur kernels and mixed noise are simulated;

[0020] The Gaussian blur kernel, the mixed noise, and real multi-signal-to-noise ratio wide-field images are combined to construct a semi-synthetic data set covering subcellular structures.

[0021] Preferably, the diffusion model is trained using the second preprocessing result until convergence is achieved, comprising:

[0022] In the forward stage of the diffusion model, Gaussian noise is gradually added to the sampling target of the diffusion model until the sampling target becomes pure noise, the sampling target being a true value distribution;

[0023] In the reverse stage of the diffusion model, the second preprocessing result is input into the diffusion model, and the diffusion model gradually restores the sampling target by predicting the Gaussian noise added in the forward stage to complete the training of the diffusion model.

[0024] The second aspect of the embodiment of the application discloses a super-resolution fluorescence image reconstruction device based on a diffusion model, the device comprising:

[0025] an acquisition unit configured to acquire a to-be-reconstructed microscopic image;

[0026] a preprocessing unit configured to input the to-be-reconstructed microscopic image into a pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result;

[0027] a reconstruction unit configured to input the first preprocessing result into a reverse stage of a pre-trained diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the to-be-reconstructed microscopic image;

[0028] The residual channel attention network and the diffusion model are trained by a training unit based on a preset semi-synthetic data set.

[0029] Preferably, the preprocessing unit is specifically configured to input the to-be-reconstructed microscopic image into the pre-trained residual channel attention network for denoising and prediction to obtain the first preprocessing result, the first preprocessing result at least including a first denoised image obtained by denoising and a first point spread function obtained by prediction.

[0030] Preferably, the training unit includes:

[0031] a construction module configured to construct a semi-synthetic data set covering subcellular structures based on an optical degradation model of a microscopic imaging system, the semi-synthetic data set at least including original images of different signal-to-noise ratios and resolutions;

[0032] a first training module configured to train a residual channel attention network using the semi-synthetic data set and a loss function until convergence;

[0033] an acquisition module configured to acquire a second preprocessing result, the second preprocessing result being obtained by preprocessing original images in the semi-synthetic data set by the converged residual channel attention network, the second preprocessing result at least including a second denoised image obtained by denoising and a second point spread function obtained by prediction;

[0034] a second training module configured to train a diffusion model using the second preprocessing result until convergence.

[0035] A third aspect of the embodiment of the present application discloses an electronic device, comprising a processor and a memory, the processor and the memory are connected through a communication bus; wherein the processor is used to call and execute the program stored in the memory; the memory is used to store the program, the program is used to realize the super-resolution fluorescence image reconstruction method based on the diffusion model disclosed in the first aspect of the embodiment of the present application.

[0036] The fourth aspect of the embodiment of the present application discloses a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the super-resolution fluorescence image reconstruction method based on a diffusion model disclosed in the first aspect of the embodiment of the present application.

[0037] Based on the super-resolution fluorescence image reconstruction method based on a diffusion model and the related equipment provided by the embodiment of the present application, the method is as follows: obtaining a to-be-reconstructed microscopic image; inputting the to-be-reconstructed microscopic image into a pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result; and inputting the first preprocessing result into a reverse stage of a pre-trained diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the to-be-reconstructed microscopic image. The present scheme uses the progressive generation mechanism of the diffusion model and the preprocessing result of the residual channel attention network to perform image reconstruction, effectively overcomes the problem of neural network spectral bias, exhibits better detail restoration capability in image reconstruction, and improves the image reconstruction effect. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0039] Figure 1 A frequency spectrum bias phenomenon example diagram of a conventional end-to-end deep learning reconstruction method provided by the embodiment of the present application;

[0040] Figure 2 A flowchart of the super-resolution fluorescence image reconstruction method based on a diffusion model provided by the embodiment of the present application;

[0041] Figure 3 A flowchart of training a residual channel attention network and a diffusion model provided by the embodiment of the present application;

[0042] Figure 4 A principle example diagram of the super-resolution fluorescence image reconstruction method based on a diffusion model provided by the embodiment of the present application;

[0043] Figure 5 A reconstruction effect diagram for a local data set provided by the embodiment of the present application;

[0044] Figure 6 A reconstruction effect diagram for an external data set provided by the embodiment of the present application;

[0045] Figure 7A reconstruction effect picture in two-color dynamic imaging of living cells provided by the embodiment of the present application;

[0046] Figure 8 A structural block diagram of a super-resolution fluorescence image reconstruction device based on a diffusion model provided by the embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0048] In the present application, the term "comprising", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitation, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0049] Super-resolution fluorescence microscopy breaks through the constraint of the optical diffraction limit, and provides a powerful observation tool for exploring the physiological activities of nanoscale subcellular structures. However, super-resolution imaging is not "what you see is what you get", but is severely dependent on the reconstruction processing of the post-processing algorithm, which puts high requirements on the performance of the algorithm and the quality of the original imaging results.

[0050] With the maturity of different types of super-resolution fluorescence microscopes in optical principles and hardware systems, the reconstruction algorithm has become the main factor restricting the performance of super-resolution imaging. Most of the traditional physical-based image reconstruction algorithms require accurate optical parameters and time-consuming iterative operations, and are greatly affected by noise, aberration and other adverse factors in actual application, which greatly limits the observation ability of super-resolution microscopes under the demand of low light toxicity and photobleaching.

[0051] It is found through research that in recent years, deep learning has been widely applied to super-resolution reconstruction of microscopic images, and has shown significant robustness, high efficiency and flexibility superior to traditional methods in low photon budget and large volume data reconstruction scenarios. However, current deep learning methods are mostly based on end-to-end supervised learning strategies, such as Figure 1An example of spectral bias of the traditional end-to-end deep learning reconstruction method is shown. The end-to-end supervised learning strategy makes these deep learning methods inevitably subject to the spectral bias inherent in the neural network, and the reconstruction effect of high-frequency detail information is still not ideal, resulting in a large room for improvement in the resolution enhancement performance of these methods. In addition, due to the lack of variety and quantity of training data, the generalization of such data-driven image reconstruction methods is extremely limited, so that deep learning-based reconstruction methods can only be applied to specific data modalities locally.

[0052] With the development of computer vision technology, the paradigm of super-resolution reconstruction task has also ushered in new breakthroughs. Generative artificial intelligence technologies such as generative adversarial networks and diffusion models have shown significantly better detail reasoning ability than past methods when applied to super-resolution reconstruction, greatly improving the resolution deficiency, inconsistent distribution, and loss of weak signals of super-resolution reconstruction. Whether and how advanced deep learning models based on such super-resolution reconstruction algorithms can be used for fluorescence microscopy has gradually attracted the attention of researchers and has become a major research hotspot and development route for artificial intelligence-enabled microscopy.

[0053] Based on this, the present scheme proposes a super-resolution fluorescence image reconstruction method based on a diffusion model and related equipment, which uses the progressive generation mechanism of the diffusion model and the preprocessing results of the residual channel attention network to perform image reconstruction, effectively overcoming the problem of neural network spectral bias, and showing better detail restoration ability in image reconstruction, improving the image reconstruction effect.

[0054] Further, the present scheme combines the probability generation ability of the diffusion model with the adaptive optimization mechanism of the self-boosting degradation learning, aiming to achieve high-fidelity and high-robustness super-resolution reconstruction, break through the resolution limit of traditional methods, and reduce the dependence on training data modalities, improving the applicability of the model in complex scenarios such as low signal-to-noise ratio and dynamic live cell imaging.

[0055] After applying the present scheme, the following problems can be solved:

[0056] (1) Traditional physical-based algorithms rely on accurate optical parameters and complex iterative operations, making it difficult to deal with noise, aberration and other interference, resulting in low reconstruction efficiency and insufficient robustness;

[0057] (2) Existing deep learning methods have poor high-frequency detail reconstruction due to spectral bias, and are limited by the size and modalities of training data, with weak generalization ability;

[0058] (3) Generative models (such as GAN) are prone to introduce artifacts or lose weak signals in fluorescence microscopy image reconstruction, making it difficult to balance detail reasoning and distribution consistency.

[0059] Overall, the scheme filters out the noise of the original imaging result adaptively and predicts the corresponding point spread function (PSF), guides the diffusion model to sample reasonable high-resolution reconstruction results, and the following embodiments will be described in detail.

[0060] Referring to Figure 2 , a flowchart of a super-resolution fluorescence image reconstruction method based on a diffusion model is shown, and the super-resolution fluorescence image reconstruction method comprises the following steps:

[0061] Step S201: Obtain a micro image to be reconstructed.

[0062] In the process of implementing step S201, a micro image to be reconstructed is obtained, which is a low-quality micro image (original imaging result) obtained in actual imaging application.

[0063] Step S202: input the micro image to be reconstructed into a pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result.

[0064] In some embodiments, before step S202 is performed, a semi-synthetic data set is constructed, and the residual channel attention network (RCAN) and the diffusion model (also known as denoising diffusion probability model, DDPM) are trained using the constructed semi-synthetic data set, that is, the residual channel attention network and the diffusion model are trained based on the pre-set semi-synthetic data set, and subsequently Figure 3 How to construct a semi-synthetic data set, how to train RCAN, and how to train DDPM are described in detail.

[0065] In the process of implementing step S202, the micro image to be reconstructed is input into the pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result, wherein the preprocessing includes denoising and PSF prediction processing.

[0066] Specifically, the micro image to be reconstructed is input into the pre-trained residual channel attention network (RCAN) for denoising and prediction to obtain a first preprocessing result, and the first preprocessing result at least includes a first denoised image obtained by denoising and a first point spread function (first PSF) obtained by prediction.

[0067] That is, the micro image to be reconstructed is inferred by RCAN to obtain a first denoised image and a first point spread function output by RCAN.

[0068] Step S203: input the first pre-processing result into the reverse stage of the pre-trained diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the to-be-reconstructed microscopic image.

[0069] In the implementation of step S203, the first pre-processing result is input into the reverse stage of the diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image (a super-resolution fluorescence image) corresponding to the to-be-reconstructed microscopic image.

[0070] Specifically, in the reverse stage (reverse denoising stage) of the diffusion model, the first denoised image output by the residual channel attention network (RCAN) and the first point spread function are input as conditions, and the Unet in the diffusion model iteratively denoises and samples a distribution of high-resolution data, which is the reconstructed super-resolution fluorescence image corresponding to the to-be-reconstructed microscopic image.

[0071] It should be noted that, in the process of training the diffusion model, the case of gradually adding Gaussian noise in the forward stage of the diffusion model will be involved (which will be described in detail in Figure 3 later), based on which, in actual application, the first denoised image and the first point spread function enter the diffusion model in a direction process for the Unet to iteratively denoise and sample a distribution of high-resolution data that conforms to the training, and the specific content of the distribution of high-resolution data is shown in formula (1).

[0072] (1);

[0073] In formula (1), x t-1 and x t are the results of the previous time step and the current time step in the iterative denoising process, z is the distribution of the Gaussian noise sampling, is the noise predicted by the network, x deno is the denoising result of the RCAN (such as the first denoised image), p is the PSF prediction result of the RCAN (such as the first point spread function), t is the noise adding time step, is the noise adding scheduling factor (generally set as a decreasing arithmetic sequence), T is the total noise adding time step, is the cumulative multiplication, .

[0074] The first term in formula (1) is equivalent to the mean value of the sampling distribution at time t-1, which is predicted by the Unet. The in the second term in formula (1) is equivalent to the standard deviation of the distribution, which can be selected according to the certainty requirement of the reconstruction result.

[0075] It should be noted that the reconstruction paradigm based on the DDPM given by the above formula (1) naturally constitutes a probabilistic reconstruction, which can model the pixel-level reconstruction uncertainty while giving the fluorescence intensity distribution. The change distribution of multiple different sampling results of the same field of view can quickly show the different reconstruction reliabilities of different regions, helping users identify potential reasoning errors.

[0076] In the embodiment of the application, the image reconstruction is performed using the progressive generation mechanism of the diffusion model and the preprocessing result of the residual channel attention network, effectively overcoming the problem of neural network spectral bias, exhibiting better detail restoration capability in image reconstruction, and improving the image reconstruction effect.

[0077] For the above-mentioned embodiments of the application Figure 2 The residual channel attention network and the diffusion model involved in steps S202 and S203 are described in detail in the following Figure 3 , a flowchart for training the residual channel attention network and the diffusion model is shown, including the following steps:

[0078] Step S301: Based on the optical degradation model of the microscopic imaging system, a semi-synthetic data set covering subcellular structures is constructed.

[0079] In the process of implementing step S301, based on the optical degradation model of the microscopic imaging system, different Gaussian blur kernels (bandwidth, orientation adjustable) and mixed noise (such as Poisson shot noise + Gaussian read noise) are simulated; combined with the Gaussian blur kernel, the mixed noise and the real multi-signal-to-noise ratio wide-field image (such as a dynamic sequence of living cells), a semi-synthetic data set covering subcellular structures is constructed, which at least includes original images of different signal-to-noise ratios and resolutions.

[0080] It should be noted that the cell structure includes but is not limited to clathrin-coated pits, endoplasmic reticulum, microtubules, etc.

[0081] The real multi-signal-to-noise ratio wide-field image means that PSF degradation is performed on the real noise, that is, optical simulation degradation is performed using the blur kernel.

[0082] In some embodiments, the optical degradation model of the microscopic imaging system can be represented by formula (2).

[0083] (2);

[0084] In formula (2), I is the actual imaging result, I0 is the ideal imaging, P is the PSF of the microscopic imaging system, △p is the defocus, coma and other aberrations, and N is the photoelectron noise.

[0085] To fully simulate the degradation model under different systems and different optical parameters, the scheme adds slight noise to the Gaussian blur kernel with different bandwidths and different orientations to cover all possible PSFs of real systems. N is modeled as a mixed noise of different degrees of Poisson shot noise and Gaussian readout noise.

[0086] In addition to the simulation data, paired wide-field images with different signal-to-noise ratios collected under different excitation light levels are also retained in the training of the model, enhancing the model's learning ability for real degradation. Depending on the structural complexity, the constructed semi-synthetic data set contains five types of subcellular structures: clathrin-coated pits, endoplasmic reticulum, microtubules, myosin, and actin fibers.

[0087] Step S302: Train the residual channel attention network using the semi-synthetic data set and the loss function until convergence.

[0088] It should be noted that the residual channel attention network (RCAN) is a cascade combination of multiple residual modules based on a convolutional neural network, and each residual module contains multiple sub-residual modules. In order to enhance the extraction ability of key features, each sub-residual module is fused with a channel attention mechanism to adaptively scale the weights of different levels of manifold in different channels.

[0089] The scheme uses a residual channel attention network (RCAN) as a preprocessing module, which adaptively extracts key image features by stacking multiple residual groups. In the residual block, a channel attention module (SE-block) is embedded to dynamically adjust the feature channel weight and enhance the preservation ability of high-frequency details.

[0090] In the process of implementing step S302, the semi-synthetic data set constructed and the loss function are used to train the residual channel attention network until convergence.

[0091] In some specific embodiments, during the training of the residual channel attention network, the residual channel attention network is trained to denoise the "test original image" (such as the collected structured illumination microscopy image) and predict the corresponding PSF.

[0092] The loss function for supervised learning of the residual channel attention network is detailed in equation (3).

[0093] L=||x-y||1+α||p-p y ||1(3);

[0094] In equation (3), x and p are the dual-channel outputs of the residual channel attention network, and x and p can represent the denoising result (i.e., the denoised image) and the PSF prediction result, respectively. y and p yrespectively, and a is a weight factor balancing the learning of the two terms, and L is the loss.

[0095] In practical applications of reconstructed images, the pre-processing of RCAN reduces the risk of photoelectron noise being calculated as artifacts in the reconstruction process, and the simultaneously predicted PSF characterizes the distance between different modalities, different parameters of input and target distribution, laying a foundation for high-quality sampling of subsequent diffusion models.

[0096] Step S303: Obtain a second pre-processing result.

[0097] In the process of specifically implementing step S303, a second pre-processing result is obtained, which is obtained by pre-processing the original image in the semi-synthetic data set by the converged residual channel attention network, and the second pre-processing result at least includes a second denoising image obtained by denoising and a second point spread function obtained by prediction.

[0098] That is, the second pre-processing result is that the original image in the semi-synthetic data set is input into the converged residual channel attention network for pre-processing, and the second denoising image and the second point spread function output by the residual channel attention network.

[0099] Step S304: training the diffusion model using the second pre-processing result until convergence.

[0100] In the process of specifically implementing step S304, a diffusion model is constructed, and in the forward stage of the diffusion model, Gaussian noise is gradually added to the sampling target of the diffusion model until the sampling target becomes pure noise, and the sampling target is the true value distribution.

[0101] Specifically, the high-resolution true value distribution is used as the sampling target of the diffusion model (i.e., the target distribution, denoted as x0), as shown in formula (4), Gaussian noise (denoted as ε) is gradually added to x0 in the forward stage of the diffusion model until x0 becomes pure noise.

[0102] (4);

[0103] In formula (4), is multiplied, is a noise adding scheduling factor (usually set as a decreasing arithmetic sequence), T is the total time step of noise adding, t is the noise adding time step, and x t is the result of x0 after noise adding.

[0104] In the reverse stage of the diffusion model, the second pre-processing result is input into the diffusion model, so that the diffusion model gradually restores the sampling target by predicting the Gaussian noise ε added in the forward stage, to complete the training of the diffusion model.

[0105] That is, based on the second preprocessed result (the second denoised image and the second point spread function) output by the residual channel attention network in the reverse stage of the diffusion model, the sampling target is gradually restored by predicting the Gaussian noise ε added in the forward stage.

[0106] Specifically, in the reverse stage of the diffusion model, the second preprocessed result output by the residual channel attention network is input as a condition, and the generation process is constrained by the cross-attention mechanism to ensure that the reconstruction result conforms to the real physical degradation model. The training target of the diffusion model is represented as formula (5).

[0107] (5);

[0108] In formula (5), θ is the noise prediction network of the Unet structure, is the noise predicted by the network, t is the noise time step, x deno and p are the denoising result (such as the second denoised image) and the PSF prediction result (such as the second point spread function) of RCAN, respectively.

[0109] Through the above method, the training of the diffusion model is completed.

[0110] In summary, the present scheme synthesizes original images of different signal-to-noise ratios and resolutions based on the classical model of image degradation, and combines real paired wide-field images of high and low signal-to-noise ratios to guide RCAN for denoising and degradation prediction supervised learning. Subsequently, the diffusion model is trained to sample the reconstruction result conforming to the true value distribution from the Gaussian noise based on the output of RCAN as a condition.

[0111] To better understand the overall process of the present scheme, the following processes A1-A4 are used as examples.

[0112] A1, construct a semi-synthetic dataset, wherein the construction process of the semi-synthetic dataset is described in detail in the above embodiment of the present application Figure 3 The content in step S301.

[0113] A2, based on the semi-synthetic dataset, train RCAN, and the specific training process is described in detail in the above embodiment of the present application Figure 3 The content in step S302.

[0114] A3, train the diffusion model, and the specific training process is described in detail in the above embodiment of the present application Figure 3 The content in step S303.

[0115] A4, for the low-quality microscopic image obtained in the actual imaging application, first input the RCAN inference to denoise the clean result (the first denoised image) and obtain the first point spread function, and then enter the reverse stage of the diffusion model to let the Unet iteratively denoise and sample the distribution of the high-resolution data used for training, so as to obtain the reconstructed super-resolution fluorescence image. The specific content of the distribution of the high-resolution data is shown in formula (1).

[0116] Overall, as Figure 4 The principle example diagram of the super-resolution fluorescence image reconstruction method based on the diffusion model is shown. The above processes A1-A4 can be divided into a training stage and an inference stage.

[0117] In the training stage, the data required for training includes true value, simulated wide-field image, simulated noisy wide-field image, real noisy wide-field image and real wide-field image; RCAN is trained using the data required for training, and the output of RCAN is used as the input condition of the diffusion model to train the diffusion model.

[0118] In the inference stage, the RCAN denoises and predicts the original input, and the denoised result and the predicted point spread function are used as the input condition of the diffusion model, so as to obtain the super-resolution reconstruction result.

[0119] In order to reflect the reconstruction effect of the image of the present scheme, the following content is used for example.

[0120] As Figure 5 The reconstruction effect diagram for the local data set is shown. The images in the "original input" column are the microscopic images to be reconstructed, the images in the "super-resolution reconstruction" column are the super-resolution fluorescence images reconstructed by the present scheme (corresponding to the original images), and the images in the "true value" column are the real images.

[0121] As Figure 6 The reconstruction effect diagram for the external data set is shown. The images in the "original input" column are the microscopic images to be reconstructed, the images in the "traditional structured light microscope reconstruction" are the images reconstructed by the prior art, and the images in the "super-resolution reconstruction" column are the super-resolution fluorescence images reconstructed by the present scheme.

[0122] As Figure 7 The reconstruction effect diagram in live cell two-color dynamic imaging is shown. In live cell two-color dynamic imaging, the images in the "original input" row are the microscopic images to be reconstructed, and the images in the "super-resolution reconstruction" row are the super-resolution fluorescence images reconstructed by the present scheme.

[0123] The above Figures 5-7As can be seen from the content shown, the scheme has better detail restoration capability in image reconstruction, has better image reconstruction effect, and in detail, the scheme has the following beneficial effects:

[0124] 1. The scheme proposes a progressive generation mechanism of the diffusion model combined with the PSF condition constraint, effectively overcomes the neural network spectrum bias problem, and exhibits better detail restoration capability in the reconstruction of subcellular structures (such as microtubule edges and actin fibers);

[0125] 2. The self-heuristic degradation learning trains the RCAN by combining the semi-synthetic data set with the real data, so that the RCAN can adaptively filter out mixed noise and predict diversified PSF, significantly reduces the dependence of the model on specific data modalities, and still maintains high accuracy in the reconstruction of unknown samples (out-of-domain data sets);

[0126] 3. The combination of the physical-based degradation modeling and the data-driven generation strategy can be extended to multiple microscopic imaging modes such as structured light and confocal, and provides a unified framework for cross-platform super-resolution reconstruction.

[0127] Corresponding to the super-resolution fluorescence image reconstruction method based on the diffusion model provided by the above-mentioned embodiment of the present application, referring to Figure 8 The embodiment of the present application also provides a structural diagram of a super-resolution fluorescence image reconstruction device based on a diffusion model, which comprises an acquisition unit 100, a preprocessing unit 200 and a reconstruction unit 300.

[0128] The acquisition unit 100 is used to acquire a microscopic image to be reconstructed.

[0129] The preprocessing unit 200 is used to input the microscopic image to be reconstructed into a pre-trained residual channel attention network for preprocessing to obtain a first preprocessing result.

[0130] In a specific implementation, the preprocessing unit 200 is specifically configured to input the microscopic image to be reconstructed into the pre-trained residual channel attention network for denoising and prediction to obtain the first preprocessing result, and the first preprocessing result at least includes a first denoised image obtained by denoising and a first point spread function obtained by prediction.

[0131] The reconstruction unit 300 is used to input the first preprocessing result into a pre-trained reverse stage of the diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the microscopic image to be reconstructed.

[0132] Among them, the residual channel attention network and the diffusion model are trained by the training unit based on the pre-set semi-synthetic data set.

[0133] In the embodiment of the present application, the progressive generation mechanism of the diffusion model and the preprocessing result of the residual channel attention network are used for image reconstruction, effectively overcoming the problem of neural network spectral bias, exhibiting better detail restoration capability in image reconstruction, and improving the image reconstruction effect.

[0134] Preferably, based on the content of the super-resolution fluorescence image reconstruction device described above, the training unit includes a construction module, a first training module, an acquisition module and a second training module, and the execution principles of each module are as follows:

[0135] The construction module is configured to construct a semi-synthetic data set covering subcellular structures based on an optical degradation model of a microscopic imaging system, and the semi-synthetic data set at least includes original images of different signal-to-noise ratios and resolutions.

[0136] The construction module is specifically configured to simulate different Gaussian blur kernels and mixed noise based on the optical degradation model of the microscopic imaging system; and combine the Gaussian blur kernel, the mixed noise and the real multi-signal-to-noise ratio wide-field image to construct the semi-synthetic data set covering the subcellular structures.

[0137] The first training module is configured to train the residual channel attention network using the semi-synthetic data set and a loss function until convergence.

[0138] The acquisition module is configured to acquire a second preprocessing result, which is obtained by preprocessing the original images in the semi-synthetic data set by the converged residual channel attention network, and the second preprocessing result at least includes a second denoising image obtained by denoising and a second point spread function obtained by prediction.

[0139] The second training module is configured to train the diffusion model using the second preprocessing result until convergence.

[0140] The second training module is specifically configured to: in the forward stage of the diffusion model, gradually add Gaussian noise to the sampling target of the diffusion model until the sampling target becomes pure noise, and the sampling target is a true value distribution; in the reverse stage of the diffusion model, input the second preprocessing result into the diffusion model, so that the diffusion model gradually restores the sampling target by predicting the Gaussian noise added in the forward stage, to complete the training of the diffusion model.

[0141] Preferably, the embodiment of the present application further provides an electronic device, which comprises a processor and a memory connected through a communication bus; wherein the processor is configured to call and execute a program stored in the memory; and the memory is configured to store the program, and the program is configured to implement the super-resolution fluorescence image reconstruction method based on the diffusion model provided by the method embodiment.

[0142] Preferably, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the super-resolution fluorescence image reconstruction method based on a diffusion model provided by the method embodiment.

[0143] To sum up, the embodiment of the present application provides a super-resolution fluorescence image reconstruction method based on a diffusion model and related equipment, which uses the progressive generation mechanism of the diffusion model and the preprocessing result of the residual channel attention network to perform image reconstruction, effectively overcomes the problem of neural network spectral bias, and exhibits better detail restoration capability in image reconstruction, thereby improving the image reconstruction effect.

[0144] Each embodiment in the specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, it is described more simply, and the related parts can be referred to the part of the method embodiment. The system and system embodiment described above are only illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0145] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical scheme. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0146] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for super-resolution reconstruction of fluorescence images based on a diffusion model, characterized in that, The method comprises: acquiring a to-be-reconstructed microscopic image; inputting the to-be-reconstructed microscopic image into a pre-trained residual channel attention network for denoising and prediction to obtain a first preprocessing result, the first preprocessing result at least comprising a first denoised image obtained by denoising and a first point spread function obtained by prediction; inputting the first preprocessing result into a reverse stage of a pre-trained diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the to-be-reconstructed microscopic image; wherein the residual channel attention network and the diffusion model are trained based on a preset semi-synthetic data set; The process of training the residual channel attention network and the diffusion model based on the preset semi-synthetic data set comprises: based on an optical degradation model of a microscopic imaging system, constructing a semi-synthetic data set covering subcellular structures, the semi-synthetic data set at least comprising original images of different signal-to-noise ratios and resolutions; training the residual channel attention network using the semi-synthetic data set and a loss function until convergence; obtaining a second preprocessing result, the second preprocessing result being obtained by preprocessing the original images in the semi-synthetic data set by the converged residual channel attention network, the second preprocessing result at least comprising a second denoised image obtained by denoising and a second point spread function obtained by prediction; training the diffusion model using the second preprocessing result until convergence, wherein in the reverse stage of the diffusion model, the generation process is constrained by a cross-attention mechanism.

2. The method of claim 1, wherein, Based on the optical degradation model of the microscopic imaging system, the semi-synthetic data set covering the subcellular structure is constructed, comprising: based on the optical degradation model of the microscopic imaging system, simulating different Gaussian blur kernels and mixed noise; combining the Gaussian blur kernel, the mixed noise and the real multi-signal-to-noise ratio wide-field image to construct the semi-synthetic data set covering the subcellular structure.

3. The method of claim 1, wherein, Training the diffusion model using the second preprocessing result until convergence, comprising: in the forward stage of the diffusion model, gradually adding Gaussian noise to the sampling target of the diffusion model until the sampling target becomes pure noise, the sampling target being a true value distribution; in the reverse stage of the diffusion model, inputting the second preprocessing result into the diffusion model, so that the diffusion model gradually restores the sampling target by predicting the Gaussian noise added in the forward stage to complete the training of the diffusion model.

4. An apparatus for super-resolution reconstruction of a fluorescence image based on a diffusion model, characterized in that, The device comprises: an acquisition unit configured to acquire a to-be-reconstructed microscopic image; a preprocessing unit configured to input the to-be-reconstructed microscopic image into a pre-trained residual channel attention network for denoising and prediction to obtain a first preprocessing result, the first preprocessing result at least comprising a first denoised image obtained by denoising and a first point spread function obtained by prediction; a reconstruction unit configured to input the first preprocessing result into a reverse stage of a pre-trained diffusion model for iterative denoising sampling to obtain a reconstructed super-resolution fluorescence image corresponding to the to-be-reconstructed microscopic image; wherein the residual channel attention network and the diffusion model are trained by a training unit based on a preset semi-synthetic data set; The training unit comprises: a construction module, configured to construct a semi-synthetic dataset covering subcellular structures based on an optical degradation model of a microscopic imaging system, the semi-synthetic dataset comprising at least original images of different signal-to-noise ratios and resolutions; a first training module, configured to train a residual channel attention network using the semi-synthetic dataset and a loss function until convergence; an acquisition module, configured to acquire a second preprocessing result, the second preprocessing result being obtained by preprocessing the original images in the semi-synthetic dataset by the residual channel attention network after convergence, the second preprocessing result comprising at least a second denoising image obtained by denoising and a second point spread function obtained by prediction; a second training module, configured to train a diffusion model using the second preprocessing result until convergence, wherein, in a reverse stage of the diffusion model, a cross-attention mechanism is used to constrain a generation process.

5. An electronic device, comprising: comprise: a processor and a memory, which are connected through a communication bus; wherein the processor is used to call and execute a program stored in the memory; the memory is used to store a program, and the program is used to implement the super-resolution fluorescence image reconstruction method based on a diffusion model according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the super-resolution fluorescence image reconstruction method based on a diffusion model according to any one of claims 1-3.

Images