A fluorescence image super-resolution restoration method and system based on a diffusion model

Abstract

This invention discloses a method and system for super-resolution restoration of fluorescence images based on a diffusion model, comprising: acquiring a noisy low-resolution image to be processed; fusing the noisy low-resolution image to be processed with randomly generated Gaussian-distributed noise data as input to a trained diffusion model; progressively performing inverse inference denoising on the noisy low-resolution image through the diffusion model; and outputting a restored super-resolution fluorescence image. This invention separately calculates the high-frequency components (representing the edges and details of the observed object) and low-frequency components (representing the main structure of the observed object) of the input image using multi-head attention. This separate processing allows for enhancement of high-frequency components when processing the high-frequency components, making these important details clearer, while avoiding over-processing of the entire image, which could lead to artifacts or unnecessary detail enhancement in the low-frequency components; it also avoids wasting computational resources on low-frequency background components that do not need processing.

Description

Technical Field

[0001] This invention relates to the field of computer vision. It discloses a method for reconstructing low-quality fluorescence microscopy images using a diffusion model. Specifically, it involves collecting various types of fluorescence microscopy image datasets and training the image with artificially synthesized datasets to improve resolution and remove noise from the fluorescence microscopy images. Background Technology

[0002] Microscopic images have important applications in fields such as medicine, materials science, and biology. However, obtaining high-quality fluorescence microscopic images is often challenging due to limitations in microscope hardware: high-resolution images often contain high noise, while low-noise images often do not meet the ideal resolution requirements.

[0003] In recent years, with the development of deep learning, it has demonstrated powerful capabilities in image processing, especially in improving image resolution and denoising. Building optical microscopes typically requires complex optical systems and incurs high costs, while deep learning methods can achieve resolution improvements at a lower cost and with simpler hardware. The success of this deep learning approach is expected to have a significant impact on the life sciences and microscopy technologies. It may provide researchers with a wider range of more economical and user-friendly high-resolution imaging tools, helping to advance scientific research.

[0004] The diffusion model, proposed by Jonathan Ho in 2020, is a generative model that generates data samples from pure noise through a progressive denoising process. Each step makes subtle adjustments based on the state of the previous step, gradually reconstructing a high-quality image or other data format. The diffusion model is now widely used in image, audio, and text processing, achieving good results. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and system for super-resolution restoration of fluorescence images based on a diffusion model.

[0006] Due to the sparsity of fluorescence microscopy images, most of the image consists of background, with only a small portion representing the observed object. This results in relatively independent high-frequency and low-frequency features. The low-frequency component typically contains large-scale structures and overall intensity variations, often representing the background and larger biological tissues or structures; while the high-frequency component mainly contains detailed information and edge features, and is also where noise is concentrated. Feeding the entire image into the network would waste significant computational resources due to the background occupying most of the image. Based on these characteristics of microscopic images, this invention primarily designs a module for extracting image features. This module can separately compute the high-frequency components (representing the edges and details of the observed object) and the low-frequency components (representing the main structure of the observed object) of the input image using multi-head attention. This separate processing allows for enhancement of the high-frequency components, making these important details clearer, while avoiding over-processing the entire image, which could lead to artifacts or unnecessary detail enhancement in the low-frequency components. It also avoids wasting significant computational resources on the low-frequency background that doesn't need processing. The technical solution adopted in this invention is as follows:

[0007] A fluorescence image super-resolution restoration method based on a diffusion model includes:

[0008] Acquire the noisy, low-resolution image to be processed;

[0009] The noisy low-resolution image to be processed is fused with a randomly generated Gaussian-distributed noise data as input to a trained diffusion model. The diffusion model is then used to progressively infer and denoise the noisy low-resolution image, and the output is a restored super-resolution fluorescence image.

[0010] The diffusion model includes a noise prediction network and a denoising network. The noise prediction network is used to predict noise based on the input image. The noise prediction network includes several encoder layers, intermediate layers, decoder layers, and an output layer. The first encoder layer contains an embedded attention module, which includes parallel first and second channels. The first channel contains a sliding window module, a query-key-value calculation module, a scaled dot product attention module, a connection layer, and a projection layer connected in sequence. The second channel contains a query-key-value calculation module, a sliding window dilation attention module, a connection layer, and a projection layer connected in sequence. One channel is used to segment the high-frequency part of the input feature map into multiple local windows. Within each window, a linear transformation is performed to obtain the query q, key k, v value vector, and local attention is calculated to capture local detail information. The second channel is used to perform a linear transformation on the low-frequency part of the input feature map to obtain the query q, key k, v value vector, and then perform attention calculation through multiple dilation convolutions with different dilation rates, fusing feature information from different receptive fields. The information output from the first and second channels is fused and used as the output of the attention module. The denoising network is used to denoise the input image based on the noise predicted by the noise prediction network to obtain the image for the next step.

[0011] Experiments have shown that by incorporating the module of this invention into the diffusion model, even with the SR3 network depth as a baseline halved, network performance is improved, significantly reducing the required computing resources while maintaining network performance.

[0012] Furthermore, the denoising network performs denoising in the following manner:

[0013]

[0014] Where, α t =1-β t Let z represent a random noise that follows a Gaussian distribution. T represents the number of denoising steps, and x0 represents the restored super-resolution fluorescence image output by the diffusion model; x T Represents the noisy, low-resolution image to be processed, ∈ θ It is a noise prediction network, ∈ θ (*) indicates the output of the noise prediction network.

[0015] Furthermore, the diffusion model is obtained by training the dataset by minimizing the difference between the noise output of the noise prediction network and the real noise.

[0016] Furthermore, each sample in the dataset contains an image pair consisting of a noisy low-resolution image and a high-resolution image. Further, the loss function representing the difference between the noise output by the noise prediction network and the true noise is expressed as follows:

[0017]

[0018] Where ∈ θ It is a noise prediction network, ∈ θ (*) represents the output of the noise prediction network, ∈0 is the noise distribution taken from the image pair at medium to high resolution, x t This represents the input image of the diffusion model at step t, where x0 represents x. t The image undergoes progressive inverse inference denoising, and the diffusion model ultimately outputs a restored super-resolution fluorescence image; q(x0) represents the probability distribution of the x0 image, U{1,T} represents a uniform distribution from 1 to T, and q(x t |x0) represents the condition that x0 is known for x. t Perform distribution prediction, where E represents the expectation, ||*||1 represents the L1 loss, and ||*||2 represents the L2 loss.

[0019] Furthermore, the encoder layer, intermediate layer, and decoder layer are each composed of several residual blocks (ResBlocks).

[0020] Furthermore, the residual block ResBlock is composed of GroupNorm layer, Swish layer, 2D convolutional layer, affine transformation layer, GroupNorm layer, Swish layer, Dropout layer, and 2D convolutional layer connected by residuals.

[0021] A fluorescence image super-resolution restoration system based on a diffusion model includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the fluorescence image super-resolution restoration method based on a diffusion model.

[0022] A storage medium containing computer-executable instructions that, when executed by a computer processor, implement the aforementioned diffusion-based fluorescence image super-resolution restoration method.

[0023] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the diffusion-based fluorescence image super-resolution restoration method.

[0024] This invention can enhance a noisy, low-resolution image to generate a low-noise, high-resolution image. The invention uses multi-head attention to separately compute the high-frequency components (representing the edges and details of the observed object) and low-frequency components (representing the main structure of the observed object) of the input image. This separate processing allows for enhancement of high-frequency components, making these important details clearer, while avoiding over-processing the entire image, which could lead to artifacts or unnecessary detail enhancement in the low-frequency components. It also avoids wasting computational resources on low-frequency background areas that don't need processing. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating the forward diffusion process of a diffusion model according to an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating the reverse diffusion process of a diffusion model according to an embodiment of the present invention, i.e., the implementation of the method of the present invention.

[0027] Figure 3 This is a schematic diagram of the structure of a noise prediction network according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of the attention module in one embodiment of the method of the present invention;

[0029] Figure 5 The diagram illustrates the generation effect of the method and comparative model of this invention. Detailed Implementation

[0030] To improve the image quality of fluorescence microscopy and address the issues of high noise in high-resolution fluorescence microscopy images and insufficient resolution in low-noise images, this invention proposes a super-resolution restoration method for fluorescence microscopy images based on a diffusion model. This method leverages the sparsity of fluorescence microscopy images by embedding an attention module into the diffusion model. The high-frequency components (representing the edges and details of the observed object) and low-frequency components (representing the main structure of the observed object) of the input image are calculated separately using multi-head attention. Then, the trained diffusion model is used to generate high-resolution, low-noise images.

[0031] For initial use, the method of this invention requires training and then using the trained diffusion model to generate high-resolution, low-noise images; subsequently, the trained diffusion model can be used directly to generate high-resolution, low-noise images. The following embodiment, using initial use as an example, describes a diffusion model-based fluorescence microscopy image super-resolution restoration method of this invention, comprising the following steps:

[0032] Training steps for the diffusion model:

[0033] Step 1: Construct a dataset in which each sample consists of an image pair consisting of a noisy low-resolution image (LR) and a high-resolution image (HR);

[0034] For fluorescence images, the original images obtained by a real microscope can be preprocessed to obtain .tif format image files; low-resolution images can be acquired by confocal microscopy, while high-resolution images can be acquired by high-stretch microscopy.

[0035] In one specific implementation, a portion of the images can also be artificially synthesized to expand the dataset.

[0036] Step 2: Establishing the diffusion model:

[0037] In step 2, the established diffusion model is an SR3-based diffusion model, including a noise prediction network and a denoising network. The noise prediction network is used to predict noise based on the input image, and an attention module is embedded in the noise prediction network. Based on the multi-head attention improvement, the feature map is divided into high-frequency and low-frequency parts for calculation. The high-frequency part first divides the input tensor into multiple local windows, then performs a linear transformation within each window to obtain q, k, v, and performs local attention calculation, reducing computational complexity while still capturing local detail information. The low-frequency part incorporates multiple dilated convolutions with different dilation rates. Specifically, the attention module includes a first channel and a second channel in parallel. The first channel contains a sliding window module, a query-key-value calculation module, a scaled dot product attention module, a connection layer, and a projection layer connected in sequence. The second channel contains a query-key-value calculation module, a sliding window dilated attention module, a connection layer, and a projection layer connected in sequence. The first channel divides the high-frequency part of the input feature map into multiple local windows, performs a linear transformation within each window to obtain query q, key k, v value vectors, and performs local attention calculation to capture local detail information. The second channel performs a linear transformation on the low-frequency part of the input feature map to obtain query q, key k, v value vectors, then performs attention calculation through multiple dilated convolutions with different dilation rates, and fuses feature information from different receptive fields. The information from the first and second channels is fused and used as the output of the attention module. Figure 4The diagram illustrates the structure of an attention module in a specific embodiment. In the diagram, the sliding window module first divides the input tensor into multiple local windows. The window size can be determined experimentally; in this embodiment, a 2×2 window is used. Attention is calculated between these small windows. The query-key-value (QKV) module performs query-key-value vector calculation on the input features. The scaled dot product attention module performs local attention calculation on the input. The sliding window dilation attention module performs attention calculation on low-frequency components through multiple dilation convolutions with different dilation rates. The concatenation layer uses concatenation to combine the results of multiple attention heads from the scaled dot product attention module. Finally, the projection layer uses linear projection to map the dimension of the feature map to the original dimension. This attention module better integrates feature mappings from different receptive fields, effectively expanding the receptive field at each spatial location and enhancing feature extraction from targets at different scales, enabling the diffusion model to generate more refined images.

[0038] In a specific implementation scheme, the noise prediction network is as follows: Figure 3 As shown, the structure consists of 12 layers, including an encoder layer, intermediate layers, a decoder layer, and an output layer. Layers 1-5 are encoder layers, each consisting of a residual block (ResBlock). The ResBlock is constructed using GroupNorm + Swish + 2D convolution + affine transformation.

[0039] The system consists of GroupNorm, Swish, Dropout, 2D convolution, and residual connections. The first layer embeds the attention module designed in this invention; that is, the first encoder layer structure is ResBlock + attention module. The sixth layer is an intermediate layer, consisting of two...

[0040] The layers consist of ResBlocks; the last 5 layers are decoder layers, each consisting of one ResBlock; the output layer consists of...

[0041] Composed of GroupNorm, Swish, and 2D convolution.

[0042] Step 3: Training the diffusion model:

[0043] The diffusion model works by starting with a purely noisy image and gradually reducing the noise through a series of steps to eventually obtain a clear image. The main principle is as follows:

[0044] 1. Forward Diffusion Process:

[0045] 1.1 Sampling real images from a real image data distribution

[0046] 1.2 such as Figure 1 As shown, in each step, noise is gradually added to the image until a purely noisy image is obtained. This can be represented as a series of steps. in It is a purely noisy image.

[0047] 1.3 This process of adding noise is usually described using a Markov chain, where each transition is achieved by adding a small amount of Gaussian noise, i.e. Where β t It is a predefined noise scheduling parameter, which is set to 1×10 in this embodiment. -6 Up to 1×10 -2 The uniform distribution of length 2000, t is the number of steps, and ∈ is noise sampled from the standard normal distribution.

[0048] 2. Reverse diffusion process, i.e., the generation process:

[0049] 2.1 As Figure 2 As shown, the goal of the reverse diffusion process is to diffuse the pure noise image x T The noise is gradually removed, eventually generating a realistic image x0.

[0050] 2.2 This process is also implemented through Markov chains, but for each step of noise prediction and removal, a Transformer-based U-Net neural network is selected, namely the noise prediction network and the denoising network.

[0051] 2.3 Specifically, the low-resolution image in each sample image pair in the dataset is fused with randomly generated Gaussian-distributed noise data as input to the diffusion model. The output of the diffusion model is used as input to the next diffusion model. In each step t, the denoising network adjusts the noise based on the current image x. t Predict the denoised image x t-1 ,Right now

[0052]

[0053] Where, α t =1-β t Let z represent a random noise that follows a Gaussian distribution. ∈ θ It is a noise prediction network, ∈ θ (*) indicates the output of the noise prediction network, where the parameter θ is optimized by minimizing the loss function until the set number of iterations is reached:

[0054]

[0055] Where ∈0 is taken from the noise distribution of the high-resolution image in the image pair. Finally, from x t The iterative formula is repeated until x0 is reached, ultimately yielding the super-resolution image x0. q(x0) represents the probability distribution of the x0 image, where U{1,T} represents a uniform distribution from 1 to T. t |x0) represents the condition that x0 is known for x. t Perform distribution prediction, where E represents the expectation, ||*||1 represents the L1 loss, and ||*||2 represents the L2 loss.

[0056] In this embodiment, the training period is 500 epochs, and the best checkpoint is taken as the final result.

[0057] Steps to generate high-resolution, low-noise images using a trained diffusion model:

[0058] Acquire the noisy, low-resolution image to be processed;

[0059] Randomly generated Gaussian-distributed noisy data is fused with the noisy low-resolution image to be processed as input to the diffusion model. The diffusion model is then used to progressively infer and denoise the noisy low-resolution image, resulting in a restored super-resolution fluorescence image.

[0060] The diffusion model of this invention is compared with four deep learning methods (including three methods based on non-diffusion models: UNiEMIR, CARE, and SRGAN, and one method based on a diffusion model: SR3) in the super-resolution restoration problem. The model structure designed in this invention has good denoising function while ensuring a certain reconstruction accuracy. Figure 5 The visualization results of the reconstructed image show that the image restored by the present invention has clear structural details and no obvious artifacts.

[0061] The resolution evaluation metric used is the parameter-free image resolution estimation based on decorrelation analysis method published in Nature Methods in 2019; the signal-to-background ratio evaluation metric first defines 0 and its surrounding pixels as the background using a kernel of size 5. Where FG(i) represents the foreground pixel and BG(j) represents the background pixel. As shown in Table 2, the results of this invention in terms of signal-to-background ratio are superior to those of other models.

[0062] Table 1: Comparison of resolution between the present invention and the comparative method at a magnification of X2.

[0063]

[0064] Table 2: Comparison of signal-to-background ratio when the magnification is X2 for the present invention and the comparative method.

[0065]

[0066] In summary, this invention provides a diffusion-based super-resolution image processing method for fluorescence microscopy. The resulting super-resolution image possesses rich detail and texture features while also removing noise. The success of this deep learning method is expected to have a significant impact on the fields of life sciences and microscopy. It may provide researchers with a wider range of more economical and easier-to-use high-resolution imaging tools, contributing to the advancement of scientific research.

[0067] Corresponding to the aforementioned embodiment of a fluorescence image super-resolution restoration method based on a diffusion model, the present invention also provides an embodiment of a fluorescence image super-resolution restoration system based on a diffusion model.

[0068] The present invention provides a fluorescence image super-resolution restoration system based on a diffusion model, comprising one or more processors for implementing a fluorescence image super-resolution restoration method based on a diffusion model as described in the above embodiments.

[0069] An embodiment of the fluorescence image super-resolution restoration system based on a diffusion model of the present invention can be applied to any device with data processing capabilities, such as a computer or other equipment or apparatus.

[0070] The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device that houses the device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, it includes a processor, memory, network interface, and non-volatile memory. In addition, the data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be elaborated further.

[0071] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0072] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0073] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a fluorescence image super-resolution restoration method based on a diffusion model as described in the above embodiments.

[0074] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., mounted on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0075] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A fluorescence image super-resolution restoration method based on a diffusion model, characterized in that, include: Acquire the noisy, low-resolution image to be processed; The noisy low-resolution image to be processed is fused with a randomly generated Gaussian-distributed noise data as input to a trained diffusion model. The diffusion model is then used to progressively infer and denoise the noisy low-resolution image, and the output is a restored super-resolution fluorescence image. The diffusion model includes a noise prediction network and a denoising network. The noise prediction network is used to predict noise based on the input image. The noise prediction network includes several encoder layers, intermediate layers, decoder layers, and an output layer. The first encoder layer contains an embedded attention module, which includes parallel first and second channels. The first channel contains a sliding window module, a query-key-value calculation module, a scaled dot product attention module, a connection layer, and a projection layer connected in sequence. The second channel contains a query-key-value calculation module, a sliding window dilation attention module, a connection layer, and a projection layer connected in sequence. One channel is used to segment the high-frequency part of the input feature map into multiple local windows. Within each window, a linear transformation is performed to obtain the query q, key k, v value vector, and local attention is calculated to capture local detail information. The second channel is used to perform a linear transformation on the low-frequency part of the input feature map to obtain the query q, key k, v value vector, and then perform attention calculation through multiple dilation convolutions with different dilation rates, fusing feature information from different receptive fields. The information output from the first and second channels is fused and used as the output of the attention module. The denoising network is used to denoise the input image based on the noise predicted by the noise prediction network to obtain the image for the next step.

2. The method according to claim 1, characterized in that, The denoising network performs denoising in the following manner: Where, α t =1-β t ,β t is a predefined noise scheduling parameter, where z represents random noise that follows a Gaussian distribution. T represents the number of denoising steps, and x0 represents the restored super-resolution fluorescence image output by the diffusion model; x t Represents the input image of the diffusion model corresponding to step t, ∈ θ It is a noise prediction network, ∈ θ (*) indicates the output of the noise prediction network.

3. The method according to claim 1, characterized in that, The diffusion model is obtained by training the dataset by minimizing the difference between the noise output by the noise prediction network and the actual noise.

4. The method according to claim 3, characterized in that, Each sample in the dataset contains an image pair consisting of a noisy low-resolution image and a high-resolution image.

5. The method according to claim 3, characterized in that, The loss function representing the difference between the noise output by the noise prediction network and the actual noise is expressed as follows: Where ∈ θ It is a noise prediction network, ∈ θ (*) represents the output of the noise prediction network, ∈0 is the noise distribution taken from the image pair at medium to high resolution, x t This represents the input image of the diffusion model at step t, where x0 represents x. t The image undergoes progressive inverse inference denoising, and the diffusion model ultimately outputs a restored super-resolution fluorescence image; q(x0) represents the probability distribution of the x0 image, U{1,T} represents a uniform distribution from 1 to T, and q(x t |x0) represents the condition that x0 is known for x. t Perform distribution prediction, where E represents the expectation, ||*||1 represents the L1 loss, and ||*||2 represents the L2 loss.

6. The method according to claim 1, characterized in that, The encoder layer, intermediate layer, and decoder layer are each composed of several residual blocks (ResBlocks).

7. The method according to claim 1, characterized in that, The ResBlock is composed of GroupNorm layers, Swish layers, 2D convolutional layers, affine transformation layers, GroupNorm layers, Swish layers, Dropout layers, and 2D convolutional layers connected by residuals.

8. A fluorescence image super-resolution restoration system based on a diffusion model, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a fluorescence image super-resolution restoration method based on a diffusion model as described in any one of claims 1-7.

9. A storage medium containing computer-executable instructions, which, when executed by a computer processor, implement a diffusion-based fluorescence image super-resolution restoration method as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the fluorescence image super-resolution restoration method based on the diffusion model as described in any one of claims 1-7.

Images