Microscopic image super-resolution reconstruction method and device based on improved diffusion model

Abstract

The invention discloses a microscopic image super-resolution reconstruction method based on an improved diffusion model. The microscopic image super-resolution reconstruction method comprises the following steps: acquiring paired high-resolution images and corresponding low-resolution images as a data set; training an improved diffusion model by using the training set; a given low-resolution image is subjected to up-sampling by using an interpolation algorithm to obtain a high-resolution image, the low-resolution image and the corresponding high-resolution image are input into a trained improved diffusion model to obtain a corresponding super-resolution image, an efficient diffusion model is introduced, the reasoning efficiency is improved, and the reconstruction effect is good.

Description

Technical Field

[0001] The invention relates to the field of microscopic image processing, and in particular to a microscopic image super-resolution reconstruction method and device based on an improved diffusion model. Background Art

[0002] Image super-resolution technology is an active technology in the field of deep learning. It aims to reconstruct higher quality super-resolution (SR) images by reasoning about the high-frequency information of low-resolution (LR) images. With the development of deep learning, super-resolution technology based on convolutional neural networks has made great progress.

[0003] Denoising Diffusion Probabilistic Models, or simply Diffusion Model. This model is a Markov chain that uses variational inference to generate samples that are consistent with the original data distribution after a finite time. Figure 1 , the diffusion model consists of two steps:

[0004] (1) A fixed (or preset) forward diffusion process q: This process gradually adds Gaussian noise to the image, forming a series of data states x1, x2, …, x T . Until finally pure noise is obtained. This process can be defined as follows:

[0005]

[0006] where t∈(1,…,T) represents the diffusion step, β t is the variance hyperparameter associated with each step, and I is the unit moment. N(x; μ; δ) represents the normal distribution with mean μ and covariance δ that generates x. q(x t |x t-1 ) is a single-step operation to directly sample an arbitrary noisy version x from the original image t , that is, x t It can be obtained by the original data x0 and the variance plan β t Direct sampling.

[0007] (2) Trainable reverse denoising diffusion process: Train a neural network to gradually denoise from pure noise until a real image is obtained. Train a neural network to simulate the process of going from a noisy state x T ~N(0,I) is the inverse process of generating the original data in reverse. In practice, this is usually done by minimizing the variational lower bound. This process can be defined as follows:

[0008] p θ (x t-1 |x t)=N(x t-1 ,μ θ (x t ,t),∑ θ (x t ,t)) (2)

[0009] The number of forward and backward steps is defined by the subscript t, and there is a predefined total number of steps T (1000 in the original DDPM paper). When t=0, it is a real picture sampled from the dataset, and when t=T, it is approximately a pure noise. Through this iterative denoising and reconstruction process, DDPM is able to generate samples that are extremely similar to the original data, with very high quality and detail fidelity. The success of this method depends on carefully designed diffusion steps and efficient network structure, as well as the ability to accurately estimate the probability distribution during training.

[0010] SRDiff (Super-Resoluiton with Diffusion Probabilistic Models) model, which completes the super-resolution of a single image based on a diffusion probability model. This model method can effectively solve problems such as overly smooth output, mode collapse, and large model footprint. SRDiff converts Gaussian white noise into SR predictions through a Markov chain, is not affected by mode collapse, and can generate diverse and high-quality SR results; compared with GAN-driven methods, SRDiff is stable in training under a single loss and does not require any additional modules (such as discriminator, which is only used in training). Compared with process-based methods, SRDiff has no architectural restrictions and therefore benefits from a small footprint and fast training speed; SRDiff can use diffusion and reverse processes for flexible image processing, including latent space interpolation and content fusion, and has broad application prospects.

[0011] The SRDIFF model is a single image super-resolution model based on the denoising diffusion model. However, its x0 is not the SR image or HR image to be predicted. In the diffusion process, x0 is the difference between the HR image and the upsampled LR image. r =x H -UP(x L ) expression. In the reverse inference process, the final x0 is added to the upsampled LR image to obtain the predicted super-resolution image x SR =x0+UP(x L ). Using residual prediction instead of direct prediction can speed up training, better optimize the network, and pay more attention to high-frequency information that is easily lost in images.

[0012] Inspired by the denoising diffusion probabilistic model (DDPM) and denoising score matching, SR3 (Image Super-Resoluiton via Iterative Refinement), a new method for conditional image generation, is proposed. SR3 works by learning to transform a standard normal distribution into an empirical data distribution through a series of iterative refinement steps, similar to Langevin dynamics. The network structure of the model is a U-Net architecture, which is trained with a denoising objective to iteratively remove various levels of noise from the output. The denoising diffusion model is used for conditional image generation. SR3 is a method for image super-resolution via iterative refinement. SR3 proves effective for super-resolution of faces and natural images at different magnification factors.

[0013] SR3 improves the Unet of the diffusion model and modifies SDE based on the original diffusion model. The original DDPM residual block is replaced with the residual block of BigGAN, and the connection is rescaled to In addition, the number of residual blocks is increased, as well as channel multiplications at different resolutions. SR3 shows strong performance on super-resolution tasks with different upsampling multiples for faces and natural images.

[0014] Diffusion-based image super-resolution (SR) methods are limited by the speed of inference because hundreds or even thousands of sampling steps are required. Existing accelerated sampling techniques inevitably sacrifice performance to a certain extent, resulting in super-blurry SR results. Diffusion models perform worse than generative models based on likelihood functions in maximum likelihood estimation. The data generalization ability is weak. Diffusion models can only process continuous data such as pictures at first, and the effect of directly applying discrete data or other data types is poor. This limits the application scenarios of diffusion models. Summary of the invention

[0015] The main purpose of the present invention is to propose a method and device for super-resolution reconstruction of microscopic images based on an improved diffusion model, introduce an efficient diffusion model, improve the reasoning efficiency, and achieve good reconstruction effect.

[0016] The present invention adopts the following technical solution:

[0017] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0018] The present invention constructs an improved diffusion model, which realizes the conversion between high-resolution images and low-resolution images by moving the residual between high-resolution images and low-resolution images, greatly improving the conversion efficiency. At the same time, the U-Net of the diffusion model is improved, and the original self-attention layer is replaced with Swin Transformer Block to enhance the robustness of the model.

[0019] In the present invention, a Markov chain is constructed to achieve the conversion between high-resolution images and low-resolution images by moving the residual between the high-resolution images and the low-resolution images, which greatly improves the conversion efficiency and eliminates the need for post-acceleration and its related performance degradation in the inference process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the schematic diagram of the existing diffusion model;

[0021] Figure 2 The improved diffusion model schematic diagram of the present invention;

[0022] Figure 3 It is the U-Net structure of the predictor of the present invention;

[0023] Figure 4 This is the structure diagram of Swin Transformer Block;

[0024] Figure 5 A diagram of the process of reconstruction using a predictor according to the present invention;

[0025] Figure 6 Comparing the input image and the output image of the model of the present invention;

[0026] Figure 7 Comparison of input image and output image details for the model of the present invention.

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0028] The present invention is further described below through specific implementation modes.

[0029] A microscopic image super-resolution reconstruction method based on an improved diffusion model of the present invention comprises the following steps:

[0030] 1) Obtain pairs of high-resolution images and corresponding low-resolution images as a dataset.

[0031] In the present invention, a microscope is used to collect paired high-resolution images HR and low-resolution images LR microscopic cell images for use as a training set. Specifically, a high-performance fully automatic digital scanning microscope can be used to collect high-quality microscopic images. The microscope has a fast and accurate autofocus function to ensure that the image clarity is always maintained during the scanning process. The microscope adopts a high-quality infinity-corrected optical system to provide clear, distortion-free images. The optical system adopts an advanced correction design to minimize common optical aberrations such as spherical aberration, chromatic aberration, and coma, and provide accurate imaging quality. At the same time, the system undergoes strict color correction and transmission curve optimization to ensure uniform light transmission at different wavelengths and reduce color deviation and light loss.

[0032] Specifically, in practical applications, the collected images also need to be processed. For example, a microscope is used to simultaneously capture cell images under 40x and 20x objective lenses, and the collected images are center cropped. The HR image is cropped to 1024x1024, and because the objective lens relationship is 2x, the LR image is cropped to 512x512. When making the data set, there are problems such as insufficient accuracy in matching HR and LR images, so the BD (Bicubic Interpolation) downsampling method is used to obtain the corresponding LR images. The LR images collected in real life are used for reconstruction comparison. A total of four types of microscopic cell images were collected, each with a different structure. 800 pairs of training data and 200 pairs of test reconstruction data were used to make the data set.

[0033] 2) Use the training set to train the improved diffusion model.

[0034] The improved diffusion model of the present invention comprises:

[0035] In the forward process, the residual between the high-resolution image x0 and the corresponding low-resolution image y0 is recorded as e0, and the residual e0 is gradually moved from the high-resolution image x0 to the low-resolution image y0 through a Markov chain of length T, that is, e0 = y0-x0 to obtain the converted low-resolution image y0. Gaussian noise is continuously added to the high-resolution image x0. After t steps, it is x t Get high-resolution image x t .

[0036] In the forward process, the constructed Markov chain serves as a bridge between HR and LR images. The task of reconstructing super-resolution images can be completed by reverse sampling the Markov chain of any given LR image. Specifically:

[0037] Introducing a displacement sequence It increases monotonically with the time step T, satisfying n1→0 and n T →1; then the transition distribution is derived according to the shift order as follows:

[0038] q(x t |x t-1 ,y0)=N(x t ;x t-1 +α t e0,α t I),t∈1,2,…,T (3)

[0039] Where q(.) represents the forward process, x t represents the high-resolution image after adding noise to the high-resolution image x0 when the time step is t, N(.) represents the Gaussian distribution, α t =n t -n t-1 , when t>1 and α1=n1, I is the identity matrix; the marginal distribution of any time step t is analytically integrable, that is:

[0040] q(x t |x0,y0)=N(x t ;x0+n t e0,n t I),t∈1,2,…,T. (4)

[0041] The design of the transition distribution is based on two main principles. The first one is related to the standard deviation, which aims to promote x t and x t-1 This is because assuming the image data falls within the range [0,1], then x t With x t-1 The expected distance between To limit, that is:

[0042]

[0043] where max[·] represents the pixel-by-pixel maximization operation. The second principle is about the mean parameter, i.e. x0+α t e0, which can approximately derive the marginal distribution in formula (4). In addition, x1 and x T The marginal distribution of converges to δx0(·) and N(·; y0,I), which are the two approximate distributions of HR image and LR image respectively. By constructing the Markov chain in this way, the SR task can be handled by reverse sampling of a given LR image y0.

[0044] The reverse process is to transform the low-resolution image y0 after conversion and the high-resolution image x after adding noise. t and time step t is input to the predictor f θ De-noising and reconstructing super-resolution images.

[0045] In the reverse process, the predictor estimates the posterior distribution p(x0|y0) by the following formula, removes the noise and reconstructs the super-resolution image:

[0046]

[0047] p(x0|y0) represents the reverse process; p(x T |y0)≈N(x T |y0,I),;p θ (x T-1 |x T ,y0) is the T to x T-1 The inverse transfer kernel with learnable parameters θ, p θ (x t-1 |x t ,y0)=N(x t-1 ;μ θ (x t ,y0,t),∑ θ (x t ,y0,t)),μ θ (x t ,y0,t) is the mean parameter. The optimization of θ is achieved by minimizing the negative evidence. t It has nothing to do with y0. Let

[0048] For the mean parameter μ θ (x t ,y0,t), and reparameterize it as follows:

[0049]

[0050] Among them, f θ is a predictor composed of a deep neural network with parameters θ, and the objective function is simplified to in

[0051] See also Figure 3 , the present invention uses the predictor f θ Predict super-resolution image, f θ The U-Net structure is used as the main body, and its input includes 3 channels x t , diffusion time step t∈{1,2,…,T} and low-resolution image y0. Unet as a whole includes the first convolution block, the contraction part, the middle part, the expansion part and the second convolution block. The first convolution block converts the input image x tTransformed into hidden state and fused with the input transformed low-resolution image y0 to obtain the feature map. Transformer's sinusoidal position encoding is used to transform time step t into time step embedding t e , feature map and t e It is successively input into the contraction step, the intermediate step and the expansion step. The contraction part includes a plurality of contraction units connected in sequence to reduce the spatial size of the input feature map, for example, including four contraction units. The intermediate part is located between the contraction part and the expansion part and is provided with two residual blocks; the expansion part includes a plurality of expansion units connected in sequence, for example, including four expansion units, and the input of each expansion unit is also connected to the output of the corresponding contraction unit to realize splicing, and each expansion unit amplifies the spliced ​​feature map. The second convolution module processes the feature map output by the expansion part and outputs a predicted super-resolution image, and the output super-resolution image is at time step t-1.

[0052] The first convolution module includes a 2D convolution layer and a Mish activation function. The second convolution module includes a 2D convolution layer and a Mish activation function.

[0053] The contraction unit includes two residual blocks and a downsampling layer, and the residual block includes two convolutional layers and a Swin Transformer block. The expansion unit includes two residual blocks and an upsampling layer, and the residual block includes two convolutional layers and a Swin Transformer block. In the present invention, in order to reduce the size of the diffusion model, the channel size is doubled only in the second and fourth contraction units, and the spatial size of the feature map is halved in each contraction unit. The downsampling layer of the contraction unit is a 2D convolution with a stride of 2, while the upsampling layer in the expansion unit is a 2D transposed convolution.

[0054] In this invention, the self-attention layer of U-Net is replaced by Swin Transformer Block, see Figure 4 , replacing the standard multi-head self-attention module (MSA) with a shifted window-based multi-head self-attention module (W-MSA / SW-MSA) and keeping other parts unchanged can enhance the robustness of the model. In the Swin Transformer Block, the combination of MLP (Multi-Layer Perceptron) and LN (Layer Normalization) can improve the expressiveness and stability of the model. MLP is responsible for the nonlinear transformation of features, while LN helps stabilize the training process and prevent gradient disappearance or explosion problems.

[0055] The predictor of the present invention is easy to train and stable due to multi-scale skip connections. In addition, it combines local and global information through the contraction part and the expansion part.

[0056] When the dataset obtained in step 1 is used for training, the input high-resolution image HR is randomly cropped to 192*192, and randomly rotated by 90, 180 and 270 degrees. The low-resolution image LR is obtained by downsampling the corresponding high-resolution image HR by 2 times, and the size is 96*96. The learning rate is set to 5e-5, the batch_size is set to 16, and the training is iterated 400,000 times. The training is completed in about 15 days.

[0057] 3) Using an interpolation algorithm, the given low-resolution image is upsampled to a high-resolution image, and the low-resolution image and the corresponding high-resolution image are input into the trained improved diffusion model to obtain the corresponding super-resolution image.

[0058] After the improved diffusion model is trained, the weights obtained by training are used to input the low-resolution image LR into the improved diffusion model to obtain the corresponding super-resolution image SR. In the present invention, an improved diffusion model with only 4 sampling steps is used, thereby significantly improving the inference efficiency. Figure 5 The process of reconstructing an image by removing noise through Unet is shown, from left to right.

[0059] Figure 6 The overall effect diagram is shown. Figure 7 The improved diffusion model reconstructs detailed images. From a subjective perspective, the detailed texture of the reconstructed image is almost the same as that of the HR image. Objectively, the PSNR and SSIM of the reconstructed image are 38.7 and 0.95, which is enough to prove that the reconstruction effect is very good.

[0060] Based on this, the present invention also proposes a microscopic image super-resolution reconstruction device based on an improved diffusion model, the device adopts the above-mentioned microscopic image super-resolution reconstruction method based on an improved diffusion model, and comprises

[0061] The data set construction module acquires a pair of high-resolution images and corresponding low-resolution images as a data set. The data set construction module is used to execute step 1) of the above-mentioned microscopic image super-resolution reconstruction method based on the improved diffusion model.

[0062] A training module is used to train the improved diffusion model using the training set. The training module is used to execute step 2) of the above-mentioned microscopic image super-resolution reconstruction method based on the improved diffusion model.

[0063] The reconstruction module uses an interpolation algorithm to upsample a given low-resolution image to a high-resolution image, and inputs the low-resolution image and the corresponding high-resolution image into the trained improved diffusion model to obtain a corresponding super-resolution image. The reconstruction module is used to perform step 3) in the above-mentioned microscopic image super-resolution reconstruction method based on the improved diffusion model.

[0064] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A microscopic image super-resolution reconstruction method based on an improved diffusion model, characterized in that: include: Obtain pairs of high-resolution images and corresponding low-resolution images as a dataset; Using the training set to train an improved diffusion model; An interpolation algorithm is used to upsample a given low-resolution image to a high-resolution image, and the low-resolution image and the corresponding high-resolution image are input into a trained improved diffusion model to obtain a corresponding super-resolution image.

2. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 1, characterized in that: The improved diffusion model includes: In the forward process, the residual between the high-resolution image x0 and the corresponding low-resolution image y0 is recorded as e0, and the residual e0 is gradually moved from the high-resolution image x0 to the low-resolution image y0 through a Markov chain of length T, that is, e0 = y0-x0 to obtain the converted low-resolution image y0, and Gaussian noise is continuously added to the high-resolution image x0 to obtain the high-resolution image x0 with added noise. t ; The reverse process is to transform the low-resolution image y0 after conversion and the high-resolution image x after adding noise. t Input predictor f θ De-noising and reconstructing super-resolution images.

3. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 2, characterized in that: The predictor f θ The U-Net structure is used as the main body, which includes a first convolution block, a contraction part, a middle part, an expansion part and a second convolution block; the first convolution block converts the input image x t The contraction part comprises a plurality of contraction units connected in sequence to reduce the spatial size of the input feature map; the middle part is located between the contraction part and the expansion part and is provided with two residual blocks; the expansion part comprises a plurality of expansion units connected in sequence, the input of each expansion unit is also connected to the output of the corresponding contraction unit to realize splicing, and each expansion unit amplifies the spliced ​​feature map; the second convolution module processes the feature map output by the expansion part to output the predicted super-resolution image.

4. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 3, characterized in that: The first convolution module includes a 2D convolution layer and a Mish activation function; the second convolution module includes a 2D convolution layer and a Mish activation function.

5. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 4, characterized in that: The contraction unit includes two residual blocks and a downsampling layer, and the residual block includes two convolutional layers and a Swin Transformer block.

6. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 4, characterized in that: The expansion unit includes two residual blocks and an upsampling layer, and the residual block includes two convolutional layers and a Swin Transformer block.

7. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 2, characterized in that: In the forward process, the Markov chain constructed is specifically: Introducing a displacement sequence It increases monotonically with the time step T, satisfying n1→0 and n T →1; then the transition distribution is derived according to the shift order as follows: q(x t |x t-1 ,y0)=N(x t ;x t-1 +α t e0,α t I),t∈1,2,…,T Where q(.) represents the forward process, x t represents the high-resolution image after adding noise to the high-resolution image x0 when the time step is t, N(.) represents the Gaussian distribution, α t =n t -n t-1 , when t>1 and α1=n1, I is the identity matrix; the marginal distribution of any time step t is analytically integrable, that is: q(x t |x0,y0)=N(x t ;x0+n t e0,n t I),t∈1,2,...,T。 8. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 7, characterized in that: In the reverse process, the predictor estimates the posterior distribution p(x0|y0) by the following formula, removes the noise and generates a reconstructed image: p(x0|y0) represents the reverse process, x t represents the image after adding noise to the high-resolution image x0 when the time step is t; p(x T |y0)≈N(x T |y0,I), ;p θ (x T-1 |x T , y0) is the value from x T to x T-1 The inverse transfer kernel with learnable parameters θ, p θ (x t-1 |x t , y0)=N(x t-1 ;μ θ (x t ,y0,t),∑ θ (x t , y0, t)), N(.) represents Gaussian distribution, μ θ (x t , y0, t) is the mean parameter.

9. The method for super-resolution reconstruction of microscopic images based on an improved diffusion model as claimed in claim 8, characterized in that: For the mean parameter μ θ (x t , y0, t), reparameterized as follows: Among them, f θ is a predictor composed of a deep neural network with parameters θ, and the objective function is simplified to in 10. A microscopic image super-resolution reconstruction device based on an improved diffusion model, characterized in that: include A dataset construction module obtains pairs of high-resolution images and corresponding low-resolution images as datasets; A training module, using the training set to train an improved diffusion model; The reconstruction module uses an interpolation algorithm to upsample a given low-resolution image to a high-resolution image, and inputs the low-resolution image and the corresponding high-resolution image into the trained improved diffusion model to obtain the corresponding super-resolution image.

Images