Image reconstruction method

Abstract

The invention provides an image reconstruction method, which comprises the following steps of: 1, acquiring a blurred image INPUT and a point spread function F of an imaging system; 3, the momentum M (i) = X (i)-X (i-1) of the (i + 1) th iteration is calculated, wherein i is larger than or equal to 1 and smaller than or equal to N-1; 4, using the convolutional neural network CNN to predict an extrapolation vector V (i) = CNN (M (i)) of the (i + 1) th iteration; 5, calculating a prediction starting point # imgabs0 # of the (i + 1) th iteration according to the extrapolation vector V (i) of the (i + 1) th iteration and the reconstruction result X (i) of the ith iteration; and 6, calculating a reconstruction result # imgabs1 # of the (i + 1) th iteration by using an RL algorithm. According to the method, image reconstruction can be rapidly completed with high quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image reconstruction method for restoring an original image from a blurred image. Background Art

[0002] Without using super-resolution technology, the limit resolution of a fluorescence microscope can be calculated by the Abbe imaging theory, and inevitable interference factors in the imaging process, such as aberration, noise, and defocus signals, will further reduce the contrast and resolution of the image. To address this problem, deconvolution is a widely adopted technique in the field of image processing, and its goal is to restore the original image from the observed degraded image (blurred image).

[0003] Based on the transparency of the reconstruction process as the division criterion, image deconvolution algorithms can be roughly divided into three categories. Early traditional algorithms are based on physical models, construct image reconstruction formulas based on prior knowledge, and use mathematical methods such as maximum likelihood to solve the target image, which is also called an explicit algorithm. Optical microscopy imaging methods usually use detectors (photomultiplier tubes, CMOS cameras, etc.) to count photons to obtain image pixel values. The emission process of photons is random, that is, within any given time, the emission of photons is a random process following a Poisson distribution. Therefore, the images obtained by optical microscopy imaging methods are generally affected by Poisson noise, which has promoted extensive research on image deconvolution algorithms based on the statistical characteristics of Poisson noise.

[0004] One of the most commonly used explicit algorithms was proposed by Richardson and Lucy, generally known as the Richardson-Lucy Algorithm (RL Algorithm for short). The following formula gives the general form (multiplicative form) of the RL algorithm:

[0005]

[0006] where X (i) represents the output result of the i-th iteration of the iterative algorithm, y represents the blurred image, represents the convolution operation, F represents the point spread function of the system, F TIt is obtained by rotating F by 180°. The RL algorithm is relatively simple and easy to implement, which makes it a classic algorithm in the field of image restoration and is widely used in teaching and research. In addition, the RL algorithm has two characteristics: energy conservation and non-negativity, that is, the RL algorithm does not change the overall energy of the image (expressed as the sum of all pixel values) and ensures that each pixel value is non-negative. This conforms to the actual situation of imaging, because the degradation and blurring process of the image does not change the total number of detected photons, and at the same time, physically, the photon count cannot be negative. However, in practical applications, there are still many problems with the RL algorithm, such as slow convergence speed, easy amplification of noise at high iteration times, and sometimes the generation of ring artifacts, etc.

[0007] The Accelerated Richardson-Lucy Algorithm (ARL algorithm for short) improves the RL algorithm and greatly improves the convergence speed. The idea of the ARL algorithm is similar to that of the gradient descent method. It uses the difference between X (i) and X (i-1) to calculate the direction vector (also called momentum) of the (i + 1)-th iteration. The ARL algorithm uses this vector to estimate the gradient direction and, together with X (i) is used to predict the starting point of the (i + 1)-th iteration. The specific implementation steps are shown as follows:

[0008]

[0009] where M (i) represents the momentum of the (i + 1)-th iteration, V (i) represents the extrapolation vector of the (i + 1)-th iteration, α i is the acceleration factor, represents the predicted starting point of the (i + 1)-th iteration. Since it no longer directly uses the output result of the previous iteration as the starting point of the RL algorithm in the next iteration, but predicts a more reasonable starting point based on the existing calculation results, the ARL algorithm has a faster convergence speed than the original RL algorithm, that is, it achieves similar reconstruction performance with fewer iteration times.

[0010] ARL has good interpretability and generalization, but when the prior knowledge is insufficient or the physical model cannot correctly reflect the actual imaging process, the reconstruction performance may drop significantly. In addition, due to the influence of noise, it is necessary to determine a reasonable number of iterations, otherwise it may converge to a wrong solution.

[0011] The second type of methods are data-driven deep learning algorithms, which utilize a large amount of data to train complex neural network models and learn the mapping relationship from blurred images to clear images. Compared with the first type of methods, the data-driven algorithms achieve better reconstruction performance and reduce the artifacts in the reconstruction results. However, data-driven methods are generally regarded as black-box models, whose internal structures and parameters are difficult to intuitively understand. Moreover, the neural network models implement deconvolution by learning image feature mappings, and these mappings are usually highly abstract and complex, making it difficult for people to explain the working principle of the algorithms. The reconstruction performance of data-driven algorithms is strongly correlated with the types of image features and is prone to overfitting, and their generalization performance is poor.

[0012] In recent years, fusion methods have received increasing attention and can be called model-driven deep learning algorithms. They combine physical models with neural network structures, improving the reconstruction performance while achieving better interpretability and generalization compared to end-to-end networks. However, most of the existing model-driven algorithms directly extend the models originally acting on the image space to high-dimensional feature spaces. Compared with traditional model algorithms, performing model-based calculations in high-dimensional feature spaces brings a considerable computational burden, which leads to a significant increase in the time taken for each iteration of these model-driven algorithms. Summary of the Invention

[0013] Based on the foregoing deficiencies of the prior art, the present invention provides an image reconstruction method with good reconstruction quality and high speed.

[0014] To achieve the above object, the present invention provides an image reconstruction method for recovering an original image from a blurred image, including the following steps:

[0015] Step 1: Obtain a blurred image INPUT and the point spread function F of the imaging system, and manually set the iteration starting point X of the image deconvolution algorithm (0) ;

[0016] Step 2: Use the RL algorithm to calculate the reconstruction result of the first iteration represents the convolution operation, and F T is obtained by rotating F by 180°;

[0017] Step 3: Calculate the momentum M of the (i + 1)-th (1 ≤ i ≤ N - 1) iteration (i) = X (i) - X (i-1) , where N is a positive integer and is set according to the actual situation;

[0018] Step 4: Use the convolutional neural network CNN to predict the extrapolation vector V of the (i + 1)-th iteration (i) = CNN(M (i) );

[0019] Step 5: Calculate the prediction starting point of the (i + 1)-th iteration using the extrapolation vector V of the (i + 1)-th iteration (i) and the reconstruction result X of the i-th iteration (i)

[0020]

[0021]

[0022] Step 7: Repeat steps 3 to 7 for (N - 1) times;

[0023] Step 8: Output the reconstruction result of the N-th iteration as the final reconstruction result Output = X (N) , forming the restored original image.

[0024] In one embodiment, in step 1, the iteration starting point X (0) is set to X (0) = INPUT or X (0) = all-ones matrix × mean(INPUT).

[0025] In one embodiment, in step 4, the convolutional neural network CNN is an ultra-lightweight convolutional neural network composed of two trainable convolutional layers and activation layers.

[0026] In one embodiment, the convolutional layer kernel size of the convolutional layer is 5×5, and the feature domain dimension is 16; the activation layer is Leaky ReLu.

[0027] In one embodiment, in step 7, the convolutional neural network CNN used in each iteration does not share network parameters.

[0027] In one embodiment, the training of the image deconvolution algorithm includes: first preparing the point spread function F and the clear image GT, then obtaining the blurred image INPUT by convolving GT with F and adding noise as needed. GT and INPUT form a training image pair for neural network model training. Use the default initialization method of Pytorch and fix all initial random number seeds. Use the L1 loss function to optimize the network. The optimizer uses the Adam optimizer provided by Pytorch, with default parameters, and the batch size Batch Size is set to 1.

[0028] The image reconstruction method of the present invention uses an image deconvolution algorithm that designs a new type of ultra-lightweight convolutional neural network based on the ARL algorithm to cooperate, which can quickly and high-quality complete the image deconvolution task. Compared with the first type of method (traditional method based on models), the present invention significantly accelerates the convergence speed, improves the reconstruction performance and significantly reduces artifacts; compared with the second type of method (data-driven deep learning algorithm), the present invention has better interpretability and generalization; compared with the third type of method (deep learning algorithm based on models), the present invention has the advantages of fast reconstruction speed, small number of network parameters, and strong small-sample learning ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure of the present invention in any way. Additionally, the shapes and proportional dimensions of the components in the figures are only schematic and are used to assist in understanding the present invention, and do not specifically limit the shapes and proportional dimensions of the components of the present invention. Those skilled in the art can, under the teachings of the present invention, select various possible shapes and proportional dimensions according to specific circumstances to implement the present invention. In the drawings:

[0030] Figure 1 is a flowchart of a method for image reconstruction provided in the first embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of a method for image reconstruction provided in the first embodiment of the present invention;

[0032] Figure 3 is a comparison diagram of the effects of the first experimental example;

[0033] Figure 4 is a comparison diagram of the effects of the second experimental example;

[0034] Figure 5 is a comparison diagram of the effects of the fourth experimental example;

[0035] Figure 6 is a comparison diagram of the effects of the fifth experimental example. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0037] Please refer to Figure 1 、Figure 2 As shown in Figure 2 , the first embodiment of the present invention provides an image reconstruction method for recovering an original image from a blurred image, including the following steps:

[0038] Step 1: Obtain the blurred image INPUT and the point spread function F of the imaging system, and manually set the iteration starting point X of the image deconvolution algorithm (0) ;

[0039] Step 2: Use the RL algorithm to calculate the reconstruction result of the first iteration represents the convolution operation, and F T is obtained by rotating F by 180°;

[0040] Step 3: Calculate the momentum M of the (i + 1)-th (1 ≤ i ≤ N - 1) iteration (i) = X (i) - X (i-1) , where N is a positive integer and is set according to the actual situation;

[0041] Step 4: Use the convolutional neural network CNN to predict the extrapolation vector V of the (i + 1)-th iteration (i) = CNN(M (i) );

[0042] Step 5: Calculate the prediction starting point of the (i + 1)-th iteration from the extrapolation vector V of the (i + 1)-th iteration (i) and the reconstruction result X of the i-th iteration (i)

[0043] Step 6: Use the RL algorithm to calculate the reconstruction result of the (i + 1)-th iteration

[0044] Step 7: Repeat steps 3 to 7 (N - 1) times;

[0045] Step 8: Output the reconstruction result of the N-th iteration as the final reconstruction result Output = X (N) , forming the recovered original image.

[0046] In one embodiment, the iteration starting point X (0) is set to X (0) = INPUT or X (0) = all-ones matrix × mean(INPUT).

[0047] In one embodiment, the convolutional neural network CNN is an ultra-lightweight convolutional neural network composed of two trainable convolutional layers and activation layers. In a specific embodiment, the convolutional kernel size of the convolutional layer is 5×5, and the feature domain dimension is 16; the activation layer is Leaky ReLu.

[0048] In one embodiment, in step 7, the convolutional neural network (CNN) used in each iteration does not share network parameters.

[0049] Briefly, for the image reconstruction method of the present invention, the image deconvolution algorithm used is designed based on the ARL algorithm with a new type of ultra-lightweight convolutional neural network (Accelerated Richardson-Lucy Network, ARLN) for cooperation. Therefore, the image deconvolution algorithm of the present invention can also be simply referred to as the ARLN algorithm. Please refer to Figure 3 As shown, the inventor conducted the first experimental example, a comparison of the reconstruction result ARLN obtained by the image reconstruction method of the present invention with the blurred image INPUT and the clear image GT (Ground Truth). Figure 3 In it, subfigure (a) is the overall image of the blurred image INPUT, subfigure (b) magnifies and shows the part indicated by the yellow dashed box in (a), subfigure (c) is the enlarged view of the blue dashed box in (b), the clear image GT, and the reconstruction result ARLN obtained by the image reconstruction method of the present invention, and subfigure (d) is the normalized gray value curve at the white dashed line in (c). From Figure 3 it can be seen that the reconstruction result ARLN of the present invention has a significant improvement in resolution and signal-to-background ratio compared to the blurred image INPUT, and can recover the peaks that have been completely blurred in the blurred image INPUT ( Figure 3 (indicated by the black arrow in subfigure (d)).

[0050] The ARLN algorithm of the present invention uses a convolutional neural network (CNN) to predict the extrapolation vector V. Compared with the manually designed calculation logic in the ARL algorithm, the convolutional neural network (CNN) of the present invention can autonomously learn the non-linear mapping relationship from the momentum M to the extrapolation vector V from the training data. Therefore, it can provide a more accurate extrapolation vector V, thus providing a better acceleration and convergence effect. Please refer to Figure 4 As shown, the inventor conducted the second experimental example, a comparison of the reconstruction result ARLN obtained by the image reconstruction method of the present invention with the reconstruction result ARL obtained by the ARL algorithm, the blurred image IN, and the clear image GT (Ground Truth), where both the present invention and the ARL algorithm were performed with different numbers of iterations. Figure 4 In it, part (a) shows the blurred image IN, the clear image GT, and the reconstruction results of the ARL algorithm and the present invention with different numbers of iterations, and part (b) shows the quantitative indicators of the reconstruction results of the ARL algorithm and the present invention with different numbers of iterations. From Figure 4 it can be seen that the reconstruction result ARLN of the present invention obtains a reconstruction quality similar to that of the ARL algorithm when only using 1 / 10 of the number of iterations.

[0051] The image reconstruction method of the present invention only involves high-dimensional feature domain calculations in step 4, while the model-based calculation process is retained in the low-dimensional image spatial domain. Therefore, the present invention has a significant computational speed advantage compared to other model-based deep learning algorithms. Moreover, the present invention uses an ultra-lightweight neural network structure, so the number of network parameters used in the present invention is also significantly reduced. As shown in Table 1 below, the inventors conducted a third experimental example. When the reconstruction quality is comparable, the reconstruction speed of the present invention's ARLN is 6 times faster than that of DWDN and 9 times faster than that of RLB compared to other model-based deep learning algorithms (including Deep Wiener Deconvolution (DWDN), Richardson–Lucy Block (RLB), and Richardson-Lucy Network (RLN)). In terms of the number of network parameters, the number of network parameters of the present invention's ARLN is 18% of that of DWDN and RLB, and 4% of that of RLN. This means that the present invention's ARLN occupies less memory and computational resources, which provides potential advantages for deploying the model in resource-constrained environments (such as mobile devices and embedded systems).

[0052] Table 1 Comparison between the present invention and other model-based deep learning algorithms

[0053]

[0054] The present invention decomposes the entire image deconvolution task into different steps and uses multiple convolutional layers to fit a small part of the non-linear mapping relationship in the entire task. Compared with other deep learning algorithms that use complex neural network structures to fit the total mapping relationship, the present invention only requires a very small number of training samples to complete learning. Please refer to Figure 5 As shown, the inventors conducted a fourth experimental example to compare the reconstruction performance of the present invention with other deep learning algorithms under different training conditions. Figure 5 Among them, part (a) shows the comparison between the blurred image RAW and the reconstructed images of each algorithm under different training conditions. Part (b) shows the gray value normalization curve of the reconstructed results of the present invention's ARLN (trained with 200 pairs of training data) and the traditional method at the yellow dashed line. Part (c) shows the gray value normalization curve of the reconstructed results of each algorithm at the yellow dashed line under different training conditions (in part (c), the icons 5, 25, and 200 in each figure correspond to the algorithms ARLN, CNN, DWDN, RLB, RLN, etc. shown in the corresponding figures in the same row of part (a)). From Figure 5 it can be seen that during the process of gradually reducing the number of training samples from 200 pairs to 5 pairs, the performance loss of the present invention is negligible compared to other deep learning algorithms (the performance loss of other algorithms is relatively obvious as the number of training samples decreases).

[0055] Traditional model-based algorithms contain prior knowledge of the problem, while deep learning algorithms capture complex mapping relationships by learning a large amount of data. Model-driven deep learning algorithms combine the two, introducing model-based prior knowledge into the neural network structure to provide better constraints and guidance for model training, which helps the model better generalize to new data types. Algorithms such as RLB, RLN, and DWDN adopt the strategy of first extracting image features and then directly transplanting model-based calculations onto the feature domain. The excessive use of abstract features limits the improvement in generalization brought by introducing the model. Therefore, their generalization performance is only better than that of data-driven deep learning algorithms, but there is still a gap compared with traditional algorithms. The present invention pays more attention to the properties of physical models and the image domain, minimizes the use of unnecessary abstract features, and enhances its generalization performance. Please refer to Figure 6 as shown. The inventor conducted the sixth experimental example, using training data with neuron cell bodies as image features (as shown in part (a) of Figure 6 ) to train a neural network model, and applying the trained model to a test image with nerve fibers as image features. The reconstruction results are as shown in part (b) of Figure 6 . Obvious stripe artifacts appeared in the reconstruction results of DWDN, while the reconstruction ability of CNN, RLB, and RLN for weak signals was very poor (as shown within the yellow dashed box in part (b) of Figure 6 ). Part (c) shows the full width at half maximum (Gaussian model fitting) of the gray value curves of each algorithm at the yellow dashed line in part (b). Part (d) shows the comparison of the normalized gray value curves of the ARLN of the present invention and traditional model algorithms at the yellow dashed line. It can be seen from this that although the reconstruction ability of the ARLN of the present invention also decreased, it was still comparable to the reconstruction performance of the ARL algorithm after 6 iterations. At this time, the calculation speed of the ARLN of the present invention was still slightly faster than that of traditional iterative algorithms. In addition, as shown in part (e), by comparing the normalized gray value curves of the ARLN of the present invention and other deep learning algorithms at the yellow dashed line, the resolution and signal-to-background ratio of the reconstruction results of the present invention are better than those of other deep learning algorithms.

[0056] The present invention also provides an ARLN algorithm training embodiment. The ARLN algorithm training process needs to prepare a point spread function F and a clear image (Ground Truth, GT) first, and then add noise as needed after GT convolution F to obtain a blurred image (INPUT). GT and INPUT constitute a training image pair for neural network model training. In this embodiment, the number of iterations N is set to 4. The software environment used is Python3.9 and Pytorch 1.8.1. The training and evaluation process of the algorithm is completed on a DELL7920 workstation. The core hardware configuration of the workstation is NVIDIA 3090GPU and INTEL CORE I9 10900XCPU. The number of feature domain dimensions is set to 16, the convolution layer uniformly uses a 5×5 convolution kernel, and the model parameters use the default initialization method of Pytorch and fix all initial random number seeds. The model uses the L1 loss function to optimize the network during training. The optimizer uses the Adam optimizer (default parameters) that comes with Pytorch, and the batch size (Batch Size) is set to 1 to obtain better reconstruction performance.

[0057] The image reconstruction method of the present invention adopts an image deconvolution algorithm which is designed based on the ARL algorithm and cooperates with a new type of ultra-lightweight convolutional neural network, which can quickly and high-quality complete the image deconvolution task. Compared with the first type of method (traditional model-based method), the present invention greatly accelerates the convergence speed, improves the reconstruction performance and significantly reduces artifacts; compared with the second type of method (data-driven deep learning algorithm), the present invention has better interpretability and generalization; compared with the third type of method (model-based deep learning algorithm), the present invention has the advantages of fast reconstruction speed, less network parameters, and strong small sample learning ability.

[0058] It should be understood that the above description is for illustration and not for limitation. Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of the present teachings should not be determined with reference to the above description, but rather with reference to the foregoing claims and the full scope of equivalents to which such claims are entitled. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the foregoing claims is not intended to be a waiver of such subject matter, nor should it be considered that the applicant has not considered such subject matter to be part of the disclosed application subject matter.

Claims

1. An image reconstruction method for restoring an original image from a blurred image, characterized in that: The steps include: Step 1: Get the blurred image INPUT and the point spread function F of the imaging system, and manually set the iteration starting point X of the image deconvolution algorithm. (0) ; Step 2: Use the RL algorithm to calculate the reconstruction result of the first iteration represents the convolution operation, F T Obtained by rotating F by 180°; Step 3: Calculate the momentum M of the i+1th (1≤i≤N-1)th iteration (i) =X (i) -X (i-1) , N is a positive integer, set according to actual conditions; Step 4: Use the convolutional neural network CNN to predict the extrapolated vector V of the i+1th iteration (i) =CNN(M (i) ); Step 5: Extrapolate the vector V from the i+1th iteration (i) and the reconstruction result X of the i-th iteration (i) Calculate the predicted starting point for the i+1th iteration Step 6: Use the RL algorithm to calculate the reconstruction result of the i+1th iteration Step 7, repeat step 3 to step 7 (N-1) times; Step 8: Output the reconstruction result of the Nth iteration as the final reconstruction result Output = X (N) , forming the restored original image.

2. The image reconstruction method according to claim 1, characterized in that: In step 1, the iteration starting point X (0) Set to X (0) =INPUT or X (0) =all-1 matrix×mean(INPUT).

3. The image reconstruction method according to claim 1, characterized in that: In step 4, the convolutional neural network CNN is an ultra-lightweight convolutional neural network consisting of two trainable convolutional layers and an activation layer.

4. The image reconstruction method according to claim 3, characterized in that: The convolution layer kernel size of the convolution layer is 5×5, and the feature domain dimension is 16; the activation layer is Leaky ReLu.

5. The image reconstruction method according to claim 2, characterized in that: In step 7, the convolutional neural network CNN used in each iteration does not share network parameters.

6. The image reconstruction method according to claim 1, characterized in that: The training of the image deconvolution algorithm includes: first preparing a point spread function F and a clear image GT, then convolving GT with F and adding noise as needed to obtain a blurred image INPUT, GT and INPUT form a training image pair for neural network model training, using the default initialization method of Pytorch and fixing all initial random number seeds, using the L1 loss function to optimize the network, and using the Adam optimizer that comes with Pytorch as the optimizer. The parameters are default, and the batch size Batch Size is set to 1.

Images