Fast sparse inverse convolution super-resolution reconstruction method for single image

Abstract

A fast super-resolution inverse convolution reconstruction method for a single two-dimensional image comprises the following steps of: 1, inputting data; reading an observation image and a convolution template; 2, carrying out linear transformation on a value range of observation image pixels, and carrying out normalization processing on a convolution template signal; and 3, carrying out rapid sparse inverse convolution restoration processing. Constructing a two-dimensional image rapid sparse inverse convolution method restoration model, and obtaining an inverse convolution restoration image; the method is used for improving the spatial resolution of an image, and has the characteristics of high processing speed and strong anti-noise capability.

Description

Technical Field

[0001] The present invention belongs to the field of image restoration processing, and in particular to a single image super-resolution reconstruction method. Specifically, a single image fast sparse inverse convolution super-resolution reconstruction method is proposed. The present invention is particularly applicable to the resolution improvement of microscopic optical images. Background Art

[0002] In many imaging systems such as infrared imagers and CCD cameras, the image quality may be degraded and blurred due to the inherent density of the sensor array or interference in the process of collecting wide and fast field of view images, which affects monitoring and observation. If the image resolution is improved by increasing the density of the sensor array, the cost may be very high. Therefore, in recent years, the spatial resolution of the image has been improved by using signal processing software, that is, super-resolution technology, to reconstruct images with better quality and higher spatial resolution.

[0003] Image super-resolution (SR) reconstruction is a technique that reconstructs a high-resolution (HR) image from one or a sequence of low-resolution (LR) images through signal processing or digital image processing technology. There are many types of super-resolution reconstruction algorithms, which can be generally divided into two categories according to the source of the reconstructed image: sequential image super-resolution reconstruction method and single image super-resolution reconstruction method. At present, the sequential image super-resolution reconstruction algorithm is widely used. It mainly uses the complementary and redundant information between low-resolution sequential images to reconstruct a high-resolution image from a low-resolution image sequence. The sequential image reconstruction algorithm requires that the displacement between image sequences meets the sub-pixel level requirements, but in actual operations it is difficult to obtain a low-resolution image sequence in the same scene. Even if multiple images of the same scene are obtained, the overlap between adjacent images may be small or non-existent, and the requirement that there must be differences between the images cannot be met, which leads to the meaningless image sequence. In addition, the sequential image super-resolution reconstruction algorithm involves motion estimation between adjacent images. This process is computationally intensive, and its accuracy will seriously affect the reconstruction quality and has poor robustness. In addition, Baker S also theoretically proved in "Limits on super-resolution and how to break them" that the super-resolution reconstruction algorithm of sequential images can only improve the resolution by about two times. As the resolution is improved, the quality of the reconstructed image drops sharply. Most importantly, after the motion estimation and registration are completed, the core and difficulty of the subsequent steps of the super-resolution reconstruction algorithm of sequential images are still concentrated on the super-resolution reconstruction of a single image. In addition, in many cases, we cannot obtain multiple low-resolution images of the same scene, such as old photos, reduced pictures on web pages, etc. Therefore, the research on the super-resolution reconstruction technology of a single image is of great significance.

[0004] Super-resolution technology can increase the spatial resolution of existing images, thereby overcoming the inherent resolution limitations of sensors and optical imaging. Compared with multi-image super-resolution technology, a single image has only a small amount of information, and the problem of ill-conditioned reconstruction is more complicated. Today's single image super-resolution reconstruction technology can be divided into the following three types: interpolation-based methods, reconstruction-based methods, and learning-based methods.

[0005] The super-resolution restoration of a single image refers to estimating the corresponding high-resolution (HR) image from a low-resolution (LR) image. The super-resolution reconstruction of a single image is processed only for one input image, with little known information, and is a typical ill-posed problem.

[0006] Single image super-resolution restoration is an ill-posed inverse problem that requires regularization constraints using prior knowledge of the image. One type of super-resolution restoration method is to learn prior knowledge (such as total variation, sparsity) from external low-resolution-high-resolution image pairs, while another type of method is to use the intrinsic prior of the image itself (such as self-similarity) to reconstruct a high-resolution image.

[0007] Super-resolution reconstruction is an image restoration technology. Image restoration technology requires the establishment of an image degradation model, and then restoration is performed based on the inverse process of the degradation model. Single-image super-resolution reconstruction is a technology that reconstructs a high-resolution image based on only one low-resolution image to be reconstructed and combined with prior knowledge. The present invention is based on the typical downsampling, blurring, denoising and degradation process, and the imaging model of the image is as shown in formula (1), which performs fast and efficient single-image super-resolution reconstruction.

[0008] G=SAF+N (1)

[0009] Where G is the degraded low-resolution image to be reconstructed, F is the original high-resolution clear image, N is the noise signal, A represents the blur operator matrix, AF is the convolution of the convolution function and the original image, and S is the downsampling matrix. Summary of the invention

[0010] The purpose of the present invention is to restore the original image F from the observed image G according to the degradation model of formula (1), and provide a fast method for super-resolution deconvolution reconstruction of a single two-dimensional image to improve the spatial resolution of the image. The method is characterized by fast processing speed and strong noise resistance.

[0011] In order to accurately estimate the high-resolution image F, some prior or regular constraints of the image are introduced, and the solution process is transformed into an optimization problem, which is expressed as

[0012]

[0013] in is the estimated high-resolution image, is the observation error term, ||F||1 is the intrinsic regularization constraint term, and λ is the balance adjustment parameter. ||·||2 and ||·||1 are the L2 norm and L1 norm. We define

[0014] F1(F)=λ||F||1 and

[0015] Formula (2) can be rewritten as

[0016]

[0017] The solution of equation (3) is solved by the alternating iteration method. For a given λ>0, any initial matrix F is selected 0Initial high-resolution image, for k = 0, 1, 2, ..., F k Iterative solution of F k+1 , mainly according to (4) to minimize F2(F) and (5) to minimize F1(F) alternately iteratively to obtain F k+1 .

[0018]

[0019] F k+1 =A λ (β k )(5)

[0020] where δ k is the k-iteration step size; is the gradient, find its gradient A according to F2 T S T (SAF k -G), and project the result; A λ is defined as follows:

[0021] A λ ([α1,α2,...,α n ] T )=[t λ (α 1) ),t λ (α2),...,t λ (α n )] T , t λ (α i )=(|λ-α i |)sgn(α i ) (6)

[0023] The above are the key technologies for implementing the above model. The present invention provides a fast algorithm for super-resolution reconstruction of two-dimensional images, comprising the following steps:

[0024] Step 1 Data input. Read the observed image G <m×n> , resolution improvement factor k and convolution template T <(2p+1)×(2q+1)> .

[0025] Step 2: Coarse super-resolution reconstruction. This process is the coarse reconstruction part of the super-resolution reconstruction algorithm. The same regularization parameter λ is used in each iteration process. The fast iterative shrinkage threshold algorithm (FISTA) is used in the iteration process to speed up the convergence speed. This process quickly obtains the overall information and features of the super-resolution reconstructed image. The specific process is:

[0026] Step 2.1 Initialization. Set the regularization parameter λ, the maximum number of iterations K and the convergence threshold ε, the counter t = 1, and set the initial value of the reconstructed high-resolution image. Set the variable t1 = 1 for iterative acceleration. And the simulated high-resolution image matrix F that records the results of the previous iteration p , and let X = F 0 , iterative step control quantity l.

[0027] Step 2.2: Minimize F2(F). This process uses the gradient projection algorithm combined with the fast convolution algorithm to minimize the L2 norm in the model. The algorithm implementation includes the following 5 steps:

[0028] S1 Observed image G minus reconstructed image F k The difference R is obtained by convolving the downsampled image with the template T<m×n> , that is, to achieve G-SAF k .

[0029] The present invention realizes a fast discrete convolution algorithm for spatial two-dimensional images. In this step, this function is used to complete the reconstruction of the high-resolution image matrix F k Convolution with convolution template T.

[0030] The present invention realizes a fast discrete convolution algorithm for spatial two-dimensional images. In this step, this function is used to complete the deconvolution of the image matrix F k Convolution with convolution template T.

[0031] The steps of fast convolution operation in the spatial domain of a two-dimensional image are as follows:

[0032] ① Find the size of the two-dimensional image F, expressed as m×n, and the size of the convolution template T, expressed as (2p+1)×(2q+1), where p and q can be decimals;

[0033] ② Set the initial value of the convolution result ConvF, ConvF is a zero matrix of size (m+2p)×(n+2q);

[0034] ③ Select elements F(i,j) (1≤i≤m,1≤j≤n) from the two-dimensional image matrix F and multiply them with the template T; and add the product to the corresponding modules between rows i to i+2p and columns j to j+2q in ConvF to obtain a new ConvF;

[0035] ④ Repeat step ③ until all elements in F have completed the operation of step ③;

[0036] ⑤Output the convolution result ConvF of F and T.

[0037] The present invention defines a sparse matrix product form of calculating SX by downsampling the image matrix X as shown in formula (7).

[0038] like

[0039] Among them C i =(11L 1) 1×k ,i=1,2,L,m,n.

[0040] This process SAF k The first step is to downsample the rows and then downsample the columns. k The k×k pixel blocks in are merged into one pixel, which can achieve the purpose of fast calculation.

[0041] S2 further completes the negative gradient based on S1 The calculation of A T S T (G-SAF k ).

[0042] The present invention defines S T The operation is still the same as (7), which is a sparse matrix product. T The X operation is defined as in formula (8).

[0043] like

[0044] Among them C i =(11L 1) 1×k ,i=1,2,L,m,n.

[0045] This calculation is S of the difference image R T The result is that the difference image R is magnified by k×k, and each element corresponds to a k×k pixel block, and S is obtained. T (G-SAF k );A T S T (G-SAF k ) is calculated by flipping the convolution kernel T by 180° and S T (G-SAF k ) convolution, and then extract the matrix of size (km-2p)×(kn-2q) from 2p+1 to km and columns 2q+1 to kn in the convolution result, which is the negative gradient of F2(F)

[0046] S3 gradient projection further corrects the negative gradient. Elements in F that are less than 0 and correspond to k The values ​​less than 0 are set to 0, that is,

[0047]

[0048] S4 finds the optimal step length parameter along the descending direction of the echelon so that the step length is large and the reconstructed image is closer to the observed image

[0049]

[0050] Among them, ||·|| F is the Frobenius norm, l is a small constant, (·) + Defined as It is understood as the mean square error between the observed image and the reconstructed image.

[0051] S5 Calculation of β k . in In step 2.2, an iterative threshold reduction algorithm is used to process the regularization constraint term to minimize the L1 norm, that is, to minimize F1(F). According to formula (5) λ (β k ) Solved to get

[0052] Step 2.3 Combine the previous iteration result F k Correction F k+1 . Modify the accelerated iteration according to (13) to (16).

[0053]

[0054] t1=t2 (16)

[0055] Step 2.4 Judgment Or whether the maximum number of iterations has been reached; if so, let F k =F k+1 Go to step 4; otherwise, let F k =F k+1 , t=t+1, go to step 2.2.

[0056] Step 3: Coarse super-resolution reconstruction. This process is the coarse reconstruction part of the super-resolution reconstruction algorithm. The same regularization parameter λ is used in each iteration process. The fast iterative shrinkage threshold algorithm (FISTA) is used in the iteration process to speed up the convergence speed. This process quickly obtains the overall information and features of the super-resolution reconstructed image. The specific process is:

[0057] Step 3.1: Set the convergence condition threshold ξ = 10^(-9), the number of weighted adjustments p = 15, and the number of sparse super-resolution reconstructions q = 15 for each weighted adjustment, and set the corresponding counting variables m = 1 and n = 1;

[0058] Step 3.2 Update the weight (i, j) parameter based on the obtained F^k. (i, j) = ((i, j)) / (F^k (i, j) + τ), τ = 0.001;

[0059] Step 3.3 Iterate and solve F^(k+1). The iterative solution is based on equations (5) and (6). The specific method is the same as steps 2.2 to 2.2 of the third step;

[0060] Step 3.4 determines whether m is less than q. If so, proceed to step 3.3; otherwise, proceed to step 3.5; if or when the maximum number of iterations is reached in step 3.5, terminate the iteration and proceed to step 5; otherwise, proceed to step 3.6; in this implementation, when p=14, the constraint condition is met and the loop is terminated in advance and the step 5 is entered;

[0061] Step 3.6: Determine whether n is less than p. If so, set [F^k=F]^(k+1) and go to step 3.2; otherwise, go to step 5.

[0062] Step 4: Fine super-resolution reconstruction. In step 3, the regularization parameter λ is set to a constant, and each pixel in the iteratively reconstructed high-resolution image is treated equally, and the pixel information with a larger contribution cannot be highlighted. The weighted L1 minimization model uses larger weights to prevent near-zero terms in the reconstructed signal, and smaller weights to encourage non-zero terms, further sparsely regulating the reconstruction of the high-resolution image.

[0063] Step 5 Output super-resolution reconstructed image BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of fast sparse inverse convolution image super-resolution reconstruction provided by the present invention;

[0065] Figure 2 It is the original high-resolution standard test grayscale image Lena image in the example of the present invention;

[0066] Figure 3 is a degraded observation image with noise added after fuzzy downsampling in the example of the present invention;

[0067] Figure 4 It is the result image of super-resolution reconstruction in the example of the present invention. DETAILED DESCRIPTION

[0068] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and implementation examples. It should be noted that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention. Figure 1This is the process of the fast sparse deconvolution two-dimensional image restoration method provided by the present invention. The specific implementation method is as follows:

[0069] Step 1: Data input. Input the observed image and convolution template. The present invention has no special requirements for the convolution kernel and noise distribution. The following is just an example. Figure 2 The standard grayscale test image LENA of size 256×256 pixels is convolved with the following template T, and 2×2 pixels are merged into one pixel to obtain a new image of size 131×131 degraded image, and then Gaussian noise with a mean of 0 and a variance of 0.1 is added as the observed image G, as shown in Figure 3 The template used in this implementation is

[0070]

[0071] Step 2: Initialization.

[0072] Step 2.1 Initialization, set the regularization parameter λ = 0.003, the maximum number of iterations K = 100 and the convergence threshold ε = 10 -8 , counter t = 1, set the initial value of reconstructing high-resolution image Set the variable t1=1 used for iterative acceleration and the simulated high-resolution image matrix F that records the results of the previous iteration p , and let F p =F 0 , the iterative step control value l = 0.01.

[0073] Step 2.2: Minimize F2(F). This process uses the gradient projection algorithm combined with the fast convolution algorithm to minimize the L2 norm in the model. The algorithm implementation includes the following 5 steps:

[0074] S1 Super-resolution reconstruction estimation image F k (k=0,1,2…,K) and convolution kernel T and downsample to achieve G-SAF k . Super-resolution reconstruction estimation image F k (k=0,1,2…,K) and convolution kernel T are convolved according to the dictionary arrangement method to complete AF k , A.F. k The size is 262×262, and then AF k The downsampling calculation SX of the image matrix X is defined as shown in formula (7). This downsampling calculation is for AF k Downsampling, converted into the product form of the sparse matrix (7), SAF k It is to downsample first and then perform column downsampling. Due to sparseness, AF is implemented kThe 2×2 pixels in the image are merged into one pixel, which can achieve the purpose of fast calculation. The difference R is obtained by subtracting the downsampled image from the observed image G. <131×131> , that is, to achieve G-SAF k ;

[0075] S2 completes negative gradient The calculation of A T S T (G-SAF k ). T The operation is the same as in formula (8), which is still a sparse matrix product. This calculation is the S of the difference image R T The result is that the difference image R is magnified by 2×2, and each element corresponds to a 2×2 block to obtain S T (G-SAF k );A T S T (G-SAF k ) is calculated by flipping the convolution kernel T by 180° and S T (G-SAF k ) convolution, the convolution result is a 256×256 matrix with size 7 to 262 and columns 7 to 262 as the desired result, that is, the negative gradient of F2(F)

[0076] S3 gradient projection. Elements in F that are less than 0 and correspond to k Those less than 0 are set to 0.

[0077] S4 finds the optimal step length parameter along the echelon descent direction according to formula (10)

[0078] S5 Calculation of β k . in In step 2.2, an iterative threshold reduction algorithm is used to minimize the L1 norm, that is, to minimize F1(F). According to formula (5) λ (β k ) Solved to get

[0079] Step 2.3 Combine the previous iteration result F k Correction F k+1 . Modify the accelerated iteration according to equations (13) to (16).

[0080]

[0081] t1=t2 (16)

[0082] Step 2.4 Judgment Or whether the maximum number of iterations has been reached; if so, let F k =F k +1 Go to step 4; otherwise, let F k =F k+1 , t=t+1, and go to step 2.2. In this implementation, the convergence condition is met when t=67, and the loop ends early and goes to step 4.

[0083] Step 3: Coarse super-resolution reconstruction.

[0084] Step 4: Fine super-resolution reconstruction. In step 3, the regularization parameter λ is set to a constant, and each pixel in the iteratively reconstructed high-resolution image is treated equally, and the pixel information with a larger contribution cannot be highlighted. The weighted L1 minimization model uses larger weights to prevent near-zero terms in the reconstructed signal, and smaller weights to encourage non-zero terms, further sparsely regulating the reconstruction of the high-resolution image.

[0085] Step 5 Output super-resolution reconstructed image The reconstructed high-resolution image is shown in (4). The peak signal-to-noise ratio of the super-resolution restored image and the real image is 29.1215.

[0086] The above is only a preferred embodiment of the present invention, but the present invention should not be limited to the contents disclosed in the embodiment and the drawings. Therefore, any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A single image fast sparse inverse convolution super-resolution reconstruction method, characterized in that: The steps include: Step 1: Data input, read the observed image G <m×n> , resolution improvement factor k and convolution template T <(2p+1)×(2q+1)> ; Step 2: Initialization. The specific process is: Step 2.1 Initialization: set the regularization parameter λ, the maximum number of iterations K and the convergence threshold ε, the counter t = 1, and set the initial value F of the reconstructed high-resolution image. 0 <(km-2p)×(kn-2q)> , set the variable t1 = 1 used for iteration acceleration, and the simulated high-resolution image matrix F that records the results of the previous iteration p , and let X = F 0 , iterative step length control amount l; Step 2.2 minimizes F2(F); Step 2.3 Combine the previous iteration result F k Correction F k+1 ; Step 2.4 Judgment Or whether the maximum number of iterations has been reached; if so, let F k =F k+1 Go to step 4; otherwise, let F k =F k+1 , t=t+1, go to step 2.2; Step 3: Coarse super-resolution reconstruction; the specific process is as follows: Step 3.1: Set the convergence condition threshold ξ = 10^(-9), the number of weighted adjustments p = 15, and the number of sparse super-resolution reconstructions q = 15 for each weighted adjustment, and set the corresponding counting variables m = 1 and n = 1; Step 3.2: Update the weight (i, j) parameter based on the obtained F^k, (i, j) = ((i, j)) / (F^k(i, j) + τ), τ = 0.001; Step 3.3 Iterate and solve F^(k+1. The iterative solution is based on equations (5) and (6). The specific method is the same as steps 2.2 to 2.2 of the third step. Step 3.4 determines whether m is less than q. If so, proceed to step 3.3; otherwise, proceed to step 3.

5. If the maximum number of iterations is reached in step 3.5, the iteration is terminated and the process goes to step 5. Otherwise, the process goes to step 3.

6. In this implementation, when p=14, the constraint condition is met and the loop is terminated in advance and the process goes to step 5. Step 3.6 Determine whether n is less than p. If it is less than p, let Then go to step 3.2; otherwise go to step 5; Step 4: Fine super-resolution reconstruction. In step 3, the regularization parameter λ is set to a constant. Each pixel in the iteratively reconstructed high-resolution image is treated equally, and the pixel information with a larger contribution cannot be highlighted. The weighted L1 minimization model uses larger weights to prevent near-zero terms in the reconstructed signal, and smaller weights to encourage non-zero terms, further sparsely regulating the reconstruction of the high-resolution image. Step 5 Output super-resolution reconstructed image 2. The method for fast sparse inverse convolution super-resolution reconstruction of a single image according to claim 1, characterized in that: In step 2.2, we minimize F2(F). The algorithm implementation consists of the following five steps: S1 Observed image G minus reconstructed image F k The difference R is obtained by convolving the downsampled image with the template T<m×n> , that is, to achieve G-SAF k ; The present invention realizes a fast discrete convolution algorithm for spatial two-dimensional images. In this step, this function is used to complete the reconstruction of the high-resolution image matrix F k Convolution with convolution template T; The present invention realizes a fast discrete convolution algorithm for spatial two-dimensional images. In this step, this function is used to complete the deconvolution of the image matrix F k Convolution with convolution template T; The steps of fast convolution operation in the spatial domain of a two-dimensional image are as follows: ① Find the size of the two-dimensional image F, expressed as m×n, and the size of the convolution template T, expressed as (2p+1)×(2q+1), where p and q can be decimals; ② Set the initial value of the convolution result ConvF, ConvF is a zero matrix of size (m+2p)×(n+2q); ③ Select elements F(i,j) (1≤i≤m,1≤j≤n) from the two-dimensional image matrix F and multiply them with the template T; And add the product to the modules corresponding to rows i to i+2p and columns j to j+2q in ConvF to get a new ConvF; ④ Repeat step ③ until all elements in F have completed the operation of step ③; ⑤Output the convolution result ConvF of F and T; The present invention defines the product form of the sparse matrix SX as shown in formula (7) for downsampling the image matrix X; like but (7) Among them C i =(11L 1) 1×k ,i=1,2,L,m,n. This process SAF k First, the row is downsampled, and then the column is downsampled; due to sparseness, SAF is implemented k The k×k pixel blocks in the image are merged into one pixel, which can achieve the purpose of fast calculation. S2 further completes the negative gradient based on S1 The calculation of A T S T (G-SAF k ); The present invention defines S T The operation is still the same as (7), which is a sparse matrix product; S T The X operation is defined as in formula (8); like but Among them C i =(1 1 L 1) 1×k ,i=1,2,L,m,n. This calculation is S of the difference image R T The result of the operation is that the difference image R is magnified by k×k, and each element corresponds to a k×k pixel block, and S is obtained. T (G-SAF k );A T S T (G-SAF k ) is calculated by flipping the convolution kernel T by 180° and S T (G-SAF k ) convolution, and then extract the matrix of size (km-2p)×(kn-2q) from 2p+1 to km and columns 2q+1 to kn in the convolution result, which is the negative gradient of F2(F) S3 gradient projection further corrects the negative gradient. Elements in F that are less than 0 and correspond to k The values ​​less than 0 are set to 0, that is, S4 finds the optimal step length parameter along the descending direction of the echelon so that the step length is large and the reconstructed image is closer to the observed image Among them, ||·|| F is the Frobenius norm, l is a small constant, (·) + Defined as It is understood as the mean square error between the observed image and the reconstructed image; S5 Calculation of β k . in 3. The method for fast sparse inverse convolution super-resolution reconstruction of a single image according to claim 1, characterized in that: In step 2.2, an iterative threshold reduction algorithm is used to process the regularization constraint term to minimize the L1 norm, that is, to minimize F1(F).

Images