A dual-channel sparse coding method for image super-resolution reconstruction

Abstract

The present invention discloses a dual-channel sparse coding image super-resolution reconstruction method, which first uses bicubic interpolation and bilinear interpolation algorithms to preprocess LR images, and uses a dictionary training method based on non-local self-similarity to train dictionaries for the preprocessed image blocks; then proposes an improved reconstruction model based on sparse representation, establishes an adaptive dual channel, designs two adaptive coefficients to control the respective proportions of the two channels, and calculates the sparse representation coefficients according to the model; finally, reconstructs the HR image blocks according to the sparse representation coefficients, and fuses all HR image blocks to obtain the final HR image. The present invention improves the robustness of the reconstruction process to a large extent through the improvement and optimization of the model, and improves the quality of the HR image, and has good reconstruction effects both qualitatively and quantitatively.

Description

Technical Field

[0001] The invention relates to a dual-channel sparse coding image super-resolution reconstruction method, belonging to the technical field of image reconstruction. Background Art

[0002] Image super-resolution reconstruction is a very classic application in the field of machine vision. It refers to reconstructing a low-resolution target image into a high-resolution image through corresponding software or hardware methods, that is, improving the resolution of the original target image through this technology. Image super-resolution reconstruction can be divided into two categories: the first category is called super-resolution reconstruction of a single image. It only needs to refer to the current low-resolution target image without considering the super-resolution of other related images. The second category is the super-resolution reconstruction technology of multi-frame videos (multi-images), which requires super-resolution reconstruction technology involving multiple related target images or multiple video frames. Here we focus on the super-resolution reconstruction of a single picture.

[0003] In real life, image super-resolution reconstruction has a wide range of applications, such as security monitoring equipment, medical image analysis, biometric information recognition, microscopic imaging, video restoration, etc., all of which have extremely important application value.

[0004] At present, there are two common image super-resolution reconstruction algorithms: traditional reconstruction methods and learning-based reconstruction methods. Interpolation-based image super-resolution reconstruction methods and reconstruction-based image super-resolution reconstruction methods belong to traditional reconstruction methods. The most widely used interpolation methods include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. The resolution of low-resolution images can be greatly improved through interpolation, but due to the defects of interpolation, the edges of the reconstructed image are blurred. Traditional image reconstruction techniques also include frequency domain-based methods and spatial domain-based methods. SR algorithms based on the spatial domain, including MAP, POCS, IBP, etc., can overcome the shortcomings of frequency domain methods. Therefore, it is widely used.

[0005] The interpolation-based image super-resolution reconstruction method is to calculate the value of a point based on the values ​​of several known points around a certain point and the positional relationship between the surrounding points and the point according to a certain formula. This is the interpolation method. Regarding how to place the points of the original image in the new image and determine the specific coordinates; when calculating unknown points, how many surrounding points are involved, how to calculate the formula, and different options, the interpolation algorithms involved are somewhat different. The use of these interpolation algorithms for super-resolution reconstruction often has limited improvement in image details, so they are less used. Generally speaking, reconstruction through interpolation algorithms between multiple images is also a means.

[0006] In order to solve the shortcomings of traditional reconstruction methods, which lead to blurred edges of generated images, loss of high-frequency detail information, and inability to process images and complex structures, researchers applied deep learning to image super-resolution reconstruction. Due to satisfactory experimental results, image super-resolution reconstruction based on deep learning has gradually become a mainstream method. Starting from the earliest proposed super-resolution SRCNN method, researchers began to discover the shortcomings of the algorithm and gradually improved it. Method structures such as FSRCNN, ESPCN, VDSR, and RCAN were designed and used. After consulting more literature, it was found that the image super-resolution reconstruction of deep convolutional networks and image super-resolution reconstruction of multi-level fusion networks currently used are emerging in an endless stream. Because the thinking of deep learning is very flexible, researchers can design various more satisfactory structural algorithms through continuous in-depth research. The image super-resolution reconstruction method based on deep learning is currently a widely used method and has good development potential in the future.

[0007] At the same time, the development of compressed sensing (CS) theory has provided new ideas for SR technology. According to CS theory, under appropriate conditions, a high-probability downsampling operation is used to reconstruct the LR image generated by the HR image into an HR image. In 2009, Sen et al. applied CS theory to SR image restoration. This method uses the prior knowledge of sparse coding of face images under the wavelet basis. Then, by adding a blur filter, the correlation between the downsampling matrix and the wavelet basis is reduced. However, since natural images are rich in details and have complex structures, the wavelet basis cannot provide them with the optimal sparse coding.

[0008] Under a specific dictionary pair, the LR image patch has the same sparse coefficient as the reconstructed HR image patch. Therefore, some researchers have used this prior constraint to reconstruct the HR image and achieved good results. Traditional information retrieval methods based on sparse representation use a unified HR dictionary and HR dictionary, which limits the performance of information retrieval results. Therefore, Liu et al. introduced a K-nearest neighbor (KNN)-based dictionary in each LRpatch to reconstruct the HR image.

[0009] Fang et al. proposed a shape-adaptive SR reconstruction method that can better reflect the complex structure of multispectral images. Li et al. simultaneously considered the edge and texture features of the image and used the fast sparse coding (FSC) algorithm based on feature classification to learn an over-complete sparse dictionary of HR and LR image patches. Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide a dual-channel sparse coding image super-resolution reconstruction method, which greatly improves the robustness of the reconstruction process and improves the quality of HR images through the improvement and optimization of the model.

[0011] The present invention adopts the following technical solutions to solve the above technical problems:

[0012] A dual-channel sparse coding image super-resolution reconstruction method comprises the following steps:

[0013] Step 1, preprocess the original LR image to obtain a training set consisting of several image blocks;

[0014] Step 2, training the training set using a dictionary training method based on non-local self-similarity, and constructing a dictionary using a K-SVD algorithm;

[0015] Step 3: Based on the dictionary obtained in step 2, an image SR reconstruction model based on dual-channel sparse representation is constructed, and the model is solved by the steepest descent method to obtain the sparse representation coefficients;

[0016] Step 4: Reconstruct each image block using the dictionary and sparse representation coefficients to obtain HR image blocks, and fuse all HR image blocks to obtain the final HR image.

[0017] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0018] The original LR images are interpolated using bicubic interpolation and bilinear interpolation algorithms respectively to obtain the training images X1 and X2 corresponding to each algorithm. Image blocks are extracted from the training images X1 and X2 to form the training set Q h .

[0019] As a preferred solution of the present invention, the specific process of step 3 is as follows:

[0020] Based on the dictionary obtained in step 2, the image SR reconstruction model based on dual-channel sparse representation is constructed as follows:

[0021]

[0022] Where α represents the sparse representation coefficient, c1 and c2 represent the adaptive channel coefficients, Y represents the HR image block after sparse processing, H represents the degradation matrix, and D h is a dictionary, λ is a positive constant, β is a constant, w is a constant, τ i is the local adaptive coefficient, x i (u,v) means in D h α and The i-th element of the different matrices between represents the L1 norm, represents the L2 norm, M×N represents the size of the HR image block;

[0023] The steepest descent method is used to solve the model, and the sparse representation coefficient is obtained through iterative calculation. The adaptive channel coefficient of the nth iteration is calculated according to the following formula: and

[0024]

[0025]

[0026] Among them, T (n) is the intermediate variable of the nth iteration, is the adaptive channel coefficient of the n-1th iteration, h is the amplification factor, U is the adaptive channel threshold, R (n) is the residual index of the nth iteration, It represents the average value of the results of the first K iterations before the n-1th iteration, where K is a constant;

[0027] When the iteration termination condition is met, the sparse representation coefficient of the nth iteration is output as:

[0028] α (n+1) =α (n) -r (n) R (n)

[0029] Among them, α (n) is the sparse representation coefficient of the nth iteration, r (n) is the learning rate of the nth iteration,

[0030]

[0031]

[0032] in, represents the shift matrix that shifts the image horizontally by u pixels, Represents a shift matrix that shifts the image vertically by v pixels.

[0033] As a preferred solution of the present invention, the residual index is defined as follows:

[0034]

[0035] is defined as follows:

[0036]

[0037] Among them, R represents the residual index, represents the average value of the results of the first K iterations before the nth iteration, R (n+1-i) represents the residual index of the n+1-ith iteration, α (n+1-i) Represents the sparse representation coefficients of the n+1-ith iteration.

[0038] As a preferred solution of the present invention, the HR image block in step 4 is represented as follows:

[0039] Y=HD h α+N

[0040] Among them, D h is a dictionary, α represents the sparse representation coefficient, H represents the degradation matrix, N represents Gaussian white noise, and Y represents the HR image block after sparse processing.

[0041] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0042] The present invention designs a training method based on non-local self-similarity to train a dictionary, which makes full use of the redundant characteristics of the image. A reconstruction model based on sparse representation is improved, including a fidelity term and a regularization term. In terms of fidelity, an adaptive dual-channel optimization model is established, which not only ensures good robustness but also obtains high-quality reconstruction. Two adaptive coefficients are designed to control the respective proportions of the two channels. In addition, an adaptive regularization term is constructed based on the image spatial features to sharpen the edges and suppress noise. Simulation results show that the method of the present invention has good reconstruction effects both qualitatively and quantitatively. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an overall architecture diagram of a dual-channel sparse coding image super-resolution reconstruction method of the present invention.

[0044] Figure 2 This is a schematic diagram of a model based on sparse representation. DETAILED DESCRIPTION

[0045] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0046] In order to ensure that the network model has good robustness and that the reconstructed image can solve the problems of edge sharpening and noise suppression, the present invention proposes a dual-channel sparse coding image super-resolution reconstruction method. Figure 1 As shown, the specific steps are as follows:

[0047] Step 1: Use the bicubic interpolation algorithm and the bilinear interpolation algorithm to interpolate the original LR image to obtain a training set.

[0048] Use the bicubic interpolation and bilinear interpolation algorithms to interpolate the original LR images respectively to obtain the training set images X1 and X2. Use X1 and X2 to form the training set Q h =[x1,x2,…,x q ], x i Represents the vector form of the i-th image block.

[0049] Step 2: Use the dictionary training method based on non-local self-similarity to train the training set, and use the K-SVD algorithm to construct the dictionary D h , making full use of the redundant characteristics of the image.

[0050] Step 3: Establish an image SR reconstruction model based on dual-channel sparse representation. The L1 norm and L2 norm are used simultaneously in the model, and adaptive parameters are introduced to solve the model to obtain the sparse representation coefficients.

[0051] The two-channel model is:

[0052]

[0053] Where c1 and c2 are adaptive channel coefficients (ACCs). ACCs are used to control the ratio of L1 norm channels to L2 norm channels.

[0054] In order to determine ACCs and make SR reconstruction more adaptable, according to the previous expression, the residual index is introduced as:

[0055]

[0056] R is the residual index. When R is relatively large, the robustness of the algorithm needs to be enhanced.

[0057] Introduction represents the average value of the K iteration results before this iteration, giving is defined as follows:

[0058]

[0059] Where K is a constant.

[0060] Finally, the ACCs of the nth iteration can be calculated.

[0061] The following is a brief introduction to sparse representation. Figure 2 This is a schematic diagram of a model based on sparse representation.

[0062] In the general image degradation model, the HR image Z is represented by the degradation matrix H and the noise N as:

[0063] Y=HZ+N

[0064] Z represents the original HR image, Y represents the LR image, H represents the degradation matrix, and N represents Gaussian white noise.

[0065] The solution of the image degradation equation is a highly uncertain problem and its solution is not unique. CS theory points out that Z can be obtained in an overcomplete dictionary D h Sparse coding is performed in, which can be expressed as:

[0066] Z=D h α

[0067] Then we can get:

[0068] Y=HD h α+N

[0069] In the framework of maximum a posteriori probability theory, the image SR reconstruction problem is modeled as a sparse regularized optimization problem:

[0070]

[0071] α is the sparse representation coefficient of the image block x. ||α||1 is the L1 norm of α, which represents the sum of the absolute values ​​of all elements in α, and λ is a positive constant.

[0072] In order to further improve the reconstruction effect, a new spatial domain constraint is added, which is expressed as follows:

[0073]

[0074] ν(X) represents the reconstruction of the spatial constraint term, and β is a constant used to balance the proportion of each term. ν(X) is a very important parameter that determines whether the performance of the network model can be further improved.

[0075] The bilateral TV (BTV) regularization term is proposed as a feature transformation on T1, expressed as:

[0076]

[0077] is the shift matrix that can shift the image X by u and v pixels horizontally and vertically, w is a constant whose value is set to 3, and τ is a constant determined by experiments.

[0078] By combining and improving the above formula, we can get:

[0079]

[0080] Next, we need to take some measures to sharpen the edges. It can be seen that the larger the τ, the sharper the edges of the reconstructed HR image, but the greater the noise. On the contrary, a smaller τ can smooth the image and reduce noise, but blur the edges.

[0081] An adaptive regularization term is given below:

[0082]

[0083] τ i is the local adaptive coefficient,

[0084] Based on the above analysis, the image reconstruction problem is transformed into the following model optimization problem:

[0085]

[0086] There is no closed form solution for the minimization problem above. The iterative solution framework is:

[0087] α (n+1) =α (n) -r (n) R (n)

[0088] α (n) is the sparsity coefficient in the n-th iteration, r (n) is the learning rate calculated at the nth iteration.

[0089] Calculate the adaptive channel coefficient according to the formula and

[0090]

[0091]

[0092]

[0093] By setting To determine the learning rate:

[0094]

[0095]

[0096] Step 4: Use the sparse representation coefficients to reconstruct the image blocks in the training set to obtain HR image blocks, and fuse all the HR image blocks to obtain the final HR image.

[0097] Example:

[0098] Step 1: Select Yang91 as the training set and Set5 and Set14 as the test sets.

[0099] Step 2: In order to make the comparison experiment more convenient and intuitive, all images are evenly cut into the same size, i.e. 256 × 256. In order to blur the original image and obtain its LR image, the size of the image block is set to 10 × 10 with an overlap of 2 pixels.

[0100] Step 3: Add Gaussian noise and perform downsampling. The Gaussian kernel size is 7×7, and the downsampling factor is set to ×2, ×3, and ×4 according to different reconstruction experiments.

[0101] Step 4: Load the training set and start training the network model.

[0102] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A dual-channel sparse coding image super-resolution reconstruction method, characterized in that: The steps include: Step 1, preprocess the original LR image to obtain a training set consisting of several image blocks; Step 2, training the training set using a dictionary training method based on non-local self-similarity, and constructing a dictionary using a K-SVD algorithm; Step 3: Based on the dictionary obtained in step 2, an image SR reconstruction model based on dual-channel sparse representation is constructed, and the model is solved by the steepest descent method to obtain the sparse representation coefficients. The specific process is as follows: Based on the dictionary obtained in step 2, the image SR reconstruction model based on dual-channel sparse representation is constructed as follows: Where α represents the sparse representation coefficient, c1 and c2 represent the adaptive channel coefficients, Y represents the HR image block after sparse processing, H represents the degradation matrix, and D h is a dictionary, λ is a positive constant, β is a constant, w is a constant, τ i is the local adaptive coefficient, x i (u,v) means in D h α and The i-th element of the different matrices between represents the L1 norm, represents the L2 norm, M×N represents the size of the HR image block; The steepest descent method is used to solve the model, and the sparse representation coefficient is obtained through iterative calculation. The adaptive channel coefficient of the nth iteration is calculated according to the following formula: and Among them, T (n) is the intermediate variable of the nth iteration, is the adaptive channel coefficient of the n-1th iteration, h is the amplification factor, U is the adaptive channel threshold, R (n) is the residual index of the nth iteration, It represents the average value of the results of the first K iterations before the n-1th iteration, where K is a constant; When the iteration termination condition is met, the sparse representation coefficient α of the n+1th iteration is output (n+1) for: α (n+1) ＝α (n) -r (n) R (n) Among them, α (n) is the sparse representation coefficient of the nth iteration, r (n) is the learning rate of the nth iteration, in, represents the shift matrix that shifts the image horizontally by u pixels, Represents a shift matrix that moves the image vertically by v pixels; Step 4: Reconstruct each image block using the dictionary and sparse representation coefficients to obtain HR image blocks, and fuse all HR image blocks to obtain the final HR image.

2. According to the dual-channel sparse coding image super-resolution reconstruction method of claim 1, it is characterized in that: The specific process of step 1 is as follows: The original LR images are interpolated using bicubic interpolation and bilinear interpolation algorithms respectively to obtain the training images X1 and X2 corresponding to each algorithm. Image blocks are extracted from the training images X1 and X2 to form the training set Q h .

3. The image super-resolution reconstruction method of dual-channel sparse coding according to claim 1 is characterized in that: The residual index is defined as follows: is defined as follows: Among them, R represents the residual index, represents the average value of the results of the first K iterations before the nth iteration, R (n+1-i) represents the residual index of the n+1-ith iteration, α (n+1-i) Represents the sparse representation coefficients of the n+1-ith iteration.

4. The image super-resolution reconstruction method of dual-channel sparse coding according to claim 1, characterized in that: The HR image block in step 4 is represented as follows: Y=HD h α+N Among them, D h is a dictionary, α represents the sparse representation coefficient, H represents the degradation matrix, N represents Gaussian white noise, and Y represents the HR image block after sparse processing.

Images