A blind image super-resolution method based on cross-scale low-rank constraints

Abstract

The present invention discloses a blind image super-resolution method based on cross-scale low-rank constraints, which realizes super-resolution reconstruction of low-resolution images by alternately iteratively estimating high-resolution images and blur kernels. The above method includes: screening pixels in the current high-resolution image that are greater than the gradient threshold in all directions to participate in blur kernel estimation; constructing a cross-scale similar image block group for image blocks in the high-resolution image, and performing low-rank estimation on the group matrix; and using the low-rank matrix to estimate the high-resolution image. The blur kernel and the high-resolution image are jointly modeled, and the blur kernel and the high-resolution image are updated and corrected at the same time; cross-scale similar image blocks are used to provide potential detail information for reconstructing image blocks, and low-rank constraints are performed on the similar image block group matrix to represent the global structure of the data, which is robust to noise. The method of the present invention can accurately estimate the blur kernel and realize super-resolution reconstruction of noisy low-resolution images.

Description

Technical Field

[0001] The present invention relates to the field of image super-resolution, and more specifically to an image blind super-resolution method based on cross-scale low-rank constraints. Background Art

[0002] In the process of acquiring images by imaging devices, it is difficult to directly obtain high-resolution images due to the turbulence effect of the atmosphere, the diffraction of the optical system in the camera equipment, the long imaging distance, and the high cost of high-resolution camera equipment. The higher the resolution of the image, the finer the details, the clearer the texture, and the more useful information it provides. The image super-resolution method uses multiple or single low-resolution images to reconstruct high-resolution images, restore or reconstruct reasonable high-frequency components for low-resolution images, break through the limitations of the inherent sampling frequency of the sensor, and achieve the expansion of low-resolution images in spatial resolution or cutoff frequency, thereby achieving the purpose of improving the spatial resolution of the image. Image super-resolution technology has been widely used in many fields such as remote sensing monitoring, video surveillance, and medical image processing.

[0003] Since it is difficult to obtain multiple low-resolution images of the same scene at the same time in actual situations, the single image super-resolution method is more universal. The single image super-resolution method uses image processing to restore the original high-resolution image from only a single low-resolution image when the degradation process is unknown, effectively improving the resolution of the image. The process of acquiring an image by an imaging system is a positive problem, and restoring the potential original image from a degraded image is a typical image inverse problem. Image super-resolution methods are divided into two categories based on whether the degradation model is known: if the degradation matrix is ​​known, it is called a non-blind super-resolution problem; if the degradation matrix is ​​unknown, it is called a blind super-resolution problem. In practical applications, it is usually impossible to know the degradation model or degradation parameters of the observed image, so the image blind super-resolution method is more practical.

[0004] The image blind super-resolution problem is a serious underdetermined inverse problem. The number of unknown variables to be solved is greater than the number of known equations, and the solution is not unique. In order to find an accurate solution, it is necessary to introduce prior knowledge about the image, called the image prior model, and add it as a regularization constraint to the reconstruction process. This process is called regularization. Regularization constraints provide additional information for solving ill-posed problems and constrain the space of feasible solutions. Most of the existing image super-resolution algorithms are non-blind, and there is not much research on image blind super-resolution. ZSSR uses a predefined or estimated blur kernel to construct a downsampled version of a low-resolution image. Based on the self-similarity of a single image, a convolutional neural network with specific parameters is trained for each image through the low-resolution image and the downsampled image. Although this method can introduce a predefined or estimated blur kernel in the reconstruction process, it does not estimate the blur kernel. Strictly speaking, it does not belong to the image blind super-resolution method. Existing image blind super-resolution algorithms can be roughly divided into two categories according to the process of estimating the blur kernel: independent solution and joint modeling. The independent solution method first estimates the blur kernel using a low-resolution image, and then uses a non-blind super-resolution method to reconstruct a high-resolution image based on the estimated blur kernel. Michaeli et al. estimated the blur kernel by maximizing the similarity of image blocks at different scales in the low-resolution image. Recently, Bell-Kligler et al. proposed a KernelGAN model based on a generative adversarial network to independently estimate the blur kernel. The model trains the generator to generate a downsampled version of the low-resolution image. When the discriminator cannot distinguish between the low-resolution image and the downsampled image, the generator fits the degradation process of the real high-resolution image to estimate the real blur kernel. This method is essentially to estimate the blur kernel by maximizing the self-similarity of the image across scales. Both of these image blind super-resolution methods use the multi-scale self-similarity of the image itself and reconstruct reasonable high-frequency components by obtaining additional information between non-local image blocks. In the independent solution method, blur kernel estimation and high-resolution image estimation are two independent stages. The error of blur kernel estimation will affect the result of subsequent high-resolution image estimation, while the result of high-resolution image estimation cannot correct the error of blur kernel estimation. The above two methods only estimate the blur kernel, and it is necessary to further use non-blind super-resolution methods to reconstruct the high-resolution image. Most of the existing image blind super-resolution algorithms assume a low noise level. The image blind super-resolution problem is an ill-posed inverse problem. In the absence of prior knowledge of noise, image blind super-resolution reconstruction will amplify the noise, resulting in an inability to accurately estimate the blur kernel.

[0005] The present invention discloses an image blind super-resolution method based on cross-scale low-rank constraints. A joint modeling method is used to jointly model blur kernels and high-resolution images. The blur kernel and high-resolution image estimation problem is expressed as an optimization problem of two decision variables. The blur kernel and high-resolution image are estimated simultaneously through alternating iterations. The high-resolution image and blur kernel are estimated by alternating iterations, and the two are gradually updated and corrected. The present invention utilizes the cross-scale self-similarity between the high-resolution image, the low-resolution image and the downsampled image thereof. For each image block in the low-resolution image, multiple similar image blocks are searched in the downsampled image, and the parent block corresponding to the input image block in the high-resolution image is merged with the parent block corresponding to the similar image block in the low-resolution image to construct a similar image block group matrix. Since the cross-scale similar image blocks in the low-resolution image can provide potential detail information for the reconstructed image blocks, the similar image block group matrix is ​​subjected to low-rank constraints and added as a regular term to the objective function. In the alternating iterative solution process, the reconstructed image is forced to restore the high-frequency components, and the blur kernel and high-resolution image are estimated simultaneously. By constraining the matrix of similar image blocks to a low rank, the global structure of the data can be retained while suppressing noise, so that the reconstructed high-resolution image gradually approaches the real high-resolution image during the iterative solution process, and the estimated blur kernel also gradually approaches the real blur kernel. The method disclosed in the present invention can process low-resolution images interfered by noise, accurately estimate blur kernels and high-resolution images, and reconstruct edges and details of high-resolution images. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides an image blind super-resolution method based on cross-scale low-rank constraints to achieve blind super-resolution reconstruction of low-resolution images.

[0007] In order to achieve the above object, the embodiment of the present invention provides the following solution:

[0008] A blind image super-resolution method based on cross-scale low-rank constraints, characterized by comprising the following steps:

[0009] Step 1. Read in the low-resolution image and initialize the high-resolution image;

[0010] Read in the low-resolution image and use the interpolation estimate of the low-resolution image as the initial estimate of the high-resolution image. Set the downsampling factor a, input image block size n, and similar block search control constant γ δ , search for the upper limit of the number of similar image blocks T h With lower limit T l , singular value threshold control constant γ σ , regularization parameter λ h , g and λ c and the blur kernel size.

[0011] Step 2. Estimate blur kernel;

[0012] Fix the estimate x of the current high-resolution image (k) , solve the fuzzy kernel h (k+1) The blur kernel h is updated by solving the following linear equations: (k+1) Estimates:

[0013] [(DX (k) ) T DX (k) +λ h I]h (k+1) =(DX (k) ) T y

[0014] Where D is the downsampling matrix, X (k) For high-resolution images x (k) The corresponding block circulant matrix, y represents the low-resolution image, λ h is the regularization parameter. Since the downsampling matrix D in the above formula cannot be calculated in the frequency domain, the present invention uses the double conjugate gradient algorithm to solve the blur kernel h in the above formula (k+1) In the above formula, the present invention adopts a common method to eliminate the interference of the image smooth area on the blur kernel estimation, that is, the high-resolution image x (k) The gradient of the pixel whose gradient is less than a certain threshold is set to 0. Let the gradient threshold be τ and the blur kernel size be s. Then the selection method of τ is: first divide the image gradient into 4 groups according to its direction, and then set the value of τ to ensure that each group retains at least Pixels are used to estimate the blur kernel. As the number of iterations increases, in order to gradually add more pixels in the reconstructed image to the blur kernel estimation process, the value of τ is reduced to 1.1 times of the previous iteration in each iteration.

[0015] Step 3. Estimate the high-resolution image;

[0016] Fix the current blur kernel estimate h (k+1) , given the estimate x of the current high-resolution image (k) , update the high-resolution image x for the next iteration (k+1) .

[0017] Step 3.1 construct a cross-scale similar image block group matrix;

[0018] Use the current blur kernel to estimate h (k+1) The low-resolution image is downsampled to obtain a downsampled image. and They represent the vector form of the current estimated high-resolution image, low-resolution image, and downsampled image respectively.j x (k) Represents the high-resolution image x (k) The jth image block extracted from is the extraction matrix, and the extracted image block size is n. Indicates Q j x (k) The corresponding sub-block in the low-resolution image y, where is the extraction matrix, the extracted image block size is n / a 2 For any image block in the low-resolution image y There are multiple similar image blocks in the downsampled image, and the KD tree search method is used to search for multiple similar image blocks in the downsampled image. Suppose that the downsampled image y a Search for m-1 similar blocks in Its corresponding parent block in the low-resolution image y is Q j x (k) and Merge to form a matrix of similar image blocks across scales , denoted as Where n is the size of the image block, m is the number of image blocks.

[0019] For detail blocks of different degrees, the number of similar image blocks to be searched is different. The present invention sets the lower limit of the number of similar blocks to be searched as T l , the upper limit is T h , that is, the number of similar blocks satisfies T l ≤m≤T h If the number of similar blocks found is less than T l , then the image block does not participate in the reconstruction of the low-rank constraint; if the number of similar blocks searched is greater than T h , then only select the first T h Similar image blocks are used to construct similar image block groups.

[0020] Step 3.2 Solve the low-rank structure

[0021] Solve the L subproblem, that is, fix the high-resolution image x (k) , solving low-rank structures Using singular value soft threshold operator to group matrices of similar image blocks across scales Low rank matrix estimation of Solve it, and its closed-form solution is

[0022]

[0023] in, is a soft threshold operator, defined as

[0024]

[0025] Among them, Ω j =diag(ω j，1 , ..., ω j，r ) is the weight diagonal matrix, ω j，i， i=1, ..., γ is an adaptive weight.

[0026] The present invention will L j The weight corresponding to the i-th singular value in is expressed as:

[0027]

[0028] Where m is the number of similar image blocks, ∈ = 10 -16 The role is to avoid the denominator being 0, σ j，i (L j ) is a low-rank matrix L j The i-th singular value of is the standard deviation of the noise, where γ σ is the control coefficient.

[0029] Step 3.3 Solve for the high-resolution image x (k+1) ;

[0030] Solve the X subproblem, that is, fix the low-rank structure Solve for high-resolution image x (k+1) Update x by solving the following linear equations (k+1) Estimates:

[0031]

[0032] Where D is the downsampling matrix, H (k+1) is the blur kernel h (k+1) The corresponding block circulant matrix, and The difference operator and The matrix representation of For L j The reconstruction result of the first column of image blocks is: Indicates that the image block will be reconstructed The extracted position is put back to the corresponding position in the image. The present invention adopts the double conjugate gradient method to solve the above linear equations to obtain x (k+1) .

[0033] Step 4. Determine convergence and output the estimate of blur kernel and high-resolution image;

[0034] Through steps 2 and 3, an iterative solution of the objective function is performed to obtain an estimate of the blur kernel h (k+1) , and the estimated x of the high-resolution image (k) Update to x (k+1) If the maximum number of iterations is reached or the iterations converge, the iterations are stopped and the final blur kernel and high-resolution image estimation are output; otherwise, let k = k + 1 and repeat steps 2 and 3.

[0035] Preferably, the input image block size n is 5×5.

[0036] Preferably, the upper limit T of the number m of similar image blocks to be searched is h is 25, the lower limit T l is 5.

[0037] Preferably, the singular value threshold control constant γ σ =0.2.

[0038] Preferably, the regularization parameter λ h 5×10 -5 a 2 N, λ g 1.5×10 -4 and λ c 1.5×10 -6 .

[0039] The present invention combines the cross-scale self-similarity prior of the image with the low-rank prior, and proposes an image blind super-resolution method based on cross-scale low-rank constraints, and adopts a joint modeling method to simultaneously estimate the blur kernel and the high-resolution image. By utilizing the cross-scale self-similarity between the high-resolution image, the low-resolution image and its downsampled image, for each image block in the low-resolution image, multiple similar image blocks are searched in the downsampled image, and the parent block corresponding to the input image block in the high-resolution image is merged with the parent block corresponding to the similar image block in the low-resolution image to construct a similar image block group matrix. The introduction of additional spatial structure information using cross-scale similar image blocks is conducive to the reconstruction of the spatial structure of the image block. By performing an overall low-rank constraint on the similar image block group matrix, the low-rank constraint can represent the global structure of the data and suppress noise, and the blur kernel and the high-resolution image are estimated simultaneously in the alternating iterative process, and the two are gradually updated and corrected each other. The method disclosed in the present invention can process low-resolution images interfered by noise, accurately estimate the blur kernel and the high-resolution image, and reconstruct the edges and details of the high-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of blind super-resolution reconstruction of an image provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of multi-scale self-similarity of an image provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the association between cross-scale self-similar blocks provided by an embodiment of the present invention;

[0044] Figure 4 A flowchart of an image blind super-resolution method based on cross-scale low-rank constraints provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of constructing a cross-scale similar image block group provided by an embodiment of the present invention;

[0046] Figure 6 The average PSNR and SSIM of various methods on the DIVRK dataset provided by the embodiments of the present invention are compared. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The degradation process from a high-resolution image to a low-resolution image is usually represented as a convolutional model as shown below:

[0049] y=(h*x)↓ a +n (1)

[0050] Among them, y represents the low-resolution image, x represents the high-resolution image, * represents the convolution operation, h is the blur kernel, a represents the downsampling factor, and n is the noise. Under the convolution model, the image blind super-resolution method studies how to simultaneously estimate the blur kernel h and the high-resolution image x from the low-resolution image y, such as Figure 1 shown.

[0051] The image blind super-resolution problem is an underdetermined inverse problem. The number of unknown variables to be solved is greater than the number of known equations, and the solution is not unique. In order to find an accurate solution, it is necessary to introduce prior knowledge about the image, called the image prior model, expressed as Ψ(x), and add it to the objective function as a regularization constraint. This process is called regularization. Regularization constraints provide additional information for solving ill-posed problems and constrain the solution space of the objective function. The regularization method is usually expressed as the following optimization problem:

[0052]

[0053] Where, the first term represents the data fidelity term, the second term is the regularization constraint of the high-resolution image, and the third term is the regularization constraint of the blur kernel. c and λ h is the regularization parameter.

[0054] Image prior models play an important role in the problem of blind image super-resolution. How to design effective regularization constraints to describe the prior knowledge or additional information of the image is a key issue in the blind image super-resolution method. Previous work mainly used prior knowledge about image gradients. Image gradients represent the relationship between adjacent pixels of an image and are not sufficient to represent larger-scale image structures. In recent years, some image blind super-resolution methods have used prior information about image blocks. Compared with gradient priors, image blind super-resolution methods based on image block prior knowledge have better accuracy and robustness. Since the image blind super-resolution algorithm realizes the expansion of low-resolution images in spatial resolution or cutoff frequency, additional additional information is required to restore or reconstruct reasonable high-frequency components for low-resolution images, thereby achieving the purpose of improving the spatial resolution of the image. The present invention mines prior knowledge from the multi-scale self-similarity of the image itself, and effectively adds the additional information contained in the multi-scale self-similar structure to the image reconstruction by constructing regularization constraints, converting the image blind super-resolution problem into an optimization problem of finding a solution that satisfies specific constraints.

[0055] like Figure 2As shown in , there are widespread multi-scale self-similar structures in images. This self-similarity is specifically manifested as similar image blocks of the same scale and different scales in the image. That is, when an image block is extracted from an image, similar image blocks can be found in the original scale image and images of other scales. The self-similarity of images at the same scale is called same-scale self-similarity, and the self-similarity at different scales is called cross-scale self-similarity. Similar image blocks of the same scale provide complementary information of sub-pixel displacement, and similar image blocks of different scales provide mapping relationships between high-resolution and low-resolution images. Cross-scale self-similarity can provide additional information for blind super-resolution reconstruction of images. For the input image block in the low-resolution image, similar image blocks are searched in the downsampled image. The blur kernel between the associated input image block and the parent block of its similar block in the low-resolution image is the blur kernel between the associated high-resolution and low-resolution images in the super-resolution degradation model. As shown in Figure 3 As shown in , if the blur kernel that associates the high-resolution image with the low-resolution image is h, and the downsampling factor is a, the image block Q in the high-resolution image corresponds to the image block q in the low-resolution image. If the low-resolution image is convolved with the blur kernel h and a Downsampling generates a downsampled image, and we can get r a Similar to q. Then for the image block q in the low-resolution image, search for a similar image block r in the downsampled image a , then the similar block r a The parent block r in the low-resolution image is similar to the parent block Q of the input image block q in the high-resolution image, providing additional information for reconstructing Q.

[0056] set up and They represent the vector forms of high-resolution images, low-resolution images, and downsampled images, respectively, where N is the number of pixels in the low-resolution image and a is the downsampling factor. j x represents the jth image patch extracted from the high-resolution image x, where is the extraction matrix, and the extracted image block size is n. Indicates Q j The sub-block corresponding to x in the low-resolution image y, where is the extraction matrix, the extracted image block size is n / a 2 Since there are widely similar image blocks across different scales, that is, for any image block in the low-resolution image y In the downsampled image y a The similar image blocks in are in is the extraction matrix, the extracted image block size is n / a 2 , The corresponding parent block in the low-resolution image y is in is the extraction matrix, and the size of the extracted image block is n. Suppose that the downsampled image y a Search for m-1 similar blocks in Its corresponding parent block in the low-resolution image y is Q j x and Similar, Q j x and Merge to form a cross-scale similar image block group matrix P j , denoted as Where n is the size of the image block, and m is the number of image blocks.

[0057] The matrix P of similar image blocks across scales j A low-rank constraint is performed, and this cross-scale low-rank prior is added as a regularization constraint on the image to the objective function of image blind super-resolution reconstruction. The objective function can be expressed as:

[0058]

[0059] Where x and y represent high-resolution and low-resolution images, respectively, h represents blur kernel, a represents downsampling factor, * represents two-dimensional convolution operation, ||·||2 represents l2 norm, P j Represents the image block Q j x and its similar image patch in the low-resolution image The group matrix L j Denotes the observation matrix P j The latent low-rank structure of ||·|| F represents the Frobenius norm of the matrix, rank(·) is the rank function, t is a constant that limits the rank of the matrix, and λ c , h , g is the regularization parameter. The first term in formula (3) is the data fidelity term, which ensures that the reconstructed image conforms to the degradation model; the second term is the cross-scale low-rank constraint term, which provides additional information for estimating the blur kernel and reconstructing the high-resolution image; the third term is the regularization constraint term of the blur kernel; the fourth term is the gradient constraint term, which uses the l2 norm to constrain the smoothness of the image edge. Since the image has cross-scale self-similarity, cross-scale similar image blocks can provide effective additional information for reconstructing high-resolution image blocks and estimating blur kernels, and the low-rank constraint can represent the global structure of the data and suppress noise.

[0060] The present invention solves the estimation of the blur kernel by the optimization problem shown in formula (3). Since formula (3) is non-convex and has no closed solution, the present invention solves the optimization problem shown in formula (3) by an alternating iteration method, that is, the current estimation result x of the high-resolution image is fixed.(k) , update the blur kernel estimate h (k+1) , then fix h (k+1) , update x (k+1) , and iterates until the estimation result converges or the maximum number of iterations is reached.

[0061] Based on the multi-scale self-similarity of the image itself, this embodiment discloses an image blind super-resolution algorithm based on cross-scale low-rank constraints to achieve blind super-resolution reconstruction of low-resolution images. Figure 4 , the above method at least comprises the following steps:

[0062] Step 1. Read in the low-resolution image and initialize the high-resolution image;

[0063] Read in the low-resolution image and use the interpolation estimate of the low-resolution image as the initial estimate of the high-resolution image. Set the downsampling factor a, input image block size n, and similar block search control constant γ δ , search for the upper limit of the number of similar image blocks T h With lower limit T l , singular value threshold control constant γ σ and the regularization parameter λ h , g and λ c Since the blind super-resolution method does not know the actual size of the blur kernel, it is necessary to estimate the blur kernel size or pre-set the blur kernel size before blind super-resolution reconstruction.

[0064] Step 2. Estimate blur kernel;

[0065] Fixed high resolution image x (k) , solve the fuzzy kernel h (k+1) , then the objective function is simplified to:

[0066]

[0067] Convert the above formula into matrix-vector product form:

[0068]

[0069] Where D is the downsampling matrix, X (k) For high-resolution images x (k) The corresponding block circulant matrix, y represents the low-resolution image, λ h is the regularization parameter. Equation (5) is a quadratic function about h and has a closed solution. Let the derivative of equation (5) with respect to h be 0, we can get:

[0070] [(DX (k) ) T DX (k) +λh I]h (k+1) =(DX (k) ) T y (6)

[0071] Since the downsampling matrix D in equation (6) cannot be calculated in the frequency domain, the present invention uses the double conjugate gradient algorithm to solve the blur kernel h in equation (6): (k+1) In formula (6), the present invention adopts a common method to eliminate the interference of the image smooth area on the blur kernel estimation, that is, the high-resolution image x (k) The gradient of the pixel whose gradient is less than a certain threshold is set to 0. Let the gradient threshold be τ and the blur kernel size be s. Then the selection method of τ is: first divide the image gradient into 4 groups according to its direction, and then set the value of τ to ensure that each group retains at least pixels are used to estimate the blur kernel. As the number of iterations increases, in order to gradually add more pixels in the reconstructed image to the blur kernel estimation process, τ The value of is reduced to 1.1 times of the previous iteration.

[0072] Step 3. Estimate the high-resolution image;

[0073] Fix the current blur kernel estimate h (k+1) , given the estimate x of the current high-resolution image (k) , update the high-resolution image x for the next iteration (k+1) , then the objective function is simplified to:

[0074]

[0075] Since L in formula (7) j and x are unknown quantities to be solved, and the unknown low-rank representation L j Depends on the solution x of the equation (k +1) , so the equation has no closed solution. The optimization problem shown in equation (7) is decomposed into L subproblems and X subproblems and solved separately.

[0076] Step 3.1 construct a cross-scale similar image block group;

[0077] For an image block in a low-resolution image, multiple similar image blocks are searched in the downsampled image, and the parent block corresponding to the input image block in the high-resolution image is merged with the parent block corresponding to the similar image block in the low-resolution image to construct a cross-scale similar image block group matrix. A low-rank constraint is imposed on the cross-scale similar image block group matrix, so that the reconstructed high-resolution image gradually approaches the real high-resolution image during the iterative solution process, and the estimated blur kernel also gradually approaches the real blur kernel.

[0078] Use the current blur kernel to estimate h (k+1) The low-resolution image is downsampled to obtain a downsampled image. and They represent the vector form of the current estimated high-resolution image, low-resolution image, and downsampled image respectively. j x (k) Represents the high-resolution image x (k) The jth image block extracted from is the extraction matrix, and the extracted image block size is n. Indicates Q j x (k) The corresponding sub-block in the low-resolution image y, where is the extraction matrix, the extracted image block size is n / a 2 Since there are a wide range of similar image blocks across different scales, that is, for any image block in the low-resolution image y There are multiple similar image blocks in the downsampled image, and the KD tree search method is used to search for multiple similar image blocks in the downsampled image. Suppose that the downsampled image y a Search for m-1 similar blocks in Its corresponding parent block in the low-resolution image y is Q j x (k) and Merge to form a matrix of similar image blocks across scales Recorded as Where n is the size of the image block and m is the number of image blocks. Figure 5 The above process is intuitively illustrated.

[0079] For blocks of detail of different degrees, the number of similar image blocks to be searched is different. The present invention uses the Euclidean distance between the image block and the image block with sub-pixel displacement as the adaptive threshold for judging the similarity of the image blocks. Interpolate and shift the high-resolution image x to generate an image with 1 / 2 sub-pixel displacement. Then for each input image block q i ,exist Find the image block at the corresponding position Let the adaptive threshold be δ d , γ δ As the control coefficient, select the Euclidean distance less than the self-similarity threshold δ d The image block is taken as the similar block of the original image block. In addition, for detail blocks of different degrees, the number of similar image blocks to be searched is different. The present invention sets the lower limit of the number of similar blocks to be searched as T l , the upper limit is T h, that is, the number of similar blocks satisfies T l ≤m≤T h If the number of similar blocks found is less than T l , then the image block does not participate in the reconstruction of the low-rank constraint; if the number of similar blocks searched is greater than T h , then only select the first T h Similar image blocks are used to construct similar image block groups.

[0080] Step 3.2 Solve the low-rank structure

[0081] Solve the L subproblem, that is, fix the high-resolution image x (k) , solving low-rank structures At this time, the objective function shown in formula (7) is simplified to:

[0082]

[0083] in, is the matrix of similar image blocks across scales, L j for low-rank structure.

[0084] The present invention uses the weighted nuclear norm to relax the rank function to constrain L j The low rank property of , formula (8) can be expressed as the following weighted nuclear norm minimization problem:

[0085]

[0086] In the formula, the low-rank term σ j，i is a low-rank matrix L j The i-th singular value of j，i is the singular value σ j，i The corresponding weights. The singular values ​​of the matrix contain important structural information, and the importance is positively correlated with the size of the singular value. The weighted nuclear norm is used to assign different weights to singular values ​​of different sizes, thereby retaining important rank components.

[0087] Using the singular value soft threshold operator to solve equation (9), firstly, the matrix of cross-scale similar image blocks is Perform singular value decomposition:

[0088]

[0089] Among them, U j and V j is a standard orthogonal matrix, ∑ j =diag(σ j，1 , ..., σ j，r ) is the singular value diagonal matrix, σ j，i, i=1,...,r is a matrix The singular values ​​of r = min (n, m), n and m represent the group matrix The number of rows and columns of the low-rank matrix L in formula (9) j It is expressed as the following closed-form solution:

[0090]

[0091] Among them, the soft threshold operator for:

[0092]

[0093] Among them, Ω j =diag(ω j，1 , ..., ω j，r ) is the weight diagonal matrix, w j,i ,i=1,...,r is the adaptive weight.

[0094] Low rank matrix L j The weight corresponding to the singular value of is inversely proportional to the size of the singular value. j The weight corresponding to the i-th singular value in is expressed as:

[0095]

[0096] Where m is the number of similar image blocks, ∈ = 10 -16 The role is to avoid the denominator being 0, σ j，i (L j ) is a low-rank matrix L j The i-th singular value of is the standard deviation of the noise, where γ σ is the control coefficient. Since the calculation weight w j，i When j，i (L j ) is an unknown quantity, so we need to j Initialize the singular values ​​of

[0097] Step 3.3 Solve for the high-resolution image x (k+1) ;

[0098] Solve the X subproblem, that is, fix the low-rank structure Solve for high-resolution image x (k+1) , then the objective function shown in formula (7) is simplified to:

[0099]

[0100] In order to establish the relationship between the expression and the image x, we replace Written as In the form of j The first column is the image block Q j x, The first column of image blocks is denoted as Converting equation (14) into the form of matrix-vector product, it can be expressed as:

[0101]

[0102] Where D is the downsampling matrix, H (k+1) is the blur kernel h (k+1) The corresponding block circulant matrix, and The difference operator and The matrix representation of . Let the derivative of equation (15) with respect to x be 0, then:

[0103]

[0104] In the formula, For L j The reconstruction result of the first column of image blocks is: Indicates that the image block will be reconstructed The extracted position is put back to the corresponding position in the image. The present invention adopts the double conjugate gradient method to solve equation (16) to obtain x (k+1) .

[0105] Step 4. Determine convergence and output the estimate of blur kernel and high-resolution image;

[0106] Through steps 2 and 3, an iterative solution of the objective function is performed to obtain an estimate of the blur kernel h (k+1) , and the estimated x of the high-resolution image (k) Update to x (k+1) If the maximum number of iterations is reached or the iterations converge, the iterations are stopped and the final blur kernel and high-resolution image estimation are output; otherwise, let k = k + 1 and repeat steps 2 and 3.

[0107] Preferably, the input image block size n = 5 × 5, similar block search control constant γ δ is 8, the upper limit T of the number of similar image blocks to search h is 25, the lower limit T l is 5, the singular value threshold control constant γ σ = 0.2, and the regularization parameter λh is 5×10 -5 a 2 N, λg 1.5×10 -4 and λ c 1.5×10 -6 .

[0108] The disclosed method is verified on the DIVRK dataset. Due to video memory reasons, the present invention creates a blind image super-resolution dataset DIVRK according to the method of constructing DIV2KRK. The dataset selects 20 images of different categories with sizes ranging from 816×816 to 904×904 from the DIV2K dataset, including animals, sculptures, aerial photography, architecture, plants, text and people. The DIVRK dataset randomly generates an anisotropic Gaussian blur kernel for each image, convolves it with the image and performs downsampling operations to generate the corresponding simulated low-resolution image. The generation process of the anisotropic Gaussian blur kernel is to randomly set the variance λ1, λ2~U(0.35, 10) in the horizontal and vertical directions, randomly rotate the angle θ~U(-π, π), and then add signal-related additive noise. The noise is positively correlated with the blur kernel coefficient, and the scale factor obeys U(-0.4, 0.4). It is normalized and finally truncated to 11×11, where U(b1, b2) represents a uniform distribution on the interval (b1, b2).

[0109] The present invention uses peak signal-to-noise ratio PSNR and structural self-similarity SSIM as quantitative evaluation indicators. The larger the PSNR and SSIM between the restored image and the true image, the closer the restored image is to the true image. There are not many image blind super-resolution methods at present. The methods proposed by Michael et al. and KernelGAN are two generally recognized image blind super-resolution methods. These two methods actually only perform blur kernel estimation. ZSSR can be combined with real or estimated blur kernels to achieve non-blind super-resolution reconstruction. Michaeli et al. + ZSSR and KernelGAN + ZSSR represent methods that combine the blur kernel estimated by Michaeli et al. and KernelGAN with ZSSR to achieve blind super-resolution reconstruction. Figure 6 The average PSNR and SSIM of the method disclosed in the present invention and the two methods on the DIVRK dataset are quantitatively compared. It can be seen that no matter the resolution is increased by 2 times or 4 times, the average PSNR and SSIM of the method disclosed in the present invention are better than those of the methods of Michaeli et al. + ZSSR and KernelGAN + ZSSR.

[0110] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present invention may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present invention, but will conform to the widest scope consistent with the principles and novel features disclosed in the present invention.

Claims

1. A blind image super-resolution method based on cross-scale low-rank constraints, characterized in that: The following steps are involved: Step 1. Read in the low-resolution image and initialize the high-resolution image; Read in the low-resolution image and use the interpolation estimate of the low-resolution image as the initial estimate of the high-resolution image; set the downsampling factor a, the input image block size n, and the similar block search control coefficient γ δ , search for the upper limit of the number of similar image blocks T h With lower limit T l , singular value threshold control constant γ σ , regularization parameter λ h , g and λ c and blur kernel size; Step 2. Estimate blur kernel; Fix the current high resolution image x (k) Estimate the fuzzy kernel h (k+1) ; Update the blur kernel h by solving the following linear equations (k+1) Estimates: [(DX (k)T DX (k) +λ h I]h (k+1) =(DX (k) ) T y Where D is the downsampling matrix, X (k) For high-resolution images x (k) The block circulant matrix corresponding to the estimation of , y represents the low-resolution image, λ h is the regularization parameter; since the downsampling matrix D in the above formula cannot be calculated in the frequency domain, the double conjugate gradient algorithm is used to solve the blur kernel h (k+1) Estimation; the high-resolution image x (k) The gradient of the pixel whose gradient is less than a certain threshold is set to 0; let the gradient threshold be τ, and the blur kernel size be s, then the selection method of τ is: first divide the image gradient into 4 groups according to its direction, and then set the value of τ to ensure that each group retains at least pixels are used to estimate the blur kernel; as the number of iterations increases, in order to gradually add more pixels in the reconstructed image to the blur kernel estimation process, the value of τ is reduced to 1.1 times of the previous iteration in each iteration; Step 3. Estimate the high-resolution image; Fixed current blur kernel h (k+1) Estimation of, given the current high-resolution image x (k) Estimates of the next iteration of the high-resolution image x (k+1) ; Step 3.1 construct a cross-scale similar image block group matrix; Use the current blur kernel h (k+1) It is estimated that the low-resolution image is downsampled to obtain a downsampled image; and They represent the vector forms of the current estimated high-resolution image, low-resolution image, and downsampled image respectively; Q j x (k) Represents the high-resolution image x (k) The jth image block extracted from is the extraction matrix, and the extracted image block size is n; Indicates Q j x (k) The corresponding sub-block in the low-resolution image y, where is the extraction matrix, the extracted image block size is n / a 2 ; For any image block in the low-resolution image y There are multiple similar image blocks in the downsampled image, and the KD tree search method is used to search for multiple similar image blocks in the downsampled image; Suppose that the downsampled image y a Search for m-1 similar blocks in Its corresponding parent block in the low-resolution image y is Q j x(k) and Merge to form a matrix of similar image blocks across scales Recorded as Where n is the size of the image block, and m is the number of image blocks; For detail blocks of different degrees, the number of similar image blocks to be searched is different. The lower limit of the number of similar blocks to be searched is set to T l , the upper limit is T h , that is, the number of similar blocks satisfies T l ≤m≤T h ; If the number of similar blocks found is less than T l , then the image block does not participate in the reconstruction of the low-rank constraint; if the number of similar blocks searched is greater than T h , then only select the first T h Similar image blocks construct a similar image block group; Step 3.2 Solve the low-rank structure Solve the L subproblem, that is, fix the high-resolution image x (k) , solving low-rank structures Using singular value soft threshold operator to group matrices of similar image blocks across scales Low rank matrix estimation of Solve it, and its closed-form solution is in, is a soft threshold operator, defined as Among them, Ω j =diag(ω j，1 , ..., ω j，r ) is the weight diagonal matrix, ω j，i , i = 1, ..., r is the adaptive weight; L j The weight corresponding to the i-th singular value in is expressed as: Where m is the number of similar image blocks, ∈ = 10 -16 The role is to avoid the denominator being 0, σ j，i (L j ) is a low-rank matrix L j The i-th singular value of is the standard deviation of the noise, where γ σ is the singular value threshold control constant; Step 3.3 Solve for the high-resolution image x (k+1) ; Solve the X subproblem, that is, fix the low-rank structure Solve for high-resolution image x (k+1) ; Update x by solving the following linear equations (k+1) Estimates: Where D is the downsampling matrix, H (k+1) is the blur kernel h (k+1) The corresponding block circulant matrix, and The difference operator and The matrix representation of For L j The reconstruction result of the first column of image blocks is: Indicates that the image block will be reconstructed Put the extracted position back to the corresponding position in the image; use the double conjugate gradient method to solve the above linear equations to obtain x (k+1) ; Step 4. Determine convergence and output the estimate of blur kernel and high-resolution image; Through steps 2 and 3, an iterative solution of the objective function is performed to obtain an estimate of the blur kernel h (k+1) , and the estimated x of the high-resolution image (k) Update to x (k+1) ; If the maximum number of iterations is reached or the iterations converge, stop the iterations and output the final blur kernel and high-resolution image estimate; otherwise, set k = k + 1 and repeat steps 2 and 3.

2. The image blind super-resolution method based on cross-scale low-rank constraint according to claim 1, characterized in that: The Euclidean distance between the image block and the image block with sub-pixel displacement is used as the adaptive threshold for judging the similarity of the image blocks; the high-resolution image x is interpolated and shifted to generate an image with 1 / 2 sub-pixel displacement Then for each input image block q i ,exist Find the image block at the corresponding position Let the adaptive threshold be δ s , γ δ To search for the control coefficient of similar blocks, select the Euclidean distance less than the self-similarity threshold δ d The image block is taken as a similar block of the original image block; the similar block search control coefficient γ δ =8.

3. The image blind super-resolution method based on cross-scale low-rank constraint as claimed in claim 1, characterized in that: The input image block size n is 5×5.

4. The image blind super-resolution method based on cross-scale low-rank constraint according to claim 1, characterized in that: The upper limit T of the number m of similar image blocks to be searched h is 25, the lower limit T l is 5.

5. The image blind super-resolution method based on cross-scale low-rank constraint according to claim 1, characterized in that: The singular value threshold control constant γ σ =0.

2.

6. The image blind super-resolution method based on cross-scale low-rank constraint according to claim 1, characterized in that: The regularization parameter λ h 5×10 -5 a 2 N, λ g 1.5×10 -4 and λ c 1.5×10 -6 .

Images