A POCS image super-resolution reconstruction method based on multiple features

Abstract

The present invention discloses a POCS image super-resolution reconstruction method based on multiple features, and belongs to the field of image reconstruction technology. First, a reference frame is constructed according to the local similarity features of the image and the gradient features of the image; then the upsampled sequence LR image and the reference frame are registered using the LK optical flow method; at the same time, the PSF of the reference frame is estimated using the ratio sparse constraint model, and the center coordinates of the PSF action window and the range of the PSF action window are obtained; finally, the reference frame and the PSF are used to generate an initial estimated image of the reconstructed image, and the reference frame is corrected according to the reconstructed image residual to obtain the final reconstructed image. The method of the present invention constructs a multi-feature reference frame image based on the local similarity features of the image and the image gradient features, and uses the ratio sparse constraint model to estimate the PSF of the multi-feature reference frame image, thereby improving the edge retention ability and texture detail information of the reconstructed image, and can effectively achieve the purpose of improving image quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution reconstruction, and in particular relates to a POCS image super-resolution reconstruction method based on multiple features. Background Art

[0002] Image super-resolution reconstruction refers to an image processing technology that uses computers to process low-resolution images or image sequences to reconstruct high-resolution images without upgrading the imaging hardware system. It is widely used in medical image analysis, security monitoring, remote sensing imaging and other fields.

[0003] Currently, super-resolution reconstruction methods based on sequence images are mainly divided into: reconstruction based on traditional algorithms and reconstruction based on deep learning methods. Super-resolution reconstruction based on traditional algorithms mainly includes: interpolation-based methods (reference: DeepuRajan, Subhasis Chaudhuri. Generalized interpolation and its application in super-resolution imaging [J]. Image and Vision Computing, 2001, 19 (13).), iterative back-projection-based methods (reference: Irani M, Peleg S. Improving resolution by image registration [J]. GVGIP: Graphical Models and Image Processing, 1991, 53 (3): 231-239.), maximum a posteriori probability-based methods (Schultz RR, Stevenson R L. Improved definition video frame enhancement [C]. International Conference on Acoustics. IEEE, 1995.) and convex set projection-based methods (Stark H, Oskoui P. High-resolution image recovery from image-plane arrays, using convex projections [J]. Journal of the Optical Society of America A Optics & Image Science, 1989, 6(11): 1715.) In recent years, image super-resolution reconstruction based on deep learning has become a hot topic of research. In 2014, Chao et al. proposed a network with three convolutional layers - SRCNN, which was the first time that deep learning was used for image super-resolution reconstruction (reference: Chao D, Chen CL, He K, et al. Learning a Deep Convolutional Network for Image Super-Resolution[J]. ECCV, 2014.) This method has excellent reconstruction effect and can achieve a large magnification, but at the same time, it has high requirements on the design of the network structure and the training process, requires a large amount of data for training, and the amount of calculation is usually very large (reference: Xie Haiping, Xie Kaili, Yang Haitao. Research Progress of Image Super-Resolution Methods[J]. Computer Engineering and Applications, 2020, 56(19): 34-41.).Compared with deep learning-based methods, traditional methods can add image prior information and do not require a large number of training samples, which has unique advantages. Among many traditional reconstruction methods, the POCS algorithm is a more commonly used reconstruction method because of its intuitive algorithm principle, flexible imaging model, and convenient introduction of a large amount of prior information (reference: Fang Yaoxin, Guo Baofeng, Ma Chao. Super-resolution reconstruction of remote sensing images based on improved point spread function [J]. Laser Surgery, 2019, 43(05): 123-128.). However, the POCS algorithm uses an interpolation method to construct a reference frame during the reconstruction process, and the PSF (Point spread function, PSF) is not estimated during the reconstruction process, resulting in poor results in the reconstructed image in terms of edge and detail information.

[0004] In summary, super-resolution reconstruction based on the traditional POCS algorithm and super-resolution reconstruction based on deep learning each have their own advantages and disadvantages. The advantages of super-resolution reconstruction based on the traditional POCS algorithm are that the algorithm principle is intuitive, the imaging model is flexible, and image prior information can be added, which has unique advantages. The disadvantage is that the image is not effective in terms of edge and detail information. On the contrary, super-resolution reconstruction based on deep learning has good effects and can achieve a large multiple of magnification. The disadvantage is that it has high requirements for the design of the network structure and the training process, requires a large amount of data for training, and the amount of calculation is usually very huge. Summary of the invention

[0005] In view of the above technical problems, the purpose of the present invention is to provide a multi-feature POCS image super-resolution reconstruction method, construct a multi-feature reference frame image based on the local similarity features of the image and the gradient features of the image, use the LK optical flow method for image registration, and adopt the ratio sparse constraint model to estimate the PSF of the image, which not only improves the edge preservation ability of the reconstructed image but also increases the texture detail information of the reconstructed image, thereby effectively improving the image quality.

[0006] To achieve the above object, the present invention provides a multi-feature based image super-resolution reconstruction method, comprising the following steps:

[0007] S1: Input sequence LR image g i (i=1,2,3,4);

[0008] S2: read a frame g1 of the sequence LR image, construct a reference frame f1 according to the local similarity features of the image, construct a reference frame f2 according to the gradient features of the image, and calculate f1 and f2 by weighted mean of pixel values ​​to obtain a multi-feature reference frame image f;

[0009] S3: Upsample the LR images of the other frame sequences except the first frame LR image by 2 times, and use the LK optical flow method to align the upsampled sequence images with the multi-feature reference frame image f, find the center coordinates of the PSF action window of the multi-feature reference frame image, and determine the range of the PSF action window;

[0010] S4: using the ratio sparse constraint model to estimate the PSF of the multi-feature reference frame image f, and calculating the initial estimated value of the pixels within the PSF window range of the reconstructed image based on the multi-feature reference frame image f and the estimated PSF;

[0011] S5: Calculate the residual R between the pixel estimation value within the PSF window range and the pixel value of the current original sequence resolution image;

[0012] S6: According to the relationship between the residual R and the threshold T, the multi-feature reference frame image f is corrected to obtain the reconstructed high-resolution images P1 and P2. The final reconstructed image P is obtained by weighted calculation of P1 and P2 according to the mean pixel value.

[0013] The expression for generating the multi-feature reference frame image f from the reference frame f1 and the reference frame f2 in step S2 is:

[0014]

[0015] Among them, f represents a multi-feature reference frame image, f1 represents a reference frame image f1, f2 represents a reference frame image f2, a1 represents the average pixel value of the reference frame image f1, and b1 represents the average pixel value of the reference frame image f2.

[0016] In step S3, the LK optical flow method, namely the Lucas-Kanade optical flow method, is used to align the upsampled sequence image with the multi-feature reference frame image f.

[0017] The PSF expression of the image reconstructed by using the ratio sparse constraint estimation in step S4 is:

[0018]

[0019] Where C is the cost function, x is a high-frequency clear image in the unknown space, y is the observed image, k is the PSF, λ and ψ are regularization parameters used to control the weights of k and the regularization term.

[0020] The initial estimated value of the pixels within the PSF window range of the reconstructed image is calculated from the multi-feature reference frame image f and the estimated PSF:

[0021]

[0022] Where: denotes the initial estimate of the pixels within the PSF window of the reconstructed image, and f denotes the multi-feature reference frame image. denotes the convolution operation, and k represents the point spread function.

[0023] In step S5, the residual expression between the pixel estimate within the PSF range and the pixel value of the current original sequence resolution image is:

[0024]

[0025] In the formula: R represents the residual between the pixel estimate within the PSF range and the pixel value of the current original sequence resolution image, and L represents the pixel value of the original sequence resolution image. denotes the pixel estimate within the PSF range.

[0026] In step S6, the expression for correcting the multi-feature reference frame image f according to the magnitude relationship between the residual R and the threshold T is:

[0027] R > T, P1 = f + (R - T)

[0028] R < T, P2 = f + (R + T)

[0029] In the formula: R represents the residual, T represents the pre-set threshold, f represents the multi-feature reference frame image, P1 represents the reconstruction result when R > T, and P2 represents the reconstruction result when R < T.

[0030] The expression for obtaining the final reconstructed image P based on P1 and P2 is:

[0031]

[0032] In the formula: P represents the final reconstructed image, P1 represents the reconstruction result when R > T, P2 represents the reconstruction result when R < T, c1 represents the mean pixel value of the reconstructed image P1, and d1 represents the mean pixel value of the reconstructed image P2.

[0033] As described above, the multi-feature-based POCS image super-resolution reconstruction method proposed by the present invention is improved in the construction of high-resolution reference frames and image reconstruction based on PSF. A multi-feature reference frame image is constructed, and a ratio sparse constraint model is used to estimate the PSF of the image, improving the edge preservation ability and texture detail information of the reconstructed image. Compared with other traditional reconstruction algorithms, the reconstruction result of the present invention is significantly better than that of other algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of a multi-feature-based POCS image super-resolution reconstruction method of the present invention;

[0035] Figure 2 It is the sequence of low-resolution real satellite remote sensing image data used in the present invention;

[0036] Figure 3 It is a comparison diagram of the image reconstruction results of the present invention and the reconstruction results of the upsampling method, the iterative back-projection algorithm, and the traditional POCS algorithm; DETAILED DESCRIPTION

[0037] See below Figure 1 to Figure 3 The multi-feature POCS image super-resolution reconstruction method of the present invention is described in detail.

[0038] like Figure 1 As shown, in order to obtain better image reconstruction results, a multi-feature based POCS image super-resolution reconstruction method is provided, comprising the following steps:

[0039] Step S1: Input sequence LR image g i (i=1,2,3,4);

[0040] Step S2: read a frame g1 of the sequence LR image, construct a reference frame f1 according to the local similarity feature of the image, construct a reference frame f2 according to the gradient feature of the image, and calculate f1 and f2 by weighted mean of pixel values ​​to obtain a multi-feature reference frame image f;

[0041] The expression for generating a multi-feature reference frame image f from reference frames f1 and f2 is:

[0042]

[0043] Among them, f represents a multi-feature reference frame image, f1 represents a reference frame image f1, f2 represents a reference frame image f2, a1 represents the average pixel value of the reference frame image f1, and b1 represents the average pixel value of the reference frame image f2.

[0044] Step S3: perform 2-fold upsampling on the LR images of the other frame sequences except the first frame LR image, use the LK optical flow method to align the upsampled sequence images with the multi-feature reference frame image f, find the center coordinates of the PSF action window of the multi-feature reference frame image, and determine the range of the PSF action window;

[0045] Step S4: using the ratio sparse constraint model to estimate the PSF of the multi-feature reference frame image f, and calculating the initial estimated value of the pixels within the PSF window range of the reconstructed image based on the multi-feature reference frame image f and the estimated PSF;

[0046] The PSF expression of the reconstructed image using ratio sparse constraint estimation is:

[0047]

[0048] Where: C is the cost function, x is the high-frequency clear image in the unknown space, y is the observed image, k is the PSF, and λ and ψ are regularization parameters used to control the weights of k and the regularization term.

[0049] The expression for the initial estimate of the pixels within the PSF window of the reconstructed image calculated from the multi-feature reference frame image f and the estimated PSF is:

[0050]

[0051] Where: represents the initial estimate of the pixels within the PSF window of the reconstructed image, f represents the multi-feature reference frame image, represents the convolution operation, and k represents the point spread function.

[0052] Step S5: Calculate the residual between the pixel estimate within the PSF window and the pixel value of the current original sequence resolution image;

[0053] The expression for calculating the residual between the pixel estimate within the PSF range and the pixel value of the current original sequence resolution image is:

[0054]

[0055] Where: R represents the residual between the pixel estimate within the PSF range and the pixel value of the current original sequence resolution image, L represents the pixel value of the original sequence resolution image, represents the pixel estimate within the PSF range.

[0056] Step S6: Modify the multi-feature reference frame image f according to the magnitude relationship between the residual R and the threshold T to obtain the reconstructed high-resolution images P1 and P2, and the final reconstructed image P is calculated by weighted averaging based on the pixel value means of P1 and P2.

[0057] The expression for modifying the multi-feature reference frame image f according to the magnitude relationship between the residual R and the threshold T is:

[0058] R > T, P1 = f + (R - T)

[0059] R < T, P2 = f + (R + T)

[0060] Where: R represents the residual, T represents the pre-set threshold, f represents the multi-feature reference frame image, P1 represents the reconstruction result when R > T, and P2 represents the reconstruction result when R < T.

[0061] The expression for obtaining the final reconstructed image P from P1 and P2 is:

[0062]

[0063] Where: P represents the final reconstructed image, P1 represents the reconstruction result when R>T, P2 represents the reconstruction result when R<T, c1 represents the average pixel value of the reconstructed image P1, and d1 represents the average pixel value of the reconstructed image P2.

[0064] The method of the present invention will be further described below through implementation examples.

[0065] Dataset: Real satellite remote sensing image data is used.

[0066] Evaluation metrics: Information entropy (IE) and average gradient (MG) are used as evaluation metrics.

[0067] Implementation steps:

[0068] The method of the present invention, the upsampling method, the iterative back-projection algorithm, and the traditional POCS algorithm are respectively used for super-resolution reconstruction of real satellite remote sensing image data. Among them, the parameters of the method of the present invention in the experiment are: k = 25, T = 1, and the parameters for estimating the image PSF by the ratio sparse constraint model are: kernel_size = 25, kernel_init = 3.

[0069] For the real satellite remote sensing image data, the image reconstruction results are as Figure 3 shown, the real satellite remote sensing image data (see Figure 2 (a-d)), the reconstruction result of the upsampling method (see Figure 3 (a)), the reconstruction result of the iterative back-projection algorithm (see Figure (3b)), the reconstruction result of the traditional POCS algorithm (see Figure 3 (c)), the reconstruction result of the method of the present invention (see Figure 3 (d)). After quality evaluation, Figure 3 (a), Figure (3b), Figure 3 (c), Figure 3 (d) have information entropies of 6.21, 6.23, 6.30, and 6.56 respectively, Figure 3 (a), Figure (3b), Figure 3 (c), Figure 3 (d) have average gradients of 2.56, 3.55, 2.49, and 3.30 respectively.

[0070] The multi-feature-based POCS image super-resolution reconstruction method of the present invention solves the problem of poor edge preservation of the reconstructed image by the traditional POCS algorithm, and at the same time increases the texture detail information of the reconstructed image. The reconstruction results of the present invention are respectively compared with the reconstruction results of several algorithms such as upsampling, iterative back projection, and convex set projection using information entropy and average gradient for quality evaluation. The comparative analysis concludes that the image reconstruction results of the method of the present invention have been significantly improved in terms of vision and indicators. The processing process of the method of the present invention is not approximated, and the reconstruction process is mainly based on real satellite remote sensing image data.

[0071] The above is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and changes can be made without departing from the principle of the present invention, and these improvements and changes are also regarded as the protection scope of the present invention.

Claims

1. A multi-feature based POCS image super-resolution reconstruction method, characterized in that: It includes the following steps: S1: Input sequence LR image g i (i=1,2,3,4); S2: Read a frame g1 of the sequence LR image, construct a reference frame f1 according to the local similarity features of the image, construct a reference frame f2 according to the gradient features of the image, and calculate the multi-feature reference frame image f by weighted calculation of f1 and f2 according to the mean value of pixel values; S3: Upsample other frame sequences of the LR image except the first frame by 2 times, use the LK optical flow method to register the upsampled sequence image and the multi-feature reference frame image f, find the center coordinates of the PSF action window of the multi-feature reference frame image, and determine the range of the PSF action window; S4: Estimate the PSF of the multi-feature reference frame image f using the ratio sparse constraint model, and calculate the initial estimated value of the pixels within the PSF window of the reconstructed image from the multi-feature reference frame image f and the estimated PSF; S5: Calculate the residual R between the pixel estimated value within the PSF window and the pixel value of the current original sequence resolution image; S6: Correct the multi-feature reference frame image f according to the size relationship between the residual R and the threshold T to obtain the reconstructed high-resolution images P1 and P2, and calculate the final reconstructed image P by weighted calculation of P1 and P2 according to the mean value of pixel values.

2. The multi-feature based POCS image super-resolution reconstruction method according to claim 1, characterized in that: In the step S2, the expression for generating the multi-feature reference frame image f from the reference frame f1 and the reference frame f2 is: The expression for generating the multi-feature reference frame image f from the reference frame f1 and the reference frame f2 is: Where f represents the multi-feature reference frame image, f1 represents the reference frame image f1, f2 represents the reference frame image f2, a1 represents the mean value of pixel values of the reference frame image f1, and b1 represents the mean value of pixel values of the reference frame image f2.

3. The multi-feature based POCS image super-resolution reconstruction method according to claim 1, characterized in that: In the step S3, the LK optical flow method, that is, the Lucas-Kanade optical flow method, is used to register the upsampled sequence image and the multi-feature reference frame image f.

4. The multi-feature based POCS image super-resolution reconstruction method according to claim 1, characterized in that: In the step S4, the expression for estimating the PSF of the reconstructed image using the ratio sparse constraint model is: In the formula: C is the cost function, x is the high-frequency clear image in the unknown space, y is the observed image, k is the PSF, and λ and ψ are regularization parameters used to control the weights of k and the regularization term; The expression for calculating the initial estimated value of the pixels within the PSF window of the reconstructed image from the multi-feature reference frame image f and the estimated PSF is: Where: represents the initial estimated value of the pixels within the PSF window of the reconstructed image, f represents the multi-feature reference frame image, represents the convolution operation, and k represents the point spread function.

5. The multi-feature based POCS image super-resolution reconstruction method according to claim 1, characterized in that: In the step S5, the expression for calculating the residual between the pixel estimated value within the PSF range and the pixel value of the current original sequence resolution image is: Where: R represents the residual between the estimated pixel value within the PSF range and the pixel value of the current original sequence resolution image, L represents the pixel value of the original sequence resolution image, Represents the estimated value of the pixel within the range of the PSF.

6. The multi-feature based POCS image super-resolution reconstruction method according to claim 1, characterized in that: In the step S6, the expression for correcting the multi-feature reference frame image f according to the size relationship between the residual R and the threshold T is: R > T, P1 = f + (R - T) R < T, P2 = f + (R + T) In the formula: R represents the residual, T represents the pre-set threshold, f represents the multi-feature reference frame image, P1 represents the reconstruction result when R > T, and P2 represents the reconstruction result when R < T; The expression for obtaining the final reconstructed image P according to P1 and P2 is: In the formula: P represents the final reconstructed image, P1 represents the reconstruction result when R > T, P2 represents the reconstruction result when R < T, c1 represents the mean value of pixel values of the reconstructed image P1, and d1 represents the mean value of pixel values of the reconstructed image P2.

Images