A deep image super-resolution method based on non-local and local prior

Abstract

The application discloses a kind of depth image super-resolution reconstruction methods based on non-local prior and local prior.The steps are as follows: the high-resolution training image set is blocked outside, find similar block and form similar block group according to the Euclidean distance for each image block, learn and optimize the Gaussian mixture model (GMM) suitable for depth edge discontinuity characteristics, extract non-local prior of training set and construct non-local constraint.Again, analyze the local structure of image in gradient domain and spatial domain: gradient domain introduces adaptive weighted super Laplace distribution based on the correlation between depth and color image local structure, models gradient to enhance depth edge alignment and detail retention;Spatial domain proposes adaptive autoregressive model (AR) based on local gradient similarity hard switching mechanism, dynamically controls color guide weight to obtain depth image local correlation.Finally, the two kinds of prior are fused in the unified optimization framework, and the high-resolution depth image is iteratively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of based on non-local and local priori depth image super-resolution method, specifically related to the image super-resolution reconstruction problem in the field of depth image restoration, belong to computer vision and intelligent information processing field. BACKGROUND

[0002] In recent years, the rapid development of 3D imaging field promotes the emergence of 3D scanning device, 3D scanning device can obtain the depth information of scene in real time, which is used as auxiliary data, plays an important role in three-dimensional reconstruction, virtual reality and other practical applications.However, compared with the corresponding color image, the quality of depth image obtained by depth sensor is relatively low.However, many practical applications require accurate depth information, limited resolution and low-quality image greatly limit the practical application of depth image.

[0003] Depth image super-resolution is an effective technique to recover high-quality depth image from low-resolution input.The commonly used model is based on learning method and based on reconstruction method.Learning-based methods train a dictionary to extract the latent image features of depth image, and represent the depth features through the extracted structure.For reconstruction-based methods, depth image reconstruction is represented as an optimization problem, and the solution is regularized by priori.For example, Markov random field model, autoregressive model, weighted least squares model and total variation model.These methods can flexibly adapt to the geometric structure of depth image, and have stronger robustness to noise.However, they only focus on the relationship between adjacent pixels in local area, and lose the ability to utilize high-order correlation.According to the nature of depth image, depth image super-resolution reconstruction method can be divided into local method and non-local method.Local method up-samples depth image by constraining the relationship between target pixel and its neighbors, in which the local geometric structure of depth image is considered.Non-local self-similarity shows that local structure often repeats multiple times at different positions within the image.According to the advantages of reconstruction method and learning method, and considering the non-locality and locality of image, a depth image super-resolution method based on learning non-local priori and enhanced local priori is proposed, which aims to improve and reconstruct the quality of depth image. SUMMARY

[0004] In order to improve the resolution of depth image, under the premise of reducing artifacts and edge blur, accurately reconstruct high-resolution depth image, a depth image super-resolution method based on learning non-local priori and enhanced local priori is proposed.

[0005] The proposed method includes four parts: non-local priori learning, non-local constraint establishment, local constraint establishment and joint model establishment.

[0006] The non-local priori learning part includes:

[0007] (1) The high-resolution training image set is divided into blocks, and for each image block, m most similar blocks are found according to the Euclidean distance to form a similar block group.

[0008] (2) The similar block group is learned by using an EM algorithm to obtain a Gaussian mixture model with k Gaussian distributions.

[0009] The non-local constraint establishing part comprises:

[0010] (1) The input low-resolution image is divided into blocks, and for each block, similar blocks are found to form a similar block group;

[0011] (2) For each similar block group, its most suitable Gaussian distribution is selected from k Gaussian distributions, and the singular value decomposition of the covariance matrix of the Gaussian distribution is performed to obtain a singular value vector;

[0012] (3) The matrix composed of the singular value vector extracts the basic structure of the image, which can be used as a dictionary, so that the sparse representation of the similar block group under it is solved, and the non-local constraint is constructed.

[0013] The local constraint establishing part comprises:

[0014] (1) The gradient distribution of the image is analyzed, and a super Laplace distribution is used to fit it to establish a gradient constraint;

[0015] (2) The structural similarity of color images and depth images is analyzed, and an AR model based on structural similarity is established to constrain the spatial domain of the depth image.

[0016] The joint model establishing part comprises:

[0017] (1) The non-local constraint and the local constraint are combined to construct a unified framework for the reconstruction of the depth image, and the SBI algorithm is used for solving.

[0018] Compared with the prior art, the beneficial effects of the present application are:

[0019] 1) The present application combines the learning-based method with the reconstruction-based method, which can fully utilize the advantages of the two methods. In addition, the present application simultaneously considers the non-locality and locality of the depth image, and fully utilizes the information possessed by the image. The proposed method performs well in reducing artifacts and restoring fine edges.

[0020] 2) Unlike existing methods that only use the information of the input low-resolution depth image to construct the non-local prior model, the present application proposes to learn a non-local self-similarity model from the external high-resolution depth image based on the Gaussian mixture model of similar block groups. Then the learned non-local model is used to explore the non-local properties of the input depth image. The proposed non-local model can capture the high-order correlation in the image block and restore fine details. In addition, since the non-local model is learned from the external high-resolution depth image, it is effective in suppressing noise.

[0021] 3) For the local information of the depth image, the present application explores it in the gradient domain and the spatial domain, and proposes an enhanced local regularization method. For the gradient domain, a weighted hyper-Laplacian prior is used to fit the gradient distribution of the depth image. For the spatial domain, an AR model based on structural similarity is proposed, and a selective color image guided weighting scheme is proposed according to the structural correlation between the depth image and its corresponding color image. The proposed enhanced local model can flexibly adapt to the spatial variation of the depth image structure, and can well restore sharp edges, reduce texture replication artifacts and edge blurring. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the gradient distribution map of the depth image.

[0023] Figure 2 is the visual effect comparison chart of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0025] 1. Image observation model:

[0026] Depth image super-resolution refers to the technology of reconstructing a high-resolution depth image from a low-resolution observation image. Let x be the high-resolution depth image and y be the low-resolution observation image. The relationship between them can be represented as:

[0027] y = Hx + n

[0028] where H represents the degradation matrix and n represents the additive Gaussian white noise. Since estimating the high-resolution depth image from the low-resolution depth image is an ill-posed problem, effective prior information is usually used to make the problem well-posed. By introducing prior knowledge, the depth image restoration problem can be represented as the following minimization problem:

[0029]

[0030] where, is a data item; E NL (x) is a non-local regularization term; β, λ1, and λ2 are regularization parameters. SSAR (x) + E GD (x) is a local regularization term; β, λ1, and λ2 are regularization parameters.

[0031] 2. Non-local prior learning:

[0032] For each depth image in the training set, it is divided into p x p blocks. Then, for each block, M most similar blocks are searched in a L x L window, and a similar block group with M similar blocks is constructed. For a group of training images, N similar block groups can be obtained For each similar block group its mean vector is To facilitate learning, each similar block group is subtracted by its mean μ n i.e. is used to learn the non-local self-similarity prior. With N similar block groups, a Gaussian mixture model {N(μ k ,∑ k ), k = 1,.., K} with K Gaussian distributions can be learned, and all image blocks in the same similar block group follow the same Gaussian distribution. For each similar block group, its likelihood distribution can be represented as:

[0033]

[0034] By assuming independent distribution among all block groups, the likelihood function can be represented as Then, GMM learning can be achieved by maximizing the following likelihood function:

[0035]

[0036] To optimize the above formula, a hidden variable {γ nk | n = 1,.., N; k = 1,.., K} is introduced. When a similar block group is generated by the k-th Gaussian component, γ nk = 1, otherwise γ nk = 0. It can be solved by optimizing the E-step and M-step iteratively by EM algorithm.

[0037] E-step: For each similar block group, the probability of belonging to each Gaussian model is calculated respectively:

[0038]

[0039] M-step: Maximize the objective function, thus obtaining the estimate of Gaussian parameters:

[0040]

[0041]

[0042] 3. Nonlocal constraint establishment:

[0043] Given a low-resolution depth image, it is first divided into multiple image patches, and for each image patch, its M most similar image patches are searched to form a similar patch group. Then, the most suitable Gaussian component is selected by evaluating the posterior probability of each similar patch group generated by the kth Gaussian component:

[0044]

[0045] The likelihood equation of the above formula can be expressed as:

[0046]

[0047] Assuming that the kth Gaussian component is the most suitable component for the similar patch group , the dictionary can be obtained by singular value decomposition of its covariance matrix:

[0048] ∑ k = D k Λ D k T

[0049] where D k is an orthogonal matrix; Λ is a diagonal matrix of eigenvalues. Since the eigenvectors in D k capture the statistical structure of the nonlocal self-similar patches in the training depth images, D k can be used to represent the structure of the similar patch group. Given the dictionary D k , the sparse representation of the input patch in the similar patch group X can be obtained by the following formula:

[0050]

[0051] 4. Local constraint establishment:

[0052] In order to fully preserve the local geometric structure of the depth image, the invention explores the local correlation in the gradient domain and the spatial domain.

[0053] Gradient distribution model: Many methods formulate the gradient of the depth image as a Laplace distribution. However, investigations show that the mapping is most suitable for the super-Laplace case, rather than the Gaussian or Laplace distribution (as shown in Figure 1 ). Therefore, in order to well fit the gradient distribution, the invention models it as a super-Laplace distribution:

[0054]

[0055] in and W represents the gradient extraction operator along the horizontal and vertical directions. i It is a weighted vector used to assign different weights to each pixel in the horizontal and vertical directions; p (0≤p≤1) is a parameter that controls the sparsity of the super Laplacian operator.

[0056] Because depth images and their corresponding color images are highly correlated in terms of geometric structure, depth images can be reconstructed using color images. Therefore, the weights are defined by the color image weighting term and the depth image weighting term as follows:

[0057] W i =W i d ·W i c

[0058] Where · represents the dot product; W i c It is a color image weighting term that utilizes the rich high-frequency information of color images; W i d It is a depth-weighted term that can adaptively change as the recovered depth image is updated. W i c and W i d The definition is as follows:

[0059]

[0060]

[0061] Where C represents a color image; κ1, κ2, θ c θ d It is a constant.

[0062] AR models based on structural similarity: AR models express the relationships between local pixels, and their objective function is written as:

[0063]

[0064] in

[0065] The guidance of color image is based on the consistency of depth discontinuity and edges of color image. However, when this assumption does not hold, texture replication artifacts are generated in the reconstructed depth image. To solve this difficulty, the present invention proposes an AR model based on structural similarity. Considering the structural consistency between depth image and corresponding color image, it can be divided into the following cases: (1) the pixel is located in the region where the depth image and the color image have the same structure. (2) the pixel is located in the region where the structure of the depth image and the color image is inconsistent with each other. When a pixel is located in the edge region of the depth image, while the corresponding pixel is also located in the edge of the color image, the guidance of the color image can promote the recovery of the depth image. When a pixel is located in the edge of the color image, while its corresponding pixel is located in the smooth region of the depth image, texture replication artifacts will be generated under the guidance of the color image. The present invention measures the similarity between the depth image and the corresponding color image, and uses it to control the guidance of the color image. Therefore, the color image weight is set according to the structural similarity between the depth image and the color image. The measure of structural similarity can be formulated as:

[0066]

[0067] Where p i is the depth image block, z i is the color image block; f(p i ) is the concatenation of gradients in horizontal and vertical directions; <·,·> represents the dot product operation; |·| represents the absolute value operator; ||·||2 is the Euclidean norm of the vector.

[0068] Then, the color image weight term is defined as follows:

[0069]

[0070] Where T represents the threshold value. When the structural similarity is less than T, the color image weight is set to 1 to avoid the influence of the color image. When the structural similarity is greater than T, the influence of the color image can be adjusted according to the structural similarity. i is the index of the target pixel, and j is the index of the adjacent pixel. w c,ij Can be defined as follows:

[0071]

[0072] 5. Algorithm optimization solution:

[0073] The generated super-resolution reconstruction image recovery model is:

[0074]

[0075] Where β, γ, λ1 and λ2 are regularization parameters; A is the weight coefficient {a i，j} corresponding weight matrix; Λ is the set of patch sparse representation; D is the set of subset D k The first term is the data term to ensure that the reconstructed image should be consistent with the degraded low-resolution depth image; the second and third terms are the non-local priors based on external samples, which exploit the non-local redundancy to enhance each local patch; the last two terms are the local priors, which preserve the local structure in the gradient domain and spatial domain. The model is iteratively optimized by Split-Bregman iteration method, and the specific process is as follows:

[0076] (1) Sparse representation of the patch:

[0077]

[0078] Where D is the concatenation of all D k , and Λ is the concatenation of all α km . Since the dictionary is orthogonal, its closed-form solution can be obtained. For each patch x m in a patch group, the solution is:

[0079]

[0080] (2) Image reconstruction:

[0081]

[0082] The optimization problem in the above formula is the p-norm constraint. Using the Split-Bregman algorithm to solve, auxiliary variables are introduced, and the final solution can be obtained by iterating the following steps.

[0083]

[0084]

[0085] In order to verify the effectiveness of the method described in the present application, the following will be compared in detail by experiment. The present application compares the root mean square error (RMSE) under 8 times sampling rate on the Middlebury dataset (the smaller the value, the better the performance). The main comparison methods are:

[0086] AR: J. Yang, X. Ye, K. Li, C. Hou, and Y. Wang, “Color-guided depth recovery from rgb-d data using an adaptive autoregressive model,” IEEE transactions on image processing., vol. 23, no. 8, pp. 3443-3458, 2014.

[0087] TGV: D. Ferstl, C. Reinbacher, R. Ranftl, M. Ruether, and H. Bischof, “Image guided depth upsampling using anisotropic total generalized variation,” in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 993-1000.

[0088] MSJF: X. Shen, C. Zhou, L. Xu, and J. Jia, “Mutual-structure for joint filtering,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3406-3414.

[0089] EIEM: Y. Zuo, Q. Wu, J. Zhang, P. An, Explicit edge inconsistency evaluation model for color-guided depth map enhancement, IEEE Transactions on Circuits and Systems for Video Technology. 28(2) (2018) 439-453

[0090] Liu: X. Liu, D. Zhai, R. Chen, X. Ji, D. Zhao, and W. Gao, “Depth super resolution via joint color-guided internal and external regularizations,” IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 1636-1645, 2019.

[0091] Jiang: Z. Jiang, Y. Hou, H. Yue, J. Yang, and C. Hou,“Depth super-resolution from rgb-d pairs with transform and spatial domain regularization,”IEEE Transactions on Image Processing., vol. 27, no. 5, pp. 2587-2602, 2018.

[0092] DKN: B. Kim, J. Ponce, and B. H. and,“Deformable kernel networks for joint image filtering,”International Journal of Computer Vision, vol. 129, no. 2, p. 579-600, 2021.

[0093] The test results are shown in Table 1. Compared with other methods, the method of the present application always obtains the best results in terms of RMSE at all three scaling factors. At the same time Figure 2 The subjective visual effect comparison chart of the present application at the sampling rate of 8 times is shown. It can be found that the method of the present application can restore better image details and other information.

[0094] Table 1 Comparison of results of various super-resolution reconstruction methods (RMSE)

[0095]

[0096] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for super-resolution reconstruction of depth images based on learning-based nonlocal priors and enhanced local priors, characterized in that, The method comprises four parts: nonlocal prior learning, nonlocal constraint establishment, local constraint establishment, and joint model establishment. The nonlocal prior learning part uses the EM algorithm to learn a Gaussian mixture model on an external high-resolution training image set to extract nonlocal priors; The nonlocal constraint establishment section is used to construct nonlocal prior constraints from the learned Gaussian mixture model, process low-resolution depth images into blocks, and search for the most similar image for each block. Image patches are grouped into similar patch groups; then, for each similar patch group, its similarity is evaluated by the first image patch. The posterior probability generated by each Gaussian component is used to select the most suitable Gaussian component. ， Assume the first The Gaussian component is the most suitable for similar block groups. The components can be obtained by using the singular value decomposition of its covariance matrix to obtain a dictionary: ， in It is an orthogonal matrix; It is a diagonal matrix of eigenvalues; ultimately, using To represent the structure of a similar block group, the blocks in the similar block group... Sparse representation of: ， in These are weight parameters; The local constraint establishment part uses a hyper-Laplacian distribution to fit the gradient distribution of the image, and analyzes the structural similarity between the color image and the depth image. An AR model based on structural similarity is established to constrain the spatial domain of the depth image, and an adaptively weighted hyper-Laplacian distribution is used to constrain the gradient distribution of the image. This adaptively weighted hyper-Laplacian distribution utilizes the strong correlation between the depth image and the corresponding color image in terms of geometric structure to determine the weighting vector. It is represented as: ， in and This represents gradient extraction operators along the horizontal and vertical directions; These are parameters that control the sparsity of the hyperLaplace operator; the weighting vector It is obtained by combining color image weighting terms and depth image weighting terms in the following forms: ， in It is a depth image weighting term that can adaptively change as the reconstructed depth map is updated; It is a weighted term for color images, which utilizes the rich high-frequency information in color images; Represents a color image. , , , It is a constant; Its characteristic is that it first uses structural similarity measurement: ， The structural similarity between depth images and color images was explored, among which... It is a depth image patch. It is a color image block. It is a cascade of gradients in the horizontal and vertical directions. This represents the dot product operation. Represents the absolute value operator. It is the Euclidean norm of the vector; Then, the obtained structural similarity is used to construct an AR model based on structural similarity to constrain the spatial information of the depth image. The AR model based on structural similarity is represented as follows: ， in The color weight term is defined as follows: ， in Indicates the threshold; The joint model establishment section integrates the aforementioned prior constraints into a unified optimization framework to obtain an estimate of the high-resolution image. The resulting super-resolution reconstructed image restoration model is as follows: ， in Represents the degeneracy matrix, It is a high-resolution depth image to be recovered. These are low-resolution observations; , and It is a regularization parameter; It is related to the weighting coefficient The corresponding weight matrix; It is a set with sparse representation; It is a subset The set, .

Images