Depth image super-resolution method based on non-local and local prior

Abstract

The invention discloses a depth image super-resolution reconstruction method based on non-local prior and local prior. The method mainly comprises the following steps of: firstly, carrying out blocking processing on a high-resolution training image set, searching M similar blocks of each image block according to an Euclidean distance to form similar blocks, and carrying out learning on a Gaussian mixture model by using a similar block group through an EM (Expectation Maximization) algorithm so as to extract non-local priori of the training set; and then carrying out non-local feature constraint on the input low-resolution image by using the learned non-local priori. Analyzing the local structure of the image in a gradient domain and a spatial domain, and for the gradient domain, introducing hyper-Laplacian prior to model gradient distribution; for a spatial domain, an autoregression model based on structural similarity is provided to acquire local correlation of a depth image. And finally, fusing the learned non-local priori and the enhanced local priori in a unified optimization framework to obtain estimation of a high-resolution image.

Description

Technical Field

[0001] The present invention relates to a deep image super-resolution method based on non-local and local priors, and specifically to an image super-resolution reconstruction problem in the field of deep image restoration, belonging to the field of computer vision and intelligent information processing. Background Art

[0002] In recent years, the rapid development of 3D imaging has promoted the emergence of 3D scanning devices, which can obtain the depth information of the scene in real time. This information plays an important role as auxiliary data in practical applications such as 3D reconstruction and virtual reality. However, compared with the corresponding color images, the quality of the depth images obtained by the depth sensor is relatively low. However, many practical applications require accurate depth information, and the limited resolution and low-quality images greatly restrict the practical application of depth images.

[0003] Deep image super-resolution is an effective technique for restoring high-quality deep images from low-resolution inputs. Commonly used models are learning-based methods and reconstruction-based methods. Learning-based methods train dictionaries to extract latent image features of deep images and represent deep features by the extracted structures. For reconstruction-based methods, deep image reconstruction is formulated as an optimization problem and the solution is regularized with a prior. For example, Markov random field models, autoregressive models, weighted least squares models, and total variation models. These methods can flexibly adapt to the geometric structure of the depth image and are more robust to noise. However, they only focus on the relationship between neighboring pixels in local areas and lose the ability to exploit high-order correlations. According to the properties of the depth image, deep image super-resolution reconstruction methods can be divided into local methods and non-local methods. Local methods upsample the depth image by constraining the relationship between the target pixel and its neighbors, in which the local geometric structure of the depth image is taken into account. Non-local self-similarity indicates that local structures are often repeated multiple times at different locations within the image. According to the advantages of reconstruction methods and learning methods, and considering the non-locality and locality of images at the same time, a depth image super-resolution method based on learned non-local prior and enhanced local prior is proposed, aiming to improve and reconstruct the quality of depth images. Summary of the invention

[0004] In order to improve the resolution of depth images and accurately reconstruct high-resolution depth images while reducing artifacts and edge blur, the present invention proposes a depth image super-resolution method based on learned non-local prior and enhanced local prior.

[0005] The proposed method includes four parts: non-local prior learning, non-local constraint establishment, local constraint establishment and joint model establishment;

[0006] The non-local prior learning part includes:

[0007] (1) Divide the high-resolution training image set into blocks, and for each image block, find the m most similar blocks to form a similar block group based on the Euclidean distance;

[0008] (2) Use the EM algorithm to learn similar block groups to obtain a Gaussian mixture model with k Gaussian distributions.

[0009] The non-local constraint establishment part includes:

[0010] (1) Divide the input low-resolution image into blocks and search for similar blocks for each block to construct a similar block group;

[0011] (2) For each similar block group, the most suitable Gaussian distribution is selected from the k Gaussian distributions, and the covariance matrix of the Gaussian distribution is singularly decomposed to obtain a singular value vector;

[0012] (3) The matrix composed of singular value vectors extracts the basic structure of the image and can be used as a dictionary to solve the sparse representation of similar block groups and use it to construct non-local constraints.

[0013] The local constraint establishment part includes:

[0014] (1) Analyze the gradient distribution of the image and fit it with the hyper-Laplacian distribution to establish gradient constraints;

[0015] (2) The structural similarity between color images and depth images is analyzed, and an AR model based on structural similarity is established to constrain the spatial domain of the depth image.

[0016] The establishment part of the joint model includes:

[0017] (1) Non-local constraints and local constraints are combined to build a unified framework for depth image reconstruction, which is solved using the SBI algorithm.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1) The present invention combines the learning-based method with the reconstruction-based method, which can fully utilize the advantages of both methods. In addition, the present invention simultaneously considers the non-locality and locality of the depth image, making full use of the information possessed by the image. The proposed method performs well in reducing artifacts and restoring fine edges.

[0020] 2) Unlike existing methods that only use the information of the input low-resolution depth image to build a non-local prior model, the present invention proposes to learn a non-local self-similar model from an external high-resolution depth image through a Gaussian mixture model based on similar block groups. The learned non-local model is then used to explore the non-local properties of the input depth image. The proposed non-local model can capture high-order correlations in image patches and restore fine details. In addition, since the non-local model is learned from an external high-resolution depth image, it is effective in suppressing noise.

[0021] 3) For the local information of the depth image, the present invention explores it in the gradient domain and the spatial domain, and proposes an enhanced local regularization method. For the gradient domain, a weighted hyper-Laplacian prior is used to fit the gradient distribution of the depth image. For the spatial domain, an AR model based on structural similarity is proposed, and a selective color image guided weighting scheme is proposed according to the structural correlation between the depth image and its corresponding color image. The proposed enhanced local model can flexibly adapt to the spatial variation of the depth image structure, can well restore sharp edges, and reduce texture replication artifacts and edge blur. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the depth image gradient distribution map.

[0023] Figure 2 This is a comparison chart of the visual effects of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0025] 1. Image observation model:

[0026] Depth image super-resolution refers to the technology of reconstructing a high-resolution depth image from a low-resolution observation image. Let x be the high-resolution depth image and y be the low-resolution observation image. The relationship between them can be expressed as:

[0027] y=Hx+n

[0028] Where H represents the degradation matrix and n represents additive Gaussian white noise. Since estimating a high-resolution depth image from a low-resolution depth image is an ill-posed problem, effective prior information is usually used to make the problem well-posed. By introducing prior knowledge, the problem of depth image restoration can be expressed as the following minimization problem:

[0029]

[0030] in, is a data item; E NL (x) is the non-local regularization term; E SSAR (x)+E GD (x) is the local regularization term; β, λ 1 and λ 2 is the regularization parameter.

[0031] 2. Non-local prior learning:

[0032] For each depth image in the training set, it is divided into p×p blocks. Then, for each block, the M most similar blocks are searched in the L×L window, and a similar block group with M similar blocks is constructed. For a set of training images, N similar block groups can be obtained. For each similar block group Its mean vector is To facilitate learning, each group of similar blocks Subtract its mean μ n Right now Used to learn non-local self-similarity priors. Using N similar block groups, we can learn a Gaussian mixture model with K Gaussian distributions {N(μ k ,∑ k )}, k = 1, .., K, and all image blocks in the same similar block group obey the same Gaussian distribution. For each similar block group, its likelihood distribution can be expressed as:

[0033]

[0034] By assuming that all block groups are independently distributed, the likelihood function can be expressed as Then, GMM learning can be achieved by maximizing the following likelihood function:

[0035]

[0036] In order to optimize the above formula, a hidden variable {γ nk |n=1,…,N;k=1,…,K}. When generated by the kth Gaussian component γ nk =1, otherwise γ nk = 0. It can be optimized by iteratively solving E-step and M-step through the EM algorithm.

[0037] E-step: For each similar block group, calculate the probability that it belongs to each Gaussian model:

[0038]

[0039] M-step: Maximize the objective function to estimate the Gaussian parameters:

[0040]

[0041]

[0042] 3. Establishment of non-local constraints:

[0043] Given a low-resolution depth image, it is first divided into multiple image blocks, and for each image block, its most similar M image blocks are searched to form a similar block group. Then, the most appropriate Gaussian component is selected by evaluating the posterior probability generated by the kth Gaussian component for each similar block group:

[0044]

[0045] The likelihood equation of the above formula can be expressed as:

[0046]

[0047] Assume that the kth Gaussian component is the most suitable for similar blocks The components of can be obtained by singular value decomposition of its covariance matrix:

[0048] ∑ k =D k ΛD k T

[0049] Where D k is an orthogonal matrix; Λ is a diagonal matrix of eigenvalues. k The feature vector in captures the statistical structure of non-local self-similar patches in the training depth image, so D k It can be used to represent the structure of similar block groups. Given a dictionary D k , the input blocks in the similar block group X can be obtained by the following formula The sparse representation of is:

[0050]

[0051] 4. Establishment of local constraints:

[0052] In order to fully preserve the local geometric structure of the depth image, the present invention explores the local correlation in the gradient domain and the spatial domain.

[0053] Gradient distribution model: Many methods formulate the gradient of the depth image as a Laplacian distribution. However, investigations show that the mapping works best for the super-Laplacian case, rather than Gaussian or Laplacian distributions (e.g. Figure 1Therefore, in order to fit the gradient distribution well, the present invention models it as a super Laplace distribution:

[0054]

[0055] in and represents the gradient extraction operator along the horizontal and vertical directions; W i is a weighting vector used to assign different weights to each pixel in the horizontal and vertical directions; p (0≤p≤1) is a parameter that controls the sparsity of the super Laplacian operator.

[0056] Since the depth image and its corresponding color image have a strong correlation in terms of geometric structure, the depth image can be reconstructed with the help of the color image. Therefore, the weights are defined by the color image weighting term and the depth image weighting term as follows:

[0057] W i =W i d ·W i c

[0058] where · represents the dot product; W i c is the color image weighting term, which utilizes the rich high-frequency information of the color image; W i d W is a depth weighting term, which can be adaptively changed as the restored depth image is updated. i c and W i d is defined as follows:

[0059]

[0060]

[0061] Where C represents the color image; κ 1 , κ 2 ,θ c ,θ d is a constant.

[0062] AR model based on structural similarity: The AR model expresses the relationship between local pixels, and its objective function is written as:

[0063]

[0064] in

[0065] The guidance of color images is based on the consistency of depth discontinuities with the edges of color images. However, when this assumption does not hold, texture duplication artifacts will be generated in the reconstructed depth image. To address this difficulty, the present invention proposes an AR model based on structural similarity. Considering the structural consistency between the depth image and the corresponding color image, it can be divided into the following cases: (1) The pixel is located in the region where the depth image and the color image have the same structure. (2) The pixel is located in the region where the structures of the depth image and the color image are inconsistent with each other. When a pixel is located in the edge region of the depth image and the corresponding pixel is also located at the edge of the color image, the guidance of the color image can promote the recovery of the depth image. When a pixel is located at the edge of the color image and its corresponding pixel is located in the smooth region of the depth image, texture duplication artifacts will be generated under the guidance of the color image. The present invention measures the similarity between the depth image and the corresponding color image and uses it to control the guidance of the color image. Therefore, the color image weight is set according to the structural similarity between the depth image and the color image. The measure of structural similarity can be formulated as:

[0066]

[0067] where p i is the depth image patch, z i is a color image block; f(p i ) is the cascade of gradients in the horizontal and vertical directions; <·, ·> represents the dot product operation; |·| represents the absolute value operator; ||·|| 2 is the Euclidean norm of the vector.

[0068] Then, the color image weight term is defined as follows:

[0069]

[0070] Where T is the threshold. When the structural similarity is less than T, the color image weight is set to 1 to avoid the influence of the color image. When the structural similarity is greater than T, the influence of the color image can be adjusted according to the structural similarity. i is the index of the target pixel and j is the index of the adjacent pixel. w c,ij It can be defined as follows:

[0071]

[0072] 5. Algorithm optimization solution:

[0073] The generated super-resolution reconstruction image restoration model is:

[0074]

[0075] Among them, β, γ, λ 1 and λ2 is the regularization parameter; A is the weight coefficient {a i，j} corresponding weight matrix; Λ is the set of sparse representations of slices; D is the subset D k The first item is a data item to ensure that the reconstructed image should be consistent with the degraded low-resolution depth image; the second and third items are non-local priors based on external samples, using non-local redundancy to enhance each local block; the last two items are local priors, which maintain local structures in the gradient domain and spatial domain. The model is iteratively optimized using the Split-Bregman iterative method. The specific process is as follows:

[0076] (1) Sparse representation of blocks:

[0077]

[0078] Where D is all D k cascade, Λ is the sum of all α km Since the dictionary is orthogonal, its closed-form solution can be obtained. For each block x in a block group m , which is solved as follows:

[0079]

[0080] (2) Image reconstruction:

[0081]

[0082] The optimization problem in the above formula is a p-norm constraint. The Split-Bregman algorithm is used to solve it and introduce auxiliary variables The final solution can be obtained by iterating the following steps.

[0083]

[0084]

[0085] In order to verify the effectiveness of the method described in the present invention, a detailed comparison will be conducted through experiments below. The present invention compares the root mean square error (RMSE) at 8 times the sampling rate on the Middlebury data set (the smaller the value, the better the performance). The main comparison methods are:

[0086] AR:J.Yang,X.Ye,K.Li,C.Hou,and Y.Wang,“Color-guided depth recoveryfrom rgb-d data using an adaptive autoregressive model,”IEEE transactions onimage processing.,vol.23,no.8,pp.3443-3458,2014.

[0087] TGV:D.Ferstl,C.Reinbacher,R.Ranftl,M.Ruether,and H.Bischof,“Imageguided depth upsampling using anisotropic total generalized variation,”inProceedings of the IEEE International Conference on Computer Vision,2013,pp.993-1000.

[0088] MSJF:X.Shen,C.Zhou,L.Xu,and J.Jia,“Mutual-structure for jointfiltering,”in Proceedings of the IEEE International Conference on ComputerVision,2015,pp.3406-3414.

[0089] EIEM:Y.Zuo,Q.Wu,J.Zhang,P.An,Explicit edge inconsistency evaluationmodel for color-guided depth map enhancement,IEEE Transactions on Circuitsand Systems for Video Technology.28(2)(2018)439-453

[0090] Liu: X. Liu, D. Zhai, R. Chen, X. Ji, D. Zhao, and W. Gao, "Depth superresolution via joint color-guided internal and external regularizations," IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 1636-1645, 2019.

[0091] Jiang: Z. Jiang, Y. Hou, H. Yue, J. Yang, and C. Hou, "Depth super-resolution from rgb-d pairs with transform and spatial domain regularization," IEEE Transactions on Image Processing., vol.27, no.5, pp.2587-2602, 2018.

[0092] DKN: B.Kim, J.Ponce, and BHand, "Deformable kernel networks for jointimage filtering," International Journal of Computer Vision, vol.129, no.2, p.579-600, 2021.

[0093] The test results are shown in Table 1. Compared with other methods, the proposed method always obtains the best results in all three scaling factors in terms of RMSE. Figure 2 A comparison chart of the subjective visual effects of the present invention at an 8-fold sampling rate is shown, and it can be found that the method of the present invention can restore better image details and other information.

[0094] Table 1 Comparison of various super-resolution reconstruction methods (RMSE)

[0095]

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep image super-resolution reconstruction method based on learned non-local prior and enhanced local prior, characterized in that The invention method includes four parts: non-local prior learning, non-local constraint establishment, local constraint establishment and joint model establishment: the non-local prior learning part uses the EM algorithm to learn a Gaussian mixture model for an external high-resolution training image set to extract non-local priors; the non-local constraint establishment part is used to construct non-local prior constraints with the learned Gaussian mixture model; the local constraint establishment part uses a hyper-Laplace distribution to fit the gradient distribution of the image, and analyzes the structural similarity between the color image and the depth image, and establishes an AR model based on structural similarity to constrain the spatial domain of the depth image; the joint model establishment part jointly constructs the above-mentioned prior constraints into a unified optimization framework to obtain an estimate of the high-resolution image.

2. The method according to claim 1, characterized in that The non-local self-similarity model is learned from the external high-resolution depth image using the GMM model.

3. The method according to claim 1, characterized in that: The non-local constraint construction process is: The low-resolution depth image is processed in blocks, and the most similar M image blocks are searched for each image block to form a similar block group; then, the most appropriate Gaussian component is selected for each similar block group by evaluating its posterior probability generated by the kth Gaussian component: Assume that the kth Gaussian component is the most suitable for similar blocks The components of can be obtained by singular value decomposition of its covariance matrix: ∑ k =D k ΛD k T Where D k is an orthogonal matrix; Λ is a diagonal matrix of eigenvalues; then use D k Represents the structure of similar block groups, then the input block in the similar block group X The sparse representation of is: Where γ is the weight parameter.

4. The method according to claim 1, characterized in that The gradient distribution of the image is constrained by a super Laplace distribution, which is expressed as: in and represents the gradient extraction operator along the horizontal and vertical directions; W i is a weighting vector used to assign different weights to each pixel in the horizontal and vertical directions; p (0≤p≤1) is a parameter that controls the sparsity of the super Laplacian operator.

5. The method according to claim 1, characterized in that First, use the structural similarity measure: The structural similarity between depth images and color images is explored, where p i is the depth image patch, z i is a color image block, f(p i ) is the cascade of gradients in the horizontal and vertical directions, <·,·> represents the dot product operation, |·| represents the absolute value operator, and ‖·‖2 is the Euclidean norm of the vector; then the obtained structural similarity is used to construct an AR model based on structural similarity to constrain the spatial information of the depth image. The AR model based on structural similarity is expressed as: in The color weight term is defined as follows: Where T represents the threshold.

6. The method according to claim 1, characterized in that The generated super-resolution reconstruction image restoration model is: Where β, γ, λ1 and λ2 are regularization parameters; A is the weight coefficient {a i,j }The corresponding weight matrix; Λ is the set of sparse representations; D is the subset D k A set of , k = 1, 2,…, K.

Images