A depth map enhancement method based on autoregressive model

Abstract

The present invention discloses a depth map enhancement method based on an autoregressive model. Given an image and parameters input by a user, the method first obtains a corresponding confidence based on the depth image, and then performs correction and upsampling operations on the depth map. For depth map correction, a lower confidence is given to unreliable edge areas, and a higher confidence is given to credible flat areas. For depth map upsampling, a low-resolution depth map is first interpolated to obtain a high-resolution depth map, a lower confidence is given to unreliable interpolated depth values, and a higher confidence is given to credible original depth values; finally, the image to be processed, the obtained confidence, and the corresponding reference image are input into our model and the model is solved to achieve the goal of depth map enhancement. Experimental results show that the method in this paper can overcome artifacts and edge-preserving smoothness while ensuring processing speed, and achieves a better depth map enhancement effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of computational photography, and specifically is a depth map enhancement method based on an autoregressive model. Background Art

[0002] In the field of computer vision, depth information has always played an irreplaceable role. Existing methods for obtaining depth information include laser rangefinders, Kinect cameras, TOF cameras, etc. However, these methods all have various problems. Although the laser rangefinder has high accuracy, it has poor real-time performance and data missing. The Kinect camera has fast speed, but the distance measurement is short, and the depth information obtained has "hole phenomenon". The TOF camera has strong real-time performance, but the depth image obtained has low resolution and a large amount of random noise. In addition to using instruments to obtain depth data, obtaining depth maps through deep learning is also a trend in the current computer vision field, but the depth images obtained in this way often have more differences. In order to process the depth images obtained by the imaging device and the depth images obtained by deep learning, improve the resolution and correct the erroneous depth values ​​so that they can meet the requirements of related applications, depth map enhancement technology is needed.

[0003] Existing depth map enhancement methods can be divided into two categories: traditional methods and solvers. Classic traditional methods include bilateral filtering, guided image filtering, and weighted median filtering. These methods are flexible, fast, and efficient, but they do not perform well when performing depth map correction tasks and cannot achieve good correction results. The fast bilateral solver is a representative method of the solver depth map enhancement method. This method is fast and can be applied to a variety of computer vision tasks, but the depth map obtained by the solver will produce artifacts.

[0004] The present invention proposes a new depth map enhancement method based on an autoregressive model, which can not only better correct the depth image and realize depth map upsampling, but also better overcome the artifact phenomenon caused by the fast bilateral solver and has high computational efficiency. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention proposes a depth map enhancement method based on an autoregressive model, which can achieve upsampling of the depth map and correction of the depth value while ensuring efficiency, and can effectively overcome the artifact phenomenon.

[0006] A depth map enhancement method based on an autoregressive model comprises the following steps:

[0007] Step 1: Set the matrix grid parameter σ l (brightness bandwidth), σ u,v (color bandwidth), σ x,y(spatial bandwidth) and smoothing parameter λ, select the depth image to be enhanced.

[0008] Step 2, solving the objective function of the depth image, enhancing the depth image, and obtaining a processed depth map;

[0009] Furthermore, the objective function used to enhance the depth image is as follows:

[0010]

[0011] Where O is the depth value that needs to be enhanced; C is the confidence of the depth value at that location. The calculation of confidence is divided into two cases: depth map correction and depth map upsampling.

[0012] Furthermore, the simplified bilateral affine matrix solution is:

[0013] According to the definition of bilateral affine matrix:

[0014]

[0015] in represents the spatial position of pixel i, Represent the values ​​of the l, u, and v channels of pixel i respectively. From the definition of the bilateral affine matrix, it can be seen that the closer the distance between pixel i and pixel j, the closer the color, and the corresponding bilateral affine matrix value A i,j The bigger it is.

[0016] However, it is unrealistic to solve the objective function of the depth image to be enhanced based on the definition of the bilateral affine matrix. This is because the depth values ​​of a depth map are usually tens of thousands, and the time spent on solving the objective function directly by the definition of the bilateral affine matrix is ​​unimaginable. Therefore, it is necessary to simplify the bilateral affine matrix, which is also the key to solving the objective function.

[0017] According to previous research, the bilateral affine matrix can be approximated by the following formula:

[0018] A≈S T BS

[0019] S T , B, S represent Splat, Blur, and Slice processes respectively. Now suppose there is a pixel value X that needs to be processed. A pixel point is determined by the position (x, y) and the color value (l, u, v). Therefore, X can be regarded as

[0020] P=(o x ,o y ,o l ,p u ,p v) point set. Based on the previous analysis, it is not advisable to directly solve the bilateral affine matrix of X because it is too time-consuming. The approximate result of the bilateral affine matrix is ​​completed by the following three steps:

[0021] Splat: Splat each pixel value x i Project to the closest p i on the vertex.

[0022] Blur: Performs a smoothing operation on vertex values ​​in bilateral space.

[0023] Slice: From the closest p i The new pixel value is interpolated based on the blurred value of the vertex.

[0024] Used in the objective function Defined by:

[0025]

[0026] Where D m , D n is a diagonal matrix.

[0027] If it is depth map correction, the confidence level is low when it is an edge area, and high when it is a smooth area. Specifically, the confidence level solution formula is:

[0028]

[0029] Among them, I is the depth image to be corrected, G is the color guide image, σ c is a smoothing parameter, which is set to 0.125 in the method of the present invention.

[0030] If the depth map is upsampled, the low-resolution depth map is first interpolated to the corresponding resolution, so the confidence at the interpolation point is naturally low, and the greater the difference from the true depth value, the lower the confidence. To solve the confidence in this case, the present invention mainly adopts the Gaussian function envelope method. Specifically, the local confidence solution formula is:

[0031]

[0032] Where f is the upsampling multiple, and F is a matrix with f rows and f columns. f Copy m copies vertically and n copies horizontally to obtain the confidence level C at the corresponding resolution. a is the length of the depth map to be enhanced, b is the width of the depth map to be enhanced. Specifically, its confidence is

[0033]

[0034] I is the actual depth value of a certain location in the depth map to be processed; λ is an adjustable smoothing parameter; It is a bilateral affine matrix, and an approximate form of the bilateral affine matrix is ​​used in the objective function.

[0035] Furthermore, the method of simplifying the objective function and solving the depth value of the corresponding area according to the simplified bilateral affine matrix is:

[0036] The approximate form of the bilateral affine matrix is Replace the target formula Then for O i Find the partial derivative and set it to 0, and further simplify the formula of the objective function:

[0037]

[0038] Writing the above formula in matrix form yields:

[0039]

[0040] The above formula can be rearranged as follows:

[0041]

[0042] Then let O = S T Y, It turns out that:

[0043]

[0044] Multiplying both sides of the above equation by S, we get the following equation:

[0045]

[0046] Among them, SS T =D m ,Sdiag(C)S T =diag(SC),diag(C)I=c⊙I, so the above formula is equivalent to:

[0047]

[0048] Finally, we only need to perform a slice operation on the solved Y to get the required depth value.

[0049] Furthermore, the above process performs a slice operation on the solved Y, that is, multiplying the solved Y matrix by the transpose of the S matrix, calculated by the following formula:

[0050] O=S T Y

[0051] The beneficial effects of the present invention are:

[0052] Depth images directly obtained by the prior art have problems such as low resolution and incorrect depth values. Therefore, before applying these depth images to computer vision tasks, they need to be repaired through depth map enhancement technology so that they can meet the requirements of related applications. The present invention can achieve a better enhancement effect on the depth map, that is, correct the incorrect depth value, and also better overcome the generation of artifacts in the sampling of the depth map, and has a faster processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the method of the present invention.

[0054] Figure 2 It is a comparison diagram of different upsampling multiples of the present invention, bilateral filtering and fast bilateral solver.

[0055] Figure 2 (a) is the result of 2 times upsampling of the present invention, Figure 2 (b) is the result of 4 times upsampling of the present invention, Figure 2 (c) is the result of 8 times upsampling of the present invention. Figure 2 (d) is the result of bilateral filtering upsampling by a factor of 2. Figure 2 (e) is the result of 4 times upsampling by bilateral filtering. Figure 2 (f) is the result of 8 times upsampling by bilateral filtering. Figure 2 (g) The result of 2x upsampling of the fast bilateral solver, Figure 2 (h) is the result of 4 times upsampling by the fast bilateral solver. Figure 2 (i) is the result of 8-fold upsampling by the fast bilateral solver. The parameters of the present invention are λ=5,σ l =10,σ u,v =5,σ x,y =30, the bilateral filter parameters are r = 16, the global variance is 16, the local variance is 0.8, and the parameters of the fast bilateral solver are λ = 5, σ l =10,σ u,v =5,σ x,y =30.

[0056] Figure 3 This is a comparison chart of the present invention and other methods for depth map correction.

[0057] Figure 3 (a) is the depth map predicted by the residual network; 3(b) is the result after correction by the present invention, with parameters λ=15,σ l =5,σ u,v =10,σ x,y =30, Figure 3(c) is the domain conversion filter DF, σ_s=8σ_r=4.6 corrected result, parameter is r=16, global variance is 16, local variance is 0.8, Figure 3 (d) is the result after weighted median filtering WMF correction, with parameters r = 10, σ = 15.5, w = 1; Figure 3 (e) is the corrected result of the fast bilateral solver with parameters λ=15,,σ l =5,σ u,v =10,σ x,y =30, Figure 3 (f) is the result after the guide image is filtered and corrected, with the parameter r=10. DETAILED DESCRIPTION

[0058] Implementation of the present invention: Given an image and parameters input by a user, the method first obtains the corresponding confidence according to the depth image, and then performs correction and upsampling operations on the depth image. For depth map correction, since the edge part of the depth map predicted by the full convolution residual network has a large error, while the prediction result of the image smooth area is relatively accurate, when the method of the present invention is used to correct the depth map, a lower confidence is given to the unreliable image edge part, and a higher confidence is given to the reliable image smooth area. For depth map upsampling, the low-resolution depth map is first interpolated to obtain a high-resolution depth map, and the depth value obtained by interpolation is unreliable, and a lower confidence value is given accordingly, while the original depth value has a higher confidence, and a higher confidence value is given accordingly; then the simplified bilateral affine matrix in the method is solved; finally, the image to be processed, the obtained confidence and the corresponding reference image are processed by the method proposed by the present invention, and the effect of correction and upsampling is achieved. Experimental results show that the method of the present invention can overcome artifacts and edge-preserving smoothing while ensuring the processing speed, and achieve a better depth map enhancement effect.

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] like Figure 1 As shown, the depth map enhancement method based on the autoregressive model proposed in the present invention has the following specific process:

[0061] Step 1, set the matrix grid parameters, σ l (brightness bandwidth), σ u,v (color bandwidth), σ x,y(spatial bandwidth) and smoothing parameter λ, select the depth image to be enhanced and the corresponding color reference image. The objective function for depth enhancement of the target depth image is established as follows:

[0062]

[0063] Among them, I i is the actual depth value of a certain location in the depth map to be processed; λ is an adjustable smoothing parameter; It is a bilateral affine matrix. An approximate form of the bilateral affine matrix is ​​used in the objective function. o is the depth value to be enhanced, i and j represent the position of the pixel in the image; C is the confidence of the depth value at that location, which is divided into two cases: depth map correction and depth map upsampling.

[0064] Step 2: Solve the simplified bilateral affine matrix and confidence in the objective function

[0065] For the objective function proposed in the above steps, we first need to solve two main problems: one is to solve the simplified bilateral affine matrix, and the other is to solve the confidence.

[0066] The method to solve the simplified bilateral affine matrix is:

[0067] According to the definition of bilateral affine matrix:

[0068]

[0069] in represents the spatial position of pixel i, Represent the values ​​of the l, u, and v channels of pixel i, respectively, x,y , σ l , σ uv Refers to three adjustable parameters. From the definition of the bilateral affine matrix, it can be seen that the closer the distance between pixel i and pixel j, the closer the color, and the corresponding bilateral affine matrix value A i,j The bigger it is.

[0070] However, the objective function proposed based on the definition of the bilateral affine matrix is ​​unrealistic. This is because the depth values ​​of a depth map are usually tens of thousands, and the time spent on solving the objective function directly based on the definition of the bilateral affine matrix is ​​unimaginable. Therefore, it is necessary to simplify the bilateral affine matrix, which is also the key to solving the objective function.

[0071] The bilateral affine matrix is ​​approximated by the following formula:

[0072] A≈S T BS

[0073] S T, B, S represent Splat, Blur, and Slice processes respectively. Assume that there is a pixel value X to be processed. A pixel point is determined by the position (x, y) and the color value (l, u, v). Therefore, X can be regarded as P = (p x ,p y ,p l ,p u ,p v ) point set. According to the previous analysis, it is not advisable to directly solve the bilateral affine matrix of X because it is too time-consuming. Therefore, the approximate result of the bilateral affine matrix is ​​completed by the following three processes:

[0074] Splat: Splat each pixel value x i Project to the closest p i on the vertex.

[0075] Blur: Performs a smoothing operation on vertex values ​​in bilateral space.

[0076] Slice: From the closest p i The new pixel value is interpolated based on the blurred value of the vertex.

[0077] Used in the objective function Defined by:

[0078]

[0079] Where D m , D n is a diagonal matrix.

[0080] If it is depth map correction, the confidence level is low when it is an edge area, and high when it is a smooth area. Specifically, the confidence level solution formula is:

[0081]

[0082] Among them, I is the depth image to be corrected, G is the color guide image, σ c is a smoothing parameter, which is set to 0.125 in the present invention.

[0083] If the depth map is upsampled, the low-resolution depth map is first interpolated to the corresponding resolution, so the confidence at the interpolation point is naturally low, and the greater the difference from the true depth value, the lower the confidence. The present invention mainly adopts the Gaussian function envelope method. Specifically, the local confidence solution formula is:

[0084]

[0085] Wherein, f is the upsampling multiple, and F is a matrix with f rows and f columns.

[0086] Then, C f Copy n copies vertically and n copies horizontally to get the confidence level C at the corresponding resolution. a is the length of the depth map to be enhanced, b is the width of the depth map to be enhanced. Specifically, its confidence is

[0087]

[0088] The method to simplify the objective function according to the simplified bilateral affine matrix is:

[0089] The approximate form of the bilateral affine matrix is Replace the target formula Then for O i By finding the partial derivative and setting it to 0, the formula of the objective function can be further simplified as follows:

[0090]

[0091] Writing the above formula in matrix form yields:

[0092]

[0093] The above formula can be rearranged as follows:

[0094]

[0095] Then let o = S T Y, It turns out that:

[0096]

[0097] Multiplying both sides of the above equation by S, we get the following equation:

[0098]

[0099] Among them, SS T =D m ,Sdiag(C)S T =diag(SC),diag(C)I=C⊙I,

[0100] Therefore, the above formula is equivalent to:

[0101]

[0102] Finally, we only need to perform a slice operation on the solved Y to get the required depth value.

[0103] The above process performs a slice operation on the solved Y, that is, multiplying the solved Y matrix by the transpose of the S matrix, calculated by the following formula:

[0104] O=S T Y

[0105] Step 3: The depth image to be enhanced is used as the target image, and the corresponding color image is used as the reference image to be input into the solver. The solver is built with the above objective function, solves the corresponding confidence for two different situations, and finally constructs an output enhanced depth image.

[0106] In addition, the size of the grid of the bilateral space mentioned in step 2 can be controlled. The larger the grid, the faster the speed. However, too large a grid will affect the effect of depth map enhancement. In the embodiment, on a machine with an Intel i5-4200H CPU @ 2.80GHz and 16G memory, the time for enhancing a depth map of 1390 pixels × 1110 pixels is about 1 second, which can meet the requirements for depth map repair calculation efficiency in real applications.

[0107] like Figure 2 The figure shows a comparison of different upsampling multiples of the present invention and bilateral filtering and fast bilateral solver. Compared with the other two methods, the method of the present invention can better retain image details and sharpen edges at the same upsampling multiple, achieving a better upsampling effect. Figure 3 This is a comparison chart of the present invention and other methods for depth map correction. The present invention is compared with four methods: domain conversion filtering, weighted median filtering, fast bilateral solver, and guided graph filtering. It can be seen from the comparison chart that the correction effect of the present invention is better than other methods, especially in the edge area of ​​the residual network predicted depth map, the correction effect is more obvious. The following table shows the comparison results of the present invention and other methods in terms of running time. The image size processed by the present invention and other methods is 1390 pixels × 1110 pixels. It can be seen that the present invention has high efficiency and can meet the real-time requirements in practical applications.

[0108] method Run time(s) The present invention 1.07 Bilateral filtering 4.07 Fast bilateral solver 1 Domain conversion filtering 0.25 Weighted Median Filter 1.21 Guided graph filtering 11.96

[0109] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A depth map enhancement method based on an autoregressive model, characterized in that: include: S1, set the matrix grid parameters, including the brightness bandwidth σ l , color bandwidth σ u,v , spatial bandwidth σ x,y and smoothing parameter λ, select the depth image to be enhanced, and establish the objective function of the depth image; S2, solving the objective function of the depth image, enhancing the depth image, and obtaining a processed depth map; In S1, the objective function of the depth image to be enhanced is established as follows: Where I is the actual depth value at the preset depth map input to be processed, λ is an adjustable smoothing parameter, is a bilateral affine matrix, O is the depth value to be enhanced, i, j represents the position of the pixel in the image, and C is the confidence of the depth value at that location, which is divided into two cases: depth map correction and depth map upsampling; If it is depth map correction, when it is an edge area, the credibility is low, and when it is a smooth area, the credibility is high. The confidence solution formula is: Among them, I is the depth image to be corrected, G is the color guide image, σ c is the confidence smoothing parameter, which is set to 0.125; The solution for the simplified bilateral affine matrix is ​​as follows: Define a bilateral affine matrix: in represents the spatial position of pixel i, Represent the values ​​of the l, u, and v channels of pixel i, respectively, x,y , σ l , σ u,v Refers to three adjustable parameters. From the definition of the bilateral affine matrix, it can be seen that the closer the distance between pixel i and pixel j, the closer the color, and the corresponding bilateral affine matrix value A i,j the bigger it is; Approximate the bilateral affine matrix: A≈S T BS S T , B, S represent Splat, Blur, and Slice processes respectively, as follows: Splat: Splat each pixel value x i Project to the closest p i On the vertex of Blur: Performs smoothing operations on vertex values ​​in bilateral space; Slice: From the closest p i The blurred value of the vertex is used to interpolate the new pixel value; Therefore, the Defined by: Where D m , D n is a diagonal matrix; The objective function is simplified according to the simplified bilateral affine matrix. The specific method is as follows: The approximate form of the bilateral affine matrix is Replace the objective function formula Then for O i Find the partial derivative and set it to 0, then further simplify the objective function described in step S1 to: Write the above formula in matrix form: The above formula can be rearranged as follows: Then let O = S T Y, It turns out that: Multiplying both sides of the above equation by S, we get the following equation: Among them, SS T =D m ,Sdiag(C)S T =diag(SC),diag(C)I=C⊙I, Therefore, the above formula is equivalent to: Finally, we only need to perform a Slice operation on the solved Y to get the required depth value. The Slice operation is to multiply the solved Y matrix by the transpose of the S matrix. The calculation expression is as follows: O=S T AND.

2. The depth map enhancement method based on the autoregressive model according to claim 1, characterized in that: If the depth map is upsampled, the low-resolution depth map is first interpolated to the corresponding resolution. Then the confidence at the interpolation point is low, and the greater the difference from the true depth value, the lower the confidence. At this time, the Gaussian function envelope method is used to calculate the local confidence: Where, f is the upsampling multiple, and F is a matrix with f rows and f columns; Then, C f Copy m copies vertically and n copies horizontally to obtain the confidence level C at the corresponding resolution. a is the length of the depth map to be enhanced, b is the width of the depth map to be enhanced, and its confidence is

Images