A depth map upsampling method and device based on hierarchical clustering and boundary enhancement

Abstract

The present invention discloses a depth map upsampling method and device based on hierarchical clustering and boundary enhancement, and relates to the technical field of depth map upsampling. The present invention not only pre-processes the low-resolution depth map, divides it into a depth continuous region and a depth discontinuous region, directly interpolates the seed pixels of the depth continuous region, and the target pixels of the depth discontinuous region are what the present invention needs to process, thereby reducing the number of targets and improving computational efficiency. The present invention also uses an initial sketch algorithm to pre-process the high-resolution color image, uses an anisotropic filter to obtain a smooth uniform region and an enhanced image boundary, and uses the boundary information of the sketch to process the depth discontinuous region with similar colors, which is helpful for subsequent fine image segmentation. When upsampling the depth map, a hierarchical classification upsampling method is proposed, which can classify and interpolate the target pixels, effectively preventing depth confusion artifacts in the depth discontinuous region.

Description

Technical Field

[0001] The present invention relates to the technical field of depth map upsampling, and in particular to a depth map upsampling method and device based on hierarchical clustering and boundary enhancement. Background Art

[0002] In recent years, the acquisition of 3D depth information plays an important role in various 3D applications, such as intelligent human-computer interaction, 3D scene reconstruction, and robot autonomous navigation. However, due to the high manufacturing cost, the resolution of depth sensors has developed slowly and is far behind the demand. The laser scanning method is not suitable for real-time applications, and the equipment is very expensive, while the passive stereo vision method can neither eliminate occlusion nor solve the problem of texturelessness in the rendering process. The fusion camera system consisting of a color camera and a depth sensor is widely used in depth acquisition due to its low cost. Typical depth cameras are Time-of-flight (ToF) cameras and structured light cameras (such as Kinect), but the depth maps they acquire have problems such as low resolution, loss of depth information, and noise interference, which limits their application and development. The depth map upsampling method can reconstruct a high-quality depth map from the low-quality depth map obtained by the depth camera by using algorithm processing under the current situation that the hardware cannot make a breakthrough. It has high research value and practical significance. Existing upsampling methods usually use the low-resolution depth map obtained by the depth camera and the high-resolution color image of the same scene to reconstruct a high-resolution depth map. Since deep upsampling can use the registered high-resolution color image as auxiliary guidance information, its sampling rate is relatively high and the upsampling depth is more accurate compared to ordinary image super-resolution reconstruction.

[0003] Common depth map upsampling methods include global optimization-based and filtering-based methods. Global optimization-based methods mainly include Markov random field-based methods and total generalized variational methods. High-resolution depth maps are obtained by optimizing energy functions. These methods solve the super-smoothness problem, but error propagation occurs during the optimization process, resulting in jagged artifacts in the boundary area. In addition, complex iterative calculations make such methods impossible to apply in real time.

[0004] The mainstream filtering-based method is the joint bilateral upsampling (JBU) method proposed by Kopf et al. (DOI: 10.1145 / 1239451.1239547). The low-resolution depth is interpolated into the high-resolution grid by multiplying two Gaussian kernels (spatial kernel and color kernel). The spatial kernel represents the spatial distance in the depth map, and the color kernel represents the color difference in the optical color image. This method is very fast, but it will produce critical artifacts in some boundary areas, resulting in the generation of high-resolution depth maps with blurred edges. A lot of research work is focused on improving this method. For example, YSHo et al. (DOI: 10.1049 / el.2013.3956) proposed a joint bilateral plus local minimum (JBLM) filtering method, which mainly proposed the idea of ​​boundary processing. Specifically, the depth map is divided into two parts: homogeneous areas and depth discontinuous areas. The JBU method is used in the homogeneous area, and the local minimum filter is used in the boundary area, that is, the depth discontinuous area. This method reduces the depth mixing phenomenon in the boundary area without increasing the computational complexity and storage cost. Ming-Yu Liu et al. (DOI: 10.1109 / CVPR.2013.29) proposed the classic joint geodesic upsampling JGU method and gave the definition of the geodesic distance from the target pixel to the seed pixel. In the high-resolution grid map, the seed pixel is a known point in the low-resolution depth map, and the target pixel is a point with unknown depth and to be interpolated. First, the geodesic distance from the current target to all seed pixels is calculated, and the nearest seeds to the current target are found, and the values ​​of these seeds are passed to the current target using Gaussian filtering. As a global algorithm, it can find seeds that are closer to the current target pixel and use these seeds to further provide accurate depth boundaries to complete the interpolation. However, when the image across the depth boundary is very rich in color, serious depth mixing artifacts will occur.

[0005] In 2020, the "Depth Image Upsampling Method and System Based on Depth Edge Points and Color Images" with publication number CN111489383A first marks and corrects unreliable points in the low-resolution depth map according to different situations, and then performs bicubic interpolation to obtain an initialized upsampled depth map. The Sobel operator is then used to extract edge points to obtain a low-resolution depth edge map, which is mapped to a high-resolution grid. At the same time, the pixels of the initialized upsampled depth map are classified based on structural consistency, and the real edge pixels are mapped to the edge-enhanced low-resolution depth map. The depth reliable pixel area is corrected by setting the influence factor to obtain a high-resolution depth map. This method can effectively enhance the detailed edge area and complete the reconstruction of the depth missing part. However, the processing process is cumbersome and complicated, and the preprocessing process is only for the low-resolution depth map, which reduces the efficiency, and the effect decreases when the upsampling factor increases.

[0006] In 2021, Junan Wu (DOI: 10.3788 / IRLA20200081) used bicubic interpolation to initially upsample the low-resolution depth map and used it as the input image for the guided filter. Multi-scale Canny edge detection was used to extract reliable edge regions, and common edge regions were extracted based on edge similarity with the high-resolution grayscale image. For the pixels in the low-resolution depth map that was initialized and upsampled and belonged to the common edge region, they were classified according to their edge position relationship with the high-resolution grayscale image, and the coefficients of the guided filter were re-weighted to improve the quality of the upsampled depth map. This method has high computational efficiency and can provide reliable data for real-time applications, but the reconstruction effect of complex texture information is poor.

[0007] In 2021, Wan Qin (DOI: 10.3778 / j.issn.1002-8331.1912-0463) proposed a hierarchical joint bilateral filtering depth map restoration algorithm based on depth confidence. Based on the depth degradation model, depth confidence measurement was proposed, and the left-right consistency between single pixels was extended to neighborhood calculation. Thresholds were set to divide pixels into high-confidence, low-confidence, and occluded pixels. Based on the depth confidence, pixel weights are dynamically estimated, the corresponding filter size is selected, and hierarchical joint bilateral filtering is used for depth restoration to obtain high-quality depth maps. The hierarchical nature of this method refers to redefining the weights of the joint bilateral filter according to the depth pixel classification, thereby simplifying the algorithm and improving efficiency. However, this method has poor repair effects on depth maps with similar colors. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a depth map upsampling method and device based on hierarchical clustering and boundary enhancement, which effectively improves the efficiency and accuracy of depth map upsampling by preprocessing the low-resolution depth map and the high-resolution color image respectively, and then uses the clustering algorithm to perform hierarchical interpolation on the depth map, thereby preventing depth confusion artifacts in depth discontinuous areas.

[0009] In a first aspect, the present invention provides a depth map upsampling method based on hierarchical clustering and boundary enhancement, comprising: an image preprocessing process and a hierarchical depth map upsampling process;

[0010] The image preprocessing process includes:

[0011] The low-resolution depth map is preprocessed and divided into depth continuous areas and depth discontinuous areas to obtain the preprocessed depth map D I ; Using pixels located in the depth continuous area as seed pixels for interpolation; Using pixels located in the depth discontinuous area as target pixels, and interpolating the target pixels through the seed pixels;

[0012] The high-resolution color image is processed using the initial sketch model to obtain an original sketch and a sketchable image, and the high-resolution color image is smoothed using the sketchable image to achieve fast and effective clustering; the original sketch is added to the smoothed high-resolution color image to enhance the image boundary and obtain a high-resolution grayscale image I R ;

[0013] The layered depth map upsampling process includes:

[0014] For the preprocessed depth map D I Perform hierarchical interpolation, and first use the k-means clustering method to cluster the high-resolution grayscale image I R Divide into k n Each class contains several discontinuous regions, and then the nearest neighbor joint bilateral NJB interpolation method is used to interpolate the target pixel in each region, where k n is the number of categories of the nth layer clustering; when the last layer of interpolation is completed, almost all target pixels are interpolated, and the remaining target pixels that are not interpolated are deleted using the post-smoothing operation.

[0015] Furthermore, during the image preprocessing process, the low-resolution depth map is preprocessed, and bilinear filtering and a threshold function are used to perform region division.

[0016] Furthermore, during the image preprocessing process, the initial sketch model is specifically as follows:

[0017] p(I)=p(I Φ )p(Φ sk )p(Φ nsk )

[0018] Among them, p(Φ sk ) represents the sketchable part, p(Φ nsk ) represents the non-sketchable part, p(I Φ ) represents a Gestalt field model.

[0019] Furthermore, the number of categories k of the k-means clustering n It decreases layer by layer, realizing the segmentation of the image from fine to coarse.

[0020] Furthermore, during the upsampling of the layered depth map, the target pixel is interpolated using the nearest neighbor joint bilateral (NJB) interpolation method for each region, specifically:

[0021] Assume that the current interpolation area contains T target pixels and S seed pixels. For each target pixel, first find several nearest seeds around it and ensure that the selected seeds are among the S seed pixels. Then use the selected seeds to interpolate the current target pixel through joint bilateral filtering:

[0022]

[0023] Among them, L represents the number of selected seeds, g d (t,s l ) and g c (t,s l ) represent the Gaussian kernels for spatial distance and color difference, respectively, as follows:

[0024]

[0025]

[0026] Among them, d(t,s l ) is the current target pixel t and the selected seed pixel s l The geometric distance between σ and c σ is a given scale parameter.

[0027] In a second aspect, the present invention provides a depth map upsampling device based on hierarchical clustering and boundary enhancement, comprising: an image preprocessing module and a hierarchical depth map upsampling module;

[0028] The image preprocessing module is used to preprocess the low-resolution depth map and divide it into a depth continuous area and a depth discontinuous area to obtain a preprocessed depth map D I ; taking pixels located in the depth continuous area as seed pixels for interpolation; taking pixels located in the depth discontinuous area as target pixels, and interpolating the target pixels through the seed pixels; processing the high-resolution color image using the initial sketch model to obtain an original sketch and a sketchable image, and using the sketchable image to smooth the high-resolution color image to achieve fast and effective clustering; adding the original sketch to the smoothed high-resolution color image to enhance the image boundary and obtain a high-resolution grayscale image I R ;

[0029] The layered depth map upsampling module is used to upsample the preprocessed depth map D I Perform hierarchical interpolation, and first use the k-means clustering method to cluster the high-resolution grayscale image I R Divide into k nEach class contains several discontinuous regions, and then the nearest neighbor joint bilateral NJB interpolation method is used to interpolate the target pixel in each region, where k n is the number of categories of the nth layer clustering; when the last layer of interpolation is completed, almost all target pixels are interpolated, and the remaining target pixels that are not interpolated are deleted using the post-smoothing operation.

[0030] Furthermore, in the image preprocessing module, the low-resolution depth map is preprocessed, and bilinear filtering and threshold function are used for region division.

[0031] Furthermore, in the image preprocessing module, the initial sketch model is specifically as follows:

[0032] p(I)=p(I Φ )p(Φ sk )p(Φ nsk )

[0033] Among them, p(Φ sk ) represents the sketchable part, p(Φ nsk ) represents the non-sketchable part, p(I Φ ) represents a Gestalt field model.

[0034] Furthermore, the number of categories k of the k-means clustering n It decreases layer by layer, realizing the segmentation of the image from fine to coarse.

[0035] Furthermore, in the layered depth map upsampling module, the target pixel is interpolated for each region using the nearest neighbor joint bilateral NJB interpolation method, specifically:

[0036] Assume that the current interpolation area contains T target pixels and S seed pixels. For each target pixel, first find several nearest seeds around it and ensure that the selected seeds are among the S seed pixels. Then use the selected seeds to interpolate the current target pixel through joint bilateral filtering:

[0037]

[0038] Among them, L represents the number of selected seeds, g d (t,s l ) and g c (t,s l ) represent the Gaussian kernels for spatial distance and color difference, respectively, as follows:

[0039]

[0040]

[0041] Among them, d(t,s l ) is the current target pixel t and the selected seed pixel s l The geometric distance between σ and c σ is a given scale parameter.

[0042] The embodiments of the present invention have at least the following technical effects or advantages:

[0043] 1. When performing image preprocessing, this patent is different from the existing methods. It not only preprocesses the low-resolution depth map, but also uses bilinear filtering and threshold functions to divide the region into depth continuous regions and depth discontinuous regions (i.e., boundary regions). The seed pixels in the continuous region are directly interpolated, and the target pixels in the discontinuous region are what this patent needs to process, reducing the number of targets and improving computational efficiency. At the same time, this patent also uses the initial sketch algorithm to preprocess the high-resolution color image, dividing it into sketches and sketchable images, and uses anisotropic filters to obtain smooth uniform regions and enhanced image boundaries. For depth discontinuous regions with similar colors, the boundary information of the sketch is used for processing, which is helpful for subsequent fine image segmentation.

[0044] 2. When upsampling the depth map, this patent proposes a hierarchical classification upsampling method that divides the image into multiple disconnected regions and interpolates the target pixels with seed pixels in the same region. The method first uses the k-means clustering algorithm to hierarchically classify the high-resolution color image. The first layer needs to divide the image into relatively more categories to ensure that all depth discontinuities are divided into different regions, and the number of categories decreases layer by layer. Since the high-resolution color image is smoothed in homogeneous areas and enhanced in boundary areas during the preprocessing process, the target pixels can be classified and interpolated, effectively preventing depth confusion artifacts in depth discontinuous areas.

[0045] 3. When interpolating the target pixel, a nearest neighbor joint bilateral interpolation method is proposed. For each target pixel, several nearest neighbor seed pixels are selected for joint bilateral filtering interpolation. The process of selecting seed pixels is to search with the target pixel as the center, and the maximum search radius decreases with the number of layers. For some bad pixels that cannot be filled in the layered interpolation, a method similar to bilateral filtering is also used to process them, effectively improving the efficiency and accuracy of depth map upsampling.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0048] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0049] Figure 2 It is a schematic diagram of the technical framework of the overall solution of the present invention;

[0050] Figure 3 A flowchart of the layered depth map upsampling process of the present invention;

[0051] Figure 4 Examples of searching for seed and target pixels for the present invention;

[0052] Figure 5 A schematic diagram of searching L seed pixels for interpolation of the current target according to the present invention;

[0053] Figure 6 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0054] The embodiment of the present application provides a depth map upsampling method and device based on hierarchical clustering and boundary enhancement, pre-processes the low-resolution depth map and the high-resolution color image respectively, and then uses a clustering algorithm to perform hierarchical interpolation on the depth map, thereby effectively improving the efficiency and accuracy of depth map upsampling and preventing depth confusion artifacts in depth discontinuous areas.

[0055] The technical solution in the embodiment of the present application has the following general idea:

[0056] (1) Preprocess the low-resolution depth map and divide it into depth continuous areas and depth discontinuous areas. Use different methods to upsample them respectively to save processing time and improve efficiency.

[0057] (2) The high-resolution color image is processed using the primal sketch algorithm to obtain a primitive sketch and a sketchable image. On the one hand, the sketchable image is used to smooth the high-resolution color image to achieve fast and effective clustering. On the other hand, the primitive sketch is added to the smoothed high-resolution color image to enhance the image boundary, which helps to maintain the depth boundary and make it overlap with the boundary of the high-resolution color image as much as possible.

[0058] (3) Perform layered interpolation on the depth map. Each layer consists of two steps:

[0059] 1) Use the k-means clustering method to divide the preprocessed high-resolution color image into k categories, each of which contains several discontinuous regions;

[0060] 2) For each region, the target pixel is interpolated using the nearest neighbor joint bilateral (NJB) interpolation method.

[0061] In order to find the most accurate seed for interpolation, the image needs to be segmented from fine to coarse, that is, the number of categories k in k-means clustering is decreasing.

[0062] (4) When the last layer of interpolation is completed, almost all target pixels are interpolated, and some remaining targets are deleted using post-smoothing operations.

[0063] The technical block diagram of the overall solution is as follows: Figure 2 As shown, it is divided into two processes, the image preprocessing process and the layered depth map upsampling process.

[0064] 1. Image preprocessing

[0065] It mainly includes two image preprocessing processes, which act on low-resolution depth maps and high-resolution color images respectively. Preprocessing A separates depth discontinuous areas, i.e., boundary areas, from continuous areas. Pixels located in continuous areas are called seed pixels and are used for interpolation; pixels located in discontinuous areas are called target pixels and are interpolated through seed pixels. Preprocessing B introduces the initial sketch algorithm for image smoothing and boundary enhancement of high-resolution color images.

[0066] 1) Preprocessing of low-resolution depth maps A

[0067] In view of the simple texture and less color information of the depth map, bilinear filtering and threshold function are used for region division, which increases the number of seed pixels and reduces the number of target pixels, greatly saving calculation time. First, the low-resolution depth map is enlarged to the same size as the high-resolution color image using bilinear interpolation, which is called the intermediate upsampled depth map. The sliding window function shown in formula (1) is used to divide it into depth discontinuous areas and continuous areas.

[0068]

[0069] Among them, N d (i) represents a rectangular box centered at pixel i; Th D is the threshold used to estimate N d (i) Is it a continuous region or a discontinuous region? d (i) The maximum depth difference of the pixels is greater than the threshold, then N d (i) It belongs to the depth discontinuous area, and the pixels in the area are target pixels. Otherwise, N d (i) Belongs to the depth continuous region, N dThe pixels in (i) are seed pixels, which are directly interpolated to obtain the preprocessed depth map D I .

[0070] 2) Preprocessing of high-resolution color images B

[0071] Since images in the YCbCr color space are more superior in describing human vision, the initial sketch algorithm is used to perform fine segmentation on the grayscale image of the high-resolution color image (high-resolution grayscale image). The initial sketch model can be simply described as:

[0072] p(I)=p(I Φ )p(Φ sk )p(Φ nsk ) (2)

[0073] Among them, p(Φ sk ) represents the sketchable part, p(Φ nsk ) represents the non-sketchable part, i.e. the original sketch, p(I Φ ) represents a Gestalt field model.

[0074] Through the initial sketch model, the original sketch S is obtained g And can sketch i . Original Sketch S g Provides clear image boundaries and can be used to sketch S i Describes the structure of the image and blurs the interior of the structure. Therefore, they are used to smooth the high-resolution grayscale image through anisotropic filters to obtain enhanced image boundaries and smooth uniform areas:

[0075]

[0076] Where I is a high-resolution grayscale image, N(i) represents the eight neighborhoods of pixel i, and f ij is the anisotropic kernel function:

[0077]

[0078] In the formula, pixel i represents the current point that needs to be smoothed, pixel j represents one of the neighboring pixels of i, and j p represents one of the ω neighboring pixels along a direction of pixel i, σ A , σ B and C t For three given parameters.

[0079] The first case indicates that the current pixel and its neighboring pixels are not on the target boundary, that is, they are in the same homogeneous area. Smoothing the current pixel i, in order to increase the weight of pixel j, can be done by ASet it relatively large.

[0080] The second case only restricts the current pixel i to be located on the target boundary, so it includes the case where the neighboring pixel j is located on the boundary. At the same time, in order to strengthen the boundary, the parameter σ B It should not be set too large.

[0081] The third case means that the current pixel i is not located at the target boundary but its neighboring pixel j is located at the boundary. In this case, C t It should be large enough to reduce the influence of boundary pixel j on current pixel i.

[0082] Thus, the filtered high-resolution image I can be obtained. F , the uniform regions are smoothed and the boundaries are effectively enhanced.

[0083] The above process helps the subsequent k-means based image segmentation. However, in some depth discontinuity areas, the colors are very similar and the pixels will be mistakenly clustered into the same category. This is because k-means clustering is based on pixels, that is, pixels with similar values ​​are divided into the same category. In other words, when areas with similar colors belong to different depth fields, it will lead to wrong segmentation. To solve this problem, sketch S g The boundary information will be added to the filtered high-resolution image I F middle,

[0084]

[0085] Therefore, the area across the depth boundary will be segmented into disconnected areas. Finally, the preprocessed image I is obtained. R Used for subsequent image segmentation.

[0086] 2. Layered Depth Map Upsampling Process

[0087] A layered depth map upsampling method is proposed, which uses seed pixels to preprocess the depth map D I In order to avoid depth mixing artifacts and obtain accurate depth boundaries, the image is first segmented into multiple disconnected regions, and then the target in each region is interpolated using the seeds in the same region.

[0088] 1) High-resolution grayscale image segmentation based on k-means

[0089] First, based on the k-means clustering algorithm, the high-resolution grayscale image I R Split into k n class, where k n From the given set of category numbers. Figure 2The flowchart of this method is given. This method is completed layer by layer, and N represents the total number of layers. In the first layer, the image needs to be segmented into relatively more categories to ensure that all depth discontinuities are segmented into different regions. However, there is also the problem that the target area that does not contain the seed pixel cannot be interpolated. Therefore, the number of categories k n It decreases layer by layer.

[0090] Assume that the training set is {x1,x2,…,x y}, k-means clustering is to divide the observations y into k categories C = {C 1, C 2, …,C k}(k≤y), to minimize the intra-class sum of squares, and its goal is to find C such that

[0091]

[0092] In the formula, μ p Represents C p It can be seen that the k-means algorithm needs to constantly adjust the sample classification and calculate new cluster centers. When the training set contains a large amount of data, the algorithm converges very slowly. Therefore, the input high-resolution image I can be pre-processed. R Random sampling is performed to improve efficiency. Since homogeneous areas of high-resolution color images are smoothed and boundaries are enhanced in advance, k-means segmentation can obtain clearer boundaries.

[0093] 2) Search for the corresponding seeds and targets

[0094] Assume that at layer n, the high-resolution grayscale image I R is divided into k n class, and get the clustering image Ω, such as Figure 4 (a) As shown in the cluster image Ω, each gray level represents a different category Ω i ={Ω|Ω(j)=i}, i=1~k n Then each Ω i Divide into M discontinuous regions. Figure 3 The method shown uses the im The seed pixels (m=1,…,M) are interpolated for the target in the same area. Figure 4 The dark gray pixels in (b) represent Ω i The seeds in , and the white pixels represent the corresponding targets. These methods can effectively search for the corresponding seeds and targets. In order to save interpolation time, some very small areas containing only dozens of pixels will be ignored, and they are easy to fill in the final smoothing process.

[0095] 3) Interpolation method based on nearest neighbor joint bilateral

[0096] Assume that the current interpolation area contains T target pixels and S seed pixels. Figure 3 As shown, for each target pixel First, find several nearest seeds around it and make sure that the selected seed is among the previous S seed pixels. Then use the selected seed to interpolate the current target pixel through joint bilateral filtering:

[0097]

[0098] Where L represents the number of selected seeds. d (t,s l ) and g c (t,s l ) represent the Gaussian kernels for spatial distance and color difference respectively:

[0099]

[0100]

[0101] In the formula, d(t,s l ) is the current target pixel t and the selected seed pixel s l The geometric distance between σ and c σ is a given scale parameter. It is worth mentioning that s l In the same region as t, the key question is how to determine the current region Ω im There are L corresponding seed pixels.

[0102] For example, enlarge Figure 4 The gray box area in (b) is as follows Figure 5 As shown, assuming Figure 5 The black intersection point is the current target pixel In order to get the L nearest seed pixels, Generate a square search window with the center as . Starting from the search radius ω = 1, search for pixels that are both seeds and pixels that are also in the current area Ω as the current target pixel. im If no pixel that meets the conditions is found, the search radius ω is increased by a step of 1. Once such a pixel is found, the search stops. All pixels that meet the above conditions and are located in the current search window ( Figure 4 The pixels on the blue box are used for interpolation. The radius of the search window ω should be decreasing and not greater than the given maximum search radius W n , n represents the nth layer. This means that W n With k nThere is a one-to-one correspondence. n It decreases as the number of layers increases. This is because the seeds are relatively few in the beginning and farther from the target. But as the amount of interpolation increases, the distance to the target gradually decreases.

[0103] Image segmentation based on k-means can be divided into at least two categories, namely k n ≥2, which will cause some bad pixels to be unable to be filled in the hierarchical interpolation process. In order to remove these bad pixels, a method similar to bilateral filtering is proposed to process the bad pixels. Assume that u is one of the bad pixels,

[0104]

[0105] Where N s (u) is a rectangular search window centered on u, and its size is determined empirically. d(u,v) represents the geometric distance, and I is the original high-resolution color image. In this way, an upsampled depth map with clear boundaries can be obtained.

[0106] Embodiment 1

[0107] This embodiment provides a depth map upsampling method based on hierarchical clustering and boundary enhancement, such as Figure 1 As shown, it includes: image preprocessing process and layered depth map upsampling process;

[0108] The image preprocessing process includes:

[0109] The low-resolution depth map is preprocessed and divided into depth continuous areas and depth discontinuous areas to obtain the preprocessed depth map D I ; Using pixels located in the depth continuous area as seed pixels for interpolation; Using pixels located in the depth discontinuous area as target pixels, and interpolating the target pixels through the seed pixels;

[0110] The high-resolution color image is processed using the initial sketch model to obtain an original sketch and a sketchable image, and the high-resolution color image is smoothed using the sketchable image to achieve fast and effective clustering; the original sketch is added to the smoothed high-resolution color image to enhance the image boundary and obtain a high-resolution grayscale image I R ;

[0111] The layered depth map upsampling process includes:

[0112] For the preprocessed depth map D I Perform hierarchical interpolation, and first use the k-means clustering method to cluster the high-resolution grayscale image I R Divide into k nEach class contains several discontinuous regions, and then the nearest neighbor joint bilateral NJB interpolation method is used to interpolate the target pixel in each region, where k n is the number of categories of the nth layer clustering; when the last layer of interpolation is completed, almost all target pixels are interpolated, and the remaining target pixels that are not interpolated are deleted using the post-smoothing operation.

[0113] In a possible implementation, during the image preprocessing process, the low-resolution depth map is preprocessed, and bilinear filtering and a threshold function are used to perform region division.

[0114] Specifically, during the image preprocessing process, the initial sketch model is as follows:

[0115] p(I)=p(I Φ )p(Φ sk )p(Φ nsk )

[0116] Among them, p(Φ sk ) represents the sketchable part, p(Φ nsk ) represents the non-sketchable part, p(I Φ ) represents a Gestalt field model.

[0117] Preferably, the number of categories k of the k-means clustering is n It decreases layer by layer, realizing the segmentation of the image from fine to coarse.

[0118] In a possible implementation, during the upsampling of the layered depth map, the target pixel is interpolated using the nearest neighbor joint bilateral (NJB) interpolation method for each region, specifically:

[0119] Assume that the current interpolation area contains T target pixels and S seed pixels. For each target pixel, first find several nearest seeds around it and ensure that the selected seeds are among the S seed pixels. Then use the selected seeds to interpolate the current target pixel through joint bilateral filtering:

[0120]

[0121] Among them, L represents the number of selected seeds, g d (t,s l ) and g c (t,s l ) represent the Gaussian kernels for spatial distance and color difference, respectively, as follows:

[0122]

[0123]

[0124] Among them, d(t,s l ) is the current target pixel t and the selected seed pixel s l The geometric distance between σ and c σ is a given scale parameter.

[0125] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0126] Embodiment 2

[0127] In this embodiment, a depth map upsampling device based on hierarchical clustering and boundary enhancement is provided. Figure 6 As shown, it includes: an image preprocessing module and a layered depth map upsampling module;

[0128] The image preprocessing module is used to preprocess the low-resolution depth map and divide it into a depth continuous area and a depth discontinuous area to obtain a preprocessed depth map D I ; taking pixels located in the depth continuous area as seed pixels for interpolation; taking pixels located in the depth discontinuous area as target pixels, and interpolating the target pixels through the seed pixels; processing the high-resolution color image using the initial sketch model to obtain an original sketch and a sketchable image, and using the sketchable image to smooth the high-resolution color image to achieve fast and effective clustering; adding the original sketch to the smoothed high-resolution color image to enhance the image boundary and obtain a high-resolution grayscale image I R ;

[0129] The layered depth map upsampling module is used to upsample the preprocessed depth map D I Perform hierarchical interpolation, and first use the k-means clustering method to cluster the high-resolution grayscale image I R Divide into k n Each class contains several discontinuous regions, and then the nearest neighbor joint bilateral NJB interpolation method is used to interpolate the target pixel in each region, where k n is the number of categories of the nth layer clustering; when the last layer of interpolation is completed, almost all target pixels are interpolated, and the remaining target pixels that are not interpolated are deleted using the post-smoothing operation.

[0130] In a possible implementation, in the image preprocessing module, the low-resolution depth map is preprocessed, and bilinear filtering and a threshold function are used for region division.

[0131] Specifically, in the image preprocessing module, the initial sketch model is as follows:

[0132] p(I)=p(I Φ )p(Φ sk )p(Φ nsk )

[0133] Among them, p(Φ sk ) represents the sketchable part, p(Φ nsk ) represents the non-sketchable part, p(I Φ ) represents a Gestalt field model.

[0134] Preferably, the number of categories k of the k-means clustering is n It decreases layer by layer, realizing the segmentation of the image from fine to coarse.

[0135] In a possible implementation, in the layered depth map upsampling module, the target pixel is interpolated for each region using the nearest neighbor joint bilateral (NJB) interpolation method, specifically:

[0136] Assume that the current interpolation area contains T target pixels and S seed pixels. For each target pixel, first find several nearest seeds around it and ensure that the selected seeds are among the S seed pixels. Then use the selected seeds to interpolate the current target pixel through joint bilateral filtering:

[0137]

[0138] Among them, L represents the number of selected seeds, g d (t,s l ) and g c (t,s l ) represent the Gaussian kernels for spatial distance and color difference, respectively, as follows:

[0139]

[0140]

[0141] Among them, d(t,s l ) is the current target pixel t and the selected seed pixel s l The geometric distance between σ and c σ is a given scale parameter.

[0142] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0143] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0144] 1. When performing image preprocessing, this patent is different from the existing methods. It not only preprocesses the low-resolution depth map, but also uses bilinear filtering and threshold functions to divide the region into depth continuous regions and depth discontinuous regions (i.e., boundary regions). The seed pixels in the continuous region are directly interpolated, and the target pixels in the discontinuous region are what this patent needs to process, reducing the number of targets and improving computational efficiency. At the same time, this patent also uses the initial sketch algorithm to preprocess the high-resolution color image, dividing it into sketches and sketchable images, and uses anisotropic filters to obtain smooth uniform regions and enhanced image boundaries. For depth discontinuous regions with similar colors, the boundary information of the sketch is used for processing, which is helpful for subsequent fine image segmentation.

[0145] 2. When upsampling the depth map, this patent proposes a hierarchical classification upsampling method that divides the image into multiple disconnected regions and interpolates the target pixels with seed pixels in the same region. The method first uses the k-means clustering algorithm to hierarchically classify the high-resolution color image. The first layer needs to divide the image into relatively more categories to ensure that all depth discontinuities are divided into different regions, and the number of categories decreases layer by layer. Since the high-resolution color image is smoothed in homogeneous areas and enhanced in boundary areas during the preprocessing process, the target pixels can be classified and interpolated, effectively preventing depth confusion artifacts in depth discontinuous areas.

[0146] 3. When interpolating the target pixel, a nearest neighbor joint bilateral interpolation method is proposed. For each target pixel, several nearest neighbor seed pixels are selected for joint bilateral filtering interpolation. The process of selecting seed pixels is to search with the target pixel as the center, and the maximum search radius decreases with the number of layers. For some bad pixels that cannot be filled in the layered interpolation, a method similar to bilateral filtering is also used to process them, effectively improving the efficiency and accuracy of depth map upsampling.

[0147] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0151] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A depth map upsampling method based on hierarchical clustering and boundary enhancement, characterized in that: include: Image preprocessing and layered depth map upsampling; The image preprocessing process includes: Preprocess the low-resolution depth map and divide it into depth continuous areas and depth discontinuous areas to obtain a preprocessed depth map D I ; Using pixels located in the depth continuous area as seed pixels for interpolation; Using pixels located in the depth discontinuous area as target pixels, and performing target pixel interpolation on the low-resolution depth map through the seed pixels; The high-resolution color image is processed using the initial sketch model to obtain an original sketch and a sketchable image. The high-resolution color image is smoothed using the sketchable image to achieve fast and effective clustering. The original sketch is added to the smoothed high-resolution color image to enhance the image boundary and obtain a high-resolution grayscale image. I R ; The layered depth map upsampling process includes: Preprocess the depth map D I Perform hierarchical interpolation, and each layer first uses the k-means clustering method to cluster the high-resolution grayscale image I R Divide k n Classes, each class contains several discontinuous regions, and then the nearest neighbor joint bilateral NJB interpolation method is used to interpolate the target pixel in each region, where k n For the n The number of categories for layer clustering; when the last layer of interpolation is completed, almost all target pixels are interpolated, and the remaining target pixels that are not interpolated are deleted using the post-smoothing operation.

2. The method according to claim 1, characterized in that: In the image preprocessing process, the low-resolution depth map is preprocessed, and bilinear filtering and threshold function are used to perform region division.

3. The method according to claim 1, characterized in that: In the image preprocessing process, the initial sketch model is as follows: in, Indicates the sketchable part. Indicates the part that cannot be sketched. Represents a Gestalt field model.

4. The method according to claim 1, characterized in that: The number of categories of the k-means clustering k n It decreases layer by layer, realizing the segmentation of the image from fine to coarse.

5. The method according to claim 1, characterized in that: During the upsampling of the layered depth map, the target pixel is interpolated using the nearest neighbor joint bilateral NJB interpolation method for each region, specifically: Assume that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, first find several nearest seeds around it and ensure that the selected seeds are within the S The selected seeds are then used to interpolate the current target pixel through joint bilateral filtering: in, L Indicates the number of selected seeds. and They represent the Gaussian kernels about spatial distance and color difference, respectively, as follows: in, The current target pixel t and the selected seed pixel s l The geometric distance between and is a given scale parameter.

6. A depth map upsampling device based on hierarchical clustering and boundary enhancement, characterized in that: include: Image preprocessing module and layered depth map upsampling module; The image preprocessing module is used to preprocess the low-resolution depth map and divide it into a depth continuous area and a depth discontinuous area to obtain a preprocessed depth map. D I ; Pixels located in the depth continuous area are used as seed pixels for interpolation; Pixels located in the depth discontinuous area are used as target pixels, and the target pixels of the low-resolution depth map are interpolated through the seed pixels; The high-resolution color image is processed using the initial sketch model to obtain an original sketch and a sketchable image, and the high-resolution color image is smoothed using the sketchable image to achieve fast and effective clustering; The original sketch is added to the smoothed high-resolution color image to enhance the image boundary and obtain a high-resolution grayscale image I R ; The layered depth map upsampling module is used to pre-process the depth map D I Perform hierarchical interpolation, and each layer first uses the k-means clustering method to cluster the high-resolution grayscale image I R Divide k n Classes, each class contains several discontinuous regions, and then the nearest neighbor joint bilateral NJB interpolation method is used to interpolate the target pixel in each region, where k n For the n The number of categories for layer clustering; when the last layer of interpolation is completed, almost all target pixels are interpolated, and the remaining target pixels that are not interpolated are deleted using the post-smoothing operation.

7. The device according to claim 6, characterized in that: In the image preprocessing module, the low-resolution depth map is preprocessed, and bilinear filtering and threshold function are used for region division.

8. The device according to claim 6, characterized in that: In the image preprocessing module, the initial sketch model is as follows: in, Indicates the sketchable part. Indicates the part that cannot be sketched. Represents a Gestalt field model.

9. The device according to claim 6, characterized in that: The number of categories of the k-means clustering k n It decreases layer by layer, realizing the segmentation of the image from fine to coarse.

10. The device according to claim 6, characterized in that: In the layered depth map upsampling module, the target pixel is interpolated for each region using the nearest neighbor joint bilateral NJB interpolation method, specifically: Assume that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, first find several nearest seeds around it and ensure that the selected seeds are within the S The selected seeds are then used to interpolate the current target pixel through joint bilateral filtering: in, L Indicates the number of selected seeds. and They represent the Gaussian kernels about spatial distance and color difference, respectively, as follows: in, The current target pixel t and the selected seed pixel s l The geometric distance between and is a given scale parameter.

Images