Depth map hole filling method and system based on adaptive superpixel maximum entropy clustering segmentation

Abstract

The application relates to the technical field of space image processing, and discloses a depth map hole compensation method and system based on adaptive superpixel maximum entropy clustering segmentation, which comprises the following steps: acquiring an RBG image, a depth map and an amplitude map through a ToF camera; after a pretreated RBG image I1 is sequentially subjected to downsampling, gradient processing, gradient reconstruction and a watershed operation, the optimal classification number n of the RBG image is obtained by using an AISNS algorithm c ; finally, the SPMEC algorithm is adopted to perform n c class scene clustering segmentation on the image I1, so that a scene clustering segmentation image is obtained; a sliding window is used to traverse the depth map and the corresponding amplitude map, and according to the scene clustering classification results of corresponding pixel points in the scene clustering segmentation image, each hole pixel point is determined as an isolated point, an edge point or an internal point; according to the categories of the hole pixel points, adaptive compensation strategies are selected, the classification information in the scene clustering segmentation image is combined, the depth values of the pixel points are estimated, and the depth map hole compensation is completed.

Description

Technical Field

[0001] This invention relates to the field of spatial image processing technology, and more specifically, to a method and system for depth map hole compensation based on adaptive superpixel maximum entropy clustering segmentation. Background Technology

[0002] A depth map is a carrier of three-dimensional information about a scene, and its value represents the distance from the detector to the target object. Currently, Time-of-Flight (ToF) ranging imaging technology has attracted much attention due to its advantages such as low computational complexity, high real-time performance, insensitivity to lighting conditions, high measurement accuracy, and long measurement distance. The basic working mechanism of a ToF camera is to emit several light signals towards the target object and measure the round-trip flight time of the emitted signals after reflection. The advantage of this technology is that it can simultaneously acquire the distance information of each pixel in the ToF camera sensor. However, limited by the physical implementation conditions of ToF sensors, the imaging resolution of ToF cameras is generally low, usually not exceeding 100,000 pixels. In addition, the depth maps output by current ToF cameras still have many problems. For example, interference from ambient light sources in the scene, reflection from smooth object surfaces, absorption from object surfaces, and exceeding the maximum measurement distance range can cause missing depth values ​​of pixels in the depth map, resulting in holes. The appearance of holes leads to invalid blind spots of varying sizes on the depth map, which not only seriously affects the quality of the depth image but also affects the subsequent processing of the target object using the depth information. Therefore, research on hole compensation techniques for depth maps has become crucial.

[0003] Existing hole compensation schemes cannot adequately balance compensation performance and time complexity. While the Joint Bilateral Filter (JBF) algorithm can repair holes quickly, it may fail with depth images containing large holes. Compared to bilateral filtering-based compensation methods, clustering algorithms are more advantageous in finding matching pixels. A Fuzzy C-Mean (FCM) clustering-guided depth image hole compensation scheme has also been proposed. This scheme considers the relationship between the clustering segmentation information of the color image and the original depth information, incorporating the clustering segmentation information into the bilateral filtering algorithm to compensate for hole pixels in the depth image. This scheme can effectively recover object edge information in the depth image, but it increases algorithm complexity and requires longer processing time, making it difficult to use in dynamic scenes. Summary of the Invention

[0004] To overcome the shortcomings of poor compensation effect and high compensation time cost in the existing depth map hole compensation algorithms, this invention provides a depth map hole compensation method and system based on adaptive superpixel maximum entropy clustering segmentation.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] The depth map hole compensation method based on adaptive superpixel maximum entropy clustering segmentation includes the following steps:

[0007] Acquire RGB, depth, and amplitude images using a ToF camera;

[0008] Preprocess the RGB image;

[0009] Preprocessed RGB image I After sequentially performing downsampling, gradient processing, gradient reconstruction, and watershed operations, the optimal number of classifications for the RGB image is obtained using the Adaptive Image Segmentation Number Selection (AISNS) algorithm. n c Finally, the SuperPixel Maximum Entropy Clustering (SPMEC) algorithm was used to segment the image. I 1. Proceed n c Scene clustering and segmentation of classes are performed to obtain a scene clustering and segmentation map;

[0010] By using a sliding window to traverse the depth map and its corresponding amplitude map, and combining the scene clustering classification results of the corresponding pixels in the scene clustering segmentation map, each hole pixel is determined as an isolated point, an edge point, or an internal point.

[0011] Based on the category of each hole pixel, an appropriate compensation strategy is selected, and the pixel depth value is estimated by combining the classification information in the scene clustering segmentation map to complete the depth map hole compensation.

[0012] Furthermore, this invention proposes a depth map hole compensation system based on adaptive superpixel maximum entropy clustering segmentation to apply the depth map hole compensation method proposed in this invention. The system includes:

[0013] A ToF camera is used to acquire RGB images, depth images, and amplitude images;

[0014] The preprocessing module is used to preprocess the RGB image;

[0015] The scene clustering and segmentation module is used to process preprocessed RGB images.I After sequentially performing downsampling, gradient processing, gradient reconstruction, and watershed operation, the optimal number of classifications for the RGB image is obtained using an adaptive image segmentation number selection algorithm. n c Finally, the superpixel maximum entropy clustering segmentation algorithm is used to segment the image. I 1. Proceed n c The scene clustering and segmentation are performed, and the scene clustering and segmentation map is output.

[0016] The pixel discrimination module is used to traverse the depth map and its corresponding amplitude map using a sliding window, and combine the scene clustering classification results of the corresponding pixels in the scene clustering segmentation map to output the category result of each hole pixel being determined as an isolated point, an edge point, or an internal point.

[0017] The hole compensation module is used to select an appropriate compensation strategy based on the category of each hole pixel, and estimate the depth value of the pixel by combining the classification information in the scene clustering segmentation map to complete the hole compensation of the depth map.

[0018] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0019] This invention utilizes the AISNS and SPMEC algorithms for scene clustering and segmentation, fully considering the scene clustering and segmentation information, spatial location information, and correlation between pixels of holed pixels. It does not require pre-setting the number of scene segments, thus improving the flexibility and practicality of the solution.

[0020] From the perspective of the spatial correlation of holes, hole pixels are classified into isolated points, edge points, and internal points. This allows for better utilization of the spatial correlation characteristics of holes and further adoption of appropriate compensation strategies based on the morphological characteristics of hole pixels. This approach is more targeted and effective, achieving a significant reduction in system processing latency while maintaining good compensation results. Attached Figure Description

[0021] Figure 1 This is a flowchart of the depth map hole compensation method in Example 1.

[0022] Figure 2 This is a decision-making diagram of an RGB image in Example 1.

[0023] Figure 3 This is a schematic diagram of an isolated point in Example 1.

[0024] Figure 4 This is a schematic diagram of the edge points in Example 1.

[0025] Figure 5 This is a schematic diagram of the internal points of Example 1.

[0026] Figure 6 This is a schematic diagram of the random void generation process in Example 2.

[0027] Figure 7 This is a schematic diagram of a dataset example for Example 2.

[0028] Figure 8 This is a schematic diagram of the randomly generated void diagram in Example 2.

[0029] Figure 9 This is a schematic diagram of RGB image clustering and segmentation in Example 2.

[0030] Figure 10 This is a schematic diagram of the experimental results of the simulation dataset in Example 2.

[0031] Figure 11 This is a schematic diagram showing the algorithm processing time of each scheme in Example 2 executed on the same hardware platform.

[0032] Figure 12 This is a schematic diagram showing the test results of various void compensation schemes in a real-world scenario in Example 2.

[0033] Figure 13 Examples of depth map compensation results for each scheme in Example 2.

[0034] Figure 14 This is an architecture diagram of the depth map hole compensation system in Example 3. Detailed Implementation

[0035] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent; it is understandable that some descriptions in the drawings may be omitted to better illustrate this embodiment.

[0036] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] Example 1

[0038] This embodiment proposes a depth map hole compensation method based on adaptive superpixel maximum entropy clustering segmentation, such as... Figure 1 The diagram shown is a flowchart of the depth map hole compensation method in this embodiment.

[0039] The depth map hole compensation method based on adaptive superpixel maximum entropy clustering segmentation proposed in this embodiment includes the following steps:

[0040] S1. Acquire RGB image, depth image and amplitude image through ToF camera.

[0041] S2. Preprocess the RGB image.

[0042] S3. Process the pre-processed RGB image IAfter sequentially performing downsampling, gradient processing, gradient reconstruction, and watershed operations, the AISNS algorithm is used to obtain the optimal number of classifications for the RGB image. n c Finally, the SPMEC algorithm was used to analyze the graph. I 1. Proceed n c The scene is clustered and segmented to obtain a scene clustering segmentation map.

[0043] S4. Using a sliding window, traverse the depth map and its corresponding amplitude map, and combine the scene clustering classification results of the corresponding pixels in the scene clustering segmentation map to determine each hole pixel as an isolated point, edge point, or internal point.

[0044] S5. Select an appropriate compensation strategy based on the category of each hole pixel, and estimate the pixel depth value by combining the classification information in the scene clustering segmentation map to complete the depth map hole compensation.

[0045] The RGB image contains rich scene information, providing global information about pixels, such as spatial and color information, as well as neighborhood relationships, which can guide hole compensation for objects in the depth map. This embodiment utilizes the clustering and segmentation information provided by the RGB image to assist in handling scenes with large-area holes or complex textures.

[0046] On the other hand, in order to adaptively select the optimal number of scene classifications and quickly achieve scene clustering and segmentation, this embodiment combines the AISNS algorithm and the SPMEC algorithm to obtain a more ideal scene clustering and segmentation map, which provides scene clustering and segmentation information for each pixel. This not only effectively compensates for large-scale holes, but also preserves the edge information of objects well, improving the accuracy of the compensated depth map. At the same time, through the adaptive image segmentation number selection strategy, it is not necessary to pre-set the number of scene segments, which improves the flexibility and practicality of the solution.

[0047] Furthermore, this embodiment classifies pixels by combining the scene clustering classification results of corresponding pixels in the scene clustering segmentation map, fully considering the scene clustering segmentation information, spatial location information and correlation between pixels of the empty pixels, and then selects an appropriate compensation strategy according to the category of the pixel. That is, different compensation strategies are adopted according to the classification characteristics of the empty pixels, which can significantly reduce the system processing latency while maintaining a good compensation effect.

[0048] In an optional embodiment, step S1, when preprocessing the RGB image, includes the following steps:

[0049] S101. In the same scene, calibrate the RGB sensor and the ToF sensor in the ToF camera respectively, and obtain the extrinsic parameter matrices of the RGB sensor and the ToF sensor; the extrinsic parameter matrices include rotation matrices and translation vectors.

[0050] S102. Using the rotation matrix and translation vector in the extrinsic parameter matrix, the coordinate system of the depth map is transformed to the coordinate system of the RGB map, and the RGB map and the depth map are overlapped, registered and aligned.

[0051] In this embodiment, considering that the resolutions of the RGB sensor and the ToF sensor in the ToF camera are different, the scene images captured by the RGB sensor and the ToF sensor cannot be completely overlapped. Therefore, before performing scene clustering and segmentation, it is necessary to overlap the RGB image and the depth image, that is, to perform a registration and alignment preprocessing operation on the RGB image.

[0052] Furthermore, in an optional embodiment, the preprocessed RGB image... I 1. Perform scene clustering and segmentation, including the following steps:

[0053] S301, Let the downsampling coefficient be... , for the diagram I 1. Perform downsampling to obtain the image. I 2; its expression is:

[0054] .

[0055] S302, Regarding the diagram I The R, G, and B channels of image 2 are subjected to Sobel edge feature extraction in the vertical and horizontal directions, respectively, to obtain the gradient map after image gradient processing. I 3.

[0056] S303, Diagram I 3. Morphological gradient reconstruction (MGR) is performed using structuring elements at multiple scales to obtain multiple reconstructed gradient images. For any pixel, the gradient value corresponding to the gradient value in the multiple reconstructed gradient images is selected as the maximum gradient value for that pixel, thus obtaining a better gradient map. I' 3.

[0057] S304, Regarding the diagram I' 3. Perform a Watershed Transform (WT) operation on the graph. I' Divide the image into N regions to obtain the segmented image. And record The collection of each region in is ; Statistical set Average pixel value of each region Its expression is:

[0058]

[0059]

[0060] in, Representing regions Average pixel values ​​of the R, G, and B channels. ; It is a region The number of pixels within; Indicates the region Inner i Channel of 1 pixel t The pixel value.

[0061] S305. Process segmented images using the Density Peak (DP) algorithm. Various regions in China Average pixel value Resulting in the decision map of the RGB image I DG Then, the Density Balance (DB) algorithm is used to process the decision graph. I DG Update the decision graph and then analyze the updated graph. I DG The distance between each point in the image and its neighboring points is calculated, and the classification threshold range is found based on the interval corresponding to the maximum distance between these points. The optimal number of classifications for the current RGB image is then determined based on the classification threshold range. The value of .

[0062] S306, Set the number of coarse superpixel segmentation regions to... ,in The Simple Non-Iterative Clustering (SNIC) algorithm was used to analyze the graph. I 1. Perform coarse superpixel segmentation, and then apply the Maximum Entropy Clustering (MEC) algorithm to the superpixel segmentation image. Fine-tuning of class segmentation yields a scene clustering segmentation map.

[0063] In this embodiment, by analyzing the gradient map I3. Morphological gradient reconstruction is performed to preserve the contour information of objects in the image and remove noise and redundant gradient information to avoid oversegmentation. Considering that the structure element (SE) size of the MGR is uniform, oversegmentation will still occur if the SE size is too small, while undersegmentation is likely if the size is too large, failing to adaptively meet the needs of different images. This embodiment combines SEs of multiple scales to obtain multiple reconstructed gradient images, where each scale of SE generates a gradient image. The maximum gradient value point-by-point of these reconstructed images is calculated, and the maximum gradient value is selected as the gradient value of that pixel. This achieves the goal of removing most useless local minima while preserving important edge information, thereby reconstructing a better gradient image. I' 3.

[0064] Furthermore, for the better gradient image I' 3. Perform the Wt.T. operation, and then use the DP and DB algorithms to determine the optimal number of categories for the current RGB image. The value of allows for adaptive selection of the number of image segments, eliminating the need to pre-set the number of scene segments and improving the flexibility and practicality of the solution.

[0065] As an example, suppose a decision graph of an RGB image is used. I DG like Figure 2 As shown, the horizontal axis represents the number of cluster splits. The vertical axis represents the category factor. .calculate Figure 2 The decision diagram shown [ ]、[ ]、[ ]、…、[ Interval distance , , Then, in the interval distance set Find the maximum interval distance, as shown in the figure. If the threshold for the number of categories falls within the interval [ [Inside. From] Figure 2 As can be seen, there are three points exceeding the threshold, as indicated by the gray blocks in the image. Therefore, the optimal number of classifications for this RGB image is: .

[0066] Furthermore, adaptive selection As The optimal number of classifications is used for image segmentation. This embodiment considers that the SNIC algorithm uses only one iteration in the coarse superpixel segmentation of RGB images, resulting in low time cost. On the other hand, the MEC algorithm can preserve the information of the original image to the greatest extent and has good convergence and robustness. Therefore, this embodiment combines SNIC and MEC to achieve complementary enhancement. Compared with the hole compensation scheme of Fuzzy C-Mean Depth Hole Compensation (FCM-DHC), this embodiment can significantly reduce the overall time complexity of the system processing while obtaining a higher quality scene segmentation map.

[0067] Furthermore, in an optional embodiment, the process of determining each hole pixel as an isolated point, edge point, or internal point in step S4 includes the following steps:

[0068] S401. Using a sliding window of a preset size, traverse the depth map and the corresponding amplitude map, and determine whether a pixel is a hole pixel based on whether its amplitude value is 0. Pixels with an amplitude value of 0 are identified as hole pixels, and pixels with an amplitude value not of 0 are identified as non-hole pixels; the non-hole pixels are the valid pixels.

[0069] S402. For any hole pixel, based on the scene clustering classification result of the corresponding pixel in the scene clustering segmentation map, determine the current hole pixel. Are the scene clustering classification results consistent with the scene clustering classification results of its neighbors?

[0070] If the current hole pixel If the scene clustering classification result of the current hole pixel is different from the scene clustering classification results of its neighbors, then the current hole pixel will be... It was determined to be an isolated point;

[0071] If the current hole pixel If the scene clustering classification result of a given point is partially the same as the scene clustering classification result of its neighboring points, then the current holed pixel will be... It was determined to be an edge point;

[0072] If the current hole pixel If the scene clustering classification result of the current hole pixel is the same as the scene clustering classification result of its neighboring scenes, then the current hole pixel will be classified as a hole pixel. It is determined to be an interior point. Among them, Represents image coordinates.

[0073] This embodiment classifies each hole pixel into categories such as isolated points, edge points, and internal points based on scene clustering and segmentation information of hole pixels in the hole depth map. Then, it adopts an appropriate compensation strategy based on the morphological characteristics of the hole pixels and estimates the depth value of each point by combining location information and the correlation between pixels.

[0074] Furthermore, in an optional embodiment, the compensation strategy selected for isolated points includes estimating the current holed pixel based on a weighting factor based on the spatial location information of effective pixels in the neighborhood. depth value Its expression is:

[0075]

[0076]

[0077] in, Represents empty pixels The neighborhood, For the neighborhood The internal coordinates are Valid pixels, Indicates the weighting factor of spatial location information. For pixels The depth value; is the standard deviation of the spatial location information weighting factor.

[0078] As an example, such as Figure 3 The diagram shown is a schematic of an isolated point in this embodiment. Figure 3 (a) is a classification result image. Figure 3 (b) is the corresponding cavity depth map. From Figure 3 (a) It can be seen that there are no identical scene clustering classification results in the neighborhood of the target pixel, that is, there is no usable information in the neighborhood of the isolated point. Therefore, the depth value of the point can be estimated directly through the spatial location information weight factor.

[0079] Furthermore, in an optional embodiment, the compensation strategy selected for edge points includes estimating the current holed pixel based on the spatial location of effective pixels in the neighborhood and scene clustering classification results. depth value Its expression is:

[0080]

[0081]

[0082] in, Indicates the weighting factor of classification information; For the Dirac function, and These represent the empty pixels. and effective pixels Scene clustering classification results This serves as a control factor for the classification results.

[0083] As an example, such as Figure 4 The diagram shown is a schematic representation of the edge points in this embodiment. Figure 4 (a) is a classification result image. Figure 4 (b) is the corresponding cavity depth map. From Figure 4 (a) It can be seen that there are some identical scene clustering classification results in the neighborhood of the target pixel. The depth estimate of the target pixel will be affected by the same type of points and different types of points. Therefore, it is necessary to combine the spatial location of the pixel and the classification result information to comprehensively judge its depth value.

[0084] Furthermore, in an optional embodiment, for internal points, the selected compensation strategy includes based on the spatial location of effective pixels within the neighborhood and hole pixels. Estimate the current hole pixel based on the similarity to the effective pixels in the neighborhood. depth value Its expression is:

[0085]

[0086]

[0087] in, Represents empty pixels effective pixels in the neighborhood Similarity weighting factors; , , These represent the pixel values ​​of the R, G, and B channels for the holed and valid pixels in the corresponding RGB image, respectively. Similarity weighting factor Standard deviation; These are the weighting factors for the R, G, and B channels, respectively, and satisfy the following conditions: .

[0088] As an example, such as Figure 5 The diagram shown is a schematic representation of the internal points in this embodiment. Figure 5 (a) is a classification result image. Figure 5 (b) is the corresponding cavity depth map. From Figure 5 (a) It can be seen that the neighborhood of the target pixel consists of pixels with the same scene clustering classification results. To further improve the classification of these similar hollow pixels... The confidence level of the depth value, in this embodiment, is determined by the spatial location information weighting factor. Add similarity weight factor on the basis The greater the similarity between a pixel and a valid neighboring pixel, the greater the weight of the depth value of that valid neighboring pixel in determining the depth value of the hole pixel.

[0089] Furthermore, in an optional embodiment, during step S4, when traversing the depth map and its corresponding amplitude map using a sliding window, the discrete coefficients of the current sliding window are calculated. for:

[0090]

[0091] in, This represents the number of valid pixels within the sliding window. The average amplitude value of the sliding window. Indicates the first [number]th ... The amplitude value of each effective pixel.

[0092] The discrete coefficients of the current sliding window With preset threshold Comparison:

[0093] If the discrete coefficients Less than the threshold If so, all pixels within the current sliding window are considered to belong to the same surface, and the estimated depth value of the center pixel of the sliding window is used to compensate for the voids of all other empty pixels within the current sliding window.

[0094] If the discrete coefficients Greater than or equal to the threshold Then, the depth value of each hole pixel in the current sliding window is estimated to complete the hole compensation.

[0095] It should be noted that in this embodiment, a sliding window is used to traverse every pixel of the entire depth map, and the center pixel of the sliding window is the currently traversed pixel. In this embodiment, the sliding window slides directly over a non-hole point when the center point is not a hole point. Only when the center point of the sliding window is a hole pixel does it stop to perform a compensation operation, and the discrepancy coefficient is used to determine whether to compensate the remaining hole points in the window to the depth value of the center point.

[0096] In this embodiment, for cases identified as belonging to the same surface, the remaining empty pixels within the sliding window are automatically compensated to the depth value of the empty pixel. This avoids repeatedly compensating for similar empty pixels, thereby reducing processing time costs.

[0097] Example 2

[0098] This embodiment performs performance testing on the depth map hole compensation method (AS-DHC) based on AISNS and SPMEC segmentation proposed in Embodiment 1.

[0099] In real-world scenarios, depth maps acquired by ToF cameras often contain holes, making them suitable for directly testing the performance of hole compensation algorithms. However, acquiring a large number of real-world depth maps using ToF cameras is time-consuming. Therefore, it's advisable to first test algorithms using publicly available depth map simulation datasets. However, currently available depth map simulation datasets contain limited depth... Figure 1 Generally, it does not have holes. However, hole maps obtained by artificially adding holes usually have unnatural holes and obvious traces of human manipulation, which makes the algorithm test more limited and biased.

[0100] This embodiment obtains a depth map simulation dataset, which includes several ideal depth maps without holes. The ideal depth maps are then pre-holeed to obtain a more realistic random hole map compared to a hole map with manually added holes. This random hole map is then added to the depth map simulation dataset for performance testing of the depth map hole compensation method.

[0101] The preparation of the random void map includes the following steps:

[0102] S601. Generate an empty map with the same dimensions as the ideal depth map.

[0103] S602. Randomly generate several shapes with indefinite areas on the empty map.

[0104] S603. For the shapes generated on the empty map, a random depth information is assigned to each shape using a random height function to generate a template with depth information, and a median filter and a Butterworth filter are used to smooth the template; repeat step S603 up to a preset number of times to obtain the hole template.

[0105] S604. Based on a preset hole threshold, valleys in the hole template that are below the hole threshold are set as holes, and the ideal depth map is multiplied by the hole template to obtain a random hole map.

[0106] like Figure 6The diagram illustrates the random hole generation process in this embodiment. Addressing the inherent uncertainty and high randomness of hole generation in real-world scenarios, this embodiment generates holes of arbitrary shape and area at any possible location within the depth map of the simulation dataset. This statistically better simulates the random generation of holes in real-world scenarios. Compared to traditional hole generation methods that manually add holes to an ideal simulation dataset, this mechanism makes the hole simulation experiment scenario closer to reality, thus better validating the effectiveness of the hole compensation method.

[0107] Furthermore, this embodiment combines experimental results from the "Zaragoza-Deep ToF dataset" simulation dataset with experimental results from real-world scenarios to comprehensively evaluate the performance of the depth map hole compensation method based on adaptive superpixel maximum entropy clustering segmentation.

[0108] First, the performance evaluation metrics for this embodiment are determined to include:

[0109] 1) Mean Square Error (MSE): This measures the difference between a depth map containing random holes (or a depth map after hole compensation) and an ideal depth map. It is defined as follows:

[0110]

[0111] in: , These are the number of rows and columns in the depth map, respectively. For an ideal depth map, This is either a random hole depth map or a compensated depth map.

[0112] 2) Peak Signal-to-Noise Ratio (PSNR): Used to measure the gain performance of a depth map containing random holes (or a depth map after hole compensation), defined as follows:

[0113]

[0114] 3) Algorithm Processing Time: This measures the computational complexity performance of processing depth maps containing random holes (or depth maps after hole compensation). The evaluation method is as follows:

[0115] a) The number of operations performed during the algorithm's processing is the normalized time complexity of the algorithm.

[0116] b) The average execution time of the statistical algorithm was measured on a computer with an Intel® i7-6700 CPU and MATLAB version R2020b, which is used to quantitatively describe the running time of the algorithm.

[0117] This embodiment uses the dataset shown in Table 1 below, such as Figure 7 The image shown is a schematic diagram of an example dataset.

[0118] Table 1 Dataset Characteristics

[0119]

[0120] Using the method for preparing random hole maps proposed in this embodiment, random holes are generated in the ideal depth map of the simulation dataset. The size of the hole template is set to 256. 256, the number of random rectangles is =50. Each run of the randomized simulated hole generation method will produce a new random hole. For example... Figure 8 The diagram shown is a schematic representation of the voids generated in this embodiment. Figure 8 (a) is the hole map generated randomly for the first time. Figure 8 (b) shows the second randomly generated hole map, where the black areas represent holes. As can be seen from the map, the generated random hole phenomenon is relatively natural and close to the actual situation.

[0121] Furthermore, pseudo-grayscale images are generated based on the amplitude data of amplitude maps for each scene in the dataset. Then, an RGB generation algorithm is used to process the pseudo-grayscale images to synthesize RGB images. The synthesized RGB images are then subjected to downsampling, gradient processing, gradient reconstruction, and watershed operations in sequence. Finally, an adaptive image segmentation number selection algorithm is used to obtain the optimal number of classifications for the RGB image. n c Finally, the superpixel maximum entropy clustering segmentation algorithm is used to segment the image. I 1. Proceed n c Clustering and segmentation of scenarios based on class. For example... Figure 9 The image shown is a schematic diagram of RGB image clustering and segmentation. Figure 9 (a) is an RGB image. Figure 9 (b) is a schematic diagram of the corresponding scene segmentation effect.

[0122] Furthermore, let the sliding window size be 5. 5. Hole generation and scene clustering segmentation were performed on the ideal depth maps of all 1704 images in the entire dataset. The Joint Bilateral Filter Depth Hole Compensation (JBF-DHC) and Fuzzy C-Mean Depth Hole Compensation (FCM-DHC) schemes were selected as comparative schemes to the proposed AS-DHC scheme. Random hole depth maps were processed using these schemes, and the mean square error, peak signal-to-noise ratio, and running time of the processing results for each image were statistically analyzed. Finally, the results for all images were averaged to obtain the following result: Figure 10 The diagram shows the experimental results of the simulation dataset.

[0123] from Figure 10 As shown in Figure (a), the mean squared error (MSE) results of the three schemes show varying degrees of reduction compared to the MSE of the random cavity depth map. The FCM-DHC and AS-DHC schemes exhibit the most significant and similar reductions. Specifically, compared to the MSE of the random cavity depth map, the JBF-DHC scheme reduces MSE by 61.2%, the FCM-DHC scheme by 98.36%, and the AS-DHC scheme by 98.83%.

[0124] from Figure 10 As shown in Figure (b), the peak signal-to-noise ratio (PSNR) results of all three schemes show varying degrees of improvement compared to the PSNR of the random cavity depth map. The FCM-DHC and AS-DHC schemes exhibit the most significant and similar improvements. Specifically, compared to the PSNR of the random cavity depth map, the JBF-DHC scheme improves by 61.94%, the FCM-DHC scheme by 182.3%, and the AS-DHC scheme by 184.6%.

[0125] Furthermore, from the perspective of algorithm complexity metrics, Table 2 lists the number of operations performed for each hole compensation scheme, i.e., the normalized time complexity of the algorithm.

[0126] Table 2 Algorithm complexity of different compensation schemes

[0127]

[0128] in, This represents the number of iterations for each element during the clustering process. Let represent the number of cluster splits, and satisfy . ; This represents the number of superpixel regions after processing by the SNIC algorithm, and satisfies... ; This indicates the size of the sliding window.

[0129] As shown in Table 2, the schemes with normalization complexity from low to high are JBF-DHC, AS-DHC, and FCM-DHC, indicating that the AS-DHC scheme proposed in this invention has a moderate normalization time complexity. Furthermore, Figure 11 The algorithm processing time of each scheme executed on the same hardware platform is presented. As can be seen from the figure, the system processing time of the AS-DHC scheme is a compromise between the other two comparative schemes, and is relatively closer to the lower of the two. Based on the simulation data tests above, it is evident that compared with existing typical schemes, the AS-DHC scheme proposed in this invention can achieve superior performance gains with relatively lower computational complexity.

[0130] Furthermore, based on experimental verification in real-world scenarios, the parameters of the ToF camera used in this embodiment are shown in Table 3.

[0131] Table 3 ToF Camera Parameter Table

[0132]

[0133] like Figure 12 The figure shows the experimental results of mean square error, peak signal-to-noise ratio and time complexity of various hole compensation schemes in real-world scenarios.

[0134] Specifically, from Figure 12 As shown in the schematic diagram of the mean square error results in (a), compared with the MSE of the random cavity depth map, the JBF-DHC scheme reduced it by 82.25%, the FCM-DHC scheme reduced it by 99.79%, and the AS-DHC scheme reduced it by 99.81%.

[0135] from Figure 12 As shown in the schematic diagram of the peak signal-to-noise ratio results in (b), compared with the PSNR of the random hole depth map, the JBF-DHC scheme improved by 112.90%, the FCM-DHC scheme improved by 382.95%, and the AS-DHC scheme improved by 387.16%.

[0136] from Figure 12 As shown in the time complexity diagram in (c), the AS-DHC scheme significantly reduces time consumption by approximately 80.02% compared to the FCM-DHC scheme, while its time cost is about twice that of the JBF-DHC scheme.

[0137] Furthermore, Figure 13Examples of depth map compensation results for each scheme are given, among which... Figure 13 (a) is the RGB image of the actual scene; Figure 13 (b) is a random cavity depth map, which is generated by... Figure 13 The ideal depth map shown in (f) was obtained using a random simulation method to generate holes. Figure 13 (c) Figure 13 (d) Figure 13 (e) are the experimental results of the JBF-DHC scheme, FCM-DHC scheme and AS-DHC scheme respectively. Figure 13 The ideal depth map shown in (f) was acquired by adjusting the shooting angle of the ToF camera so that there are no holes.

[0138] Specifically, from Figure 13 (a) and Figure 13 (b) It can be seen that the experimental scene is relatively complex, with smooth objects, dark objects, and interference from ambient light sources, and the hole area of ​​the random hole depth map is relatively large.

[0139] from Figure 13 (c) It can be seen that the JBF-DHC scheme cannot adequately compensate for the cavity depth map under these circumstances, and a small number of cavity areas remain unprocessed. From... Figure 13 (d) and Figure 13 (e) It can be seen that the FCM-DHC scheme and the AS-DHC scheme can effectively deal with the problem of large hole area in the depth map. However, based on the above performance index analysis, the AS-DHC scheme has more advantages and practicality due to its lower time complexity.

[0140] Therefore, the verification results of each scheme in the real-world scenario are largely consistent with the verification results using the simulation dataset, and they can corroborate each other. In both types of tests, the AS-DHC scheme achieved better performance gains with less processing time, thus exhibiting the best overall performance.

[0141] Example 3

[0142] This embodiment proposes a depth map hole compensation system based on adaptive superpixel maximum entropy clustering segmentation, applying the depth map hole compensation method proposed in Embodiment 1. For example... Figure 14 The diagram shown is an architecture diagram of the depth map hole compensation system in this embodiment.

[0143] The depth map hole compensation system based on adaptive superpixel maximum entropy clustering segmentation proposed in this embodiment includes:

[0144] A ToF camera is used to acquire RGB images, depth images, and amplitude images.

[0145] The preprocessing module is used to preprocess RGB images.

[0146] The scene clustering and segmentation module is used to process preprocessed RGB images. I After sequentially performing downsampling, gradient processing, gradient reconstruction, and watershed operation, the optimal number of classifications for the RGB image is obtained using an adaptive image segmentation number selection algorithm. n c Finally, the superpixel maximum entropy clustering segmentation algorithm is used to segment the image. I 1. Proceed n c The scene clustering and segmentation are performed, and the scene clustering and segmentation map is output.

[0147] The pixel discrimination module is used to traverse the depth map and its corresponding amplitude map using a sliding window, and combine the scene clustering classification results of the corresponding pixels in the scene clustering segmentation map to output the category result of each hole pixel being determined as an isolated point, an edge point, or an internal point.

[0148] The hole compensation module is used to select an appropriate compensation strategy based on the category of each hole pixel, and estimate the depth value of the pixel by combining the classification information in the scene clustering segmentation map to complete the hole compensation of the depth map.

[0149] In an optional embodiment, when the preprocessing module preprocesses the RGB image, it calibrates the RGB sensor and the ToF sensor in the ToF camera respectively, obtaining the extrinsic parameter matrices of the RGB sensor and the ToF sensor; the extrinsic parameter matrices include rotation matrices and translation vectors. Then, the coordinate system of the depth map is transformed to the coordinate system of the RGB image using the rotation matrix and translation vectors in the extrinsic parameter matrices, and the RGB image and the depth map are aligned and registered to output the preprocessed RGB image. I 1.

[0150] In an optional embodiment, the scene clustering and segmentation module performs preprocessed RGB image processing. I 1. In the process of scene clustering and segmentation, the first step is to analyze the graph. I 1. Perform downsampling to obtain the image. I 2. Then, regarding the diagram... I The R, G, and B channels of image 2 are subjected to Sobel edge feature extraction in the vertical and horizontal directions, respectively, to obtain the gradient map after image gradient processing. I 3. Regarding the diagram I 3. Morphological gradient reconstruction is performed using structuring elements at multiple scales to obtain multiple reconstructed gradient images. For any pixel, the maximum gradient value among these multiple reconstructed gradient images is selected as the gradient value for that pixel, thus reconstructing a better gradient map. I' 3. Regarding the diagram I'3. Perform a watershed operation on the graph. I' Divide the image into N regions to obtain the segmented image. And statistically analyze the segmented images. The average pixel value of each region. Image segmentation is processed using a density peaking algorithm. The decision map for the RGB image is obtained by averaging the pixel values ​​of each region. I DG Then, the density balance algorithm is used to process the decision graph. I DG Update the decision graph and then analyze the updated graph. I DG The distance between each point in the image and its neighboring points is calculated, and the classification threshold range is found based on the interval corresponding to the maximum distance between these points. The optimal number of classifications for the current RGB image is then determined based on the classification threshold range. The value of is then used. Finally, a simple non-iterative clustering algorithm is used to analyze the graph. I 1. Perform coarse superpixel segmentation, and then use the maximum entropy clustering algorithm to further segment the superpixel image. Fine-grained optimization of class segmentation outputs a scene clustering segmentation map.

[0151] In one optional embodiment, during the process of determining whether a pixel is an isolated point, an edge point, or an internal point, the pixel discrimination module traverses the depth map and the corresponding amplitude map using a sliding window of a preset size, and determines whether the corresponding pixel is a hole pixel based on whether its amplitude value is 0. Pixels with an amplitude value of 0 are determined to be hole pixels, while pixels with a non-zero amplitude value are determined to be non-hole pixels. The non-hole pixels are the valid pixels.

[0152] Furthermore, for any hole pixel, based on the scene clustering classification result of the corresponding pixel in the scene clustering segmentation map, it is determined whether the scene clustering classification result of the current hole pixel is consistent with the scene clustering classification results of its neighbors:

[0153] If the scene clustering classification result of the current hole pixel is different from the scene clustering classification results of its neighbors, then the current hole pixel is determined to be an isolated point.

[0154] If the scene clustering classification result of the current hole pixel is partially the same as the scene clustering classification result of its neighbors, then the current hole pixel is determined to be an edge point.

[0155] If the scene clustering classification result of the current hole pixel is the same as the scene clustering classification result of its neighbors, then the current hole pixel is determined to be an internal point.

[0156] The pixel discrimination module outputs the category determination result of each hole pixel to the hole compensation module, so that the hole compensation module can select an appropriate compensation strategy according to the category of each hole pixel.

[0157] In an optional embodiment, the hole compensation module selects a compensation strategy for isolated points that includes estimating the depth value of the current hole pixel based on a weight factor of the spatial location information of effective pixels in the neighborhood.

[0158] The compensation strategy selected for edge points includes estimating the depth value of the current hole pixel based on the spatial location of effective pixels in the neighborhood and the scene clustering classification results.

[0159] For internal points, the selected compensation strategy includes estimating the depth value of the current hole pixel based on the spatial location of the effective pixels in the neighborhood and the similarity between the hole pixel and the effective pixels in the neighborhood.

[0160] The terminology used in the accompanying drawings is for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0161] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A depth map hole compensation method based on adaptive superpixel maximum entropy clustering segmentation, characterized in that, Includes the following steps: Acquire RGB, depth, and amplitude images using a ToF camera; Preprocess the RGB image; Preprocessed RGB image I After sequentially performing downsampling, gradient processing, gradient reconstruction, and watershed operation, the optimal number of classifications for the RGB image is obtained using an adaptive image segmentation number selection algorithm. n c Finally, the superpixel maximum entropy clustering segmentation algorithm is used to segment the image. I 1. Proceed n c Scene clustering and segmentation of classes are performed to obtain a scene clustering and segmentation map; By using a sliding window to traverse the depth map and its corresponding amplitude map, and combining the scene clustering classification results of the corresponding pixels in the scene clustering segmentation map, each hole pixel is determined as an isolated point, an edge point, or an internal point. Based on the category of each hole pixel, an appropriate compensation strategy is selected, and the pixel depth value is estimated by combining the classification information in the scene clustering segmentation map to complete the depth map hole compensation. Preprocessed RGB image I 1. Perform scene clustering and segmentation, including: Let the downsampling coefficient be , for the diagram I 1. Perform downsampling to obtain the image. I 2; its expression is: ； For the picture I The R, G, and B channels of image 2 are subjected to Sobel edge feature extraction in the vertical and horizontal directions, respectively, to obtain the gradient map after image gradient processing. I 3; For the picture I 3. Morphological gradient reconstruction is performed using structuring elements at multiple scales to obtain multiple reconstructed gradient images. For any pixel, the gradient value corresponding to the gradient value in the multiple reconstructed gradient images is selected as the maximum gradient value for that pixel, thus obtaining a better gradient map. I' 3; For the picture I' 3. Perform a watershed operation on the graph. I' Divide the image into N regions to obtain the segmented image. And record The collection of each region in is ; Statistical set Average pixel value of each region Its expression is: in, Representing regions Average pixel values ​​of the R, G, and B channels. ; It is a region The number of pixels within; It is a region Inner i Channel of 1 pixel t pixel values, ; Image segmentation is processed using the density peak algorithm. Various regions in China Average pixel value Resulting in the decision map of the RGB image I DG Then, the density balance algorithm is used to process the decision graph. I DG Update the decision graph and then analyze the updated graph. I DG The distance between each point in the image and its neighboring points is calculated, and the classification threshold range is found based on the interval corresponding to the maximum distance between these points. The optimal number of classifications for the current RGB image is then determined based on the classification threshold range. The possible values ​​of ; Set the number of coarse superpixel segmentation regions to ,in A simple non-iterative clustering algorithm was used to cluster the graph. I 1. Perform coarse superpixel segmentation, and then use the maximum entropy clustering algorithm to further segment the superpixel image. Fine-tuning of class segmentation yields a scene clustering segmentation map.

2. The depth map hole compensation method according to claim 1, characterized in that, Preprocessing of RGB images includes: The RGB sensor and the ToF sensor in the ToF camera are calibrated respectively to obtain the extrinsic parameter matrices of the RGB sensor and the ToF sensor; the extrinsic parameter matrices include rotation matrices and translation vectors. The coordinate system of the depth map is transformed to the coordinate system of the RGB map using the rotation matrix and translation vector in the extrinsic parameter matrix, and the RGB map and the depth map are aligned and registered.

3. The depth map hole compensation method according to claim 1, characterized in that, Each hole pixel is classified as an isolated point, an edge point, or an internal point, including: The depth map and the corresponding amplitude map are traversed using a sliding window of a preset size. The pixel is determined to be a hole pixel based on whether its amplitude value is 0. Pixels with an amplitude value of 0 are identified as hole pixels, while pixels with an amplitude value other than 0 are identified as non-hole pixels. The non-hole pixels are the valid pixels. For any hole pixel, determine the current hole pixel based on the scene clustering classification result of the corresponding pixel in the scene clustering segmentation map. Are the scene clustering classification results consistent with the scene clustering classification results of its neighbors? If the current hole pixel If the scene clustering classification result of the current hole pixel is different from the scene clustering classification results of its neighbors, then the current hole pixel will be... It was determined to be an isolated point; If the current hole pixel If the scene clustering classification result of a given point is partially the same as the scene clustering classification result of its neighboring points, then the current holed pixel will be... It was determined to be an edge point; If the current hole pixel If the scene clustering classification result of the current hole pixel is the same as the scene clustering classification result of its neighboring scenes, then the current hole pixel will be classified as a hole pixel. It is determined to be an interior point; in, Represents image coordinates.

4. The depth map hole compensation method according to claim 3, characterized in that, For isolated points, the selected compensation strategy includes estimating the current holed pixel based on a weight factor that uses spatial location information of effective pixels in the neighborhood. depth value ; Its expression is: in, Represents empty pixels The neighborhood, For the neighborhood The internal coordinates are Valid pixels, Indicates the weighting factor of spatial location information. Valid pixels The depth value; is the standard deviation of the spatial location information weighting factor.

5. The depth map hole compensation method according to claim 3, characterized in that, For edge points, the selected compensation strategy includes estimating the current holed pixel based on the spatial location of effective pixels in the neighborhood and the scene clustering classification results. depth value ; Its expression is: in, Represents empty pixels The neighborhood, For the neighborhood The internal coordinates are Valid pixels, Indicates the weighting factor of spatial location information. Indicates the weighting factor of classification information; For the Dirac function, and These represent the empty pixels. and effective pixels Scene clustering classification results This serves as a control factor for the classification results.

6. The depth map hole compensation method according to claim 3, characterized in that, For internal points, the selected compensation strategy includes based on the spatial location of effective pixels in the neighborhood and hole pixels. Estimate the current hole pixel based on the similarity to the effective pixels in the neighborhood. depth value ; Its expression is: in, Represents empty pixels The neighborhood, For the neighborhood The internal coordinates are Valid pixels, Indicates the weighting factor of spatial location information. Represents empty pixels effective pixels in the neighborhood Similarity weighting factors; , , These represent the pixel values ​​of the R, G, and B channels for the holed and valid pixels in the corresponding RGB image, respectively. Similarity weighting factor Standard deviation; These are the weighting factors for the R, G, and B channels, respectively, and satisfy the following conditions: .

7. The depth map hole compensation method according to any one of claims 1 to 6, characterized in that, It also includes the following steps: During the process of traversing the depth map and its corresponding amplitude map using a sliding window, the discrete coefficients of the current sliding window are calculated. for: in, This represents the number of valid pixels within the sliding window. The average amplitude value of the sliding window. Indicates the first [number]th ... The amplitude value of each valid pixel; The discrete coefficients of the current sliding window With preset threshold Comparison: If the discrete coefficients Less than the threshold If so, all pixels within the current sliding window are considered to belong to the same surface, and the estimated depth value of the center pixel of the current sliding window is used to compensate for the voids of all other empty pixels within the current sliding window. If the discrete coefficients Greater than or equal to the threshold Then, the depth value of each hole pixel in the current sliding window is estimated to complete the hole compensation.

8. The depth map hole compensation method according to any one of claims 1 to 6, characterized in that, It also includes the following steps: After completing the depth map hole compensation, performance testing is performed using a depth map simulation dataset. The depth map simulation dataset includes several ideal depth maps without holes, and several random hole maps obtained by pre-holeing the ideal depth maps. The preparation of the random hole maps includes the following steps: Generate an empty map with the same dimensions as the ideal depth map; Several shapes with indefinite areas are randomly generated on the empty map; For the shapes generated on the empty map, a random depth information is assigned to each shape using a random height function to generate a template with depth information. The template is then smoothed using a median filter and a Butterworth filter. This step is repeated up to a preset number of times to obtain the hole template. Based on a preset hole threshold, troughs in the hole template that are below the hole threshold are set as holes, and the ideal depth map is multiplied by the hole template to obtain a random hole map.

9. A depth map hole compensation system based on adaptive superpixel maximum entropy clustering segmentation, applying the depth map hole compensation method according to any one of claims 1 to 8, characterized in that, include: A ToF camera is used to acquire RGB images, depth images, and amplitude images; The preprocessing module is used to preprocess the RGB image; The scene clustering and segmentation module is used to process preprocessed RGB images. I After sequentially performing downsampling, gradient processing, gradient reconstruction, and watershed operation, the optimal number of classifications for the RGB image is obtained using an adaptive image segmentation number selection algorithm. n c Finally, the superpixel maximum entropy clustering segmentation algorithm is used to segment the image. I 1. Proceed n c The scene clustering and segmentation are performed, and the scene clustering and segmentation map is output. The pixel discrimination module is used to traverse the depth map and its corresponding amplitude map using a sliding window, and combine the scene clustering classification results of the corresponding pixels in the scene clustering segmentation map to output the category result of each hole pixel being determined as an isolated point, an edge point, or an internal point. The hole compensation module is used to select an appropriate compensation strategy based on the category of each hole pixel, and estimate the depth value of the pixel by combining the classification information in the scene clustering segmentation map to complete the hole compensation of the depth map.

Images