A GPU-based depth map filtering method, system, device and storage medium

Abstract

The application provides a GPU-based depth map filtering method, comprising: dividing a depth map to be filtered into a plurality of equal thread blocks; determining a label value of each pixel in any thread block to obtain an initial label map equal to the depth map; for any thread block, starting from the first pixel in the upper left corner of the thread block, comparing the depth similarity of the pixel with adjacent pixels; if there is a depth value difference less than a preset threshold, it is determined that the similarity is high, and the label value corresponding to the current pixel is updated to the label value corresponding to an adjacent pixel with high similarity; otherwise, it is determined that the similarity is low, and the current pixel is not processed; after traversal, an updated label map is obtained; repeating the step until a preset iteration number is reached, and outputting an optimized label map; determining a connected region in the depth map according to the optimized label map, and performing bilateral filtering on the depth map based on the connected region; wherein the similarity comparison step is performed in parallel on all thread blocks by using a GPU.

Description

Technical Field

[0001] This invention relates to the field of mapping image processing technology, and more specifically, to a GPU-based depth map filtering method, system, device, and storage medium. Background Technology

[0002] After obtaining the depth map, the multi-view stereo matching algorithm often encounters irregular noise regions due to image occlusion and insufficient visible images. This results in poor visual quality and affects accuracy. Therefore, post-processing of the depth map, such as filtering, is necessary, including median filtering and connected component detection. Median filtering effectively removes isolated noise points in the disparity map, while connected component detection detects small blobs caused by noise in the disparity map, facilitating the removal of small noisy connected regions or interpolation to fill holes.

[0003] Traditional connected component detection methods are CPU-based and use image corner diffusion to globally search for large and small connected components. However, this method is time-consuming and inefficient when the image resolution is high or the scene range is large. Moreover, the depth map filtering method based on image corner diffusion relies heavily on the information of neighboring pixels, and the calculation between adjacent pixels is not independent. From the perspective of diffusion principle, it is not suitable for parallel design. Summary of the Invention

[0004] To overcome the shortcomings of existing depth map filtering methods, which are time-consuming and inefficient when image resolution is high or scene range is large, this invention provides a GPU-based depth map filtering method and system.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] A GPU-based depth map filtering method includes the following steps:

[0007] S1. Divide the depth map to be filtered into several thread blocks of equal size;

[0008] S2. For any thread block, determine the label value of each pixel in the thread block to obtain an initial label map of the same size as the depth map;

[0009] S3. For any thread block, starting from the first pixel in the top left corner of the thread block, compare its depth similarity with that of its neighboring pixels: if there is a depth value difference less than a preset threshold, it is determined to be highly similar, and the label value corresponding to the current pixel is updated to the label value corresponding to a neighboring pixel with high similarity; otherwise, it is determined to be low similarity, and no processing is done on the current pixel; after completing the traversal, the updated label map is obtained.

[0010] S4. Repeat step S3 until the preset number of iterations is reached, and output the optimized marker map;

[0011] S5. Determine the connected regions in the depth map based on the optimized marker map, and perform bilateral filtering on the depth map based on the connected regions;

[0012] Specifically, the GPU is used to execute steps S2 to S4 in parallel on all thread blocks.

[0013] As a preferred embodiment, step S2, which involves determining the marker value of each pixel in the thread block, includes:

[0014] S2.1 For any thread block, determine the initial index value label based on the number of rows r and columns c of the pixels; its expression is:

[0015] Label = W × r + c

[0016] In the formula, W is the width value of the depth map;

[0017] S2.2. Use the initial index value label as the label value of the current pixel; after traversing all pixels in the thread block, generate an initial label map that matches the depth map.

[0018] As a preferred embodiment, in step S3, the neighboring pixels include the pixels located above the current pixel and the pixels located to the left of the current pixel.

[0019] As a preferred embodiment, step S5, the step of determining the connected regions in the depth map based on the optimized marker map, includes: taking all pixels with the same marker value as connected regions based on the optimized marker map, and performing bilateral filtering on the depth map based on the connected regions.

[0020] As a preferred embodiment, step S5, which identifies all pixels with the same marker value as connected regions based on the optimized marker map, further includes the following steps: for any connected region, searching the optimized marker map to determine if there is an adjacent connected region in the upper-left direction; if so, updating the marker values ​​of all pixels in the current connected region with the corresponding marker values ​​of the adjacent connected region in the upper-left direction, thus confirming the update of the connected region; repeating the above steps until a preset update threshold is reached to obtain the updated connected region.

[0021] As a preferred embodiment, step S5 further includes the following steps: counting the number of pixels with the same label value in each connected region (Count) and comparing it with a preset area threshold; if Count is less than or equal to the preset area threshold, the connected region is determined to be a fragmented connected region, and the fragmented connected region is interpolated and filled using the nearest depth value; otherwise, the connected region is retained.

[0022] Furthermore, this invention also proposes a GPU-based depth map filtering system, applied to the GPU-based depth map filtering method proposed in this invention. The depth map filtering system includes:

[0023] The preprocessing module is used to divide the depth map to be filtered into several equally sized thread blocks;

[0024] The GPU module includes several labeling units and several similarity comparison units, and the labeling units and similarity comparison units work in parallel;

[0025] The marking unit is used to determine the marking value of each pixel in the current thread block;

[0026] The similarity comparison unit is used to traverse the pixels in the current thread block, starting from the first pixel in the upper left corner of the thread block, and compare its depth similarity with that of its neighboring pixels: if there is a depth value difference less than a preset threshold, it is determined to be highly similar, and the label value corresponding to the current pixel is updated to the label value corresponding to a neighboring pixel with high similarity; otherwise, it is determined to be low similarity, and no processing is performed on the current pixel.

[0027] The similarity comparison unit iterates until a preset number of iterations is reached, and outputs an optimized label map of the same size as the depth map;

[0028] The filtering module is used to determine the connected regions in the depth map based on the optimized marker map, and to perform bilateral filtering on the depth map based on the connected regions, and output the filtered depth map.

[0029] As a preferred embodiment, the filtering module further includes a fragmented region filtering unit, which is used to count the number of pixels with the same label value in each connected region (Count) and compare it with a preset area threshold: if Count is less than or equal to the preset area threshold, the connected region is determined to be a fragmented connected region, and the fragmented connected region is interpolated and filled using the nearest depth value; otherwise, the connected region is retained.

[0030] Furthermore, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the GPU-based depth map filtering method proposed in the present invention.

[0031] Furthermore, the present invention also proposes a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the GPU-based depth map filtering method proposed in this invention.

[0032] Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows: The present invention divides the depth map into several equally sized thread blocks, and further uses the GPU to process all thread blocks in parallel. For depth maps with high image resolution or large scene range, the depth map filtering efficiency can be effectively improved. The present invention uses a label value to mark pixels with high similarity, which facilitates the search of connected regions and reduces the computational resource requirements. Attached Figure Description

[0033] Figure 1 This is a flowchart of the GPU-based depth map filtering method in Example 1.

[0034] Figure 2 This is a schematic diagram of the thread block partitioning process in Example 1.

[0035] Figure 3 This is a reference diagram for the marking diagram of Example 1.

[0036] Figure 4 This is a reference diagram of the optimized labeling diagram for Example 1.

[0037] Figure 5 This is a schematic diagram of the filtering effect in Example 1.

[0038] Figure 6 This is an architecture diagram of the GPU-based depth map filtering system in Example 2. Detailed Implementation

[0039] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0040] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0041] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] Example 1

[0043] This embodiment proposes a GPU-based depth map filtering method, such as... Figure 1 The diagram shown is a flowchart of the GPU-based depth map filtering method in this embodiment.

[0044] The GPU-based depth map filtering method proposed in this embodiment includes the following steps:

[0045] S1. Divide the depth map to be filtered into several thread blocks of equal size.

[0046] S2. For any thread block, determine the label value of each pixel in the thread block to obtain an initial label map of the same size as the depth map.

[0047] S3. For any thread block, starting from the first pixel in the top left corner of the thread block, compare its depth similarity with that of its neighboring pixels: if there is a depth value difference less than a preset threshold, it is determined to be highly similar, and the label value corresponding to the current pixel is updated to the label value corresponding to a neighboring pixel with high similarity; otherwise, it is determined to be low similarity, and no processing is done on the current pixel; after completing the traversal, the updated label map is obtained.

[0048] S4. Repeat step S3 until the preset number of iterations is reached, and output the optimized marker map.

[0049] S5. Determine the connected regions in the depth map based on the optimized marker map, and perform bilateral filtering on the depth map based on the connected regions.

[0050] In this embodiment, the GPU is used to execute steps S2 to S4 in parallel on all thread blocks to achieve more efficient connected component search.

[0051] In this embodiment, the depth map is first divided into several equally sized thread blocks for pixel depth similarity comparison, thereby completing the connected component search. This embodiment uses a tagging method for recording, marking pixels with high similarity with the same label to facilitate the determination of connected components based on the tag values. Furthermore, this application uses a GPU to process all thread blocks in parallel, which can effectively improve the depth map filtering efficiency for depth maps with high image resolution or large scene areas.

[0052] In one optional embodiment, the depth map is divided into multiple 32*32 pixel thread blocks according to the size of the depth map to be filtered. Figure 2 The diagram shown illustrates the thread block partitioning process.

[0053] In an alternative embodiment, the step of determining the marker value for each pixel in the thread block includes:

[0054] S2.1 For any thread block, determine the initial index value label based on the number of rows r and columns c of the pixels; its expression is:

[0055] Label = W × r + c

[0056] In the formula, W is the width value of the depth map.

[0057] S2.2. Use the initial index value label as the label value of the current pixel; after traversing all pixels in the thread block, generate an initial label map that matches the depth map.

[0058] like Figure 3 The diagram shown is a marker diagram of a specific embodiment, where the boxed portion represents the marker diagram corresponding to a thread block.

[0059] Furthermore, for any thread block, starting from the first pixel in the top left corner of the thread block, its depth similarity is compared with that of its neighboring pixels.

[0060] In one optional embodiment, the neighboring pixels include pixels located above the current pixel and pixels located to the left of the current pixel. Priority is given to comparing with pixels located above the current pixel, followed by comparison with pixels located to the left of the current pixel, ensuring that the marker propagation direction is from top to bottom and from left to right.

[0061] In the specific implementation process, for Figure 3 (a) shows the initial labeled image. When the depth values ​​of all pixels within the thread block meet the high similarity criterion, each pixel in the thread block, except for the top left corner, is relabeled with the label value of the top left pixel. That is, 9, 10, 11, 15, 16, 17, 18, 22, 23, 24, and 25 are updated to 8 (left), 9 (left), 10 (left), 8 (top), 9 (top), 10 (top), 11 (top), 15 (top), 16 (top), 17 (top), and 18 (top), resulting in the following: Figure 3 (b) shows the updated marker diagram.

[0062] After iterative processing, the result is as follows: Figure 4 The optimized label map shown here has connected regions with similar depth values ​​in a label map of the same size as the depth map, and these regions are labeled with the same label value.

[0063] Furthermore, connected regions in the depth map are determined based on the optimized label map. Specifically, based on the optimized label map, all pixels with the same label value are taken as connected regions, and bilateral filtering is performed on the depth map based on the connected regions.

[0064] Furthermore, considering that each thread block is independent of the others and the connected regions within each thread block do not match the actual connected regions, it is necessary to update and confirm the connected regions in each thread block based on the distribution of connected regions in the labeled graph. That is, the labeled values ​​in the thread block are updated according to the labeled values ​​of the connected regions adjacent to each other in the upper left direction, and the final global connected region labeled graph is constructed.

[0065] In an optional embodiment, according to the optimized labeling map, all pixels with the same label value are taken as connected regions, and the method further includes the following steps: for any connected region, check whether there is an adjacent connected region in the upper left direction from the optimized labeling map. If so, update the label value of all pixels in the current connected region with the corresponding label value of the adjacent connected region in the upper left direction, and confirm the update of the connected region; repeat the above steps until a preset update threshold is reached to obtain the updated connected region.

[0066] This embodiment utilizes the high-performance computing capabilities of GPU memory to accelerate the search of all connected components. Secondly, it addresses the issues of incomplete and discontinuous connected components caused by GPU block search through updating the labeled graph, thereby ensuring the correctness of the results.

[0067] Further, in an optional embodiment, the number of pixels with the same marker value in each connected region is counted (Count) and compared with a preset area threshold: if Count is less than or equal to the preset area threshold, the connected region is determined to be a fragmented connected region, and the fragmented connected region is interpolated and filled using the nearest depth value; otherwise, the connected region is retained.

[0068] In this embodiment, fragmented connected regions in the depth map are identified as depth map noise. Specifically, the number of pixels with the same marker value (Count) within a connected region is used to determine whether the current connected region belongs to a fragmented connected region. If it is, the depth values ​​around the fragmented connected region are interpolated to fill the gap; otherwise, no processing is performed. This filters all fragmented connected regions in the depth map, further improving the depth map filtering effect.

[0069] In a specific implementation process, such as Figure 5 The image shown illustrates the filtering effect. Figure 5 (a) is the initial depth map, and the area highlighted in the map is a region of fine noise. Figure 5 (b) shows the result after depth map filtering. The GPU-based depth map filtering method proposed in this embodiment is used for filtering. Compared with the traditional CPU method, the filtering efficiency for connected component detection is shown in Table 1 below:

[0070] Table 1

[0071] 3000*2000 81.83 11.987 4000*3000 170.22 18.377 5000*5000 253.26 24.087

[0072] As shown in the table above, compared with the traditional CPU method, the connected component detection filtering efficiency of this embodiment is improved by 8-10 times.

[0073] Furthermore, the final depth map obtained by using bilateral filtering can be further used for point cloud fusion and 3D reconstruction.

[0074] Example 2

[0075] This embodiment proposes a GPU-based depth map filtering system, applying the GPU-based depth map filtering method proposed in Embodiment 1. For example... Figure 6 The diagram shown is an architecture diagram of the GPU-based depth map filtering system in this embodiment.

[0076] The GPU-based depth map filtering system proposed in this embodiment includes:

[0077] The preprocessing module is used to divide the depth map to be filtered into several equally sized thread blocks.

[0078] The GPU module includes several labeling units and several similarity comparison units, and the labeling units and similarity comparison units work in parallel.

[0079] The marking unit is used to determine the marking value of each pixel in the current thread block.

[0080] The similarity comparison unit is used to traverse the pixels in the current thread block, starting from the first pixel in the upper left corner of the thread block, and compare its depth similarity with that of the neighboring pixels: if there is a depth value difference less than a preset threshold, it is determined to be highly similar, and the label value corresponding to the current pixel is updated to the label value corresponding to a neighboring pixel with high similarity; otherwise, it is determined to be low similarity, and no processing is performed on the current pixel.

[0081] The similarity comparison unit iterates until a preset number of iterations is reached, and outputs an optimized marker map of the same size as the depth map.

[0082] The filtering module is used to determine the connected regions in the depth map based on the optimized marker map, and to perform bilateral filtering on the depth map based on the connected regions, and output the filtered depth map.

[0083] In a specific implementation process, the depth map to be filtered is first divided into several equally sized thread blocks by the preprocessing module, and then transmitted to the GPU module, which allocates computing resources to achieve parallel depth map filtering processing.

[0084] In this process, a thread block is input into the marking unit to determine the marking value of each pixel in the current thread block.

[0085] In an alternative embodiment, the step of determining the marker value for each pixel in the thread block includes:

[0086] For any thread block, the initial index value label is determined based on the number of rows r and columns c of the pixels; its expression is: Label = W × r + c; where W is the width value of the depth map; the initial index value label is used as the label value of the current pixel; after traversing all pixels in the thread block, an initial label map matching the depth map is generated.

[0087] The similarity comparison unit performs a similarity comparison on the input thread block. Preferably, it compares the depth similarity of the current pixel with the pixel above the current pixel and the pixel to the left of the current pixel. If there is a depth value difference less than a preset threshold, it is determined that the similarity is high, and the label value corresponding to the current pixel is updated to the label value corresponding to a neighboring pixel with high similarity. Otherwise, it is determined that the similarity is low, and no processing is performed on the current pixel.

[0088] After several similarity comparison units in the GPU module complete the traversal and iteration of the thread blocks, they output the corresponding label map of the thread block, and then perform corresponding matching based on the depth map, outputting an optimized label map of the same size as the depth map to the filtering module.

[0089] The filtering module determines connected regions in the depth map based on the optimized marker map, and performs bilateral filtering on the depth map based on the connected regions, outputting a filtered depth map. Specifically, the filtering module identifies all pixels with the same marker value as connected regions based on the optimized marker map, and performs bilateral filtering on the depth map based on these connected regions, outputting a filtered depth map.

[0090] In an optional embodiment, after determining the connected regions in the depth map, the filtering module searches the optimized label map to see if there is an adjacent connected region in the upper-left direction for any connected region. If so, it updates the label values ​​of all pixels in the current connected region with the label value of the adjacent connected region in the upper-left direction to confirm the update of the connected region. The above steps are repeated until a preset update threshold is reached to obtain the updated connected region, which is further used to perform bilateral filtering on the depth map and output the filtered depth map.

[0091] Furthermore, in an optional embodiment, the filtering module further includes a fragmented region filtering unit, used to count the number of pixels with the same label value in each connected region (Count) and compare it with a preset area threshold. During implementation, if Count is less than or equal to the preset area threshold, the connected region is determined to be a fragmented connected region, and the fragmented connected region is interpolated and filled using neighboring depth values; otherwise, the connected region is retained.

[0092] Example 3

[0093] This embodiment proposes a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the GPU-based depth map filtering method proposed in Embodiment 1.

[0094] Example 4

[0095] This embodiment proposes a storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the GPU-based depth map filtering method proposed in Embodiment 1.

[0096] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A GPU-based depth map filtering method, characterized in that, Includes the following steps: S1. Divide the depth map to be filtered into several thread blocks of equal size; S2. For any thread block, determine the label value of each pixel in the thread block to obtain an initial label map of the same size as the depth map; S3. For any thread block, starting from the top-left pixel in the thread block, compare its depth similarity with that of its neighboring pixels: if there is a depth value difference less than a preset threshold, it is determined to be highly similar, and the label value corresponding to the current pixel is updated to the label value corresponding to a neighboring pixel with high similarity; otherwise, it is determined to be low similarity, and no processing is performed on the current pixel. After completing the traversal, an updated marker map is obtained; the neighboring pixels include the pixels above the current pixel and the pixels to the left of the current pixel; S4. Repeat step S3 until the preset number of iterations is reached, and output the optimized marker map; S5. Determine connected regions in the depth map based on the optimized marker map, and perform bilateral filtering on the depth map based on the connected regions; including: Based on the optimized label map, all pixels with the same label value are considered as connected regions, and bilateral filtering is performed on the depth map based on the connected regions; For any connected region, check whether there is an adjacent connected region in the upper left direction from the optimized labeling map. If so, update the labeling value of all pixels in the current connected region with the corresponding labeling value of the adjacent connected region in the upper left direction to confirm the update of the connected region. Repeat the above steps until the preset update threshold is reached to obtain the connected region that has been updated. Specifically, the GPU is used to execute steps S2 to S4 in parallel on all thread blocks.

2. The GPU-based depth map filtering method according to claim 1, characterized in that, In step S2, the step of determining the marker value of each pixel in the thread block includes: S2.1 For any thread block, determine the initial index value label based on the number of rows r and columns c of the pixels; its expression is: Label=W×r+c In the formula, W is the width value of the depth map; S2.

2. Use the initial index value label as the label value of the current pixel; after traversing all pixels in the thread block, generate an initial label map that matches the depth map.

3. The GPU-based depth map filtering method according to claim 2, characterized in that, Step S5 also includes the following steps: The number of pixels with the same marker value in each connected region is counted (Count) and compared with a preset area threshold. If Count is less than or equal to the preset area threshold, the connected region is determined to be a fragmented connected region, and the fragmented connected region is filled by interpolation using the nearest depth value; otherwise, the connected region is retained.

4. A GPU-based depth map filtering system, applied to the GPU-based depth map filtering method according to any one of claims 1 to 3, characterized in that, The system includes: The preprocessing module is used to divide the depth map to be filtered into several equally sized thread blocks; The GPU module includes several labeling units and several similarity comparison units, and the labeling units and similarity comparison units work in parallel; The marking unit is used to determine the marking value of each pixel in the current thread block; The similarity comparison unit is used to traverse the pixels in the current thread block, starting from the first pixel in the upper left corner of the thread block, and compare its depth similarity with that of its neighboring pixels: if there is a depth value difference less than a preset threshold, it is determined to be highly similar, and the label value corresponding to the current pixel is updated to the label value corresponding to a neighboring pixel with high similarity; otherwise, it is determined to be low similarity, and no processing is performed on the current pixel. The similarity comparison unit iterates until a preset number of iterations is reached, and outputs an optimized label map of the same size as the depth map; The filtering module is used to determine the connected regions in the depth map based on the optimized marker map, and to perform bilateral filtering on the depth map based on the connected regions, and output the filtered depth map.

5. The GPU-based depth map filtering system according to claim 4, characterized in that, The filtering module also includes a fragmented region filtering unit, which is used to count the number of pixels with the same label value in each connected region (Count) and compare it with a preset area threshold: if Count is less than or equal to the preset area threshold, the connected region is determined to be a fragmented connected region, and the fragmented connected region is interpolated and filled using the nearest depth value; otherwise, the connected region is retained.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the GPU-based depth map filtering method according to any one of claims 1 to 3.

7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the GPU-based depth map filtering method as described in any one of claims 1 to 3.

Images