Layered minimum curved surface reconstruction-based laser radar depth completion method

Abstract

The invention discloses a laser radar depth completion method based on hierarchical minimum surface reconstruction, and relates to the technical field of computer vision and image processing, and the method comprises the steps: carrying out the depth value inversion and morphological expansion processing of a sparse depth map, obtaining a scene expansion depth map and a full-resolution effective mask, and obtaining a full-resolution effective mask; then performing down-sampling to obtain a coarse scale scene expansion image and a coarse scale mask, and obtaining a coarse scale filling image by using an iterative convolution kernel of a Laplace operator; performing up-sampling on the coarse scale filling graph to obtain an up-sampled depth graph, and fusing the up-sampled depth graph with the scene expansion depth graph to obtain a fused depth graph; positioning and fusing all residual holes in the depth map through a connected domain analysis method, calculating a depth mean value of effective pixels in an annular pixel band around each residual hole, and filling the corresponding residual holes with the depth mean value to obtain a hole-free depth map; and after global Gaussian blur, executing a depth value inversion operation to obtain a dense depth map of the scene, thereby realizing high-precision three-dimensional reconstruction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image processing, and particularly relates to a laser radar depth completion method based on hierarchical minimum surface reconstruction. BACKGROUND

[0002] As an image containing distance information of the surface of scene objects, a depth map plays an irreplaceable role in cutting-edge fields such as three-dimensional reconstruction, automatic driving environment perception, robot navigation and obstacle avoidance, and augmented reality (AR) interaction. However, due to the imaging principle of hardware, the data collected by the existing mainstream depth acquisition devices such as laser radar (LiDAR) is essentially sparse three-dimensional point cloud. When the three-dimensional point cloud is projected onto the camera imaging plane through coordinate transformation, the sparse depth image (Sparse Depth Map) formed will have a large number of unsampled zero-pixel regions. Therefore, the laser radar depth completion (Depth Completion) technology, that is, using an algorithm to recover a dense, continuous and structurally complete depth map from sparse and incomplete depth data, is a prerequisite for the implementation of the above-mentioned applications.

[0003] The existing depth completion methods are mainly divided into two categories: depth learning-based methods and non-learning traditional image processing methods. Although the depth learning-based methods have made great progress in accuracy, they often rely on huge computing resources and massive training data, and the generalization ability is limited to the scenes covered by the training set. In contrast, the non-learning traditional depth completion methods still occupy an important position in the edge computing scene with extremely high real-time requirements due to their small amount of calculation, no need for pre-training, and low requirement for hardware resources, or are often used as input preprocessing modules of complex deep neural networks (DNNs) to provide reliable scene structure prior for subsequent networks.

[0004] However, the existing conventional depth completion algorithm has significant technical limitations in processing indoor scenes with complex geometric structures. For example, the IP-Basic algorithm, which is widely used as a benchmark in current mainstream depth completion methods, mainly relies on morphological dilation and closing operations on the depth map to mechanically spread the sparse effective depth values to the surrounding empty regions. This strategy is still acceptable when processing outdoor sparse data at a long distance (such as the KITTI dataset), but it exposes its limitations in indoor scenes. Since the algorithm essentially assumes that the depth in the local region is piecewise constant, it ignores the continuous and gradual change of depth values in three-dimensional space, especially at close distances. Therefore, when dealing with continuous surfaces with inclined angles commonly found in indoor scenes (such as extended corridor floors and inclined walls), simple morphological diffusion cannot fit the linear variation of depth values, easily producing obvious step effects and block artifacts in the completed regions, severely damaging the smoothness and realism of the scene's geometric structure, and failing to meet the needs of high-precision three-dimensional reconstruction.

[0005] Therefore, how to provide a laser radar depth completion method for an indoor scene that can meet the needs of high-precision three-dimensional reconstruction has become an important problem. SUMMARY

[0006] To solve the above problems in the prior art, the present application provides a laser radar depth completion method based on hierarchical minimum surface reconstruction.

[0007] The technical problem to be solved by the present application is solved by the following technical scheme: In a first aspect, the present application provides a laser radar depth completion method based on hierarchical minimum surface reconstruction, which comprises: performing depth value inversion and morphological dilation processing on the sparse depth map of the laser radar system to obtain a scene dilated depth map and a full-resolution effective mask; performing down-sampling on the scene dilated depth map and the full-resolution effective mask respectively to obtain a coarse-scale scene dilated map and a coarse-scale mask, and then performing an iterative filling operation using an iterative convolution kernel of Laplace operator to obtain a coarse-scale filled map; performing up-sampling on the coarse-scale filled map to obtain an up-sampled depth map, and fusing the up-sampled depth map with the scene dilated depth map to obtain a fused depth map; locating all remaining holes in the fused depth map by connected component analysis method, calculating the depth mean of the effective pixels in the annular pixel band around each remaining hole, and filling the corresponding remaining hole using the depth mean to obtain a hole-free depth map; After global Gaussian blur is performed on the non-hole depth map, a depth value inversion operation is performed to obtain a dense scene depth map.

[0008] In a second aspect, the present application provides a laser radar depth completion device based on layered minimum surface reconstruction, the laser radar depth completion device comprising: A processing module is configured to perform depth value inversion and morphological dilation processing on a sparse depth map of a laser radar system to obtain a scene dilation depth map and a full-resolution effective mask. A first filling module is configured to perform down-sampling on the scene dilation depth map and the full-resolution effective mask respectively to obtain a coarse-scale scene dilation map and a coarse-scale mask, and then perform an iterative filling operation using an iterative convolution kernel of a Laplacian operator to obtain a coarse-scale filling map. A fusion module is configured to perform up-sampling on the coarse-scale filling map to obtain an up-sampled depth map, and fuse the up-sampled depth map with the scene dilation depth map to obtain a fused depth map. A second filling module is configured to locate all remaining holes in the fused depth map by a connected domain analysis method, calculate a depth mean value of effective pixels in a ring-shaped pixel band around each remaining hole, and fill the corresponding remaining hole using the depth mean value to obtain a non-hole depth map. An inversion module is configured to perform a global Gaussian blur on the non-hole depth map, and then perform a depth value inversion operation to obtain a dense scene depth map.

[0009] In a third aspect, the present application provides an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the computer program stored on the memory to implement the method steps of any one of the above laser radar depth completion methods based on layered minimum surface reconstruction.

[0010] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method steps of any one of the above laser radar depth completion methods based on layered minimum surface reconstruction.

[0011] The application provides a laser radar depth completion method based on layered minimum surface reconstruction, wherein an iterative convolution based on a Laplace operator is used for diffusion filling to replace a traditional large core inflation operation, so that the smooth geometric structure of a large-area continuous surface can be restored while the calculation amount is significantly reduced, and "blocky" artifacts can be effectively avoided. All remaining holes in the fused depth map are located by using a connected domain analysis method, and the depth average of the effective pixels in the annular pixel band around each remaining hole is calculated, and the corresponding remaining hole is filled by using the depth average, so that the small "pinhole" artifacts that are easily left in the reconstruction process are accurately removed, and the accuracy of local details is ensured. The most time-consuming iterative filling process is placed in the coarse scale space after downsampling, so that the operation cost is greatly reduced, and the detail repair at full resolution is only for a small number of remaining holes, so that the calculation load is low. The layered strategy makes the algorithm achieve real-time processing frame rate on the CPU, and has a low deployment cost.

[0012] In addition, as a non-learning method based on image processing and numerical calculation principles, the method does not require training data, has universal and stable processing capability for different scenes and noise modes, and fundamentally avoids the overfitting and generalization failure problems commonly seen in the depth learning mode.

[0013] The application will be further described in detail below with reference to the accompanying drawings and the application. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of a laser radar depth completion method based on layered minimum surface reconstruction provided by an embodiment of the application; Figure 2 is a flowchart of depth value inversion and morphological dilation processing provided by an embodiment of the application; Figure 3 is a flowchart of depth value inversion and self-defined kernel inflation calculation provided by an embodiment of the application; Figure 4 is a flowchart of multi-scale Laplace diffusion provided by an embodiment of the application; Figure 5 is a flowchart of upsampling and depth fusion provided by an embodiment of the application; Figure 6 is a flowchart of remaining hole filling provided by an embodiment of the application; Figure 7 is a flowchart of blur noise reduction and depth value inversion provided by an embodiment of the application; Figure 8 is a qualitative comparison result diagram of the method of the application and Ip_Basic in three different indoor scenes; Figure 9is a schematic diagram of the effect comparison between the method of the present application and the diffusion process of the prior art; Figure 10 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0015] The present application will be further described in detail below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0016] In order to solve the problem that the existing laser radar depth completion method is prone to produce obvious step effect and block-like artifacts in the completion area when processing the continuous surface with an inclined angle commonly seen in indoor scenes, seriously damaging the smoothness and authenticity of the scene geometry, and unable to meet the demand of high-precision three-dimensional reconstruction, an embodiment of the present application provides a laser radar depth completion method based on layered minimum surface reconstruction, as shown in Figure 1 , Figure 1 is a flowchart of the laser radar depth completion method based on layered minimum surface reconstruction provided by an embodiment of the present application, and specifically includes the following steps: Step S101, performing depth value inversion and morphological dilation processing on the sparse depth map of the laser radar system to obtain a scene dilated depth map and a full-resolution valid mask.

[0017] Referring to Figure 2 , Figure 2 is a flowchart of the depth value inversion and morphological dilation processing provided by an embodiment of the present application. The sparse depth map is an image containing distance information of the surface of scene objects, wherein each pixel value represents the distance of the corresponding scene point, and can be generated by a depth acquisition device such as a laser radar (LiDAR).

[0018] Among them, the resolution of the sparse depth map can be .

[0019] In an embodiment of the present application, the depth value inversion and morphological dilation processing are performed on the sparse depth map of the laser radar system to obtain a scene dilated depth map and a full-resolution valid mask, including: Performing a depth value inversion operation on the sparse depth map by using a predefined initial valid pixel mask to obtain an inverted sparse depth map; Performing morphological dilation processing on the inverted sparse depth map by using a preset dilation kernel to obtain a scene dilated depth map; Extracting a full-resolution valid mask based on the scene dilated depth map.

[0020] In the embodiments of the present application, the depth value of the sparse depth map is reversed, so that the pixels in the near distance, i.e. the pixels with smaller depth values, are converted to higher values, and the pixels in the far distance, i.e. the pixels with larger depth values, are converted to lower values. This conversion makes the subsequent morphological operation preferentially propagate the depth information in the near distance. See Figure 3 , Figure 3 is a schematic diagram of the depth value reversal and self-defined kernel inflation calculation process provided by the embodiments of the present application, assuming that the maximum depth value obtained is 10, and the conversion formula is , , is the original depth value, is the reversed depth value. For example, for a pixel with an original depth value of 2.0, the reversed depth value is 8.0, and for a pixel with an original depth value of 9.0, the reversed depth value is 1.0.

[0021] In the embodiments of the present application, based on the sparse depth map, an initial valid pixel mask can be predefined according to the pixel positions with depth values Then, the depth value reversal operation is performed to obtain the reversed sparse depth map ; wherein, max_depth represents the maximum valid depth, which can be set by the technician according to the requirements, such as 10.0 m; sparse_depth represents the sparse depth map; The depth value reversal operation is only performed on the valid pixels, see Figure 2 , and the invalid pixels, i.e. the pixels with a value of 0, remain 0. Among them, the valid pixels refer to the pixels with a depth value greater than 0.1.

[0022] In the embodiments of the present application, a morphological closing operation is performed on the reversed sparse depth map using a smaller preset inflation kernel to connect the adjacent valid pixels and fill in the small holes. The above operation is called self-defined kernel inflation. The self-defined kernel inflation includes: performing at least once a small kernel inflation operation and once a closing operation on the reversed sparse depth map in sequence to obtain a scene inflation depth map, and the specific process is as follows: On the basis of the reversed sparse depth map, a full kernel with a smaller size, such as is first used to inflate the reversed sparse depth map, diffuse the high-value pixels, and expand the valid area; Then, the same full kernel is used to perform the closing operation to fill in the small holes, close the small gaps, and smooth the boundaries, so as to avoid excessive inflation, thereby obtaining the scene inflation depth map.

[0023] ​​Then, for pixels with a depth value that is not 0 in the scene dilation depth map, the value is set to 1, and for pixels with a depth value of 0, the value is set to 0, thus obtaining a full-resolution effective mask.

[0024] See Figure 3 The custom kernel expansion section allows for a custom kernel size that can also be set to [value]. The final filled result, i.e., the scene dilation depth map, is obtained by using a custom kernel to check the inverted depth of the sparse depth map after inversion.

[0025] Step S102: Downsample the scene dilation depth map and the full-resolution effective mask to obtain the coarse-scale scene dilation map and the coarse-scale mask. Then, use the iterative convolution kernel of the Laplacian operator to perform iterative filling operation to obtain the coarse-scale filled map.

[0026] In this embodiment of the invention, the process of downsampling the scene dilation depth map and the full-resolution effective mask to obtain the coarse-scale scene dilation map and the coarse-scale mask is as follows: First, set the downsampling factor. For example, it can be 4 times. And calculate the coarse-scale image size, for example... Pixels. Then, the scene dilation depth map is calculated using nearest neighbor interpolation. and full-resolution effective mask Downsampling was performed separately to obtain coarse-scale scene inflation maps. and coarse-scale mask The nearest neighbor interpolation method is used to ensure that the mask boundaries are clear and the inflation value is not changed at a coarse scale.

[0027] See Figure 4 , Figure 4 This is a schematic diagram of the multi-scale Laplacian diffusion process provided in an embodiment of the present invention. Iterative filling operations are performed using the iterative convolution kernel of the Laplacian operator to obtain a coarse-scale filled map, including: Duplicate the coarse-scale scene dilation map to obtain the initial coarse-scale fill map; Perform a Laplacian convolution on the initial coarse-scale filled map to obtain the average map; Using an iterative update formula, the coarse-scale scene dilation map is filled based on the average value map and the coarse-scale mask to obtain an intermediate coarse-scale filled map; The intermediate coarse-scale filled image is used as the initial coarse-scale filled image, and the process of performing a Laplacian convolution on the initial coarse-scale filled image to obtain the average value image is repeated until the preset number of iterations is reached, at which point the coarse-scale filled image is obtained.

[0028] Specifically, first, we define the iterative convolution kernel of the Laplacian operator. An exemplary Laplacian operator iterative convolution kernel can be defined as a kernel with a size of , a center value of 0, four orthogonal neighborhood values of 0.25, and the rest of the corner values of 0. This convolution kernel is used to calculate the average value of the 4-neighborhood of the center pixel. The number of iterations of the coarse scale is set to, for example, 50, and the coarse scale scene dilation map is copied to obtain an initial coarse scale filling map , which is used as the starting point of the iteration.

[0029] Then, an iteration loop is entered. In each iteration, the Laplacian convolution is performed on the initial coarse scale filling map to obtain an average value map .

[0030] The coarse scale scene dilation map is filled based on the average value map and the coarse scale mask using an iterative update formula to obtain an intermediate coarse scale filling map.

[0031] The iterative update formula is as follows: ; ; wherein, avg denotes the average value map; int denotes the intermediate coarse scale filling map; K denotes the iterative convolution kernel of the Laplacian operator; dil denotes the coarse scale scene dilation map; and mask denotes the coarse scale mask.

[0032] In the iterative update formula, the region with a valid value in the coarse scale scene dilation map is forced to retain its original value to avoid being affected by the iteration process, and the region with an invalid value is updated to the average value of its neighborhood, i.e. After 50 iterations, the value of the hollow region is smoothly diffused from its boundary to form a minimum surface interpolation, thereby recovering the global structure.

[0033] In step S103, the coarse scale filling map is up-sampled to obtain an up-sampled depth map, and the up-sampled depth map is fused with the scene dilation depth map to obtain a fused depth map.

[0034] Referring to Figure 5 , Figure 5 is a flowchart of the up-sampling and depth fusion provided by the embodiments of the present application. The coarse scale filling map is up-sampled to obtain an up-sampled depth map, which includes: The coarse scale filling map is up-sampled back to the original resolution by using the bilinear interpolation method to obtain the up-sampled depth map . ​​​​​​

[0035] wherein the original resolution refers to the resolution of the sparse depth map, such as .

[0036] In the embodiment of the present application, the up-sampled depth map is fused with the scene dilated depth map to obtain a fused depth map, comprising: ; wherein, represents the fused depth map; represents the scene dilated depth map; represents the full-resolution valid mask; represents the up-sampled depth map; represents the inverse mask; The up-sampling result is fused with the scene dilated depth map, that is, the up-sampled depth map is used to fill the hole region in the scene dilated depth map, while the original valid pixels in S101, that is, the original valid values, are completely retained, to obtain a fused depth map. Through the fusion processing, all high-resolution original dilated information is losslessly retained, and the calculated low-resolution global structure, that is, the up-sampled depth map, is only used to fill the large-area hole in S101 which is not covered, that is, the position represented by.

[0037] In step S104, all remaining holes in the fused depth map are located by the connected component analysis method, and the depth mean value of the annular valid pixel band around each remaining hole is calculated, and the corresponding remaining hole is filled by using the depth mean value, to obtain a non-hole depth map.

[0038] The fused depth map obtained by step S103 restores the macro structure, but may leave small remaining holes at the edge of the original valid value or the interpolation region. How to clean the remaining holes will be described below, referring to Figure 6 , Figure 6 is a flowchart of the remaining hole filling provided by the embodiment of the present application: In the embodiment of the present application, all remaining holes in the fused depth map are located by the connected component analysis method, and the depth mean value of the valid pixels in the annular pixel band around each remaining hole is calculated, and the corresponding remaining hole is filled by using the depth mean value, to obtain a non-hole depth map, comprising: locating all remaining holes in the fused depth map; performing connected component analysis on all remaining holes, and assigning an integer label to each remaining hole; traversing all integer labels, and performing dilation on the remaining holes with a binary mask of 1 using two morphological kernels of different sizes to obtain a corresponding first dilated region and a second dilated region; The difference set of the first expanded region and the second expanded region is calculated to obtain a ring-shaped pixel band; The depth mean of all valid pixels in the ring-shaped pixel band is calculated, and the depth mean is used to fill the remaining holes of the binary mask to obtain a non-hole depth map.

[0039] Specifically, first, all remaining holes are obtained, and connected domain analysis is performed on the remaining holes. Each independent hole is assigned a unique integer label.

[0040] Each integer label is traversed to obtain the binary mask of the current remaining hole. The remaining holes with a binary mask of 1 are expanded using two different size morphological kernels to obtain a first expanded region and a second expanded region. For example, the remaining holes are expanded using a small size morphological kernel (such as a full kernel) to obtain the first expanded region .

[0041] The remaining holes are expanded again using a large size morphological kernel (such as a full kernel) to obtain the second expanded region .

[0042] The difference set of the first expanded region and the second expanded region is calculated : ; This difference set is a ring-shaped pixel band around the current remaining hole. The depth mean of all valid pixels in the ring-shaped pixel band is calculated, where the valid pixels are pixels with a pixel value greater than 0. Then, after filling the corresponding remaining hole using the depth mean, the step of traversing each integer label is executed until all integer labels are traversed, to obtain a non-hole depth map .

[0043] In step S105, after performing global Gaussian blur on the non-hole depth map, a depth value inversion operation is performed to obtain a scene dense depth map.

[0044] Referring to Figure 7 , Figure 7 is a flowchart of the blur noise reduction and depth value inversion provided by the embodiments of the present application. Global Gaussian blur is performed on the non-hole depth map to smooth the slight blockiness that may be caused by the upsampling operation and the edge abruptness that may be caused by the hole filling operation. The valid pixels in the blurred non-hole depth map are marked as 1, and the invalid pixels are marked as 0 to obtain an effective mask .

[0045] Finally, a depth value inversion operation is performed to restore the depth values to the original domain, i.e., the near value is small and the far value is large. Thus, a dense, smooth structure and no small artifacts of the scene dense depth map are obtained, which effectively improves the robustness and visual quality of depth completion, and then high-precision three-dimensional reconstruction can be realized based on the scene dense depth map. The scene dense depth map is: ; wherein, represents the scene dense depth map; represents a preset maximum effective depth value; represents a non-hole depth map.

[0046] In the embodiment of the present application, the iterative convolution based on the Laplace operator is used for diffusion filling, instead of the traditional large kernel inflation operation, which can significantly reduce the calculation amount while restoring the smooth geometry of large-area continuous surfaces, effectively avoiding "blocky" artifacts. By using the connected domain analysis method, all remaining holes in the fused depth map are located, and the depth mean of the effective pixels in the annular pixel band around each remaining hole is calculated. The corresponding remaining hole is filled by using the depth mean, which accurately removes the small "pinhole" artifacts easily left in the reconstruction process, ensuring the accuracy of local details. In the present application, the most time-consuming iterative filling process is placed in the coarse scale space after downsampling, which greatly reduces the computational cost. The detail repair at full resolution is only for a small number of remaining holes, and the calculation load is low. This hierarchical strategy makes the algorithm achieve real-time processing frame rate on the CPU (Central Processing Unit), with lower deployment cost.

[0047] In addition, as a non-learning method based entirely on image processing and numerical calculation principles, the present application does not require training data and has universal and stable processing capabilities for different scenes and noise patterns, fundamentally avoiding the overfitting and generalization failure problems commonly seen in deep learning methods.

[0048] The simulation experiment of the laser radar depth completion method based on hierarchical minimum surface reconstruction provided by the embodiment of the present application is as follows: In the simulation experiment, an AMD EPYC 7542 processor is selected as the hardware platform, and an NYU Depth v indoor depth dataset is selected as the test data, and the following preprocessing steps are performed to construct the standard input: first, the sparse depth map of the laser radar system is scaled to resolution; then, a center cropping operation is performed to obtain an image with a resolution of Finally, 500 pixel points are randomly sampled in the image as sparse depth inputs to simulate extremely sparse laser radar scan data.

[0049] In the selection of evaluation indexes, the experiment adopts the root mean square error (RMSE) and the mean absolute error (MAE) to quantify the completion accuracy of the depth map, and adopts the floating point operation number (FLOPs) and the single frame inference time (ms) to evaluate the running efficiency of the algorithm. In order to eliminate the influence of randomness on the experimental results and ensure the reliability of the data, the average value of the results of 5 independent repeated experiments is taken as all the experimental indexes. Referring to Table 1, Table 1 is a comparison table of the results of the present application and the traditional method.

[0050] Table 1 Comparison table of results of the present application and the traditional method

[0051] A detailed analysis of the experimental data in Table 1 shows that the laser radar depth completion method based on hierarchical minimum surface reconstruction provided by the embodiment of the present application has a breakthrough performance advantage in image reconstruction quality. Specifically, the RMSE index of the laser radar depth completion method based on hierarchical minimum surface reconstruction provided by the embodiment of the present application is reduced to 172.5 mm, and the MAE index is reduced to 71.8 mm. Compared with the traditional method (IP-Basic) which is the best in the prior art, the present application improves the accuracy of depth completion to more than 1.7 times of the prior art while maintaining very low error. This significant performance leap is mainly due to the hierarchical strategy of the present application: the global geometric structure of the scene is effectively recovered through coarse-scale minimum surface reconstruction, avoiding the structural distortion caused by traditional large kernel inflation; and the residual holes are filled to accurately repair the small holes, thereby greatly reducing the overall error while retaining more realistic edge details.

[0052] In terms of algorithm running efficiency, although the theoretical calculation amount and single frame inference time of the present application increase slightly in numerical value compared with the benchmark method, the computing power of the mainstream edge side hardware platform such as ARM Cortex-A78AE can reach 280 GFLOPs, and this slight increase in computing load can be completely ignored. The measured inference speed of 7.526 ms is much lower than the 33 ms time threshold of conventional real-time processing, and will not become a running bottleneck of the system at all. Therefore, the present application exchanges the nearly doubled completion accuracy for the extremely low and non-real-time power cost (only increases about 1 ms of time consumption), which fully proves that the present application has high robustness and cost performance in actual engineering application, and can fully meet the performance requirements of modern computing platforms.

[0053] In order to intuitively show the depth completion capability of the present application in complex indoor scenes, Figure 8This is a schematic diagram showing the qualitative comparison results between the method of this invention and Ip_Basic in three different indoor scenarios. The comparison diagram includes the original truth map, the Ip_Basic completion result, the completion result of this invention (Ours), and the absolute difference map between each and the truth.

[0054] From the perspective of surface smoothness reconstruction, the Ip_Basic method, when processing large continuous surfaces such as walls and floors, exhibits significant stair-step effects and blocky artifacts on planes with tilt angles due to its algorithm's assumption of a local constant depth, thus disrupting the geometric continuity of the scene. In contrast, the depth map generated by the method of this invention demonstrates excellent smoothness in these areas, with depth values ​​showing a natural gradient transition. This proves that the minimum surface reconstruction strategy based on the Laplacian operator used in this invention can effectively adapt to the physical characteristics of linear changes in depth with spatial distance in indoor scenes, successfully eliminating blocky artifacts.

[0055] In terms of restoring the details of object edges, this invention is also superior to existing technologies. See also Figure 8 In traditional Chinese programming, object outlines in Ip_Basic often exhibit blurring or jagged edges when filling in the edges. This invention, however, uses residual hole filling and morphological optimization to fill tiny holes while more accurately maintaining a clear boundary between the object and the background, without introducing excessive smoothing or structural distortion.

[0056] Furthermore, observation of the difference maps reveals a large number of gray-white noise points in the Ip_Basic difference map, indicating a widespread depth error globally. In contrast, the difference map corresponding to this invention is predominantly black, with only slight errors in a very few areas with extremely complex textures. This visual result aligns with the significant decrease in RMSE and MAE error indices in Table 1, intuitively confirming that the scene-dense depth map generated by this invention significantly outperforms existing mainstream methods in both structural integrity and numerical accuracy.

[0057] Those skilled in the art should understand that the parameters in the above embodiments, such as the sampling factor, the number of coarse-scale iterations, and the kernel size for filling remaining holes, are preferred embodiments of the present invention and not limitations thereof. These parameters can be modified or replaced without departing from the spirit and principle of the present invention to adapt to different application requirements. For example, iterative solutions can be replaced with other numerical methods for solving the Poisson equation.

[0058] This invention provides a lidar depth completion method based on hierarchical minimum surface reconstruction, employing a hybrid strategy. It introduces a hierarchical minimum surface reconstruction mechanism based on downsampling space to address the performance bottlenecks and structural distortion issues of traditional methods. For example... Figure 9 As shown, Figure 9is a schematic diagram of effect comparison between the method of the present application and the diffusion process of the prior art. The mechanism simulates the physical heat diffusion process in mathematics, so that the depth value can be naturally propagated in the hole area according to the smoothing constraint defined by the Laplace operator. This filling method based on partial differential equation essentially constructs a minimum surface that satisfies the boundary condition, so as to accurately fit the inclined wall or extended ground commonly seen in indoor scenes in geometry, effectively eliminating the step-like artifacts caused by the local constant assumption of the traditional morphological large-core inflation method. And using a multi-scale strategy, the most computationally intensive iterative solving process is placed in a low-dimensional space, so that the amount of data to be processed decreases geometrically, greatly reducing the number of floating point operations, ensuring that the algorithm maintains high-precision smooth surface reconstruction while achieving the running efficiency of the CPU end real-time application scenario.

[0059] Based on the same inventive concept, the embodiment of the present application also provides a laser radar depth completion device based on hierarchical minimum surface reconstruction, the laser radar depth completion device comprising: A processing module is configured to perform depth value inversion and morphological inflation processing on the sparse depth map of the laser radar system to obtain a scene inflation depth map and a full-resolution effective mask. A first filling module is configured to perform down-sampling on the scene inflation depth map and the full-resolution effective mask respectively to obtain a coarse-scale scene inflation map and a coarse-scale mask, and then perform iterative filling operation by using an iterative convolution kernel of a Laplace operator to obtain a coarse-scale filling map. A fusion module is configured to perform up-sampling on the coarse-scale filling map to obtain an up-sampled depth map, and fuse the up-sampled depth map with the scene inflation depth map to obtain a fused depth map. A second filling module is configured to locate all remaining holes in the fused depth map by using a connected domain analysis method, calculate the depth mean of the effective pixels in the annular pixel band around each remaining hole, and fill the corresponding remaining hole by using the depth mean to obtain a hole-free depth map. An inversion module is configured to perform global Gaussian blur on the hole-free depth map, and then perform depth value inversion operation to obtain a scene dense depth map.

[0060] In the embodiment of the present application, the iterative convolution based on the Laplace operator is used for diffusion filling, instead of the traditional large-core inflation operation, which can significantly reduce the calculation amount while restoring the smooth geometric structure of the large-area continuous surface, effectively avoiding the "block" artifacts. By using the connected domain analysis method to locate all remaining holes in the fused depth map, and calculating the depth mean of the effective pixels in the annular pixel band around each remaining hole, the corresponding remaining hole is filled by using the depth mean, which accurately removes the small "pinhole" artifacts easily left in the reconstruction process, ensuring the accuracy of local details. The most time-consuming iteration filling process is placed in the coarse scale space after downsampling, which greatly reduces the computational overhead, and the detail repair at full resolution is only for a small amount of remaining holes, and the calculation load is low. This hierarchical strategy makes the algorithm achieve real-time processing frame rate on the CPU, and has lower deployment cost.

[0061] In addition, as a non-learning method based entirely on image processing and numerical calculation principles, the method does not require training data and has universal and stable processing capabilities for different scenes and noise patterns, fundamentally avoiding the overfitting and generalization failure problems commonly seen in deep learning methods.

[0062] The embodiment of the present application also provides an electronic device, such as Figure 10 As shown in the figure, Figure 10 It is a structural schematic diagram of an electronic device provided by the embodiment of the present application, comprising a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 complete mutual communication through the communication bus 1004, The memory 1003 is used for storing computer programs; The processor 1001 is used for executing the programs stored in the memory 1003, and realizes the method steps of the above-mentioned any kind of laser radar depth completion method based on hierarchical minimum surface reconstruction.

[0063] The communication bus mentioned in the above-mentioned electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0064] The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0065] The memory can include random access memory (RAM) and can also include non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0066] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0067] The application further provides a computer readable storage medium. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the method steps of any one of the above laser radar depth completion methods based on layered minimum surface reconstruction are implemented.

[0068] Optionally, the computer readable storage medium can be a non-volatile memory (NVM), for example, at least one disk memory.

[0069] Optionally, the computer readable storage medium can also be at least one storage device located away from the processor.

[0070] In another embodiment of the application, a computer program product containing instructions, which, when run on a computer, causes the computer to perform the method steps of any one of the above laser radar depth completion methods based on layered minimum surface reconstruction.

[0071] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the application. Rather, they are merely examples of devices and methods consistent with some aspects of the application.

[0072] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the description of the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0073] Although the present application is described herein in conjunction with various embodiments, those skilled in the art, with reference to the drawings and the disclosure, can understand and implement other variations of the disclosed embodiments in the implementation of the claimed application. In the description of the present application, the word "comprising" does not exclude other components or steps, "one" or "an" does not exclude a plurality, and "plurality" means two or more, unless otherwise explicitly specified. In addition, some measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0074] The method provided by the embodiments of the present application can be applied to electronic devices. Specifically, the electronic device can be: desktop computer, portable computer, smart mobile terminal, server, etc. Herein, any electronic device that can implement the present application belongs to the protection scope of the present application.

[0075] For device / electronic device / storage medium embodiments, because they are basically similar to method embodiments, the description is relatively simple, and the relevant part can be referred to the part of the method embodiment.

[0076] It should be noted that the device, electronic device and storage medium of the embodiments of the present application are respectively the device, electronic device and storage medium of the above-mentioned one kind of laser radar depth completion method based on layered minimum surface reconstruction, and all embodiments of the above-mentioned one kind of laser radar depth completion method based on layered minimum surface reconstruction are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0077] The above is a further detailed description of the present application in conjunction with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be regarded as falling within the protection scope of the present application.

Claims

1. A method for lidar depth completion based on hierarchical minimum surface reconstruction, characterized in that, The laser radar depth completion method comprises: The sparse depth map of the laser radar system is subjected to depth value inversion and morphological dilation processing to obtain a scene dilation depth map and a full-resolution effective mask; The scene dilation depth map and the full-resolution effective mask are respectively down-sampled to obtain a coarse-scale scene dilation map and a coarse-scale mask, and then an iterative convolution kernel of a Laplace operator is used to perform an iterative filling operation to obtain a coarse-scale filling map; The coarse-scale filling map is up-sampled to obtain an up-sampled depth map, and the up-sampled depth map is fused with the scene dilation depth map to obtain a fused depth map; All remaining holes in the fused depth map are located by a connected domain analysis method, and the depth average of the effective pixels in the annular pixel band around each remaining hole is calculated, and the corresponding remaining hole is filled by using the depth average to obtain a non-hole depth map; After the non-hole depth map is subjected to global Gaussian blurring, a depth value inversion operation is performed to obtain a scene dense depth map.

2. The lidar depth completion method of claim 1, wherein, The sparse depth map of the laser radar system is subjected to depth value inversion and morphological dilation processing to obtain a scene dilation depth map and a full-resolution effective mask, comprising: The sparse depth map is subjected to a depth value inversion operation by using a predefined initial effective pixel mask to obtain an inverted sparse depth map; The inverted sparse depth map is subjected to morphological dilation processing by using a preset dilation kernel to obtain a scene dilation depth map; The full-resolution effective mask is extracted based on the scene dilation depth map.

3. The lidar depth completion method of claim 1, wherein, An iterative filling operation is performed by using an iterative convolution kernel of a Laplace operator to obtain a coarse-scale filling map, comprising: The coarse-scale scene dilation map is copied to obtain an initial coarse-scale filling map; The initial coarse-scale filling map is subjected to Laplace convolution to obtain an average value map; The coarse-scale scene dilation map is filled based on the average value map and the coarse-scale mask by using an iterative update formula to obtain an intermediate coarse-scale filling map; The intermediate coarse-scale filling map is taken as the initial coarse-scale filling map, and the step of performing Laplace convolution on the initial coarse-scale filling map to obtain an average value map is returned to be executed until the number of iterations reaches a preset number of times, and a coarse-scale filling map is obtained.

4. The lidar depth completion method of claim 3, wherein, The iterative update formula is: ； ； wherein, denotes the average map; denotes the intermediate coarse scale filling map; denotes the iterated convolution kernel of the Laplacian operator; denotes the coarse scale scene dilation map; denotes the coarse scale mask.

5. The lidar depth completion method of claim 1, wherein, The coarse-scale filling map is up-sampled to obtain an up-sampled depth map, comprising: The coarse-scale filling map is up-sampled back to the original resolution by using a bilinear interpolation method to obtain an up-sampled depth map.

6. The lidar depth completion method of claim 1, wherein, The up-sampled depth map is fused with the scene dilation depth map to obtain a fused depth map, comprising: ； wherein, represents the fused depth map; represents the scene dilated depth map; represents the full resolution valid mask; represents the upsampled depth map.

7. The lidar depth completion method of claim 1, wherein, All remaining holes in the fused depth map are located by a connected domain analysis method, and the depth average of the effective pixels in the annular pixel band around each remaining hole is calculated, and the corresponding remaining hole is filled by using the depth average to obtain a non-hole depth map, comprising: All remaining holes in the fused depth map are located; All remaining holes are subjected to connected domain analysis, and each remaining hole is assigned an integer label; All integer labels are traversed, and two different size morphological kernels are used to dilate the remaining holes with a binary mask of 1 to obtain a corresponding first dilation region and a second dilation region; A difference set of the first dilated region and the second dilated region is calculated to obtain a ring-shaped pixel band; A depth mean of all valid pixels in the ring-shaped pixel band is calculated, and the depth mean is used to fill the remaining holes of the binary mask with 1 to obtain a non-hole depth map.

8. A device for depth completion of a LiDAR based on layered minimum surface reconstruction, characterized in that, The laser radar depth completion device comprises: A processing module is configured to perform depth value inversion and morphological dilation processing on a sparse depth map of a laser radar system to obtain a scene dilated depth map and a full-resolution valid mask; A first filling module is configured to perform down-sampling on the scene dilated depth map and the full-resolution valid mask respectively to obtain a coarse-scale scene dilated map and a coarse-scale mask, and then perform an iterative filling operation by using an iterative convolution kernel of a Laplacian operator to obtain a coarse-scale filling map; A fusion module is configured to perform up-sampling on the coarse-scale filling map to obtain an up-sampled depth map, and fuse the up-sampled depth map with the scene dilated depth map to obtain a fused depth map; A second filling module is configured to locate all remaining holes in the fused depth map by using a connected domain analysis method, calculate a depth mean of valid pixels in a ring-shaped pixel band around each remaining hole, and fill the corresponding remaining hole by using the depth mean to obtain a non-hole depth map; An inversion module is configured to perform a global Gaussian blur on the non-hole depth map, and then perform a depth value inversion operation to obtain a scene dense depth map.

9. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the computer program stored in the memory to implement the laser radar depth completion method based on hierarchical minimum surface reconstruction according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the laser radar depth completion method based on hierarchical minimum surface reconstruction according to any one of claims 1-7. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the laser radar depth completion method based on hierarchical minimum surface reconstruction according to any one of claims 1-7.

Images