Laser radar depth completion method based on layered minimum surface reconstruction

Abstract

This invention discloses a lidar depth completion method based on hierarchical minimum surface reconstruction, relating to the fields of computer vision and image processing technology. The method includes: performing depth value inversion and morphological dilation on a sparse depth map to obtain a scene dilated depth map and a full-resolution effective mask; then downsampling to obtain a coarse-scale scene dilated map and a coarse-scale mask; using an iterative convolution kernel of the Laplacian operator to obtain a coarse-scale filled map; upsampling the coarse-scale filled map to obtain an upsampled depth map, and then fusing it with the scene dilated depth map to obtain a fused depth map; locating all remaining holes in the fused depth map using connected component analysis, and calculating the mean depth of the effective pixels within the annular pixel band around each remaining hole; using this mean depth to fill the corresponding remaining holes to obtain a hole-free depth map; and after global Gaussian blurring, performing a depth value inversion operation to obtain a scene dense depth map, thereby achieving high-precision 3D reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a lidar depth completion method based on hierarchical minimum surface reconstruction. Background Technology

[0002] Depth maps, as images containing distance information about the surfaces of objects in a scene, play an irreplaceable role in cutting-edge fields such as 3D reconstruction, autonomous driving environmental perception, robot navigation and obstacle avoidance, and augmented reality (AR) interaction. However, limited by hardware imaging principles, existing mainstream depth acquisition devices, such as LiDAR, essentially collect sparse 3D point clouds. When these 3D point clouds are projected onto the camera's imaging plane through coordinate transformation, the resulting sparse depth map contains a large number of unsampled zero-value pixel regions. Therefore, LiDAR depth completion technology, which uses algorithms to recover dense, continuous, and structurally complete depth maps from sparse and incomplete depth data, has become a crucial prerequisite for realizing the aforementioned applications.

[0003] Existing deep completion methods are mainly divided into two categories: deep learning-based methods and traditional non-learning image processing methods. Although deep learning-based methods have made significant progress in accuracy, they often rely on massive computing resources and huge amounts of training data, and their generalization ability is limited by the scenes covered by the training set. In contrast, traditional non-learning deep completion methods, with their advantages of low computational cost, no need for pre-training, and low hardware resource requirements, still occupy an important position in edge computing scenarios with extremely high real-time requirements, or are often used as input preprocessing modules for complex deep neural networks (DNNs), aiming to provide reliable scene structure priors for subsequent networks.

[0004] However, existing traditional depth completion algorithms have significant technical limitations when dealing with geometrically complex indoor scenes. Taking the IP-Basic algorithm, widely used as a benchmark among current mainstream depth completion methods, as an example, this method mainly relies on large-kernel morphological dilation and closing operations on the depth map to mechanically diffuse sparse effective depth values ​​into surrounding void regions. This strategy is reasonably efficient when processing sparse outdoor data from distant locations (such as the KITTI dataset), but it reveals its flaws in indoor scenes. Because this algorithm essentially assumes that the depth within a local region is a piecewise constant, it ignores the continuous, gradual change in depth values ​​with spatial distance in 3D space, especially from a close-up perspective. Therefore, when dealing with continuous surfaces with tilted angles commonly found in indoor scenes (such as extended corridor floors or sloping walls), simple morphological diffusion cannot fit the linear change in depth values, easily producing obvious stair-step effects and blocky artifacts in the completed area. This severely damages the smoothness and realism of the scene's geometry, failing to meet the requirements of high-precision 3D reconstruction.

[0005] Therefore, how to provide a LiDAR depth completion method that can meet the requirements of high-precision 3D reconstruction for indoor scenes has become an important issue. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a lidar depth completion method based on hierarchical minimum surface reconstruction.

[0007] The technical problem to be solved by this invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention provides a lidar depth completion method based on hierarchical minimum surface reconstruction, the lidar depth completion method comprising:

[0009] The sparse depth map of the lidar system is subjected to depth value inversion and morphological dilation to obtain a scene dilated depth map and a full-resolution effective mask.

[0010] After downsampling the scene dilation depth map and the full-resolution effective mask to obtain the coarse-scale scene dilation map and the coarse-scale mask, iterative filling operation is performed using the iterative convolution kernel of the Laplacian operator to obtain the coarse-scale filled map.

[0011] The coarse-scale fill map is upsampled to obtain an upsampled depth map, and the upsampled depth map is fused with the scene dilation depth map to obtain a fused depth map;

[0012] All remaining holes in the fused depth map are located by connected component analysis, and the mean depth of the effective pixels within the annular pixel band around each remaining hole is calculated. The mean depth is then used to fill the corresponding remaining holes to obtain a hole-free depth map.

[0013] After applying a global Gaussian blur to the hole-free depth map, a depth value inversion operation is performed to obtain a scene dense depth map.

[0014] Secondly, the present invention provides a lidar depth completion device based on hierarchical minimum surface reconstruction, the lidar depth completion device comprising:

[0015] The processing module is used to perform depth value inversion and morphological dilation processing on the sparse depth map of the lidar system to obtain a scene dilated depth map and a full-resolution effective mask.

[0016] The first filling module is used to downsample the scene dilation depth map and the full-resolution effective mask respectively to obtain a coarse-scale scene dilation map and a coarse-scale mask. Then, iterative filling operation is performed using the iterative convolution kernel of the Laplacian operator to obtain a coarse-scale filled map.

[0017] The fusion module is used to upsample the coarse-scale fill map to obtain an upsampled depth map, and then fuse the upsampled depth map with the scene dilation depth map to obtain a fused depth map.

[0018] The second filling module is used to locate all remaining holes in the fused depth map using connected component analysis, calculate the average depth of effective pixels within the annular pixel band around each remaining hole, and fill the corresponding remaining holes using the average depth to obtain a hole-free depth map.

[0019] The inversion module is used to perform a depth value inversion operation on the hole-free depth map after applying a global Gaussian blur to obtain a scene dense depth map.

[0020] Thirdly, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0021] Memory, used to store computer programs;

[0022] When a processor executes a computer program stored in memory, it implements the method steps of any of the above-mentioned lidar depth completion methods based on hierarchical minimum surface reconstruction.

[0023] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method steps of any of the above-mentioned lidar depth completion methods based on hierarchical minimum surface reconstruction.

[0024] This invention provides a lidar depth completion method based on hierarchical minimum surface reconstruction. It employs iterative convolution based on the Laplacian operator for diffusion filling, replacing the traditional large kernel dilation operation. This significantly reduces computational cost while restoring the smooth geometry of large-area continuous surfaces, effectively avoiding "blocky" artifacts. By using connected component analysis to locate all remaining holes in the fused depth map and calculating the mean depth of effective pixels within the annular pixel band surrounding each remaining hole, the corresponding remaining holes are filled using this mean depth. This precisely eliminates the tiny "pinhole" artifacts easily left behind during the reconstruction process, ensuring the accuracy of local details.

[0025] By placing the most time-consuming iterative filling process in the coarse-scale space after downsampling, computational overhead is greatly reduced; while detail restoration at full resolution only targets a small number of remaining holes, resulting in low computational load. This hierarchical strategy enables the algorithm to achieve real-time frame rates on the CPU, with low deployment costs.

[0026] Furthermore, as a non-learning method based entirely on the principles of image processing and numerical computation, this method requires no training data and has universal and stable processing capabilities for different scenes and noise patterns, fundamentally avoiding the overfitting and generalization failure problems common in deep learning methods.

[0027] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a lidar depth completion method based on hierarchical minimum surface reconstruction provided by an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the depth value inversion and morphological dilation processing provided in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the depth value inversion and custom kernel expansion calculation process provided in an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the multi-scale Laplace diffusion process provided in an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the upsampling and depth fusion process provided in an embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram of the process for filling remaining voids provided in an embodiment of the present invention;

[0034] Figure 7 This is a schematic diagram of the blur denoising and depth value inversion process provided in an embodiment of the present invention;

[0035] Figure 8 This is a schematic diagram showing the qualitative comparison results between the method of this invention and Ip_Basic in three different indoor scenarios;

[0036] Figure 9 This is a schematic diagram comparing the diffusion process effects of the method of the present invention and the prior art;

[0037] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0039] To address the problem that existing LiDAR depth completion methods easily produce noticeable stair-step effects and blocky artifacts in the completed area when processing continuous surfaces with tilted angles, which are common in indoor scenes and severely compromise the smoothness and realism of the scene's geometry, thus failing to meet the requirements of high-precision 3D reconstruction, this invention provides a LiDAR depth completion method based on hierarchical minimum surface reconstruction. (See [link to relevant documentation]). Figure 1 , Figure 1 This is a flowchart illustrating a lidar depth completion method based on hierarchical minimum surface reconstruction provided by an embodiment of the present invention, specifically including the following steps:

[0040] Step S101: Perform depth value inversion and morphological dilation processing on the sparse depth map of the lidar system to obtain a scene dilated depth map and a full-resolution effective mask.

[0041] See Figure 2 , Figure 2 This is a schematic diagram of the depth value inversion and morphological dilation processing provided in the embodiments of the present invention. A sparse depth map is an image that contains distance information of the surface of objects in a scene, where each pixel value represents the distance of the corresponding scene point, and can be generated by depth acquisition devices such as LiDAR.

[0042] The resolution of the sparse depth map can be... .

[0043] In this embodiment of the invention, the sparse depth map of the lidar system is subjected to depth value inversion and morphological dilation processing to obtain a scene dilated depth map and a full-resolution effective mask, including:

[0044] The sparse depth map is inverted by using a predefined initial valid pixel mask to obtain the inverted sparse depth map.

[0045] Morphological dilation is performed on the inverted sparse depth map using a preset dilation kernel to obtain a scene dilated depth map;

[0046] Extract the full-resolution effective mask based on the scene dilation depth map.

[0047] In this embodiment of the invention, a depth value inversion is performed on the sparse depth map, causing nearby pixels (those with smaller depth values) to be converted to higher values, and distant pixels (those with larger depth values) to be converted to lower values. This conversion ensures that subsequent morphological operations preferentially propagate the depth information from nearby pixels. See also Figure 3 , Figure 3 This is a schematic diagram of the depth value inversion and custom kernel expansion calculation process provided in an embodiment of the present invention, assuming the maximum depth value obtained is... The value is 10, and the conversion formula is: , This is the original depth value. This is the inverted depth value. For example, a pixel with an original depth value of 2.0 has an inverted depth value of 8.0, and a pixel with an original depth value of 9.0 has an inverted depth value of 1.0.

[0048] In embodiments of the present invention, the depth values ​​can be based on a sparse depth map. The pixel position, predefine an initial valid pixel mask. Then, a depth value inversion operation is performed to obtain the inverted sparse depth map. :

[0049] ;

[0050] in, This indicates the maximum effective depth, and its value can be set by technicians according to their needs, such as 10.0m; Represents a sparse depth map;

[0051] The depth value inversion operation is performed only on valid pixels; see [link to documentation]. Figure 2 Invalid pixels, i.e., pixels with a value of 0, remain at 0. Valid pixels refer to pixels with a depth value greater than 0.1.

[0052] In this embodiment of the invention, a small preset dilation kernel is used to perform a morphological closing operation on the inverted sparse depth map, connecting adjacent valid pixels and filling tiny holes. This operation is called custom kernel dilation. Custom kernel dilation includes: sequentially performing at least one small kernel dilation operation and one closing operation on the inverted sparse depth map to obtain a scene dilated depth map, as detailed below:

[0053] Based on the inverted sparse depth map, a smaller full-core is first used, such as... After full-core inversion, the sparse depth map is dilated, spreading high-value pixels and expanding the effective area.

[0054] Then use the same The entire core performs a closing operation to fill smaller holes, thereby closing tiny gaps and smoothing boundaries to avoid excessive expansion, thus obtaining a scene expansion depth map.

[0055] Then, for pixels with a depth value that is not 0 in the scene dilation depth map, the value is set to 1, and for pixels with a depth value of 0, the value is set to 0, thus obtaining a full-resolution effective mask.

[0056] See Figure 3 The custom kernel expansion section allows for a custom kernel size that can also be set to [value]. The final filled result, i.e., the scene dilation depth map, is obtained by using a custom kernel to check the inverted depth of the sparse depth map after inversion.

[0057] Step S102: Downsample the scene dilation depth map and the full-resolution effective mask to obtain the coarse-scale scene dilation map and the coarse-scale mask. Then, use the iterative convolution kernel of the Laplacian operator to perform iterative filling operation to obtain the coarse-scale filled map.

[0058] In this embodiment of the invention, the process of downsampling the scene dilation depth map and the full-resolution effective mask to obtain the coarse-scale scene dilation map and the coarse-scale mask is as follows:

[0059] First, set the downsampling factor. For example, it can be 4 times. And calculate the coarse-scale image size, for example... Pixels. Then, the scene dilation depth map is calculated using nearest neighbor interpolation. and full-resolution effective mask Downsampling was performed separately to obtain coarse-scale scene inflation maps. and coarse-scale mask The nearest neighbor interpolation method is used to ensure that the mask boundaries are clear and the inflation value is not changed at a coarse scale.

[0060] See Figure 4 , Figure 4This is a schematic diagram of the multi-scale Laplacian diffusion process provided in an embodiment of the present invention. Iterative filling operations are performed using the iterative convolution kernel of the Laplacian operator to obtain a coarse-scale filled map, including:

[0061] Duplicate the coarse-scale scene dilation map to obtain the initial coarse-scale fill map;

[0062] Perform a Laplacian convolution on the initial coarse-scale filled map to obtain the average map;

[0063] Using an iterative update formula, the coarse-scale scene dilation map is filled based on the average value map and the coarse-scale mask to obtain an intermediate coarse-scale filled map;

[0064] The intermediate coarse-scale filled image is used as the initial coarse-scale filled image, and the process of performing a Laplacian convolution on the initial coarse-scale filled image to obtain the average value image is repeated until the preset number of iterations is reached, at which point the coarse-scale filled image is obtained.

[0065] Specifically, first, we define the iterative convolution kernel of the Laplacian operator. For example, the iterative convolution kernel of the Laplacian operator can be defined as a kernel of size [size missing]. The kernel has a center value of 0, four orthogonal neighbors (top, bottom, left, and right) with values ​​of 0.25, and all other corner points with values ​​of 0. This convolution kernel is used to calculate the average of the four neighbors of the center pixel. The number of coarse-scale iterations is set. For example, 50 times, and copy the coarse-scale scene inflation map. The initial coarse-scale filled map is obtained. This serves as the starting point for the iteration.

[0066] Then, the iteration loop begins. In each iteration, a Laplacian convolution is performed on the initial coarse-scale filled map to obtain the average map. .

[0067] Using an iterative update formula, the coarse-scale scene expansion map is filled based on the average value map and the coarse-scale mask to obtain the intermediate coarse-scale filled map.

[0068] The iterative update formula is as follows:

[0069] ;

[0070] ;

[0071] in, Represents an average value chart; This indicates a coarse-scale filled plot. This represents the iterative convolution kernel of the Laplacian operator; This represents a coarse-scale scene expansion map; This represents a coarse-scale mask.

[0072] In the iterative update formula, regions with valid values ​​in the coarse-scale scene dilation map are forced to retain their original values ​​to avoid being affected by the iteration process, while regions with invalid values ​​are updated to the average value of their neighborhood. After 50 iterations, The values ​​in the hollow region diffuse smoothly in from its boundary, forming a minimal surface interpolation, thus restoring the global structure.

[0073] Step S103: Upsample the coarse-scale fill map to obtain an upsampled depth map, and fuse the upsampled depth map with the scene dilation depth map to obtain a fused depth map.

[0074] See Figure 5 , Figure 5 This is a schematic diagram of the upsampling and depth fusion process provided in an embodiment of the present invention. Upsampling is performed on a coarse-scale filled map to obtain an upsampled depth map, including:

[0075] Bilinear interpolation is used to upsample the coarse-scale filled map back to its original resolution, resulting in an upsampled depth map. .

[0076] Here, the original resolution refers to the resolution of the sparse depth map, such as... .

[0077] In this embodiment of the invention, the upsampled depth map and the scene dilation depth map are fused to obtain a fused depth map, including:

[0078] ;

[0079] in, Represents the fused depth map; Represents the scene expansion depth map; Indicates the full-resolution effective mask; Represents the upsampled depth map; Indicates the reverse mask;

[0080] The upsampled result is then fused with the scene dilation depth map. This involves filling the hole areas in the scene dilation depth map with the upsampled depth map while preserving all original valid pixels (original valid values) from S101, resulting in a fused depth map. This fusion process ensures that all high-resolution original dilation information is preserved. The calculated low-resolution global structure, i.e., the upsampled depth map, is preserved without loss, and is only used to fill the large uncovered holes in S101. The indicated location.

[0081] Step S104: Locate all remaining holes in the fused depth map using connected component analysis, calculate the mean depth of the annular effective pixel band around each remaining hole, and fill the corresponding remaining hole with the mean depth to obtain a hole-free depth map.

[0082] Although the fused depth map obtained through step S103 restores the macroscopic structure, it may leave small residual holes at the edges of the original effective values ​​or in the interpolation regions. The following will explain how to clean up these residual holes; see [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the process for filling remaining voids provided in an embodiment of the present invention:

[0083] In this embodiment of the invention, all remaining holes in the fused depth map are located using connected component analysis, and the mean depth of effective pixels within the annular pixel band surrounding each remaining hole is calculated. This mean depth is then used to fill the corresponding remaining holes to obtain a hole-free depth map, including:

[0084] Locate and merge all remaining holes in the depth map;

[0085] Perform connected component analysis on all remaining holes and assign an integer label to each remaining hole;

[0086] Iterate through all integer labels and dilate the remaining holes with a binary mask of 1 using two morphological kernels of different sizes to obtain the corresponding first and second dilated regions.

[0087] The difference between the first and second dilated regions is calculated to obtain the annular pixel band.

[0088] Calculate the average depth of all valid pixels within the annular pixel band, and use this average depth to fill the remaining holes with a binary mask of 1, to obtain a hole-free depth map.

[0089] Specifically, first, all remaining holes are obtained, and then connected component analysis is performed on the remaining holes. Each independent hole is assigned a unique integer label.

[0090] Iterate through each integer label, obtain the binary mask of the current remaining hole, and use two morphological kernels of different sizes to expand the remaining hole with a binary mask of 1 to obtain the first expansion region and the second expansion region.

[0091] For example, using small-sized morphological kernels (such as...) The remaining voids in the whole core are expanded to obtain the first expansion region. .

[0092] Use large-sized morphological nuclei (such as...) The entire core expands again, resulting in a second expansion region. .

[0093] Calculate the difference between the first and second dilated regions. :

[0094] ;

[0095] This difference set is the ring-shaped pixel band surrounding the current remaining hole;

[0096] Calculate the average depth of all valid pixels within the annular pixel band, where valid pixels are those with a pixel value greater than 0;

[0097] Then, after filling the corresponding remaining holes using the average depth value, return to the step of traversing each integer label until all integer labels have been traversed, and obtain the hole-free depth map. .

[0098] Step S105: After applying a global Gaussian blur to the holeless depth map, perform a depth value inversion operation to obtain a dense scene depth map.

[0099] See Figure 7 , Figure 7 This is a flowchart of blur denoising and depth value inversion provided in an embodiment of the present invention. Global Gaussian blur is applied to the holeless depth map to smooth out the slight blockiness that may be caused by the upsampling operation and the edge abruptness that may be caused by the hole filling operation.

[0100] The effective mask is obtained by marking valid pixels as 1 and invalid pixels as 0 in the blurred hole-free depth map. .

[0101] Finally, a depth value inversion operation is performed to restore the depth values ​​to their original domain, where values ​​are smaller for closer objects and larger for farther objects. This results in a dense scene depth map that is structurally smooth and free of minor artifacts, effectively improving the robustness and visual quality of depth completion. This allows for high-precision 3D reconstruction based on the scene depth map. The scene dense depth map is as follows:

[0102] ;

[0103] in, Represents a dense depth map of the scene; This indicates the preset maximum effective depth value; This represents a depth map without holes.

[0104] In this embodiment of the invention, iterative convolution based on the Laplacian operator is used for diffusion filling instead of the traditional large kernel expansion operation. This significantly reduces the computational cost while restoring the smooth geometric structure of a large-area continuous surface, effectively avoiding "blocky" artifacts. All remaining holes in the fused depth map are located using connected component analysis, and the mean depth of the effective pixels within the annular pixel band surrounding each remaining hole is calculated. This mean depth is then used to fill the corresponding remaining holes, precisely eliminating the tiny "pinhole" artifacts that are easily left behind during the reconstruction process, ensuring the accuracy of local details.

[0105] In this invention, the most time-consuming iterative filling process is performed in the coarse-scale space after downsampling, greatly reducing computational overhead; while detail restoration at full resolution only targets a small number of remaining holes, resulting in low computational load. This hierarchical strategy enables the algorithm to achieve real-time processing frame rates on a CPU (Central Processing Unit) with low deployment costs.

[0106] Furthermore, as a non-learning method based entirely on the principles of image processing and numerical computation, this invention requires no training data and has universal and stable processing capabilities for different scenarios and noise patterns, fundamentally avoiding the overfitting and generalization failure problems common in deep learning methods.

[0107] The simulation experiment of a lidar depth completion method based on hierarchical minimum surface reconstruction provided by the embodiments of the present invention is as follows:

[0108] In the simulation experiment, an AMD EPYC 7542 processor was selected as the hardware platform, and the NYU Depth v indoor depth dataset was used as the test data. The following preprocessing steps were performed to construct the standard input: First, the sparse depth map of the LiDAR system was scaled to... Resolution; then, a center cropping operation is performed to obtain... Finally, 500 pixels are randomly sampled from the image to serve as sparse depth input, simulating extremely sparse LiDAR scanning data.

[0109] For the evaluation metrics, the experiment used root mean square error (RMSE) and mean absolute error (MAE) to quantify the depth map completion accuracy, and floating-point operations (FLOPs) and single-frame inference time (ms) to evaluate the algorithm's running efficiency. To eliminate the influence of randomness on the experimental results and ensure the reliability of the data, all listed experimental metrics are the average of five independent repeated experiments. See Table 1, which compares the results of this invention with those of traditional methods.

[0110] Table 1. Comparison of results between the present invention and traditional methods

[0111]

[0112] A detailed analysis of the experimental data in Table 1 reveals that the LiDAR depth completion method based on hierarchical minimum surface reconstruction provided by this invention exhibits a breakthrough performance advantage in image reconstruction quality. Specifically, the RMSE index of the LiDAR depth completion method based on hierarchical minimum surface reconstruction provided by this invention is reduced to 172.5 mm, and the MAE index is reduced to 71.8 mm. Compared with the best-performing traditional method (IP-Basic) in the prior art, this invention improves the accuracy of depth completion to more than 1.7 times that of the prior art while maintaining extremely low error. This significant performance leap is mainly due to the hierarchical strategy of this invention: the global geometric structure of the scene is effectively restored through coarse-scale minimum surface reconstruction, avoiding the structural distortion caused by traditional large kernel expansion; and with residual hole filling, tiny holes are accurately repaired, thereby significantly reducing the overall error while preserving more realistic edge details.

[0113] Regarding algorithm efficiency, although the theoretical computational load and single-frame inference time of this invention are slightly increased compared to the benchmark method, the computing power of mainstream edge hardware platforms, such as the ARM Cortex-A78AE, can reach 280 GFLOPs. This slight increase in computational load is negligible. The measured inference speed of 7.526ms is far lower than the 33ms time threshold of conventional real-time processing and will not become a bottleneck for the system. Therefore, this invention achieves a near-doubling improvement in completion accuracy at a very low computational cost that does not affect real-time performance (only an increase of about 1ms in time). This fully demonstrates that this invention has extremely high robustness and cost-effectiveness in practical engineering applications and can fully meet the performance requirements of modern computing platforms.

[0114] To visually demonstrate the depth completion capabilities of this invention in complex indoor scenes Figure 8 This is a schematic diagram showing the qualitative comparison results between the method of this invention and Ip_Basic in three different indoor scenarios. The comparison diagram includes the original truth map, the Ip_Basic completion result, the completion result of this invention (Ours), and the absolute difference map between each and the truth.

[0115] From the perspective of surface smoothness reconstruction, the Ip_Basic method, when processing large continuous surfaces such as walls and floors, exhibits significant stair-step effects and blocky artifacts on planes with tilt angles due to its algorithm's assumption of a local constant depth, thus disrupting the geometric continuity of the scene. In contrast, the depth map generated by the method of this invention demonstrates excellent smoothness in these areas, with depth values ​​showing a natural gradient transition. This proves that the minimum surface reconstruction strategy based on the Laplacian operator used in this invention can effectively adapt to the physical characteristics of linear changes in depth with spatial distance in indoor scenes, successfully eliminating blocky artifacts.

[0116] In terms of restoring the details of object edges, this invention is also superior to existing technologies. See also Figure 8 In traditional Chinese programming, object outlines in Ip_Basic often exhibit blurring or jagged edges when filling in the edges. This invention, however, uses residual hole filling and morphological optimization to fill tiny holes while more accurately maintaining a clear boundary between the object and the background, without introducing excessive smoothing or structural distortion.

[0117] Furthermore, observation of the difference maps reveals a large number of gray-white noise points in the Ip_Basic difference map, indicating a widespread depth error globally. In contrast, the difference map corresponding to this invention is predominantly black, with only slight errors in a very few areas with extremely complex textures. This visual result aligns with the significant decrease in RMSE and MAE error indices in Table 1, intuitively confirming that the scene-dense depth map generated by this invention significantly outperforms existing mainstream methods in both structural integrity and numerical accuracy.

[0118] Those skilled in the art should understand that the parameters in the above embodiments, such as the sampling factor, the number of coarse-scale iterations, and the kernel size for filling remaining holes, are preferred embodiments of the present invention and not limitations thereof. These parameters can be modified or replaced without departing from the spirit and principle of the present invention to adapt to different application requirements. For example, iterative solutions can be replaced with other numerical methods for solving the Poisson equation.

[0119] This invention provides a lidar depth completion method based on hierarchical minimum surface reconstruction, employing a hybrid strategy. It introduces a hierarchical minimum surface reconstruction mechanism based on downsampling space to address the performance bottlenecks and structural distortion issues of traditional methods. For example... Figure 9 As shown, Figure 9This diagram illustrates a comparison of the diffusion process effects of the method of this invention and existing technologies. This mechanism mathematically simulates the physical thermal diffusion process, allowing depth values ​​to propagate naturally within the cavity region according to the smoothness constraints defined by the Laplace operator. This partial differential equation-based filling method essentially constructs a minimal surface that satisfies the boundary conditions, thus geometrically accurately fitting common sloping walls or extended floors in indoor scenes. This effectively eliminates the step-like artifacts caused by the local constant assumptions in traditional morphological large-kernel expansion methods. Furthermore, by employing a multi-scale strategy, the most computationally intensive iterative solution process is placed in a low-dimensional space, resulting in a geometric reduction in the amount of data to be processed. This significantly reduces the number of floating-point operations, ensuring that the algorithm maintains high-precision smooth surface reconstruction while achieving the operational efficiency required for real-time CPU applications.

[0120] Based on the same inventive concept, this invention also provides a lidar depth completion device based on hierarchical minimum surface reconstruction, the lidar depth completion device comprising:

[0121] The processing module is used to perform depth value inversion and morphological dilation processing on the sparse depth map of the lidar system to obtain a scene dilated depth map and a full-resolution effective mask.

[0122] The first filling module is used to downsample the scene dilation depth map and the full-resolution effective mask respectively to obtain the coarse-scale scene dilation map and the coarse-scale mask. Then, iterative filling operation is performed using the iterative convolution kernel of the Laplacian operator to obtain the coarse-scale filled map.

[0123] The fusion module is used to upsample the coarse-scale fill map to obtain an upsampled depth map, and then fuse the upsampled depth map with the scene dilation depth map to obtain a fused depth map.

[0124] The second filling module is used to locate all remaining holes in the fused depth map using connected component analysis, and to calculate the average depth of the effective pixels within the annular pixel band around each remaining hole. The average depth is then used to fill the corresponding remaining holes to obtain a hole-free depth map.

[0125] The inversion module is used to perform a depth value inversion operation on a holeless depth map after applying a global Gaussian blur to obtain a dense depth map of the scene.

[0126] In this embodiment of the invention, iterative convolution based on the Laplacian operator is used for diffusion filling instead of the traditional large kernel expansion operation. This significantly reduces the computational cost while restoring the smooth geometric structure of a large-area continuous surface, effectively avoiding "blocky" artifacts. All remaining holes in the fused depth map are located using connected component analysis, and the mean depth of the effective pixels within the annular pixel band surrounding each remaining hole is calculated. This mean depth is then used to fill the corresponding remaining holes, precisely eliminating the tiny "pinhole" artifacts that are easily left behind during the reconstruction process, ensuring the accuracy of local details.

[0127] By placing the most time-consuming iterative filling process in the coarse-scale space after downsampling, computational overhead is greatly reduced; while detail restoration at full resolution only targets a small number of remaining holes, resulting in low computational load. This hierarchical strategy enables the algorithm to achieve real-time frame rates on the CPU, with low deployment costs.

[0128] Furthermore, as a non-learning method based entirely on the principles of image processing and numerical computation, this method requires no training data and has universal and stable processing capabilities for different scenes and noise patterns, fundamentally avoiding the overfitting and generalization failure problems common in deep learning methods.

[0129] This invention also provides an electronic device, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, including a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0130] Memory 1003 is used to store computer programs;

[0131] When the processor 1001 executes the program stored in the memory 1003, it implements the method steps of any of the above-mentioned lidar depth completion methods based on hierarchical minimum surface reconstruction.

[0132] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0133] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0134] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0135] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0136] The present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program implements the method steps of any of the above-described lidar depth completion methods based on hierarchical minimum surface reconstruction.

[0137] Optionally, the computer-readable storage medium may be non-volatile memory (NVM), such as at least one disk storage device.

[0138] Optionally, the aforementioned computer-readable storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0139] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the method steps of any of the above-described lidar depth completion methods based on hierarchical minimum surface reconstruction.

[0140] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0141] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0142] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0143] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0144] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0145] It should be noted that the device, electronic device and storage medium in the embodiments of the present invention are respectively the device, electronic device and storage medium for applying the above-mentioned method for lidar depth completion based on hierarchical minimum surface reconstruction. Therefore, all embodiments of the above-mentioned method for lidar depth completion based on hierarchical minimum surface reconstruction are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0146] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A lidar depth completion method based on hierarchical minimum surface reconstruction, characterized in that, The laser radar depth completion method includes: The sparse depth map of the lidar system is subjected to depth value inversion and morphological dilation to obtain a scene dilated depth map and a full-resolution effective mask. After downsampling the scene dilation depth map and the full-resolution effective mask to obtain the coarse-scale scene dilation map and the coarse-scale mask, iterative filling operation is performed using the iterative convolution kernel of the Laplacian operator to obtain the coarse-scale filled map. The coarse-scale fill map is upsampled to obtain an upsampled depth map, and the upsampled depth map is fused with the scene dilation depth map to obtain a fused depth map; All remaining holes in the fused depth map are located by connected component analysis, and the mean depth of the effective pixels within the annular pixel band around each remaining hole is calculated. The mean depth is then used to fill the corresponding remaining holes to obtain a hole-free depth map. After applying a global Gaussian blur to the hole-free depth map, a depth value inversion operation is performed to obtain a scene dense depth map. The iterative filling operation using the Laplacian operator's iterative convolution kernel to obtain a coarse-scale filled map includes: Copy the coarse-scale scene dilation map to obtain the initial coarse-scale fill map; Perform a Laplacian convolution on the initial coarse-scale filled map to obtain the average map; Using an iterative update formula, the coarse-scale scene expansion map is filled based on the average value map and the coarse-scale mask to obtain an intermediate coarse-scale filled map; The intermediate coarse-scale filled image is used as the initial coarse-scale filled image, and the process of performing a Laplacian convolution on the initial coarse-scale filled image to obtain the average value image is repeated until the preset number of iterations is reached to obtain the coarse-scale filled image. The process involves locating all remaining holes in the fused depth map using connected component analysis, calculating the average depth of effective pixels within the annular pixel band surrounding each remaining hole, and filling the corresponding remaining holes with this average depth to obtain a hole-free depth map. This includes: Locate all remaining holes in the fused depth map; Perform connected component analysis on all remaining holes and assign an integer label to each remaining hole; Iterate through all integer labels and dilate the remaining holes with a binary mask of 1 using two morphological kernels of different sizes to obtain the corresponding first and second dilated regions. The difference between the first dilated region and the second dilated region is calculated to obtain the annular pixel band; Calculate the average depth of all valid pixels within the annular pixel band, and use this average depth to fill the remaining holes with a binary mask of 1, to obtain a hole-free depth map.

2. The lidar depth completion method according to claim 1, characterized in that, The sparse depth map of the LiDAR system is subjected to depth value inversion and morphological dilation processing to obtain a scene dilated depth map and a full-resolution effective mask, including: The sparse depth map is inverted by using a predefined initial valid pixel mask to obtain the inverted sparse depth map. The inverted sparse depth map is subjected to morphological dilation using a preset dilation kernel to obtain a scene dilated depth map; Extract the full-resolution effective mask based on the scene dilation depth map.

3. The lidar depth completion method according to claim 1, characterized in that, The iterative update formula is: ； ； in, This represents the average value graph; This represents the intermediate coarse-scale filled map; This represents the iterative convolution kernel of the Laplacian operator; This represents the coarse-scale scene expansion map; This represents the coarse-scale mask.

4. The lidar depth completion method according to claim 1, characterized in that, Upsampling the coarse-scale filled map to obtain an upsampled depth map includes: The coarse-scale filled map is upsampled back to its original resolution using bilinear interpolation to obtain an upsampled depth map.

5. The lidar depth completion method according to claim 1, characterized in that, The upsampled depth map is fused with the scene dilation depth map to obtain a fused depth map, including: ； in, This represents the fused depth map; This represents the scene's dilation depth map; This represents the full-resolution effective mask; This represents the upsampled depth map.

6. A lidar depth completion device based on hierarchical minimum surface reconstruction, characterized in that, The lidar depth completion device includes: The processing module is used to perform depth value inversion and morphological dilation processing on the sparse depth map of the lidar system to obtain a scene dilated depth map and a full-resolution effective mask. The first filling module is used to downsample the scene dilation depth map and the full-resolution effective mask respectively to obtain a coarse-scale scene dilation map and a coarse-scale mask. Then, iterative filling operation is performed using the iterative convolution kernel of the Laplacian operator to obtain a coarse-scale filled map. The fusion module is used to upsample the coarse-scale fill map to obtain an upsampled depth map, and then fuse the upsampled depth map with the scene dilation depth map to obtain a fused depth map. The second filling module is used to locate all remaining holes in the fused depth map using connected component analysis, calculate the average depth of effective pixels within the annular pixel band around each remaining hole, and fill the corresponding remaining holes using the average depth to obtain a hole-free depth map. The inversion module is used to perform a depth value inversion operation on the hole-free depth map after global Gaussian blurring to obtain a scene dense depth map; The first filling module performs iterative filling operations using the iterative convolution kernel of the Laplacian operator to obtain a coarse-scale filled map, including: Copy the coarse-scale scene dilation map to obtain the initial coarse-scale fill map; Perform a Laplacian convolution on the initial coarse-scale filled map to obtain the average map; Using an iterative update formula, the coarse-scale scene expansion map is filled based on the average value map and the coarse-scale mask to obtain an intermediate coarse-scale filled map; The intermediate coarse-scale filled image is used as the initial coarse-scale filled image, and the process of performing a Laplacian convolution on the initial coarse-scale filled image to obtain the average value image is repeated until the preset number of iterations is reached to obtain the coarse-scale filled image. The second filling module is specifically used to locate all remaining holes in the fused depth map; perform connected component analysis on all remaining holes and assign an integer label to each remaining hole; traverse all integer labels and dilate the remaining holes with a binary mask of 1 using two morphological kernels of different sizes to obtain the corresponding first dilated region and second dilated region; calculate the difference between the first dilated region and the second dilated region to obtain an annular pixel band; calculate the depth mean of all effective pixels in the annular pixel band and use the depth mean to fill the remaining holes with a binary mask of 1 to obtain a hole-free depth map.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a computer program stored in memory, it implements the lidar depth completion method based on hierarchical minimum surface reconstruction as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the lidar depth completion method based on hierarchical minimum surface reconstruction as described in any one of claims 1-5.

Images