Image-guided point cloud compensation method and system for failure region of lidar

Abstract

The application discloses a laser radar failure area-oriented image-guided point cloud compensation method and system, relates to the related technical field of image processing, and comprises the following steps: collecting original laser point clouds and RGB image sequences of the surface of a target coal pile through a laser radar and an RGB camera; obtaining a despeckling image through floating particle attenuation preprocessing and outputting a failure area mask; performing geometric ambiguity correction to obtain a geometric consistent depth map; generating a guide point cloud through a dynamic calibration matrix projection; performing spatiotemporal consistency fusion and outputting a compensation point cloud; and outputting a three-dimensional reconstruction model after local surface optimization and continuous surface model output. The application solves the technical problems that, in the prior art, when the laser radar is locally invalid on the surface of a coal pile, the point cloud data has holes and outliers, and visual depth estimation is affected by geometric ambiguity and dust scattering, and it is difficult to realize high-precision continuous surface reconstruction, and achieves the technical effects of improving the integrity and precision of coal pile three-dimensional reconstruction in a laser radar failure scenario.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image-guided point cloud compensation method and system for lidar failure areas. Background Technology

[0002] In bulk material management scenarios such as mines, ports, and power plants, changes in the volume and shape of coal piles are key to inventory management, production scheduling, and trade settlement. Traditional manual measurement suffers from low efficiency, poor accuracy, and high safety risks. LiDAR-based 3D reconstruction can directly acquire high-precision 3D point cloud data of coal pile surfaces, unaffected by ambient lighting. However, the complex and harsh environment at coal pile sites presents numerous challenges for LiDAR in practical applications. Coal pile surfaces often have areas obscured by water, mud, or dust. Water surfaces cause specular reflection of the laser beam, while dust particles cause beam scattering and attenuation, preventing LiDAR from obtaining effective echoes in these areas. This results in point cloud voids or outliers, severely impacting the integrity and accuracy of 3D reconstruction. While RGB cameras provide rich texture information, they are susceptible to weak textures, shadows, and dust scattering on the coal pile surface, leading to significant uncertainty in depth estimation and making direct use for high-precision geometric compensation difficult. High concentrations of suspended dust particles are commonly found in coal pile work areas, forming pseudo-points in the LiDAR point cloud and further degrading the reliability of visual depth estimation. Furthermore, the complex surface morphology of coal piles, with irregular features such as collapses, sharp edges, and abrupt slope changes, makes it difficult for current methods to balance local details with global efficiency, resulting in large errors in volume calculation.

[0003] Therefore, in the current related technologies, there are technical problems such as the presence of voids and outliers in the point cloud data when the lidar fails locally on the surface of the coal pile, and the visual depth estimation being affected by geometric ambiguity and dust scattering, making it difficult to achieve high-precision continuous surface reconstruction. Summary of the Invention

[0004] This application provides an image-guided point cloud compensation method and system for lidar failure areas, which solves the technical problems in the prior art where point cloud data has holes and outliers when lidar fails locally on the surface of a coal pile, and visual depth estimation is affected by geometric ambiguity and dust scattering, making it difficult to achieve high-precision continuous surface reconstruction. It achieves the technical effect of improving the integrity and accuracy of three-dimensional reconstruction of coal piles in lidar failure scenarios.

[0005] This application provides an image-guided point cloud compensation method for lidar failure areas. The method includes: using hardware-triggered pulses to drive a lidar and an RGB camera to perform spatiotemporally synchronized acquisition of original lidar point clouds and RGB image sequences on the surface of a target coal pile; performing suspended particle attenuation preprocessing on the RGB image sequences to obtain descattered images, and then performing failure area determination based on multimodal confidence fusion in conjunction with the original lidar point clouds to output a failure area mask; performing geometric ambiguity correction on the descattered images to obtain a geometrically consistent depth map; projecting the geometrically consistent depth map onto the lidar coordinate system using a dynamic calibration matrix to generate a guide point cloud; using the topological boundary of the failure area mask as a constraint, performing spatiotemporal consistency fusion based on adaptive confidence weights on the original lidar point cloud and the guide point cloud to output a compensated point cloud; performing local surface optimization based on moving least squares smoothing and local Poisson reconstruction on the compensated point cloud to output a continuous surface model, and then performing a feedback calibration closed loop based on volume calculation errors to output a three-dimensional reconstruction model.

[0006] In a possible implementation, the original laser point cloud and the guiding point cloud are fused based on adaptive confidence weights using the topological boundary of the failure region mask as a constraint to output a compensated point cloud. The method includes: performing morphological boundary tracking on the failure region mask to obtain a set of boundary pixel contours, and then performing polygon fitting to obtain a closed topological boundary representation; based on the joint calibration matrix of the intrinsic and extrinsic parameters of the lidar and RGB camera, registering the guiding point cloud to the lidar coordinate system through rigid body transformation, and then synchronizing the original laser point cloud and the guiding point cloud according to the hardware triggering timing of the lidar and RGB camera to generate a registered geometrically completed point cloud; constructing an adaptive confidence weight function based on the closed topological boundary representation and the depth information of the registered geometrically completed point cloud; and, within the Bayesian maximum a posteriori probability estimation framework, using the adaptive confidence weight function as the inverse variance of the observation noise, performing optimal weighted fusion of the original laser point cloud and the registered geometrically completed point cloud under uncertainty propagation to output the compensated point cloud.

[0007] In a possible implementation, an adaptive confidence weight function is constructed based on the closed topological boundary representation and the depth information of the registered geometrically completed point cloud. The method includes: using the closed topological boundary representation as a two-dimensional projection reference, performing back projection calculations in conjunction with the depth information of the registered geometrically completed point cloud to construct a three-dimensional boundary surface model in the lidar coordinate system; calculating the spatial oriented distance field of the three-dimensional boundary surface model, dividing the three-dimensional spatial domain covered by the original lidar point cloud into a certainty valid region, a certainty invalid region, and a boundary transition region; and constructing the adaptive confidence weight function based on the regional division of the certainty valid region, the certainty invalid region, and the boundary transition region.

[0008] In a possible implementation, the compensated point cloud is optimized using local surface optimization based on moving least squares smoothing and local Poisson reconstruction. After outputting a continuous surface model, a feedback calibration closed loop is performed based on volume calculation error to output a three-dimensional reconstruction model. The method includes: fitting the compensated point cloud to a local surface based on adaptive neighborhood radius selection using moving least squares; projecting the compensated point cloud onto the local surface for smoothing to output a smoothed point cloud; calculating a uniform normal vector field of the smoothed point cloud on the local surface; extracting the spatial position and normal vector of multiple points from the smoothed point cloud and the uniform normal vector field, constructing a three-dimensional vector contribution point by point through kernel function mapping, and accumulating the contributions to synthesize a global indicator function gradient field; solving the Poisson equation with the global indicator function gradient field as a constraint to obtain a three-dimensional spatial scalar indicator function, extracting the isosurface of the three-dimensional spatial scalar indicator function to obtain the continuous surface model; and performing a feedback calibration closed loop based on volume calculation error on the continuous surface model to output the three-dimensional reconstruction model.

[0009] In a possible implementation, a feedback calibration closed loop based on volume calculation error is performed on the continuous surface model to output the three-dimensional reconstruction model. The method includes: calculating the geometric volume of the continuous surface model as the model volume value, comparing it with the calibration volume value, and calculating the relative volume error; predefining multiple key parameters for local surface optimization of the continuous surface model, and constructing a parameterized response model; performing model volume error comparison and iteration under the parameterized response model with the multiple key parameters adjusted until the volume error meets the preset accuracy requirements, and outputting the three-dimensional reconstruction model.

[0010] In a possible implementation, the image-guided point cloud compensation method for lidar failure areas further includes: constructing a dual-task network model based on end-to-end training, wherein the dual-task network model includes a shared feature encoding layer, a depth estimation branch, and a normal prediction branch, and the output of the shared feature encoding layer is connected in parallel to the depth estimation branch and the normal prediction branch.

[0011] In a possible implementation, geometric ambiguity correction is performed on the descattered image to obtain a geometrically consistent depth map. The method includes: inputting the descattered image into the shared feature encoding layer of the dual-task network model for multi-level convolutional feature extraction to obtain a shared high-level semantic feature map; then, performing monocular depth regression and surface normal regression in parallel on the shared high-level semantic feature map in the depth estimation branch and the normal prediction branch to obtain an initial depth map and a surface normal map; establishing a differential geometric constraint relationship between the surface normal map and the initial depth map based on the pixel-level geometric orientation information of the surface normal map; constructing a normal consistency optimization function using the differential geometric constraint relationship; performing iterative geometric correction on the initial depth map based on a numerical optimization algorithm; and outputting the geometrically consistent depth map; wherein the geometrically consistent depth map and the surface normal map satisfy the differential geometric constraint relationship.

[0012] In a possible implementation, the image-guided point cloud compensation method for lidar failure areas further includes: performing time-domain weighted mean calculation on the RGB image sequence with multi-frame pixel alignment to obtain a dust transient suppression image; and performing physical separation of specular reflection and diffuse reflection components on the dust transient suppression image through polarization difference analysis to obtain the descattering image.

[0013] In a possible implementation, the image-guided point cloud compensation method for lidar failure areas further includes: after extracting point cloud-level local density distribution features and echo intensity attenuation features from the original lidar point cloud, performing point cloud physical failure determination by combining dynamic dual thresholds to generate a point cloud failure probability map; injecting the descattering image into the specular reflection feature extraction branch and dust occlusion feature extraction branch of a pre-constructed dual-channel semantic segmentation model, performing parallel segmentation of water surface specular reflection area and suspended particle coverage area, obtaining water surface area probability map and dust area probability map, and then performing pixel-level probability fusion under channel weighted constraints in the feature fusion layer to output the failure area mask.

[0014] This application also provides an image-guided point cloud compensation system for lidar failure areas. The system includes: a point cloud image acquisition module, used to drive the lidar and RGB camera to perform spatiotemporally synchronous acquisition of original lidar point clouds and RGB image sequences on the surface of a target coal pile via hardware-triggered pulses; an image sequence processing module, used to perform suspended particle attenuation preprocessing on the RGB image sequence to obtain a descattered image, and then perform failure area determination based on multimodal confidence fusion in conjunction with the original lidar point cloud, outputting a failure area mask; and an image ambiguity correction module, used to perform geometric ambiguity correction on the descattered image to obtain several... The system comprises: a geometrically consistent depth map; a guiding point cloud generation module, used to project the geometrically consistent depth map onto the lidar coordinate system through a dynamic calibration matrix to generate a guiding point cloud; a compensation point cloud generation module, used to perform spatiotemporal consistency fusion of the original lidar point cloud and the guiding point cloud based on adaptive confidence weights, using the topological boundary of the failure region mask as a constraint condition, to output a compensation point cloud; and a 3D reconstruction model output module, used to perform local surface optimization of the compensation point cloud based on moving least squares smoothing and local Poisson reconstruction, output a continuous surface model, and then perform feedback calibration closed loop based on volume calculation error to output a 3D reconstruction model.

[0015] This application proposes an image-guided point cloud compensation method and system for lidar failure areas. The method involves acquiring original lidar point clouds and RGB image sequences from the surface of a target coal pile using lidar and an RGB camera; preprocessing with suspended particles to obtain a descattered image and performing failure area determination, outputting a failure area mask; correcting geometric ambiguity to obtain a geometrically consistent depth map; generating a guiding point cloud through dynamic calibration matrix projection; performing spatiotemporal consistency fusion to output a compensated point cloud; and optimizing local surfaces to output a continuous surface model, followed by a 3D reconstruction model. This method solves the technical problems in existing technologies where, when lidar fails locally on the coal pile surface, point cloud data contains voids and outliers, and visual depth estimation is affected by geometric ambiguity and dust scattering, making high-precision continuous surface reconstruction difficult. It achieves the technical effect of improving the completeness and accuracy of 3D reconstruction of coal piles in lidar failure scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments of this disclosure will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1This is a schematic flowchart of an image-guided point cloud compensation method for lidar failure areas provided in an embodiment of this application.

[0018] Figure 2 A schematic diagram of the structure of an image-guided point cloud compensation system for lidar failure areas provided in an embodiment of this application.

[0019] Figure labeling: Point cloud image acquisition module 10, image sequence processing module 20, image ambiguity correction module 30, guided point cloud generation module 40, compensated point cloud generation module 50, 3D reconstruction model output module 60. Detailed Implementation

[0020] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on the specific implementation methods, structures, features and effects of the present invention.

[0021] This application provides an image-guided point cloud compensation method for lidar failure areas, such as... Figure 1 As shown, the method includes: Step S100: The original laser point cloud and RGB image sequence are spatiotemporally acquired on the surface of the target coal pile by hardware triggering pulse drive lidar and RGB camera.

[0022] Preferably, the hardware trigger pulse refers to using a hardware signal generator, such as a microcontroller, FPGA, or dedicated clock circuit, to send electrical pulse signals at fixed time intervals. This pulse signal serves as a synchronous trigger source and is connected to the control interfaces of the LiDAR and RGB camera. After receiving the hardware trigger pulse, the LiDAR and RGB camera immediately initiate a data acquisition action upon the arrival of each pulse, no longer relying on their respective internal independent clocks for software time synchronization. Since the LiDAR and RGB camera are triggered by the same pulse signal, they both begin acquiring a frame of data at the same time, ensuring that each frame of laser point cloud and each frame of RGB image are strictly aligned in time. Through the rigid body transformation matrix between the LiDAR coordinate system and the camera coordinate system obtained through pre-calibration, the acquired point cloud and image pixels correspond in spatial geometry. The LiDAR emits a laser beam and receives the echo, measuring the three-dimensional coordinates (X, Y, Z) of each point and the echo intensity value to obtain the original laser point cloud. The RGB camera simultaneously captures color images of the coal pile surface, with each pixel containing the brightness values ​​of the red, green, and blue channels, thus determining the RGB image sequence.

[0023] Step S200: Perform suspended particle attenuation preprocessing on the RGB image sequence to obtain a descattered image, and then perform failure region determination based on multimodal confidence fusion in conjunction with the original laser point cloud to output a failure region mask.

[0024] Step S200 further includes: performing time-domain weighted mean calculation on the RGB image sequence with multi-frame pixel alignment to obtain a dust transient suppression image; and performing physical separation of specular reflection and diffuse reflection components on the dust transient suppression image through polarization difference analysis to obtain the descattering image.

[0025] Preferably, image processing is performed to address the optical attenuation caused by airborne dust particles in the RGB image sequence. Specifically, multiple frames are continuously extracted from the RGB image sequence. Due to potential minor camera shake or slight displacement of the coal pile surface, image registration methods such as optical flow or phase correlation are used to remap the pixel coordinates of each frame to a common reference coordinate system, ensuring that pixels representing the same physical point in different frames fall at the same image coordinate position. For pixels at the same coordinate position in each aligned frame, their brightness values ​​(e.g., grayscale values ​​of the red channel) on the time axis are weighted and averaged. The weighting coefficient is determined based on the time distance between the frame and the current moment, with frames closer in time receiving higher weights and frames farther in time receiving lower weights. The calculated weighted average replaces the original value of the pixel. After pixel-by-pixel processing, a dust transient suppression image is obtained. Since dust particles move rapidly in the air and appear randomly in different frames, their weighted average contribution is low and effectively suppressed. Meanwhile, the static or slowly moving coal pile surface is preserved after weighted averaging because its brightness value is stable in each frame.

[0026] Preferably, the transient suppression image of dust is physically separated by polarization difference analysis. This involves utilizing the difference in polarization characteristics between specular and diffuse light. Specular light has strong polarization, while diffuse light is approximately unpolarized. Specifically, a rotatable linear polarizer is installed in front of the RGB camera lens, and an image is acquired at two mutually orthogonal polarization angles (e.g., 0° and 90°). The difference between the two images (0° polarization image minus 90° polarization image) is calculated to obtain the specular reflection component image. In this image, the specular reflection area exhibits a high brightness value, while the diffuse reflection area has a brightness value close to zero. The specular reflection component is subtracted from the original image according to the physical model to obtain the diffuse reflection component image. The diffuse reflection component image is then extracted as a descattering image. In this image, specular highlights generated by the water surface or wet, slippery coal pile surface are removed, while the diffuse reflection information reflecting the true color and texture of the coal pile surface is retained. At the same time, the scattered light caused by dust is further suppressed.

[0027] Furthermore, step S200 also includes, after extracting point cloud-level local density distribution features and echo intensity attenuation features from the original laser point cloud, performing point cloud physical failure determination by combining dynamic dual thresholds to generate a point cloud failure probability map; injecting the descattering image into the specular reflection feature extraction branch and dust occlusion feature extraction branch of the pre-constructed dual-channel semantic segmentation model, performing water surface specular reflection region segmentation and suspended particle coverage region segmentation in parallel, obtaining water surface region probability map and dust region probability map, and then performing pixel-level probability fusion under channel weighted constraints in the feature fusion layer to output the failure region mask.

[0028] Preferably, for each point in the original laser point cloud, a three-dimensional spherical neighborhood is defined centered on that point. The number of points contained within this neighborhood is counted, and the local point cloud density is calculated. The local density value of each point is used as the point cloud-level local density distribution feature. Due to missing or scattering echoes, the failure area of ​​the lidar usually exhibits a significantly lower local point cloud density than the normal coal pile surface. Then, the echo intensity value carried by each laser point is read. For each point, its echo intensity is compared with the standard intensity value calibrated by the lidar manufacturer (for targets with the same distance and reflectivity), and the attenuation is calculated to obtain the echo intensity attenuation feature. Water surface absorption and specular reflection of the laser, as well as dust scattering of the laser, can all lead to abnormal attenuation of the echo intensity. High and low thresholds are set for point cloud density and echo intensity, respectively. For the local density or echo intensity of each point, if it is lower than the low threshold, it is determined to be a confirmed failure; if it is between the low and high thresholds, it is determined to be a suspected failure; if it is higher than the high threshold, it is determined to be valid. The high and low thresholds are dynamically calculated based on the global statistical distribution of the point cloud in the current frame, for example, taking multiples of the mean or percentiles. Then, the 3D point cloud space is projected onto the 2D image plane using camera calibration parameters. The failure probability of points in the corresponding area is recorded at each pixel position. The probability value is determined by the ratio of the number of failed points in the area to the total number of points, combined with the confidence weighting of dynamic double thresholds. The point cloud failure probability map is output, and each pixel value represents the possibility of LiDAR failure at that position.

[0029] Preferably, a pre-built dual-channel semantic segmentation model based on deep learning is used, comprising two parallel feature extraction branches: a specular reflection feature extraction branch for identifying specular reflection regions in the image, and a dust occlusion feature extraction branch for identifying regions in the image where textures are blurred due to dust coverage. Then, water surface specular reflection region segmentation and suspended particle coverage region segmentation are performed in parallel. The descattered image is input into the specular reflection feature extraction branch, which outputs a water surface region probability map, where each pixel value represents the probability that the location belongs to a water surface specular reflection region. The descattered image is input into the dust occlusion feature extraction branch, which outputs a dust region probability map. Each pixel value represents the probability that the location belongs to a dust-covered area. Then, in the feature fusion layer of the dual-channel semantic segmentation model, learnable weight coefficients are assigned to the water surface probability map and the dust probability map respectively. The weight coefficients are automatically determined through network training and reflect the relative importance of the two failure types in the current scene. For each pixel location in the image, the failure probability after fusion is calculated by weighting. A threshold (e.g., 0.5) is set for the fused probability map. Pixels with probability values ​​greater than the threshold are marked as 1 (failure area), and pixels with probability values ​​less than or equal to the threshold are marked as 0 (effective area), thereby determining the failure area mask.

[0030] Step S300: Perform geometric ambiguity correction on the descattered image to obtain a geometrically consistent depth map.

[0031] Step S300 further includes constructing a dual-task network model based on end-to-end training, wherein the dual-task network model includes a shared feature encoding layer, a depth estimation branch, and a normal prediction branch, and the output of the shared feature encoding layer is connected in parallel to the depth estimation branch and the normal prediction branch.

[0032] Preferably, a dual-task network model is constructed based on end-to-end training to simultaneously estimate depth and surface normals from a single RGB image. Specifically, the descattered image input to the network and the final output depth map and normal map are considered as a complete differentiable computation process. During training, the prediction errors of the depth estimation branch and the prediction errors of the normal prediction branch are backpropagated simultaneously to uniformly update all parameters of the entire network (shared coding layer and two branches). The dual-task network model includes a shared feature coding layer, a depth estimation branch, and a normal prediction branch. The shared feature coding layer is a convolutional neural network, such as ResNet or VGG, which consists of multiple convolutional layers, pooling layers, and activation function layers stacked together. The input is the descattered image. This layer transforms the original pixels into an abstract high-dimensional feature map step by step through multiple convolutional operations. The output of the shared feature coding layer is connected in parallel to the depth estimation branch and the normal prediction branch, and the depth estimation branch and the normal prediction branch perform computations in parallel.

[0033] Preferably, the depth estimation branch is a decoder network consisting of transposed convolutional layers or upsampling layers. Its input is a high-dimensional feature map output from a shared feature encoding layer. Through progressive upsampling, the feature map is mapped to a depth map of the same size as the input image. Each pixel value in the depth map represents the distance of the corresponding spatial point from the camera's optical center, and the output is a single-channel initial depth map. The normal prediction branch is also a decoder network with a similar structure to the depth estimation branch but with independently trained parameters. Its input is also a high-dimensional feature map output from a shared feature encoding layer, and the output is a surface normal map. Each pixel's vector consists of three floating-point numbers representing the unit vector of the surface orientation at that location. The three components of the normal are typically visualized using RGB pseudo-color encoding, and the output is a surface normal map. The training data requires an RGB image dataset simultaneously annotated with ground truth depth and normal values. The loss function is depth loss + normal loss, which represents the mean square error or absolute error between the predicted depth map and the ground truth depth map, and the angular error between the predicted normal map and the ground truth normal map, respectively.

[0034] Furthermore, step S300 also includes: inputting the descattered image into the shared feature encoding layer of the dual-task network model for multi-level convolutional feature extraction to obtain a shared high-level semantic feature map; then performing monocular depth regression and surface normal regression processing on the shared high-level semantic feature map in parallel on the depth estimation branch and the normal prediction branch to obtain an initial depth map and a surface normal map; establishing a differential geometric constraint relationship between the surface normal map and the initial depth map based on the pixel-level geometric orientation information of the surface normal map; constructing a normal consistency optimization function using the differential geometric constraint relationship; performing iterative geometric correction on the initial depth map based on a numerical optimization algorithm; and outputting the geometrically consistent depth map; wherein the geometrically consistent depth map and the surface normal map satisfy the differential geometric constraint relationship.

[0035] Preferably, the descattered image is input into the shared feature encoding layer of the dual-task network model. After passing through multiple convolutional layers of the shared feature encoding layer, each convolutional layer sequentially extracts features from low to high level. Shallow convolutional kernels extract low-level features such as edges, corners, and textures, while deep convolutional kernels extract semantic features such as body parts and surface regions, outputting a shared high-level semantic feature map. The depth estimation branch performs an upsampling operation on the shared high-level semantic feature map, i.e., upsampling by transposed convolution or bilinear interpolation followed by convolution, regressing depth values ​​pixel by pixel, and outputting an initial depth map. Each pixel value is a floating-point number representing the distance from the point to the camera optical center. The normal prediction branch performs an upsampling operation on the shared high-level semantic feature map in parallel, regressing the three components of the surface normal vector pixel by pixel and normalizing them to ensure that the magnitude is 1, outputting a surface normal map.

[0036] Preferably, for a continuous surface in three-dimensional space, the surface normal vector and depth satisfy the following differential relationship: the tangent plane of the surface at a certain point is determined by the partial derivatives of the depth with respect to x and y; there are fixed geometric constraints between the surface normal vector and the depth gradient. Specifically, in the camera coordinate system, if the depth value represents the distance from the spatial point corresponding to the pixel to the camera optical center and the camera intrinsic parameters are known, then the relationship between the normal vector and the depth gradient can be established. For example, the x-component of the normal vector is proportional to the partial derivative of the depth with respect to the u direction, the y-component of the normal vector is proportional to the partial derivative of the depth with respect to the v direction, and the z-component of the normal vector is related to the depth value itself. For each pixel in the surface normal map, its normal component is read. For the same pixel in the initial depth map, its approximate partial derivative with respect to the adjacent pixels is calculated. The calculated normal component must be equal to the theoretical normal component derived from the depth gradient. Then, a normal consistency optimization function is constructed using differential geometric constraints. For each pixel, a consistency error is defined, which is the square of the absolute value of the difference between the normal value output by the normal prediction branch and the normal value derived from the initial depth map through differential calculation. A smoothing regularization term of the depth map is added to prevent over-correction from amplifying noise. A total loss function is constructed for the entire image. The final output geometrically consistent depth map is required to have a normal map obtained by differential calculation that is as close as possible to the normal value of the corresponding pixel in the surface normal map.

[0037] Preferably, the initial depth map is subjected to iterative geometric correction based on a numerical optimization algorithm. Specifically, the initial depth map is used as the initial value of the optimization variable. The depth value is updated pixel by pixel using the gradient descent method. In the k-th iteration, the normal map corresponding to the current depth map is calculated, and the consistency error between the current depth map and the target normal map (surface normal map) is calculated. Based on the derivative (gradient) of the error with respect to the depth value, the depth value of each pixel is adjusted in the direction of reducing the error, and the depth value is updated. When the number of iterations reaches a preset upper limit (e.g., 50 times) or the change in depth value between two adjacent iterations is less than a preset threshold (e.g., 1 mm), the iteration terminates, and a geometrically consistent depth map is output. The depth gradient is calculated by spatial difference of the depth map, and the normal component obtained by conversion is highly consistent with the surface normal map predicted by the dual-task network at the pixel level. Moreover, the two satisfy the differential geometric constraint relationship, that is, there is no geometric ambiguity between the depth map and the normal map.

[0038] Step S400: Project the geometrically consistent depth map onto the lidar coordinate system using a dynamic calibration matrix to generate a guide point cloud.

[0039] Preferably, the calibration matrix is ​​a 3×4 projection matrix or a 4×4 rigid body transformation matrix, which includes the intrinsic parameter matrix (3×3) of the RGB camera, namely focal length, principal point coordinates, distortion coefficient, and the extrinsic parameter matrix from the camera to the LiDAR. The three-dimensional point coordinates in the RGB camera coordinate system are transformed to the LiDAR coordinate system. The dynamic calibration matrix is ​​updated in real time according to various factors, such as equipment deformation caused by thermal expansion and contraction due to temperature changes, and extrinsic parameter drift caused by minor vibrations of the equipment transportation or installation base. The matrix parameters are periodically corrected through closed-loop calibration feedback, and online calibration updates are performed using stable natural features in the scene (such as the edge of a coal pile or fixed reference objects). Then, the geometrically consistent depth map is projected onto the LiDAR coordinate system using a dynamic calibration matrix. For each pixel and its depth value in the depth map, the pixel coordinates are converted into image physical coordinates. The depth values ​​are used to construct the 3D point coordinates in the camera coordinate system. Using the extrinsic parameters in the dynamic calibration matrix, a projection operation is performed on each pixel in the geometrically consistent depth map, generating a 3D point for each pixel. The 3D coordinates of each point are determined. The color value of the original pixel (RGB values ​​extracted from the corresponding position in the descattered image) is used as the color attribute of the point cloud, and the normal information (extracted from the corresponding position in the surface normal map) is used as the normal attribute of the point cloud. Finally, a guiding point cloud is output, containing multiple 3D points. Each point contains coordinates (x, y, z) and additional attributes. This point cloud is spatially aligned with the original LiDAR point cloud in the LiDAR coordinate system to compensate for missing data in the failure area (hole) of the original LiDAR point cloud.

[0040] Step S500: Using the topological boundary of the failure region mask as a constraint, perform spatiotemporal consistency fusion of the original laser point cloud and the guiding point cloud based on adaptive confidence weight, and output the compensated point cloud.

[0041] Step S500 further includes: performing morphological boundary tracking on the failure region mask to obtain a set of boundary pixel contours, then performing polygon fitting to obtain a closed topological boundary representation; based on the joint calibration matrix of the intrinsic and extrinsic parameters of the lidar and RGB camera, registering the guiding point cloud to the lidar coordinate system through rigid body transformation; synchronizing the original lidar point cloud and the guiding point cloud in time according to the hardware triggering timing of the lidar and RGB camera to generate a registered geometrically completed point cloud; constructing an adaptive confidence weight function based on the closed topological boundary representation and the depth information of the registered geometrically completed point cloud; and, within the Bayesian maximum a posteriori probability estimation framework, using the adaptive confidence weight function as the inverse variance of the observation noise, performing optimal weighted fusion of the original lidar point cloud and the registered geometrically completed point cloud under uncertainty propagation to output the compensated point cloud.

[0042] Preferably, the topological boundary of the failure region mask is used as a spatial constraint to perform adaptive weighted Bayesian fusion on the original laser point cloud and the guide point cloud. Specifically, an eight-neighborhood boundary tracing algorithm is used to perform morphological boundary tracing on the failure region mask, extracting boundary pixels of all regions with a value of 1 from the mask, and outputting a set of boundary pixel contours containing multiple closed contours, each contour consisting of an ordered set of pixel coordinate points; the Douglas-Peucker polygon approximation algorithm is used to simplify the set of boundary pixel contours, retaining the main inflection points of the contours and deleting collinear or nearly collinear intermediate points, outputting a closed topological boundary representation, consisting of one or more polygons, each polygon defined by several vertex coordinates, representing the spatial boundary between the failure region and the effective region. Using the joint calibration matrix of the intrinsic and extrinsic parameters of the LiDAR and RGB camera (i.e., the rotation matrix and translation vector), a rigid body transformation is performed on each point in the guide point cloud, and the guide point cloud transformed to the LiDAR coordinate system is output. Then, the original LiDAR point cloud and the guide point cloud are synchronized according to the hardware trigger timing of the LiDAR and RGB camera. For each frame of the original LiDAR point cloud, the arrival time of its trigger pulse is recorded. The guide point cloud acquired at the same time is selected. If the frame rates of the LiDAR and the camera are different, for example, the LiDAR is 10Hz and the camera is 30Hz, the nearest neighbor time matching is used to select the guide point cloud frame with a time difference of less than half a frame period, and the time-aligned and registered geometrically completed point cloud is output.

[0043] Preferably, the input is the depth information of the closed topological boundary representation and the registered geometrically completed point cloud. The polygon representing the closed topological boundary is back-projected into 3D space. Combined with the depth information, a 3D spatial boundary surface in the lidar coordinate system is constructed. For any 3D point in the lidar coordinate system, the directed distance from it to the 3D boundary surface model is calculated. A positive distance value indicates that the point is on one side of the failure region and needs compensation. A negative distance value indicates that the point is on one side of the effective region and the original lidar point cloud is reliable. A distance value of zero indicates that the point is on the boundary. This process further divides the region into a reliable effective region, a boundary transition region, and a reliable failure region. Then, an adaptive confidence weight function is constructed, usually an sigmoid function or a piecewise linear function. The weight distribution is dynamically adjusted according to the failure region morphology of the current frame. The Bayesian maximum posterior probability is set. The estimation framework uses the ground truth value of the real surface point cloud as the estimator. The observation data includes the original laser point cloud observations and the registered geometrically completed point cloud observations. An adaptive confidence weight function is used as the inverse variance of the observation noise to perform optimal weighted fusion of the original laser point cloud and the registered geometrically completed point cloud under uncertainty propagation. For each spatial location, the observation values ​​of the corresponding points in the original laser point cloud and the registered geometrically completed point cloud are taken, and the fused estimated value is calculated. This variance represents the uncertainty of the fused point cloud at that location. Finally, a compensated point cloud is output. Each point contains the estimated value calculated by Bayesian fusion and the uncertainty measure (variance). Within the failure region mask, the guiding point cloud information is mainly used, while outside the failure region mask, the original laser point cloud information is mainly used. The fusion is smoothly transitioned in the boundary region.

[0044] Furthermore, step S500 also includes: using the closed topological boundary as a two-dimensional projection reference, performing back projection calculations in conjunction with the depth information of the registered geometric completion point cloud, and constructing a three-dimensional boundary surface model in the lidar coordinate system; calculating the spatial oriented distance field of the three-dimensional boundary surface model, dividing the three-dimensional spatial domain covered by the original lidar point cloud into a certainty valid region, a certainty invalid region, and a boundary transition region; and constructing the adaptive confidence weight function based on the regional division of the certainty valid region, the certainty invalid region, and the boundary transition region.

[0045] Preferably, a 3D boundary surface model is constructed using the depth information of the closed topological boundary representation and the completed point cloud. A spatial directed distance field is used to divide the spatial region, and an adaptive confidence weight function is constructed. Specifically, the depth information of the closed topological boundary representation and the registered geometrically completed point cloud is input. For each boundary pixel in the closed topological boundary representation, the corresponding 3D coordinates of that pixel are found in the registered geometrically completed point cloud. Since the guiding point cloud is generated pixel-by-pixel projection, each pixel has a unique corresponding 3D point. The 3D coordinates of the boundary pixel are obtained, and the 3D coordinates of all boundary pixels are collected to obtain a point set in 3D space. Using the boundary point set as constraints, a continuous 3D surface is constructed using surface interpolation or triangulation algorithms. This surface represents the interface between the failure region and the effective region in 3D space, and the 3D boundary surface model is output. For any 3D point in the lidar coordinate system, a directed distance to the 3D boundary surface model is defined, the magnitude of which is the shortest Euclidean distance from the point to the boundary surface. For each point in the original lidar point cloud, the distance from the point to the 3D boundary surface model is calculated, and the minimum value of the distance from the point to each triangle face is taken. Then, a failure confidence threshold and a valid confidence threshold are preset. For each point, a confidence failure zone, a boundary transition zone, and a confidence valid zone are divided according to its directed distance. Finally, a piecewise function is constructed as an adaptive confidence weight function. In the confidence valid zone, the confidence weight of the point cloud is low, and in the confidence failure zone, the confidence weight of the point cloud is high. In the boundary transition zone, the weight smoothly transitions from 0 to 1 to avoid abrupt changes in the fusion boundary. The weight function is dynamically determined based on the geometric information of the current frame, such as dynamically calculated based on the curvature of the boundary surface model or the point cloud density.

[0046] Step S600: Perform local surface optimization on the compensated point cloud based on moving least squares smoothing and local Poisson reconstruction, output a continuous surface model, perform feedback calibration closed loop based on volume calculation error, and output a three-dimensional reconstruction model.

[0047] Step S600 further includes: after fitting the compensated point cloud to a local surface based on adaptive neighborhood radius selection using the moving least squares method, projecting the compensated point cloud onto the local surface for smoothing processing, and outputting a smoothed point cloud; calculating the uniformized normal vector field of the smoothed point cloud on the local surface; extracting the spatial position and normal vector of multiple points from the smoothed point cloud and the uniformized normal vector field, constructing a three-dimensional vector contribution point by point through kernel function mapping, and accumulating the contributions to synthesize a global indicator function gradient field; solving the Poisson equation with the global indicator function gradient field as a constraint condition to obtain a three-dimensional spatial scalar indicator function, extracting the isosurface of the three-dimensional spatial scalar indicator function to obtain the continuous surface model; performing a feedback calibration closed loop based on volume calculation error on the continuous surface model, and outputting the three-dimensional reconstruction model.

[0048] Preferably, the input compensation point cloud is used for local surface fitting based on adaptive selection of neighborhood radius. For each point in the point cloud, the neighborhood radius is dynamically determined according to its local point cloud density. A smaller neighborhood radius is used for high-density areas, and a larger neighborhood radius is used for low-density areas. The radius is gradually expanded with the point as the center until the number of points in the neighborhood reaches a preset value (e.g., 20). The moving least squares method is used for fitting, and a weighted polynomial function is used to fit the local surface in the neighborhood, outputting the local fitted surface corresponding to each point. Then, each point in the compensation point cloud is projected along the normal direction onto its corresponding local fitted surface, and a smooth point cloud is output. In this case, the noise of the new point position after projection is eliminated, and the surface is smoother. Next, a uniform normal vector field for smoothing the point cloud is calculated on the local surface. For each point, principal component analysis is performed using its neighboring points to construct the covariance matrix of the neighboring points. The eigenvector corresponding to the smallest eigenvalue of the covariance matrix is ​​then solved and used as the normal vector of that point. The center of the point cloud is then selected as the reference point, and its normal vector is set to point in the direction of the camera. The normal direction is propagated using a breadth-first search. If the dot product of the normal vectors of two adjacent points is negative, the direction of one of them is flipped, and a uniform normal vector field is output, that is, all normal vectors point to the same side of the surface.

[0049] Preferably, the gradient value at the surface boundary is non-zero and its direction is consistent with the normal vector. For each point and its normal vector in the smooth point cloud, a Gaussian kernel function is used to construct the contribution of that point to the gradient field, centered on that point. Each point affects the gradient field value in its surrounding neighborhood. A three-dimensional spatial grid is initialized, and the gradient of all grids is initialized to (0, 0, 0). All points are traversed, and the gradient contribution calculated by the kernel function for each point is accumulated into the corresponding grid cell. The global indicator function gradient field is output, which is a vector field defined on the three-dimensional grid. Then, the global indicator function gradient field is used as a constraint. The Poisson equation is solved by using the gradient field as a constraint to construct a sparse linear system of equations, which is then solved using an iterative solver to obtain the scalar indicator function value for each grid cell, and outputting the 3D spatial scalar indicator function. The MarchingCubes algorithm is used to extract isosurfaces, with the median or average value set as the isosurface threshold. All grid cells are traversed, and the existence of isosurfaces within a cell is determined based on the relationship between the scalar indicator function values ​​of the eight vertices and the threshold. The intersection points of the isosurfaces and the cell edges are calculated by linear interpolation, connected to form triangular patches, and outputting a continuous surface model. Then, a feedback calibration closed loop based on volume calculation error is performed, including volume integration of the continuous surface model, outputting the model volume value, converting the actual weighing data to obtain the calibration volume value, and calculating the relative volume error; establishing an approximate functional relationship between parameter combination and volume error, determining the response model, iteratively optimizing based on the current volume error, repeatedly generating a new continuous surface model, repeatedly calculating the volume error until the error is less than a preset threshold (such as 1%), and outputting a three-dimensional reconstruction model that meets the accuracy requirements. In this process, the reconstruction parameters are corrected in reverse by external weighing data to eliminate long-term drift and ensure the accuracy of volume measurement.

[0050] Furthermore, step S600 also includes: calculating the geometric volume of the continuous surface model as the model volume value, comparing it with the calibration volume value, and calculating the relative volume error; predefining multiple key parameters for local surface optimization of the continuous surface model, and constructing a parameterized response model; performing model volume error comparison and iteration under the parameterized response model with the multiple key parameters adjusted until the volume error meets the preset accuracy requirements, and outputting the three-dimensional reconstruction model.

[0051] Preferably, a continuous surface model is input, and the volume enclosed by the triangular mesh closed surface is calculated using the divergence theorem (Gauss's formula). Specifically, a reference plane is set with the ground or a reference horizontal plane. For each triangular facet in the mesh, the volume of the tetrahedron enclosed between the triangle and the reference plane is calculated. All tetrahedral volumes are summed with signs, and the absolute value is taken to obtain the final model volume value. The calibration volume value is converted from actual weighing data. The total mass of the coal pile is measured by a weighbridge or belt scale and divided by the pre-determined bulk density of the coal to obtain the calibration volume value. The difference is calculated and the relative volume error value is output. Several key parameters are predefined for local surface optimization of the continuous surface model, including a neighborhood radius of 0.5cm to 5cm, a polynomial order of 1 or 2, a Poisson reconstruction mesh depth of 6 to 12, an isosurface extraction threshold of 0.3 to 0.7, and a normalized neighborhood radius of 1cm to 3cm. A parameterized response model is constructed to describe the mapping relationship between the key parameter combination and the output volume error. This includes selecting N different parameter combinations in the parameter space, calculating the corresponding volume error for each parameter combination, fitting an approximate function based on sampled data, and outputting the parameterized response model. Then, a continuous surface model is generated using the default parameter combination, the initial volume error is calculated, the adjustment direction of each parameter is predicted based on the current parameterized response model, the partial derivative of the response model with respect to each parameter is calculated using the gradient descent method, the parameter values ​​are adjusted along the error reduction direction, and a new continuous surface model is generated iteratively using the new parameter combination until the volume error meets the preset accuracy requirement of ≤1%. Finally, a 3D reconstruction model is output, and each output model has a corresponding volume error index.

[0052] In the above text, refer to Figure 1 A method for image-guided point cloud compensation for lidar failure areas according to an embodiment of the present invention is described in detail. Next, reference will be made to... Figure 2 An image-guided point cloud compensation system for lidar failure areas is described according to an embodiment of the present invention.

[0053] The image-guided point cloud compensation system for lidar failure areas according to embodiments of the present invention addresses the technical problems in the prior art where, when lidar fails locally on the surface of a coal pile, point cloud data contains holes and outliers, and visual depth estimation is affected by geometric ambiguity and dust scattering, making it difficult to achieve high-precision continuous surface reconstruction. This system improves the integrity and accuracy of three-dimensional reconstruction of coal piles in lidar failure scenarios. Figure 2 As shown, the image-guided point cloud compensation system for lidar failure areas includes: a point cloud image acquisition module 10, an image sequence processing module 20, an image ambiguity correction module 30, a guided point cloud generation module 40, a compensated point cloud generation module 50, and a three-dimensional reconstruction model output module 60.

[0054] The point cloud image acquisition module 10 is used to perform spatiotemporal synchronous acquisition of original laser point clouds and RGB image sequences on the surface of the target coal pile by using hardware-triggered pulse-driven lidar and RGB camera; the image sequence processing module 20 is used to perform suspended particle attenuation preprocessing on the RGB image sequence to obtain a descattered image, and then perform failure region determination based on multimodal confidence fusion in conjunction with the original laser point cloud, outputting a failure region mask; the image ambiguity correction module 30 is used to perform geometric ambiguity correction on the descattered image to obtain a geometrically consistent depth map; and the guided point cloud generation module 40... The module 50 is used to project the geometrically consistent depth map onto the lidar coordinate system through a dynamic calibration matrix to generate a guiding point cloud; the compensation point cloud generation module 50 is used to perform spatiotemporal consistency fusion of the original lidar point cloud and the guiding point cloud based on adaptive confidence weights, using the topological boundary of the failure region mask as a constraint condition, and output the compensation point cloud; the 3D reconstruction model output module 60 is used to perform local surface optimization of the compensation point cloud based on moving least squares smoothing and local Poisson reconstruction, output a continuous surface model, and then perform feedback calibration closed loop based on volume calculation error to output a 3D reconstruction model.

[0055] The specific configuration of the compensation point cloud generation module 50 will be described in detail below. The compensation point cloud generation module 50 further includes: performing morphological boundary tracking on the failure region mask to obtain a set of boundary pixel contours, then performing polygon fitting to obtain a closed topological boundary representation; based on the joint calibration matrix of the intrinsic and extrinsic parameters of the LiDAR and RGB camera, registering the guiding point cloud to the LiDAR coordinate system through rigid body transformation, then synchronizing the original LiDAR point cloud and the guiding point cloud according to the hardware triggering timing of the LiDAR and RGB camera to generate a registered geometrically completed point cloud; constructing an adaptive confidence weight function based on the closed topological boundary representation and the depth information of the registered geometrically completed point cloud; and, within the Bayesian maximum a posteriori probability estimation framework, using the adaptive confidence weight function as the inverse variance of the observation noise, performing optimal weighted fusion of the original LiDAR point cloud and the registered geometrically completed point cloud under uncertainty propagation to output the compensation point cloud.

[0056] The specific configuration of the compensation point cloud generation module 50 will be described in detail below. The compensation point cloud generation module 50 further includes: using the closed topological boundary as a two-dimensional projection reference, performing back projection calculations in conjunction with the depth information of the registered geometrically completed point cloud, and constructing a three-dimensional boundary surface model in the lidar coordinate system; calculating the spatial oriented distance field of the three-dimensional boundary surface model, dividing the three-dimensional spatial domain covered by the original lidar point cloud into a certainty valid region, a certainty invalid region, and a boundary transition region; and constructing the adaptive confidence weight function based on the regional division of the certainty valid region, the certainty invalid region, and the boundary transition region.

[0057] The specific configuration of the 3D reconstruction model output module 60 will be described in detail below. The 3D reconstruction model output module 60 further includes: applying a moving least squares method to the compensated point cloud for local surface fitting based on adaptive neighborhood radius selection; projecting the compensated point cloud onto the local surface for smoothing processing to output a smoothed point cloud; calculating a uniform normal vector field of the smoothed point cloud on the local surface; extracting the spatial positions and normal vectors of multiple points from the smoothed point cloud and the uniform normal vector field; constructing 3D vector contributions point-by-point through kernel function mapping; accumulating these contributions to synthesize a global indicator function gradient field; solving the Poisson equation using the global indicator function gradient field as a constraint to obtain a 3D spatial scalar indicator function; extracting the isosurface of the 3D spatial scalar indicator function to obtain the continuous surface model; performing a feedback calibration closed loop based on volume calculation error on the continuous surface model; and outputting the 3D reconstruction model.

[0058] The specific configuration of the 3D reconstruction model output module 60 will be described in detail below. The 3D reconstruction model output module 60 further includes: calculating the geometric volume of the continuous surface model as the model volume value, comparing it with the calibration volume value, and calculating the relative volume error; predefining multiple key parameters for local surface optimization of the continuous surface model, and constructing a parametric response model; performing iterative comparison of the model volume error under the parameter tuning of the multiple key parameters based on the parametric response model until the volume error meets the preset accuracy requirements, and outputting the 3D reconstruction model.

[0059] The specific configuration of the 3D reconstruction model output module 60 will be described in detail below. The 3D reconstruction model output module 60 further includes: a dual-task network model constructed based on end-to-end training, wherein the dual-task network model includes a shared feature encoding layer, a depth estimation branch, and a normal prediction branch, and the output of the shared feature encoding layer is connected in parallel to the depth estimation branch and the normal prediction branch.

[0060] The specific configuration of the image ambiguity correction module 30 will be described in detail below. The image ambiguity correction module 30 further includes: inputting the descattered image into the shared feature encoding layer of the dual-task network model for multi-level convolutional feature extraction to obtain a shared high-level semantic feature map; then, performing monocular depth regression and surface normal regression processing on the shared high-level semantic feature map in parallel on the depth estimation branch and the normal prediction branch to obtain an initial depth map and a surface normal map; establishing a differential geometric constraint relationship between the surface normal map and the initial depth map based on the pixel-level geometric orientation information of the surface normal map; constructing a normal consistency optimization function using the differential geometric constraint relationship; performing iterative geometric correction on the initial depth map based on a numerical optimization algorithm; and outputting the geometrically consistent depth map; wherein the geometrically consistent depth map and the surface normal map satisfy the differential geometric constraint relationship.

[0061] The specific configuration of the image sequence processing module 20 will be described in detail below. The image sequence processing module 20 further includes: performing time-domain weighted mean calculation on the RGB image sequence with multi-frame pixel alignment to obtain a dust transient suppression image; and performing physical separation of specular reflection and diffuse reflection components on the dust transient suppression image through polarization difference analysis to obtain the descattering image.

[0062] The specific configuration of the image sequence processing module 20 will be described in detail below. The image sequence processing module 20 further includes: after extracting point cloud-level local density distribution features and echo intensity attenuation features from the original laser point cloud, performing point cloud physical failure determination using dynamic dual thresholds to generate a point cloud failure probability map; injecting the descattered image into the specular reflection feature extraction branch and dust occlusion feature extraction branch of a pre-constructed dual-channel semantic segmentation model, performing parallel segmentation of the water surface specular reflection region and suspended particle coverage region, obtaining the water surface region probability map and the dust region probability map, and then performing pixel-level probability fusion under channel weighted constraints in the feature fusion layer to output the failure region mask.

[0063] The image-guided point cloud compensation system for lidar failure areas provided in this embodiment of the invention can execute the image-guided point cloud compensation method for lidar failure areas provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An image-guided point cloud compensation method for lidar failure areas, characterized in that, The method includes: The original laser point cloud and RGB image sequence were acquired spatiotemporally and synchronously on the surface of the target coal pile by using a hardware-triggered pulse-driven lidar and RGB camera. After performing suspended particle attenuation preprocessing on the RGB image sequence to obtain a descattered image, the original laser point cloud is combined to perform failure region determination based on multimodal confidence fusion, and a failure region mask is output. The descattered image is then subjected to geometric ambiguity correction to obtain a geometrically consistent depth map; The geometrically consistent depth map is projected onto the lidar coordinate system using a dynamic calibration matrix to generate a guide point cloud. Using the topological boundary of the failure region mask as a constraint, the original laser point cloud and the guiding point cloud are subjected to spatiotemporal consistency fusion based on adaptive confidence weight, and a compensated point cloud is output. The compensated point cloud is optimized for local surfaces based on moving least squares smoothing and local Poisson reconstruction. After outputting a continuous surface model, a feedback calibration loop is performed based on the volume calculation error to output a three-dimensional reconstruction model.

2. The image-guided point cloud compensation method for lidar failure areas as described in claim 1, characterized in that, Using the topological boundary of the failure region mask as a constraint, the original laser point cloud and the guiding point cloud are subjected to spatiotemporal consistency fusion based on adaptive confidence weights to output a compensated point cloud. The method includes: Morphological boundary tracing is performed on the mask of the failure area to obtain the set of boundary pixel contours. Then, polygon fitting is performed to obtain a closed topological boundary representation. Based on the joint calibration matrix of the intrinsic and extrinsic parameters of the lidar and RGB camera, the guiding point cloud is registered to the lidar coordinate system through rigid body transformation. Then, the original lidar point cloud and the guiding point cloud are synchronized in time according to the hardware triggering timing of the lidar and RGB camera to generate a registered geometrically complete point cloud. Based on the closed topological boundary representation and the depth information of the registered geometric completion point cloud, an adaptive confidence weight function is constructed. Within the Bayesian maximum a posteriori probability estimation framework, the adaptive confidence weight function is used as the inverse variance of the observation noise to perform optimal weighted fusion of the original laser point cloud and the registered geometric completion point cloud under uncertainty propagation, and the compensated point cloud is output.

3. The image-guided point cloud compensation method for lidar failure areas as described in claim 2, characterized in that, Based on the closed topological boundary representation and the depth information of the registered geometrically completed point cloud, an adaptive confidence weight function is constructed, the method comprising: Using the closed topological boundary as a two-dimensional projection reference, and combining the depth information of the registered geometric completion point cloud, back projection calculation is performed to construct a three-dimensional boundary surface model in the lidar coordinate system; Calculate the spatial oriented distance field of the three-dimensional boundary surface model, and divide the three-dimensional spatial domain covered by the original laser point cloud into a certain effective region, a certain ineffective region, and a boundary transition region; Based on the regional division of the confidence valid region, confidence invalid region, and boundary transition region, the adaptive confidence weight function is constructed.

4. The image-guided point cloud compensation method for lidar failure areas as described in claim 1, characterized in that, The method involves performing local surface optimization based on moving least squares smoothing and local Poisson reconstruction on the compensated point cloud, outputting a continuous surface model, and then performing a feedback calibration loop based on the volume calculation error to output a three-dimensional reconstructed model. After performing local surface fitting on the compensated point cloud using the moving least squares method based on adaptive neighborhood radius selection, the compensated point cloud is projected onto the local surface for smoothing processing, and a smoothed point cloud is output. Calculate the uniform normal vector field of the smoothed point cloud on the local surface; The spatial positions and normal vectors of multiple points are extracted from the smoothed point cloud and the uniform normal vector field. After constructing the three-dimensional vector contribution point by point through kernel function mapping, the contributions are accumulated to synthesize the global indicator function gradient field. Using the global indicator function gradient field as a constraint, the Poisson equation is solved to obtain the three-dimensional spatial scalar indicator function. Then, the isosurface of the three-dimensional spatial scalar indicator function is extracted to obtain the continuous surface model. A feedback calibration closed loop based on volume calculation error is performed on the continuous surface model to output the three-dimensional reconstruction model.

5. The image-guided point cloud compensation method for lidar failure areas as described in claim 4, characterized in that, The method involves performing a feedback calibration closed loop based on volume calculation error on the continuous surface model to output the three-dimensional reconstructed model. The geometric volume of the continuous surface model is calculated as the model volume value, and compared with the calibration volume value to calculate the relative volume error; Several key parameters for local surface optimization of the continuous surface model are predefined, and a parameterized response model is constructed. Based on the parameterized response model, the model volume error is compared and iterated under the multiple key parameter tuning until the volume error meets the preset accuracy requirements, and then the three-dimensional reconstruction model is output.

6. The image-guided point cloud compensation method for lidar failure areas as described in claim 1, characterized in that, A dual-task network model is constructed based on end-to-end training. The dual-task network model includes a shared feature encoding layer, a depth estimation branch, and a normal prediction branch. The output of the shared feature encoding layer is connected in parallel to the depth estimation branch and the normal prediction branch.

7. The image-guided point cloud compensation method for lidar failure areas as described in claim 6, characterized in that, The method involves performing geometric ambiguity correction on the descattered image to obtain a geometrically consistent depth map, the method comprising: The descattered image is input into the shared feature encoding layer of the dual-task network model for multi-level convolutional feature extraction to obtain a shared high-level semantic feature map. Then, monocular depth regression and surface normal regression are performed in parallel on the shared high-level semantic feature map in the depth estimation branch and the normal prediction branch to obtain the initial depth map and surface normal map. Based on the pixel-level geometric orientation information of the surface normal map, a differential geometric constraint relationship between the surface normal map and the initial depth map is established; The differential geometric constraint relationship is used to construct a normal consistency optimization function, and the initial depth map is subjected to iterative geometric correction based on a numerical optimization algorithm to output the geometrically consistent depth map; The geometrically consistent depth map and surface normal map satisfy the differential geometric constraint relationship.

8. The image-guided point cloud compensation method for lidar failure areas as described in claim 1, characterized in that, The method further includes: The RGB image sequence is subjected to multi-frame pixel-aligned temporal weighted mean calculation to obtain a dust transient suppression image; The descattering image is obtained by physically separating the specular and diffuse reflection components of the dust transient suppression image through polarization difference analysis.

9. The image-guided point cloud compensation method for lidar failure areas as described in claim 8, characterized in that, The method further includes: After extracting the point cloud-level local density distribution features and echo intensity attenuation features from the original laser point cloud, a point cloud physical failure determination is performed by combining dynamic dual thresholds to generate a point cloud failure probability map. The descattered image is injected into the specular reflection feature extraction branch and the dust occlusion feature extraction branch of the pre-constructed dual-channel semantic segmentation model. The water surface specular reflection region segmentation and suspended particle coverage region segmentation are performed in parallel to obtain the water surface region probability map and the dust region probability map. Then, pixel-level probability fusion under channel weighted constraints is performed in the feature fusion layer to output the failure region mask.

10. An image-guided point cloud compensation system for lidar failure areas, characterized in that, The system is used to implement the image-guided point cloud compensation method for lidar failure areas as described in any one of claims 1 to 9, and the system comprises: The point cloud image acquisition module is used to perform spatiotemporal synchronous acquisition of raw laser point cloud and RGB image sequences on the surface of the target coal pile by using hardware-triggered pulses to drive the lidar and RGB camera. The image sequence processing module is used to perform suspended particle attenuation preprocessing on the RGB image sequence to obtain a descattered image, and then perform failure region determination based on multimodal confidence fusion in conjunction with the original laser point cloud, and output a failure region mask. The image ambiguity correction module is used to perform geometric ambiguity correction on the descattered image to obtain a geometrically consistent depth map; The guide point cloud generation module is used to project the geometrically consistent depth map onto the lidar coordinate system through a dynamic calibration matrix to generate a guide point cloud. The compensation point cloud generation module is used to perform spatiotemporal consistency fusion of the original laser point cloud and the guide point cloud based on adaptive confidence weight, using the topological boundary of the failure region mask as a constraint condition, and output the compensation point cloud. The 3D reconstruction model output module is used to perform local surface optimization on the compensated point cloud based on moving least squares smoothing and local Poisson reconstruction. After outputting a continuous surface model, a feedback calibration closed loop is performed based on the volume calculation error to output the 3D reconstruction model.

Images