Multi-sensor fusion coal conveying belt coal blockage detection method and device and storage medium

Abstract

The invention relates to the technical field of multi-sensor detection, and discloses a multi-sensor fusion coal conveying belt coal blockage detection method and device and a storage medium, and the method comprises the steps: collecting a laser point cloud, a first visual image and millimeter wave volume information of a coal flow of a coal conveying belt transfer point, and carrying out the dust environment restoration of the first visual image, obtaining a second visual image; registering the laser point cloud with the second visual image to obtain first fusion data; performing surface projection transformation on the millimeter wave volume information to obtain surface projection data, and integrating the surface projection data with the first fusion data to obtain second fusion data; performing time compensation on the second fusion data to obtain synchronous coal flow data; the coal flow accumulation abnormity identification is performed based on the synchronous coal flow data, the coal flow blockage early warning signal is generated, the accurate identification of the coal flow accumulation abnormity state is realized, and the reliability and the accuracy of blockage detection are improved.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor detection technology, and in particular to a method, device, and storage medium for detecting coal blockage in coal conveyor belts using multi-sensor fusion. Background Technology

[0002] Coal conveyor belt transfer points are critical nodes in coal conveying systems. During coal transfer, real-time monitoring and anomaly identification of the three-dimensional morphology of the coal flow directly affect the safe and stable operation of the entire conveying system. However, the transfer point environment is complex, with high dust concentrations and variable coal flow states. Traditional single-sensor detection technologies are insufficient to comprehensively and accurately acquire the three-dimensional morphology information of the coal flow, easily leading to false alarms and missed detections, which seriously affects the reliability and timeliness of coal blockage early warning. Summary of the Invention

[0003] This invention provides a method, device, and storage medium for detecting coal blockage in coal conveyor belts using multi-sensor fusion. This invention enables accurate identification of abnormal coal flow accumulation states and improves the reliability and accuracy of blockage detection.

[0004] In a first aspect, the present invention provides a multi-sensor fusion method for detecting coal blockage in a coal conveyor belt, the multi-sensor fusion method for detecting coal blockage in a coal conveyor belt comprising: Laser point cloud, first visual image and millimeter wave volume information of coal flow at the coal conveyor belt transfer point are collected, and dust environment is restored from the first visual image to obtain the second visual image; The laser point cloud is registered with the second visual image to obtain the first fused data; The millimeter-wave volume information is subjected to surface projection transformation to obtain surface projection data, and the surface projection data is integrated with the first fused data to obtain the second fused data; Time compensation is applied to the second fused data to obtain synchronized coal flow data; Based on the synchronous coal flow data, coal flow accumulation anomalies are identified, and a coal flow blockage early warning signal is generated.

[0005] In conjunction with the first aspect, in a first implementation of the first aspect of the present invention, the step of acquiring laser point cloud, first visual image, and millimeter-wave volume information of coal flow at the coal conveyor belt transfer point, and performing dust environment restoration on the first visual image to obtain a second visual image, includes: The coal flow at the transfer point of the coal conveyor belt is scanned by lidar to form a lidar point cloud. The coal flow is captured by an industrial camera to form a first-person visual image. The internal structure of the coal flow is detected by millimeter-wave radar to form millimeter-wave volume information. Based on the surface ranging data in the laser point cloud and the depth information of the corresponding pixels in the first visual image, a dust thickness mapping map is established. Based on the dust thickness mapping, the light scattering characteristics of coal particles of different sizes are calculated to determine the scattering attenuation coefficient and the surface reflectivity correction value. Using the scattering attenuation coefficient and the surface reflectivity correction value, the brightness and contrast of each pixel in the first visual image are corrected to obtain the second visual image.

[0006] In conjunction with the first aspect, in a second implementation of the first aspect of the present invention, the step of registering the laser point cloud with the second visual image to obtain the first fused data includes: The laser normal vector is calculated based on the neighborhood geometry of each three-dimensional coordinate point in the laser point cloud, and the visual normal vector at the corresponding position is derived based on the pixel gradient change of the second visual image. Establish a consistency constraint relationship between the laser normal vector and the visual normal vector; Based on the consistency constraint relationship, the first matching point pair that satisfies the coal flow surface continuity condition is identified; Extract the curvature variation features of the first matching point pair on the coal flow surface profile, and determine the corresponding second matching point pair based on the curvature variation features; Set the distance matching weight, curvature matching weight, and normal vector matching weight as the first coal flow detection ratio, and use the second matching point pair and the first coal flow detection ratio to calculate the optimal spatial transformation relationship; Based on the optimal spatial transformation relationship, the laser three-dimensional coordinates and the image two-dimensional coordinates are mapped to form the first fused data.

[0007] In conjunction with the first aspect, in a third implementation of the first aspect of the present invention, the step of performing a surface projection transformation on the millimeter-wave volume information to obtain surface projection data, and integrating the surface projection data with the first fused data to obtain second fused data, includes: The density distribution data of the coal flow from the surface to the interior at each depth layer is extracted from the millimeter-wave volume information, and the density stratification parameters are calculated based on the density distribution data; The density stratification parameters are applied to the depth integration operation to project and convert the millimeter-wave volume information into a two-dimensional surface density distribution map, thereby generating surface projection data. Adjust the scale difference between the millimeter-level resolution of the surface projection data and the sub-millimeter-level resolution of the first fused data to form a fused data pair; The laser surface weighting coefficient and the millimeter wave internal weighting coefficient are determined as the second coal flow detection ratio, and the fused data is used to perform weighted fusion with the second coal flow detection ratio to generate the second fused data.

[0008] In conjunction with the first aspect, in a fourth implementation of the first aspect of the present invention, the step of applying the density stratification parameter to a depth integral operation to project and convert the millimeter-wave volume information into a two-dimensional surface density distribution map, generating surface projection data, includes: Establish the exponential decay function corresponding to the density stratification parameter, set the integral boundary condition from the coal flow surface to the maximum accumulation depth, and construct the volume density integral model based on the exponential decay function and the integral boundary condition. Identify the true bulk density distribution data of coal in the millimeter-wave volume information; The volume density integral model is run to perform layer-by-layer depth integration on the real volume density distribution data to form a two-dimensional surface density distribution map. By comparing the grid size of the two-dimensional surface density distribution map with the spatial resolution of the laser surface measurement grid, a correspondence between millimeter-wave projection coordinates and laser measurement coordinates is established, and surface projection data is generated based on the correspondence.

[0009] In conjunction with the first aspect, in a fifth implementation of the first aspect of the present invention, the step of performing time compensation on the second fused data to obtain synchronized coal flow data includes: Analyze the position change information of the coal flow in the second fused data, and calculate the instantaneous velocity vector and acceleration vector of the coal flow during the belt conveying process based on the position change information; Based on the instantaneous velocity vector and the acceleration vector, the coal flow trajectory is predicted to form trajectory prediction data; Calculate the timestamp deviation value between lidar, industrial camera and millimeter-wave radar, use the motion trajectory prediction data to perform position compensation correction on the timestamp deviation value, and generate a timestamp correction mapping table; The timestamp correction mapping table is used to perform time alignment on the second fused data to obtain synchronized coal flow data.

[0010] In conjunction with the first aspect, in the sixth implementation of the first aspect of the present invention, the step of identifying coal flow accumulation anomalies based on the synchronous coal flow data and generating a coal flow blockage early warning signal includes: Extract the three-dimensional height distribution data of the coal flow from the synchronous coal flow data, and calculate the second-order partial derivative of the coal flow surface based on the three-dimensional height distribution data; When the second-order partial derivative exceeds the coal flow-specific curvature threshold, the curvature anomaly region is marked and a surface curvature change rate feature is generated. The instantaneous change in coal flow volume over time in the synchronous coal flow data is measured, and the instantaneous change is multiplied by the coal flow volume change sensitivity coefficient to form a volume gradient; Based on the volume gradient, abnormal accumulation regions are identified and volume gradient change features are generated; The surface curvature change rate feature and the volume gradient change feature are fused using dual criteria to determine the stacking state determination result; The corresponding coal flow morphology characteristic values ​​are calculated by using the surface curvature, volume gradient, and surface roughness of the abnormal accumulation area. A coal flow blockage early warning signal is generated based on the accumulation state determination result and the coal flow morphology characteristic value.

[0011] In conjunction with the first aspect, in the seventh implementation of the first aspect of the present invention, the step of generating a coal flow blockage early warning signal based on the accumulation state determination result and the coal flow morphology characteristic value includes: When the accumulation state determination result shows that the coal flow accumulation is abnormal, the ratio of the current accumulation height to the critical blockage height is calculated based on the coal flow morphology characteristic value to obtain the accumulation height coefficient. At the same time, the ratio of the coal flow lateral diffusion velocity to the critical diffusion velocity is calculated to obtain the diffusion velocity coefficient. The duration coefficient is calculated based on the accumulation duration. A risk assessment is performed on the accumulation height coefficient, the diffusion rate coefficient, and the duration coefficient to obtain a risk level, and a corresponding coal flow blockage early warning signal is output based on the risk level.

[0012] Secondly, the present invention provides a multi-sensor fusion coal conveyor belt blockage detection device, the multi-sensor fusion coal conveyor belt blockage detection device comprising: The acquisition module is used to acquire laser point cloud, first visual image and millimeter wave volume information of coal flow at the coal conveyor belt transfer point, and to restore the dust environment of the first visual image to obtain the second visual image. A registration module is used to register the laser point cloud with the second visual image to obtain first fused data; An integration module is used to perform surface projection transformation on the millimeter-wave volume information to obtain surface projection data, and to integrate the surface projection data with the first fused data to obtain second fused data; The time compensation module is used to perform time compensation on the second fused data to obtain synchronized coal flow data; The coal flow accumulation anomaly identification module is used to identify coal flow accumulation anomalies based on the synchronous coal flow data and generate a coal flow blockage early warning signal.

[0013] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described multi-sensor fusion method for detecting coal blockage in a coal conveyor belt.

[0014] The technical solution provided by this invention establishes a coal particle scattering attenuation model and calculates the light scattering characteristics based on Weibull statistical laws, realizing a dedicated image restoration technology for high-concentration dust environments at coal transfer points. This effectively solves the problem of severe image quality degradation in dusty environments under traditional visual inspection. A registration algorithm based on coal flow surface normal vector consistency constraints and contour curvature feature matching establishes a sub-pixel-level precise correspondence between laser 3D points and image 2D pixels, overcoming the bottleneck of insufficient registration accuracy in existing technologies and achieving high-precision spatial unification of data from different modal sensors. By establishing a coal density stratified decreasing model and a depth integral transformation algorithm, the spatial scale matching problem between millimeter-wave millimeter-level resolution and laser sub-millimeter-level resolution is innovatively solved, achieving the organic fusion of volume information and surface information. A time compensation model based on coal flow trajectory prediction is established, considering the coal particle friction coefficient and belt drive characteristics, effectively eliminating the timing deviation caused by different sampling frequencies of lidar, visual cameras, and millimeter-wave radar, achieving millisecond-level precise synchronization of multi-sensor data. By establishing a dual-criteria fusion identification algorithm based on the rate of change of surface curvature and volume gradient, and combining it with the physical characteristics of coal flow to set a dedicated weight ratio, the algorithm can accurately identify the abnormal state of coal flow accumulation, significantly improving the reliability and accuracy of blockage detection. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the steps of the multi-sensor fusion method for detecting coal blockage in a coal conveyor belt in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the coal conveyor belt blockage detection device with multi-sensor fusion in an embodiment of the present invention. Detailed Implementation

[0017] This invention provides a method, apparatus, and storage medium for detecting coal blockage in a coal conveyor belt using multi-sensor fusion. The terms "first," "second," "third," "fourth," etc. (if applicable) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to the present invention includes: Step S11: Collect the laser point cloud, first visual image and millimeter wave volume information of the coal flow at the coal conveyor belt transfer point, and restore the dust environment of the first visual image to obtain the second visual image; Specifically, a high-frequency scan of the coal flow surface at the transfer point of the coal conveyor belt is performed using lidar to generate 3D lidar point cloud data containing height and spatial structure information. Simultaneously, an industrial camera captures the same area in visible light to obtain a first-view image, showcasing the texture and brightness details of the coal flow surface. Millimeter-wave radar is used to penetrate dust interference, detect the internal accumulation structure of the coal flow, and generate volume information, achieving multi-source sensing coverage of both the external surface and internal structure of the coal flow. To establish a dust thickness mapping map, a depth correlation is established between the surface ranging values ​​obtained by lidar and the brightness variations of corresponding pixels in the industrial camera image. Based on the geometric mapping relationship between the lidar point cloud and image coordinates, corresponding point pairs are extracted, and the degree of brightness attenuation is calculated. This allows the derivation of the degree of dust occlusion experienced by pixels in the first-view image, constructing a two-dimensional dust thickness distribution map. Based on the dust thickness map and the particle size distribution parameters of coal particles, the light scattering behavior of particles with different sizes is modeled and calculated. Referring to the diffuse reflection and absorption characteristics of coal in a dusty environment, the scattering attenuation coefficient and surface reflectance correction value for each pixel are estimated. The scattering coefficient value ranges from 0.15 to 0.25 depending on the coal type, reflecting the intensity loss of light as it passes through the dust layer. The reflectance correction value is used to restore the true surface brightness of the obscured area. The scattering attenuation coefficient and surface reflectance correction value are applied to the pixel-by-pixel correction of the first visual image. Taking into account both the brightness reduction caused by scattering attenuation and the color shift caused by changes in surface material reflection, the brightness and contrast of the image are reconstructed and enhanced point-by-point, outputting the second visual image.

[0019] Step S12: Register the laser point cloud with the second visual image to obtain the first fused data; Specifically, based on the neighborhood structure information of each 3D coordinate point in the laser point cloud, principal component analysis is used to perform covariance matrix operations on several surrounding points to extract the principal direction of the local surface where the 3D coordinate point is located, and the laser normal vector is calculated, reflecting the geometric sectional direction of the coal flow surface at that location. Simultaneously, in the second visual image, the grayscale gradient of the image is used to calculate the two-dimensional spatial derivative, and a gradient vector field is constructed using the Sobel operator. The grayscale gradient is then mapped to the 3D coordinate system using image-space mapping rules, thereby deriving the visual normal vector of the corresponding pixel in the visual image. A consistency constraint relationship is established between the two sets of normal vectors, i.e., an angle threshold of 15° is set to filter laser normal vector and visual normal vector point pairs with an angle less than the threshold, ensuring that these point pairs have directional consistency on the actual coal flow surface, conforming to the physical consistency assumption of continuous coal accumulation, thus obtaining the first matching point pair. The curvature change characteristics of the first matching point pair on the coal flow surface contour are extracted, and combined with the principle of curvature continuity of the coal flow surface, a second matching point pair with coordinated surface geometry and consistent curvature characteristics is identified. A comprehensive evaluation function is constructed based on three indicators for matching point pairs. Distance matching weight minimizes the Euclidean error between the 3D point and its image mapping point; curvature matching weight measures local contour consistency; and normal vector matching weight measures surface orientation consistency. These three weights are set to 0.5, 0.3, and 0.2 according to the first coal flow detection ratio, reflecting their actual importance in the surface characteristics of the coal flow. By minimizing the weighted error function, the optimal spatial transformation relationship between the laser point cloud and the 2D image in 3D space is calculated. This optimal spatial transformation relationship includes a rotation matrix and a translation vector, used to map the laser 3D coordinates to the visual image coordinate system, achieving coordinate system consistency. Based on the optimal spatial transformation relationship, coordinate reconstruction is performed on all laser point cloud data, forming first fused data aligned with the second visual image.

[0020] Step S13: Perform surface projection transformation on the millimeter-wave volume information to obtain surface projection data, and integrate the surface projection data with the first fused data to obtain the second fused data; Specifically, the coal flow volume information acquired by millimeter-wave radar is structured to extract the reflection intensity and density distribution data of the coal flow at different depth levels, reflecting the density variation trend of coal accumulation from the surface to the interior, exhibiting a gradual decrease characteristic from the surface to the interior. Based on this trend, a density stratification model of the coal flow is established, constructing a parametric expression in the form of density decreasing with depth, and calculating density stratification parameters. The density stratification parameters are then used to longitudinally integrate the millimeter-wave volume data, compressing the three-dimensional density field along the depth direction to a two-dimensional plane, generating the average density value at the corresponding location on the coal flow surface, and constructing a two-dimensional surface density distribution map, i.e., surface projection data, on all surface pixels, presenting the density distribution pattern of the coal flow on the horizontal plane. The resolution of the surface projection data is scaled, and the original millimeter-level accuracy of the millimeter-wave data is matched with the sub-millimeter-level laser data through interpolation or resampling, forming a one-to-one corresponding fused data pair between the two data sources, and a unified coordinate framework is established on this basis. A weighting strategy is set based on the second coal flow detection ratio, with a weighting coefficient of 0.7 for laser surface data and 0.3 for millimeter-wave internal data, reflecting the relative importance of surface geometric accuracy and internal structural signals in coal flow deposition analysis. The laser-visual fusion value and millimeter-wave surface projection value of each pair of corresponding pixels are weighted and averaged according to the set weights to form a fused density field, outputting second fused data that simultaneously includes surface morphology, texture details, and internal deposition state.

[0021] Step S14: Perform time compensation on the second fused data to obtain synchronized coal flow data; Specifically, continuous positional change information of the coal flow during the conveying process is extracted from the second fused data. This continuous positional change information is obtained based on the temporal changes in the geometric contour and internal density field of the coal flow as reflected by laser point clouds, visual images, and millimeter-wave volume data at different times. By analyzing the characteristics of these positional evolutions over time, the instantaneous velocity and acceleration vectors of the coal flow during belt conveying are calculated. The velocity vector is obtained by removing the spatial position from the fused data of two consecutive moments by a time interval, while the acceleration vector is derived from the rate of change of velocity over time. The acceleration is then corrected by incorporating the mechanical parameters of the belt conveyor (such as drive power, friction coefficient, and conveying angle). Using the velocity and acceleration information, a two-dimensional accelerated motion trajectory model of the coal flow is established to predict its position over a future period, forming motion trajectory prediction data containing the theoretical spatial position information corresponding to the triggering time of each sensor. The time difference between different data frames is calculated, and combined with the trajectory prediction results of the coal flow during these time differences, the data position at each sensor acquisition moment is compensated and corrected. A timestamp correction mapping table is established to record the unified time reference position corresponding to each sensor timestamp. Based on the timestamp correction mapping table, the second fusion data obtained by different sensors under asynchronous sampling is aligned and resampled. The second fusion data is then uniformly mapped to the same reference time point through interpolation or translation transformation to obtain synchronous coal flow data.

[0022] Step S15: Identify coal flow accumulation anomalies based on synchronous coal flow data and generate a coal flow blockage early warning signal.

[0023] Specifically, the three-dimensional height distribution information of the coal flow surface is extracted from synchronous coal flow data to reflect the accumulation profile and local undulations of the coal flow during belt conveying. Spatial gradient analysis is performed on the three-dimensional height distribution, and the second-order partial derivatives of the three-dimensional height distribution in the x and y directions are calculated to obtain the rate of curvature change of the coal flow surface in different directions. When the rate of curvature change of a certain region exceeds a preset coal flow-specific curvature threshold (e.g., 0.15 m), the curve is considered to be at a certain value. -1When the current area is identified as a curvature anomaly region, its spatial distribution characteristics are extracted to generate surface curvature change rate features, identifying morphological changes that may be caused by accumulation, such as local protrusions, bulges, or collapses on the coal flow surface. Combining the time dimension of the synchronous data, the volume change of the coal flow per unit time is measured and multiplied with a coal flow volume change sensitivity coefficient (e.g., 1.2) to form a temporal gradient value of the coal flow volume. This reflects whether the coal flow exhibits abnormal accumulation, rapid increase in pile height, or nonlinear trends such as obstruction of discharge within a short period. By analyzing the spatial diffusion region of the volume gradient, the accelerated accumulation zone within the coal flow is identified, forming volume gradient change features. The surface curvature change rate features and volume gradient change features are fused using dual criteria, and a weighted strategy or classifier model is employed for joint analysis to improve the accuracy and robustness of anomaly detection, outputting the accumulation state determination result. Simultaneously, within each abnormal accumulation region, the corresponding surface curvature value, volume gradient value, and surface roughness value are extracted. The variance of the roughness function is calculated to reflect the uniformity and undulation of particle distribution on the coal flow surface. These indicators collectively constitute the three-dimensional morphological characteristic values ​​of the coal flow, used to comprehensively describe the appearance and structural characteristics of the coal flow under accumulation conditions. The accumulation state determination results are combined with the morphological characteristic values ​​and input into a preset risk assessment model to generate quantitative coal flow blockage early warning signals. These signals are then categorized into different warning levels (e.g., yellow, orange, or red) based on morphological characteristic levels and the severity of the anomaly.

[0024] In one specific embodiment, the process of performing step S11 may specifically include the following steps: The coal flow at the transfer point of the coal conveyor belt is scanned by lidar to form a lidar point cloud. The coal flow is captured by an industrial camera to form a first-person visual image. The internal structure of the coal flow is detected by millimeter-wave radar to form millimeter-wave volume information. A dust thickness mapping map is established based on the surface ranging data in the laser point cloud and the depth information of the corresponding pixels in the first visual image. Based on the dust thickness mapping diagram, the light scattering characteristics of coal particles of different sizes are calculated to determine the scattering attenuation coefficient and the surface reflectivity correction value. The brightness and contrast of each pixel in the first visual image are corrected using the scattering attenuation coefficient and the surface reflectance correction value to obtain the second visual image.

[0025] Specifically, three types of heterogeneous sensing devices are deployed at the transfer points of the coal conveyor belt: lidar, industrial vision cameras, and millimeter-wave radar. LiDAR performs non-contact scanning of the coal flow surface at high frequency, acquiring three-dimensional laser point cloud data containing spatial depth, morphological structure, and surface undulations. Simultaneously, the industrial camera performs visible light imaging of the coal flow surface via video stream, generating a two-dimensional image with realistic color and texture information, thus obtaining the first visual image. Millimeter-wave radar, utilizing its insensitivity to dust, penetrates the high-concentration dust layer covering the coal flow surface, probing the internal structure of the coal flow and outputting three-dimensional volumetric reflectance intensity data to construct a volumetric model of the coal flow's interior. The ranging information of the laser point cloud is geometrically mapped to the pixel information in the first visual image. Through system calibration, the spatial transformation matrix between the lidar's field of view and the industrial camera's viewpoint is obtained. Using extrinsic parameter transformation and intrinsic parameter calibration relationships, the three-dimensional points are projected onto the image plane, establishing the true spatial depth data corresponding to each pixel in the image. By comparing the difference between the standard distance measured by lidar and the brightness attenuation of pixels in the visual image, the degree of dust occlusion of pixels during the shooting process is deduced. A two-dimensional dust thickness map is constructed across the entire image area to reflect the dust concentration distribution at different locations, revealing the spatial boundary characteristics between occluded and unoccluded areas in the image. Based on the dust thickness map, a physical model of the light scattering behavior of coal particles is established. Considering the significant impact of different coal types and their particle size distributions on optical properties, a Weibull distribution function is used to describe the particle size probability density based on the statistical distribution characteristics of coal particle size. Combined with the multiple scattering and absorption processes of light in dust, a scattering attenuation model is established. Based on the shape and size parameters of coal particles, the scattering cross-section and corresponding attenuation effect under different particle sizes are calculated, thereby quantifying the degree of attenuation accumulated on the dust penetration path of each pixel and obtaining the scattering attenuation coefficient corresponding to each pixel. Simultaneously, the coal surface exhibits different reflective characteristics due to varying material roughness. Based on the surface structure information provided by laser point clouds, its average roughness index is derived, and combined with gray-level gradient analysis and reflectivity regression models, the surface reflectivity correction value of each pixel is calculated. These two values ​​are used to compensate for the brightness attenuation caused by dust and the change in reflection intensity caused by differences in coal material, respectively. The scattering attenuation coefficient and the surface reflectivity correction value are jointly applied to the pixel-by-pixel brightness restoration and contrast adjustment process of the first visual image. Specifically, the original brightness value of each pixel undergoes inverse attenuation calculation based on an exponential enhancement model. This involves amplifying the attenuation caused by dust and correcting the reflectivity error to restore the true brightness that the pixel should present when there is no dust obstruction. Simultaneously, a local contrast enhancement algorithm is used to improve the recognition of image edges and details, effectively suppressing the visual degradation phenomenon of the entire image in a dusty environment. The processed image is the second visual image.

[0026] The process involves comparing surface ranging data in the laser point cloud with the depth information of corresponding pixels in the first visual image, and establishing a dust thickness mapping map based on the depth information. This includes: establishing a geometric calibration transformation relationship between the laser radar ranging coordinate system and the industrial camera imaging coordinate system; projecting the three-dimensional ranging data in the laser point cloud onto the two-dimensional pixel coordinates of the first visual image using a spatial coordinate transformation matrix to form a laser-visual spatial correspondence; identifying pixel regions in the first visual image that are severely affected by dust, and marking pixels as dust-occluded when the difference between the measured laser distance and the visually estimated distance at the corresponding pixel location exceeds a dust occlusion discrimination threshold. When the difference is less than the clear imaging discrimination threshold, it is marked as a clear imaging pixel, and a dust occlusion area distribution map is generated. The deviation between the laser measured distance and the visually estimated distance at each occluded pixel position in the dust occlusion area distribution map is calculated. Combining the suspension characteristics of coal particles in the air and the law of gravity settling, a nonlinear mapping function between dust particle thickness and distance deviation is established, and the dust layer thickness value is calculated pixel by pixel. The dust particle thickness value is integrated with the geometric morphology information of the coal flow surface, taking into account the airflow distribution at the transfer point and the dust diffusion mode. The dust thickness value of the unmeasured area is filled in by the spatial interpolation algorithm, and a dust thickness mapping map covering the entire coal flow surface is constructed.

[0027] In one specific embodiment, the process of performing step S12 may specifically include the following steps: The laser normal vector is calculated based on the neighborhood geometry of each 3D coordinate point in the laser point cloud, and the visual normal vector at the corresponding position is derived based on the pixel gradient changes of the second visual image. Establish a consistency constraint relationship between the laser normal vector and the visual normal vector; Based on the consistency constraint relationship, the first matching point pair that satisfies the condition of continuity of the coal flow surface is identified; Extract the curvature variation features of the first matching point pair on the coal flow surface profile, and determine the corresponding second matching point pair based on the curvature variation features; Set the distance matching weight, curvature matching weight, and normal vector matching weight as the first coal flow detection ratio, and use the second matching point pair and the first coal flow detection ratio to calculate the optimal spatial transformation relationship; Based on the optimal spatial transformation relationship, the laser's three-dimensional coordinates are mapped to the image's two-dimensional coordinates to form the first fused data.

[0028] Specifically, local geometric structure analysis is performed on each 3D coordinate point in the laser point cloud. Each target point and its neighboring point set within a certain range are selected in the point cloud, and a covariance matrix is ​​constructed based on the local point set. Principal component analysis is used to extract the surface normal vector of the region. The eigenvector corresponding to the smallest eigenvalue is the laser normal vector of the target point, reflecting the local directional characteristics of the surface where the 3D point is located. Simultaneously, in the second visual image after dust restoration processing, gradient calculation is performed on each pixel. The Sobel or Scharr operator is used to extract the horizontal and vertical grayscale change rates, forming a two-dimensional gradient vector field. The two-dimensional gradient information is combined with camera intrinsic parameters and the viewpoint matrix and back-projected into 3D space to derive the visual normal vector corresponding to each pixel, representing the spatial directional trend of the target edge or texture region in the image. A consistency constraint relationship is established between the laser normal vector and the visual normal vector, quantifying the similarity of the spatial directions of two different sources at the same location. The degree of consistency is measured by calculating the angle between the normal vectors. If the angle is less than a set threshold (e.g., 15°), the two normal vectors are considered to have consistent directions. In the laser point cloud and visual image, all pixel-laser point combinations that satisfy the normal vector consistency condition are searched and marked as the first matching point pairs. Curvature analysis is performed on the contour features of the first matching point pairs on the coal flow surface. Curvature change information of the region where the first matching points are located is extracted. For each point, the curvature value in its neighborhood is calculated to determine its geometric state in the contour edge, depression, or ridge region. Based on this, point pairs with consistent curvature change patterns are selected to obtain the second matching point pairs. A registration optimization function fusing three types of error weights is constructed to calculate the optimal spatial transformation relationship between the laser point cloud and the image plane. Three weight parameters are set: distance matching weight is used to measure the Euclidean distance deviation of the matching point pairs in three-dimensional space, curvature matching weight reflects the consistency of the surface contour morphology, and normal vector matching weight is used to measure the consistency of the surface orientation. The combination of these three constitutes the first coal flow detection ratio, set to 0.5:0.3:0.2. An error cost function is constructed based on weighted proportions. Least squares fitting or robust optimization algorithms are used to solve for the spatial transformation parameters, obtaining the optimal rigid transformation matrix describing the mapping from the laser point cloud to image coordinates. This matrix includes a rotation matrix and a translation vector. The obtained spatial transformation relationship is applied to each 3D coordinate point in the laser point cloud, projecting it onto the 2D plane of the image. This establishes a spatial correspondence with the pixel positions in the visual image, achieving alignment and fusion of multi-source data within a unified spatial framework, forming the first fused data.

[0029] The optimization ratio for coal flow detection is set as follows: distance matching weight, curvature matching weight, and normal vector matching weight. The optimal spatial transformation relationship is calculated using the second matching point pair and the optimized coal flow detection ratio. This includes: statistically analyzing the Euclidean distance error distribution between the laser 3D coordinates and the image 2D coordinates in the second matching point pair; calculating the mean and standard deviation of the distance error; determining the distance matching reliability coefficient based on the coal particle size distribution characteristics; and establishing an adaptive adjustment function for the distance matching weight. The consistency of the coal flow surface curvature at corresponding positions of the second matching point pair is analyzed, and the correlation between the laser-measured curvature and the visually estimated curvature is calculated. Based on the coefficients and the natural angle of repose characteristics of coal accumulation, a dynamic optimization function for curvature matching weight is established; the angular deviation distribution between the laser surface normal vector and the visual surface normal vector at the second matching point is evaluated, and the normal vector consistency evaluation index is calculated. Based on the physical constraints of coal flow surface continuity, an intelligent adjustment mechanism for normal vector matching weight is constructed; a multi-objective optimization algorithm is run to jointly optimize the distance matching weight, curvature matching weight, and normal vector matching weight, with the objective functions of minimizing registration residuals and maximizing coal flow geometric consistency, and iteratively calculating the coal flow detection optimization ratio adapted to the current coal flow state.

[0030] In one specific embodiment, the process of performing step S13 may specifically include the following steps: Density distribution data of coal flow from the surface to the interior at various depths are extracted from millimeter-wave volume information, and density stratification parameters are calculated based on the density distribution data; Density stratification parameters are applied to depth integration calculations to project millimeter-wave volume information into a two-dimensional surface density distribution map, generating surface projection data. Adjust the scale difference between the millimeter-level resolution of the surface projection data and the sub-millimeter-level resolution of the first fused data to form a fused data pair; The laser surface weighting coefficient and the millimeter wave internal weighting coefficient are determined as the second coal flow detection ratio, and the fused data is used to perform weighted fusion on the second coal flow detection ratio to generate the second fused data.

[0031] Specifically, the three-dimensional volume information acquired by millimeter-wave radar is analyzed layer by layer to extract density distribution data of the coal flow at various depths from the surface to the interior. Based on the changes in reflection intensity and propagation loss of the millimeter-wave echo signal at different depths, combined with the frequency response characteristics, dielectric constant range (3.2 to 4.8), and loss tangent parameter (0.02 to 0.05) of the coal medium, the energy attenuation of the millimeter wave in different penetration layers is corrected, thereby deducing the coal packing density at each depth layer. The density distribution data forms a multi-layer voxel array, with each layer representing the compaction degree or particle concentration of the coal flow within a specific depth range. The multi-layer density data is modeled and processed to construct a density stratification parameter expression. The density stratification parameter expression is based on the gravity-dominated compaction trend during coal accumulation and adopts an exponential decay model, i.e., the density gradually decreases from the surface to the interior, reflecting the natural distribution state of coal particles after accumulation. Key parameters such as surface density value, maximum pile height, and density attenuation coefficient are extracted from this, and a unified depth stratification function is established. Density stratification parameters are applied to depth integral calculations, mapping millimeter-wave volume data from three-dimensional space to a two-dimensional planar coordinate system consistent with the image coordinates. The density function is integrated along the vertical direction (i.e., the depth direction) at each horizontal position to obtain the projected value of the sum of densities at all depth layers at that horizontal position, forming a two-dimensional surface density image, i.e., surface projection data. The surface projection data undergoes scale adjustment processing. Through image processing methods such as bilinear interpolation, bicubic interpolation, or convolutional upsampling, the millimeter-resolution data is resampled to the same spatial resolution and coordinate grid as the first fused data, forming a one-to-one corresponding fused data pair in spatial dimension. Based on the established spatially aligned data, to achieve data fusion and introduce comprehensive perception of the coal flow surface and internal structure, weighting coefficients are set according to the second coal flow detection ratio. Laser surface information is given a higher weight due to its high precision and low interference characteristics, for example, set to 0.7; while millimeter-wave internal information, although slightly blurred in terms of surface features, has advantages in identifying the hidden accumulation trend inside the coal pile, and is given a weight of 0.3. These two weighting parameters work together in the weighted calculation process of the fused data pairs. Specifically, at each corresponding pixel location, the laser-vision fusion data and millimeter-wave surface projection data are numerically integrated using a weighted average, so that the output simultaneously reflects the surface spatial features and volume density features at that location. The second fused data is obtained through this weighted fusion process.

[0032] In one specific embodiment, the process of applying density stratification parameters to depth integration calculations to project and convert millimeter-wave volume information into a two-dimensional surface density distribution map, and generating surface projection data, can specifically include the following steps: Establish the exponential decay function corresponding to the density stratification parameter, set the integral boundary condition from the coal flow surface to the maximum accumulation depth, and construct the volume density integral model based on the exponential decay function and the integral boundary condition. Identify the true bulk density distribution data of coal in millimeter-wave volume information; Run the volume density integral model to perform layer-by-layer depth integration on the real volume density distribution data to form a two-dimensional surface density distribution map; By comparing the grid size of the two-dimensional surface density distribution map with the spatial resolution of the laser surface measurement grid, a correspondence between millimeter-wave projection coordinates and laser measurement coordinates is established, and surface projection data is generated based on this correspondence.

[0033] Specifically, an exponential decay function corresponding to the density stratification parameters is established. This function is modeled based on the physical laws governing coal particle accumulation. Its core concept is that coal flow, under gravity, forms a gradually compacted structure from top to bottom. The surface particles are relatively loose, while the lower particles are more compacted due to the pressure from above. Therefore, the coal packing density exhibits a non-linear increasing characteristic with depth. However, from the perspective of millimeter-wave signals, this density change should manifest as an echo response that gradually decays from the surface to the interior. To simulate this process, a density stratification function ρ(h) = ρ0·exp(-γ·h / h) is constructed. max ), where ρ(h) is the density value at any depth h, ρ0 is the surface density of the coal flow, γ is the coal-specific density attenuation coefficient, and h max The maximum stacking height is defined. The density stratification function is obtained through statistical fitting of the measured dielectric properties, particle size distribution, and echo response of the coal medium, where γ reflects the difference in density gradient among different coal types under gravity compaction conditions. Integral boundary conditions are set in the model, i.e., the integration range is from h=0 (i.e., the coal flow surface) to h=h. max (i.e., the bottom of the coal flow or the deepest point detectable by millimeter waves), thus defining the spatial vertical boundary of the entire integration region and constructing a volumetric density integral model. The true volumetric density distribution data of coal is identified from the raw volumetric data of the millimeter-wave radar. Based on the energy attenuation of millimeter-wave echoes at different depth levels, and combined with the medium response model of coal, the reflection intensity is inverted. After removing background noise and interference from non-coal materials, a physically meaningful volumetric density data cube is obtained, with the data structure V(x, y, h), where x and y are the horizontal positions, and h is the depth dimension. The volumetric density cube reflects the coal density and packing conditions at different levels from the surface to the bottom. The true volumetric density distribution data is input as the integrand into the volumetric density integral model, and for each x, y coordinate from h=0 to h=h... maxLayer-by-layer integration is performed along the depth direction. The integration operation, combined with an exponential decay weighting function, emphasizes the dominant role of density near the surface in the overall projected density, while preserving the decay contribution of deeper structures. Each integration result is the surface density projection value of the corresponding x, y point on the two-dimensional plane. As the entire volume space is fully scanned and integrated, a two-dimensional surface density distribution map is formed. Scale adjustment and spatial registration operations are performed on the two-dimensional surface density map. By comparing the grid size of the two-dimensional surface density distribution map with the spatial resolution of the laser surface measurement grid, it is determined whether the spatial sampling interval of the millimeter-wave projection map overlaps or offsets with the surface grid distribution of the laser point cloud. Through image scaling, grid resampling, interpolation fitting, and other methods, the millimeter-wave projection map is reconstructed into a coordinate system consistent with the laser grid, forming a unified spatial reference. A spatial correspondence between the millimeter-wave projection coordinates and the laser measurement coordinates is constructed. Through coordinate mapping functions or common-view plane transformation, each millimeter-wave projection data point is mapped to its corresponding position in the laser point cloud coordinates, while maintaining the original density information without distortion. Based on the spatial correspondence, surface projection data aligned with the laser surface data is generated.

[0034] The process involves identifying the dielectric constant and loss tangent frequency response characteristics of coal medium in millimeter-wave volumetric information, calculating the energy attenuation compensation factor when millimeter-wave signals penetrate coal particles, and generating a true volumetric density distribution. This includes: extracting signal intensity data of different frequency components from the millimeter-wave volumetric information; measuring the propagation delay and phase change of millimeter waves in the coal medium; establishing a dielectric constant frequency response model based on the chemical composition and moisture content characteristics of coal; calculating the real and imaginary parts of the complex dielectric constant of coal medium at different depths; analyzing the variation of signal attenuation with propagation distance in the millimeter-wave volumetric information, and combining this with the geometric scattering characteristics of coal particles and dielectric loss. The mechanism establishes a correlation function between the loss tangent and the particle size and density distribution of coal, and determines the frequency-dependent loss tangent parameters. It constructs an electromagnetic wave propagation model of millimeter waves in coal particle medium, considering the multiple scattering effect between coal particles and interface reflection loss. By solving Maxwell's equations, it calculates the attenuation coefficients of millimeter waves at different frequencies, generating a frequency-depth two-dimensional attenuation compensation matrix. The frequency-depth two-dimensional attenuation compensation matrix is ​​then applied to correct the original millimeter wave detection data frequency by frequency component, compensating for signal attenuation and phase distortion caused by the coal medium, restoring the true reflection intensity of each depth layer within the coal flow, and reconstructing the true volume density distribution.

[0035] In one specific embodiment, the process of performing step S14 may specifically include the following steps: Analyze the position change information of coal flow in the second fused data, and calculate the instantaneous velocity vector and acceleration vector of coal flow during belt conveying based on the position change information; Based on instantaneous velocity vectors and acceleration vectors, coal flow trajectory prediction is performed to generate trajectory prediction data; Calculate the timestamp deviation between lidar, industrial camera and millimeter-wave radar, use motion trajectory prediction data to perform position compensation correction on the timestamp deviation, and generate a timestamp correction mapping table; The second fused data is time-aligned using a timestamp correction mapping table to obtain synchronized coal flow data.

[0036] Specifically, the position change information of the coal flow during the belt conveyor process is extracted from the second fused data. This position change information originates from the three-dimensional spatial point set fused from multiple sensors and its pose changes at different sampling times, including dynamic changes in the surface morphology, volume distribution, and internal density of the coal flow. By comparing the differences in the three-dimensional coordinates of corresponding feature points or feature regions in two consecutive frames of fused data, the displacement vector of the coal flow per unit time is calculated, and the displacement vector is removed by dividing the time interval to obtain the instantaneous velocity vector. Simultaneously, by differentiating the rate of change of the instantaneous velocity vector over time, the acceleration vector of the coal flow at the current moment is obtained. This acceleration vector is then filtered and corrected using the mechanical characteristic parameters of the conveying system to eliminate measurement noise and non-physical abrupt changes. Based on the instantaneous velocity and acceleration vectors, a motion trajectory prediction model for the coal flow is constructed. This model uses the current position, current velocity, and current acceleration of the coal flow as initial conditions and extrapolates in the time domain using kinematic formulas to predict the position distribution of the coal flow at any future time offset, thus forming motion trajectory prediction data. The timestamp deviation between LiDAR, industrial camera, and millimeter-wave radar is calculated by reading and comparing the system clocks of each sensor, combined with sampling trigger records, to accurately measure the time delay difference between them. Since the sampling frequencies of the three sensors are different (e.g., 100Hz for LiDAR, 30Hz for industrial camera, and 50Hz for millimeter-wave radar), the timestamp deviation includes both a fixed delay component and a dynamic offset in the sampling phase. Using motion trajectory prediction data, the position of one sensor at its sampling time is extrapolated or pushed back to the time reference point of another sensor, thereby performing position compensation correction on the timestamp deviation, generating a timestamp correction mapping table that records the target unified time and its position correction information corresponding to each original sampling time. The timestamp correction mapping table is used to perform time alignment operations on the second fused data, remapping the multimodal data collected by each sensor at different times to the same reference time point. Data at missing times is estimated using interpolation methods, or the data is directly moved to a unified time coordinate using a combination of translation and registration, generating synchronized coal flow data.

[0037] The process involves predicting the trajectory of coal flow based on coal flow motion state parameters and coal particle friction coefficients, generating trajectory prediction data. This includes: establishing a multibody dynamics model of coal particles during belt conveying, considering sliding friction between coal particles and the belt surface, internal friction between coal particles, and air resistance; establishing a layered friction parameter matrix based on the friction coefficient differences of coal particles of different sizes; analyzing the driving characteristics and load variation of the belt conveyor, measuring belt speed fluctuations and acceleration changes, and establishing a belt-coal flow coupled motion equation by combining coal flow weight distribution and belt tension parameters, calculating the boundary conditions of coal flow motion under belt constraints; constructing a nonlinear prediction model of coal flow motion, using a Kalman filter algorithm to fuse coal flow motion state parameters and layered friction parameter matrices, predicting the spatial coordinates and velocity of feature points at different locations in the coal flow at future times, and generating a three-dimensional trajectory prediction field; verifying the accuracy of the three-dimensional trajectory prediction field by comparing the deviation between the predicted trajectory and the actual measured trajectory, and when the deviation exceeds a preset threshold, activating an adaptive update mechanism for model parameters to dynamically adjust the friction coefficient and boundary condition parameters, and outputting corrected trajectory prediction data.

[0038] In one specific embodiment, the process of performing step S15 may specifically include the following steps: Extract the three-dimensional height distribution data of the coal flow from the synchronous coal flow data, and calculate the second-order partial derivative of the coal flow surface based on the three-dimensional height distribution data; When the second-order partial derivative exceeds the specific curvature threshold of the coal flow, the curvature anomaly region is marked and the surface curvature change rate feature is generated. The instantaneous change in coal flow volume over time in synchronous coal flow data is measured, and the instantaneous change is multiplied by the coal flow volume change sensitivity coefficient to form a volume gradient; Identify anomalous accumulation regions based on volume gradient and generate volume gradient change features; The surface curvature change rate feature and the volume gradient change feature are fused together to determine the stacking state judgment result. The corresponding coal flow morphology characteristic values ​​are calculated by considering the surface curvature, volume gradient, and surface roughness of the abnormal accumulation region. A coal flow blockage early warning signal is generated based on the results of the accumulation state determination and the characteristic values ​​of the coal flow morphology.

[0039] Specifically, three-dimensional height distribution data of the coal flow is extracted from synchronous coal flow data. This data originates from the fusion results after multi-sensor spatial registration and temporal alignment, reflecting the surface undulations and accumulation morphology of the coal flow on the conveyor belt within a unified coordinate system. By performing spatial differentiation calculations on the three-dimensional height field, the second-order partial derivatives in the conveying direction (x-axis) and transverse direction (y-axis) are calculated to obtain the curvature change rate distribution of the coal flow surface at different locations. Since the second-order partial derivatives increase significantly at abrupt changes or sharp bends in the surface, the calculation results must exceed a pre-set coal flow-specific curvature threshold (e.g., 0.15 m). -1 When the surface curvature change rate is abnormal, the region is marked as a curvature anomaly region, and surface curvature change rate feature data is generated to reflect the geometric changes such as abnormal bulging, collapse, or accumulation on the coal flow surface. The volume change characteristics of the coal flow are analyzed. By comparing the volume of the three-dimensional spatial point cloud or density grid at continuous sampling times of synchronous coal flow data, the instantaneous change in coal flow volume over time is obtained. To enhance the sensitivity of volume change features to accumulation anomalies, the instantaneous change is multiplied by a coal flow volume change sensitivity coefficient (e.g., 1.2) to form a volume gradient index. Using the spatial distribution of the volume gradient, regions with significant accumulation growth anomalies are identified, and volume gradient change features are generated. The surface curvature change rate feature and the volume gradient change feature are fused using dual criteria. During the fusion process, weighted superposition, threshold joint judgment, or classifier models are used to comprehensively analyze the two types of features on the same spatial grid or feature points. When both features simultaneously exceed their respective anomaly thresholds in the same region, the confidence of the accumulation judgment is significantly improved, thereby effectively reducing the misjudgments and omissions that may be caused by a single feature. The system uses a dual-criteria fusion method to output the coal accumulation status assessment results and classifies the coal flow accumulation risk at different locations. After obtaining the accumulation status assessment results, coal flow morphology characteristic values ​​are calculated for each area identified as abnormal accumulation. These include three core parameters: surface curvature, used to quantify the degree of curvature of the accumulation surface; volume gradient, used to reflect the rate and magnitude of volume change in the accumulation; and surface roughness, which describes the uniformity and volatility of surface particle distribution by statistically analyzing the root mean square error between the accumulation surface height value and its average value. The accumulation status assessment results and coal flow morphology characteristic values ​​are input into the coal flow blockage risk assessment module, which uses established risk classification standards and early warning generation rules to quantify the risk of the accumulation area. When the risk value is in the low range, a yellow early warning signal is generated, prompting an increase in monitoring frequency; when the risk value enters the medium range, an orange early warning signal is generated, triggering preparation for manual intervention; when the risk value reaches the high range, a red early warning signal is generated, initiating emergency shutdown and on-site cleanup measures.

[0040] In one specific embodiment, the process of generating a coal flow blockage early warning signal based on the accumulation state determination result and coal flow morphology characteristic value may specifically include the following steps: When the accumulation status determination result shows that the coal flow accumulation is abnormal, the ratio of the current accumulation height to the critical blockage height is calculated based on the coal flow morphology characteristic value to obtain the accumulation height coefficient. At the same time, the ratio of the coal flow lateral diffusion velocity to the critical diffusion velocity is calculated to obtain the diffusion velocity coefficient. The duration coefficient is calculated based on the accumulation duration. A weighted risk assessment is performed on the accumulation height coefficient, diffusion velocity coefficient, and duration coefficient to obtain the risk level, and a corresponding coal flow blockage early warning signal is output based on the risk level.

[0041] Specifically, based on the surface height data contained in the morphological feature values, the highest accumulation height of the current accumulation area is extracted and compared with a pre-set critical blockage height to obtain the accumulation height coefficient. The critical blockage height is set based on physical parameters such as the trough structure of the conveyor belt, the geometric limitations of the transfer point, and the angle of repose of coal particles. When the ratio is close to or exceeds 1, it means that the accumulation height has reached or exceeded the limit for safe operation of the conveying system. Simultaneously, the velocity information reflecting the lateral diffusion of coal flow in the morphological feature values ​​is processed to calculate the instantaneous diffusion velocity of the coal flow in the lateral direction, and this velocity is compared with a set critical diffusion velocity to obtain the diffusion velocity coefficient. The critical diffusion velocity is statistically determined by the velocity distribution of the coal flow under normal evacuation and spreading conditions. When the diffusion velocity coefficient is significantly lower than 1, it indicates that the coal flow's ability to diffuse laterally is weakened, the accumulation density tends to increase, and the risk of blockage is significantly increased. The accumulation duration coefficient is calculated based on the system's continuous monitoring data. This involves obtaining the duration of the current accumulation state and performing a logarithmic or proportional conversion with a preset baseline duration to obtain a coefficient value that reflects the cumulative effect over time. This coefficient increases significantly when the accumulation duration is long, thus holding significant weight in risk assessment. The accumulation height coefficient, diffusion velocity coefficient, and duration coefficient are used as input parameters for weighted risk assessment. The weighting coefficients are set based on statistical analysis of extensive historical operating data to reflect the different contributions of each parameter to the blockage risk. For example, the accumulation height coefficient is weighted at 0.4, the diffusion velocity coefficient at 0.35, and the duration coefficient at 0.25. These are then weighted and summed to form a single risk index. According to the assessment model, when the risk index is between 0 and 0.3, it is classified as a low-risk level, and a yellow warning signal is output, prompting operators to pay attention to the coal flow status and appropriately increase the sensor sampling frequency. When the risk index is between 0.3 and 0.7, it is classified as a medium-risk level, corresponding to an orange warning signal, and a manual intervention preparation procedure is initiated, including adjusting the belt speed and clearing localized accumulations. When the risk index exceeds 0.7, it is classified as a high-risk level, corresponding to a red warning signal, immediately triggering emergency shutdown control and initiating a rapid clearing process for the conveyor system to prevent the blockage from escalating into a large-scale shutdown accident.

[0042] The above describes the multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to embodiments of the present invention. The following describes the multi-sensor fusion device for detecting coal blockage in a coal conveyor belt according to embodiments of the present invention. Please refer to [link / reference]. Figure 2 One embodiment of the multi-sensor fusion coal conveyor belt blockage detection device of the present invention includes: The acquisition module 21 is used to acquire the laser point cloud, first visual image and millimeter wave volume information of the coal flow at the coal conveyor belt transfer point, and to restore the dust environment of the first visual image to obtain the second visual image. Registration module 22 is used to register the laser point cloud with the second visual image to obtain the first fused data; Integration module 23 is used to perform surface projection transformation on millimeter-wave volume information to obtain surface projection data, and integrate the surface projection data with the first fused data to obtain the second fused data; Time compensation module 24 is used to perform time compensation on the second fused data to obtain synchronous coal flow data; The coal flow accumulation anomaly identification module 25 is used to identify coal flow accumulation anomalies based on synchronous coal flow data and generate a coal flow blockage early warning signal.

[0043] Through the collaborative efforts of the aforementioned components, and by establishing a coal particle scattering attenuation model and calculating the light scattering characteristics using Weibull statistical laws, a dedicated image restoration technology for high-concentration dust environments at coal transfer points was achieved. This effectively solves the problem of severe image quality degradation in dusty environments under traditional visual inspection. A registration algorithm based on coal flow surface normal vector consistency constraints and contour curvature feature matching was employed to establish a sub-pixel-level precise correspondence between laser 3D points and image 2D pixels, overcoming the bottleneck of insufficient registration accuracy in existing technologies and achieving high-precision spatial unification of data from different modal sensors. By establishing a coal density stratified decreasing model and a depth integral transform algorithm, the spatial scale matching problem between millimeter-wave millimeter-level resolution and laser sub-millimeter-level resolution was innovatively solved, achieving the organic fusion of volume and surface information. A time compensation model based on coal flow trajectory prediction was established, considering the coal particle friction coefficient and belt drive characteristics, effectively eliminating the timing deviations caused by different sampling frequencies of lidar, visual cameras, and millimeter-wave radar, and achieving millisecond-level precise synchronization of multi-sensor data. By establishing a dual-criteria fusion identification algorithm based on the rate of change of surface curvature and volume gradient, and combining it with the physical characteristics of coal flow to set a dedicated weight ratio, the algorithm can accurately identify the abnormal state of coal flow accumulation, significantly improving the reliability and accuracy of blockage detection.

[0044] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0045] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the present invention and embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0046] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0047] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0048] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting coal blockage in a coal conveyor belt using multi-sensor fusion, characterized in that, include: Laser point cloud, first visual image and millimeter wave volume information of coal flow at the coal conveyor belt transfer point are collected, and dust environment is restored from the first visual image to obtain the second visual image; The laser point cloud is registered with the second visual image to obtain the first fused data; The millimeter-wave volume information is subjected to surface projection transformation to obtain surface projection data, and the surface projection data is integrated with the first fused data to obtain the second fused data; Time compensation is applied to the second fused data to obtain synchronized coal flow data; Based on the synchronous coal flow data, coal flow accumulation anomalies are identified, and a coal flow blockage early warning signal is generated.

2. The multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to claim 1, characterized in that, The process involves acquiring laser point clouds, a first visual image, and millimeter-wave volume information of the coal flow at the coal conveyor belt transfer point, and then performing dust environment reconstruction on the first visual image to obtain a second visual image, including: The coal flow at the transfer point of the coal conveyor belt is scanned by lidar to form a lidar point cloud. The coal flow is captured by an industrial camera to form a first-person visual image. The internal structure of the coal flow is detected by millimeter-wave radar to form millimeter-wave volume information. Based on the surface ranging data in the laser point cloud and the depth information of the corresponding pixels in the first visual image, a dust thickness mapping map is established. Based on the dust thickness mapping, the light scattering characteristics of coal particles of different sizes are calculated to determine the scattering attenuation coefficient and the surface reflectivity correction value. Using the scattering attenuation coefficient and the surface reflectivity correction value, the brightness and contrast of each pixel in the first visual image are corrected to obtain the second visual image.

3. The multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to claim 1, characterized in that, The step of registering the laser point cloud with the second visual image to obtain the first fused data includes: The laser normal vector is calculated based on the neighborhood geometry of each three-dimensional coordinate point in the laser point cloud, and the visual normal vector at the corresponding position is derived based on the pixel gradient change of the second visual image. Establish a consistency constraint relationship between the laser normal vector and the visual normal vector; Based on the consistency constraint relationship, the first matching point pair that satisfies the coal flow surface continuity condition is identified; Extract the curvature variation features of the first matching point pair on the coal flow surface profile, and determine the corresponding second matching point pair based on the curvature variation features; Set the distance matching weight, curvature matching weight, and normal vector matching weight as the first coal flow detection ratio, and use the second matching point pair and the first coal flow detection ratio to calculate the optimal spatial transformation relationship; Based on the optimal spatial transformation relationship, the laser three-dimensional coordinates and the image two-dimensional coordinates are mapped to form the first fused data.

4. The multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to claim 1, characterized in that, The step of performing a surface projection transformation on the millimeter-wave volume information to obtain surface projection data, and integrating the surface projection data with the first fused data to obtain second fused data, includes: The density distribution data of the coal flow from the surface to the interior at each depth layer is extracted from the millimeter-wave volume information, and the density stratification parameters are calculated based on the density distribution data; The density stratification parameters are applied to the depth integration operation to project and convert the millimeter-wave volume information into a two-dimensional surface density distribution map, thereby generating surface projection data. Adjust the scale difference between the millimeter-level resolution of the surface projection data and the sub-millimeter-level resolution of the first fused data to form a fused data pair; The laser surface weighting coefficient and the millimeter wave internal weighting coefficient are determined as the second coal flow detection ratio, and the fused data is used to perform weighted fusion with the second coal flow detection ratio to generate the second fused data.

5. The multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to claim 4, characterized in that, The step of applying the density stratification parameters to depth integration calculation, projecting the millimeter-wave volume information into a two-dimensional surface density distribution map, and generating surface projection data includes: Establish the exponential decay function corresponding to the density stratification parameter, set the integral boundary condition from the coal flow surface to the maximum accumulation depth, and construct the volume density integral model based on the exponential decay function and the integral boundary condition. Identify the true bulk density distribution data of coal in the millimeter-wave volume information; The volume density integral model is run to perform layer-by-layer depth integration on the real volume density distribution data to form a two-dimensional surface density distribution map. By comparing the grid size of the two-dimensional surface density distribution map with the spatial resolution of the laser surface measurement grid, a correspondence between millimeter-wave projection coordinates and laser measurement coordinates is established, and surface projection data is generated based on the correspondence.

6. The multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to claim 1, characterized in that, The step of performing time compensation on the second fused data to obtain synchronized coal flow data includes: Analyze the position change information of the coal flow in the second fused data, and calculate the instantaneous velocity vector and acceleration vector of the coal flow during the belt conveying process based on the position change information; Based on the instantaneous velocity vector and the acceleration vector, the coal flow trajectory is predicted to form trajectory prediction data; Calculate the timestamp deviation value between lidar, industrial camera and millimeter-wave radar, use the motion trajectory prediction data to perform position compensation correction on the timestamp deviation value, and generate a timestamp correction mapping table; The timestamp correction mapping table is used to perform time alignment on the second fused data to obtain synchronized coal flow data.

7. The multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to claim 1, characterized in that, The step of identifying coal flow accumulation anomalies based on the synchronized coal flow data and generating a coal flow blockage early warning signal includes: Extract the three-dimensional height distribution data of the coal flow from the synchronous coal flow data, and calculate the second-order partial derivative of the coal flow surface based on the three-dimensional height distribution data; When the second-order partial derivative exceeds the coal flow-specific curvature threshold, the curvature anomaly region is marked and a surface curvature change rate feature is generated. The instantaneous change in coal flow volume over time in the synchronous coal flow data is measured, and the instantaneous change is multiplied by the coal flow volume change sensitivity coefficient to form a volume gradient; Based on the volume gradient, abnormal accumulation regions are identified and volume gradient change features are generated; The surface curvature change rate feature and the volume gradient change feature are fused using dual criteria to determine the stacking state determination result; The corresponding coal flow morphology characteristic values ​​are calculated by using the surface curvature, volume gradient, and surface roughness of the abnormal accumulation area. A coal flow blockage early warning signal is generated based on the accumulation state determination result and the coal flow morphology characteristic value.

8. The multi-sensor fusion method for detecting coal blockage in a coal conveyor belt according to claim 7, characterized in that, The step of generating a coal flow blockage early warning signal based on the accumulation state determination result and the coal flow morphology characteristic value includes: When the accumulation state determination result shows that the coal flow accumulation is abnormal, the ratio of the current accumulation height to the critical blockage height is calculated based on the coal flow morphology characteristic value to obtain the accumulation height coefficient. At the same time, the ratio of the coal flow lateral diffusion velocity to the critical diffusion velocity is calculated to obtain the diffusion velocity coefficient. The duration coefficient is calculated based on the accumulation duration. A risk assessment is performed on the accumulation height coefficient, the diffusion rate coefficient, and the duration coefficient to obtain a risk level, and a corresponding coal flow blockage early warning signal is output based on the risk level.

9. A multi-sensor fusion coal conveyor belt blockage detection device, characterized in that, The coal conveyor belt blockage detection method for performing multi-sensor fusion as described in any one of claims 1-8, wherein the multi-sensor fusion coal conveyor belt blockage detection device comprises: The acquisition module is used to acquire laser point cloud, first visual image and millimeter wave volume information of coal flow at the coal conveyor belt transfer point, and to restore the dust environment of the first visual image to obtain the second visual image. A registration module is used to register the laser point cloud with the second visual image to obtain first fused data; An integration module is used to perform surface projection transformation on the millimeter-wave volume information to obtain surface projection data, and to integrate the surface projection data with the first fused data to obtain second fused data; The time compensation module is used to perform time compensation on the second fused data to obtain synchronized coal flow data; The coal flow accumulation anomaly identification module is used to identify coal flow accumulation anomalies based on the synchronous coal flow data and generate a coal flow blockage early warning signal.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when run by a processor, causes the processor to execute the multi-sensor fusion method for detecting coal blockage in a coal conveyor belt as described in any one of claims 1 to 8.

Images