Coal quantity detection method and detection device based on laser radar and vision fusion

Abstract

The invention provides a coal quantity detection method and device based on laser radar and vision fusion, and belongs to the technical field of coal quantity detection of underground coal mine conveying belts. According to the technical scheme, the coal quantity detection method based on laser radar and vision fusion comprises the following steps: S1, sensor joint calibration; s2, data acquisition; s3, identifying a coal quantity area; s4, complementing the depth image; and S5, calculating the coal quantity. The invention further provides a coal quantity detection device which comprises a speed sensor, a laser radar, an industrial camera, a computer and a human-computer interface. The method has the beneficial effects that through multi-sensor data fusion and a deep learning algorithm, underground complex environment interference is overcome, and high-precision real-time detection of the coal quantity is realized; the detection device has the explosion-proof and dust-resistant characteristics, and the detection method comprises environment self-adaptive correction and lightweight calculation, and can be widely applied to intelligent mine transportation systems.

Description

Technical Field

[0001] The present invention relates to the technical field of coal quantity detection for underground conveyor belts in coal mines, and particularly to a coal quantity detection method and detection device based on the fusion of lidar and vision. Background Art

[0002] Efficient and accurate transportation is of indispensable significance to the continuous production of coal mines. However, affected by factors such as the volume of coal material and the running speed of the belt, the conveying capacity of belt conveyors has certain uncertainties and dynamic variations. Traditional contact coal flow detection devices are mainly electronic belt scales and nuclear belt scales. Electronic belt scales have limitations such as high cost, complex maintenance, and large environmental interference. Nuclear belt scales involve radioactive substances and are difficult to meet the development needs of modern industrial greenization and intelligentization. In recent years, with the rapid development of vision technology, non-contact detection technology has gradually become a research hotspot in the field of coal quantity detection due to its characteristics of high efficiency, safety, and strong adaptability. The environment in underground coal mines is relatively harsh, with a large amount of dust and water mist. Single lidar and single industrial camera modes often cannot perform high-precision coal quantity detection on the coal flow of underground belt conveyors. By fusing lidar and industrial cameras, the deficiencies of single sensors can be made up for, data on the surrounding environment can be collected better, and the detection accuracy of the coal quantity of belt conveyors can be improved. Summary of the Invention

[0003] The purpose of the present invention is to overcome the problems in the background art and provide a detection method and detection device based on the fusion of lidar and vision. This detection method overcomes the interference of complex underground environments through multi-sensor data fusion and deep learning algorithms, and realizes high-precision real-time detection of coal quantity. This detection device has explosion-proof and dust-resistant characteristics. The detection method includes environmental adaptive correction and lightweight calculation, and can be widely applied to intelligent mine transportation systems.

[0004] In order to achieve the above-mentioned invention purpose, the technical solution adopted by the present invention is specifically as follows: A coal quantity detection method based on the fusion of lidar and vision includes the following steps:

[0005] S1. Joint calibration of sensors;

[0006] S2. Data acquisition: Use industrial cameras, lidar, and speed sensors to collect information related to coal flow, providing basic data for subsequent analysis;

[0007] S3. Identification of coal quantity area, determining the area where the coal flow is located, preparing for subsequent calculations;

[0008] S4. Depth Image Completion: A depth image completion network guided by a color map is adopted. The depth image completion network includes a color-dominated branch and a depth-dominated branch. Both the color-dominated branch and the depth-dominated branch are encoder-decoder structures and are composed of convolutional layers, residual blocks, and deconvolutional layers.

[0009] S5. Coal Quantity Calculation: After preprocessing the depth map, it is converted into a point cloud, and the coal quantity is calculated through an algorithm.

[0010] Further, in step S1, the sensor joint calibration is as follows:

[0011] Establish a LiDAR coordinate system The origin is the physical center of the LiDAR The X-axis points horizontally in the running direction of the belt conveyor, The Y-axis is horizontally perpendicular to the X-axis and points to the right side of the conveyor, The Z-axis points vertically upward from the ground;

[0012] Establish an industrial camera coordinate system , the origin is the optical center of the camera The X-axis points horizontally to the right corresponding to the U-axis of the pixel coordinate system, The Y-axis points vertically downward corresponding to the V-axis of the pixel coordinate system, The Z-axis is the optical axis direction;

[0013] Pixel coordinate system The origin is the upper left corner of the image, The U-axis increases horizontally by pixels, The V-axis increases vertically by pixels;

[0014] World coordinate system The origin is a custom reference point, The X-axis is along the running direction of the conveyor, The Y-axis is perpendicular to the conveyor plane, The Z-axis points vertically upward from the ground;

[0015] A joint calibration model of the LiDAR and the industrial camera is established through the rotation matrix R and the translation vector T. The formula is: where is the corresponding pixel coordinate, is the 3D point cloud coordinate of the LiDAR. The internal parameter matrix K of the industrial camera is known data. The rotation matrix R and the translation vector t from the LiDAR to the industrial camera are unknown parameters. By solving this external parameter matrix, the LiDAR point cloud is projected onto the image plane to achieve multi-modal data space alignment;

[0016] The internal parameter matrix K of the industrial camera includes the focal length and the principal point coordinates .

[0017] Further, in step S2, the data acquisition specifically includes the following steps:

[0018] S2.1. Industrial camera image acquisition: Set the frame rate of the industrial camera , , the conveyor speed , the length of the coal flow covered by a single frame of image . Since each sampling length , the industrial camera skips frames to acquire image information, and the number of skipped frames is frames. The image data starts to be acquired from the th frame, and its data set is ;

[0019] S2.2. LiDAR point cloud acquisition: Use a 128-line LiDAR to acquire the coal flow height information. The LiDAR frame rate f = 30 frames / s, which is synchronized with the industrial camera for continuous sampling of 10 frames. Starting from the th frame, the point cloud data set is represented as ;

[0020] S2.3. Speed data acquisition: The speed sensor measures the real-time operating speed of the belt conveyor.

[0021] Further, in step S3, the coal quantity area identification specifically includes the following steps:

[0022] S3.1. LiDAR point cloud area separation: Acquire the complete three-dimensional point cloud of the conveyor belt in the no-load state

[0023] . With the help of the Poisson reconstruction algorithm, construct a high-precision reference surface, and perform voxel-level registration operations on the acquired load point cloud data set and the reference point cloud . Eliminate the points in the point cloud data set that cannot be matched with the reference through a spatial filtering strategy, and separate the coal flow area and non-coal area on the conveyor belt to obtain a point cloud data set containing coal flow information;

[0024] S3.2. Industrial camera image area positioning: Acquire the RGB color image of the conveyor belt in the no-load state, and extract the surface texture features of the conveyor belt and the conveyor belt area . When processing the acquired load color map data set I, for the part that is consistent with the no-load conveyor belt area , perform texture comparison, and use the structural similarity index to locate the coal flow area of the belt conveyor. The specific determination method is as follows: Suspicious coal area.

[0025] Further, in step S4, the depth image completion specifically includes the following steps:

[0026] S4.1. Color-dominated branch processing: The input of the color-dominated branch is the color map and the sparse depth map. The spatial pyramid pooling module is used to capture multi-scale context information, and the skip connection is used to fuse the low-level edge features. The input images RGB and depth D are concatenated and operated through a 3×3 convolutional layer to obtain multi-dimensional features, and the RGB features and geometric features are feature-fused and feature-extracted to obtain different-scale fused features;

[0027] S4.2. Depth-dominated branch processing: The depth-dominated branch takes the rough depth map and the sparse depth map output by the color-dominated branch as the input, extracts sparse features through a 3×3 convolutional layer, and fuses them with the RGB geometric features extracted by the color-dominated branch. The two branches respectively predict the depth map and the confidence map , and finally output the fused rough depth map:

[0028] , represents a pixel,

[0029] represents the fused depth map, represent the CD depth map and the DD depth map in sequence,

[0030] represent the CD confidence and the DD confidence in sequence;

[0031] S4.3. Dynamic optimization and acceleration:

[0032] The dynamic optimization and acceleration network takes the predicted rough depth map as the input, learns to generate variable convolution kernel parameters related to the spatial position through the convolutional layer, and realizes the adaptive receptive field adjustment;

[0033] Integrate the conditional space propagation mechanism, spread the effective observations of the sparse point cloud to the non-observed area through the bidirectional propagation strategy, and at the same time introduce the spatial attention module to generate the attention weight matrix based on the depth gradient map and weight-enhance the features in the edge area;

[0034] The hierarchical loss supervision strategy synchronously constrains the global depth error and the local structural similarity to ensure the accuracy of the high-frequency edges;

[0035] The CSPN results predicted by the variable convolution kernel multiple times are weighted and summed to obtain the final predicted fine depth map of the network.

[0036] Further, in step S5, the coal quantity calculation specifically includes the following steps:

[0037] S5.1. Depth map preprocessing:

[0038] Based on the refined depth map repaired by the depth completion network, preprocess the no-load depth map and the loaded depth map with coal material;

[0039] The no-load depth map eliminates environmental interference by establishing a background template through the Gaussian mixture model;

[0040] The loaded depth map adopts a combined strategy of bilateral filtering and statistical outlier removal algorithm to suppress noise;

[0041] The preprocessed depth data is converted into a three-dimensional point cloud through a non-linear mapping from polar coordinates to Cartesian coordinates;

[0042] S5.2. Point cloud processing and volume calculation:

[0043] Point cloud stitching uses an improved iterative closest point algorithm, and reduces the initial registration error by introducing a feature point matching strategy with curvature weighting. The registered point cloud generates a three-dimensional reconstruction model through Poisson reconstruction;

[0044] The three-dimensional reconstruction model is sliced in layers along the vertical direction with a fixed layer spacing d and slice thickness for hierarchical slicing processing, and the hierarchical density is dynamically encrypted in the region of curvature mutation;

[0045] By extracting the closed contour boundary in the slice projection plane and performing numerical integration on the cross-sectional area based on Green's formula, , the total volume V is realized by discrete integral accumulation .

[0046] The present invention also provides a coal quantity detection device, including a speed sensor, a lidar, an industrial camera, a computer and a human-machine interface. The speed sensor is installed on the belt conveyor for measuring the real-time running speed of the belt conveyor. The industrial camera is installed directly above the belt conveyor for collecting image data of the coal flow in combination with the belt conveyor speed information. The lidar is used for real-time collecting the coal flow data information passing through the belt conveyor. The computer is used for integrating data, performing depth image completion and coal quantity calculation processing. The human-machine interface includes a display screen and an input module, and the human-machine interface is used for displaying data and operation control.

[0047] Further, the signal output end of the speed sensor, the signal output end of the lidar, and the signal output end of the industrial camera are all connected to the signal input end of the computer, and the signal control end of the computer is connected to the signal end of the human-machine interface.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] 1. This technical solution adopts a non-contact detection method that combines lidar and vision. It does not need to directly contact the coal flow. Through steps such as sensor joint calibration, data sampling, depth image completion, and coal volume calculation, it avoids the disadvantages of traditional contact detection, reduces equipment costs and maintenance difficulties, meets the requirements of modern industrial green development, and overcomes the technical problems in the prior art that electronic belt scales have high costs, complex maintenance, and are greatly affected by the environment, and nuclear belt scales involve radioactive substances.

[0050] 2. This method combines lidar to provide accurate depth information and industrial cameras to provide rich texture and color information. The two complement each other. Reasonable parameter settings are made in the data sampling link to ensure effective information collection. The depth image completion network improves the image quality, effectively reduces the influence of environmental factors, and improves the detection accuracy and real-time performance, overcoming the technical problems in the prior art that monocular vision detection has large errors, binocular vision is greatly affected by environmental light and dust, and has poor real-time performance.

[0051] 3. This technical solution uses lidar and vision fusion detection. The high-precision three-dimensional coordinates and depth information of lidar, as well as the texture and color information of vision, are more stable and reliable in the complex underground coal mine environment; through accurate algorithms such as coal volume area recognition, depth image completion, and coal volume calculation, it ensures that the measurement results are stable and accurate, effectively overcoming the technical problems in the prior art that ultrasonic detection is greatly affected by environmental factors and the measurement is unstable. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention.

[0053] Figure 1 It is a schematic structural diagram of the device of the present invention.

[0054] Figure 2 It is a schematic diagram of the principle of joint calibration of lidar and industrial camera.

[0055] Figure 3 It is a schematic diagram of the coal volume calculation process.

[0056] Figure 4 It is a schematic diagram of the complete process of the detection method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0058] Embodiment

[0059] As Figure 1 shown, this embodiment provides a coal quantity detection device, which mainly includes a 128-line lidar, an industrial camera, a computer, a human-machine interface, and a speed sensor;

[0060] Connection and position relationship: The speed sensor measures the real-time running speed of the conveyor and transmits the data to the computer. The industrial camera collects coal flow image data in combination with the speed information and transmits the image data to the computer. The lidar collects coal flow height information and also transmits the data to the computer. The computer processes these data and displays the relevant results on the human-machine interface.

[0061] Functions of each component: The speed sensor obtains the speed of the conveyor; the industrial camera collects the surface texture and color information of the coal flow; the lidar obtains the coal flow height information; the computer integrates the data and performs processing such as depth image completion and coal quantity calculation; the human-machine interface is used to display data and perform operation control.

[0062] The speed sensor is installed on the belt conveyor and is used to measure the real-time running speed of the belt conveyor. The industrial camera is installed directly above the belt conveyor and collects image data of the coal flow in combination with the speed information of the belt conveyor. The lidar can obtain the coal flow height information transported by the belt conveyor at the corresponding moment by collecting the coal flow data information passing through the belt conveyor in real time. The computer performs corresponding depth image completion on the depth map based on the collected image information and coal flow height information, combined with the speed and time information of the belt conveyor; the depth image completion adopts the PENet network architecture, and based on the characteristics of the collected lidar and industrial camera data information, experiments on the extraction and fusion mechanism of color, sparse, and geometric features are carried out, and a corresponding confidence evaluation mechanism is constructed to realize depth image completion through effective fusion of the depth map; the coal quantity detection establishes a coal quantity calculation model, calculates the depth difference between the depth maps under no-load and load conditions, slices the point cloud data, segments the coal material contour boundary, and realizes the cross-sectional area calculation; according to the difference between the calculated area value and the actual value, the volume calculation result is corrected in combination with the error feedback mechanism, and the total volume of the coal quantity of the belt conveyor is accurately measured by accumulating the cross-sectional areas. This system fuses the lidar and industrial camera information, thereby complementing and improving the integrity and accuracy of the depth information, and can work stably under harsh working conditions and environments such as underground coal mines, and accurately calculate the volume of coal transported by the belt conveyor.

[0063] As Figures 2 - 4 shown, the present invention provides a coal quantity detection method based on lidar and vision fusion, and the specific content is as follows:

[0064] S1. Sensor joint calibration:

[0065] Establish the coordinate system of the lidar , with the origin at the physical center of the lidar The x-axis points horizontally in the running direction of the belt conveyor The y-axis is horizontally perpendicular to the x-axis and points to the right side of the conveyor The z-axis points vertically upward from the ground; establish the coordinate system of the industrial camera , with the origin at the optical center of the camera The u-axis points horizontally to the right corresponding to the U-axis of the pixel coordinate system The v-axis points vertically downward corresponding to the V-axis of the pixel coordinate system The w-axis is in the direction of the optical axis; the pixel coordinate system has the origin at the upper left corner of the image The u-axis increases in pixels horizontally The v-axis increases in pixels vertically; the world coordinate system has the origin at a custom reference point The X-axis is along the running direction of the conveyor The Y-axis is perpendicular to the plane of the conveyor The Z-axis points vertically upward from the ground

[0066] Establish a joint calibration model of the lidar and the industrial camera through the rotation matrix R and the translation vector T. The formula is: , where is the corresponding pixel coordinate is the three-dimensional point cloud coordinate of the lidar. The internal parameter matrix K of the industrial camera (including the focal length and the principal point coordinate ) is known data. The rotation matrix R and the translation vector t from the lidar to the industrial camera are unknown parameters. By solving this external parameter matrix, project the lidar point cloud onto the image plane to achieve multi-modal data space alignment

[0067] S2. Data acquisition: Use various devices to collect coal flow-related information from different dimensions to provide basic data for subsequent analysis

[0068] S2.1. Industrial camera image acquisition: The industrial camera is installed directly above the belt conveyor. Combine the speed information of the belt conveyor to collect coal flow images. Set the frame rate 、 of the industrial camera. When the conveyor speed is, the length of the coal flow covered by a single frame of image

[0069] . Since each sampling length , so the industrial camera skips frames to collect image information. The number of frames between skips is frames. The image data starts to be collected from the th frame, and its data set is ;

[0070] S2.2, LiDAR point cloud collection: 128-line LiDAR collects coal flow height information, LiDAR frame rate f=30 frames / s, synchronized with industrial camera, continuous sampling 10 frames, if from the Frame collection starts, the point cloud dataset can be expressed as ;

[0071] S2.3. Speed data collection: The speed sensor is installed on the belt conveyor to measure the real-time running speed of the belt conveyor.

[0072] S3. Coal flow area identification: Using LiDAR point cloud data and industrial camera image data, we accurately determine the coal flow area to prepare for subsequent calculations.

[0073] S3.1. LiDAR point cloud region separation: Collecting the complete 3D point cloud of the conveyor belt in the empty state , using the Poisson reconstruction algorithm, a high-precision reference surface is constructed to collect the load point cloud data set , and the benchmark point cloud Perform voxel-level registration operations and eliminate point cloud datasets through spatial filtering strategies Medium and Benchmark For points that cannot be matched, the coal flow area and the non-coal area on the conveyor belt are accurately separated to obtain a point cloud dataset containing only coal flow information;

[0074] S3.2. Industrial camera image area positioning: Collect RGB color images of the conveyor belt in the empty state and extract the surface texture features of the conveyor belt. and conveyor belt area When processing the collected load color map data set I, the area with no-load conveyor belt is For the consistent parts, texture comparison is performed and the coal flow area of the belt conveyor is located by using the structural similarity index (SSIM). The specific determination method is as follows: Suspected coal area.

[0075] S4. Deep Image Completion: A color-map-guided deep image completion network is used. The network consists of a color-dominant branch and a depth-dominant branch. Each branch is an encoder-decoder structure consisting of a convolutional layer, a residual block, and a deconvolutional layer.

[0076] S4.1. Color-dominant branch processing: The input of the color-dominant branch is the color map and the sparse depth map. The Atrous Spatial Pyramid Pooling (ASPP) module is used to capture multi-scale context information, and skip connections are utilized to fuse low-level edge features. The input images RGB and D are concatenated and processed through a 3×3 convolutional layer to obtain multi-dimensional features. Then, the RGB features and geometric features are fused and feature-extracted to obtain fused features at different scales;

[0077] S4.2. Depth-dominant branch processing: The depth-dominant branch takes the coarse depth map and the sparse depth map output by the color-dominant branch as inputs. Sparse features are extracted through a 3×3 convolutional layer and fused with the RGB geometric features extracted by the color-dominant branch. The two branches respectively predict the depth map and the confidence map through transposed convolution, and finally output the fused rough depth map:

[0078] , represents a pixel,

[0079] represents the fused depth map, successively represent the CD depth map and the DD depth map,

[0080] successively represent the CD confidence and the DD confidence;

[0081] S4.3. Dynamic optimization and acceleration: The Dynamic Optimization and Acceleration Network (DA-CSPN++) takes the predicted rough depth map as input, learns to generate variable convolution kernel parameters related to spatial positions through convolutional layers to achieve adaptive receptive field adjustment. This module integrates the Conditional Spatial Propagation (CSP) mechanism, spreads the effective observations of the sparse point cloud to the unobserved areas through a two-way propagation strategy, and at the same time introduces the Spatial Attention Module (SAM) to generate an attention weight matrix based on the depth gradient map to weight and enhance the features in the edge regions. The hierarchical loss supervision strategy synchronously constrains the global depth error and local structural similarity to ensure the accuracy of high-frequency edges. The final predicted fine depth map of the network can be obtained by weighted summation of the CSPN results predicted by multiple variable convolution kernels.

[0082] S5. Coal quantity calculation: After preprocessing the depth map, it is converted into a point cloud, and the coal quantity is calculated through a specific algorithm;

[0083] S5.1, Depth Map Preprocessing: Based on the refined depth map repaired by the depth completion network, preprocess the no-load depth map and the loaded depth map with coal. The no-load depth map eliminates environmental interference by establishing a background template through the Gaussian Mixture Model (GMM). The loaded depth map uses a combined strategy of bilateral filtering and Statistical Outlier Removal (SOR) algorithm to suppress noise. The preprocessed depth data is converted into a three-dimensional point cloud through a non-linear mapping from polar coordinates to Cartesian coordinates to increase the point cloud density;

[0084] S5.2, Point Cloud Processing and Volume Calculation: The point cloud stitching uses an improved Iterative Closest Point (ICP) algorithm. By introducing a feature point matching strategy weighted by curvature, the initial registration error is reduced. The registered point cloud is generated into a three-dimensional reconstruction model through Poisson reconstruction. The model is sliced in the vertical direction (Z-axis) at a fixed layer spacing d and slice thickness for hierarchical slicing. In regions with sudden curvature changes (such as the edge of the coal pile), the hierarchical density is dynamically encrypted. By extracting the closed contour boundary within the slice projection plane and numerically integrating the cross-sectional area based on Green's formula, , the total volume V is realized by cumulative discrete integration .

[0085] S6, Experimental Verification and Demonstration: Verify the effectiveness of the detection method through experiments and display relevant data with the help of a human-machine interface;

[0086] S6.1, On-site Experimental Setup: Conduct experiments on-site. The point cloud depth map is collected using an ouster OSO-GEN2.0 lidar. The laser wavelength is 905nm, the scanning frequency is 30HZ, and its vertical field of view is 90° (-45° to +45°). The color map is collected using an InterRealSense D457 industrial camera. The resolution of the industrial camera is 640×480, the frame rate is 30HZ, and the working distance is 0.06m to 6m. The data is processed based on the VisualStudio2022, ROS, and ubuntu20.04 operating systems;

[0087] S6.2, Human-Machine Interface Function: Design a dedicated human-machine interface to build a monitoring system integrating data feedback, visual presentation, and early warning control. The computer transmits the coal quantity calculation value to the human-machine interface in real time. The interface generates a coal quantity change curve for each period through a data processing module to visually present the dynamic trend of the coal flow volume on the conveyor belt in a graphical manner. The interface integrates a real-time data display function and can dynamically present the real-time volume data of the coal flow. When the monitored value exceeds the preset coal flow threshold, an alarm mechanism and shutdown will be triggered; in addition, the supporting historical query interface supports retrieving historical monitoring data during the operation of the conveyor belt;

[0088] S6.3. Experimental result verification: By comparing the original image processing results processed by different algorithms, comparing the coal volume detection errors of three methods: lidar, vision industrial camera, and lidar-vision fusion, and comparing the coal flow volume curves detected by the electronic belt scale, verify the accuracy, precision, and reliability of this detection method;

[0089] ①. Comparison of multi-algorithm depth image completion effects

[0090] In the comparison of multi-algorithm depth image completion effects, the color-dominated + depth-dominated dual-branch network of the present invention has a 20% - 30% improvement in the completion accuracy in complex texture areas and a 15% reduction in edge error compared with traditional single-branch algorithms and pure vision algorithms. After enabling the DA-CSPN++ dynamic optimization module, the inference speed reaches 22 FPS, and the small target missed detection rate drops from 18% to 5%;

[0091] ②. Comparison of detection errors of multi-sensor solutions

[0092]

[0093] In an extreme environment with a dust concentration > 500 mg / m³, the error of the fusion solution is reduced by more than 50% compared with the single modality;

[0094] The three-dimensional structure constraint provided by the lidar can correct the misdetection caused by occlusion in vision, while the visual texture information makes up for the missed detection of the lidar on coal blocks with low reflectivity;

[0095] ③. Comparison and verification with the electronic belt scale

[0096] 1. Dynamic load tracking comparison

[0097] Experimental design: Set the conveyor belt load to increase stepwise from 50 t / h to 300 t / h, and simultaneously record the volume curves of the fusion solution and the electronic belt scale;

[0098] Result analysis: The Pearson correlation coefficient between the curve of the fusion solution and the electronic belt scale reaches 0.98, and the dynamic lag time < 0.5 s. The error of the electronic belt scale reaches ±5% during load mutation, while the fusion solution controls the error within ±2% through the inertial navigation compensation algorithm;

[0099] 2. Long-term operation stability comparison

[0100] Test period: Continuously operate for 30 days, and record the cumulative coal volume deviation every day;

[0101] Result: The cumulative error of the electronic belt scale reaches 1.2% due to belt wear and needs to be calibrated regularly. The cumulative error of the fusion solution is only 0.3%, thanks to the environment adaptive correction algorithm.

[0102] In summary, the specific implementation of the present invention can be summarized as follows:

[0103] 1. Build a detection system: Construct a coal quantity detection system including a 128-line lidar, an industrial camera, a computer, a touch screen, and a speed sensor. Install the lidar (tilt angle 30° - 45°) and the industrial camera (vertically looking down) on the conveyor support. The distance between the two is ≤ 0.5 m to ensure field of view overlap. The speed sensor measures the conveyor belt speed, the industrial camera captures coal flow images, the lidar obtains coal flow height information, and the computer completes data processing and depth image completion, thereby realizing accurate calculation of coal quantity.

[0104] 2. Apply the detection principle

[0105] Jointly calibrate sensors: Define the lidar, industrial camera, pixel, and world coordinate systems. Solve the extrinsic matrix through the rotation matrix R and translation vector T, project the lidar point cloud onto the image plane, and unify the coordinate systems to fuse data; Collaboratively collect data: The industrial camera samples at intervals, determines the frame skipping interval according to the frame rate and conveyor belt speed. For example, when the frame rate is 30 frames / s and the speed is 3 m / s, the length of the coal flow covered by a single frame is 0.1 m, and the frame skipping interval is 10 frames; The lidar and the industrial camera collect data synchronously, maintaining the same sampling frequency, and accurately identify the coal quantity area: For the lidar point cloud data, build a reference surface using the Poisson reconstruction algorithm, register the loaded and unloaded point clouds, and filter out invalid points to separate the coal flow area; For the industrial camera image, extract the texture and regional features of the unloaded conveyor belt, and use the structural similarity index (SSIM) to locate the coal flow area. When SSIM < 0.7, it is determined as a suspected coal area; Complete the depth image: Adopt a color map-guided depth image completion network. Its color-dominated branch captures multi-scale information and fuses edge features, and the depth-dominated branch fuses sparse and RGB-geometry features, outputting a fused rough depth map. Then, adjust the receptive field through the dynamic optimization acceleration network (DA-CSPN++) to propagate the observations and enhance the edge features to obtain a fine depth map; Accurately calculate the coal quantity: Preprocess the unloaded and loaded depth maps, respectively use the Gaussian mixture model, bilateral filtering, and statistical outlier removal algorithm to eliminate interference and suppress noise; Convert the depth data into a three-dimensional point cloud, use the improved iterative closest point (ICP) algorithm to splice the point cloud, calculate the cross-sectional area according to Green's formula after hierarchical slicing, and accumulate to obtain the total volume.

[0106] 3. Conduct experimental verification: Select the ouster OSO-GEN2.0 lidar and the InterRealSense D457 industrial camera to collect data, process the data in the Visual Studio 2022, ROS, and ubuntu 20.04 systems, design a human-machine interface to monitor the change of coal flow volume, compare the depth maps processed by different algorithms and the coal quantity volume measurement results of different detection methods, and verify the accuracy and reliability of the method.

[0107] The intelligent algorithm is designed as follows

[0108] 1. Sensor joint calibration

[0109] Import numpy as np from open3d import *

[0110] def calibrate_lidar_camera(lidar_points, img_points, K):

[0111] """

[0112] lidar_points: LiDAR point cloud (N, 3)

[0113] img_points: Corresponding pixel coordinates (N, 2)

[0114] K: Industrial camera internal parameter matrix (3, 3)

[0115] Returns: Extrinsic parameter matrix [R|t] (3, 4)

[0116] """

[0117] # Normalize point cloud coordinates (remove translation)

[0118] lidar_centroid = np.mean(lidar_points, axis = 0)

[0119] lidar_normalized = lidar_points - lidar_centroid

[0120] # Convert pixel coordinates to industrial camera coordinates (homogenize)

[0121] img_hom = np.hstack((img_points, np.ones((img_points.shape[0], 1))))

[0122] camera_points = np.linalg.inv(K) @ img_hom.T # (3, N)

[0123] camera_normalized = camera_points - np.mean(camera_points, axis = 1, keepdims = True)

[0124] # Solve for rotation matrix (SVD)

[0125] H = lidar_normalized.T @ camera_normalized

[0126] U, S, Vt = np.linalg.svd(H)

[0127] R = Vt.T @ U.T

[0128] t = -R @ lidar_centroid.reshape(3, 1) + camera_points.mean(axis = 1, keepdims = True)

[0129] return np.hstack((R, t)) # (3,4)

[0130] 2. Data Sampling and Synchronization

[0131] def sample_data(camera_frames, lidar_frames, conveyor_speed, frame_interval = 10):

[0132] """

[0133] Skip-frame sampling according to the conveyor belt speed

[0134] """

[0135] sampled_camera = camera_frames[::frame_interval]

[0136] sampled_lidar = lidar_frames[::frame_interval]

[0137] return sampled_camera, sampled_lidar

[0138] 3. Coal Quantity Region Identification

[0139] from skimage.metrics import structural_similarity as ssim

[0140] def detect_coal_region(load_img, empty_img):

[0141] """

[0142] load_img: Load RGB image (H, W, 3)

[0143] Empty_img: Empty RGB image (H, W, 3)

[0144] Returns: coal area mask (H, W, 1)

[0145] """

[0146] gray_load=rgb2gray(load_img)

[0147] gray_empty=rgb2gray(empty_img)

[0148] ssim_map=ssim(gray_load,gray_empty,full=True)[1]

[0149] coal_mask=(ssim_map<0.7).astype(np.uint8)#SSIM<0.7 is determined to be coal area

[0150] returncoal_mask

[0151] 4. Depth Completion Grid

[0152] Importtorchimporttorch.nnasnn

[0153] classDepthFusionNet(nn.Module):

[0154] def__init__(self):

[0155] super().__init__()

[0156] #CD branch and DD branch definition (omit the specific convolution layer implementation)

[0157] self.cd_encoder=nn.Sequential(...)

[0158] self.dd_encoder=nn.Sequential(...)

[0159] self.decoder=nn.Sequential(...)

[0160] defforward(self,rgb,depth_sparse):

[0161] #Feature extraction and fusion

[0162] cd_features = self.cd_encoder(torch.cat([rgb, depth_sparse], dim = 1))

[0163] dd_features = self.dd_encoder(depth_sparse)

[0164] fused_features = cd_features + dd_features

[0165] # Output depth map and confidence

[0166] depth_cd, conf_cd = self.decoder(cd_features)

[0167] depth_dd, conf_dd = self.decoder(dd_features)

[0168] # Confidence weighted fusion

[0169] exp_conf_cd = torch.exp(conf_cd)

[0170] exp_conf_dd = torch.exp(conf_dd)

[0171] depth_fused = (exp_conf_cd * depth_cd + exp_conf_dd * depth_dd) / (exp_conf_cd + exp_conf_dd)

[0172] return depth_fused

[0173] 5. Coal volume calculation

[0174] def calculate_coal_volume(depth_map, conveyor_speed, slice_thickness = 0.05):

[0175] """

[0176] depth_map: Fine depth map (H, W)

[0177] Returns: Coal volume (m³)

[0178] """

[0179] # Convert depth map to point cloud (industrial camera coordinate system)

[0180] u, v = np.meshgrid(np.arange(depth_map.shape[1]), np.arange(depth_map.shape[0]))

[0181] z = depth_map

[0182] x = (u - K[0, 2]) * z / K[0, 0]

[0183] y = (v - K[1, 2]) * z / K[1, 1]

[0184] points = np.stack([x, y, z], axis=-1).reshape(-1, 3)

[0185] # ICP registration (assuming the empty point cloud has been obtained in advance)

[0186] pcd_load = PointCloud()

[0187] pcd_load.points = Vector3dVector(points)

[0188] pcd_empty = read_point_cloud("empty.pcd")

[0189] reg_p2p = registration_icp(

[0190] pcd_load, pcd_empty, max_correspondence_distance = 0.01,

[0191] init = np.eye(4), type_registration = TransformationEstimationPointToPoint()

[0192] aligned_points = pcd_load.transform(reg_p2p.transformation)

[0193] # Slice integration to calculate volume (simplified implementation)

[0194] z_min, z_max = np.min(aligned_points[:, 2]), np.max(aligned_points[:, 2])

[0195] slices = np.arange(z_min, z_max, slice_thickness)

[0196] volume = 0.0

[0197] for i in range(len(slices) - 1):

[0198] mask = (aligned_points[:, 2] >= slices[i]) & (aligned_points[:, 2] < slices[i + 1])

[0199] slice_points = aligned_points[mask]

[0200] # Calculate cross-sectional area using Green's theorem (simplified to convex hull area)

[0201] hull = ConvexHull(slice_points[:, :2])

[0202] area = hull.volume

[0203] volume += area * slice_thickness

[0204] return volume

[0205] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A coal quantity detection method based on the fusion of lidar and vision, characterized in that, It includes the following steps: S1. Sensor joint calibration; S2. Data acquisition: Use industrial cameras, lidar, and speed sensors to collect coal flow-related information to provide basic data for subsequent analysis; S3. Coal quantity area identification to determine the area where the coal flow is located and prepare for subsequent calculations; S4. Depth image completion: Adopt a color-guided depth image completion network. This depth image completion network includes a color-dominated branch and a depth-dominated branch. Both the color-dominated branch and the depth-dominated branch are encoder-decoder structures and are composed of convolutional layers, residual blocks, and deconvolutional layers; S5. Coal quantity calculation: After preprocessing the depth map, convert it into a point cloud and calculate the coal quantity through an algorithm.

2. The coal quantity detection method based on laser radar and vision fusion according to claim 1 is characterized in that: In step S1, the specific content of the sensor joint calibration is as follows: Establish the coordinate system of the lidar , with the origin at the physical center of the lidar The x-axis points horizontally in the running direction of the belt conveyor, The y-axis is horizontally perpendicular to the x-axis and points to the right side of the conveyor, The z-axis points vertically upward from the ground; Establish the industrial camera coordinate system , with the origin at the optical center of the camera The X-axis is horizontally to the right corresponding to the U-axis of the pixel coordinate system, The Y-axis is vertically downward corresponding to the V-axis of the pixel coordinate system, The Z-axis is in the direction of the optical axis; Pixel coordinate system The origin is at the upper left corner of the image, the horizontal pixels increase along the axis, and the vertical pixels increase along the axis; World coordinate system The origin is a user-defined reference point, The axis is along the conveyor running direction, The axis is perpendicular to the conveyor plane, The axis is perpendicular to the ground and points upward; A joint calibration model of the lidar and the industrial camera is established through the rotation matrix R and the translation vector T. The formula is as follows: , where is the corresponding pixel coordinate, is the three-dimensional point cloud coordinate of the lidar. The internal parameter matrix K of the industrial camera is known data, and the rotation matrix R and the translation vector t from the lidar to the industrial camera are unknown parameters. By solving this external parameter matrix, the lidar point cloud is projected onto the image plane to achieve spatial alignment of multi-modal data; The internal parameter matrix K of the industrial camera includes the focal length and the principal point coordinates .

3. The coal quantity detection method based on lidar and vision fusion according to claim 1, characterized in that, In step S2, the specific content of the data acquisition includes the following steps: S2.

1. Industrial camera image acquisition: Set the frame rate of the industrial camera , conveyor speed Length of coal flow covered by a single-frame image Since each sampling length Therefore, the industrial camera skips frames to acquire image information, and the frame interval is Image data starts to be acquired from the frame, and its data set is ; S2.

2. LiDAR Point Cloud Acquisition: The height information of the coal flow is acquired using a 128-line LiDAR. The LiDAR frame rate f = 30 frames / s, and it is acquired synchronously with the industrial camera. 10 frames are sampled continuously. Starting from the frame, the point cloud data set is represented as ; S2.

3. Speed data acquisition: The speed sensor measures the real-time operating speed of the belt conveyor.

4. The coal quantity detection method based on laser radar and vision fusion according to claim 1 is characterized in that: In step S3, the specific content of the coal quantity area identification includes the following steps: S3.

1. Lidar point cloud area separation: Collect the complete three-dimensional point cloud of the conveyor belt in the no-load state With the Poisson reconstruction algorithm, a high-precision reference surface is constructed, and the collected load point cloud data set and the reference point cloud carry out voxel-level registration operations, and eliminate the point cloud data set through a spatial filtering strategy in the reference points that cannot be matched, separate the coal flow area and non-coal area on the conveyor belt, and obtain a point cloud data set containing coal flow information; S3.

2. Industrial camera image area positioning: Collect the RGB color image of the conveyor belt in the no-load state, and extract the texture features on the surface of the conveyor belt and the conveyor belt area When processing the collected load color map dataset I, for the part that is consistent with the no-load conveyor belt area in it, texture comparison is carried out, and the coal flow area of the belt conveyor is located by using the structural similarity index. The specific determination method is as follows: Suspected coal area.

5. A coal quantity detection method based on the fusion of lidar and vision according to claim 1, characterized in that, In step S4, the specific content of the depth image completion includes the following steps: S4.

1. Color-dominated branch processing: The input of the color-dominated branch is the color map and the sparse depth map. It captures multi-scale context information through the spatial pyramid pooling module, fuses low-level edge features using skip connections. The input images RGB and depth D are concatenated and operated through a 3×3 convolutional layer to obtain multi-dimensional features, and the RGB features and geometric features are feature-fused and feature-extracted to obtain different-scale fused features; S4.

2. Depth-dominant branch processing: The depth-dominant branch takes the coarse depth map and the sparse depth map output by the color-dominant branch as inputs, extracts sparse features through a 3×3 convolutional layer, and fuses them with the RGB geometric features extracted by the color-dominant branch. The two branches respectively predict the depth map and the confidence map through transposed convolution, and finally outputs the fused rough depth map: represents a pixel, represents a fused depth map, successively represents a CD depth map and a DD depth map, successively represents a CD confidence and a DD confidence; S4.

3. Dynamic optimization acceleration: The dynamic optimization acceleration network takes the predicted rough depth map as the input, learns to generate variable convolution kernel parameters related to spatial positions through convolutional layers, and realizes adaptive receptive field adjustment; Integrate the conditional space propagation mechanism, diffuse the effective observations of the sparse point cloud to the unobserved area through a two-way propagation strategy, and at the same time introduce a spatial attention module to generate an attention weight matrix based on the depth gradient map to enhance the features of the edge area by weighting; The hierarchical loss supervision strategy synchronously constrains the global depth error and local structural similarity to ensure the accuracy of high-frequency edges; The final predicted fine depth map of the network can be obtained by weighted summation of the CSPN results predicted by the variable convolution kernel multiple times.

6. The coal quantity detection method based on the fusion of lidar and vision according to claim 1, characterized in that, In step S5, the specific content of the coal quantity calculation includes the following steps: S5.

1. Depth map preprocessing: Based on the fine depth map repaired by the depth completion network, preprocess the no-load depth map and the loaded depth map with coal material; The no-load depth map eliminates environmental interference by establishing a background template through the Gaussian mixture model; The loaded depth map adopts a combined strategy of bilateral filtering and statistical outlier removal algorithm to suppress noise; The preprocessed depth data is converted into a three-dimensional point cloud through a non-linear mapping from polar coordinates to Cartesian coordinates; S5.

2. Point cloud processing and volume calculation: Point cloud stitching adopts an improved iterative closest point algorithm. By introducing a feature point matching strategy weighted by curvature, the initial registration error is reduced. The registered point cloud generates a three-dimensional reconstruction model through Poisson reconstruction; The three-dimensional reconstruction model is sliced vertically at a fixed layer spacing d and slice thickness for hierarchical slicing, and the hierarchical density is dynamically encrypted in the regions of abrupt curvature change; By extracting the closed contour boundary within the slice projection plane and performing numerical integration on the cross-sectional area based on Green's formula , the total volume V is achieved by cumulative discrete integration 7. A coal quantity detection device, characterized in that: It includes a speed sensor, a lidar, an industrial camera, a computer, and a human-machine interface. The speed sensor is installed on the belt conveyor and is used to measure the real-time operating speed of the belt conveyor. The industrial camera is installed directly above the belt conveyor and is used to collect image data of the coal flow in combination with the belt conveyor speed information. The lidar is used to collect real-time coal flow data information passing through the belt conveyor. The computer is used to integrate data, perform depth image completion and coal quantity calculation processing. The human-machine interface includes a display screen and an input module, and the human-machine interface is used to display data and perform operation control.

8. The coal quantity detection device according to claim 7, characterized in that, The signal output end of the speed sensor, the signal output end of the lidar, and the signal output end of the industrial camera are all connected to the signal input end of the computer, and the signal control end of the computer is connected to the signal end of the human-machine interface.

Images