Dynamic stockpiling detection system based on industrial camera and laser radar

Abstract

The invention relates to the technical field of industrial automatic intelligent detection systems, in particular to a dynamic stockpile detection system based on an industrial camera and a laser radar, which comprises a sensor module, a cloud platform module and a stockpile loading and unloading working end, and is characterized in that the sensor module comprises the industrial camera, the laser radar and a transmission unit; wherein the industrial camera and the laser radar are aligned with a data acquisition time sequence through a time synchronization module, the laser radar adopts a multi-line scanning mode, and a self-cleaning module is arranged in the laser radar to deal with dust interference; through hardware-level time sequence synchronization and space coordinate alignment of the industrial camera and the laser radar, in combination with deep fusion of the RGB image and the three-dimensional point cloud, the limitation that a traditional single sensor can only obtain contour or texture information is broken through, and through cooperative application of the improved DBSCAN algorithm and the YOLOv5 model, the precision of the contour or texture information is improved. The surface detail features and the three-dimensional morphological parameters of the stockpile can be extracted at the same time, the accuracy of volume calculation and morphological analysis is remarkably improved, and the method has more advantages especially in a complex stockpile scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial automation intelligent detection system, and particularly relates to a dynamic stockpile detection system based on an industrial camera and a laser radar. BACKGROUND

[0002] Stockpile generally refers to a pile formed by stacking mined or processed materials such as coal, gravel, and waste slag according to certain rules in mine, port, power plant and other scenes to balance production demand or for long-term storage. The cross-sectional shape of the stockpile is mostly herringbone or trapezoidal, and the height and width need to be designed in combination with the material repose angle and stockpile equipment parameters.

[0003] For example, a stockpile edge detection method and system with an application number of CN202011162327.0 and an authorization announcement date of 20230328. The method includes: obtaining two-dimensional point cloud data of a stockpile cutting surface detected by a radar; the radar is fixed to the side of the boom of the reclaimer and located at the side of the bucket wheel; the radar is used to scan the stockpile cutting surface in front of the boom of the reclaimer, and the scanning plane of the radar is a vertical plane; according to the rotation direction of the boom of the reclaimer, the two-dimensional point cloud data is expanded into three-dimensional point cloud data; the three-dimensional point cloud data is preprocessed; the processed three-dimensional point cloud data corresponding to the current frame is plane fitted; the material surface inclination angle corresponding to the current frame is calculated; it is judged whether the material surface inclination angle corresponding to the current frame is less than the material repose angle; if yes, it is determined that the scanning plane of the current frame radar is on the outside of the material pile edge; if not, it is determined that the scanning plane of the current frame radar is on the inside of the material pile edge. The present application can improve the accuracy and timeliness of stockpile edge detection.

[0004] Stockpile is the core form of material storage in mine, port, power plant and other scenes, and its form monitoring and safety management have long relied on traditional technical means. However, the traditional technical means mostly use single radar scanning, which can only detect the edge of the stockpile cutting surface and cannot synchronously obtain the surface texture information of the stockpile. Moreover, the traditional technical means are based on fixed period scanning and cannot dynamically perceive the change of the stockpile form. Therefore, it is urgent to design a dynamic stockpile detection system based on an industrial camera and a laser radar to solve the above problems. SUMMARY

[0005] The present application aims to provide a dynamic stockpile detection system based on an industrial camera and a laser radar, which realizes dynamic and accurate monitoring of the stockpile form through the combination of optical metrology and image recognition technology.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0007] A dynamic stockpile detection system based on an industrial camera and a laser radar, comprising a sensor module, a cloud platform module and a stockpile loading and unloading work end, the sensor module includes an industrial camera, a laser radar and a transmission unit, wherein:

[0008] The industrial camera and the laser radar are aligned in data collection timing through a time synchronization module, and the laser radar adopts a multi-line scanning mode, enters a 120°x120° field of view angle, and is provided with a self-cleaning module to cope with dust interference.

[0009] It should be noted that the sensor module is additionally provided with a temperature and humidity sensor, a millimeter wave radar and an inertial measurement unit, the temperature and humidity sensor monitors environmental parameters in real time, the millimeter wave radar assists in detecting moving targets such as flying birds and personnel intrusion, and the inertial measurement unit (IMU) corrects the attitude drift of the sensor.

[0010] The transmission unit supports 4G / 5G network communication, adopts TCP / UDP protocol to transmit data, and integrates data encryption and verification functions.

[0011] The cloud platform module includes a receiving unit, a preprocessing unit, a computing unit, a decision-making unit, a remote control unit, a storage unit and a notification unit, wherein:

[0012] The preprocessing unit includes an optimization module and a conversion module, the optimization module removes laser radar point cloud noise through median filtering, and optimizes industrial camera image data through Gaussian filtering;

[0013] It should be noted that the optimization module of the preprocessing unit further includes:

[0014] (1) The laser radar point cloud is down-sampled by voxel, with a resolution of 0.1m 3 , to reduce the data volume;

[0015] (2) The histogram equalization of the industrial camera image is enhanced to increase the contrast;

[0016] (3) The sliding window method is used to identify and fill in missing values during data cleaning.

[0017] The conversion module fuses the laser radar point cloud and the camera image into an RGB-D data format;

[0018] The computing unit includes a calculation module, an error compensation module and a delay monitoring module, the calculation module generates the stacking volume and the shape parameters by using a three-dimensional point cloud clustering algorithm (DBSCAN) combined with a YOLO target detection model;

[0019] It should be noted that the calculation module of the computing unit adopts the following steps:

[0020] Step S1. Align the spatial coordinates of the laser radar point cloud and the camera image through the external parameter calibration matrix;

[0021] Step S2. Extract the stacking contour point cloud and fit the minimum bounding box to calculate the volume (v i for a voxel unit volume);

[0022] Step S3. Recognize the stacking surface collapse risk area through a convolutional neural network (CNN).

[0023] The error compensation module dynamically corrects data based on environmental sensors (temperature and humidity, light intensity), compensates for dust shielding, rain scattering, and low light error;

[0024] It should be noted that the error compensation module includes the following sub-modules:

[0025] (1) Dust compensation sub-module: identify the dust area through the difference between the laser radar echo intensity and the point cloud density, and reconstruct the missing point cloud using the interpolation algorithm;

[0026] (2) Rain compensation sub-module: based on millimeter wave radar auxiliary data, correct the ranging error caused by raindrop scattering of the laser radar, and when the signal-to-noise ratio of the millimeter wave radar is continuously less than 10 dB for 5 seconds, the camera is turned off and switched to pure laser radar scanning, and at the same time, the attitude drift is compensated through the IMU data;

[0027] (3) Low light compensation sub-module: the industrial camera is switched to infrared mode and the laser radar point cloud is fused to generate a heat map.

[0028] The error compensation module fuses laser radar, IMU, and millimeter wave radar data through a Kalman filter to dynamically update the state equation:

[0029]

[0030] Where K is the Kalman gain, and H is the observation matrix. k

[0031] The operation process of the calculation unit is as follows:

[0032] Step one. Data alignment: obtain the external parameter matrix T of the industrial camera and the laser radar through the calibration board T ∈ R ∧ {4×4}, map the point cloud coordinates (x, y, z) to the image pixel coordinates (u, v), and the formula is:

[0033]

[0034] Where K is the camera intrinsic matrix, and [R|t] is the rotation and translation matrix.

[0035] Step two. Point cloud segmentation: use an improved DBSCAN algorithm to cluster point clouds, set the neighborhood radius ∈ = 0.3m and the minimum point number MinPts = 15, and accelerate the neighbor search through KD-Tree;

[0036] ​Step three. Object detection: The input image size of the YOLOv5 model is 640x640, the Backbone uses CSPDarknet53, and the output is the bounding box and class probability of the stack;

[0037] Step four. Volume calculation: An octree structure is constructed for the segmented point cloud, and the voxel resolution is 0.05m 3 Discretize the space and calculate the volume by Monte Carlo integration:

[0038]

[0039] Where k is the deformation compensation coefficient, Δh i is the surface relief height difference;

[0040] Step five. Use ResNet-34 CNN to extract features from the height map generated by the stack surface point cloud, and output the collapse probability P collapse =σ(WX+b), σ is the Sigmoid function;

[0041] The delay monitoring module automatically reduces the point cloud resolution (0.2m 3 ) and image size (320x320) when the data processing time exceeds the set threshold (500ms), and dynamically adjusts the YOLOv5 inference frame rate, prioritizing key area detection (stack top and edge).

[0042] The decision unit generates loading and unloading path planning through a multi-objective optimization algorithm, and predicts the stability risk of the stack based on historical data;

[0043] It should be noted that the solution generation of the decision unit includes:

[0044] Step A1. If the stack height exceeds the safety threshold, generate an "emergency unloading" instruction;

[0045] Step A2. If the stack shape deviates from the stable parameters, the inclination angle > 25°, generate a "stiff support" instruction;

[0046] Step A3. Combine weather forecast data to schedule a rain cover instruction in advance.

[0047] The remote control unit control instruction needs to be verified by path simulation. The simulation engine calculates the collision probability based on the stack three-dimensional model and the kinematics model of the robot arm. If the risk is > 5%, the path is re-planned. The remote control unit supports dynamic priority adjustment, and emergency tasks can interrupt the current loading and unloading process and restart detection;

[0048] It should be noted that the remote control unit supports the following control modes:

[0049] (1) Automatic mode: complete reliance on decision unit instructions execution;

[0050] (2) Semi-automatic mode: workers preview decision schemes through AR glasses and manually confirm;

[0051] (3) Emergency mode: directly take over the motor controller of the loading and unloading end.

[0052] The storage unit adopts a distributed storage architecture, data is stored according to time stamp and event type, supports compression and encryption;

[0053] It should be noted that the storage unit adopts a hierarchical storage strategy, and the original data is automatically deleted after 7 days, the key event (such as collapse warning) data is permanently stored and backed up to a remote disaster recovery center, and the log data is encrypted and uploaded to the block chain for storage.

[0054] The notification unit is a dynamic detection trigger mechanism, when the laser radar monitors the point cloud change rate in real time, if the change rate is lower than the threshold 0.5% / min, it enters the sleep mode, and when it is higher than the threshold, the industrial camera and radar joint detection are activated;

[0055] It should be noted that the multi-level warning mechanism of the notification unit includes:

[0056] First-level warning: broadcast prompt on-site workers;

[0057] Second-level warning: SMS notification to management personnel;

[0058] Third-level warning: automatically dial emergency phone and start sound and light alarm.

[0059] The dynamic detection trigger mechanism further includes:

[0060] (1) Laser radar scanning interval is dynamically adjusted according to environmental visibility, such as 1Hz in rainy days and 5Hz in sunny days;

[0061] (2) Industrial camera adopts frame difference method to detect small motion on the surface of the material pile, such as pixel change>5% to trigger detection.

[0062] The loading and unloading work end of the material pile is integrated with AGV trolley and mechanical arm, receives decision instructions, executes loading and unloading actions, and feeds back execution status to the cloud platform.

[0063] In the above technical solution, the dynamic material pile detection system based on industrial camera and laser radar provided by the application has the following beneficial effects:

[0064] (1) The application synchronizes the hardware level timing of the industrial camera and the laser radar, and combines the depth fusion of the RGB image and the three-dimensional point cloud, breaking through the limitation of traditional single sensor that can only obtain contour or texture information, and the improved DBSCAN algorithm and YOLOv5 model are applied cooperatively, so that the surface detail features and three-dimensional morphological parameters of the stacked material can be extracted at the same time, the accuracy of volume calculation and morphological analysis is significantly improved, and the application has more advantages in complex stacking scenes.

[0065] (2) The application realizes on-demand allocation of detection resources based on the dynamic dormancy and activation strategy of the point cloud change rate, and combines the intelligent downsampling technology of the delay monitoring module and the frame rate adaptive adjustment function.

[0066] (3) The application fuses multi-sensor data through a Kalman filter, combines dust shielding interpolation reconstruction, rain mode switching and low-light infrared thermal map generation technology, effectively deals with dust, rain, fog and other complex interference in industrial scenes, and the system can stably operate under different environmental conditions, ensures the integrity of data acquisition, and reduces the risk of false detection and missed detection.

[0067] (4) The decision unit of the application generates loading and unloading path planning based on a multi-objective optimization algorithm, predicts the stacking collapse risk by combining a convolutional neural network, realizes early warning of stability hazards, and a multi-level linkage alarm mechanism formed by on-site broadcasting, remote notification and emergency response and path simulation verification function form a closed-loop safety protection system, which significantly reduces the probability of operation accidents and improves the efficiency of emergency disposal.

[0068] (5) The application solves the core defects of traditional stacking detection systems such as low precision, poor real-time performance and weak environmental adaptability through multi-sensor cooperation, dynamic optimization algorithm and intelligent decision closed loop, and can greatly improve the detection efficiency in mine, port and other scenes, and also can reduce the rate of safety accidents. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0070] Fig. 1 The system block diagram of the embodiment of the application is provided.

[0071] Fig. 2A schematic diagram of a preprocessing unit provided in an embodiment of a dynamic material pile detection system based on an industrial camera and a lidar according to the present invention.

[0072] Fig. 3 A schematic diagram of a computing unit provided in an embodiment of a dynamic material pile detection system based on an industrial camera and a lidar according to the present invention. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0074] like Figs. 1-3 As shown, an embodiment of the present invention provides a dynamic pile detection system based on an industrial camera and a laser radar, including a sensor module, a cloud platform module, and a pile loading and unloading work end. The sensor module includes an industrial camera, a laser radar, and a transmission unit, wherein:

[0075] The industrial camera and LiDAR align data acquisition timing through a time synchronization module. The LiDAR uses a multi-line scanning mode, such as a 120°×120° field of view, and has a built-in self-cleaning module to deal with dust interference.

[0076] It should be noted that the sensor module is equipped with a temperature and humidity sensor, a millimeter-wave radar and an inertial measurement unit. The temperature and humidity sensor monitors environmental parameters in real time, the millimeter-wave radar assists in detecting moving targets such as birds and people intruding, and the inertial measurement unit (IMU) corrects sensor attitude drift.

[0077] The transmission unit supports 4G / 5G network communication, uses TCP / UDP protocol to transmit data, and integrates data encryption and verification functions;

[0078] The cloud platform module includes a receiving unit, a pre-processing unit, a computing unit, a decision-making unit, a remote control unit, a storage unit, and a notification unit, among which:

[0079] The preprocessing unit includes an optimization module and a conversion module. The optimization module removes the noise of the lidar point cloud through median filtering and optimizes the industrial camera image data through Gaussian filtering.

[0080] It should be noted that the optimization module of the pre-processing unit further includes:

[0081] (1) Voxel downsampling is used for the lidar point cloud with a resolution of 0.1m 3 , reduce the amount of data;

[0082] (2) Perform histogram equalization on industrial camera images to enhance contrast;

[0083] (3) When cleaning data, the sliding window method is used to identify and fill missing values.

[0084] The conversion module fuses the lidar point cloud and the camera image into an RGB-D data format;

[0085] The calculation unit includes an arithmetic module, an error compensation module, and a delay monitoring module. The arithmetic module generates the stacking volume and the morphology parameters by using a three-dimensional point cloud clustering algorithm (DBSCAN) combined with a YOLO target detection model.

[0086] It should be noted that the arithmetic module of the calculation unit adopts the following steps:

[0087] Step S1. Align the space coordinates of the lidar point cloud and the camera image through the external parameter calibration matrix.

[0088] Step S2. Extract the stacking contour point cloud and fit the minimum bounding box to calculate the volume (v i for the voxel unit volume);

[0089] Step S3. Identify the stacking surface collapse risk area through a convolutional neural network (CNN).

[0090] The error compensation module dynamically corrects the data based on the environmental sensors (temperature and humidity, light intensity) to compensate for dust shielding, rain scattering, and low light errors.

[0091] It should be noted that the error compensation module includes the following sub-modules:

[0092] (1) Dust compensation sub-module: identify the dust area by the difference between the lidar echo intensity and the point cloud density, and reconstruct the missing point cloud using an interpolation algorithm.

[0093] (2) Rain compensation sub-module: based on the auxiliary data of the millimeter wave radar, correct the ranging error caused by raindrop scattering of the lidar, and when the signal-to-noise ratio of the millimeter wave radar is less than 10 dB for 5 seconds, turn off the camera and switch to pure lidar scanning, and at the same time, compensate for the attitude drift through the IMU data.

[0094] (3) Low light compensation sub-module: switch the industrial camera to infrared mode and fuse the lidar point cloud to generate a heat map.

[0095] The error compensation module fuses the lidar, IMU, and millimeter wave radar data through a Kalman filter to dynamically update the state equation:

[0096]

[0097] where K k is the Kalman gain, and H is the observation matrix.

[0098] The operation process of the calculation unit is as follows:

[0099] Step 1. Data alignment: Obtain the extrinsic matrix T ∈ R ∧ {4×4} of the industrial camera and the lidar through the calibration board, which maps the point cloud coordinates (x, y, z) to the image pixel coordinates (u, v) as follows:

[0100]

[0101] where K is the camera intrinsic matrix and [R|t] is the rotation and translation matrix.

[0102] Step 2. Point cloud segmentation: Use the improved DBSCAN algorithm to cluster the point cloud, set the neighborhood radius ∈ = 0.3 m and the minimum point number MinPts = 15, and accelerate the nearest neighbor search through KD-Tree.

[0103] Step 3. Object detection: The input image size of the YOLOv5 model is 640×640, the Backbone uses CSPDarknet53, and the output is the stacking boundary box and class probability.

[0104] Step 4. Volume calculation: Construct an octree structure for the segmented point cloud with a voxel resolution of 0.05 m 3 to discretize the space, and calculate the volume by the Monte Carlo integration method:

[0105]

[0106] where k is the deformation compensation coefficient, Δh i is the surface relief height difference.

[0107] Step 5. Use ResNet-34 CNN to extract features from the height map generated by the stacking surface point cloud, and output the collapse probability P collapse = σ(WX+b), where σ is the Sigmoid function.

[0108] The delay monitoring module automatically reduces the point cloud resolution (0.2 m 3 ) and image size (320×320) when the data processing time exceeds the set threshold (500 ms), and dynamically adjusts the YOLOv5 inference frame rate, prioritizing key area detection (stacking top and edges).

[0109] The decision unit generates loading and unloading path planning through a multi-objective optimization algorithm, and predicts the stacking stability risk based on historical data.

[0110] It should be noted that the solution generation of the decision unit includes:

[0111] Step A1. If the stacking height exceeds the safety threshold, generate an "emergency unloading" instruction.

[0112] Step A2: If the pile shape deviates from the stable parameters and the inclination angle is greater than 25°, generate a "reinforce support" instruction;

[0113] Step A3: Based on the weather forecast data, dispatch the rainproof cloth covering instruction in advance.

[0114] The remote control unit's control instructions must be verified through path simulation. The simulation engine calculates the collision probability based on the three-dimensional model of the pile and the robot arm's kinematic model. If the risk is greater than 5%, the path is replanned. The remote control unit supports dynamic priority adjustment. Emergency tasks can interrupt the current loading and unloading process and restart the inspection.

[0115] It should be noted that the remote control unit supports the following control modes:

[0116] (1) Automatic mode: completely dependent on the instructions of the decision-making unit;

[0117] (2) Semi-automatic mode: Workers preview the decision plan through AR glasses and manually confirm it;

[0118] (3) Emergency mode: Directly take over the motor controller at the loading and unloading end.

[0119] The storage unit adopts a distributed storage architecture, data is stored by timestamp and event type, and supports compression and encryption;

[0120] It should be noted that the storage unit adopts a hierarchical storage strategy, and the original data is automatically deleted after being retained for 7 days. Key event data (such as collapse warning) is permanently stored and backed up to an off-site disaster recovery center, and the log data is encrypted and uploaded to the blockchain for evidence storage.

[0121] The notification unit is a dynamic detection trigger mechanism. When the lidar monitors the point cloud change rate in real time, if the change rate is lower than the threshold of 0.5% / min, it enters sleep mode. If the change rate is higher than the threshold, it activates the joint detection of industrial camera and lidar.

[0122] It should be noted that the multi-level warning mechanism of the notification unit includes:

[0123] Level 1 warning: Broadcast alerts workers on site;

[0124] Level 2 warning: SMS notification to management personnel;

[0125] Level 3 warning: Automatically dial the emergency number and activate the sound and light alarm.

[0126] The dynamic detection trigger mechanism further includes:

[0127] (1) The lidar scanning interval is dynamically adjusted according to the environmental visibility, such as 1Hz in rainy days and 5Hz in sunny days;

[0128] (2) Industrial camera uses frame difference method to detect small movements on the surface of the pile, such as pixel change > 5% to trigger detection.

[0129] The pile handling end integrates AGV trolley and mechanical arm, receives decision instructions, executes handling actions, and feeds back execution status to the cloud platform.

[0130] Working principle: When the system is running, the industrial camera and the laser radar realize microsecond-level time sequence alignment through the time synchronization module (PTP protocol), ensuring that the RGB image and the three-dimensional point cloud are collected at the same time. Through the calibration of the calibration board, the laser radar point cloud coordinates (x, y, z) are mapped to the camera pixel coordinates (u, v). The laser radar uses a multi-line scanning mode (120°×120° field of view), combined with the built-in self-cleaning module to eliminate dust interference. The millimeter wave radar detects moving targets such as vehicles and personnel, and the temperature and humidity sensor feeds back environmental parameters. The IMU monitors the sensor attitude deviation in real time, and compensates for the deviation error caused by mechanical vibration or wind load through coordinate system transformation. The subsequent laser radar point cloud is filtered by median filtering to remove outliers, and the voxel is down-sampled to compress the data volume. The industrial camera image is smoothed by Gaussian filtering, the histogram equalization is enhanced to enhance the low-contrast area details, the conversion module associates the RGB image pixels with the corresponding point cloud depth values, constructs the RGB-D data frame, and fuses the texture and three-dimensional geometric information. The improved DBSCAN algorithm is used to segment the pile point cloud cluster. The 640×640 image is input into the CSPDarknet53 network, and the pile boundary box and class probability are output. Based on the octree discretization of the segmented point cloud space, the Monte Carlo integral is used to calculate the pile volume. In the subsequent calculation process, the dust shielding area is identified by the difference in echo intensity, and the missing point cloud is reconstructed by radial basis function interpolation. In rainy weather, when the signal-to-noise ratio of the millimeter wave radar is <10dB, the camera is turned off, and the laser radar data is corrected by Kalman filtering to correct the raindrop scattering error. In low light, the camera switches to infrared mode, fuses the laser radar point cloud to generate a heat map, and the ResNet-34 CNN extracts the height map features to predict the collapse probability. The decision unit then generates the handling path of the AGV trolley and the mechanical arm using the NSGA-II multi-objective optimization algorithm, balancing efficiency and safety. Based on the historical pile stability data and real-time collapse probability prediction, a three-level early warning mechanism is triggered, and the process is as follows:

[0131] Pile height exceeds the limit → broadcast prompt to on-site workers;

[0132] Morphology deviates from stable parameters → send SMS to management personnel;

[0133] Collapse risk threshold triggers → start emergency phone and sound-light alarm;

[0134] In addition, the remote control unit supports path simulation verification: the collision probability is pre-calculated based on the three-dimensional point cloud model and the robotic arm kinematic model. If the risk is >5%, the path is re-planned. In emergency mode, the motor controller at the loading and unloading end is directly taken over, the current task is interrupted and the obstacle avoidance action is started; the subsequent delay monitoring module evaluates the data processing time in real time: when the threshold is exceeded, the point cloud resolution and YOLOv5 inference frame rate are reduced, and key area detection is prioritized; the point cloud change rate triggers the sleep / activation mechanism: the camera is turned off when the change rate is <1% / s, and a full sensor joint scan is started when the change rate is >5% / s; the original data is retained for 7 days and then rolled over and deleted, and key event data is permanently stored and backed up to an off-site disaster recovery center; the operation log will be encrypted and uploaded to the blockchain for evidence storage to support auditing and accident tracing.

[0135] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A dynamic pile detection system based on industrial cameras and laser radar, including a sensor module, a cloud platform module, and a pile loading and unloading work end, characterized by: The sensor module includes an industrial camera, a laser radar and a transmission unit, wherein: The industrial camera and laser radar align the data acquisition timing through a time synchronization module, and the laser radar adopts a multi-line scanning mode and a built-in self-cleaning module to deal with dust interference; The transmission unit supports 4G / 5G network communication, adopts TCP / UDP protocol to transmit data, and integrates data encryption and verification functions; The cloud platform module includes a receiving unit, a pre-processing unit, a computing unit, a decision unit, a remote control unit, a storage unit and a notification unit, wherein: The preprocessing unit includes an optimization module and a conversion module. The optimization module removes laser radar point cloud noise through median filtering and optimizes industrial camera image data through Gaussian filtering. The conversion module fuses the lidar point cloud and the camera image into RGB-D data format; The calculation unit includes a calculation module, an error compensation module and a delay monitoring module. The calculation module uses a three-dimensional point cloud clustering algorithm combined with a YOLO target detection model to generate the volume and morphological parameters of the pile; the error compensation module dynamically corrects data based on environmental sensors to compensate for dust occlusion, rain scattering and low light errors; The operation process of the computing unit is as follows: Step 1. Data alignment: Obtain the external parameter matrix T∈R∧{4×4} of the industrial camera and lidar through the calibration plate, and map the point cloud coordinates (x, y, z) to the image pixel coordinates (u, v) using the formula: Where K is the camera intrinsic parameter matrix, [R|t] is the rotation and translation matrix; Step 2. Point cloud segmentation: Use the improved DBSCAN algorithm to cluster the point cloud, set the neighborhood radius ∈ = 0.3m, the minimum number of points MinPts = 15, and use KD-Tree to accelerate the nearest neighbor search; Step 3. Object detection: The YOLOv5 model inputs an image of size 640×640, and Backbone uses CSPDarknet53 to output the bounding box and category probability. Step 4. Volume calculation: Construct an octree structure for the segmented point cloud with a voxel resolution of 0.05m 3 Discretize the space and calculate the volume using Monte Carlo integration: Where k is the deformation compensation coefficient, Δh i is the height difference of the surface relief; Step 5. Use ResNet-34CNN to extract features from the height map generated by the pile surface point cloud and output the collapse probability P collapse =σ(WX+b), σ is the Sigmoid function; The error compensation module fuses the lidar, IMU and millimeter-wave radar data through the Kalman filter and dynamically updates the state equation: Among them, K k is the Kalman gain, H is the observation matrix; When the data processing time exceeds the set threshold, the delay monitoring module automatically reduces the point cloud resolution and image size, and dynamically adjusts the YOLOv5 inference frame rate to prioritize key area detection; The decision-making unit generates loading and unloading path planning through a multi-objective optimization algorithm and predicts stockpile stability risks based on historical data; The remote control unit supports dynamic adjustment of priorities, and emergency tasks can interrupt the current loading and unloading process and restart the inspection; The storage unit adopts a distributed storage architecture, data is stored according to timestamp and event type, and supports compression and encryption; The notification unit is a dynamic detection trigger mechanism. When the lidar monitors the point cloud change rate in real time, it enters a sleep mode if the change rate is lower than the threshold. When the change rate is higher than the threshold, it activates the industrial camera and radar for joint detection. The stacking and loading and unloading work end integrates an AGV trolley and a robotic arm, executes loading and unloading actions after receiving decision instructions, and feeds back the execution status to the cloud platform.

2. The dynamic pile detection system based on industrial camera and laser radar according to claim 1 is characterized in that: The sensor module is additionally equipped with a temperature and humidity sensor, a millimeter-wave radar, and an inertial measurement unit. The temperature and humidity sensor monitors environmental parameters in real time, the millimeter-wave radar assists in detecting moving targets, and the inertial measurement unit corrects sensor attitude drift.

3. The dynamic pile detection system based on industrial camera and laser radar according to claim 1 is characterized in that: The optimization module of the pre-processing unit further includes: (1) Using voxel downsampling on the lidar point cloud to reduce the data volume; (2) Perform histogram equalization on industrial camera images to enhance contrast; (3) When cleaning data, the sliding window method is used to identify and fill missing values.

4. The dynamic pile detection system based on industrial camera and laser radar according to claim 1, characterized in that: The error compensation module includes the following submodules: (1) Dust compensation submodule: identifies dust areas by comparing the difference between the laser radar echo intensity and the point cloud density, and reconstructs the missing point cloud using an interpolation algorithm; (2) Rain compensation submodule: This module corrects the ranging error of the lidar caused by raindrop scattering based on the millimeter-wave radar auxiliary data. When the millimeter-wave radar signal-to-noise ratio is less than 10dB for 5 seconds, the camera is turned off and switched to pure lidar scanning. At the same time, the attitude drift is compensated through the IMU data. (3) Low light compensation submodule: The industrial camera switches to infrared mode and fuses the lidar point cloud to generate a thermal map.

5. The dynamic pile detection system based on industrial camera and laser radar according to claim 1, characterized in that: The calculation module of the computing unit adopts the following steps: Step S1. Align the spatial coordinates of the lidar point cloud and the camera image through the extrinsic calibration matrix; Step S2. Extract the point cloud of the pile outline and fit the minimum bounding box to calculate the volume v i is the volume of the voxel unit; Step S3: Identify the risk areas of collapse on the pile surface through a convolutional neural network.

6. The dynamic pile detection system based on industrial camera and laser radar according to claim 1, characterized in that: The solution generation of the decision unit includes: Step A1. If the pile height is detected to exceed the safety threshold, an "emergency unloading" command is generated; Step A2: If the pile shape deviates from the stable parameters, generate a "reinforce support" instruction; Step A3: Based on the weather forecast data, dispatch the rainproof cloth covering instruction in advance.

7. The dynamic pile detection system based on industrial camera and laser radar according to claim 1, characterized in that: The control instructions of the remote control unit must be verified by path simulation. The simulation engine calculates the collision probability based on the three-dimensional model of the pile and the kinematic model of the robot arm. If the risk is greater than 5%, the path is replanned. The remote control unit supports the following control modes: (1) Automatic mode: completely dependent on the instructions of the decision-making unit; (2) Semi-automatic mode: Workers preview the decision plan through AR glasses and manually confirm it; (3) Emergency mode: Directly take over the motor controller at the loading and unloading end.

8. The dynamic pile detection system based on industrial camera and laser radar according to claim 1, characterized in that: The storage unit adopts a hierarchical storage strategy, and the original data is automatically deleted after being retained for 7 days. Key event data is permanently stored and backed up to an off-site disaster recovery center, and the log data is encrypted and uploaded to the blockchain for evidence storage.

9. The dynamic pile detection system based on industrial camera and laser radar according to claim 1, characterized in that: The multi-level warning mechanism of the notification unit includes: (1) Level 1 warning: Broadcast alerts workers on site; (2) Second level warning: SMS notification to management personnel; (3) Level 3 warning: Automatically dial the emergency number and activate the sound and light alarm.

10. The dynamic pile detection system based on industrial camera and laser radar according to claim 1, characterized in that: The dynamic detection trigger mechanism further includes: (1) The laser radar scanning interval is dynamically adjusted according to the environmental visibility; (2) The industrial camera uses the frame difference method to detect tiny movements on the surface of the pile.

Images