Coal flow monitoring method for belt conveyor based on inspection robot and monocular vision

Abstract

The present invention belongs to the technical field of conveyor coal flow monitoring, and aims to solve the problem of insufficient accuracy of existing coal flow monitoring methods. A belt conveyor coal flow monitoring method based on patrol robots and monocular vision is provided. This method dynamically collects coal flow images through patrol robots, uses deep learning algorithms to achieve accurate segmentation of coal flow areas, and combines monocular depth estimation technology to reconstruct the three-dimensional point cloud of the coal flow, ultimately achieving qualitative and quantitative analysis of the coal flow. Compared with traditional methods, the present invention has the advantages of non-contact measurement, full conveyor belt coverage, high computational efficiency, and adaptability to complex environments. It can provide reliable data support for intelligent transportation systems in coal mines.

Description

Technical Field

[0001] The present invention belongs to the technical field of conveyor coal flow monitoring, and in particular relates to a belt conveyor coal flow monitoring method based on an inspection robot and monocular vision. Background Art

[0002] Currently, coal flow monitoring on underground belt conveyors in coal mines primarily relies on contact measurement methods. Electronic belt scales and nuclear scales are common contact sensors that calculate coal flow by directly measuring the weight of material on the conveyor belt. Chinese patent CN118292878A discloses a coal flow monitoring device and method for mines. This device uses a mechanical structure to break up and smooth the transported coal to improve its distribution. The coal flow is then measured through physical contact with a weighing sensor. While this method can obtain coal flow data relatively directly, such contact sensors are susceptible to dust, moisture, and mechanical vibration in the harsh underground environment, resulting in fluctuations in measurement accuracy and requiring frequent maintenance and calibration. Furthermore, contact sensors are fixed in their mounting position, making it difficult to monitor the entire length of long conveyor belts. Furthermore, the crushing device poses a risk of damaging the mechanical structure when handling non-coal foreign matter (such as anchors and gangue) in the conveyor belt coal flow.

[0003] In recent years, non-contact visual measurement methods have been gradually applied to coal flow monitoring. Binocular stereo vision systems use dual cameras to simulate the parallax principle of the human eye, reconstructing the three-dimensional shape of the coal flow and calculating its volume. Chinese patent CN118865356A discloses a coal flow monitoring method based on binocular vision. By rationally arranging a detection bracket containing a binocular camera on a scraper conveyor, the surface contour information of the coal flow is captured, and this information is reconstructed in three dimensions to obtain three-dimensional point cloud information of the coal flow. The volume of the point cloud is then calculated to obtain the coal flow size. Although this method has high accuracy, it has strict requirements for camera calibration, high computational complexity, poor real-time performance, and high hardware cost. At the same time, the complex lighting conditions and dust environment underground can affect the matching accuracy of the binocular camera.

[0004] Monocular vision methods have attracted attention due to their simple hardware and low cost. Traditional monocular methods are mainly based on two-dimensional image processing, such as edge detection, threshold segmentation, image recognition, etc. Currently, there are also some methods based on deep learning applied to coal flow monitoring tasks. Chinese patent CN119873280A discloses a real-time monitoring system and method for coal flow of scraper conveyors, which uses video AI coal flow monitoring devices on the left and right rocker arms of the coal mining machine to obtain cross-sectional information of the coal flow, and then calculates the coal flow size through mathematical modeling. This method can more conveniently obtain the approximate shape of the coal flow by shooting coal flow images, but because the images taken by the monocular camera lack depth information, only a rough estimate of the coal flow can be made. With the development of deep learning technology, monocular depth estimation methods based on neural networks have made significant progress, but their application in coal mine scenarios still faces challenges, such as insufficient model generalization capabilities and limited computing resources. Summary of the Invention

[0005] In order to solve at least one of the above-mentioned technical problems existing in the prior art, the present invention provides a method for monitoring coal flow of a belt conveyor based on an inspection robot and monocular vision.

[0006] The present invention is implemented by the following technical solution: a method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision, comprising the following steps:

[0007] Acquire a historical coal flow image of the belt conveyor, where the historical coal flow image is captured by an industrial camera carried by the inspection robot;

[0008] Performing pixel-level annotation on the historical coal flow image, dividing the area in the coal flow image into a coal flow area, a conveyor belt area, and a background area; and constructing a data set based on the annotated historical coal flow image;

[0009] Constructing an improved DeepLabV3+ semantic segmentation model, wherein the improved DeepLabV3+ semantic segmentation model is used to segment the coal flow, conveyor belt, and cluttered background in the annotated historical coal flow image and extract the coal flow region therefrom; training and optimizing the improved DeepLabV3+ semantic segmentation model based on the dataset;

[0010] Obtaining an optimal coal flow segmentation result output by the improved DeepLabV3+ semantic segmentation model, wherein the optimal coal flow segmentation result is an image containing a predicted mask of the coal flow and the conveyor belt;

[0011] Constructing a monocular depth estimation model, inputting the image containing the predicted mask of the coal flow and conveyor belt into the monocular depth estimation model to obtain a predicted depth map of the coal flow and conveyor belt; converting the predicted depth map to obtain a three-dimensional point cloud of the coal flow and conveyor belt; calculating the volume of the coal flow in the three-dimensional point cloud based on a slicing method, and calculating the coal flow rate based on the coal flow volume;

[0012] Deploy the improved DeepLabV3+ semantic segmentation model on an edge device, and deploy the edge device on the inspection robot, and use TensorRT to accelerate the inference of the improved DeepLabV3+ semantic segmentation model;

[0013] The industrial camera carried by the inspection robot obtains a real-time coal flow image of the belt conveyor, and inputs the real-time coal flow image into the DeepLabV3+ semantic segmentation model for inference to obtain an image containing a prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image;

[0014] Based on the image containing the prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image, it is judged whether the coal flow is a uniform coal flow. If the judgment result is yes, the coal flow rate is calculated by mathematical modeling. If the judgment result is no, the image containing the prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image is input into the monocular depth estimation model, and finally the coal flow rate is calculated by reconstructing the three-dimensional point cloud.

[0015] Preferably, the DeepLabV3+ network in the DeepLabV3+ semantic segmentation model consists of an encoder and a decoder, wherein the encoder part includes a backbone network, a void space pyramid pooling module, and an attention mechanism module; the decoder part is used to upsample and fuse the feature map output by the encoder part to restore the boundary information of the coal flow and the conveyor belt; the improvements to the DeepLabV3+ semantic segmentation model include:

[0016] The backbone network of the encoder is replaced with MobileNetV2, which is used to reduce the number of model parameters while maintaining accuracy. A strip convolution spatial pyramid pooling module with an attention mechanism is designed, which is used to extract feature maps output by MobileNetV2 at five different scales. An attention mechanism module is added to the encoder, which is used to enhance attention to target boundaries and internal features. A combination of cross entropy loss and Dice loss is adopted.

[0017] Preferably, the method further includes performing a preliminary qualitative judgment on the amount of coal based on the proportion of the coal flow mask in the conveyor belt mask in the image containing the prediction mask of the coal flow and the conveyor belt, and the judgment steps are as follows:

[0018] When the proportion of coal flow mask pixels exceeds 60% of the conveyor belt mask pixels, it is qualitatively judged as excessive coal;

[0019] When the coal flow mask pixel ratio is less than or equal to 60% of the conveyor belt mask pixel ratio and greater than 20% of the conveyor belt mask pixel ratio, it is qualitatively judged as normal;

[0020] When the proportion of coal flow mask pixels is less than or equal to 20% of the conveyor belt mask pixels and greater than 2% of the conveyor belt mask pixels, it is judged as insufficient coal;

[0021] When the proportion of coal flow mask pixels is less than or equal to 2% of the conveyor belt mask pixels, it is judged that there is no coal.

[0022] Preferably, the step of calculating the coal flow rate by using the monocular depth estimation model comprises:

[0023] Inputting the image containing the predicted mask of the coal flow and the conveyor belt into the monocular depth estimation model to obtain a predicted depth map of the coal flow and the conveyor belt;

[0024] According to the camera tilt angle and the conveyor belt geometric characteristics, the predicted depth map is converted into an orthographic projection view through perspective transformation;

[0025] Use Zhang Zhengyou calibration method to obtain camera intrinsic parameters and correct image distortion;

[0026] Convert the rectified orthographic projection view into a 3D point cloud using OpenCV's ReprojectImageTo3D function;

[0027] Statistical filtering and denoising are performed on the three-dimensional point cloud, and the coal flow volume is calculated by a slicing method, and the coal flow rate is estimated based on the coal flow volume.

[0028] Preferably, the coal flow in each image containing the prediction mask of the coal flow and the conveyor belt is calculated, the sampling frequency of the image is determined according to the running speed of the inspection robot and the running speed of the conveyor belt, and the calculated amount of coal flow obtained from each image is accumulated to realize the distributed calculation of the coal flow on the belt conveyor.

[0029] Preferably, in the distributed calculation of the coal flow on the belt conveyor, the relative speed of the inspection robot and the conveyor belt must be determined based on the rotary encoder installed on the inspection robot and the speed sensor on the conveyor belt, and the sampling frequency of the camera is determined based on the relative speed.

[0030] Preferably, the step of determining the sampling frequency of the camera according to the relative speed includes:

[0031] When the inspection robot and the conveyor belt move toward each other, the relative speed between the inspection robot and the conveyor belt is expressed as: , where is the speed of the conveyor belt, is the speed of the inspection robot. When the two images contain the prediction mask of the coal flow and the conveyor belt, the interval time between them is at least , where is the actual length of the distance between the upper and lower horizontal lines in the image containing the prediction mask of the coal flow and the conveyor belt, which corresponds to the actual length on the conveyor belt. The image containing the prediction mask of the coal flow and the conveyor belt has three horizontal lines, which are located at 20%, 50% and 80% of the vertical direction of the image respectively. The sampling frequency of the image when the inspection robot and the conveyor belt move towards each other is obtained as follows: ;

[0032] When the inspection robot and the conveyor belt move in the same direction, the relative speed between the inspection robot and the conveyor belt is expressed as: , when moving in the same direction, the interval between the two images containing the prediction mask of the coal flow and the conveyor belt is at least , and then the sampling frequency of the image when the inspection robot and the conveyor belt move towards each other is obtained as .

[0033] Preferably, when deploying the improved DeepLabV3+ semantic segmentation model on the edge device, it is necessary to convert the improved DeepLabV3+ semantic segmentation model into ONNX format, and use TensorRT's ONNX parser to build an optimization engine to perform FP16 quantization on the improved DeepLabV3+ semantic segmentation model; configure the CUDA and CUDNN environment on the edge device, and the edge device is connected to the camera of the industrial camera on the inspection robot to perform real-time inference on the coal flow data captured by the industrial camera.

[0034] Preferably, the method for determining whether the coal flow is uniform is:

[0035] The coal flow is uniform by judging the horizontal coordinate relationship of the intersection of the upper, middle and lower horizontal lines given in the coal flow mask and the prediction mask containing the coal flow and the conveyor belt; are the left and right intersection points between the coal flow mask and the upper horizontal line given in the image, are the left and right intersection points between the coal flow mask and the horizontal line in the middle of the image, They are the left and right intersection points between the coal flow mask and the given lower horizontal line in the image;

[0036] Since the inspection robot is shooting at an angle, the conveyor belt and the uniform coal flow appear as isosceles trapezoids in the image. When the following judgment formula is met, the coal flow is judged to be uniform; otherwise, the coal flow is judged to be non-uniform.

[0037] The determination formula is as follows:

[0038]

[0039]

[0040] Where, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points; for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, These are preset thresholds.

[0041] Preferably, the step of calculating the coal flow rate by the mathematical modeling method includes:

[0042] A mathematical model of the cross section of the conveyor belt and the coal flow is established based on the coal flow width information in the image containing the prediction mask of the coal flow and the conveyor belt, the coal flow height information obtained by 3D reconstruction, and the unloaded information of the conveyor belt;

[0043] Coal flow cross-section shape curve in the mathematical model of conveyor belt and coal flow cross section Conveyor belt cross-section curve Uniform coal flow 3D point cloud The coordinate points on the cross section are fitted by calculating the area enclosed by the two curves, and then combining the actual length of the conveyor belt corresponding to the distance between the upper and lower horizontal lines in the image containing the prediction mask of the coal flow and the conveyor belt. , and the volume of the coal flow in this section is obtained , the calculation formula is:

[0044]

[0045] Where, is the actual width of the uniform coal flow; For Perform derivation;

[0046] The corresponding coal flow rate is calculated based on the coal flow volume.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. This invention combines dynamic data collection by patrol robots with deep learning algorithms to achieve intelligent monitoring of coal flow on belt conveyors. This method overcomes the spatial limitations of traditional fixed monitoring equipment and enables comprehensive, real-time monitoring of the conveyor belt. The innovative mobile patrol mode allows for flexible adjustment of monitoring frequency and range based on actual production needs, making it particularly well-suited for the complex underground mine tunnel environment. The system possesses autonomous judgment capabilities, automatically identifying abnormal coal flow conditions and issuing early warnings, significantly enhancing the level of intelligent monitoring.

[0049] 2. This invention replaces traditional contact sensors with monocular vision technology, significantly simplifying the system architecture while maintaining measurement accuracy. Deep learning-based image processing methods effectively overcome the maintenance challenges of contact sensors, enabling stable acquisition of monitoring data. An innovative 3D reconstruction algorithm calculates coal flow volume using only a monocular camera, eliminating the complex calibration and maintenance required for binocular systems. This non-contact measurement method avoids the loss of accuracy associated with sensor wear.

[0050] 3. This invention innovatively combines semantic segmentation with monocular depth estimation to achieve integrated qualitative and quantitative analysis of coal flow. The improved DeepLabV3+ semantic segmentation model, through lightweight design and optimized attention mechanisms, significantly improves computational efficiency while maintaining segmentation accuracy. The Depth-anything monocular depth estimation algorithm accurately reconstructs the three-dimensional shape of coal flow, providing reliable point cloud data support for flow calculations. The system automatically selects the optimal calculation strategy based on the coal flow distribution characteristics, ensuring accurate measurement results under various operating conditions.

[0051] 4. This invention utilizes a distributed computing architecture that collaborates with the cloud and edge, enabling efficient operation of the monitoring system. Lightweight models deployed on edge devices can rapidly complete image processing and preliminary analysis, meeting real-time requirements. The cloud performs complex 3D reconstruction and data analysis tasks, fully leveraging computing resources. This architectural design effectively balances computing loads, reduces network transmission pressure, and enables the system to operate stably in the complex environment of underground coal mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 It is a flowchart of the method of the present invention;

[0054] Figure 2 Schematic diagram of the improved DeepLabV3+ network structure of the present invention;

[0055] Figure 3 This is the prediction result of the DeepLabV3+ semantic segmentation model;

[0056] Figure 4 This is the point cloud result after 3D reconstruction of coal flow;

[0057] Figure 5 This is a schematic diagram of the inspection robot and the conveyor belt moving towards each other;

[0058] Figure 6 This is a schematic diagram of the inspection robot moving in the same direction as the conveyor belt;

[0059] Figure 7 is a schematic diagram of an image containing a prediction mask of coal flow and a conveyor belt under uniform coal flow;

[0060] Figure 8 is a schematic diagram of an image containing a prediction mask of coal flow and a conveyor belt under non-uniform coal flow;

[0061] Figure 9 It is a fitting diagram of the coal flow cross-sectional shape curve and the conveyor belt cross-sectional curve. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention are clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other implementations derived by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0063] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention. It should be noted that in this specification, relational terms such as first and second are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0064] The present invention provides an embodiment:

[0065] like Figures 1 to 9 As shown, a method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision comprises the following steps:

[0066] S1: Obtain historical coal flow images of the belt conveyor. The historical coal flow images are captured by the industrial camera mounted on the inspection robot. The historical coal flow images include different lighting conditions, coal flow distribution states (empty, lightly loaded, fully loaded), and other working conditions.

[0067] S2: Use annotation tools such as LabelMe to perform pixel-level annotation on historical coal flow images, and divide the areas in the coal flow images into coal flow areas, conveyor belt areas, and background areas. Build a dataset based on the annotated historical coal flow images, and divide the annotated images into training, validation, and test sets in an 8:1:1 ratio. Perform data augmentation on the dataset, including random rotation, brightness adjustment, and noise addition, to improve the model's generalization ability. LabelMe is an open source image annotation tool.

[0068] S3: Build an improved DeepLabV3+ semantic segmentation model. The improved DeepLabV3+ semantic segmentation model is used to segment the coal flow, conveyor belt, and cluttered background in the annotated historical coal flow images and extract the coal flow area from them. The improved DeepLabV3+ semantic segmentation model is trained and optimized based on the dataset.

[0069] The DeepLabV3+ network in the DeepLabV3+ semantic segmentation model consists of an encoder and a decoder. The encoder part mainly includes the backbone network, the atrous spatial pyramid pooling module (ASPP) and the attention mechanism module (CBAM). The backbone network and the atrous spatial pyramid pooling module are responsible for obtaining the semantic information in the feature map and outputting the final feature map through splicing and convolution operations. The decoder part is used to upsample and fuse the feature map output by the encoder part, and fuse the low-resolution feature map from the encoder with the high-resolution feature map in the backbone network, and finally restore the boundary information between the coal flow and the conveyor belt. Since the coal flow information is continuous, the present invention optimizes the above algorithm framework, and the improvements to the DeepLabV3+ semantic segmentation model include:

[0070] 1. The original Xception backbone network was replaced with MobileNetV2, significantly reducing the number of model parameters while maintaining accuracy. Xception is a deep learning network, and MobileNetV2 is a lightweight deep learning network. 2. The strip convolutional spatial pyramid pooling module with attention mechanism (CA-SASPP) was introduced, combining the channel attention mechanism with strip void spatial pyramid pooling to improve the ability to extract coal flow edge features. 3. The attention mechanism module was introduced in the encoder part to enhance the model's focus on key areas. 4. A combination of cross-entropy loss and Dice loss was used to balance global and local feature learning. Dice loss is a loss function used to measure the similarity between predicted segmentation results and true labels.

[0071] The improved DeepLabV3+ semantic segmentation model is as follows:

[0072] Encoder part:

[0073] Mobilenetv2 is mainly divided into five layers, including the input layer, the first inverted residual block group, the second inverted residual block group, the third inverted residual block group, and the output layer.

[0074] The strip convolution spatial pyramid pooling module with attention mechanism mainly extracts the feature maps output by Mobilenetv2 at five different scales. The first scale is the normal Convolution; the second and third scales are different in terms of void ratio Depthwise separable convolution is mainly used to extract larger and smaller features; the fourth scale is composed of two strip convolutions, namely and , which can extract vertical and horizontal features in the image and speed up the calculation; the fifth scale is global average pooling. The feature maps of these five scales are spliced ​​with the feature map residual output by Mobilenetv2, and then a Convolution reduces the number of channels and recombines and weights the features through the residual; then the attention mechanism module enhances the focus on the target boundary and internal features, thereby reducing errors and finally obtaining a deep semantic feature layer.

[0075] Decoder part:

[0076] First, we obtain the shallow semantic feature layer from Mobilenetv2, and then After convolution, it is stacked with the upsampled deep semantic feature layer obtained in the encoder, and the feature fusion is performed. Convolution is used to extract features, and then upsampling is used to resize the image to the same size as the input image.

[0077] S4: Obtain the optimal coal flow segmentation result output by the improved DeepLabV3+ semantic segmentation model. The optimal coal flow segmentation result is an image containing the predicted mask of the coal flow and the conveyor belt;

[0078] It also includes a preliminary qualitative judgment of the amount of coal based on the proportion of the coal flow mask in the conveyor belt mask in the image containing the predicted mask of the coal flow and the conveyor belt. The judgment steps are as follows: when the proportion of the coal flow mask pixels exceeds 60% of the conveyor belt mask pixels, it is qualitatively judged as too much coal; when the proportion of the coal flow mask pixels is less than or equal to 60% of the conveyor belt mask pixels and greater than 20% of the conveyor belt mask pixels, it is qualitatively judged as normal; when the proportion of the coal flow mask pixels is less than or equal to 20% of the conveyor belt mask pixels and greater than 2% of the conveyor belt mask pixels, it is judged as less coal; when the proportion of the coal flow mask pixels is less than or equal to 2% of the conveyor belt mask pixels, it is judged as no coal.

[0079] S5: Construct a monocular depth estimation model and input the image containing the predicted mask of the coal flow and conveyor belt into the monocular depth estimation model to obtain a predicted depth map of the coal flow and conveyor belt; convert the predicted depth map to obtain a three-dimensional point cloud of the coal flow and conveyor belt; calculate the volume of the coal flow in the three-dimensional point cloud based on the slicing method, and calculate the coal flow rate based on the coal flow volume;

[0080] The steps for calculating coal flow using a monocular depth estimation model include:

[0081] The image containing the predicted mask of the coal flow and conveyor belt is input into the monocular depth estimation model. The predicted depth map of the coal flow and conveyor belt is obtained by the Depth-anything monocular depth estimation algorithm.

[0082] According to the camera tilt angle and the conveyor belt geometric characteristics, the predicted depth map is converted into an orthographic projection view through perspective transformation;

[0083] Use Zhang Zhengyou calibration method to obtain camera intrinsic parameters and correct image distortion;

[0084] The corrected orthographic projection view is converted into a 3D point cloud using the OpenCV ReprojectImageTo3D function. OpenCV is an open source computer vision library, and the ReprojectImageTo3D function is a function that converts an image into a 3D point cloud.

[0085] The 3D point cloud is subjected to statistical filtering and denoising, and the coal flow volume is calculated by the slicing method, and the coal flow rate is estimated based on the coal flow volume.

[0086] The system also calculates the coal flow rate for each image containing a prediction mask of the coal flow and conveyor belt, determines the image sampling frequency based on the inspection robot's operating speed and the conveyor belt's operating speed, and accumulates the coal flow calculations for each image to achieve distributed calculation of the coal flow rate on the belt conveyor. This distributed calculation of the coal flow rate on the belt conveyor requires first determining the relative speed of the inspection robot and the conveyor belt using the rotary encoder installed on the inspection robot and the speed sensor on the conveyor belt. This relative speed determines the camera's sampling frequency to ensure that the coal flow within the camera's field of view is neither duplicated nor missing.

[0087] The steps of determining the sampling frequency of the camera according to the relative speed include:

[0088] When the inspection robot and the conveyor belt move toward each other, the relative speed between the inspection robot and the conveyor belt is expressed as: , where is the speed of the conveyor belt, is the speed of the inspection robot. When the two images contain the prediction mask of the coal flow and the conveyor belt, the interval time between them is at least , where The distance between the upper and lower horizontal lines in the image containing the prediction mask of the coal flow and the conveyor belt corresponds to the actual length of the conveyor belt. There are three horizontal lines in the image containing the prediction mask of the coal flow and the conveyor belt, which are located at 20%, 50% and 80% of the vertical direction of the image respectively. Since the inspection robot is shooting at an angle, the camera has a large field of view. From the experiment, it can be concluded that the maximum length of the conveyor belt in its field of view is 2m. However, the degree of distortion at both ends of the camera is large during the 3D reconstruction process, so only the middle area is selected. In this embodiment ; Then the sampling frequency of the image when the inspection robot and the conveyor belt move towards each other is obtained as ;

[0089] When the inspection robot and the conveyor belt move in the same direction, the relative speed between the inspection robot and the conveyor belt is expressed as: , when moving in the same direction, the interval between the two images containing the prediction mask of the coal flow and the conveyor belt is at least , and then the sampling frequency of the image when the inspection robot and the conveyor belt move towards each other is obtained as .

[0090] For example: When the inspection robot and the conveyor belt move towards each other, the speed of the conveyor belt =2.5m / s, the speed of the inspection robot is =1m / s, its relative speed is =3.5m / s. Under this condition, the interval between two coal flow images is at least =0.343s, the sampling frequency of the image is Hz, that is, a speed of at least 3 frames per second is required to meet the minimum requirement for segmented measurement of coal flow.

[0091] S6: Deploy the improved DeepLabV3+ semantic segmentation model on an edge device, which is also deployed on an inspection robot. Use TensorRT to accelerate inference on the improved DeepLabV3+ semantic segmentation model. The edge device is an NVIDIA Jetson Orin Nano. When deploying the improved DeepLabV3+ semantic segmentation model on the edge device, convert the improved DeepLabV3+ semantic segmentation model to the ONNX format. Use TensorRT's ONNX parser to build an optimization engine and perform FP16 (16-bit floating point) quantization on the improved DeepLabV3+ semantic segmentation model to reduce computational complexity and memory usage. Configure the CUDA and CUDNN environments on the NVIDIA Jetson Orin Nano edge device. Connect the edge device to the industrial camera on the inspection robot and perform real-time inference on the coal flow data captured by the industrial camera. TensorRT is a deep learning model acceleration framework, NVIDIA Jetson Orin Nano is a model of edge device, ONNX is a deep learning model format, and both CUDA and CUDNN environments are deep learning-dependent environments.

[0092] S7: The industrial camera on the inspection robot captures real-time coal flow images on the belt conveyor. This real-time coal flow image is fed into the DeepLabV3+ semantic segmentation model for inference. The resulting image, containing the predicted mask of the coal flow and conveyor belt, is uploaded to the cloud server.

[0093] S8: Based on the image containing the prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image, determine whether the coal flow is a uniform coal flow. If the judgment result is yes, calculate the coal flow rate by mathematical modeling. If the judgment result is no, input the image containing the prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image into the monocular depth estimation model, and finally calculate the coal flow rate by reconstructing the three-dimensional point cloud.

[0094] The method to determine whether the coal flow is uniform is:

[0095] The coal flow is uniform by judging the horizontal coordinate relationship of the intersection of the upper, middle and lower horizontal lines given in the coal flow mask and the prediction mask containing the coal flow and the conveyor belt; are the left and right intersection points between the coal flow mask and the upper horizontal line given in the image, are the left and right intersection points between the coal flow mask and the horizontal line in the middle of the image, They are the left and right intersection points between the coal flow mask and the given lower horizontal line in the image;

[0096] Since the inspection robot is shooting at an angle, the conveyor belt and the uniform coal flow appear as isosceles trapezoids in the image. When the following judgment formula is met, the coal flow is judged to be uniform; otherwise, the coal flow is judged to be non-uniform.

[0097] The determination formula is as follows:

[0098]

[0099]

[0100] Where, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points; for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, These are preset thresholds.

[0101] In this embodiment, the resolution of a single image containing the prediction mask of the coal flow and the conveyor belt is , indicating that the image width is 3024 pixels and the length is 4032 pixels; the preset threshold is 100 pixels, It is 150 pixels.

[0102] The steps for calculating coal flow by mathematical modeling include:

[0103] Based on the coal flow width information in the image containing the prediction mask of the coal flow and the conveyor belt, the coal flow height information obtained by 3D reconstruction and the conveyor belt no-load information, a mathematical model of the conveyor belt and coal flow cross section is established; the coal flow cross section shape curve in the mathematical model of the conveyor belt and coal flow cross section Conveyor belt cross-section curve Uniform coal flow 3D point cloud The coordinate points on the cross section are fitted by calculating the area enclosed by the two curves, and then combining the actual length of the conveyor belt corresponding to the distance between the upper and lower horizontal lines in the image containing the prediction mask of the coal flow and the conveyor belt. , and the volume of the coal flow in this section is obtained , based on the volume of the coal flow Estimate the corresponding coal flow rate and coal flow volume The calculation formula is:

[0104]

[0105] Where, is the actual width of the uniform coal flow, For Perform the derivation.

[0106] The coal flow width information, coal flow height information, and conveyor belt empty load information are all obtained through three-dimensional reconstruction. Because it is a uniform coal flow, only the height and width of the center part of the point cloud are obtained (that is, the height of the position corresponding to the second horizontal line in the image containing the prediction mask of the coal flow and conveyor belt). Then, the coal flow width information and coal flow height information can be obtained by approximate calculation using mathematical modeling methods; the conveyor belt empty load information is also obtained from the point cloud. After obtaining these point cloud information, two curves are fitted, namely the coal flow cross-sectional shape curve Conveyor belt cross-section curve .

[0107] The foregoing description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed herein should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A coal flow monitoring method for a belt conveyor based on a patrol robot and monocular vision is characterized in that: The following steps are involved: Acquire a historical coal flow image of the belt conveyor, where the historical coal flow image is captured by an industrial camera carried by the inspection robot; Performing pixel-level annotation on the historical coal flow image, dividing the area in the coal flow image into a coal flow area, a conveyor belt area, and a background area; And build a dataset based on the annotated historical coal flow images; Constructing an improved DeepLabV3+ semantic segmentation model, wherein the improved DeepLabV3+ semantic segmentation model is used to segment the coal flow, conveyor belt, and cluttered background in the annotated historical coal flow image and extract the coal flow region therefrom; Training and optimizing the improved DeepLabV3+ semantic segmentation model based on the dataset; Obtaining an optimal coal flow segmentation result output by the improved DeepLabV3+ semantic segmentation model, wherein the optimal coal flow segmentation result is an image containing a predicted mask of the coal flow and the conveyor belt; Constructing a monocular depth estimation model, inputting the image containing the predicted mask of the coal flow and the conveyor belt into the monocular depth estimation model to obtain a predicted depth map of the coal flow and the conveyor belt; converting the predicted depth map to obtain a three-dimensional point cloud of the coal flow and the conveyor belt; and calculating the coal flow volume in the three-dimensional point cloud based on a slicing method, and calculating the coal flow rate based on the coal flow volume; Deploy the improved DeepLabV3+ semantic segmentation model on an edge device, and deploy the edge device on the inspection robot, and use TensorRT to accelerate the inference of the improved DeepLabV3+ semantic segmentation model; The industrial camera carried by the inspection robot obtains a real-time coal flow image of the belt conveyor, and inputs the real-time coal flow image into the DeepLabV3+ semantic segmentation model for inference to obtain an image containing a prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image; Based on the image containing the prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image, it is judged whether the coal flow is a uniform coal flow. If the judgment result is yes, the coal flow rate is calculated by mathematical modeling. If the judgment result is no, the image containing the prediction mask of the coal flow and the conveyor belt corresponding to the real-time coal flow image is input into the monocular depth estimation model, and finally the coal flow rate is calculated by reconstructing the three-dimensional point cloud.

2. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 1, characterized in that: The DeepLabV3+ network in the DeepLabV3+ semantic segmentation model consists of an encoder and a decoder, where the encoder part includes a backbone network, a void spatial pyramid pooling module, and an attention mechanism module; The decoder part is used to upsample and fuse the feature maps output by the encoder part to restore the boundary information between the coal flow and the conveyor belt; Improvements to the DeepLabV3+ semantic segmentation model include: The backbone network of the encoder is replaced with MobileNetV2, which is used to reduce the number of model parameters while maintaining accuracy. A strip convolution spatial pyramid pooling module with an attention mechanism is designed, which is used to extract feature maps output by MobileNetV2 at five different scales. An attention mechanism module is added to the encoder, which is used to enhance attention to target boundaries and internal features. A combination of cross entropy loss and Dice loss is adopted.

3. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 1, characterized in that: The method further includes performing a preliminary qualitative judgment on the amount of coal based on the proportion of the coal flow mask in the conveyor belt mask in the image containing the predicted mask of the coal flow and the conveyor belt, and the judgment steps are as follows: When the proportion of coal flow mask pixels exceeds 60% of the conveyor belt mask pixels, it is qualitatively judged as excessive coal; When the coal flow mask pixel ratio is less than or equal to 60% of the conveyor belt mask pixel ratio and greater than 20% of the conveyor belt mask pixel ratio, it is qualitatively judged as normal; When the proportion of coal flow mask pixels is less than or equal to 20% of the conveyor belt mask pixels and greater than 2% of the conveyor belt mask pixels, it is judged as insufficient coal; When the proportion of coal flow mask pixels is less than or equal to 2% of the conveyor belt mask pixels, it is judged that there is no coal.

4. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 1, characterized in that: The steps of calculating the coal flow rate by using the monocular depth estimation model include: Inputting the image containing the predicted mask of the coal flow and the conveyor belt into the monocular depth estimation model to obtain a predicted depth map of the coal flow and the conveyor belt; According to the camera tilt angle and the conveyor belt geometric characteristics, the predicted depth map is converted into an orthographic projection view through perspective transformation; Use Zhang Zhengyou calibration method to obtain camera intrinsic parameters and correct image distortion; Convert the rectified orthographic projection view into a 3D point cloud using OpenCV's ReprojectImageTo3D function; Statistical filtering and denoising are performed on the three-dimensional point cloud, and the coal flow volume is calculated by a slicing method, and the coal flow rate is estimated based on the coal flow volume.

5. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 4, characterized in that: The coal flow rate in each image containing the prediction mask of the coal flow and the conveyor belt is calculated. The image sampling frequency is determined according to the running speed of the inspection robot and the conveyor belt. The calculation amount of the coal flow calculated for each image is accumulated to realize the distributed calculation of the coal flow on the belt conveyor.

6. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 5, characterized in that: In the distributed calculation of the coal flow on the belt conveyor, it is necessary to first determine the relative speed of the inspection robot and the conveyor belt based on the rotary encoder installed on the inspection robot and the speed sensor on the conveyor belt, and determine the sampling frequency of the camera based on the relative speed.

7. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 6, characterized in that: The step of determining the sampling frequency of the camera according to the relative speed includes: When the inspection robot and the conveyor belt move toward each other, the relative speed between the inspection robot and the conveyor belt is expressed as: , where is the speed of the conveyor belt, is the speed of the inspection robot. When the two images contain the prediction mask of the coal flow and the conveyor belt, the interval time between them is at least , where is the actual length of the distance between the upper and lower horizontal lines in the image containing the prediction mask of the coal flow and the conveyor belt, which corresponds to the actual length on the conveyor belt. The image containing the prediction mask of the coal flow and the conveyor belt has three horizontal lines, which are located at 20%, 50% and 80% of the vertical direction of the image respectively. The sampling frequency of the image when the inspection robot and the conveyor belt move towards each other is obtained as follows: ; When the inspection robot and the conveyor belt move in the same direction, the relative speed between the inspection robot and the conveyor belt is expressed as: , when moving in the same direction, the interval between the two images containing the prediction mask of the coal flow and the conveyor belt is at least , and then the sampling frequency of the image when the inspection robot and the conveyor belt move towards each other is obtained as .

8. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 1, characterized in that: When deploying the improved DeepLabV3+ semantic segmentation model on the edge device, it is necessary to convert the improved DeepLabV3+ semantic segmentation model into ONNX format, and use TensorRT's ONNX parser to build an optimization engine to perform FP16 quantization on the improved DeepLabV3+ semantic segmentation model; configure the CUDA and CUDNN environment on the edge device, and connect the edge device to the camera of the industrial camera on the inspection robot to perform real-time inference on the coal flow data captured by the industrial camera.

9. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 1, characterized in that: The method for determining whether the coal flow is uniform is as follows: The coal flow is uniform by judging the horizontal coordinate relationship of the intersection of the upper, middle and lower horizontal lines given in the coal flow mask and the prediction mask containing the coal flow and the conveyor belt; are the left and right intersection points between the coal flow mask and the upper horizontal line given in the image, are the left and right intersection points between the coal flow mask and the horizontal line in the middle of the image, They are the left and right intersection points between the coal flow mask and the given lower horizontal line in the image; Since the inspection robot is shooting at an angle, the conveyor belt and the uniform coal flow appear as isosceles trapezoids in the image. When the following judgment formula is met, the coal flow is judged to be uniform; otherwise, the coal flow is judged to be non-uniform. The determination formula is as follows: Where, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points; for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, for The pixel horizontal coordinate of the point is subtracted The difference in the pixel horizontal coordinates of the points, These are preset thresholds.

10. The method for monitoring coal flow on a belt conveyor based on a patrol robot and monocular vision according to claim 9, characterized in that: The steps of calculating the coal flow rate by the mathematical modeling method include: A mathematical model of the cross section of the conveyor belt and the coal flow is established based on the coal flow width information in the image containing the prediction mask of the coal flow and the conveyor belt, the coal flow height information obtained by 3D reconstruction, and the unloaded information of the conveyor belt; Coal flow cross-section shape curve in the mathematical model of conveyor belt and coal flow cross section Conveyor belt cross-section curve Uniform coal flow 3D point cloud The coordinate points on the cross section are fitted by calculating the area enclosed by the two curves, and then combining the actual length of the conveyor belt corresponding to the distance between the upper and lower horizontal lines in the image containing the prediction mask of the coal flow and the conveyor belt. , and the volume of the coal flow in this section is obtained , the calculation formula is: Where, is the actual width of the uniform coal flow, For Perform derivation; The corresponding coal flow rate is calculated based on the coal flow volume.

Images