Underground belt deviation and tearing detection method and system based on TOF camera

Abstract

The invention belongs to the technical field of 3D visual sensing, and relates to an underground belt deviation and tearing detection method and system based on a TOF camera, and the method comprises the steps: collecting a depth image of an operation region of a belt conveyor, setting a camera coordinate system and a belt conveyor coordinate system, converting the camera coordinate system to the belt conveyor coordinate system, and obtaining a depth image of the operation region of the belt conveyor; calculating the horizontal offset of the belt edge and the reference point according to the converted coordinates, and obtaining the deviation degree of the belt by using the horizontal offset; when the belt deviation degree exceeds a target threshold value, the depth image is input into a belt tearing detection model, a tearing detection result is obtained, the belt tearing detection model is obtained by training a YOLO network model through a training set, and the training set comprises a belt tearing sample image and a tearing area label. The detection precision can be improved, the operation efficiency of the belt conveying system is improved, and the maintenance cost of equipment is reduced.

Description

Technical Field

[0001] The present invention relates to the field of 3D vision sensing technology, and particularly to a method and system for detecting belt deviation and tearing in a mine based on a TOF camera. Background Art

[0002] Belt conveyors are widely used in mine scenarios for material transportation. Belt deviation, tearing, and overloading are the most common faults during the operation of belt conveyors. These situations can not only cause serious damage to belt conveyor equipment, affecting production efficiency, but also pose potential safety hazards.

[0003] With the rapid development of computer vision technology, vision detection has become an important means in the field of conveyor detection. Currently, common vision detection methods for deviation and tearing include infrared detection, structured light detection, etc. However, these detection methods are limited by underground light, complex environments, and equipment, and the detection effects of belt deviation and tearing are not precise enough. Existing technical solutions have disclosed some methods based on vision technology, and have also disclosed methods for detecting belt conveyors using vision technology in conveyor detection:

[0004] An existing technical solution discloses a method for detecting deviation of a coal conveying belt based on computer vision, including: emitting linear structured light to a first area of the coal conveying belt, and the linear structured light forms a first light spot in the first area; collecting a first video stream of the first area in real time; extracting a first image frame from the first video stream, and performing first light spot feature extraction on the first image frame; comparing the preset light spot image with the extracted first light spot to determine whether the coal conveying belt has a torsional deviation. The method for detecting deviation of a coal conveying belt based on computer vision provided by this invention can form linear structured light at the two side edges of the coal conveying belt, and then collect the video stream of the light spot area through a camera, analyze the video stream, and at the same time, by comparing the preset light spot image with the collected light spot image, if distortion occurs, it can be determined that the coal conveying belt has a torsional deviation.

[0005] This invention determines whether the belt is deviated through the light spot formed by linear structured light. Its detection principle depends on the morphological change of the light spot, so it has high requirements for environmental conditions and requires strict control of the light projection angle and the position of the fixed light source. However, in a complex mine environment, due to the complex and changeable underground light conditions and the random distribution of materials on the belt conveyor, the light spot features are easily interfered by various factors such as environmental light changes and material occlusion. This leads to unstable extraction of light spot features, thereby reducing the reliability and stability of belt deviation detection.

[0006] The prior art solution also discloses a longitudinal tear detection device and method for a conveyor belt based on the fusion of TOF and binocular images. The device is arranged between the upper and lower belts of the conveyor belt and includes a central processing unit, a data storage module, an image processing module, and an image acquisition module; a power supply module is connected to the central processing unit, the image processing module is connected to the image acquisition module, and the central processing unit is respectively connected to the data storage module and the image processing module; this method transmits the images collected by the image acquisition module to the image processing module, and then transmits the processed data to the central processing unit; the central processing unit fuses the TOF disparity map and the binocular disparity map to calculate the depth map and analyzes and judges whether the conveyor belt has a longitudinal tear according to the depth map.

[0007] This technical solution mainly obtains the depth map through the fusion of TOF and binocular images and judges whether there is a longitudinal tear on the surface of the conveyor belt according to whether the depth value in the depth map exceeds the threshold. Although the TOF technology is used, this prior art solution is for single problem detection and does not achieve multi-functional integrated monitoring. Summary of the Invention

[0008] In order to solve the problems existing in the above prior art, the purpose of the present invention is to provide a method and system for detecting belt deviation and tear in an underground mine based on a TOF camera, which can effectively identify belt deviation and tear conditions, ensure the stability and safety of the conveying process, improve the detection accuracy, enhance the operation efficiency of the belt conveying system, and reduce the equipment maintenance cost.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A method for detecting belt deviation and tear in an underground mine based on a TOF camera includes:

[0011] Collect the depth image of the operating area of the belt conveyor, set the camera coordinate system and the belt conveyor coordinate system, transform the camera coordinate system to the belt conveyor coordinate system, calculate the horizontal offset between the belt edge and the reference point according to the transformed coordinates, and use the horizontal offset to obtain the degree of belt deviation.

[0012] When the degree of belt deviation exceeds the target threshold, input the depth image into the belt tear detection model to obtain the tear detection result. The belt tear detection model is obtained by training the YOLO network model using a training set, and the training set includes: belt tear sample images and tear area labels.

[0013] Optionally, transforming the camera coordinate system to the belt conveyor coordinate system includes:

[0014] Set a target reference point on the belt surface as the origin of the belt conveyor coordinate system, establish the belt conveyor coordinate system, and establish the camera coordinate system with the optical center of the TOF camera as the origin.

[0015] Convert the pixel coordinate system to the camera coordinate system, and convert the camera coordinate system to the belt conveyor coordinate system;

[0016] Wherein, the axis direction of the camera coordinate system is consistent with that of the belt conveyor coordinate system.

[0017] Optionally, converting the pixel coordinate system to the camera coordinate system includes:

[0018]

[0019] Wherein, f x and f y are the camera focal lengths, Z is the depth value, (c x， c y ) is the coordinate of the optical center in the pixel coordinate system, (u, v) is the pixel coordinate of the reference point in the pixel coordinate system, X c , Y c , Z c are the three-dimensional coordinates of the reference point in the camera coordinate system.

[0020] Optionally, converting the camera coordinate system to the belt conveyor coordinate system includes:

[0021]

[0022] Wherein, R is the rotation matrix and T is the translation vector.

[0023] Optionally, converting the camera coordinate system to the belt conveyor coordinate system further includes:

[0024] When the rotation matrix and the translation vector , converting the camera coordinate system to the belt conveyor coordinate system is:

[0025]

[0026] Wherein, h is the distance.

[0027] Optionally, calculating the horizontal offset between the belt edge and the reference point according to the converted coordinates includes:

[0028] Δx = |D left - D ref_left | + |D right - D ref_right |

[0029] Wherein, D left , D right are the horizontal distances from the real-time edge of the belt to the reference point, D ref_left , Dref_right It is the horizontal distance from the belt edge to the reference point in the standard model.

[0030] To achieve the above object, the present invention also provides an underground belt deviation and tear detection system based on a TOF camera, including:

[0031] An image acquisition module for acquiring depth images of the operating area of the belt conveyor;

[0032] A deviation detection module for setting the camera coordinate system and the belt conveyor coordinate system, converting the camera coordinate system to the belt conveyor coordinate system, calculating the horizontal offset between the belt edge and the reference point according to the converted coordinates, and using the horizontal offset to obtain the degree of belt deviation;

[0033] A tear detection module for, when the degree of belt deviation exceeds the target threshold, inputting the depth image into a belt tear detection model to obtain a tear detection result, where the belt tear detection model is obtained by training a YOLO network model using a training set, and the training set includes: belt tear sample images and tear area labels.

[0034] The beneficial effects of the present invention are:

[0035] The TOF camera used in the present invention adopts the time-of-flight principle and directly obtains the depth information of the scene by emitting and receiving optical signals, and can accurately capture the three-dimensional shape and position changes of the belt edge. Compared with the linear structured light method, the TOF camera has a lower dependence on external environmental light, and at the same time has a stronger anti-interference ability against environments such as noise, dust, and smoke, and can maintain high-precision detection performance in the mine environment.

[0036] The present invention can effectively detect belt deviation and tear, including methods based on depth image processing and training a belt tear detection model, to improve the accuracy and robustness of detection, combine coordinate conversion algorithms, comprehensively analyze the motion characteristics of the belt conveyor, improve the accuracy and reliability of detection, and ensure production safety and efficiency.

[0037] The present invention effectively detects belt deviation and tear, solves the problem of poor recognition of the belt running state by traditional vision methods, thereby timely discovers belt deviation and tear situations, and ensures the normal transmission of the belt conveyor. The solution of the present invention can be combined with sensor detection to improve the accuracy of prediction results, or used as a means for fault diagnosis of belt conveyors and sensors. Description of the Drawings

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for use in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0039] Figure 1 Flowchart of a method for detecting belt deviation and tearing in underground mines based on a TOF camera according to an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of a system for detecting belt deviation and tearing in underground mines based on a TOF camera according to an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of coordinate transformation according to an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of TOF camera arrangement and coordinate system establishment according to an embodiment of the present invention. Detailed implementation manners

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0044] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0045] As Figure 1 shown, this embodiment discloses a method for detecting belt deviation and tearing in underground mines based on a TOF camera, including: collecting depth images of the operating area of the belt conveyor, setting the camera coordinate system and the belt conveyor coordinate system, transforming the camera coordinate system to the belt conveyor coordinate system, calculating the horizontal offset between the belt edge and the reference point according to the transformed coordinates, and obtaining the degree of belt deviation using the horizontal offset; when the degree of belt deviation exceeds the target threshold, input the depth image into the belt tearing detection model to obtain the tearing detection result, and the belt tearing detection model is obtained by training the YOLO network model using a training set, and the training set includes: belt tearing sample images and tearing area labels.

[0046] Specifically, this embodiment discloses a method for detecting belt deviation and tearing in underground mines based on a TOF camera, including: collecting depth image data of the belt running area by arranging a TOF camera above the belt, performing coordinate transformation on the collected depth image, then preprocessing the depth image, performing edge detection based on the preprocessed depth image, calculating the offset of the belt edge, and further determining whether the belt is deviated. By collecting belt tearing image data to construct a training data set and training a belt tearing detection model based on a deep learning algorithm, the tearing degree on the belt surface is identified. This method avoids the dependence on factors such as ambient light and material occlusion in traditional methods, improves the stability and reliability of detection, and also has the ability of multi-functional detection.

[0047] Furthermore, the installation and configuration of the TOF camera include: installing the TOF camera at a position above the belt conveyor, selecting appropriate distance h and angle. Make the optical axis of the camera perpendicular to the surface of the belt conveyor to reduce the complexity of coordinate transformation. Make the image cover the complete movement path of the belt edge to ensure that the running states of both sides of the belt can be comprehensively captured.

[0048] A TOF camera is a three-dimensional imaging device based on the Time-of-Flight (TOF) technology. By emitting light pulses and measuring the time difference required for the light pulses to be emitted and reflected back, the distance between the object and the camera is calculated, thereby generating a high-precision depth image.

[0049] Furthermore, real-time data collection includes: the TOF camera collects depth images of the belt conveyor running area at a fixed frequency. The abscissa u and ordinate v of the pixel are respectively the column number and row number in its image array, and the pixel value represents the distance from the corresponding scene point to the camera (i.e., the depth value Z). The depth image is transmitted to the host computer for further processing.

[0050] Furthermore, a large number of belt tearing sample images are collected, including tears of different types, sizes and positions, and the tearing areas in the images are labeled.

[0051] Model design and training include: selecting a suitable YOLO network structure, building a YOLO network model, and at the same time adjusting appropriate model parameters and weights. Input the training data set in step 201 into the YOLO model for training, and stop training when the loss function reaches the threshold, and output the trained belt tearing detection model.

[0052] Real-time tearing detection includes: the system inputs the depth image of the belt into the trained belt tearing detection model in real time to determine whether there is a tearing area. If it is detected that the belt is torn, the system immediately triggers an alarm.

[0053] Further, converting the camera coordinate system to the belt conveyor coordinate system includes: setting a target reference point on the belt surface as the origin of the belt conveyor coordinate system, establishing the belt conveyor coordinate system, and establishing a camera coordinate system with the optical center of the TOF camera as the origin; converting the pixel coordinate system to the camera coordinate system and then converting the camera coordinate system to the belt conveyor coordinate system; wherein, the axis directions of the camera coordinate system are consistent with those of the belt conveyor coordinate system.

[0054] Specifically, establishing the coordinate system includes: selecting a specific reference point on the belt surface as the origin of the belt conveyor coordinate system and establishing the belt conveyor coordinate system. The X-axis in the belt conveyor coordinate system is along the belt running direction, the Y-axis is perpendicular to the belt width direction, and the Z-axis is perpendicular to the belt conveyor surface and upward, as Figures 3 - 4 shown.

[0055] Establish a camera coordinate system with the optical center of the TOF camera as the origin. The X, Y, and Z axis directions in the camera coordinate system are consistent with those of the belt conveyor coordinate system.

[0056] Since the belt conveyor is defaulted as the reference object in this system, the belt conveyor coordinate system is regarded as the world coordinate system.

[0057] The camera coordinate system conversion includes:

[0058] Converting the point in the pixel coordinate system to the point in the camera coordinate system P c below:

[0059]

[0060] wherein, f x and f y are the camera focal lengths, Z is the depth value, and (c x， c y ) is the coordinate of the optical center in the pixel coordinate system.

[0061] The belt conveyor coordinate system conversion includes:

[0062] Converting the point P c under the camera coordinate system to the point P c under the belt conveyor coordinate system P b :

[0063]

[0064] wherein, R is the rotation matrix, T is the translation vector, and u and v are the abscissa and ordinate of the image pixels.

[0065] Preferably, the camera optical axis is installed perpendicular to the surface of the belt conveyor, which can reduce the complexity of coordinate conversion. Since the directions of the axes of the belt conveyor coordinate system and the camera coordinate system are the same, and the origin of the belt conveyor coordinate system is directly below the camera optical center with a distance of h, the rotation matrix can be obtained. Translation vector

[0066] The conversion formula can be simplified as:

[0067] Specifically, the belt deviation detection includes:

[0068] The left and right edge lines of the belt are identified through the edge detection algorithm. Based on the installed and debugged belt position, reference lines are established with the belt edges in the image, and the deviation degree is calculated according to the distance between the detected belt and the reference line. The horizontal offset Δx between the belt edge and the reference point is calculated through the converted coordinates:

[0069] Δx = |D left - D ref_left | + |D right - D ref_right |

[0070] where D left , D right is the horizontal distance from the real-time edge of the belt to the reference point, and D ref_left , D ref_right is the horizontal distance from the belt edge to the reference point in the standard model. According to the safety management requirements, a deviation threshold X Threshold is customized. If Δx > X Threshold , it is determined that the belt is deviated, and the system immediately triggers an alarm.

[0071] The horizontal distance from the belt edge to the reference point in the standard model is: The intersection point of the perpendicular line from any point on the belt edge to the reference line and the reference line is the reference point, and the distance of this perpendicular line is the horizontal distance D from the belt edge to the reference point.

[0072] As Figure 2 shown, this embodiment also provides an underground belt deviation and tear detection system based on a TOF camera, including: an image acquisition module for acquiring depth images of the operating area of the belt conveyor; a deviation detection module for setting the camera coordinate system and the belt conveyor coordinate system, converting the camera coordinate system to the belt conveyor coordinate system, calculating the horizontal offset between the belt edge and the reference point according to the converted coordinates, and obtaining the belt deviation degree by using the horizontal offset; a tear detection module for, when the belt deviation degree exceeds the target threshold, inputting the depth image into the belt tear detection model to obtain the tear detection result, and the belt tear detection model is obtained by training the YOLO network model with a training set, and the training set includes: belt tear sample images and tear area labels.

[0073] Specifically, this embodiment also provides a downhole belt deviation and tear detection system based on a TOF camera, including:

[0074] An image acquisition module that acquires depth image data of the conveyor belt through a TOF camera, and captures the three-dimensional shape and operating state of the belt surface in real time. It also sets functions such as the camera position, the acquisition screen, and transmitting the image to the processing end.

[0075] A coordinate transformation module that converts the pixel coordinates in the depth image acquired by the TOF camera into three-dimensional coordinates in the camera coordinate system, and further converts them into three-dimensional coordinates in the belt coordinate system in combination with the spatial relationship between the camera optical center and the belt reference point for subsequent processing.

[0076] A data preprocessing module, including a data preprocessing unit, which is used to perform preprocessing on the depth image obtained from the TOF camera, including methods such as image denoising, brightness enhancement, and contrast enhancement, and transmits the processed data to the detection module.

[0077] A deviation detection module, including an edge detection unit and a deviation judgment unit, which extracts the depth feature data of the belt edge, including the edge positions and change trends on both the left and right sides of the belt. Analyze the edge features, calculate the offset of the belt, and thus determine whether the belt is deviated.

[0078] A tear detection module that adopts a deep learning network architecture based on YOLO, receives the image sequence of the belt conveyor, and is used to extract features, associate status label information, and detect the tear state of the belt conveyor.

[0079] An alarm module that triggers an audible and visual alarm or signal when the system detects a tear, reminding the operator to handle it.

[0080] The above-described embodiments are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for detecting belt deviation and tearing in underground mines based on TOF camera, characterized in that: include: Collecting a depth image of the belt conveyor running area, setting a camera coordinate system and a belt conveyor coordinate system, transforming the camera coordinate system into the belt conveyor coordinate system, calculating the horizontal offset between the belt edge and the reference point according to the transformed coordinates, and using the horizontal offset to obtain the degree of belt deviation; When the degree of belt deviation exceeds the target threshold, the depth image is input into a belt tear detection model to obtain a tear detection result. The belt tear detection model is obtained by training a YOLO network model using a training set. The training set includes: a belt tear sample image and a tear area label.

2. The method for detecting underground belt deviation and tearing based on TOF camera according to claim 1, characterized in that: Converting the camera coordinate system to the belt conveyor coordinate system includes: A target reference point is set on the belt surface as the origin of the belt conveyor coordinate system, the belt conveyor coordinate system is established, and a camera coordinate system is established with the optical center of the TOF camera as the origin; Converting a pixel coordinate system to the camera coordinate system, and converting the camera coordinate system to the belt conveyor coordinate system; Wherein, the axis direction of the camera coordinate system is consistent with the belt conveyor coordinate system.

3. The TOF camera-based underground belt deviation and tearing detection method according to claim 2, characterized in that: Converting the pixel coordinate system to the camera coordinate system comprises: Among them, f x and f y is the focal length of the camera, Z is the depth value, (c x， c y ) is the coordinate of the optical center in the pixel coordinate system, (u, v) is the pixel coordinate of the reference point in the pixel coordinate system, X c , Y c , Z c is the three-dimensional coordinate of the reference point in the camera coordinate system.

4. The method for detecting underground belt deviation and tearing based on TOF camera according to claim 1, characterized in that: Converting the camera coordinate system to the belt conveyor coordinate system includes: Among them, R is the rotation matrix and T is the translation vector.

5. The TOF camera-based underground belt deviation and tearing detection method according to claim 4, characterized in that: Converting the camera coordinate system to the belt conveyor coordinate system further includes: When the rotation matrix And the translation vector , the camera coordinate system is transformed into the belt conveyor coordinate system as follows: Where h is the distance.

6. The TOF camera-based underground belt deviation and tearing detection method according to claim 1, characterized in that: Calculating the horizontal offset between the belt edge and the reference point according to the transformed coordinates includes: <h2 style=";text-align:left;direction:ltr">Δx =|D<h2 style=";text-align:left;direction:ltr"> left <h2 style=";text-align:left;direction:ltr"> -D<h2 style=";text-align:left;direction:ltr"> ref_left <h2 style=";text-align:left;direction:ltr"> |+|D<h2 style=";text-align:left;direction:ltr"> right <h2 style=";text-align:left;direction:ltr"> -D<h2 style=";text-align:left;direction:ltr"> ref_right <h2 style=";text-align:left;direction:ltr"> | Among them, D left ,D right D is the horizontal distance from the real-time edge of the belt to the reference point. ref_left ,D ref_right It is the horizontal distance from the belt edge to the reference point in the standard model.

7. An underground belt deviation and tearing detection system based on TOF camera, characterized in that: include: An image acquisition module, used to acquire a depth image of the belt conveyor operation area; A deviation detection module is used to set a camera coordinate system and a belt conveyor coordinate system, transform the camera coordinate system into the belt conveyor coordinate system, calculate the horizontal offset between the belt edge and the reference point according to the transformed coordinates, and obtain the degree of belt deviation using the horizontal offset; The tear detection module is used to input the depth image into the belt tear detection model to obtain the tear detection result when the belt deviation degree exceeds the target threshold. The belt tear detection model is obtained by training the YOLO network model using the training set, and the training set includes: belt tear sample images and tear area labels.

Images