Mining belt conveyor belt status detection method and system based on image recognition

Abstract

The present invention relates to the technical field of coal mine belt conveyor operation monitoring, and in particular to a method and system for detecting the status of a conveyor belt of a coal mine belt conveyor based on image recognition. The method comprises: installing a video sensor on a gantry of the conveyor; first, using the video sensor to capture a video of the belt surface area of the conveyor belt in real time; and capturing the video frame by frame to obtain multiple frames of continuous images of the coal mine conveyor belt; then, the video sensor analyzes and processes the coal mine conveyor belt images; and finally, the video sensor displays the processed coal mine conveyor belt images, wherein the processed coal mine conveyor belt images include information on whether there is coal on the conveyor belt and whether the conveyor belt is operating. The present invention provides a method for detecting the status of a coal mine belt conveyor based on image recognition, which is reliable, has good real-time detection performance, is easy to maintain, is low in cost, and is convenient for practical application in coal mine scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mine belt conveyor operation monitoring, and in particular to a method and system for detecting the state of a conveyor belt of a coal mine belt conveyor based on image recognition. Background Art

[0002] Highly efficient coal mine main transportation equipment is an important component of automated and intelligent mines. Among them, mining belt conveyors are key equipment in the coal mine main transportation system. Realizing the status detection of the conveyor belt during the operation of the mining belt conveyor is of great significance for reducing manual maintenance costs and improving mining safety.

[0003] Existing belt conveyor belt status detection methods currently rely on hardware or mining conveyor belt image processing technology. Hardware-based detection methods, when applied to the harsh environment of underground coal mines, are susceptible to dust, coal slime, oil pollution, and other factors. This reduces the sensitivity of hardware sensors, resulting in high false positive and false negative rates and low detection accuracy. To reduce the false positive rate and improve detection accuracy, these methods require regular manual disassembly and calibration, resulting in high maintenance costs. Furthermore, methods based on mining conveyor belt image processing utilize complex network models, which have a large number of parameters and operate slowly, affecting the real-time performance of detection.

[0004] Although the existing conveyor belt status detection method for mining belt conveyors has played a role in conveyor belt status detection to a certain extent, due to the harsh underground production environment, the detection reliability and real-time detection cannot be guaranteed. It is urgent to develop a conveyor belt status detection method for mining belt conveyors that is reliable, has good real-time detection, is easy to maintain and has low cost. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: in order to solve the technical problems that the existing belt conveyor belt status detection method is easily affected by the environment during hardware detection, has low detection accuracy, high maintenance cost, and poor real-time performance during mining conveyor belt image processing, the present invention provides a mining belt conveyor belt status detection method based on image recognition, which has reliable detection, good real-time detection, easy maintenance and low cost, and the hardware equipment it relies on is simple, easy to deploy, reliable in operation, and the algorithm complexity is not high. It can run on mining intrinsically safe embedded equipment, which is convenient for practical application in coal mine scenarios.

[0006] The technical solution adopted by the present invention to solve its technical problem is: a method for detecting the state of a conveyor belt of a mining belt conveyor based on image recognition, wherein the state detection includes detecting whether there is coal on the conveyor belt and detecting the running state of the conveyor belt. The detection method includes:

[0007] Step S1: a video sensor collects a video of the belt surface area of the conveyor belt in real time during operation, and captures the video frame by frame to obtain a plurality of continuous frames of the mining conveyor belt image;

[0008] Step S2, the video sensor analyzes and processes the image of the mining conveyor belt, and detects whether there is coal on the conveyor belt and the running status of the conveyor belt based on the image of the mining conveyor belt;

[0009] Step S3, the video sensor displays a processed image of the mining conveyor belt according to the detection result, wherein the processed image of the mining conveyor belt includes whether there is coal on the conveyor belt and whether the conveyor belt is running;

[0010] If the detection result shows that the conveyor belt is in motion and there is coal on the conveyor belt, the process returns to step S2 to continue the detection;

[0011] If the detection result shows that the conveyor belt is in a stopped state and there is no coal on the conveyor belt, the video sensor records the image of the mining conveyor belt, sends out an alarm signal, and then returns to step S2 to continue detection.

[0012] Further, specifically, the step S2 includes performing coal material state inference analysis on the image of the mining conveyor belt through the GPU of the video sensor and performing running state detection analysis on the image of the mining conveyor belt through the CPU of the video sensor.

[0013] Further, specifically, the step S2 of analyzing the presence or absence of coal material on the mining conveyor belt image includes the following steps:

[0014] Step S211: the video sensor collects video images of whether there is coal in a portion of the belt surface area of the conveyor belt, and all the video images of whether there is coal are generated into a data set;

[0015] Step S212: labeling the data set and dividing the data set into a training set, a test set, and a validation set;

[0016] Step S213: preprocessing the data set;

[0017] Step S214: Building a ResNet18 network model based on the Pytorch deep learning framework, and converting the data set into an input vector recognized by the ResNet18 network model;

[0018] Step S215: giving the input vector and target value to the ResNet18 network model, training the ResNet18 network model, and obtaining a weight file with excellent performance;

[0019] Step S216: Use the ResNet18 network model to load the weight file to perform inference on the mine conveyor belt image, and the inference result includes that the conveyor belt is in the state with coal material or the conveyor belt is in the state without coal material.

[0020] Further, specifically, step S2 for analyzing the operating state of the mine conveyor belt image includes the following steps:

[0021] Step S221: Perform grayscale processing on the mine conveyor belt image to convert the mine conveyor belt image into a grayscale image;

[0022] Step S222: Based on the Shi-Tomasi algorithm, extract the feature points of the first grayscale image in the multi-frame continuous mine conveyor belt images to form a first set of feature points. The set of the first feature points is {(x0, y0), (x1, y1), ……(x j , y j ), where j = 0, 1…j, and use the first feature point as the initial tracking point;

[0023] Step S223: Starting from the second grayscale image to the last grayscale image, based on the feature points in the first set of feature points, use the Lucas-Kanade optical flow algorithm to perform feature point tracking on the current grayscale image to obtain a second set of feature points of the current grayscale image. The set of the second feature points is {(X0, Y0), (X1, Y1), ……(X j , Y j ), and use the second feature point as the end tracking point;

[0024] Step S224: According to the positions of the first feature point and the second feature point, calculate the movement direction of the first feature point and the second feature point, and compare the movement direction with the reference angle direction:

[0025] If the movement direction is consistent with the reference angle direction, the count value is incremented by 1;

[0026] If the movement direction is inconsistent with the reference angle direction, no count is made;

[0027] Step S225: Calculate the ratio P between the total number of the count values and the total number of all the first feature points and the second feature points;

[0028] Step S226: Compare the ratio P with the set threshold K:

[0029] If P > K, it indicates that the conveyor belt is in the operating state;

[0030] If P < K, it indicates that the conveyor belt is in the stopped state;

[0031] Step S227: taking the second feature point of the current frame grayscale image as the initial tracking point of the next frame grayscale image, and repeating steps S223 to S224 until the video sensor completes video acquisition.

[0032] Further, specifically, the step S222 includes the following steps:

[0033] S2221: The first frame grayscale image is used as the input image of the Shi-Tomasi algorithm, and the number of detected maximum feature points is set to N. The first frame grayscale image is I(x, y). First, the horizontal gradient I of I(x, y) is calculated. x and the vertical gradient I y , the calculation formula is:

[0034] I x =I*[-1 0 1];

[0035] I y =I*[-1 0 1] T ;

[0036] Next, calculate the product of the horizontal and vertical gradients using the following formula:

[0037]

[0038]

[0039] I x I y =I x *I y ;

[0040] Then, use the Gaussian function to I x I y Perform Gaussian weighting, add a window with the first frame grayscale image I(x,y) as the center to form a window image ω(x,y), and calculate the covariance matrix M corresponding to the window image ω(x,y). The calculation formula is:

[0041]

[0042]

[0043] C=∑ (x,y)∈W g(I x I y )=∑ (x,y)∈W I x I y *ω(x,y);

[0044]

[0045] The covariance matrix M is then used to calculate the corner response function R corresponding to each pixel. The calculation formula is:

[0046] R=det(M)-K(trace(M)) 2 ;

[0047] Where det(M) = λ1λ2 is the determinant of the matrix M, and trace(M) = λ1+λ2 is the trace of the matrix. λ1 and λ2 are the eigenvalues of the matrix M, and K is an empirical constant with a range of (0.04, 0.06).

[0048] Finally, the corresponding values R greater than a certain threshold t are filtered to obtain corner points, which are feature points;

[0049] Step S2222: determining the valid first feature points and the total number of the valid first feature points among the detected feature points;

[0050] If the distance between two adjacent first feature points is less than 1, the first feature points are determined to be invalid feature points and deleted;

[0051] If the distance between two adjacent first feature points is greater than or equal to 1, the first feature points are determined to be valid feature points and saved;

[0052] The step is 2223: taking all valid first feature points as initial tracking points.

[0053] Furthermore, specifically, in step 223, the first feature points and the second feature points that do not meet the requirements are screened out, and the first feature points and the second feature points that meet the requirements are saved. When the first feature point and the second feature point satisfy |X j -x j |+|Y j -y j |>2, the second feature point (X j , Y j ) meets the requirements, and saves the second feature point and the first feature point corresponding to the second feature point.

[0054] Further, specifically, the step S224 includes the following steps:

[0055] Step S2241: Calculate the arc between the first feature point and the second feature point using the following formula:

[0056] Step S2242: Convert the radian into an angle using the following calculation formula:

[0057] Step S2243: Calculating the reference angle, where the reference angle is the average value of the sum of angles formed by the set of the first feature points and the set of the second feature points;

[0058] Step S2244: Calculate the absolute value of the angle and the reference angle, and compare with the set angle threshold:

[0059] If the absolute value is less than the angle threshold, the movement direction is consistent with the reference angle direction; if the absolute value is greater than or equal to the angle threshold, the movement direction is inconsistent with the reference angle direction.

[0060] A mining belt conveyor belt status detection system based on image recognition, which adopts the mining belt conveyor belt status detection method based on image recognition as described above, includes: a conveyor, the conveyor includes a conveyor belt, and a gantry is arranged above the conveyor belt; a video sensor, the video sensor is installed in the middle of the gantry, and the inclination angle between the video sensor and the belt surface of the conveyor belt is 45° to 75°.

[0061] Furthermore, specifically, the video sensor is a hardware and software integrated device for video acquisition, video analysis and fusion computing, and a visual image processing module and an AI acceleration module are provided in the video sensor.

[0062] Further, specifically, after the video sensor is powered on, the camera of the video sensor is located above the conveyor belt, the camera is set toward the belt surface of the conveyor belt, and the camera collects the video of the belt surface area of the conveyor belt, and the width of the belt surface area accounts for 2 / 3 of the video width.

[0063] The beneficial effects of the present invention are:

[0064] (1) The present invention provides a method for detecting the state of a mining belt conveyor based on image recognition, which uses a video sensor to collect images in real time, and processes and analyzes them, with reliable detection and good real-time detection performance;

[0065] (2) The algorithm of the present invention is not complex and runs fast. The visual image processing algorithm can improve the detection accuracy and monitoring efficiency. Through the detection results, the video sensor records the image of the mining conveyor belt at this time. After the alarm signal is issued, the status of the conveyor belt can be monitored in real time and is easy to replay;

[0066] (3) The hardware equipment on which the present invention relies is simple, easy to deploy, and reliable in operation. It can run on intrinsically safe embedded equipment for mining, facilitating practical application in coal mine scenarios.

[0067] (4) The present invention can process the belt surface area in real time based on image recognition, without the need for human patrol, thus reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present invention will be further described below with reference to the accompanying drawings and examples.

[0069] Figure 1 It is a schematic structural diagram of a detection system according to an embodiment of the present invention.

[0070] Figure 2 It is a flow chart of the detection method according to an embodiment of the present invention.

[0071] Figure 3 It is a detailed flow chart of the detection method according to an embodiment of the present invention.

[0072] Figure 4 This is a detailed training flow chart of the ResNet18 network model according to an embodiment of the present invention.

[0073] Figure 5 This is a detailed inference flowchart of the ResNet18 network model according to an embodiment of the present invention.

[0074] Figure 6 Schematic diagram of initial feature points detected by the detection method according to an embodiment of the present invention.

[0075] Figure 7 The present invention is a reasonable detection method for conveying coal and is a schematic diagram of the operating status.

[0076] Figure 8 The present invention is a reasonable detection method for a conveyor belt without coal and in a stopped state.

[0077] In the figure, 1. Conveyor; 2. Conveyor belt; 3. Video sensor; 4. Gantry. DETAILED DESCRIPTION

[0078] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0079] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0080] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0081] like Figure 1 The illustrated system for detecting the state of a mining belt conveyor belt based on image recognition includes: a conveyor belt 1 and a video sensor 3. The conveyor belt 2 is provided on the conveyor belt 1, and a gantry 4 is provided above the conveyor belt 2. The video sensor 3 is inclined at an angle of 45° to 75° relative to the surface of the conveyor belt 2. The video sensor 3 is preferably positioned in the middle of the gantry 4 so that the video sensor 3 can capture images of the entire surface of the conveyor belt 2. After the video sensor 3 is powered on, the camera of the video sensor 3 is positioned above the conveyor belt 2, facing the surface of the conveyor belt 2. The camera captures a video of the conveyor belt surface area. The camera parameters of the video sensor 3 are configured, including but not limited to gain, resolution, and exposure time. The width of the belt surface area accounts for 2 / 3 of the video width. The video captured by the camera can fully cover the entire belt surface area and the rollers, preventing workers outside the conveyor belt from entering the video and interfering with the detection method and affecting the accuracy of the detection.

[0082] In this embodiment of the present invention, the video sensor 3 is an integrated hardware and software device for video acquisition, video analysis, and fusion computing. It is built into a low-power embedded system, meeting underground coal mine safety requirements and practical applications. The video sensor 3 is equipped with a visual image processing module and an AI acceleration module. Specifically, the video image processing module runs on the video sensor's CPU. It incorporates a cross-platform visual algorithm developed based on OpenCV. This visual algorithm runs as a dynamic link library within the video image processing module and is used to detect the operating status of the conveyor belt 2. The AI acceleration module is a GPU module within the video sensor. A ResNet18 network model built based on the Pytorch deep learning framework is accelerated by the GPU module for inference, and is used to detect the presence or absence of coal on the conveyor belt 2. When the conveyor belt 2 transitions from a moving state to a stopped state, or from a state with coal on the conveyor belt 2 to a state without coal on the conveyor belt 2, the video sensor 3 records 30 seconds of video images, saves them to the local disk, and issues an alarm signal. The video images facilitate real-time monitoring and playback of the conveyor belt status of the mining belt conveyor.

[0083] like Figure 2 The present invention shows a method for detecting the state of a mining belt conveyor based on image recognition. The method includes the following steps:

[0084] In step S1, the video sensor 3 collects the video of the belt surface area of the conveyor belt 2 in real time, and takes screenshots of the video frame by frame to obtain multiple frames of continuous mining conveyor belt images.

[0085] In step S2, the video sensor 3 analyzes and processes the conveyor belt image and detects the presence of coal on the conveyor belt 2 and the operating status of the conveyor belt 2 based on the conveyor belt image. Specifically, the GPU of the video sensor 3 performs coal presence inference analysis on the conveyor belt image, and the CPU of the video sensor 3 performs operating status detection and analysis on the conveyor belt image.

[0086] In step S3, the video sensor 3 displays the processed image of the mining conveyor belt according to the detection result. The processed image of the mining conveyor belt includes whether there is coal on the conveyor belt 2 and whether the conveyor belt 2 is running. If the detection result shows that the conveyor belt is in a moving state and there is coal on the conveyor belt 2, the process returns to step S2 to continue the detection. If the detection result shows that the conveyor belt is in a stopped state and there is no coal on the conveyor belt 2, the video sensor 3 records the image of the mining conveyor belt for 30 seconds, sends an alarm signal, and then returns to step S2 to continue the detection.

[0087] like Figure 3 As shown, in the embodiment of the present invention, step S2 of analyzing the presence or absence of coal material on the mining conveyor belt image includes the following steps:

[0088] Step S211: the video sensor 3 collects the video images of the presence or absence of coal on the belt surface area of the conveyor belt 2, and makes all the video images of the presence or absence of coal into a data set with a resolution of 640×480. The data set includes a data set with coal and a data set without coal.

[0089] Step S212: Label the dataset and divide it into training, test, and validation sets. Specifically, 3 / 5 of the dataset is used as the training set, 1 / 5 of the dataset is used as the test set, and the remaining 1 / 5 of the dataset is used as the validation set. Furthermore, if the dataset has a total of 642 images, after labeling the dataset, 386 images are randomly selected as the training set, 128 images are randomly selected from the remaining 256 images as the test set, and the remaining 128 images are used as the validation set.

[0090] Step S213: preprocessing the data set, including but not limited to enhancing, rotating, and scaling the images in the data set to a uniform size, by setting parameters such as saturation or exposure of the images.

[0091] Step S214: Build a ResNet18 network model based on the Pytorch deep learning framework, and convert the data set into an input vector recognized by the ResNet18 network model.

[0092] Step S215: Given the input vector and target value to the ResNet18 network model, the ResNet18 network model is trained to obtain a weight file with excellent performance. The ResNet18 network model training process is as follows: Figure 4 The specific steps are as follows:

[0093] S2151: Initialize the weights of the ResNet18 network model.

[0094] S2152: The input vector is forward propagated through the convolutional layer, downsampling layer, and fully connected layer in the ResNet18 network model to obtain the output value.

[0095] S2153: Input the output value and target value into the loss function of the ResNet18 network model to calculate the total error of the loss function. Based on the total error of the loss function, backpropagate to calculate the error and error gradient of each layer, and then update the weight of each convolution kernel.

[0096] S2154: Perform iterative training on the ResNet18 network model. If the number of training rounds reaches the preset total number of iteration rounds, end the ResNet18 network model training and save the weight file with the best performance in the total number of iteration rounds. If the number of training rounds does not reach the preset total number of iteration rounds, return to step S2151.

[0097] Step S216: Use the ResNet18 network model to load the trained weight file to infer the mine conveyor belt image, transfer the mine conveyor belt image to the ResNet18 network model, and use the trained weight file to perform inference analysis on the mine conveyor belt image. The inference results include whether the conveyor belt 2 is in a coal-filled state or a coal-free state. Figure 5 The specific steps are as follows:

[0098] S2161: Read the mining conveyor belt image in step S1, pre-process the mining conveyor belt image, and obtain image data. Specifically, each pixel point in the pre-processed image has a corresponding pixel value, and the image data is a feature vector composed of the pixel values corresponding to each pixel point in the image.

[0099] S2162: After the ResNet18 network model loads the trained weight file, the acquired image data is input into the ResNet18 network model.

[0100] S2163: Perform inference analysis on image data using the weight file of the ResNet18 network model, and output the inference results after forward propagation.

[0101] S2164: Obtain the inference results, convert the inference results into label names, and print out the label names. The inference results include 0 and 1. Set 1 to the coal state and 0 to the coal-free state. The label names are the coal state and the coal-free state. Convert and output the label names according to the inference results.

[0102] In the embodiment of the present invention, while detecting whether the conveyor belt 2 is filled with coal or not, the running status of the mining conveyor belt image is also detected, thereby improving the detection speed. Step S2 of analyzing the running status of the mining conveyor belt image includes the following steps:

[0103] Step S221: grayscale processing is performed on the mining conveyor belt image to convert the mining conveyor belt image into a grayscale image.

[0104] Step S222: Based on the Shi-Tomasi algorithm, extract the feature points of the first frame grayscale image in the multiple frames of continuous mining conveyor belt images to form a first feature point set. The first feature point set is {(x0, y0), (x1, y1), ... (x j ,y j )}, where j = 0, 1…j, and the first feature point is used as the initial tracking point.

[0105] Specifically, step S222 includes the following steps:

[0106] S2221: Use the first frame grayscale image as the input image of the Shi-Tomasi algorithm, set the number of detected maximum feature points to N, the first frame grayscale image to I(x,y), and first calculate the horizontal gradient I of I(x,y) x and the vertical gradient I y , the calculation formula is:

[0107] I x =I*[-1 0 1];

[0108] I y =I*[-1 0 1] T ;

[0109] Next, calculate the product of the horizontal and vertical gradients using the following formula:

[0110]

[0111]

[0112] I x I y =I x *I y ;

[0113] Secondly, use the Gaussian function to I x I y Gaussian weighting is performed, where σ = 2, ksize = 3, and a window image ω(x, y) is formed with the first frame grayscale image I(x, y) as the center. The covariance matrix M corresponding to the window image ω(x, y) is calculated using the following formula:

[0114]

[0115]

[0116] C=∑ (x,y)∈W g(I x I y )=∑ (x,y)∈W I x I y *ω(x,y);

[0117]

[0118] Then, use M to calculate the corner response function R corresponding to each pixel. The calculation formula is:

[0119] R=det(M)-K(trace(M)) 2 ;

[0120] Where det(M) = λ1λ2 is the determinant of the matrix M, and trace(M) = λ1+λ2 is the trace of the matrix. λ1 and λ2 are the eigenvalues of the matrix M, and K is an empirical constant with a range of (0.04, 0.06).

[0121] In the embodiment of the present invention, since the stability of the corner point of each pixel is related to the smaller eigenvalue of the matrix M, and K is an empirical constant, when K cannot be set to an optimal value, the calculation formula of the corner point response function R of each pixel is rewritten as follows:

[0122] R=min(λ1,λ2)

[0123] Finally, the response function R greater than a certain threshold t is filtered to obtain the corner point, which is the feature point.

[0124] Step S2222: Determine the valid first feature points and the total number of valid first feature points among the detected feature points; if the distance between two adjacent first feature points is less than 1, the first feature point is determined to be an invalid feature point and deleted; if the distance between two adjacent first feature points is greater than or equal to 1, the first feature point is determined to be a valid feature point and saved.

[0125] The steps are 2223: take all valid first feature points as initial tracking points, such as Figure 6 shown.

[0126] Step S223: From the second grayscale image frame to the last grayscale image frame, based on the feature points in the first feature point set, the Lucas-Kanade optical flow algorithm is used to track the feature points of the current grayscale image frame to obtain the second feature point set of the current grayscale image frame. The second feature point set is {(X0, Y0), (X1, Y1), ... (X j , Y j )}, where j = 0, 1…j, and the second feature point is used as the final tracking point. Using the Lucas-Kanade optical flow algorithm requires three assumptions: first, the brightness of the same point does not change over time; second, the position does not change suddenly over time; and finally, adjacent points in the same scene appear adjacent when projected onto the image, and their motion speeds are consistent. The technical principle of the Lucas-Kanade optical flow algorithm is: given a template T, an input I, and one or more transformations W, find a transformation W with optimal parameters that minimizes the formula: When seeking the optimal solution, assuming that the current transformation parameters Known and iteratively calculated Increment Make the updated Can make Smaller than before, The formula is modified to

[0127] In step 223, the first feature points and the second feature points that do not meet the requirements need to be screened out, and the first feature points and the second feature points that meet the requirements are saved. j -x j |+|Y j -y j When |>2, the second feature point (X j , Y j ) meets the requirements, and saves the second feature point and the first feature point corresponding to the second feature point.

[0128] Step S224: Calculate the movement direction of the first feature point and the second feature point based on the positions of the first feature point and the second feature point, and compare the movement direction with the reference angle direction. If the movement direction is consistent with the reference angle direction, the count value is increased by 1; if the movement direction is inconsistent with the reference angle direction, no count is performed.

[0129] Specifically, step S224 includes the following steps:

[0130] Step S2241: Take the first feature point as the starting point and the second feature point as the end point, and set the position of the first feature point as (x j ,y j ), the position of the second feature point (X j , Y j ), calculate the arc between the first feature point and the second feature point, the calculation formula is:

[0131] Step S2242: Convert radians into degrees using the following calculation formula:

[0132] Step S2243: Calculate a reference angle, where the reference angle is the average value of the sum of angles formed by the set of first feature points and the set of second feature points.

[0133] Step S2244: Calculate the absolute value of the angle and the reference angle, and compare them with the set angle threshold. If the absolute value is less than the angle threshold, the movement direction is consistent with the reference angle direction, and the count value is increased by 1; if the absolute value is greater than or equal to the angle threshold, the movement direction is inconsistent with the reference angle direction, and the count is not counted. The range of the angle threshold is 30° to 60°, and the angle threshold is preferably 60°. The measurement algorithm has strong robustness, low false detection rate, and high detection accuracy.

[0134] Step S225: According to the ratio P of the total number Nua of the count values to the total number Nub of all the first feature points and the second feature points, that is

[0135] Step S226: Compare P with the set threshold K. If P > K, it indicates that the conveyor belt 2 is in the running state; if P < K, it indicates that the conveyor belt 2 is in the stopped state, and on the video sensor 3.

[0136] Step S227: Take the second feature points of the current frame grayscale image as the initial tracking points of the next frame grayscale image, and repeat steps S223 - S224 until the video acquisition of the video sensor 3 is completed.

[0137] In step S3, the video sensor 3 displays the processed image of the mine conveyor belt. When the detection result is that there is coal material and it is in the running state, the second feature points in the current frame image are the end points, and the first feature points in the first frame image are the start points. The movement trajectory from the start point to the end point of the current frame image is displayed on the video sensor 3, and "beltrotation, coal" is displayed in the current frame image, as Figure 7 shown, where "belt rotation, coal" indicates that the conveyor belt 2 has coal material and is in the running state; conversely, when the detection result is that there is no coal material and it is in the stopped state, the movement trajectory from the start point to the end point of the current frame image is not displayed on the video sensor 3, but "belt stop rotation, no coal" is displayed in the current frame image, as Figure 8 shown, where "belt stop rotation, no coal" indicates that there is no coal material and it is in the stopped state.

[0138] The present invention provides a method for detecting the state of a mine belt conveyor based on image recognition. It can collect images in real time through a video sensor, process and analyze them. The detection is reliable and has good real - time performance; the algorithm complexity is not high, and the running speed is fast. The detection accuracy and monitoring efficiency can be improved through the visual image processing algorithm. Through the detection result, the video sensor records the image of the mine conveyor belt at this time. After sending an alarm signal, it can monitor the state of the conveyor belt in real time and is convenient for playback; the hardware equipment relied on is simple, easy to deploy, and works reliably. It can run on mine intrinsically safe embedded devices, which is convenient for practical applications in coal mine scenarios. In addition, based on image recognition, the belt surface area can be processed in real time, without the need for people to patrol, reducing labor costs.

[0139] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0140] Other embodiments of the present invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0141] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for detecting the state of a mining belt conveyor based on image recognition, characterized in that: The state detection includes the detection of the presence or absence of coal material on the conveyor belt (2) and the detection of the operating state of the conveyor belt (2). The detection method includes: Step S1, the video sensor (3) continuously captures the video of the belt surface area of the conveyor belt (2) during operation, and takes screenshots of the video frame by frame to obtain multiple consecutive mine conveyor belt images; Step S2, the video sensor (3) analyzes and processes the mine conveyor belt images, and detects the presence or absence of coal material on the conveyor belt (2) and the operating state of the conveyor belt (2) according to the mine conveyor belt images; Step S3, the video sensor (3) displays the processed mine conveyor belt images according to the detection results. The processed mine conveyor belt images include the presence / absence of coal material on the conveyor belt (2) and whether the conveyor belt (2) is operating or not; If the detection result is that the conveyor belt (2) is in a moving state and there is coal material on the conveyor belt (2), return to Step S2 to continue the detection; If the detection result is that the conveyor belt (2) is in a stopped state and there is no coal material on the conveyor belt (2), the video sensor (3) records the mine conveyor belt image, emits an alarm signal, and then returns to Step S2 to continue the detection; Among them, in Step S2, the CPU of the video sensor (3) performs operating state detection and analysis on the mine conveyor belt images, which specifically includes the following steps: Step S221: Perform grayscale processing on the mine conveyor belt images to convert the mine conveyor belt images into grayscale images; Step S222: Based on the Shi-Tomasi algorithm, extract the feature points of the first frame grayscale image in the multiple frames of continuous mining conveyor belt images to form a first feature point set. The first feature point set is , where j=0,1…j, the first feature point is used as the initial tracking point; Step S223: From the second frame grayscale image to the last frame grayscale image, based on the feature points in the first feature point set, the Lucas-Kanade optical flow algorithm is used to track the feature points of the current frame grayscale image to obtain the second feature point set of the current frame grayscale image. The second feature point set is , taking the second feature point as the end tracking point; Step S224: According to the positions of the first feature point and the second feature point, calculate the movement directions of the first feature point and the second feature point, and compare the movement directions with the reference angle direction: If the movement direction is consistent with the reference angle direction, the count value is incremented by 1; If the movement direction is inconsistent with the reference angle direction, no counting is performed; Step S225: Calculate the ratio P between the total number of the count values and the total number of all the first feature points and the second feature points; Step S226: Compare the ratio P with the set threshold K: If P > K, it indicates that the conveyor belt (2) is in an operating state; If P < K, it indicates that the conveyor belt (2) is in a stopped state; Step S227: Use the second feature point of the current frame grayscale image as the initial tracking point of the next frame grayscale image, and repeat Steps S223 - S224 until the video acquisition of the video sensor (3) is completed.

2. The method for detecting the state of a mining belt conveyor based on image recognition according to claim 1, wherein: Step S2 also includes performing inference and analysis on the presence or absence of coal material state of the mine conveyor belt images through the GPU of the video sensor (3).

3. The method for detecting the state of a mining belt conveyor based on image recognition according to claim 2, wherein: Step S2 for analyzing the presence or absence of coal material state of the mine conveyor belt images includes the following steps: Step S211: The video sensor (3) captures the video images of the presence or absence of coal material in the belt surface area of a part of the conveyor belt (2), and makes all the video images of the presence or absence of coal material into a data set; Step S212: Label the data set and divide the data set into a training set, a test set, and a validation set; Step S213: Perform preprocessing on the data set; Step S214: Building a ResNet18 network model based on the Pytorch deep learning framework, and converting the data set into an input vector recognized by the ResNet18 network model; Step S215: giving the input vector and target value to the ResNet18 network model, training the ResNet18 network model, and obtaining a weight file with excellent performance; Step S216: using the ResNet18 network model to load the weight file and perform inference on the mining conveyor belt image, the inference result includes whether the conveyor belt (2) is in a coal-filled state or the conveyor belt (2) is in a coal-free state.

4. The method for detecting the state of a mining belt conveyor based on image recognition according to claim 1, wherein: The step S222 includes the following steps: S2221: Use the first frame grayscale image as the input image of the Shi-Tomasi algorithm, set the number of detected maximum feature points to N, and the first frame grayscale image is , first, calculate Gradient in the horizontal direction and the vertical gradient , the calculation formula is: ； ； Next, calculate the product of the horizontal and vertical gradients using the following formula: ； ； ； Then, use the Gaussian function to , , Gaussian weighting is performed to take the first frame grayscale image Form a window image for center weighting , calculate the window image The corresponding covariance matrix M is calculated as follows: ； ； ； ； Then, the covariance matrix M is used to calculate the corner response function R corresponding to each pixel. The calculation formula is: ； in, is a matrix The determinant of is the trace of the matrix, and is a matrix The characteristic value of , K is an empirical constant, and the value range of K is (0.04, 0.06); Finally, the corresponding values R greater than a certain threshold t are filtered to obtain corner points, which are feature points; Step S2222: determining the valid first feature points and the total number of the valid first feature points among the detected feature points; If the distance between two adjacent first feature points is less than 1, the first feature points are determined to be invalid feature points and deleted; If the distance between two adjacent first feature points is greater than or equal to 1, the first feature points are determined to be valid feature points and saved; The step is 2223: taking all valid first feature points as initial tracking points.

5. The method for detecting the state of a mining belt conveyor based on image recognition according to claim 1, wherein: In step 223, the first feature points and the second feature points that do not meet the requirements are screened out, and the first feature points and the second feature points that meet the requirements are saved. When the second feature point If the requirement is met, the second feature point and the first feature point corresponding to the second feature point are saved.

6. The method for detecting the state of a mining belt conveyor based on image recognition according to claim 1, wherein: The step S224 includes the following steps: Step S2241: Calculate the arc between the first feature point and the second feature point using the following formula: ; Step S2242: Convert the radian into an angle using the following calculation formula: ; Step S2243: Calculating the reference angle, where the reference angle is the average value of the sum of angles formed by the set of the first feature points and the set of the second feature points; Step S2244: Calculate the absolute value of the angle and the reference angle, and compare with the set angle threshold: If the absolute value is less than the angle threshold, the movement direction is consistent with the reference angle direction; if the absolute value is greater than or equal to the angle threshold, the movement direction is inconsistent with the reference angle direction.

7. A detection system used in the method for detecting the state of a mining belt conveyor based on image recognition as claimed in any one of claims 1 to 6, characterized in that: The detection system comprises: A conveyor (1), the conveyor (1) comprising a conveyor belt (2), and a gantry (4) is provided above the conveyor belt (2); A video sensor (3), wherein the video sensor (3) is mounted on the gantry (4), and an inclination angle between the video sensor (3) and the conveyor belt (2) is 45° to 75°.

8. The detection system according to claim 7, wherein: The video sensor (3) is a hardware and software integrated device for video acquisition, video analysis and fusion calculation, and a visual image processing module and an AI acceleration module are provided in the video sensor (3).

9. The detection system according to claim 8, wherein: After the video sensor (3) is powered on, the camera of the video sensor (3) is located above the conveyor belt (2), and the camera is set toward the belt surface of the conveyor belt (2). The camera collects the video of the belt surface area of the conveyor belt (2), and the width of the belt surface area accounts for 2 / 3 of the width of the video.

Images