A coal mine transportation belt deviation detection method based on optical flow detection and end-to-end

Abstract

The application discloses a coal mine transportation belt deviation detection method based on optical flow detection and end-to-end, and belongs to the technical field of safety monitoring. In view of the problems that the prior art cannot realize real-time monitoring on the running condition of the belt, the inspection efficiency is low, and there is a certain lag, a data set is made by collecting a coal mine transportation video, a carrier roller is labeled with the aid of image labeling software, and target frame position information of the carrier roller is calculated by using a carrier roller detection model, a slope of a belt transportation direction is calculated by using an optical flow detection algorithm, a straight line is determined according to the slope and the target frame position information of the carrier roller, the target frame of the carrier roller is divided into left and right regions, an edge graph obtained through edge detection and the target frame of the carrier roller with the divided position are used to generate a mask mask, an interested region is extracted, straight line information in the edge graph is processed in combination with straight line detection, position information of the belt and the carrier roller is determined, and whether the belt deviates is judged.

Description

Technical Field

[0001] This invention belongs to the field of safety monitoring technology, specifically relating to a method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end connection. Background Technology

[0002] Belts are key components of belt conveyors, responsible for traction and transport, and are widely used in coal, mining, metallurgy, and chemical industries. During operation, belt misalignment frequently occurs due to various reasons, including belt scratches from coal or scrap steel, belt aging, improper installation and adjustment of the conveyor, and uneven load caused by incorrect material drop positions at transfer points. These issues jeopardize safe operation. Therefore, it is essential to monitor and track the belt conveyor process to detect and address any misalignment promptly, thereby reducing safety risks.

[0003] Current technology mainly relies on manual periodic inspections. Inspectors periodically check the condition of the belt, and when belt misalignment is found, maintenance personnel stop the machine and gradually adjust the belt position by adjusting the tail shaft. This inspection method cannot achieve real-time monitoring of belt operation, has low inspection efficiency, and has a certain lag, making it difficult to meet the needs of modern production. Summary of the Invention

[0004] To address the problems of low efficiency and certain lag in existing belt conveyor detection technologies, this invention provides a method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end integration. This method aims to achieve real-time monitoring of belt misalignment, save manual inspection time, and improve detection efficiency and accuracy.

[0005] To achieve the above objectives, the present invention employs the following technical solutions:

[0006] A method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end measurement, the method comprising the following steps:

[0007] Step 1: Collect videos of belt conveyor transport in different coal mine scenarios to create an original coal mine belt conveyor image dataset, and then annotate it using image annotation software;

[0008] Step 2: Read the real-time video data from the camera to obtain the frame images of the video data, and use the optical flow detection algorithm to obtain the slope of the belt running direction;

[0009] Step 3: Input the original coal mine conveyor belt image dataset into the idler detection model for calculation to obtain the target box position information of the idlers on both sides of the conveyor belt;

[0010] Step 4: Using the slope obtained in Step 2 and the position information of the idler target frame obtained in Step 3, determine a straight line and divide the idler target frame into left and right regions;

[0011] Step 5: Obtain the edge map of the image through edge detection, and generate a mask based on the position division of the target box of the roller in Step 4 to extract the region of interest (ROI).

[0012] Step 6: Combine the straight line detection algorithm to process the edge map in Step 5 to obtain straight line information, mark the straight lines on the belt edge, and determine whether the belt is off-center during transportation by using the position information of the belt and idler rollers.

[0013] Furthermore, the specific process of collecting belt conveyor videos from different coal mine scenarios in step 1, creating an original coal mine belt conveyor image dataset, and annotating it using image annotation software is as follows:

[0014] Step 101: Collect monitoring videos from multiple scenarios under different coal mine environments, and split the belt conveyor video frame by frame to obtain the original coal mine belt image dataset.

[0015] Step 102: Use image annotation software to annotate the original coal mine conveyor belt image dataset. The annotation object is the idler roller next to the conveyor belt. Divide the annotated coal mine conveyor belt image dataset into a training set and a test set according to the ratio of 9:1. The total number of images in the coal mine conveyor belt image dataset is no less than 1000.

[0016] Furthermore, the specific process of step 2, which involves reading real-time video data from the camera and using an optical flow detection algorithm to obtain the slope of the belt's running direction, is as follows:

[0017] Step 201: Read the real-time video data from the camera;

[0018] Step 202: Obtain the first 10 frames of video data, and use the optical flow detection algorithm to calculate the instantaneous displacement of each position in the first 10 frames of images, thereby obtaining the slope S of the belt running direction;

[0019] Optical flow detection algorithm: Calculates the instantaneous displacement of each position in the image based on the pixel correlation between video frames. The input consists of two temporally adjacent frames I... t-1 and I t Optical flow calculations shift each pixel in the frame, ensuring that the position of each pixel after shifting is the same as I in the next time step. t Consistent.

[0020] Furthermore, the specific process of step 3, which involves inputting the original coal mine conveyor belt image dataset into the idler detection model for calculation to obtain the target bounding box position information of the idlers on both sides of the conveyor belt, is as follows:

[0021] Step 301: Input the coal mine conveyor belt image dataset trained in Step 102 into an end-to-end deep learning object detection model for training (such as the YOLO detection model). The learning rate is set to 0.001 and the batch size is set to 8 for the first 50 training rounds, and the learning rate is set to 0.0001 and the batch size is set to 4 for the next 50 training rounds. The optimal model weights are obtained through 100 training rounds to obtain the idler roller detection model.

[0022] Step 302: Input the video image obtained in step 201 into the trained idler roller detection model for detection;

[0023] Step 303: Obtain the target bounding box position information calculated by the model, including the top left horizontal coordinate offset (top), the left vertical coordinate offset (left), the bottom right horizontal coordinate offset (bottom), and the right vertical coordinate offset (right).

[0024] Furthermore, the specific process of determining a straight line and dividing the target frame of the idler roller into left and right regions using the slope obtained in step 2 and the position information of the idler roller target frame obtained in step 3 in step 4 is as follows:

[0025] Step 401: Based on the target box position information output by the roller detection model in step 303, convert the target box position information into vertex coordinate information [(top,left),(bottom,right)], and calculate the center point coordinate information (x,y) of the target box. The calculation process is as follows: x=(top+bottom) / 2, y=(left+right) / 2;

[0026] Step 402: Calculate the slope of the line connecting the center points of each pair of target frames based on the coordinate information of the center points of the target frames, and subtract it from the slope S of the belt running direction calculated by the optical flow detection algorithm in step 202. Take the largest difference and calculate the coordinates (m,n) of the center point on the line connecting them. The calculation process is as follows: m = (x1 + x2) / 2, n = (y1 + y2) / 2; (x1, y1) and (x2, y2) represent the coordinates of the two idler target frame center points with the largest difference from the slope S of the belt running direction calculated by the optical flow detection algorithm.

[0027] Step 403: Based on the coordinates (m,n) obtained in Step 402 and the slope S of the belt running direction calculated by the optical flow detection algorithm in Step 202, determine a straight line L; divide the target frames of the idlers on both sides of the belt into left and right regions. The specific division method is as follows: substitute the coordinates x of the center point of each idler target frame into the equation of the straight line L to obtain... according to The relationship between y and the left and right regions is used to divide the region.

[0028] Step 404: Based on the target frames of the idler rollers after dividing them into left and right sides, take the coordinates of the center points of the two target frames on the left, z1(x1,y1) and z2(x2,y2). Based on these coordinate points, take the derived coordinate points z11 and z22. The coordinates of the derived coordinate points z11 and z22 are obtained by shifting z1 and z2 10 pixels to the right on the horizontal axis. The positions of these four coordinate points z1, z2, z11 and z22 are used to determine the region of interest of the left belt. The region of interest of the right belt is determined by the coordinates of the center point of the target frame of the right idler roller, z3(x3,y3) and z4(x4,y4) and its derived coordinate points (shifted 10 pixels to the left on the horizontal axis).

[0029] Furthermore, step 5 obtains the edge map of the image through edge detection, and generates a mask based on the position division of the target box of the idler roller in step 4. The specific process of extracting the region of interest (ROI) is as follows:

[0030] Step 501: Use a color conversion function to convert the color images in the coal mine conveyor belt image dataset trained in step 102 into grayscale images, such as the cv.cvtColor function in the OpenCV library, which implements color space conversion.

[0031] Step 502: Use a Gaussian filter to perform Gaussian filtering on the grayscale image obtained in step 501 to remove high-frequency noise, such as the cv.GaussianBlur function in the OpenCV library.

[0032] Gaussian filtering is a process of weighted averaging across the entire image. The value of each pixel is obtained by weighted averaging of its own value and the values ​​of other pixels in its neighborhood. The principle of Gaussian filtering is to perform a weighted average based on the gray values ​​of the pixel to be filtered and its neighboring pixels, according to the parameters generated by the Gaussian formula.

[0033] Step 503: Use an edge detection algorithm, such as the Canny edge detection algorithm, to extract edge lines in the image after Gaussian filtering; generate a mask matrix with the same size and dimensions as the original image and initialize it to all zeros, i.e., all black.

[0034] Step 504: Construct the region of interest on the mask matrix by referring to the original image;

[0035] Step 505: Use a polygon drawing function, such as the cv.fillpoly() function in OpenCV, to fill the defined polygon outline with a fill value of 1, i.e., all white; determine the coordinates of four vertices in each of the two regions, i.e., the two regions of interest on the left and right sides of the belt obtained in step 404, and the graphic is a trapezoid.

[0036] Step 506: Using image bitwise AND operation functions, such as the cv.bitwise_and() function in OpenCV, perform a bitwise AND operation with the edge-detected image to retain the white pixel values ​​in the corresponding region of interest in the original image, remove the black pixel values, generate a mask, and extract the region of interest (ROI).

[0037] A mask is a tool used to occlude (fully or partially) an image to be processed, using a selected image, graphic, or object. This controls the area or process of image processing. The specific image or object used for occlusion is called a mask or template. A mask is a binary image composed of 0s and 1s. When a mask is applied, regions with 1s are processed, while regions with 0s are not included in the calculation.

[0038] Furthermore, step 6 combines a straight line detection algorithm to process the edge map in step 5 to obtain straight line information, marks the straight lines on the belt edge, and determines whether the belt is misaligned during transportation based on the position information of the belt and idlers. The specific process is as follows:

[0039] Step 601: Combine line detection algorithms, such as the Hough transform function cv.HoughLinesP encapsulated in OpenCV, to process the line information in the image obtained in step 5.

[0040] Step 602: Further preprocess the line information obtained in step 601. The processing method is as follows: First, use the line L in step 403 to divide the lines in the image into left and right groups, and classify them into left and right lists respectively; Second, calculate the slope of each line information obtained in step 601, and calculate the slope lslope of the center points z1 and z2 of the left idler roller in step 404, and calculate the slope rslope of the center points z3 and z4 of the right idler roller. Use lslope and rslope to filter the lines in the left and right lists within the specified error range respectively.

[0041] Step 603: Perform least squares fitting on the lines in the left and right lists after filtering in step 602 to obtain the slope and intercept. According to the linear formula y = k*x + b, for the slope and intercept obtained by fitting the lines in the left list, take the midpoint and maximum value of the vertical axis of the image as the y coordinate, calculate the corresponding x coordinate, and connect the two coordinates to obtain the straight line of the left belt edge. For the slope and intercept obtained by fitting the lines in the right list, take the midpoint and maximum value of the horizontal axis of the image as the x coordinate, calculate the corresponding y coordinate, and connect the two coordinates to obtain the straight line of the right belt edge.

[0042] Step 604: Based on the position information of the belt and idler rollers, determine whether the belt has deviated during transportation.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] This invention enables real-time monitoring of the belt's operating status during operation, facilitating timely detection and handling of abnormalities by staff.

[0045] The method provided by this invention is easy to debug and promote, and has universality. It can be applied to belt conveyor scenarios under various conditions. It has fast detection speed and high real-time performance, which can help operators grasp the on-site situation more quickly and prevent accidents that may be caused by belt misalignment in a timely manner, thereby ensuring operational safety and reducing the waste of manpower and time. Attached Figure Description

[0046] Figure 1 This is a schematic flowchart of an overall method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end measurement.

[0047] Figure 2 This is a schematic diagram of Example 1 for belt misalignment detection;

[0048] Figure 3 This is a schematic diagram of Example 2 for belt misalignment detection;

[0049] Figure 4 This is a schematic diagram of Example 3 for belt misalignment detection. Detailed Implementation

[0050] Example 1

[0051] Step 1: Collect videos of belt conveyor transport in different coal mine scenarios to create an original coal mine belt conveyor image dataset, and then annotate it using image annotation software;

[0052] Step 101: Collect belt conveyor videos from different coal mine scenarios, and split the belt conveyor videos frame by frame to obtain the original coal mine belt image dataset.

[0053] Step 102: Use the image annotation software labellmg to annotate the original coal mine belt image dataset. The annotation object is the idler roller next to the belt (an important component of the belt conveyor).

[0054] The labeled coal mine conveyor belt image dataset was divided into a training set and a test set according to a ratio of 9:1.

[0055] Step 2: Read the real-time video data from the camera to obtain the frame images of the video data, and use the optical flow detection algorithm to obtain the slope of the belt running direction;

[0056] Step 201: Read real-time video data from underground coal mine cameras;

[0057] Step 202: Obtain the first 10 frames of video data, and use the optical flow detection algorithm to calculate the instantaneous displacement of each position in the first 10 frames, thereby obtaining the slope S = 0.4940 of the belt running direction.

[0058] Step 3: Input the original coal mine conveyor belt image dataset into the idler detection model for calculation to obtain the target box position information of the idlers on both sides of the conveyor belt;

[0059] Step 301: Input the coal mine conveyor belt image dataset trained in Step 102 into the end-to-end deep learning object detection model for training. The learning rate is set to 0.001 and the batch size is set to 8 for the first 50 training rounds, and the learning rate is set to 0.0001 and the batch size is set to 4 for the next 50 training rounds. The optimal model weights are obtained through 100 training rounds to obtain the idler roller detection model.

[0060] Step 302: Input the video image obtained in step 201 into the trained idler roller detection model for detection;

[0061] Step 303: Obtain the target bounding box position information calculated by the model, including the top left horizontal coordinate offset (top), the left vertical coordinate offset (left), the bottom right horizontal coordinate offset (bottom), and the right vertical coordinate offset (right).

[0062] Step 4 uses the slope obtained in Step 2 and the position information of the idler target frame obtained in Step 3 to determine a straight line and divide the idler target frame into left and right regions.

[0063] Step 401: Based on the target box position information output by the roller detection model in step 303, convert the target box position information into vertex coordinate information [(top,left),(bottom,right)], and calculate the center point coordinate information (x,y) of the target box. The calculation process is as follows: x=(top+bottom) / 2, y=(left+right) / 2.

[0064] Step 402: Calculate the slope of the line connecting any two target box center points based on the target box center point coordinate information, and subtract it from the slope S of the belt running direction calculated by the optical flow detection algorithm in step 202. Take the largest difference. In this implementation, the largest difference is 0.7794. Calculate the center point coordinates (m,n) on the line connecting them. The calculation process is as follows: m = (x1 + x2) / 2, n = (y1 + y2) / 2; (m,n) = (821.0, 481.5).

[0065] Step 403: Based on the coordinates (m, n) obtained in step 402 and the slope S of the belt running direction calculated by the optical flow detection algorithm in step 202, determine a straight line L; divide the target frames of the idlers on both sides of the belt into left and right regions. Specifically, the division method is as follows: substitute the coordinates x of the center point of each idler target frame into the equation of the straight line L to obtain... according to The slope difference is used to divide the area into left and right regions. If the slope difference is less than 0.30, the selected target boxes are considered to be on one side, and the center point coordinates (m, n) use the set initial values; when the slope difference is greater than 0.30, the center point coordinates are updated.

[0066] Step 404: Based on the target bounding boxes of the idlers on the left and right sides, take the coordinates of the center points of two target bounding boxes on each side. The center points of the left idler are z1: (523.5, 543.5), z2: (1098.5, 984.5), and the center points of the right idler are z3: (1118.5, 419.5), z4: (1862.5, 675.5). Based on the center points z1, z2 of the left idler and z3, z4 of the right idler, two derived coordinate points are selected by translating them 10 pixels towards the center of the belt. In this way, the regions of interest on the left and right sides are delineated respectively.

[0067] Step 5: Obtain the edge map of the image through edge detection, and generate a mask based on the position division of the target box of the roller in Step 4 to extract the region of interest (ROI).

[0068] Step 501: Use a color conversion function to convert the color images in the coal mine conveyor belt image dataset trained in Step 102 into grayscale images;

[0069] Step 502: Use a Gaussian filter to perform Gaussian filtering on the grayscale image obtained in step 501 to remove high-frequency noise;

[0070] Step 503: Use an edge detection algorithm to extract edge lines from the image after Gaussian filtering; generate a mask matrix with the same size and dimensions as the original image and initialize it to all zeros, i.e., all black.

[0071] Step 504: Construct the region of interest on the mask matrix by referring to the original image;

[0072] Step 505: Use the polygon drawing function to fill the defined polygon outline with a fill value of 1, i.e., all white; determine the coordinates of four vertices in each of the two regions, i.e., the two regions of interest on the left and right sides of the belt obtained in step 404, and the shape is a trapezoid.

[0073] Step 506: Using the image bitwise AND operation function, perform a bitwise AND operation with the edge-detected image to retain the white pixel values ​​in the corresponding region of interest in the original image, remove the black pixel values, generate a mask, and extract the region of interest (ROI).

[0074] Step 6: Combine the straight line detection algorithm to process the edge map in Step 5 to obtain straight line information, mark the straight line of the belt edge, and determine whether the belt is running off-center during transportation by using the position information of the belt and idler rollers.

[0075] Step 601: Combine the line detection algorithm to process the line information in the image edge map in step 5;

[0076] Step 602: Further preprocessing is performed on the line information obtained in step 601. The processing method is as follows: First, the lines in the image are divided into left and right groups by using the line L in step 403, and assigned to the left and right lists respectively; Second, the slope of each line information obtained in step 601 is calculated, and the slope lslope = 0.767 is calculated for the center points z1 and z2 of the left idler roller in step 404, and the slope rslope = 0.344 is calculated for the center points z3 and z4 of the right idler roller. The lines in the left and right lists are then filtered within the specified error range using lslope and rslope respectively. The error value was selected as ±0.2 after experimentation.

[0077] Step 603: Perform least squares fitting on the lines in the left and right lists after filtering in step 602 to obtain the slope and intercept. According to the linear formula y = k*x + b, for the slope and intercept obtained by fitting the lines in the left list, take the midpoint and maximum value of the vertical axis of the image as the y coordinate, calculate the corresponding x coordinate, and connect the two coordinates to obtain the straight line of the left belt edge. For the slope and intercept obtained by fitting the lines in the right list, take the midpoint and maximum value of the horizontal axis of the image as the x coordinate, calculate the corresponding y coordinate, and connect the two coordinates to obtain the straight line of the right belt edge.

[0078] Step 604: Determine whether the belt is off-center during transportation by using the position information of the belt and idler rollers.

[0079] Example 2

[0080] The creation of the coal mine underground conveyor belt dataset involves collecting conveyor belt transportation videos from different coal mine scenarios, splitting the conveyor belt transportation videos frame by frame, and obtaining the original coal mine conveyor belt image data.

[0081] The image annotation software labellmg was used to annotate the coal mine conveyor belt image dataset, with the annotation object being the idler roller, an important component of the conveyor belt device.

[0082] Dataset partitioning: For the idler roller detection model, the dataset is divided into a training set and a test set according to a ratio of 9:1.

[0083] Training model: Training the idler roller detection model.

[0084] The real-time video of the conveyor belt in the coal mine is read, the first 10 frames of the video are obtained, and the instantaneous displacement of each position in the first 10 frames is calculated using an optical flow detection algorithm, so as to obtain the slope S = 0.5133 of the belt running direction.

[0085] The real-time belt conveyor video is input into the idler detection model for calculation to obtain the target bounding box position information of the idlers on both sides of the belt, including the top left horizontal coordinate offset (top), left vertical coordinate offset (left), bottom right horizontal coordinate offset (bottom), and right vertical coordinate offset (right). The target bounding box position information is converted into vertex coordinate information [(top, left), (bottom, right)], and its center point coordinate information (x, y) is calculated as follows: x = (top + bottom) / 2, y = (left + right) / 2.

[0086] Calculate the slope of the line connecting the center points of each pair of target frames, and subtract it from the slope S of the belt running direction. Take the largest difference; in this embodiment, the largest difference is 0.7860. Calculate the coordinates of the center point on the line connecting these points: (m, n) = (832.25, 619.0). Based on the coordinates (m, n) and the slope S of the belt running direction, determine a straight line L to divide the target frames of the idler rollers on both sides of the belt into left and right sections.

[0087] If the slope difference is less than 0.30, the selected target boxes are considered to be on one side, and the center point coordinates (m,n) use the set initial values; when the slope difference is greater than 0.30, the center point coordinates will be updated.

[0088] Based on the target bounding boxes of the idler rollers after dividing them into left and right sides, the coordinates of the center points of two target bounding boxes on each side are taken. The center points of the left idler roller are z1 and z2: (28.0, 305.5) and (1142.0, 1021.5), and the center points of the right idler roller are z3 and z4: (522.5, 216.5) and (1786.5, 632.0). Based on this, two auxiliary coordinate points are selected by translating them 10 pixels towards the center of the belt, thereby delineating the regions of interest on the left and right sides respectively.

[0089] Edge lines in the image are obtained through edge detection. Based on the position division of the target box of the roller, a mask is generated to extract the region of interest (ROI).

[0090] Combining a straight line detection algorithm, the straight line information in the image edge map is processed. The slope lslope = 0.643 is calculated using the center points z1 and z2 of the left idler roller, and the slope rslope = 0.329 is calculated using the center points z3 and z4 of the right idler roller. The straight lines in the left and right lists are filtered within a certain error range using lslope and rslope respectively. The error value was selected as ±0.2 after experimentation. The straight lines on the belt edge are marked. Based on the position information of the belt and idler rollers, it is determined whether the belt is deviating during transportation.

[0091] Example 3

[0092] The creation of the coal mine underground conveyor belt dataset involves collecting conveyor belt transportation videos from different coal mine scenarios, splitting the conveyor belt transportation videos frame by frame, and obtaining the original coal mine conveyor belt image data.

[0093] The image annotation software labellmg was used to annotate the coal mine conveyor belt image dataset, with the annotation object being the idler roller, an important component of the conveyor belt device.

[0094] Dataset partitioning: For the idler roller detection model, the dataset is divided into a training set and a test set according to a ratio of 9:1.

[0095] Training model: Training the idler roller detection model.

[0096] The real-time video of the conveyor belt in the coal mine is read, the first 10 frames of the video are obtained, and the instantaneous displacement of each position in the first 10 frames is calculated using an optical flow detection algorithm, so as to obtain the slope of the belt running direction S = 0.5427.

[0097] The real-time belt conveyor video is input into the idler detection model for calculation to obtain the target bounding box position information of the idlers on both sides of the belt, including the top left horizontal coordinate offset (top), left vertical coordinate offset (left), bottom right horizontal coordinate offset (bottom), and right vertical coordinate offset (right). The target bounding box position information is converted into vertex coordinate information [(top, left), (bottom, right)], and its center point coordinate information (x, y) is calculated as follows: x = (top + bottom) / 2, y = (left + right) / 2.

[0098] Calculate the slope of the line connecting the center points of each pair of target frames, and subtract it from the slope S of the belt running direction. Take the largest difference; in this embodiment, the largest difference is 0.8388. Calculate the coordinates of the center point on the line connecting these points: (m, n) = (750.0, 508.25). Based on the coordinates (m, n) and the slope S of the belt running direction, determine a straight line L to divide the target frames of the idler rollers on both sides of the belt into left and right sections.

[0099] If the slope difference is less than 0.30, the selected target boxes are considered to be on one side, and the center point coordinates (m,n) use the set initial values; when the slope difference is greater than 0.30, the center point coordinates will be updated.

[0100] Based on the target bounding boxes of the idler rollers after dividing them into left and right sides, the coordinates of the center points of two target bounding boxes on each side are taken. The center points of the left idler roller are z1 and z2: (374.5, 338.0) and (568.5, 562.0), and the center points of the right idler roller are z3 and z4: (661.5, 280.5) and (931.5, 454.5). Based on this, two auxiliary coordinate points are selected by translating them 10 pixels towards the center of the belt, thereby delineating the regions of interest on the left and right sides respectively.

[0101] Edge lines in the image are obtained through edge detection. Based on the position division of the target box of the roller, a mask is generated to extract the region of interest (ROI).

[0102] By combining a straight line detection algorithm, the straight line information in the image edge map is processed. Using the center points z1 and z2 of the left idler roller, its slope lslope = 1.154 is calculated, and the center points z3 and z4 of the right idler roller, its slope rslope = 0.6444 is calculated. The straight lines in the left and right lists are filtered within a certain error range using lslope and rslope respectively. The error value was selected as ±0.2 after experimentation. The straight lines of the belt edge are marked. Based on the position information of the belt and idler rollers, it is determined whether the belt is deviating during transportation.

[0103] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end measurement, characterized in that, The method includes the following steps: Step 1: Collect videos of belt conveyor transport in different coal mine scenarios to create an original coal mine belt conveyor image dataset, and then annotate it using image annotation software; Step 2: Read the real-time video data from the camera to obtain the frame images of the video data, and use the optical flow detection algorithm to obtain the slope of the belt running direction; Step 3: Input the original coal mine conveyor belt image dataset into the idler detection model for calculation to obtain the target box position information of the idlers on both sides of the conveyor belt; Step 4: Using the slope obtained in Step 2 and the position information of the idler target frame obtained in Step 3, determine a straight line and divide the idler target frame into left and right regions; Step 401: Based on the target frame position information of the idlers on both sides of the belt obtained in Step 3, the target frame position information includes the horizontal coordinate offset top and vertical coordinate offset left of the upper left corner of the target frame, the horizontal coordinate offset bottom and vertical coordinate offset right of the lower right corner, convert the target frame position information into vertex coordinate information [(top, left), (bottom, right)], and calculate the center point coordinate information (x, y) of the target frame. The calculation process is as follows: x=(top+bottom) / 2, y=(left+right) / 2; Step 402: Calculate the slope of the line connecting any two target frame center points based on the target frame center point coordinate information, and subtract it from the slope S of the belt running direction calculated by the optical flow detection algorithm in step 202. Take the largest difference and calculate the center point coordinates (m, n) on the line connecting them. The calculation process is as follows: m=(x1+x2) / 2, n=(y1+y2) / 2; (x1, y1), (x2, y2) represent the center point coordinates of the two idler roller target frames with the largest difference from the slope S of the belt running direction calculated by the optical flow detection algorithm. Step 403: Based on the coordinates (m, n) obtained in Step 402 and the slope S of the belt running direction calculated by the optical flow detection algorithm in Step 202, determine a straight line L; divide the target frames of the idlers on both sides of the belt into left and right regions. The specific division method is as follows: substitute the coordinates x of the center point of each idler target frame into the equation of the straight line L to obtain... ,according to The relationship between y and the left and right regions is used to divide the region. Step 404: Based on the target frames of the idler rollers after dividing them into left and right sides, take the coordinates of the center points of the two target frames on the left, z1(x1,y1) and z2(x2,y2). Based on these coordinate points, take the derived coordinate points z11 and z22, which are obtained by shifting z1 and z2 10 pixels to the right on the horizontal axis. The positions of these four coordinate points z1, z2, z11, and z22 are used to determine the region of interest for the left belt. The region of interest for the right belt is determined by the center point coordinates of the right idler roller target frame z3(x3,y3) and z4(x4,y4) and their derived coordinate points. Step 5: Obtain the edge map of the image through edge detection, and generate a mask based on the position division of the target box of the roller in Step 4 to extract the region of interest (ROI). Step 6: Combine the straight line detection algorithm to process the edge map in Step 5 to obtain straight line information, mark the straight lines on the belt edge, and determine whether the belt is off-center during transportation by using the position information of the belt and idler rollers.

2. The method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end connection as described in claim 1, characterized in that: The specific process of collecting belt conveyor videos from different coal mine scenarios in step 1, creating an original coal mine belt conveyor image dataset, and then annotating it using image annotation software is as follows: Step 101: Collect monitoring videos from multiple scenarios under different coal mine environments, and split the belt conveyor video frame by frame to obtain the original coal mine belt image dataset. Step 102: Use image annotation software to annotate the original coal mine conveyor belt image dataset. The annotation object is the idler roller next to the conveyor belt. Divide the annotated coal mine conveyor belt image dataset into a training set and a test set according to the ratio of 9:

1.

3. The method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end connection as described in claim 2, characterized in that: The specific process of step 2, which involves reading real-time video data from the camera and using an optical flow detection algorithm to obtain the slope of the belt's running direction, is as follows: Step 201: Read the real-time video data from the camera; Step 202: Obtain the first 10 frames of video data, and use the optical flow detection algorithm to calculate the instantaneous displacement of each position in the first 10 frames, thereby obtaining the slope S of the belt running direction.

4. The method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end connection as described in claim 3, characterized in that: The specific process of step 3, which involves inputting the original coal mine conveyor belt image dataset into the idler detection model for calculation to obtain the target bounding box position information of the idlers on both sides of the conveyor belt, is as follows: Step 301: Input the coal mine conveyor belt image dataset trained in Step 102 into the end-to-end deep learning object detection model for training. The learning rate is set to 0.001 and the batch size is set to 8 for the first 50 training rounds, and the learning rate is set to 0.0001 and the batch size is set to 4 for the next 50 training rounds. The optimal model weights are obtained through 100 training rounds to obtain the idler roller detection model. Step 302: Input the video image obtained in step 201 into the trained idler roller detection model for detection; Step 303: Obtain the target bounding box position information calculated by the model, including the top left horizontal coordinate offset (top), the left vertical coordinate offset (left), the bottom right horizontal coordinate offset (bottom), and the right vertical coordinate offset (right).

5. The method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end connection as described in claim 4, characterized in that: Step 5 obtains the edge map of the image through edge detection, and generates a mask based on the position division of the target box of the roller in step 4. The specific process of extracting the region of interest (ROI) is as follows: Step 501: Use a color conversion function to convert the color images in the coal mine conveyor belt image dataset trained in Step 102 into grayscale images; Step 502: Use a Gaussian filter to perform Gaussian filtering on the grayscale image obtained in step 501 to remove high-frequency noise; Step 503: Use an edge detection algorithm to extract edge lines from the image after Gaussian filtering; generate a mask matrix with the same size and dimensions as the original image and initialize it to all zeros, i.e., all black. Step 504: Construct the region of interest on the mask matrix by referring to the original image; Step 505: Use the polygon drawing function to fill the defined polygon outline with a fill value of 1, i.e., all white; determine the coordinates of four vertices in each of the two regions, i.e., the two regions of interest on the left and right sides of the belt obtained in step 404, and the shape is a trapezoid. Step 506: Using the image bitwise AND operation function, perform a bitwise AND operation with the edge-detected image to retain the white pixel values ​​in the corresponding region of interest in the original image, remove the black pixel values, generate a mask, and extract the region of interest (ROI).

6. The method for detecting belt misalignment in coal mines based on optical flow detection and end-to-end connection as described in claim 5, characterized in that: Step 6, combined with a straight line detection algorithm, processes the edge image from step 5 to obtain straight line information, marks the straight lines on the belt edge, and uses the position information of the belt and idlers to determine whether the belt is misaligned during transportation. The specific process is as follows: Step 601: Combine the line detection algorithm to process the line information in the image edge map obtained in step 5; Step 602: Further preprocess the line information obtained in step 601. The processing method is as follows: First, use the line L in step 403 to divide the lines in the image into left and right groups, and classify them into left and right lists respectively; Second, calculate the slope of each line information obtained in step 601, and calculate the slope lslope of the center points z1 and z2 of the left idler roller in step 404, and calculate the slope rslope of the center points z3 and z4 of the right idler roller. Use lslope and rslope to filter the lines in the left and right lists within the specified error range respectively. Step 603: Perform least squares fitting on the lines in the left and right lists after filtering in step 602 to obtain the slope and intercept. According to the linear formula y=k*x+b, for the slope and intercept obtained by fitting the lines in the left list, take the midpoint and maximum value of the vertical axis of the image as the y coordinate, calculate the corresponding x coordinate, and connect the coordinates of the two points to obtain the straight line of the left belt edge. For the slope and intercept obtained by fitting the lines in the right list, take the midpoint and maximum value of the horizontal axis of the image as the x coordinate, calculate the corresponding y coordinate, and connect the coordinates of the two points to obtain the straight line of the right belt edge. Step 604: Based on the position information of the belt and idler rollers, determine whether the belt has deviated during transportation.

Images