A belt deviation detection method based on deep learning

Abstract

This invention discloses a deep learning-based method for detecting belt misalignment, belonging to the field of image processing technology. The specific steps are as follows: 1. Acquire images of the belt conveyor in operation, create a belt edge image dataset, and train a FastBeltNet network using the belt misalignment image dataset; 2. Operators pre-set corresponding misalignment warning lines for different production lines; 3. Use the trained FastBeltNet network to obtain a segmentation mask for the belt edge in the real-time monitoring image; 4. Fit the straight line equation of the belt edge and calculate the distance to the preset misalignment warning line; 5. Operators provide misalignment standards according to different production requirements and select an appropriate misalignment threshold for alarm. This invention only requires access to the on-site monitoring video of the belt conveyor, without requiring additional modifications to the original production line, achieving low-cost, rapid, and accurate detection of belt misalignment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a belt misalignment detection method based on deep learning. Background Technology

[0002] Belt misalignment is the most common fault in belt conveyors, potentially leading to serious downtime and equipment failure, causing personal injury and production accidents. Initially, belt misalignment was mainly detected manually, but this was labor-intensive and prone to errors and omissions. Therefore, contact-type devices were introduced to replace manual inspection. These devices include installing guiding rollers or misalignment switches on the conveyor belt. However, these measurement methods have low accuracy and cannot quantitatively reflect the degree of misalignment in real time. They only function when the conveyor belt comes into contact with the vertical roller or emergency switch, failing to provide early correction. This can cause wear on the belt edges and may even lead to malfunctions in the entire conveyor system.

[0003] Considering the problems of poor detection accuracy, easy damage, and inability to visualize correction data, many researchers have thought of using machine vision and image processing technology for non-contact detection.

[0004] In 2020, Zhang M et al. published "Research on deviation detection of belt conveyor based on inspection robot and deep learning" in *Applied Sciences*. They employed image processing techniques, using the Canny edge detection algorithm and Hough transform to detect conveyor belt edges. They also proposed a deviation determination method based on line laser assistance. While the use of lasers provides a reference standard for deviation measurement in dimly lit and dusty environments, this method has a relatively slow processing speed, which does not meet the real-time requirements of factories.

[0005] In 2021, Liu Y et al. published a paper in *Complexity* titled "Research on deviation detection of belt conveyor based on inspection robot and deep learning," proposing a method for detecting conveyor belt deviation based on an inspection robot and deep learning. This method compares the distance between the conveyor belt and the edge of the roller. The two edges can be obtained using Hough transform and template matching, respectively, but this method is slow. The processing time for a single image is approximately 135.5 ms, and the roller edge detection based on the template matching method is unstable. Summary of the Invention

[0006] Given that existing belt misalignment measurement methods need improvement, this invention provides a belt misalignment detection method based on deep learning. This invention uses the FastBeltNet network to identify the belt edge and obtain a prediction mask for the edge line. The left and right edge lines are fitted from the predicted left and right edge masks. Then, the positions of the left and right edges are obtained by least-squares fitting of the two edge lines, thereby calculating the degree of belt misalignment.

[0007] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0008] The present invention provides a belt misalignment detection method based on deep learning, the steps of which are as follows:

[0009] Step 1: Collect images of the belt conveyor in operation, create a belt misalignment image dataset, and use the belt misalignment image dataset to train the FastBeltNet network;

[0010] Step 2: Staff pre-set corresponding deviation warning lines for different production lines;

[0011] Step 3: Obtain the segmentation mask of the belt edge in the image using the trained FastBeltNet network;

[0012] Step 4: Fit the straight line equation of the belt misalignment mask, and then calculate the distance to the preset misalignment warning line;

[0013] Step 5: Staff members set deviation standards based on different production requirements and select appropriate deviation thresholds to trigger an alarm.

[0014] This solution acquires real-time images of the conveyor belt in operation and uses a deep learning-based semantic segmentation model to determine belt misalignment. The proposed FastBeltNet semantic segmentation model is an improvement on the BiSeNet dual-branch network. The feature fusion branch in the network supervises different positions of the context branch to extract semantic information from different stages, thus complementing the spatial branch's extraction of low-level spatial information. The detection accuracy reaches 79.64 mIoU (%), matching the level of current mainstream semantic segmentation models. Simultaneously, the network has been lightweighted, with only 1.02M parameters and a detection speed of 100.53 FPS, making it adaptable to various embedded devices for real-time detection and achieving a good balance between speed, accuracy, and parameter size. This deep learning-based belt misalignment detection method can directly input images acquired on-site into the model for judgment, requiring only pre-marking of warning lines by staff. Compared to traditional mechanical devices and later machine vision-based detection solutions, it reduces maintenance costs and significantly improves detection accuracy and speed, resulting in superior performance.

[0015] Furthermore, in step one, images of the belts at different times and locations were collected from the actual site. The left and right edges of the belts in the images were labeled using Labelme software to create a belt edge line image dataset. The obtained dataset was randomly divided into a training set and a test set to train the FastBeltNet network.

[0016] Furthermore, in step two, before the inspection, the staff pre-determines the left and right warning lines for belt misalignment. Specifically, the two preset left and right warning lines are drawn using the cv2.setMouseCallback function of OpenCV.

[0017] Furthermore, step three employs the FastBeltNet semantic segmentation network, an improvement upon the BiSeNet model. It utilizes a dual-branch semantic segmentation neural network, extracting low-level spatial information only through stacked convolutional layers in a lightweight spatial configuration. This lightweighting is achieved by adding GhostMoudle and downsampling modules to the context branch. Simultaneously, an Injection remote feature supplementation module compensates for the accuracy loss caused by the lightweighting. Finally, the low-level spatial information extracted from the spatial branch is combined with the semantic information extracted from the context branch. This network ensures both detection accuracy and real-time performance while maintaining a lightweight design.

[0018] Furthermore, in step four, the segmentation mask predicted by the FastBeltNet network is divided into three categories based on the grayscale value of each pixel: background (0), left edge (1), and right edge (2).

[0019] Furthermore, in step four, when processing the image, a Cartesian coordinate system is established with the top left corner as the origin, the horizontal direction to the right as the x-axis, and the vertical direction downward as the y-axis. At this point, each pixel of the mask on the left and right edges can be regarded as (x, y) coordinates.

[0020] Furthermore, in step four, when calculating the straight-line distance, the least squares method is first used to fit the straight-line equations of the left and right edges, and then the distances from the left and right edge lines to their respective warning lines are calculated separately.

[0021] Furthermore, steps four and five calculate the belt misalignment threshold, and the standard criteria can be adjusted according to actual requirements.

[0022] Furthermore, step five calculates the degree of belt misalignment, using the following formula:

[0023]

[0024] Where Threshold is the calculated deviation threshold, d1 is the position from the left edge of the belt to the left warning line, and d2 is the position from the right edge of the belt to the right warning line.

[0025] Compared with the prior art, the technical solution provided by the present invention has the following significant effects:

[0026] (1) The present invention provides a belt deviation detection method based on deep learning. First, the FastBeltNet network is trained using a belt image dataset. The trained network obtains the mask of the edge line in the image. Then, based on the idea that the belt edge line mask tends to be a straight line, the left and right edge lines are fitted. The deviation degree is calculated by the straight line distance from the edge line to the preset warning line. Since only two edge lines need to be predicted by the FastBeltNet network, the expected effect can be achieved with a small dataset. The FastBeltNet network has no limitation on the size of the input image and is suitable for various environments.

[0027] (2) The belt misalignment detection method based on deep learning of the present invention uses deep learning network and simple image processing calculation to predict the belt edge line. It only requires a small amount of data. The simple edge line target reduces the data annotation time. It has high accuracy, fast speed and small memory requirements. It is suitable for real-time detection environment in factories. The error is within the required range. Attached Figure Description

[0028] Figure 1 This is a flowchart of the belt misalignment measurement process in this invention;

[0029] Figure 2 This is a diagram of the FastBeltNet network structure in this invention;

[0030] Figure 3 This is the original image;

[0031] Figure 4 This is a schematic diagram of the prediction mask in this invention;

[0032] Figure 5 This is a schematic diagram of the mathematical model for calculating the deviation threshold in this invention. Detailed Implementation

[0033] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments.

[0034] Example 1:

[0035] Combination Figure 1This embodiment presents a deep learning-based belt misalignment detection method. First, a labeled belt image dataset is collected. Then, the FastBeltNet network is trained. The operator needs to pre-define two deviation warning lines on the left and right edges. The trained FastBeltNet network is used to obtain a prediction mask for the belt edge lines in the image. Then, the least squares method is used to fit the mask image to obtain the equations of the left and right edge lines. The deviation distance from the preset edge warning lines is calculated, and finally, the degree of deviation is calculated. The specific steps are as follows:

[0036] Dataset collection and annotation: Images of actual belts from different time periods and locations were collected. The left and right edges of the belts in the images were annotated using Labels to create a belt edge line image dataset. The obtained dataset was randomly divided into training and testing sets to train the FastBeltNet network.

[0037] Preset belt misalignment warning lines: Before inspection, the staff pre-determines the left and right warning lines for belt misalignment. Specifically, the two preset left and right warning lines are drawn using the cv2.setMouseCallback function of OpenCV.

[0038] Training the belt misalignment detection network: The obtained dataset is randomly divided into a training set and a test set, and the FastBeltNet network is trained using the training set.

[0039] Predicting belt images: Images of the belt conveyor in operation are acquired in the computer, and the trained FastBeltNet network is used to identify the belt images to obtain the predicted left and right masks of the belt.

[0040] Fitting the straight lines of the belt edge: Perform least squares straight line fitting on the two obtained masks to obtain the left and right edge straight lines of the predicted mask.

[0041] Least squares linear fitting formula:

[0042]

[0043]

[0044]

[0045] Where, x i With y i Let xy coordinates be the coordinates of each mask pixel. Represents all x i The average value of the data points Represents all y iThe mean of the data points, n is the number of data points, SSxx represents the sum of the squared differences between the x data points and their mean, SSyy represents the sum of the squared differences between the y data points and their mean, SSxy represents the covariance of the x and y data points and their respective means, m represents the slope of the regression line, b represents the intercept of the regression line, and y = mx + b represents the formula for the fitted mask line.

[0046] Calculate the distance from the edge line to the warning line: Calculate the distance from the two obtained left and right edge lines to the corresponding warning lines to obtain the left distance d1 and the right distance d2.

[0047] Formula for straight-line distance:

[0048]

[0049] Where y1 and y2 represent the linear formulas of the mask line and the warning line, and d represents the vertical linear distance between the two parallel lines, the edge line and the warning line.

[0050] Calculate the degree of deviation: The degree of deviation is calculated by using the left and right distances in the formula. The standard can be adjusted according to actual requirements.

[0051]

[0052] Where abs is the absolute value function.

[0053] Figure 3 The original image of the belt conveyor in operation; Figure 4 It is a mask extracted by identifying edges using the FastBeltNet network; Figure 5 Based on the extracted image mask, the least squares method is used to fit the mask curve, and the fitted straight line represents the edge line of the conveyor belt. This invention only requires the FastBeltNet network to extract the edge line of the belt, has good robustness, has no limitation on the size of the input image, and can detect conveyor belts at different times and locations. After extracting the mask from the original image, two edge straight lines are fitted using the least squares method, and the degree of deviation is calculated by calculating the straight-line distance between the two lines and the preset warning line.

[0054] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for detecting belt deviation based on deep learning, characterized in that, The method includes the following steps: Step 1: Collect images of the belt conveyor in operation, create a belt misalignment image dataset, and use the belt misalignment image dataset to train the FastBeltNet network; Step 2: Staff pre-set corresponding deviation warning lines for different production lines; Step 3: Obtain the segmentation mask of the belt edge in the image using the trained FastBeltNet network; Step 4: Fit the linear equation of the belt misalignment mask and then calculate the distance to the preset misalignment warning line; Step 5: Staff members set deviation standards based on different production requirements and select appropriate deviation thresholds to trigger an alarm. In step three, the FastBeltNet semantic segmentation network is an improvement on the BiSeNet model. It adopts a dual-branch semantic segmentation neural network. In the lightweight space, low-level spatial information is extracted only by stacking convolutional layers. The model is lightweighted by adding GhostMoudle and downsampling modules to the context branch. At the same time, an Injection feature supplementation module is added to compensate for the accuracy loss caused by the lightweighting. Finally, the low-level spatial information extracted by the spatial branch and the semantic information extracted by the context branch are combined and complemented.

2. The belt deviation detection method based on deep learning according to claim 1, characterized in that: Step 1: Collect images of the actual belt conveyor production line. Use Labelme software to annotate the left and right edges of the belt in the images to create a belt edge line image dataset. Randomly divide the obtained dataset into training and test sets to train the FastBeltNet network.

3. The belt deviation detection method based on deep learning according to claim 2, characterized in that: In step two, before the inspection, the staff pre-determines the left and right warning lines for belt misalignment. Specifically, the two preset left and right warning lines are drawn using the cv2.setMouseCallback function of OpenCV.

4. The belt deviation detection method based on deep learning according to claim 3, characterized in that: The segmentation mask predicted by the FastBeltNet network in step four is divided into three categories according to the gray value of each pixel: background 0, left edge 1, and right edge 2.

5. The belt misalignment detection method based on deep learning according to claim 4, characterized in that: In step four, when processing the image, a Cartesian coordinate system is established with the top left corner as the origin, the horizontal direction to the right as the x-axis, and the vertical direction downward as the y-axis. At this time, each pixel of the mask on the left and right edges is regarded as (x, y) coordinates.

6. The belt deviation detection method based on deep learning according to claim 5, characterized in that: Step four involves calculating the straight-line distance by first fitting the straight-line equations of the left and right edges using the least squares method, and then calculating the distances from the left and right edge lines of the belt to their respective warning lines.

7. The belt misalignment detection method based on deep learning according to claim 2, characterized in that: Step 5: Calculate the degree of belt misalignment. The specific formula is as follows: in, To calculate the degree of deviation, This is the distance from the left edge of the belt to the left warning line. is the distance from the right edge of the belt to the right warning line, and abs() is the absolute value function.

Images