A coal flow foreign matter identification method for a coal mine belt conveyor based on machine vision

Abstract

The present application belongs to the technical field of machine vision and image processing, and discloses a coal flow foreign matter identification method for a coal mine belt conveyor based on machine vision. The method comprises the following steps: S1: obtaining a coal flow foreign matter image of the belt conveyor; S2: after enhancement processing of the foreign matter image, the foreign matter in the image is labeled, and then the foreign matter image data is divided into a training set and a test set; S3: a target detection model is built based on an improved YOLOv5 algorithm, and the improvement specifically comprises: setting a CA attention module between a Backbone network and a Neck network, adopting an FReLU function to replace an activation function of an original convolution layer in the Backbone network, and simultaneously optimizing a loss function and adding a penalty term to an existing loss function; S4: obtaining a final target detection model through training and testing; and S5: obtaining a real-time conveyor image and inputting the image into the target detection model for foreign matter identification. The present application can improve the detection speed and accuracy.

Description

Technical Field

[0001] This invention belongs to the field of machine vision and image processing technology, and specifically relates to a machine vision-based method for identifying foreign objects in coal flow of a coal mine belt conveyor. Background Technology

[0002] Belt conveyors are crucial equipment for coal transportation in coal mines. With the continuous improvement of coal production efficiency, ensuring the safe and reliable operation of belt conveyors has become increasingly important. Longitudinal belt tears pose a significant hazard, primarily caused by the intrusion of non-coal foreign objects such as anchor bolts, channel steel, and gangue. Therefore, timely detection and removal of foreign objects from the belt conveyor system in the early stages is of great significance in preventing belt tear accidents. Analysis and detection of coal flow videos collected by inspection robots can promptly identify foreign objects that have not been removed in time, alerting inspection personnel to handle them promptly and preventing further damage to the conveyor belt.

[0003] Retinex image enhancement theory is a class of image enhancement methods, including single-scale Retinex (SSR), multi-scale Retinex (MSR), and multi-scale Retinex with color restoration (MSRCR). The multi-scale Retinex with color restoration adds a color adjustment factor to the MSR method, achieving good dynamic range while maintaining excellent color restoration performance.

[0004] The YOLO series of algorithms are regression-based single-stage object detection algorithms that fuse bounding box prediction and recognition, significantly improving detection speed. The existing YOLOv5 model mainly consists of four parts: Input, Backbone, Neck, and Prediction. The Input part augments the training set before feeding it into the Backbone. The Backbone includes five convolutional layers, four C3 modules, and one SPPF module for extracting image features. The Neck fused the extracted features at multiple scales. The Head predicts the extracted features at three scales, outputting a feature vector containing object category information.

[0005] The YOLOv5 algorithm boasts high detection speed and accuracy, but the high dust and humidity levels in underground coal mines, along with interference from miners' lamps and changing lighting, result in poor clarity and edge prominence in the raw images acquired from the visual sensors, impacting target detection accuracy. Furthermore, multi-scale targets in the image and reduced target saliency due to environmental factors also decrease the detection accuracy of the existing algorithm framework. Therefore, improvements to existing target detection algorithms are needed to achieve foreign object identification on belt conveyors under the harsh conditions of underground coal mine environments. Summary of the Invention

[0006] To overcome the shortcomings of existing target detection algorithms in identifying foreign objects in belt conveyors under harsh underground coal mine environments, this invention provides a machine vision-based method for identifying foreign objects in coal flow on coal belt conveyors.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: a method for identifying foreign objects in coal flow of a coal mine belt conveyor based on machine vision, comprising the following steps:

[0008] S1: Acquire images of foreign objects in the coal flow of the belt conveyor;

[0009] S2: After enhancing the foreign object image, the foreign object is labeled, and then the foreign object image data is divided into training set and test set;

[0010] S3: Build an object detection model. The object detection model is an improvement of the YOLOv5 algorithm. The improvement specifically includes: setting a CA attention module between the Backbone network and the Neck network of the YOLOv5 algorithm, using the FReLU function to replace the activation function of the original convolutional layer in the Backbone network, and optimizing the loss function.

[0011] S4: Train the object detection model using the training set data, and then test the object detection model using the test set data after training. After testing, the final object detection model is obtained.

[0012] S5. Acquire a real-time image of the conveyor, perform enhancement processing on it, and input it into the target detection model obtained in step S4 for foreign object identification.

[0013] The formula for optimizing the loss function is as follows:

[0014] L γCIOU =IOU γ L CIOU ;

[0015] Where γ is a parameter, IOU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box, and L... CIOU The original loss function is expressed in the following formula:

[0016]

[0017] α is the weighting parameter, and V represents B and B'. gt The similarity of aspect ratios, b and b gt ρ(·) represents the coordinates of the center points of the predicted bounding box and the ground truth bounding box, respectively. ρ(·) represents the distance between the two center points, and c represents the diagonal distance between the minimum bounding rectangles of the predicted bounding box and the ground truth bounding box.

[0018] The value of γ is in the range of [0, 5].

[0019] The method for enhancing the image in steps S2 and S5 is as follows:

[0020] Convert the image from RGB color space to YCrCb color space, extract the Y channel, and calculate the mean and variance of the Y channel;

[0021] Set the mean threshold and variance threshold;

[0022] Determine if the mean or variance of the image corresponds to a threshold. If it is greater than the threshold, calculate the information entropy of the Gaussian blurred image at the first three scales, and assign different weights to the three Gaussian blurred images according to the magnitude of the information entropy for image fusion. The fused image is then used as the enhanced image.

[0023] The formula for calculating the weight is:

[0024]

[0025] Among them, H k The information entropy of the image is represented by k, where k is the blur scale; N represents the number of scales.

[0026] The formula for calculating information entropy is:

[0027]

[0028] Where H represents information entropy, P(Z) j ) represents the probability of the j-th gray level appearing.

[0029] The formula for calculating the mean m of the Y channel is:

[0030]

[0031] The formula for calculating the variance of the Y channel is:

[0032]

[0033] Where m represents the mean value of the Y channel image, σ 2 Represents the variance of the Y channel, Z j P(Z) represents the gray level of the Y channel image.j ) represents the probability of gray levels appearing.

[0034] There are three CA attention modules, which are respectively set between the output of the second C3 module and the third C3 module of the Backbone network and the Neck network.

[0035] The CA attention module includes residual blocks, X-average pooling layers, Y-average pooling layers, information fusion modules, two-dimensional convolution modules, Sigmoid activation modules, and weight adjustment modules.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1. This invention provides a machine vision-based method for identifying foreign objects in coal flow from a coal mine belt conveyor. By adaptively enhancing the image and improving the YOLOv5 algorithm, a Coordinate Attention (CA) module is introduced into the backbone network, which helps the model more accurately locate and identify targets of interest, improving detection speed. Simultaneously, the FReLU function is introduced to replace the original convolutional activation function in the backbone network, further improving detection accuracy. Furthermore, a penalty term is added to the loss function to address the imbalance between positive and negative samples in the original loss function.

[0038] 2. This invention addresses the poor imaging conditions in underground coal mines by setting a threshold, calculating the mean and variance of the Y channel of the image, and determining their relationship with the threshold. Images exceeding the threshold are then processed using guided processing, while those exceeding the threshold are processed using an adaptive enhancement algorithm. This solves the problem of uneven image illumination and further improves the accuracy of foreign object detection. Attached Figure Description

[0039] Figure 1 A flowchart illustrating a machine vision-based method for identifying foreign objects in coal flow of a coal mine belt conveyor, provided as an embodiment of the present invention.

[0040] Figure 2 This is the original YOLOv5 architecture diagram;

[0041] Figure 3 This is a structural diagram of the CA attention module used in the embodiments of the present invention;

[0042] Figure 4 This is a diagram of the improved YOLOv5 structure used in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, this embodiment of the invention provides a machine vision-based method for identifying foreign objects in coal flow of a coal mine belt conveyor, comprising the following steps:

[0045] S1: Obtain images of foreign objects in the coal flow of the belt conveyor. Specifically, cameras and related data acquisition equipment carried by the belt conveyor inspection robot can be used to obtain data on common foreign objects and build a dataset.

[0046] S2: After enhancing the foreign object image, the foreign objects are labeled. Then, the foreign object image data is divided into a training set and a test set. The ratio of the training set, test set, and validation set is 8:2:1. Labels are added using the labelimg tool. The label box surrounds the object, indicating the location and size of the foreign object. The label name is filled with the type of foreign object. The labeled label file format is txt.

[0047] In this embodiment, the adaptive MSRCR method is used to enhance the image, highlight image features, and solve the problem of uneven illumination in downhole images. Specifically, in this embodiment, the method for image enhancement is as follows:

[0048] (1) Convert the image from RGB color space to YCrCb color space, extract the Y channel and calculate the mean and variance of the Y channel.

[0049] Color space conversion formula:

[0050]

[0051] R, G, and B each represent a channel of the input image.

[0052] Y-channel mean formula:

[0053]

[0054] Y-channel variance formula:

[0055]

[0056] Where m represents the mean value of the Y channel image, σ 2 Represents the variance of the Y channel, Z j P(Z) represents the gray level of the Y channel image. j) represents the probability of gray levels appearing.

[0057] (2) Set the mean threshold and variance threshold;

[0058] (3) Determine whether the mean or variance of the image corresponds to the threshold. If it is greater than the threshold, calculate the information entropy of the Gaussian blurred image at N scales, and assign different weights to the three Gaussian blurred images according to the size of the information entropy to perform image fusion. The fused image is used as the enhanced image.

[0059] The formula for calculating the weights is as follows:

[0060]

[0061] Among them, H k This represents the information entropy of the image when the blur scale is k; N represents the number of scales, typically 3; the formula for calculating information entropy is:

[0062]

[0063] Where H represents information entropy, P(Z) j ) represents the probability of the j-th gray level appearing.

[0064] Enhanced image The calculation formula is:

[0065]

[0066]

[0067]

[0068] Where α and β are empirical parameters, i represents the image channel, and I i (x, y) represents the image of the i-th channel, C i (x, y) represents the color reversion factor of the i-th channel, used to adjust the proportion of colors in the three channels, N represents the number of blur scales, k represents the k-th blur scale, and F k (x, y) is the filter function, typically a Gaussian filter function. In image enhancement, N filter functions are convolved with the original image to obtain N illuminated images; ω k Weights for each scale. This represents the enhanced image of the i-channel.

[0069] S3: Build an object detection model, which is an improvement on the YOLOv5 algorithm. The improvement specifically includes: setting a CA attention module between the Backbone network and the Neck network of the YOLOv5 algorithm, using the FReLU function to replace the activation function of the original Conv convolutional layer in the Backbone network, and optimizing the loss function.

[0070] In this embodiment, the improved YOLOv5 algorithm is used as the target detection model, such as Figure 2 As shown, the existing YOLOv5 model mainly consists of four parts: Input, Backbone, Neck, and Prediction. The Input part performs data augmentation operations such as concatenation, flipping, and cropping on the input dataset, and the augmented data is then input into the Backbone. The Backbone part includes 5 convolutional layers, 4 C3 modules, and a Spatial Pyramid Pooling Fast (SPPF) module, which is used to extract image features. The Neck part fuses the extracted features at multiple scales, increasing the information contained in the feature map. The Prediction part predicts the extracted features at three scales, outputting a feature vector containing target category information. Then, based on the obtained feature vector, it classifies and identifies the target in the image to be detected and outputs the detection result.

[0071] like Figure 4 As shown, in this embodiment, a Coordinate Attention (CA) module is set between the Backbone network and the Neck network. The attention mechanism can improve target detection accuracy and increase the algorithm's inference speed by increasing the attention paid to the target region. The CA attention mechanism encodes features in two directions, then fuses the long-range dependencies obtained in one direction with the precise location information obtained in the other, ultimately generating a feature map sensitive to both direction and location, thus enhancing the algorithm's focus on the target of interest. The structure of the CA attention module is as follows: Figure 3 As shown, the CA attention module includes a residual block, an X-average pooling layer, a Y-average pooling layer, an information fusion module, a two-dimensional convolution module, a Sigmoid activation module, and a weight adjustment module.

[0072] The implementation of the CA attention mechanism can be divided into two processes: coordinate information embedding and coordinate attention generation. Coordinate information embedding uses two convolutional kernels of size H×1 and 1×W to perform global pooling in the horizontal and vertical directions of the feature map, respectively, to obtain the positional information of the feature map in these two directions. Coordinate attention generation first concatenates the results from the horizontal and vertical directions to obtain an attention feature map, then divides the result of non-linear activation into horizontal and vertical feature maps. These feature maps are then subjected to convolution and sigmoid activation to obtain attention weights. Finally, the input feature map and the obtained attention weights are multiplied to obtain an attention feature map containing the exact location information of the object of interest. The structure of the CA attention mechanism is as follows: Figure 3 As shown.

[0073] Specifically, such as Figure 4 As shown, in this embodiment, there are three CA attention modules, which are respectively set between the output of the second C3 module of the Backbone network, the third C3 module, and the SPPF module and the Neck network.

[0074] In this embodiment, the FReLU function is used to replace the activation function of the original Conv convolutional layer in the Backbone network. The FReLU function adds spatial conditions to the ReLU / PReLU function, extending ReLU and PReLU into a visually parametric ReLU with pixel-level modeling capabilities. Figure 4 As shown, the improved convolutional layer is denoted by FConv.

[0075] In this embodiment, a penalty term is added to the original loss function to improve the imbalance of negative samples in CIOU (Complete Intersection over Union). The improved function is denoted as γCIOU, and the improved loss function is as follows:

[0076] L γCIOU =IOU γ L CIOU (9)

[0077] Where γ is a parameter, IOU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box, and L... CIOU The original loss function is expressed in the following formula:

[0078]

[0079]

[0080]

[0081]

[0082] In the above formula, B and B gt b and b' represent the predicted bounding box and the ground truth bounding box, respectively. gt Let B and B' represent the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. ρ(·) represents the distance between the two center points, c represents the diagonal distance between the minimum bounding rectangles of the predicted and ground truth bounding boxes, α is the weight parameter, and V is used to measure B' and B''''''''''''''''''''''''""""," and V' ... gt The similarity of aspect ratio, R CIOU γ represents the CIOU penalty term, and IOU represents the intersection-union ratio of the predicted bounding box and the ground truth bounding box. Experiments show that a value of γ of 0.5 yields better results.

[0083] S4: Train the object detection model using the training set data, and then test the object detection model using the test set data. After the test is completed, the final object detection model is obtained.

[0084] The test results return the bounding box coordinates of the foreign object (the center coordinates of the bounding box, the length and width information of the bounding box) and the type of foreign object. Based on the test results, the training parameters are continuously adjusted and the network structure is optimized so that the final model meets the actual needs.

[0085] S5. Acquire real-time conveyor images, perform enhancement processing, and input them into the target detection model obtained in step S4 for foreign object identification. After identifying a foreign object, mark it, issue a warning, and remind staff to handle it.

[0086] The following parameters were chosen as evaluation criteria for improving object detection algorithms: GFLOPS (representing the computational cost of the model, in billions of floating-point operations per second), mAP-50, and FPS (algorithm detection speed). mAP represents the average precision when the Intersection over Union (IOU) between the predicted and ground truth bounding boxes is greater than 0.5; it is the average precision of all categories. The formula for mAP is as follows:

[0087]

[0088]

[0089]

[0090] TP represents positive samples predicted as positive; FP represents negative samples predicted as positive; P represents precision; an IOU greater than or equal to 0.5 indicates a positive sample, otherwise it indicates a negative sample; AP represents the area under the PR curve. For n samples of a certain class, assuming there are m positive examples, each positive example corresponds to a recall R value. The maximum precision P is calculated for each recall, and then the average of these m P values ​​is calculated. j represents the j-th class, C represents the total number of classes, and p(r) represents the PR curve.

[0091] Using validation set data as an example, the improved algorithm was tested. The test results are shown in Table 1. The original YOLOv5 object detection algorithm mAP-50 achieved a detection rate of 94.6%, a detection speed of 44 FPS, 7,027,720 parameters, and a computational cost of 16.0 GFLOPS. The YOLOv5 object detection algorithm mAP-50 optimized by the attention mechanism, γCIOU, and FReLU in this embodiment of the invention achieved a detection rate of 96.9%, a detection speed of 57 FPS, 7,063,400 parameters, and a computational cost of 16.0 GFLOPS. The optimized algorithm improved by 1.9% compared to the original mAP-50 structure with a small increase in the number of parameters and no change in the computational cost. The detection speed was improved by 14 FPS compared to the original network model, meeting the requirements for real-time detection.

[0092] Table 1 Improved Algorithm Indicators

[0093]

[0094] In summary, this invention proposes a machine vision-based method for foreign object identification on coal mine belt conveyors. It employs the YOLOv5 target recognition algorithm for foreign object identification. By optimizing the YOLOv5 algorithm framework, a CA attention module is introduced into the YOLO algorithm's head part to help the model more accurately locate and identify targets of interest, improving detection speed. Furthermore, the FReLU function is used to replace the ReLU activation function of the convolutional layers in the main part of the original YOLOv5 framework, improving detection accuracy and accelerating model convergence. This invention also introduces γCIOU to optimize the loss function based on the original CIOU Loss, focusing the regression process on high-quality anchor boxes, further improving target detection accuracy. Therefore, this invention, through improvements to the YOLOv5 algorithm, produces a deep network model with good real-time performance and high accuracy, meeting the needs of real-time foreign object identification in underground mines. Furthermore, this invention proposes an adaptive image enhancement method in the image preprocessing section. It determines whether the image needs to be enhanced based on the variance and mean of the image. During image fusion, it assigns weights to images of different scales according to the information entropy of the Gaussian blurred images at different scales, thereby increasing the information content of the fused image. The original method has an image information entropy of 4.64, while the improved method has an image information entropy of 4.71, thus improving the algorithm speed of the image preprocessing section.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A machine vision-based foreign matter identification method for coal flow of a coal mine belt conveyor, characterized in that, The method comprises the following steps: S1: obtaining a coal flow foreign matter image of a belt conveyor; S2: after enhancement processing of the foreign matter image, the foreign matter in the image is labeled, and then the foreign matter image data is divided into a training set and a test set; S3: a target detection model is built, which is obtained by improving a YOLOv5 algorithm, and the improvement specifically includes: setting a CA attention module between a Backbone network and a Neck network of the YOLOv5 algorithm, replacing an activation function of an original convolution layer in the Backbone network with a FReLU function, and optimizing a loss function; the loss function optimization calculation formula is: ； where γ is a parameter, IOU represents the intersection over union of the predicted box and the real box, L CIOU denotes the original loss function, S4: the target detection model is trained by using the training set data, and after the training is completed, the target detection model is tested by using the test set data, and finally the target detection model is obtained after the testing is completed; S5: real-time conveyor images are obtained, which are subjected to enhancement processing and then input into the target detection model obtained in step S4 for foreign matter identification; The CA attention module has three, which are respectively arranged between the second C3 module, the third C3 module and the output end of the SPPF module of the Backbone network and the Neck network.

2. The method according to claim 1, characterized in that, The calculation formula of the original loss function is: ； is a weight parameter, denotes and a similarity of aspect ratios, and represent the center point coordinates of the predicted and ground truth bounding boxes, respectively, denotes the distance between the two center points, denotes the diagonal distance of the minimum enclosing rectangle of the predicted and ground truth bounding boxes.

3. The method according to claim 2, wherein, The value range of the gamma is [0, 5].

4. The method according to claim 1, characterized in that, In steps S2 and S5, the method for enhancing the image is: The image is converted from an RGB color space to a YCrCb color space, the Y channel is extracted, and the mean and variance of the Y channel are calculated; The mean threshold and the variance threshold are set; It is judged whether the mean or the variance of the image corresponds to the threshold, if greater, the information entropy of the Gaussian blurred image under the first three scales is calculated, and according to the size of the information entropy, three Gaussian blurred images are given different weights for image fusion, and the fused image is taken as the enhanced image; The weight calculation formula is: ； wherein, Hk represents the information entropy corresponding to the image when the blur scale is k; N represents the number of scales; The calculation formula of the information entropy is: ； where H denotes the information entropy, denotes the probability of occurrence of the jth gray level.

5. The method according to claim 4, wherein, The calculation formula of the Y channel mean m is: ； The calculation formula of the Y channel variance is: ； where m represents the Y channel image mean, σ 2 represents the Y channel variance, represents the Y channel image gray level, represents the gray level occurrence probability.

6. The machine vision-based foreign matter identification method for coal flow of a coal mine belt conveyor according to claim 1, characterized in that, The CA attention module includes a residual block, an X average pooling layer, a Y average pooling layer, an information fusion module, a two-dimensional convolution module, a Sigmoid activation module and a weight adjustment module.

Images