A belt conveyor multi-operation state detection method and device based on deep learning

Abstract

The present disclosure relates to a kind of deep learning-based belt conveyor multi-operation state detection method and device, method includes: real-time acquisition conveyor operation monitoring image;The feature of monitoring image is extracted using feature extraction network, and the feature is respectively input into segmentation network and prediction network;Monitoring image is carried out semantic segmentation using segmentation network, and the material load area and the edge line of both sides of conveyor belt are obtained;According to material load area, detect load, according to the edge line of both sides of conveyor belt, judge the deviation state of conveyor belt;Through prediction network, obtain the detection frame for multi-class target;According to the detection frame of multi-class target, obtain the detection result of conveying foreign matter, conveyor belt damage, conveyor belt water accumulation, conveying along line material scattering, conveying area smoke, conveying area fire.The above method makes single network model can simultaneously realize the multi-task of including load detection, conveyor belt deviation state detection, large block conveying foreign matter detection, conveyor belt damage detection and so on, provides support data for the safety control in the process of material transportation.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent coal mining equipment technology, and in particular to a method and device for detecting multiple operating states of a belt conveyor based on deep learning. Background Technology

[0002] Belt conveyors, as the main transportation method in underground and open-pit coal mines, are the preferred equipment for continuous conveying of bulk materials and are currently developing towards larger scale, higher intelligence, and greater energy efficiency. Among these developments, the intelligentization of belt conveyors is a crucial measure to actively respond to the call for energy conservation and emission reduction, reduce worker workload, improve system reliability, and achieve sustainable and efficient development for coal mines and other enterprises. From a software perspective, it mainly includes two parts: methods for rapid and accurate perception of the conveyor's operating status and corresponding adaptive control strategies. Summary of the Invention

[0003] Currently, methods for sensing the operating status of conveyors have been well developed and are gradually being applied to the comprehensive and full lifecycle management of belt conveyor systems and their key components. Among these, the conveyor belt, as one of the most important components of a belt conveyor, plays a crucial role in carrying and traction of materials. Furthermore, as the ultimate function and output of the entire belt conveyor system, its apparent operating status directly or indirectly reflects the stability of the entire system. Therefore, monitoring the operating status of the conveyor belt is of considerable significance and value.

[0004] Thanks to the development of artificial intelligence and computer technology, many advanced sensing technologies are now being applied to the monitoring of conveyor belt operation status, such as artificial intelligence, deep learning technology, smart chip technology, infrared camera technology, and weak magnetic induction technology. There are also some load monitoring and damage detection technologies based on point cloud processing or ultrasonic sensors. In addition, for key components such as idlers and rollers that are in direct contact with the conveyor belt, fault diagnosis or classification technologies based on vibration sensing or sound sensing are being gradually proposed and further applied.

[0005] From the perspective of state monitoring methods based solely on deep learning and machine vision, it can be observed that most current monitoring methods can only handle single-task detection and cannot simultaneously handle multiple tasks. This necessitates the support of multiple models capable of fulfilling different detection tasks when simultaneously monitoring multiple operating states of a belt conveyor. This method of multiple models executing together limits their application in real-world scenarios to some extent and also places higher demands on the system's hardware investment.

[0006] To address the aforementioned technical issues, this disclosure provides a method and system for detecting multiple operating states of a belt conveyor based on deep learning.

[0007] In a first aspect, this disclosure provides a deep learning-based method for detecting multiple operating states of a belt conveyor. The method includes: real-time acquisition of conveyor operation monitoring images; extraction of features from the monitoring images using a feature extraction network, and inputting the features into a segmentation network and a prediction network respectively; semantic segmentation of the monitoring images using the segmentation network to obtain the material load area and the edge lines on both sides of the conveyor belt; detection of the load based on the material load area, and determination of the conveyor belt deviation state based on the edge lines on both sides of the conveyor belt; obtaining detection boxes for multiple categories of targets through the prediction network; and obtaining detection results for conveyed foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveyor line, smoke in the conveyor area, and fire in the conveyor area based on the detection boxes for multiple categories of targets.

[0008] In some embodiments, features of a surveillance image are extracted using a feature extraction network, and the features are input into a segmentation network and a prediction network, respectively. This includes: extracting features of the surveillance image using a feature extraction network to obtain a first feature and a second feature, wherein the first feature and the second feature are feature maps of different scales output by the feature extraction network; inputting the first feature into a feature enhancement network to obtain a first enhanced feature; inputting the second feature and the first enhanced feature into a segmentation network; and inputting the first enhanced feature into a prediction network.

[0009] In some embodiments, the feature extraction network includes: a convolutional module, a residual convolutional module, and a spatial pyramid pooling module; wherein, the convolutional module is composed of convolutional layers, batch normalization layers, and loss function layers connected together; the residual convolutional module is composed of convolutional modules, residual layers, and concatenation layers connected together; a second feature is determined by the first convolutional module and the first residual convolutional module in the feature extraction network; and a first feature is determined by the second residual convolutional module, the third residual convolutional module, and the spatial pyramid pooling module.

[0010] In some embodiments, semantic segmentation of a monitoring image is performed using a segmentation network to obtain a material load area and the edge lines on both sides of the conveyor belt. This includes: performing semantic segmentation of the monitoring image using a segmentation network to obtain multiple segmentation results, and determining the category of the segmentation results; in response to the current segmentation category belonging to the left edge of the conveyor belt, determining and outputting the segmentation result as the left edge line and its corresponding equation expression in the mathematical geometric space, wherein the left edge line is the result obtained by fitting the left edge segmentation line of the conveyor belt obtained by the segmentation network; in response to the current segmentation category belonging to the right edge of the conveyor belt, determining and outputting the segmentation result as the right edge line and its corresponding equation expression in the mathematical geometric space, wherein the right edge line is the result obtained by fitting the right edge segmentation line of the conveyor belt obtained by the segmentation network; and in response to the current segmentation category not belonging to the edge of the conveyor belt, determining and outputting the segmentation result as the material load area.

[0011] In some embodiments, the left edge line is the result obtained by fitting the left edge segmentation line of the conveyor belt obtained by the segmentation network, and the right edge line is the result obtained by fitting the right edge segmentation line of the conveyor belt obtained by the segmentation network. This includes: if the state detection method is applied to the monitoring of the straight section of the belt conveyor, the fitting method is linear fitting; if the state detection method is applied to the monitoring of the curved section of the belt conveyor, the fitting method is nonlinear fitting.

[0012] In some embodiments, a segmentation network is used to perform semantic segmentation on the monitoring image to obtain multiple segmentation results. The category of the segmentation results is determined, including: if the multi-operation state detection method for belt conveyors is applied to a tubular belt conveyor, the edge segmentation result of one side of the conveyor belt is obtained by segmentation through the segmentation network, the edge segmentation line and its corresponding equation expression in the mathematical geometric space are determined and output, and the edge segmentation line is the result obtained by linear fitting of the conveyor belt edge segmentation line obtained by the segmentation network.

[0013] In some embodiments, detecting the load based on the material load area includes: determining the area of ​​the material load area, the area being determined based on the number of pixels; determining the proportion of the material load area occupying the monitoring image based on the area of ​​the material load area; and determining the load based on the proportion.

[0014] In some embodiments, determining the conveyor belt deviation state based on the two side edge lines of the conveyor belt includes: determining the offset and tilt state of the conveyor belt by using the equation expressions in the mathematical geometric space corresponding to the left and right edge lines of the conveyor belt.

[0015] In some embodiments, the offset and tilt state of the conveyor belt are determined by the equations in the mathematical geometric space corresponding to the left and right edge lines of the conveyor belt. This includes: if the multi-operational state detection method is applied to the monitoring of the straight section of the belt conveyor, the slope and position information of the edge dividing line are determined by the equation in the mathematical geometric space to determine the offset and tilt state of the conveyor belt; if the multi-operational state detection method is applied to the monitoring of the curved section of the belt conveyor, the position information or corresponding area of ​​the edge dividing line are determined by the equation in the mathematical geometric space to determine the belt misalignment state.

[0016] In some embodiments, determining the position information of the edge segmentation line or the corresponding area region, and determining the deviation state of the conveyor belt, includes: determining the intersection point of the identified inner edge curve of the conveyor belt and the diagonal of the monitoring image, and judging the deviation state based on the distance of the intersection point from the lower right corner vertex or the upper left corner vertex; or, determining the area region enclosed by the inner edge line of the conveyor belt and the image region, obtaining the size of the area region by definite integral, and judging the deviation state based on the area region.

[0017] In some embodiments, determining the conveyor belt misalignment state based on the edge lines on both sides of the conveyor belt includes: if the multi-operation state detection method for belt conveyors is applied to a tubular belt conveyor, then by using the equation expression of the obtained edge dividing line in mathematical geometric space, the slope and position information of the edge dividing line are determined, and the twisting and misalignment states of the conveyor belt are determined.

[0018] In some embodiments, the method further includes: acquiring monitoring images of the conveyor operation; labeling the monitoring images to obtain multi-task labels, the multi-task labels including segmentation labels and detection labels, wherein the segmentation labels are the material load area and conveyor belt edge line in the monitoring image; the detection labels are foreign objects being conveyed, conveyor belt damage, water accumulation on the conveyor belt, material spillage along the conveyor line, smoke in the monitored area, fire in the monitored area, and idlers on the left and right sides of the conveyor belt in the monitoring image. A multi-operational state detection network model for the belt conveyor is trained using monitoring images containing multi-task labels.

[0019] In some embodiments, a multi-operational state detection network model for a belt conveyor is trained using monitoring images containing multi-task labels, including: the loss function of the multi-operational state detection network model for the belt conveyor consists of two parts: prediction loss and segmentation loss; the prediction loss consists of three parts: classification loss, position loss, and confidence loss, and the segmentation loss consists of classification loss.

[0020] In some embodiments, the prediction network is a computational layer that connects feature maps of multiple scales.

[0021] Secondly, this disclosure also provides a deep learning-based multi-operational state detection device for belt conveyors. The device includes: an acquisition module for real-time acquisition of conveyor operation monitoring images; a feature extraction module for extracting features from the monitoring images using a feature extraction network, and inputting the features into a segmentation network and a prediction network respectively; a segmentation module for semantic segmentation of the monitoring images using the segmentation network to obtain the material load area and the edge lines on both sides of the conveyor belt; a load and conveyor belt detection module for detecting the load based on the material load area and determining the conveyor belt deviation state based on the edge lines on both sides of the conveyor belt; and a multi-category target detection module for obtaining detection boxes for multi-category targets through the prediction network; and obtaining detection results for conveyed foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveyor line, smoke in the conveyor area, and fire in the conveyor area based on the detection boxes of multi-category targets.

[0022] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: This disclosure provides a method for real-time detection of multiple operating states of a belt conveyor. The segmentation network for load detection and conveyor belt edge detection, and the prediction network for detecting objects such as conveyor foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveyor line, smoke in the monitored area, fire in the monitored area, and idlers on both sides of the conveyor belt, acquire features through the same feature extraction network and feature enhancement network. This enables feature information sharing between the segmentation network and the prediction network during feature extraction, allowing a single network model to simultaneously perform multiple tasks, including load detection, conveyor belt misalignment detection, conveyor foreign object detection, conveyor belt damage detection, conveyor belt water accumulation detection, material spillage along the conveyor line, smoke in the monitored area, fire in the monitored area, and idler identification on both sides of the conveyor belt. This provides technical support for real-time monitoring and ensuring the safety of material transportation. The method of this disclosure only requires the construction and training of a single network model to meet the needs of simultaneous multi-task detection, significantly reducing the complexity of the network model, saving model training time, and reducing hardware computing costs.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] Figure 1 A schematic flowchart of a deep learning-based method for detecting multiple operating states of a belt conveyor, according to some embodiments of this disclosure, is shown.

[0026] Figure 2 A schematic diagram of the network structure of a deep learning-based multi-operational state detection model for belt conveyors, representing some other embodiments of this disclosure, is shown.

[0027] Figure 3 A schematic diagram of the network structure of a deep learning-based multi-operational state detection model for belt conveyors, representing some other embodiments of this disclosure, is shown.

[0028] Figure 4 A schematic diagram illustrating conveyor belt misalignment calculation is shown in some embodiments of this disclosure.

[0029] Figure 5 A schematic flowchart of a multi-operational state detection method for a belt conveyor according to other embodiments of this disclosure is shown.

[0030] Figure 6 A statistical chart of sample labels from some embodiments of this disclosure is shown.

[0031] Figure 7 The tagged surveillance image, processed according to this disclosure, is shown.

[0032] Figure 8 A comparative diagram of segmentation results corresponding to various models including those disclosed herein is shown. Implementation

[0033] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the present disclosure, and are not intended to imply any limitation on the scope of the disclosure.

[0034] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment". The term "another embodiment" is to be interpreted as "at least one other embodiment".

[0035] For example, networks such as R-CNN, YOLO, SSD, and CenterNet are used to detect conveyor belt defects and damage, while algorithms like Hough, FCN, and DeepLab are used to detect or segment conveyor belt edges to determine belt misalignment. This necessitates the use of multiple networks capable of handling different detection tasks when simultaneously monitoring various operating states of a belt conveyor. The high complexity of these networks limits their application in real-world scenarios and places higher demands on system hardware.

[0036] To address the aforementioned technical problems, in some embodiments, this disclosure provides a deep learning-based method for detecting multiple operating states of a belt conveyor, such as... Figure 1 As shown, the method includes steps S1-S6, which are described in detail below.

[0037] Step S1: Real-time acquisition of monitoring images of the conveyor operation.

[0038] Cameras are installed along the belt conveyor to capture real-time images of the conveyor's operation. For example... Figure 2 As shown, the image input to the feature extraction network is the monitoring image. The camera is set to a fixed shooting angle so that the image mainly includes the conveyor belt. The monitoring image can be any frame captured from real-time video or a fixed image taken at regular intervals.

[0039] Step S2: Use the feature extraction network to extract features from the surveillance image, and input the features into the segmentation network and the prediction network respectively.

[0040] Furthermore, such as Figure 2 As shown, both the segmentation network and the prediction network obtain feature information through the same feature extraction network. In this embodiment, the input features to the segmentation network and the prediction network are the same. The segmentation network and the prediction network operate in parallel, without any temporal, spatial, or logical order. They run independently of each other, and their detection results do not affect each other. For example, when the conveyor is unloaded, the segmentation network's recognition result will not include the segmentation result for the load, but this does not affect the segmentation network's segmentation of the conveyor belt edge, nor does it affect the prediction network's detection of targets such as conveyed foreign objects or conveyor belt damage; similarly, the prediction network's detection result for potential targets will not affect the segmentation network. Since networks performing different detection tasks essentially extract and utilize information from the same image, and multiple networks share commonalities when performing lower-level image processing or feature extraction operations, it is unnecessary to repeatedly process such tasks through multiple networks to a certain extent. This could potentially reduce computational load and accelerate image analysis. When this issue is reflected in the construction of deep neural networks, this disclosure proposes to address it by combining multiple tasks when performing multi-task detection. Semantic segmentation and multiple image prediction and classification tasks share a single feature extraction network, thereby achieving feature information sharing.

[0041] In some embodiments, features of a surveillance image are extracted using a feature extraction network, and the features are input into a segmentation network and a prediction network, respectively. This includes: extracting features of the surveillance image using a feature extraction network to obtain a first feature and a second feature, wherein the first feature and the second feature are feature maps of different scales output by the feature extraction network; inputting the first feature into a feature enhancement network to obtain a first enhanced feature; inputting the second feature and the first enhanced feature into a segmentation network; and inputting the first enhanced feature into a prediction network.

[0042] An enhancement network can be added between the segmentation network, prediction network, and feature extraction network. After adding the enhancement network, the features input to the segmentation network and prediction network can be the same, or more scale feature information can be fused in the segmentation network. As can be seen in the steps of the above embodiment, the first feature and the second feature have different scales. The first enhanced feature incorporates information from the first feature in the upsampling feature enhancement network, and further incorporates information from the second feature in the segmentation network. The fusion of feature map data at more scales can further improve the segmentation accuracy of the segmentation network.

[0043] In some embodiments, the feature extraction network includes: a convolutional module, a residual convolutional module, and a spatial pyramid pooling module; wherein, the convolutional module is composed of convolutional layers, batch normalization layers, and loss function layers connected together; the residual convolutional module is composed of convolutional modules, residual layers, and concatenation layers connected together; a second feature is determined by the first convolutional module and the first residual convolutional module in the feature extraction network; and a first feature is determined by the second residual convolutional module, the third residual convolutional module, and the spatial pyramid pooling module.

[0044] like Figure 3 As shown, the feature extraction network structure consists of multiple convolutional modules (CBLs) and multiple residual convolutional modules (CSPs) connected together, followed by a spatial pyramid pooling module (SPPF). Each CBL comprises convolutional layers, batch normalization layers, and a loss function layer. The CSPs are constructed by cascading convolutional modules and adding residual modules. Finally, a concatenation layer stitches the residual results together. The SPPF extracts multi-scale feature information from the image using multiple pooling layers with different sampling rates. Figure 3 In the diagram, the first number after CSP indicates the type of residual convolutional module, and the second number indicates the number of Res residual blocks used in the module. For example, "CSP1_1" indicates that the basic structure of the residual convolutional module is as shown in CSP1_X, and the Res residual blocks contained therein are in the style of ResX (in CSP1). CSP1_X contains one Res residual block. Similarly, "CSP2_1" indicates that the basic structure of the residual convolutional module uses two Res residual blocks in the style of ResX (in CSP2). Feature extraction networks can increase network depth and enhance feature extraction capabilities by repeatedly using convolutional and residual convolutional modules.

[0045] In some embodiments, the feature enhancement network can be divided into an upsampling enhancement network and a downsampling enhancement network, such as... Figure 3 As shown, a feature extraction network is used to extract features from the surveillance image to obtain a first feature and a second feature; the first feature is input into an upsampling enhancement network to obtain a first enhanced feature; the second feature and the first enhanced feature are input into a segmentation network; the first enhanced feature is input into a downsampling enhancement network to obtain a third feature; and the third feature is input into a prediction network to obtain a multi-task bounding box.

[0046] Figure 3The schematic diagram of the network structure of the deep learning-based belt conveyor multi-state detection model also illustrates a feature enhancement method. Between the feature extraction network and the prediction network, the extracted features are enhanced by concatenating upsampled and downsampled feature layers. This allows the rich semantic information in the high-level (deep) feature layers to be integrated with the precise location information in the low-level (shallow) layers, which is more conducive to target classification and accurate positioning. Simultaneously, there is also a feature enhancement method that concatenates high and low feature layers between the feature extraction network and the prediction network, which is significant for improving the performance of the segmentation network.

[0047] Depend on Figure 3 As can be seen from this disclosure, in a specific embodiment, the segmentation network includes three upsampling stages. It is understood that those skilled in the art can increase or decrease the number of feature upsampling stages according to actual circumstances. The first enhanced feature, after passing through a convolutional module, is upsampled and then concatenated with a portion of the second feature of the same size. It then passes through a residual convolutional module, another convolutional module, and an upsampling layer before being concatenated with another portion of the second feature of the same size.

[0048] Furthermore, in one embodiment provided in this disclosure, the first feature is upsampled and enhanced by a network to obtain a first enhanced feature. This enhanced feature is then downsampled and fused at a concatenation layer using a cascaded residual convolutional module and a convolutional module. The result is then input into the next residual convolutional module, and the output of each residual convolutional module is used as the third feature and input into the prediction network. Typically, the third feature comprises feature map data at three scales.

[0049] Step S3: The segmentation network is used to perform semantic segmentation on the monitoring image to obtain the material load area and the edge lines on both sides of the conveyor belt. The main tasks of the segmentation network include load detection and edge line detection. During the conveying process, the boundary of the load area is contained within the edge lines on both sides of the conveyor belt. Therefore, the segmentation network undertakes two different segmentation tasks: load area detection is the region detection, and conveyor belt detection is the edge detection.

[0050] In some embodiments, semantic segmentation of a monitoring image is performed using a segmentation network to obtain the material load area and the edge lines on both sides of the conveyor belt. This includes: performing semantic segmentation of the monitoring image using a segmentation network to obtain multiple segmentation results, and determining the category of the segmentation results; in response to the current segmentation category belonging to the left edge of the conveyor belt, determining and outputting the segmentation result as the left edge line and its corresponding equation expression in the mathematical geometric space, wherein the left edge line is the result obtained by fitting the left edge segmentation line of the conveyor belt obtained by the segmentation network; in response to the current segmentation category belonging to the right edge of the conveyor belt, determining and outputting the segmentation result as the right edge line and its corresponding equation expression in the mathematical geometric space, wherein the right edge line is the result obtained by fitting the right edge segmentation line of the conveyor belt obtained by the segmentation network; in response to the current segmentation category not belonging to the edge of the conveyor belt, determining and outputting the segmentation result as the material load area. During load detection, the output object is the load area, which can be divided using irregular polygonal boxes based on the boundary of the load object. When segmenting edges, when the multi-operational state detection method for belt conveyors is applied to a common trough belt conveyor, the output object is the two sides of the conveyor belt, and the edge lines are the result after function fitting.

[0051] In some embodiments, the left edge line is the result obtained by fitting the left edge segmentation line of the conveyor belt obtained by the segmentation network, and the right edge line is the result obtained by fitting the right edge segmentation line of the conveyor belt obtained by the segmentation network. This includes: if the state detection method is applied to the monitoring of the straight section of the belt conveyor, the fitting method is linear fitting; if the state detection method is applied to the monitoring of the curved section of the belt conveyor, the fitting method is nonlinear fitting.

[0052] In some embodiments, a segmentation network is used to perform semantic segmentation on the monitoring image to obtain multiple segmentation results. The category of the segmentation results is determined, including: if the multi-operation state detection method for belt conveyors is applied to a tubular belt conveyor, the edge segmentation result of one side of the conveyor belt is obtained by segmentation through the segmentation network, the edge segmentation line and its corresponding equation expression in the mathematical geometric space are determined and output, and the edge segmentation line is the result obtained by linear fitting of the conveyor belt edge segmentation line obtained by the segmentation network.

[0053] Specifically, when the conveyor belt in the monitoring image is on a straight section of the conveyor, the fitting method is linear fitting; when the conveyor belt in the monitoring image is on a turning section of the conveyor, the fitting method is nonlinear fitting, usually spline fitting. When the multi-operation state detection method for belt conveyors is applied to tubular belt conveyors, due to the special structural form of the tubular belt conveyor, only one side of the conveyor belt edge exists in the camera's field of view. Similarly, the segmentation network can segment the conveyor belt edge and output the fitted edge line, obtaining its equation expression. Through the single-sided edge line, it can also be compared with the tubular belt conveyor during normal operation to further determine whether the tubular belt conveyor has experienced faults such as pipe twisting or deviation.

[0054] Step S4: Detect the load based on the material load area, and determine the conveyor belt deviation status based on the edge lines on both sides of the conveyor belt.

[0055] The load detection method primarily relies on the area ratio method, which determines the load level by comparing the proportion of the load area on the conveyor belt relative to the entire field of view. Using a segmentation network, pixels belonging to the material load can be segmented or identified from the image, their number counted, and then the ratio of this value to the total number of pixels in the image can be calculated, or the value can be calculated relative to the number of pixels belonging to the material load when the conveyor is fully loaded.

[0056] In some embodiments, detecting the load based on the material load area includes: determining the area of ​​the material load area, the area being determined based on the number of pixels; determining the proportion of the material load area occupying the monitoring image based on the area of ​​the material load area; and determining the load based on the proportion.

[0057] Specifically, in some embodiments, detecting the load based on the material load area includes: obtaining the area of ​​the fully loaded material load area when the conveyor is fully loaded, the area being determined based on the number of pixels; determining the area of ​​the current material load area; determining the area ratio between the current material load area and the fully loaded material load area; and determining the load based on the area ratio. The fully loaded material load area can be pre-defined by manually capturing and saving images of the fully loaded conveyor from a fixed camera, and then calculating and saving the area of ​​the fully loaded material load area to a database through annotation.

[0058] When determining the load, calculations can be performed by setting area ratio thresholds. A minimum area ratio threshold can be set; if the load falls below this threshold, the current load is considered too low, and a notification is sent to the management user. Conversely, a maximum area ratio threshold can be set; if the load exceeds this threshold, the current load is considered to have exceeded the safe conveyor transmission limit, and an alarm is promptly sent to the management user. Furthermore, the ratio thresholds can be modified according to actual operational needs, improving the flexibility of load detection.

[0059] Another segmentation target of the segmentation network is the edge of the conveyor belt. When the multi-operational state detection method for belt conveyors is applied to a common trough belt conveyor, its output object is both sides of the conveyor belt. When the multi-operational state detection method for belt conveyors is applied to a tubular belt conveyor, its output object is one side of the conveyor belt.

[0060] In some embodiments, determining the conveyor belt deviation state based on the two side edge lines of the conveyor belt includes: determining the offset and tilt state of the conveyor belt by using the equation expressions in the mathematical geometric space corresponding to the left and right edge lines of the conveyor belt.

[0061] In some embodiments, the offset and tilt state of the conveyor belt are determined by the equations in the mathematical geometric space corresponding to the left and right edge lines of the conveyor belt. This includes: if the multi-operational state detection method is applied to the monitoring of the straight section of the belt conveyor, the slope and position information of the edge dividing line are determined by the equation in the mathematical geometric space to determine the offset and tilt state of the conveyor belt; if the multi-operational state detection method is applied to the monitoring of the curved section of the belt conveyor, the position information or corresponding area of ​​the edge dividing line are determined by the equation in the mathematical geometric space to determine the offset and tilt state of the conveyor belt.

[0062] Specifically, when the edge segmentation line is a linear fit, the slope, intercept, and other parameters in the equation can be analyzed and compared with the corresponding parameters in the equation obtained when the conveyor is running normally to further determine the offset and tilt state of the conveyor belt. Specifically, when the multi-operational state detection method for belt conveyors is applied to the straight section of a common trough belt conveyor, the conveyor's offset state can be determined by the following method based on the equations of the conveyor belt's edge lines on both sides (assumed to be the left and right sides) obtained from the segmentation network: Based on the detected straight edge of the conveyor belt, a virtual reference line is established within the camera's field of view. This virtual reference line is parallel to the x-axis, and the distance between it and the x-axis is marked as Δy. The virtual reference line intersects the left and right edges of the camera's field of view at points L and R, respectively, and intersects the left and right edges of the conveyor belt at points M and N, respectively. Let LM be the line segment distance between points L and M, and RN be the line segment distance between points R and N. Calculate the lengths of LM and NR to determine the position of the conveyor belt within the camera's field of view. Calculate the slope and intercept of LM and NR in real time using an algorithm, compare them with the parameters under standard conditions, and set a threshold parameter to determine the belt deviation state and amount. That is, if the difference is less than the threshold, the conveyor belt is judged not to be deviating; if the difference is greater than the threshold, the conveyor belt is judged to be deviating, and an alarm message is provided to the monitoring user. In some scenarios, the camera cannot be centered in the middle of the conveyor belt and needs to be installed on one side of the conveyor. In this case, the threshold setting needs to be set according to the actual situation.

[0063] In some embodiments, determining the position information of the edge segmentation line or the corresponding area region, and determining the deviation state of the conveyor belt, includes: determining the intersection point of the identified inner edge curve of the conveyor belt and the diagonal of the monitoring image, and judging the deviation state based on the distance of the intersection point from the lower right corner vertex or the upper left corner vertex; or, determining the area region enclosed by the inner edge line of the conveyor belt and the image region, obtaining the size of the area region by definite integral, and judging the deviation state based on the area region.

[0064] When the edge segmentation line is a non-linear fit, i.e., used for conveyor belt edge line identification in turning sections, multiple points on the edge line can be labeled with coordinates. Using mathematical methods, such as the least squares method, an approximate functional expression between the variables can be obtained from these coordinates, and this expression can be compared in real time with the parameters under standard conditions. Turning sections tend to veer towards the turning center. Based on this characteristic, there are two specific measurement principles, such as... Figure 4 As shown, the conveyor belt deviation can be determined by calculating the intersection points of the left and right edge lines at the turning point with the top-left and bottom-right diagonal lines of the image, and the distances between the top-left and bottom-right corners of the image; alternatively, it can be determined by calculating the changes in the area enclosed by the identified edge lines and the image region. A deviation threshold is set; if the deviation exceeds the threshold, the conveyor belt at the turning point is considered to be deviating.

[0065] In some embodiments, determining the conveyor belt misalignment state based on the edge lines on both sides of the conveyor belt includes: if the multi-operational state detection method for belt conveyors is applied to a tubular belt conveyor, then by using the equation expression of the obtained edge segmentation line in mathematical geometric space, the slope and position information of the edge segmentation line are determined to determine the conveyor belt's twisting or misalignment state. Similarly, the segmentation network can segment the edge of one side of the conveyor belt and output the fitted edge line, obtaining the equation expression of its one-sided edge line. By analyzing the parameters such as the slope and intercept in the equation expression and comparing them with the corresponding parameters in the equation expression obtained during normal operation of the tubular belt conveyor, it is further determined whether the tubular belt conveyor has experienced faults such as twisting or misalignment.

[0066] The method disclosed herein detects conveyor edges using deep learning and integrates edge detection into a multi-operational state detection model for belt conveyors. This simplifies the extraction process of conveyor belt edge lines in complex environments and utilizes segmentation networks applicable to conveyor belt edge line recognition in turning sections, effectively ensuring the safe and efficient operation of belt conveyors.

[0067] Step S5: Obtain detection boxes for multi-class targets through the prediction network.

[0068] In some embodiments, the prediction network is a computational layer that connects feature maps of multiple scales. The prediction network makes predictions based on the feature maps and then merges the prediction results, enabling the network to adapt well to large, medium, and small targets in the surveillance image. Specifically, an 80×80×28 feature layer is used to predict small targets, a 20×20×512 feature layer is used to predict large targets, and a 40×40×256 feature layer is used to predict medium-sized targets.

[0069] Step S6: Based on the detection boxes of multiple target categories, obtain the detection results for conveyed foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveying line, smoke in the conveying area, and fire in the conveying area.

[0070] Specifically, the implementation of steps S5 and S6 depends on the prediction network. The output of the prediction network is the predicted bounding box for the above-mentioned multiple targets, which can also be called the detection box. The number of multiple target categories that the prediction network can detect depends on the target categories contained in the dataset before the network is deployed. Therefore, the more target categories contained in the dataset, the more target categories the prediction network can detect.

[0071] Furthermore, before deploying the network and performing segmentation and detection, it is necessary to train the network to obtain the corresponding weights. In some embodiments, such as... Figure 5As shown, the method further includes: step S7, for acquiring monitoring images of the conveyor operation; step S8, labeling the monitoring images to obtain multi-task labels, which include segmentation labels and detection labels, wherein the segmentation labels are the material load area and conveyor belt edge line in the monitoring image; the detection labels are conveyed foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveyor line, smoke in the monitoring area, fire in the monitoring area, and idlers on the left and right sides of the conveyor belt in the monitoring image; step S9, training a multi-operation state detection network model of the belt conveyor using monitoring images containing multi-task labels.

[0072] Because this disclosure utilizes a single model for multi-task training, and due to the specific nature of the detected targets, no dataset related to this disclosure was found in existing public datasets. Therefore, this disclosure constructs a completely new dataset using on-site data collection and manual annotation. All image data originates from monitoring videos of the conveyor's operating site, and data annotation was completed using Labelme (segmentation labels) and Labelimg (detection labels) software. The dataset contains a total of 4291 monitoring images and 18559 labels, including 4291 segmentation labels, 5699 labels for large conveyed foreign objects, and 8563 labels for conveyor belt edges. In one embodiment, a specific partial data distribution is as follows: Figure 6 As shown, some annotation styles are as follows: Figure 7 As shown. Due to the limited number of field data samples, this disclosure does not label conveyor belt damage, conveyor belt water accumulation, material spillage along the conveyor line, smoke in the monitoring area, fire in the monitoring area, and idlers on the left and right sides of the conveyor belt. However, this does not affect the detection effect of this disclosure. As mentioned before, the labeling method for these target categories is completely consistent with the labeling method for large conveyed foreign objects.

[0073] During data annotation, due to potential distortion in the monitoring images, the edges of the conveyor belt appearing in the images are not strictly straight. Therefore, during image segmentation and annotation, it is crucial that the two endpoints of the labels on the conveyor belt edge lines fall precisely on the conveyor belt edge itself. Figure 7 As shown.

[0074] In some embodiments, a multi-operational state detection network model for a belt conveyor is trained using monitoring images containing multi-task labels, including: the loss function of the multi-operational state detection network model for the belt conveyor consists of two parts: prediction loss and segmentation loss; the prediction loss consists of three parts: classification loss, position loss, and confidence loss, and the segmentation loss consists of classification loss.

[0075] Since the network disclosed herein includes both object detection and object segmentation tasks, the loss function of the improved network also comprises two parts. The object detection loss is calculated by a weighted sum of three losses: classification loss, object loss, and bounding box loss, as shown in the formula, where the loss function L... det By classifying loss L class Object loss L obj and bounding box loss L box get:

[0076]

[0077] Bounding box loss L box The cross-union ratio (IOU), also known as bounding box regression loss, is used to characterize the distance between the predicted box and the ground truth box. It is achieved by calculating the ratio of the intersection area to the union area of ​​the two bounding boxes. The larger the IOU, the higher the overlap between the predicted box and the ground truth box. The smaller the distance between them, the smaller the loss.

[0078] In some embodiments, transfer learning is used to train the multi-operational state detection model for a belt conveyor, and the pre-trained weights of the network are obtained from the original model trained on the COCO dataset. In this paper, the total number of training steps is set to 50, the batch size is set to 12, and gradient descent is performed using an annealed cosine learning rate. Simultaneously, online data augmentation strategies such as Mosaic and Cutmix are also used to enrich the background of the target and improve the network's generalization ability.

[0079] This disclosure uses Unet, Deeplabv3+, and Yolov5 networks to compare segmentation performance, while Faster R-CNN, Yolov4, and YolovX5 are used to compare object detection performance. However, as mentioned earlier, they can only perform one aspect of the task. The model in this disclosure is named DH, and the results include detection of conveyor belt edges, detection of large foreign objects, and segmentation of material flow. Figure 8 This invention demonstrates the processing results of different models in the segmentation task, and also shows the processing results of different models in the prediction task.

[0080] Prediction accuracy and prediction speed are the two most important metrics for evaluating object detection or segmentation networks. The accuracy of object detection is crucial for the safe operation of belt conveyors, especially for accurately identifying and locating large foreign objects or damage to the conveyor belt. Segmentation accuracy directly affects the quantitative measurement of real-time load, thus influencing the formulation of intelligent speed control strategies. Similarly, unreasonable speed control strategies will impact the drive motor, potentially leading to overload damage and accidents. Experiments show that the method disclosed in this paper, on a novel dataset constructed in this disclosure, can effectively achieve conveyor object detection, conveyor belt edge segmentation, and load detection, with a prediction or detection accuracy exceeding 97%. The fastest prediction speed on an RTX3080 reaches 90 frames per second. This invention more clearly demonstrates the results achieved by the method disclosed herein. It can unify the segmentation task and the prediction task through a single network and has the ability to perform multiple tasks, including load detection, conveyor belt misalignment detection, conveyor belt damage identification, conveyor belt water accumulation detection, material spillage detection along the conveyor line, smoke detection in the monitoring area, fire detection in the monitoring area, and detection of idlers on both sides of the conveyor belt.

[0081] Secondly, this invention also provides a deep learning-based multi-operational state detection device for belt conveyors. The device includes: an acquisition module for real-time acquisition of monitoring images of the conveyor operation; a feature extraction module for extracting features from the monitoring images using a feature extraction network, and inputting the features into a segmentation network and a prediction network respectively; a segmentation module for semantic segmentation of the monitoring images using the segmentation network to obtain the material load area and the edge lines on both sides of the conveyor belt; a load and conveyor belt detection module for detecting the load based on the material load area and determining the conveyor belt deviation state based on the edge lines on both sides of the conveyor belt; and a multi-category target detection module for obtaining detection boxes for multiple categories of targets through the prediction network; and obtaining detection results for conveyed foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveyor line, smoke in the conveyor area, and fire in the conveyor area based on the detection boxes of the multi-category targets.

[0082] The methods and apparatus disclosed herein can be implemented using standard programming techniques, and various method steps can be implemented using rule-based logic or other logic. It should also be noted that the terms "apparatus" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0083] Any step, operation, or procedure described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product comprising a computer-readable medium containing computer program code, which is executable by a computer processor to perform any or all of the described steps, operations, or procedures.

[0084] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0085] It can be further understood that, unless otherwise specified, "connection" includes a direct connection between two entities where no other components exist, an indirect connection between two entities where other components exist, and a communication connection that enables the transmission of information or data without a physical connection.

[0086] It is further understood that although operations are described in a specific order in the accompanying drawings in this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0087] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the corresponding claims.

[0088] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A deep learning-based method for detecting multiple operating states of a belt conveyor, characterized in that, The method includes: S1: Real-time acquisition of conveyor operation monitoring images, the monitoring images including the complete scene of the conveyor belt, material load and surrounding environment; S2: The feature extraction network is used to extract features from the surveillance image. The feature extraction network includes a convolutional module, a residual convolutional module, and a spatial pyramid pooling module. The convolutional module is composed of a convolutional layer, a batch normalization layer, and a loss function layer. The residual convolutional module is composed of the convolutional module, a residual layer, and a splicing layer. The second feature is determined by the first convolutional module and the first residual convolutional module in the feature extraction network. The first feature is determined by the second residual convolutional module, the third residual convolutional module, and the spatial pyramid pooling module. The first feature and the second feature are feature maps of different scales. S3: Input the features of the monitoring image into the segmentation network and the prediction network respectively; input the first feature into the feature enhancement network to obtain the first enhanced feature; input the second feature and the first enhanced feature into the segmentation network; input the first enhanced feature into the prediction network; use the segmentation network to perform semantic segmentation on the monitoring image to obtain the material load area and the edge lines on both sides of the conveyor belt; after fitting processing, the edge lines output the corresponding mathematical geometric space equation expression; linear fitting is used for monitoring straight sections, and nonlinear fitting is used for monitoring curved sections. S4: Determine the area ratio based on the number of pixels in the material load area, and then detect the load. Based on the equation expression of the edge lines on both sides of the conveyor belt, analyze the slope and position information or area region to determine the conveyor belt deviation status. In the case of tubular belt conveyors, determine the tube twist and deviation status by the edge line on one side. S5: Obtain detection boxes for multiple target categories through a prediction network. The prediction network is a computational layer that connects feature maps of multiple scales, and is adapted to the detection of large, medium and small targets respectively. Based on the detection boxes of the multiple target categories, obtain the detection results of conveying foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveying line, smoke in the conveying area, and fire in the conveying area.

2. The method for detecting multiple operating states of a belt conveyor according to claim 1, characterized in that: The segmentation network is used to perform semantic segmentation on the monitoring image, resulting in multiple segmentation results. These results include: responding to the current segmentation category belonging to the left edge of the conveyor belt, determining and outputting the segmentation result as the left edge line and its corresponding equation in mathematical geometric space, where the left edge line is the result obtained by fitting the left edge segmentation line of the conveyor belt obtained by the segmentation network; responding to the current segmentation category belonging to the right edge of the conveyor belt, determining and outputting the segmentation result as the right edge line and its corresponding equation in mathematical geometric space, where the right edge line is the result obtained by fitting the right edge segmentation line of the conveyor belt obtained by the segmentation network; and responding to the current segmentation category not belonging to the conveyor belt edge, determining and outputting the segmentation result as a material load area.

3. The method for detecting multiple operating states of a belt conveyor according to claim 2, characterized in that: The segmentation result also includes: if the multi-operational state detection method for belt conveyors is applied to tubular belt conveyors, then the edge segmentation result of one side of the conveyor belt is obtained by segmenting through the segmentation network, and the edge segmentation line and its corresponding equation expression in the mathematical geometric space are determined and output. The edge segmentation line is the result obtained by linear fitting of the conveyor belt edge segmentation line obtained by the segmentation network.

4. The method for detecting multiple operating states of a belt conveyor according to claim 1, characterized in that: Detecting the load based on the material load area includes: determining the area of ​​the material load area, the area being determined based on the number of pixels; determining the proportion of the material load area occupying the monitoring image based on the area of ​​the material load area; and determining the load based on the proportion.

5. The method for detecting multiple operating states of a belt conveyor according to claim 1, characterized in that, If the multi-operational state detection method is applied to the monitoring of the straight section of a belt conveyor, the slope and position information of the edge dividing line are determined by the equation expression of the obtained edge dividing line in mathematical geometric space, thereby determining the offset and tilt state of the conveyor belt; if the multi-operational state detection method is applied to the monitoring of the curved section of a belt conveyor, the position information or corresponding area of ​​the edge dividing line are determined by the equation expression of the obtained edge dividing line in mathematical geometric space, thereby determining the misalignment state of the conveyor belt.

6. The method for detecting multiple operating states of a belt conveyor according to claim 1, characterized in that: Determining the position information or corresponding area of ​​the edge segmentation line, and determining the deviation state of the conveyor belt, includes: determining the intersection point of the identified inner edge curve of the conveyor belt and the diagonal of the monitoring image, and judging the deviation state based on the distance of the intersection point from the lower right corner vertex or the upper left corner vertex; or, determining the area enclosed by the inner edge line of the conveyor belt and the image area, obtaining the size of the area area through definite integral, and judging the deviation state based on the area area.

7. The method for detecting multiple operating states of a belt conveyor according to claim 1, characterized in that: The monitoring images are labeled with multi-task tags, which include segmentation tags and detection tags. Specifically, the segmentation tags are the material load area and the edge line of the conveyor belt in the monitoring images; the detection tags are foreign objects being transported, conveyor belt damage, water accumulation on the conveyor belt, material spillage along the conveyor line, smoke in the monitored area, fire in the monitored area, and idlers on the left and right sides of the conveyor belt in the monitoring images. A multi-operational state detection network model for the belt conveyor is trained using monitoring images containing multi-task tags.

8. The method for detecting multiple operating states of a belt conveyor according to claim 7, characterized in that, The step of training a multi-operational state detection network model for a belt conveyor using monitoring images containing multi-task labels includes: the loss function of the multi-operational state detection network model for the belt conveyor consists of two parts: prediction loss and segmentation loss; the prediction loss consists of three parts: classification loss, position loss, and confidence loss; and the segmentation loss consists of classification loss.

9. A deep learning-based multi-operational state detection device for belt conveyors, characterized in that: The method described in claim 1 employs a multi-operational state detection device for a belt conveyor, comprising: a data acquisition module for real-time acquisition of monitoring images of the conveyor operation; a feature extraction module for extracting features from the monitoring images using a feature extraction network, and inputting the features into a segmentation network and a prediction network respectively; a segmentation module for performing semantic segmentation on the monitoring images using the segmentation network to obtain the material load area and the edge lines on both sides of the conveyor belt; a load and conveyor belt detection module for detecting the load based on the material load area and determining the conveyor belt deviation state based on the edge lines on both sides of the conveyor belt; and a multi-category target detection module for obtaining detection boxes for multiple categories of targets through a prediction network; and obtaining detection results for conveyed foreign objects, conveyor belt damage, conveyor belt water accumulation, material spillage along the conveyor line, smoke in the conveyor area, and fire in the conveyor area based on the detection boxes of the multiple categories of targets.

Images