Coal conveying belt deviation intelligent identification method based on deep semantic feature fusion

Abstract

The invention discloses a coal conveying belt deviation intelligent inspection system based on deep semantic feature fusion, and the system comprises collection units which are fixedly disposed at two sides of a belt, and are used for collecting videos; the intelligent inspection management platform is a Web front-end management platform used for service configuration, reference graph labeling configuration, video display, data query, result display and service alarm; the intelligent inspection service system is a background service system used for video stream pulling, video decoding and frame extraction, algorithm interface requesting and receiving, database addition, deletion, modification and checking and front-end calling interface returning; and the algorithm service system is used for receiving the system request parameters, carrying out image processing and logic judgment and returning an algorithm identification result.

Description

Technical Field

[0001] The present invention relates to an intelligent recognition method for deviation of a coal conveyor belt based on deep semantic feature fusion, which belongs to the field of intelligent inspection of coal conveyor belt transportation and is an application of computer vision and deep learning in the direction of traditional industrial inspection. Background Art

[0002] Chemical plants require a large amount of coal as raw material and coal conveyor belts as transportation equipment. Belt conveyors have various common problems in daily operation, the most typical of which is belt deviation. When the problem reaches a certain level, it will lead to various safety hazards, such as belt rollover, coal spillage, belt tearing, etc., causing system failures, safety problems or production stoppages.

[0003] However, the inventors of this application have found that the prior art has at least the following technical problems during the production process: At present, most chemical plants use manual inspections. Inspectors inspect equipment regularly and manually shut down the equipment for repairs when they find equipment failures. Manual inspections have a certain lag, making it difficult to detect and deal with them early. Some chemical plants use deviation switch sensors to monitor belt anomalies, but they have the disadvantages of unstable output, difficult maintenance, and high maintenance costs. Some chemical plants use fixed cameras and traditional computer vision technology to detect faults. They extract the edge line of the belt based on traditional image processing algorithms, and then calculate the distance between the edge line of the belt and the prior warning line to determine whether it is deviating. They have poor robustness in situations such as large changes in lighting and complex backgrounds, and are prone to missed detections and false detections.

[0004] Currently, regarding the automatic detection system for belt deviation, part of it is based on sensor monitoring, and part of it is based on image understanding and analysis.

[0005] The mainstream image understanding and analysis methods are mainly based on the detection of belt areas, using traditional edge detection, traditional segmentation or deep learning segmentation algorithms to extract the belt area, and then compare it with the pre-set false alarm line to determine whether the belt is running off track. Because the belt area itself has the problem of inaccurate extraction in uncertain and complex environments or viewing angles, the belt edge line extracted in image post-processing is also inaccurate, and the distance between the belt edge line and the pre-set false alarm line cannot be accurately measured under certain shooting perspective distortions, resulting in errors in the overall judgment of belt deviation. Summary of the invention

[0006] The purpose of the present invention is to provide an intelligent coal conveyor belt inspection system based on computer vision, deep learning and image processing technology, so as to realize the real-time and accurate fault detection and solve the defects of the existing system such as poor robustness, low safety and high labor cost.

[0007] The present application provides an intelligent inspection system for coal conveyor belt deviation based on deep semantic feature fusion, which is characterized by comprising: The acquisition unit is fixedly installed on both sides of the belt and is used to collect video; Intelligent inspection management platform, a Web front-end management platform for business configuration, reference diagram annotation configuration, video display, data query, result display and business alarm; Intelligent inspection service system, a backend service system for video streaming, video decoding and frame extraction, algorithm interface request and reception, database addition, deletion, modification and query, and front-end call interface return; The algorithm service system is used to receive the request parameters of the intelligent inspection service system, perform image processing, logical judgment and return the algorithm recognition result.

[0008] The present invention also discloses a method for intelligently identifying deviation of a coal conveyor belt based on deep semantic feature fusion, which is characterized by comprising the following steps: S1 can flexibly install a fixed-position HD camera above the belt according to the on-site environment; S2 obtains the video stream of the HD camera in real time through the network and RTSP protocol; S3 extracts a basic frame image from the high-definition video stream and performs parameter configuration on the Web front end; specifically, it includes: First, marking the boundary lines on the left and right sides of the belt running on the basic frame image; Then, select two rollers with clear dimensions on both sides of the belt and mark the center axis of each roller; The S4 server backend obtains the real-time HD video stream, decodes and extracts frames to obtain a picture sequence, and then calls the algorithm service system deployed on a high-performance GPU server for processing. S5 The algorithm service system performs preprocessing on the image, including image enhancement, image filtering, and removing the influence of lighting; S6 uses the semantic segmentation algorithm of basic deep learning to segment the pixel area of ​​the belt and the pixel area of ​​the roller; S7 performs morphological post-processing on the acquired belt pixel area, and obtains the belt edge straight line information by combining contour detection, edge detection, and belt main direction information; determines whether the belt is deeply deviated by the intersection of the belt area and the deviation boundary line; and tracks the belt deviation process and magnitude in real time by the distance between the belt edge straight line and the deviation boundary line; S8 performs morphological post-processing on the obtained roller pixel area to obtain multiple roller instance objects, and combines the position information of the four reference rollers annotated by the Web front end to obtain the exposure size and belt occlusion of the four reference rollers in the current time; S9 combines the belt logic of step S7 and the roller logic of step S8 to comprehensively determine whether the belt is deviating and accurately measure the degree of deviation in real time; The S10 algorithm service system returns the belt deviation mark and deviation measurement length information to the server. The server writes the returned information and related information into the database. The Web front end displays the belt deviation warning content in real time and sounds an alarm.

[0009] Preferably, in step S6, DeepLabV3Plus is selected as the framework for semantic segmentation of belts and rollers, the atrous space pyramid pooling ASPP architecture is applied to the processing of deep semantic features, and then the obtained deep features and shallow detail features are connected to obtain the final pixel category probability through a classifier.

[0010] Preferably, in step S7, the judgment logic of the belt deviation includes: 1) Based on the belt area and virtual boundary line, the number and length of cross pixels between the segmented belt pixel area and the belt virtual boundary line pixel area are calculated. If the number and length exceed the threshold, it is judged that the belt is seriously deviated; The PCA principal component analysis is used to calculate the running direction of the belt based on the pixel coordinates of the belt area. Ten anchor points of the boundary virtual line are evenly selected at equal distances. The distance from the anchor point to the belt area is calculated along the vertical direction of the belt running. Finally, different weights are given based on the distance of the anchor point in the scene to calculate the distance from the belt to the boundary virtual line. 2) Based on the change in the length of the center axis of the anchor roller, when the belt deviates to the left, the area of ​​the two rollers on the left that is blocked by the belt becomes larger, and the exposed roller area becomes smaller; at the same time, the area of ​​the two rollers on the right that is blocked by the belt becomes smaller, and the exposed roller area becomes larger; When the length of the center axis of the left roller is shortened by more than a threshold value and the length of the center axis of the right roller is increased by more than a threshold value, it can be determined that the belt is deviating to the left.

[0011] The technical solution of the belt deviation intelligent inspection system based on deep semantic feature fusion provided by this application has at least the following technical effects or advantages: 1) Compared with traditional manual inspection, it has the advantages of safety, real-time, stability and all-weather. 2) Compared with the deviation switch sensor, it is stable, intuitive and easy to maintain.

[0012] 3) Compared with other methods based on traditional computer vision, the coal conveyor belt deviation intelligent recognition method based on deep semantic feature fusion in the present invention can detect the belt area and smaller roller area more accurately, deeply understand the business scenario, integrate a variety of known information, and ingeniously design the business logic of belt deviation, thus realizing the accuracy, robustness, real-time and ease of operation of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a labeled example diagram of a reference diagram of a belt deviation inspection scene in an embodiment of the present invention; Figure 2 It is a workflow and system architecture diagram of the intelligent belt inspection system in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] This implementation example provides a coal conveyor belt intelligent inspection system based on deep semantic feature fusion, including: High-definition cameras, fixedly installed on both sides of the belt, are used to collect video; Intelligent inspection management platform, a Web front-end management platform for business configuration, reference diagram annotation configuration, video display, data query, result display and business alarm; Intelligent inspection service system, a backend service system for video streaming, video decoding and frame extraction, algorithm interface request and reception, database addition, deletion, modification and query, and front-end call interface return; The algorithm service system is used to receive the request parameters of the intelligent inspection service system, perform image processing, logical judgment and return the algorithm recognition results.

[0015] The present invention also discloses a method for intelligently identifying deviation of a coal conveyor belt based on deep semantic feature fusion, which is characterized by comprising the following steps: S1 can flexibly install a fixed-position HD camera above the belt according to the on-site environment; S2 obtains the video stream of the HD camera in real time through the network and RTSP protocol; S3 extracts a basic frame image from the HD video stream and configures parameters on the Web front end: First, mark the boundary lines on the left and right sides of the belt running on the basic frame image, such as Figure 1 Then select 4 clear rollers with larger scale, and place them on the left and right sides of the belt, such as Figure 1 For each roller, mark the center axis, such as the center axis 5 of roller 1 and the center axis 10 of roller 4.

[0016] The S4 server backend obtains the real-time high-definition video stream, decodes and extracts frames to obtain a picture sequence, and then calls the belt deviation recognition algorithm interface deployed on the high-performance GPU server for processing; The S5 algorithm service system first pre-processes the image, including image enhancement, image filtering, and removing the effects of lighting. S6 Then DeepLabV3Plus with the core architecture of atrous spatial pyramid pooling ASPP is used as the framework for semantic segmentation of belts and rollers, and is applied to the processing of deep semantic features. Then the obtained deep features are connected with the shallow detail features, and the final pixel category probability is obtained through the classifier to enhance the segmentation effectiveness of the belt pixel area and the roller pixel area; S7 performs morphological post-processing on the acquired belt pixel area, and obtains the belt edge straight line information by combining contour detection, edge detection, belt main direction and other information. The intersection of the belt area and the deviation boundary line can be used to determine whether the belt is deeply deviated; the distance between the belt edge straight line and the deviation boundary line can be used to track the belt deviation process and magnitude in real time; S8 Performing morphological post-processing on the obtained roller pixel area can obtain multiple roller instances. Combining the position information of the four reference rollers annotated by the Web front end, the exposure size and belt occlusion of the four reference rollers in the current time can be obtained. For example, roller No. 1 has been annotated with a rotating rectangular box for position information and a straight line for the center axis. The algorithm obtains the pixel distribution area of ​​roller No. 1. If the belt image is offset to the left, the belt will cover more rollers, the roller segmentation area will become smaller, the intersection area of ​​the center axis and the roller will become smaller, and the length of the center axis will become smaller; at the same time, the center axis of roller No. 3 becomes longer. In the process of the belt shifting to the left, the length of the center axis of roller No. 1 shows a downward trend, while the length of the center axis of roller No. 3 shows an upward trend. Based on this change process, it can be judged that the belt is deviating to the left; S9 combines S7 belt logic and S8 roller logic to comprehensively judge whether the belt is deviating and accurately measure the degree of deviation in real time; The S10 algorithm service system returns the belt deviation mark and deviation measurement length information to the server. The server writes the returned information and related information into the database. The Web front end displays the belt deviation warning content in real time and sounds an alarm.

[0017] This implementation method does not rely on a single identification target in the belt area, but simultaneously monitors the changes of 5 targets, including 4 rollers and 1 belt. By introducing the hollow space pyramid pooling ASPP network structure based on the DeepLabV3Plus semantic segmentation algorithm, the semantic segmentation accuracy of the recognition algorithm in multi-scale image perception is improved, thereby obtaining a more accurate pixel area of ​​5 targets. Then, the real-time offset parameters of the 5 targets are combined for judgment. Compared with the existing methods, the improved recognition algorithm has higher accuracy and stability, and can reflect the belt operation status through the real-time offset distance change curve.

[0018] The center line of the roller can more accurately measure the deviation of the belt than the edge line of the belt. The roller detection has an automatic error correction mechanism. The rollers on both sides of the belt will inevitably show opposite distance changes as the belt deviates. The detection of rollers is simpler and more accurate. During the operation of the belt equipment, the rollers are stationary, while the belt will deviate. The improved deep learning image segmentation algorithm has higher segmentation accuracy and faster speed than the previous algorithm. The comprehensive logic judgment based on multiple categories and multiple targets is more robust than the single belt target detection.

[0019] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention. These improvements should also be regarded as within the scope of protection of the present invention.

Claims

1. An intelligent inspection system for coal conveyor belt deviation based on deep semantic feature fusion, characterized in that: include: The acquisition unit is fixedly installed on both sides of the coal conveyor belt to collect video; Intelligent inspection management platform, a Web front-end management platform for business configuration, reference diagram annotation configuration, video display, data query, result display and business alarm; Intelligent inspection service system, a backend service system for video streaming, video decoding and frame extraction, algorithm interface request and reception, database addition, deletion, modification and query, and front-end call interface return; The algorithm service system is used to receive the request parameters of the intelligent inspection service system, perform image processing, logical judgment and return the algorithm recognition result.

2. An intelligent identification method for coal conveyor belt deviation based on deep semantic feature fusion, characterized in that: The following steps are involved: S1 Install a fixed-position HD camera above the belt according to the on-site environment; S2 obtains the camera's high-definition video stream in real time through the network and RTSP protocol; S3 extracts a basic frame image from the high-definition video stream and performs parameter configuration on the Web front end; specifically, it includes: First, marking the boundary lines on the left and right sides of the belt running on the basic frame image; Then, select two rollers with clear dimensions on both sides of the belt and mark the center axis of each roller; The S4 server backend obtains the real-time HD video stream, decodes and extracts frames to obtain a picture sequence, and then calls the algorithm service system deployed on a high-performance GPU server for processing; S5 The algorithm service system performs preprocessing on the image, including image enhancement, image filtering and removing the influence of lighting; S6 uses the semantic segmentation algorithm of basic deep learning to segment the pixel area of ​​the belt and the pixel area of ​​the roller; S7 performs morphological post-processing on the acquired belt pixel area, and obtains the belt edge straight line information by combining contour detection, edge detection and belt main direction information; determines whether the belt is deeply deviated by the intersection of the belt area and the deviation boundary line; and tracks the belt deviation process and amplitude in real time by the distance between the belt edge straight line and the deviation boundary line; S8: Perform morphological post-processing on the obtained roller pixel area to obtain multiple roller instance objects, and combine the position information of the four reference rollers annotated by the Web front end to obtain the exposure size and belt occlusion of the four reference rollers in the current time; S9 combines the belt logic of step S7 and the roller logic of step S8 to comprehensively determine whether the belt is deviating and accurately measure the degree of deviation in real time; The S10 algorithm service system returns the belt deviation mark and deviation measurement length information to the server. The server writes the return information of the identification result into the database. The Web front end displays the warning content of the belt deviation in real time and sounds an alarm.

3. The method for intelligently identifying deviation of coal conveyor belt based on deep semantic feature fusion according to claim 2 is characterized by: In step S6, DeepLabV3Plus is selected as the framework for semantic segmentation of belts and rollers, the hollow space pyramid pooling ASPP architecture is applied to the processing of deep semantic features, and then the obtained deep features and shallow detail features are connected to obtain the final pixel category probability through a classifier.

4. The method for intelligently identifying deviation of a coal conveyor belt based on deep learning according to claim 3 is characterized in that: The step S7 comprises: 1) Based on the belt area and virtual boundary line, the number and length of cross pixels between the segmented belt pixel area and the belt virtual boundary line pixel area are calculated. If the number and length exceed the threshold, it is judged that the belt is seriously deviated; The PCA principal component analysis is used to calculate the running direction of the belt based on the pixel coordinates of the belt area. Ten anchor points of the boundary virtual line are evenly selected at equal distances. The distance from the anchor point to the belt area is calculated along the vertical direction of the belt running. Finally, different weights are given based on the distance of the anchor point in the scene to calculate the distance from the belt to the boundary virtual line. 2) Based on the change in the length of the center axis of the anchor roller, when the belt deviates to the left, the area of ​​the two rollers on the left that is blocked by the belt becomes larger, and the exposed roller area becomes smaller; at the same time, the area of ​​the two rollers on the right that is blocked by the belt becomes smaller, and the exposed roller area becomes larger; When the length of the center axis of the left roller is shortened by more than a threshold value and the length of the center axis of the right roller is increased by more than a threshold value, it can be determined that the belt is deviating to the left.

Images