A coal belt deviation detection method and device based on machine vision

Abstract

A coal belt deviation detection method based on machine vision, the detection method comprising the following steps: adopting an intrinsically safe camera to perform image acquisition; performing image instance segmentation annotation on the acquired image, and training an instance segmentation model to obtain an instance segmentation result; performing structured feature extraction on the instance segmentation result, the structured feature being feature information of the belt and the roller in the image, including the area of the belt in the left and right parts of the image, the center point coordinates of the left and right edges of the belt, the number of rollers and the pixel area of different rollers; using XGBoost to establish an image classification model based on the feature vectors of the normal operation and deviation of the belt; performing online detection and recognition on the belt running image, the instance segmentation model and the image classification model to accurately judge whether the belt has deviated, if it is identified as deviating, issuing a warning and saving the image, otherwise continuing to detect.

Description

TECHNICAL FIELD

[0001] The application relates to a coal belt deviation detection method and device based on machine vision. BACKGROUND

[0002] Coal mining is to separate and transport valuable coal from underground or surface, whether open-pit mining or underground mining, first needs to carry out geological exploration, find out the distribution range of coal-bearing strata, mineable number of layers, layer thickness, dip angle, reserves, and geological structure, spontaneous combustion tendency, water, gas and other occurrence conditions and mining conditions, then reasonably plan the construction scale of the mining area, the number of mines, production and construction sequence, according to the overall design of the mining area and the design of the mine, one by one construction and then hand over to production.

[0003] In the process of coal transportation, mainly rely on the coal belt to operate, after a long time work will inevitably appear coal belt deviation phenomenon, belt deviation will affect the service life of the belt, reduce production efficiency, bring economic loss.

[0004] The existing matching deviation detection method mainly includes a detection method based on a photoelectric sensor, a detection method based on laser ranging and a detection method based on image processing; often a plurality of photoelectric switches are used for detection together, the overall detection cost is high, and meanwhile, the detection accuracy is low due to the influence of the dust under the mine; in recent years, the belt deviation detection method based on machine vision has been widely applied due to its simple deployment and non-contact measurement, a single feature of the belt in the image is selected for detection, although the implementation is easy, the robustness is poor, and the detection effect is poor in reliability if the belt edge feature is not obvious enough. SUMMARY

[0005] The application embodiment provides a coal belt deviation detection method and device based on machine vision, the structure and method design are reasonable, a multi-element fusion feature expression mode is used, and a plurality of elements related to the belt deviation are expressed in a structured feature fusion mode, compared with a single evaluation and detection mode in the prior art, noise interference can be effectively reduced to improve the detection accuracy, the detection is stable and reliable, has high robustness, and solves the problems in the prior art.

[0006] The technical scheme adopted by the application to solve the above technical problems is:

[0007] A coal belt deviation detection method based on machine vision, the detection method comprises the following steps:

[0008] S1, image acquisition is performed by using an intrinsically safe camera, the intrinsically safe camera is arranged to capture the belt running image in the direction of the belt running direction, the field of view of the image includes the belt and the rollers on both sides, and the belt running image includes two categories of normal belt running and belt deviation;

[0009] S2, image instance segmentation labeling is performed on the collected image, an instance segmentation model is trained, and an instance segmentation result is obtained, the instance segmentation model is a YolactEdge model, can run at real-time speed on a small edge device, and the belt and the roller in the image are segmented out by using instance segmentation;

[0010] S3, structured feature extraction is performed on the instance segmentation result, the structured feature is the feature information of the belt and the roller in the image, including the area of the belt in the left and right parts of the image, the center point coordinates of the left and right edges of the belt, the number of rollers and the pixel area of different rollers;

[0011] S4, an image classification model is established based on the feature vector of the normal belt running and the deviation of the belt by using XGBoost;

[0012] S5, online detection and identification are performed on the belt running image, the instance segmentation model and the image classification model to accurately judge whether the belt deviates or not, if the identification is deviation, a warning is issued and the image is saved, otherwise the detection is continued.

[0013] The conveying belt and the six rollers are segmented in the belt running image and are marked by different colors.

[0014] The structured feature extraction on the instance segmentation result includes the following steps:

[0015] S3.1, the belt running image is equally divided into left and right part sub-images, and the features of the belt are extracted in the left and right sub-images respectively;

[0016] S3.2, the pixel area of the belt in the left sub-image is extracted, denoted as S L , the center point coordinates of the edge of the belt in the left sub-image are extracted, denoted as X L , Y L , the pixel area of the belt in the right sub-image is extracted, denoted as S R , and the center point coordinates of the edge of the belt in the right sub-image are extracted, denoted as X R , Y R ;

[0017] S3.3, set the ROI region so that the ROI region contains an equal number of rollers on both sides of the belt, detect the number of specified rollers in the image and the corresponding pixel area, divide the six rollers into the target for statistics, and count the number N and the corresponding pixel area S of the six rollers i , i∈[1, 6];

[0018] S3.4, the belt running image is characterized as a feature vector with a length of 13, that is, [S L , X L , Y L , S R , X R , Y R , N, S1, S2, S3, S4, S5, S6].

[0019] The detection method comprises an offline training link and an online detection link, and the offline training link and the online detection link are synchronously matched.

[0020] A detection device using the above machine vision-based coal belt deviation detection method, the detection device comprising: an image acquisition component, a communication transmission component, a data processing component and a data storage component;

[0021] The image acquisition component is an intrinsically safe camera in a coal mine, which can collect high-definition images during the running of the transmission belt; a dustproof material is used on the surface of the intrinsically safe camera to avoid the attachment of dust in the mine to affect the imaging quality of the image;

[0022] The communication transmission component comprises a 5G transmission device, an industrial wifi transmission device and a USB3.0 Ethernet transmission device;

[0023] The data processing component is an edge processing device, which can complete image instance segmentation based on the YolactEdge model and image classification based on XGBoost;

[0024] The data storage component is an SD card or a hard disk, which is used to store images for identifying belt deviation.

[0025] The edge processing device is an AI edge computing server, which has a certain computing power and memory, and can complete image instance segmentation and image recognition classification.

[0026] The application adopts the above structure, image acquisition is performed through the intrinsic camera, image data basis is provided for the deviation detection of the coal conveying belt, image instance segmentation labeling is performed on the collected image, an instance segmentation model is trained, and an instance segmentation result is obtained, structured features are extracted from the instance segmentation result, the structured features are feature information of the belt and the roller in the image, an image classification model is established based on the feature vectors of the normal operation and deviation of the belt through XGBoost, online detection and identification are performed on the belt operation image, the instance segmentation model and the image classification model, and whether the belt appears the deviation phenomenon is accurately judged, and the application has the advantages of stability and reliability, practicality and simplicity. BRIEF DESCRIPTION OF DRAWINGS

[0027] Fig. 1 It is an implementation flowchart of the application.

[0028] Fig. 2 It is a device schematic diagram of the application. DETAILED DESCRIPTION

[0029] To clearly illustrate the technical features of the application, the application will be described in detail below through specific embodiments and in combination with the accompanying drawings.

[0030] As shown in Figs. 1-2 A coal conveying belt deviation detection method based on machine vision, the detection method comprises the following steps:

[0031] S1, an intrinsic camera is used for image acquisition, the intrinsic camera is arranged above the belt running direction and downwardly arranged to collect the belt running image, the field of view of the image includes the belt and the rollers on both sides, and the belt running image includes two categories of normal operation of the belt and deviation of the belt;

[0032] S2, image instance segmentation labeling is performed on the collected image, an instance segmentation model is trained, and an instance segmentation result is obtained, the instance segmentation model is a YolactEdge model, can run at real-time speed on a small edge device, and the belt and the roller in the image are segmented out through instance segmentation;

[0033] S3, structured features are extracted from the instance segmentation result, the structured features are feature information of the belt and the roller in the image, including the area of the belt in the left and right parts of the image, the center point coordinates of the left and right edges of the belt, the number of the rollers and the pixel area of different rollers;

[0034] S4, an image classification model is established based on the feature vectors of the normal operation and deviation of the belt through XGBoost;

[0035] S5, online detection and identification are performed on the belt running image, the instance segmentation model and the image classification model to accurately determine whether the belt running image is out of alignment, if the identification result is that the belt is out of alignment, a warning is issued and the image is saved, otherwise the detection is continued.

[0036] The transmission belt and the six rollers are segmented in the belt running image and are marked with different colors.

[0037] The structured feature extraction on the instance segmentation result comprises the following steps:

[0038] S3.1, the belt running image is equally divided into left and right sub-images, and the features of the belt are extracted in the left and right sub-images respectively;

[0039] S3.2, the pixel area of the belt in the left sub-image is extracted, denoted as S L , the center point coordinates of the edge of the belt in the left sub-image are extracted, denoted as X L , Y L ; the pixel area of the belt in the right sub-image is extracted, denoted as S R , the center point coordinates of the edge of the belt in the right sub-image are extracted, denoted as X R , Y R ;

[0040] S3.3, the ROI region is set so that the ROI region contains an equal number of rollers on both sides of the belt, the number of specified rollers in the image and the corresponding pixel area are detected, the six segmented rollers are taken as the target for statistics, the number N and the corresponding pixel area S i of the six rollers are counted, i∈[1,6];

[0041] S3.4, the belt running image is represented as a feature vector with a length of 13, namely [S L , X L , Y L , S R , X R , Y R , N, S1, S2, S3, S4, S5, S6].

[0042] The detection method comprises an offline training link and an online detection link, and the offline training link and the online detection link are synchronously matched.

[0043] A detection device using the above machine vision-based coal belt running deviation detection method, the detection device comprises an image acquisition component, a communication transmission component, a data processing component and a data storage component.

[0044] The image acquisition component is an intrinsically safe camera in a coal mine, which can collect high-definition images during the operation of the conveying belt; dust-proof material is used on the surface of the intrinsically safe camera to avoid the adhesion of dust in the mine to affect the imaging quality of the image;

[0045] The communication transmission component includes a 5G transmission device, an industrial wifi transmission device and a USB3.0 Ethernet transmission device;

[0046] The data processing component is an edge processing device, which can complete image instance segmentation based on the YolactEdge model and image classification based on XGBoost;

[0047] The data storage component is an SD card or a hard disk, which is used to store images for identifying belt deviation.

[0048] The edge processing device is an AI edge computing server, which has a certain computing power and memory, and can complete image instance segmentation and image recognition classification.

[0049] The working principle of the coal belt deviation detection method and device based on machine vision in the embodiment of the application is that a multi-element fusion feature expression method is used, and the multi-element related to the belt deviation is expressed by structural feature fusion, compared with the single evaluation and detection method in the prior art, the detection accuracy can be effectively improved by reducing noise interference, the detection is stable and reliable, and has high robustness.

[0050] Further, the intrinsically safe camera is used to pre-collect images of the normal operation of the belt and the belt deviation, and an image data set is established; the instance segmentation model is trained by using the images in the image data set, and the belt and the roller are two types of targets in the segmented images.

[0051] After instance segmentation, the structural features of the targets are extracted, and the target information in the image is expressed as a feature vector; the feature vectors of the normal operation of the belt and the belt deviation image are trained, and a classifier is established by using the xgboost model; after completing the offline training of the model, the images collected by the camera in real time are input into the instance segmentation model, and the segmentation results are expressed by structural features. The obtained feature vectors are input into the classification model for deviation identification; if the belt deviation is identified, a warning is issued and the image is stored for further training of the model; otherwise, the monitoring continues.

[0052] In the overall scheme, the detection method includes the following steps:

[0053] S1, an intrinsically safe camera is used for image acquisition, the intrinsically safe camera is arranged above the belt running direction and downwardly to collect the belt running image, the field of view of the image includes the belt and the rollers on both sides, and the belt running image includes two categories of normal operation of the belt and belt deviation;

[0054] S2, image instance segmentation labeling is performed on the collected image, an instance segmentation model is trained, and an instance segmentation result is obtained, the instance segmentation model is a YolactEdge model, can run at real-time speed on a small edge device, adopts instance segmentation, and separates out a belt and a roller in an image;

[0055] S3, structured feature extraction is performed on the instance segmentation result, the structured feature is feature information of the belt and the roller in the image, and includes an area of the belt in left and right parts of the image, center point coordinates of left and right edges of the belt, a number of the rollers, and pixel areas of different rollers;

[0056] S4, an image classification model is established based on feature vectors of normal operation and deviation of the belt by using XGBoost;

[0057] S5, online detection and identification are performed on the belt operation image, the instance segmentation model and the image classification model, to accurately determine whether the belt deviates, if the identification is that the belt deviates, a warning is sent and the image is saved, otherwise, detection is continuously performed.

[0058] In the application, the overall detection method can be divided into an offline training link and an online detection link, and the two are implemented synchronously and cooperatively to accurately identify and determine the deviation of the coal conveying belt.

[0059] On the other hand, the corresponding detection device mainly includes an image acquisition assembly, a communication transmission assembly, a data processing assembly and a data storage assembly;

[0060] The image acquisition assembly is an intrinsically safe camera in a coal mine, which can acquire high-definition images in the running process of the conveying belt; a dustproof material is used on the surface of the intrinsically safe camera to avoid the attachment of dust in the mine to affect the imaging quality of the image;

[0061] The communication transmission assembly includes a 5G transmission device, an industrial wifi transmission device and a USB3.0 Ethernet transmission device;

[0062] The data processing assembly is an edge processing device, which can complete image instance segmentation based on the YolactEdge model and image classification based on XGBoost;

[0063] The data storage assembly is an SD card or a hard disk, used for storing images of identified belt deviation.

[0064] Further, the edge processing device is an AI edge computing server, which has a certain computing power and memory, and can complete image instance segmentation and image recognition and classification.

[0065] The structured feature extraction on the instance segmentation result in the application is a core operation, specifically including the following steps:

[0066] S3.1, the belt running image is evenly divided into left and right two part images, and the features of the belt are extracted in the left and right sub-images respectively;

[0067] S3.2, the pixel area of the belt in the left sub-image is extracted, denoted as S L , the center point coordinates of the edge of the belt in the left sub-image are extracted, denoted as X L , Y L ; the pixel area of the belt in the right sub-image is extracted, denoted as S R , the center point coordinates of the edge of the belt in the right sub-image are extracted, denoted as X R , Y R ;

[0068] S3.3, set the ROI region, so that the ROI region contains an equal number of rollers on both sides of the belt, detect the number of specified rollers in the image and the corresponding pixel area, divide the six rollers into the target for statistics, and count the number N and the corresponding pixel area S i of the six rollers, i∈[1, 6];

[0069] S3.4, the belt running image is represented as a feature vector with a length of 13, that is, [S L , X L , Y L , S R , X R , Y R , N, S1, S2, S3, S4, S5, S6].

[0070] It is particularly pointed out that no matter which neural network model is used to realize instance segmentation of the collected image in the application, the essence of the scheme is not affected; no matter which machine learning method is used to realize classification, such as Logistic regression, random forest, standard gradient boosting, etc., the essence of the scheme is not affected; therefore, the application can be applied to various different application scenarios, facilitating popularization and promotion.

[0071] In summary, the coal belt deviation detection method and device based on machine vision in the embodiment of the application adopts a multi-element fusion feature expression method, and the multi-elements related to the belt deviation are expressed by structured feature fusion, which can effectively reduce noise interference and improve detection accuracy compared with the single evaluation and detection method in the prior art, and the detection is stable and reliable, and has high robustness.

[0072] The above detailed description cannot be considered as a limitation to the scope of the protection of the present application, but any modification or equivalent substitute for any part of the present application, which falls within the scope of the present application, is considered to be within the scope of the protection of the present application.

[0073] The present application is not detailed in the above, which is the known technology of the skilled in the art.

Claims

1. A coal belt deviation detection method based on machine vision, characterized in that, The detection method comprises the following steps: S1, using an intrinsically safe camera to collect images, the intrinsically safe camera collects belt running images above and below the belt running direction respectively, the field of view of the image includes the belt and the rollers on both sides, and the belt running image includes two categories of normal belt running and belt deviation; S2, image instance segmentation labeling is performed on the collected images, an instance segmentation model is trained, and an instance segmentation result is obtained, the instance segmentation model is a YolactEdge model, can run at real-time speed on a small edge device, uses instance segmentation to segment the belt and the rollers in the image; S3, structured feature extraction is performed on the instance segmentation result, the structured feature is the feature information of the belt and the rollers in the image, including the area of the belt in the left and right parts of the image, the center point coordinates of the left and right edges of the belt, the number of rollers and the pixel area of different rollers; S4, an image classification model is established based on the feature vectors of normal belt running and deviation using XGBoost; S5, online detection and identification are performed on the belt running image, the instance segmentation model and the image classification model to accurately determine whether the belt deviates, if the identification is deviation, a warning is issued and the image is saved, otherwise the detection continues.

2. The coal belt deviation detection method based on machine vision according to claim 1, characterized in that: The transmission belt and six rollers are segmented in the belt running image and marked with different colors.

3. The coal belt deviation detection method based on machine vision according to claim 1, characterized in that, The structured feature extraction on the instance segmentation result comprises the following steps: S3.1, the belt running image is equally divided into left and right part sub-images, and the features of the belt are extracted in the left and right sub-images respectively; S3.2, the pixel area of the belt in the left sub-image is extracted, denoted as S L ; the center point coordinates of the edge of the belt in the left sub-image are extracted, denoted as X L , Y L ; the pixel area of the belt in the right sub-image is extracted, denoted as S R ; the center point coordinates of the edge of the belt in the right sub-image are extracted, denoted as X R , Y R ; S3.3, set the ROI region, so that the ROI region contains an equal amount of rollers on both sides of the belt, detect the number of specified rollers in the image and the corresponding pixel area, the six rollers segmented as the target for statistics, and the number N and the corresponding pixel area S of the six rollers appearing are counted i , i ∈ [1, 6]; S3.4, the belt run image is characterized as a feature vector of length 13, i.e. [S L ,X L ,Y L ,S R ,X R ,Y R ,N, S1, S2, S3, S4, S5, S6].

4. The coal belt deviation detection method based on machine vision according to claim 1, characterized in that: The detection method comprises an offline training link and an online detection link, and the offline training link and the online detection link are synchronously matched.

5. A detection apparatus using the machine vision-based coal belt deviation detection method according to any one of claims 1 to 4, characterized by The detection device comprises an image collection component, a communication transmission component, a data processing component and a data storage component; The image collection component is an intrinsically safe camera in a coal mine, which can collect high-definition images during the running of the transmission belt; a dustproof material is used on the surface of the intrinsically safe camera to avoid the attachment of dust in the mine to affect the imaging quality of the image; The communication transmission component comprises a 5G transmission device, an industrial wifi transmission device and a USB3.0 Ethernet transmission device; The data processing component is an edge processing device, which can complete image instance segmentation based on the YolactEdge model and image classification based on XGBoost; The data storage component is an SD card or a hard disk, which is used to store the images of the identified belt deviation.

6. The detection device using the machine vision-based coal belt deviation detection method according to any one of claims 1-4, characterized in that: The edge processing device is an AI edge computing server, which has a certain computing power and memory, and can complete image instance segmentation and image recognition classification.

Images