A kind of open coal mine smoke detection method and device based on M-ST-YOLOv5 model

Abstract

The present application relates to the technical field of smoke identification, in particular to an open coal mine smoke detection method and device based on M-ST-YOLOv5 model; the present application provides a smoke detection algorithm M-ST-YOLOv5 under the open coal mine scene based on YOLOv5.The method of the present application firstly uses a camera to collect a video, then the obtained video data is cut into images at a fixed interval, then the smoke target area in the image is labeled using a labeling tool, then the labeled image data is input into the M-ST-YOLOv5 network for training, and finally the training weight is obtained.For the smoke target image to be processed, the training weight is used for prediction, and finally the smoke prediction result is output; the parameter quantity and the operation quantity are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smoke identification, in particular to an open-pit coal mine smoke detection method and device based on an M-ST-YOLOv5 model. BACKGROUND

[0002] With the continuous expansion of the coal industry, heavy dump trucks are currently used for transportation in large open-pit mines. The transportation roads are generally unconstructed dirt roads with poor road flatness, and road dust cannot be avoided. Not only does it pollute the surrounding environment and harm the health of workers, but it also shortens the service life of vehicles. In addition, dust affects the visibility of vehicle drivers, affects road safety, and reduces production efficiency. At the same time, after coal mining, an air interface is formed, which, after fully contacting with air, reaches the ignition point and locally self-ignites, causing loss of coal resources, ground subsidence, and emission of large amounts of harmful gases such as carbon monoxide and sulfur dioxide into the air, causing great pollution to the environment. Therefore, it is necessary to spray water to remove dust from the mine area.

[0003] A common method for detecting smoke is to use YOLOv5 to identify and detect smoke. However, it has a large number of parameters and a large amount of GFLOPs. Based on this, the present application proposes an open-pit coal mine smoke detection method and device based on an M-ST-YOLOv5 model. SUMMARY

[0004] The present application provides an open-pit coal mine smoke detection method and device based on an M-ST-YOLOv5 model, aiming to reduce the number of model parameters and the amount of GFLOPs when identifying smoke, and improve the recognition rate of smoke detection.

[0005] The first aspect of the present application aims to provide an open-pit coal mine smoke detection method based on an M-ST-YOLOv5 model, comprising the following steps:

[0006] Obtaining a target image of the open-pit coal mine smoke to be detected;

[0007] Inputting the image into the M-ST-YOLOv5 model for open-pit coal mine smoke detection and identification;

[0008] Outputting the detection result of the open-pit coal mine smoke.

[0009] Further, the M-ST-YOLOv5 model construction method comprises:

[0010] Step 1, collecting open-pit coal mine smoke image data;

[0011] Step 2, processing the image data of step 1 to construct a sample data set of open-pit coal mine smoke with open-pit coal mine smoke images and corresponding labels;

[0012] Step 3, build the M-ST-YOLOv5 model, and input the sample data set into the M-ST-YOLOv5 network for training; obtain the training weight parameters suitable for the open-pit coal mine dust detection, and obtain the M-ST-YOLOv5 model with open-pit coal mine dust detection.

[0013] Further, the M-ST-YOLOv5 model construction method comprises the following steps:

[0014] Step 2.1, the video is intercepted at a fixed interval frame, for example, converted into picture data at 25 frames per second;

[0015] Step 2.2, using LabelImg tool to label the open-pit coal mine dust image in step 2.1, and obtaining XML file corresponding to the image sample as a sample;

[0016] Step 2.3, divide the data set; the image data set obtained in step 2.2 is divided into training set (train), validation set (val) and test set (test) according to the proportion of 7:2:1

[0017] Step 2.4, adjust the data set format; convert the XML file of the training set (train), the validation set (val) and the test set (test) obtained by labeling into TXT file; construct the sample data set of open-pit coal mine dust with open-pit coal mine dust image and corresponding label.

[0018] All open-pit coal mine dust images obtained in step 2.1 are constructed, and new images are obtained by data enhancement, so as to expand the number of data sets; the data enhancement method includes any one or a combination of multiple combinations of translation, flip, rotation, noise addition and scaling.

[0019] Further, the M-ST-YOLOv5 model is improved compared with the YOLOv5 model, and the specific way is as follows:

[0020] Replace the CBS structure of the 1st, 3rd, 5th, 7th, 18th, 21st layer with MobileConv structure;

[0021] Replace the C3 structure of the 2nd, 4th, 6th, 13th, 17th, 20th, 23rd layer with MobileC3 structure;

[0022] Replace the C3 structure of the 8th layer with C3ST.

[0023] Furthermore, MobileConv's first layer is a CBS structure with a 1×1 kernel size, increasing the number of input channels c1 to the number of output channels c2; the second layer is a CBS structure using grouped convolution, with the number of groups g equal to the number of channels c2; the third layer uses a 1×1 kernel size convolution, and finally a normalization layer BN; the grouped convolution CBS structure used in the second layer of MobileConv can reduce network parameters, and the convolution in the third layer can enhance the mixing of information between channels.

[0024] Furthermore, the Bottleneck in C3 consists of a CBS structure with a 1×1 kernel and a 3×3 kernel; MobileC3 replaces the second CBS structure in the Bottleneck with a grouped convolutional CBS structure and adds a 1×1 convolutional kernel and a normalization layer (BN).

[0025] Furthermore, the Bottleneck in the C3 structure of layer 8 is replaced by the Swin-Transformer Block structure, which includes window multi-head self-attention (W-MSA) and shift window multi-head self-attention (SW-MSA).

[0026] Furthermore, Multi-head Self-Attention (W-MSA) divides the image into multiple windows for self-attention to reduce computational cost; Shift-Window Multi-head Self-Attention (SW-MSA) uses a shift-window approach to enhance information exchange between windows and improve the network's ability to extract information.

[0027] A second aspect of the present invention is to provide an open-pit coal mine dust detection device, characterized in that the device comprises:

[0028] The smoke and dust image acquisition module is configured to acquire target images of smoke and dust from open-pit coal mines to be detected.

[0029] The detection and recognition module is configured to input images into the M-ST-YOLOv5 model for open-pit coal mine dust detection and recognition.

[0030] The results output module is configured to output the detection results of dust from open-pit coal mines.

[0031] The beneficial effects achieved by this invention are as follows: This invention provides a smoke and dust detection algorithm, M-ST-YOLOv5, based on YOLOv5, for open-pit coal mine scenarios. The method first uses a camera to capture video, then extracts images from the acquired video data at fixed intervals. Next, the smoke and dust target regions in the images are labeled using a labeling tool. The labeled image data is then input into the M-ST-YOLOv5 network for training, finally obtaining training weights. For the smoke and dust target image to be processed, the training weights are used to predict its accuracy, and the smoke and dust prediction result is output. M-ST-YOLOv5 improves the accuracy (P) of smoke and dust detection by 1.6 percentage points and the recall (R) by 1.4 percentage points, while reducing the number of parameters by 30% and the computational cost (GFLOPs) by one-third. This invention has fewer network parameters, is suitable for practical deployment, and achieves high accuracy in smoke and dust detection. Attached Figure Description

[0032] Figure 1 This is a flowchart of the open-pit coal mine dust detection method of the present invention.

[0033] Figure 2 This is a flowchart of the M-ST-YOLOv5 model construction method of the present invention.

[0034] Figure 3 This is a schematic diagram of two adjacent images extracted by interval frames according to the present invention.

[0035] Figure 4 This is the labelImg annotation diagram used in this invention.

[0036] Figure 5 This is the present invention. Figure 4 The corresponding XML file diagram.

[0037] Figure 6 This is a schematic diagram illustrating an example of smoke and dust image data enhancement according to the present invention.

[0038] Figure 7 This is the invention Figure 3 The corresponding TXT file image.

[0039] Figure 8 This is a structural diagram of the M-ST-YOLOv5 model of the present invention.

[0040] Figure 9 This is a comparison diagram of the CBS structure and the MobileConv structure of this invention.

[0041] Figure 10 This is a comparison diagram of the C3 structure and the MobileC3 structure of the present invention.

[0042] Figure 11 This is a structural diagram of the C3ST of the present invention.

[0043] Figure 12 is a Swin-Transformer Block structure of the present application.

[0044] Figure 13 is a YOLOv5 and M-ST-YOLOv5 prediction result comparison chart of the present application. DETAILED DESCRIPTION

[0045] In order for those skilled in the art to understand the present application, the specific embodiments of the present application will be described below with reference to the accompanying drawings.

[0046] A kind of open coal mine smoke detection method based on M-ST-YOLOv5 model, as shown in Figure Figure 1 , comprising the following steps:

[0047] Obtain the target image of the open coal mine smoke to be detected;

[0048] The image is input into M-ST-YOLOv5 model, and open coal mine smoke detection identification is carried out;

[0049] Output the detection result of the open coal mine smoke.

[0050] As shown in Figure Figure 2 , the M-ST-YOLOv5 model construction method comprises:

[0051] Step 1, collect open coal mine smoke image data;

[0052] Set up a camera in the open coal mine, and collect videos containing smoke generated by coal spontaneous combustion and dust generated by coal yard production activities (such as vehicle driving).

[0053] Step 2, process the video to form a trainable sample data set: label the smoke target in the open coal mine smoke image as a label, and construct the sample data set of the open coal mine smoke with the open coal mine smoke image and the corresponding label;

[0054] Specifically, the trainable sample data set is obtained by the following method:

[0055] Step 2.1, the video is intercepted according to fixed interval frame, and the frame rate can be set by oneself, but it is necessary to ensure that the intercepted images have difference, for example, convert picture data at 25 frames per second;As shown in Figure Figure 3 , two adjacent open coal mine smoke images intercepted according to interval frame are shown;

[0056] Step 2.2, use LabelImg tool to label the smoke target of the open coal mine smoke image intercepted in step 2.1, and obtain XML file corresponding to the image sample as a sample; Figure 4For the smoke image being labeled, the box is the smoke target area, and the label is "Smoke&Dust"; the labeled information will be saved to the XML file corresponding to the image name, Figure 4 The Figure 5 corresponding XML file.

[0057] Step 2.3, divide the data set; the image data set obtained in step 2.2 is divided into a training set (train), a validation set (val), and a test set (test) in a ratio of 7:2:1, and the data of the three sets do not overlap. If there is data enhancement data, all of them are put into the training set (train). The training set is used to train the network, the validation set is used to adjust the hyperparameters (such as learning rate) in the network, and the test set is used to test the performance of the network;

[0058] Step 2.4, adjust the data set format; according to the requirements of M-ST-YOLOv5, the XML files of the training set (train), the validation set (val), and the test set (test) obtained by labeling are converted into TXT files; open-pit coal dust images and corresponding labels are used to construct a sample data set of open-pit coal dust; Figure 6 For example of TXT file, the first number is the category (Smoke&Dust), and the four data after it are the positions of the annotation box relative to the whole image.

[0059] If the number of data sets is small, it may cause overfitting of the training result, so all open-pit coal dust images obtained in step 2.1 can be used to perform data enhancement on the images to obtain new images, thereby expanding the number of data sets; the data enhancement method includes any one or a combination of multiple methods such as translation, flipping, rotation, adding noise, and scaling; Figure 7 The original image and the data enhanced image are shown, and the method used for the image is horizontal flipping. The data enhanced data also needs to be labeled.

[0060] Step 3: build M-ST-YOLOv5 model, input the sample data set in step two into M-ST-YOLOv5 network for training; obtain the training weight parameters suitable for open-pit coal dust detection, and obtain the M-ST-YOLOv5 model with open-pit coal dust detection.

[0061] M-ST-YOLOv5 is a network improved based on YOLOv5, and the specific structure is as follows Figure 8As shown: the Backbone part uses convolution to extract features from the input image; the Neck part uses Upsample to upsample the feature map and uses Concat to splice the feature maps of the same size to perform feature fusion; the Head part sets three scales to detect targets of different sizes; the specific way in which the M-ST-YOLOv5 model improves the YOLOv5 model is as follows:

[0062] (1) First, except for the 0 / 10 / 14 layers, replace all CBS structures with MobileConv structures; specifically, replace the CBS structures of layers 1, 3, 5, 7, 18, and 21 with MobileConv structures; the purpose of this is to reduce network parameters and running volume, and to reduce the difficulty of practical deployment;

[0063] (2) Second, replace all C3 structures with MobileC3 structures; specifically, replace the C3 structures of layers 2, 4, 6, 13, 17, 20, and 23 with MobileC3 structures; the purpose of this is to reduce network parameters and running volume, and to reduce the difficulty of practical deployment

[0064] (3) Third, replace the C3 structure of layer 8 with C3ST; this can improve the network's ability to recognize smoke targets without significantly increasing network parameters.

[0065] The comparison between CBS structure and MobileConv structure is shown in Figure 9 . Figure 9 The left side is the CBS structure of the YOLOv5 model, Figure 9 and the right side is the MobileConv structure of the M-ST-YOLOv5 model; the first layer of MobileConv is a CBS structure with a convolution kernel size of 1x1, which increases the input channel number c1 to the output channel number c2; the second layer is a CBS structure using grouped convolution, with the group number g being the channel number c2; the third layer uses a convolution kernel size of 1x1, and the last layer is a normalization layer BN; the grouped convolution used in the second layer of MobileConv can reduce network parameters, and the third layer convolution can enhance the information mixing between channels; in addition, the reason why layer 0 is not replaced is to preserve as much information from the input image as possible; the reason why layers 10 / 14 are not replaced is that these two layers originally use a CBS structure with a convolution kernel of 1x1, and if replaced with MobileConv, it will increase the network parameters due to the addition of a CBS structure with a convolution kernel of 3x3;

[0066] The comparison between C3 structure and MobileC3 structure is shown in Figure 10 . Figure 10 The left side is the C3 structure of the YOLOv5 model, Figure 9The right side shows the MobileC3 structure of the M-ST-YOLOv5 model; the Bottleneck in C3 consists of a CBS structure with a 1×1 kernel and a 3×3 kernel; MobileC3 replaces the second CBS structure in the Bottleneck with a grouped convolutional CBS structure, and adds a 1×1 convolutional kernel and a normalization layer (BN); MobileConv and MobileC3 have similar improvement ideas, the purpose of which is to reduce network parameters and runtime, and reduce the difficulty of practical network deployment.

[0067] like Figure 11 The C3ST shown is a structure that incorporates a Swin-Transformer; specifically, it replaces the Bottleneck in the 8th layer C3 structure with a Swin-Transformer Block structure. Figure 12 The diagram shows the Swin-TransformerBlock architecture, where the most important structures are based on W-MSA (Windows-based Multi-Head Self-Attention) and Shifted Windows Multi-Head Self-Attention (SW-MSA). W-MSA divides the image into multiple windows for self-attention, effectively reducing computational cost. SW-MSA uses a moving window approach to enhance information exchange between windows, effectively improving the network's information extraction capabilities. Furthermore, the feature map size of layer 8 is the smallest, so adding Swin-Transformer to this layer can minimize the increase in parameters and improve the network's ability to recognize smoke targets.

[0068] Experimental verification and analysis

[0069] The experiment used the PyTorch framework to build the M-ST-YOLOv5 network model, and the experimental GPU environment was a GeForce RTX 2080Ti. The weights obtained by the Ultralytics team from training on the COCO dataset were used to initialize the network parameters to speed up the training process.

[0070] The experimental parameters are set as follows: the image size img_size for training and verification is 640, the batch-size is 16, the training round number epoch is 50, the network optimizer is SGD, and the learning rate is 0.0001; Table 1 is a comparison of the experimental results of YOLOv5 and M-ST-YOLOv5 networks; as shown in the table, the accuracy P of M-ST-YOLOv5 for smoke detection is improved by 1.6 percentage points, the recall rate R is improved by 1.4 percentage points, the parameter quantity is reduced by 30%, and the operation quantity GFLOPs is reduced by 1 / 3. The network parameter quantity of the present application is less, suitable for practical deployment, and the detection accuracy of smoke is higher.

[0071] Table 1 Comparison of network experimental results

[0072]

[0073] Figure 13 The prediction results of YOLOv5 and M-ST-YOLOv5 for the same smoke image (the image is not in the training set verification set) are compared, wherein the left image is the prediction result of YOLOv5, and the right image is the prediction result of M-ST-YOLOv5; as shown in the figure, M-ST-YOLOv5 can identify more smoke areas, and the confidence is high.

[0074] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practice of the present disclosure disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the art that are not disclosed in the present disclosure; the specification and examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0075] The above-mentioned embodiments of the present application do not constitute a limitation on the scope of protection of the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the claims of the present application.

Claims

1. A method for detecting smoke and dust in an open-pit coal mine based on an M-ST-YOLOv5 model, characterized in that: The method comprises the following steps: Obtain the target image of the open-pit coal mine smoke to be detected; Input the image into the M-ST-YOLOv5 model for open-pit coal mine smoke detection and identification; The M-ST-YOLOv5 model construction method comprises: Step 1, collect open-pit coal mine smoke image data; Step 2, process the image data of step 1 to construct the sample data set of open-pit coal mine smoke with open-pit coal mine smoke images and corresponding labels; Step 3, build the M-ST-YOLOv5 model, input the sample data set into the M-ST-YOLO5 network for training; obtain the training weight parameters suitable for open-pit coal mine smoke detection, and obtain the M-ST-YOLOv5 model with open-pit coal mine smoke detection; Output the detection result of the open-pit coal mine smoke; The M-ST-YOLOv5 model is improved compared with the YOLOv5 model, and the specific mode is as follows: The CBS structure of the 1st, 3rd, 5th, 7th, 18th, 21st and 23rd layers is replaced by MobileConv structure; The C3 structure of the 2nd, 4th, 6th, 13th, 17th, 20th and 23rd layers is replaced by MobileC3 structure; The C3 structure of the 8th layer is replaced by C3ST; The first layer of the MobileConv is a CBS structure with a convolution kernel size of 1x1, which increases the input channel number c1 to the output channel number c2; the second layer is a CBS structure using grouped convolution, and the group number g is the channel number c2; the third layer adopts a convolution kernel size of 1x1, and the last is a normalization layer BN; the grouped convolution CBS structure used in the second layer of the MobileConv is used to reduce the network parameters, and the third layer convolution is used to enhance the information mixing between channels; The Bottleneck in the C3 structure is composed of a CBS structure with a convolution kernel of 1x1 and a CBS structure with a convolution kernel of 3x3; the MobileC3 replaces the second CBS structure in the Bottleneck with a CBS structure using grouped convolution, and adds a convolution kernel of 1x1 and a normalization layer BN; The Bottleneck in the C3 structure of the 8th layer is replaced by Swin-Transformer Block structure, and the Swin-Transformer Block structure includes window multi-head self-attention W-MSA and shift window multi-head self-attention SW-MSA.

2. The open-pit coal mine smoke detection method based on the M-ST-YOLOv5 model according to claim 1, characterized in that, The M-ST-YOLOv5 model construction method comprises that the trainable sample data set is obtained by the following method: Step 2.1, the video is intercepted at 25 frames per second; Step 2.2, using LabelImg tool to label the smoke target of the open-pit coal mine smoke image intercepted in step 2.1, obtaining an XML file corresponding to the image sample as a sample; Step 2.3, divide the data set; divide the image data set obtained in step 2.2 into training set, validation set and test set according to the proportion of 7:2:1; Step 2.4, adjusting the data set format; converting the XML files of the training set, validation set and test set obtained by labeling into TXT files; constructing the sample data set of the open-pit coal dust image and the corresponding label.

3. The open-pit coal mine smoke detection method based on the M-ST-YOLOv5 model according to claim 2, characterized in that, In step 2.1, all the open-pit coal dust images obtained are subjected to data enhancement to obtain new images, thereby expanding the number of data sets. The data enhancement method includes any one or a combination of multiple kinds of translation, flipping, rotation, noise addition and scaling.

4. The open-pit coal mine smoke detection method based on the M-ST-YOLOv5 model according to claim 1, characterized in that, The multi-head self-attention W-MSA divides the image into multiple windows for self-attention, which is used to reduce the amount of calculation. The shift window multi-head self-attention SW-MSA adopts a moving window mode, which is used to enhance the information exchange between windows and improve the network's ability to extract information.

5. An open cut coal mine dust detection apparatus, characterised in that, The device is configured to perform the method of any one of claims 1-4; the device comprises: A dust image acquisition module configured to perform acquisition of a target image of open-pit coal dust to be detected; A detection and recognition module configured to perform input of the image into the M-ST-YOLOv5 model for open-pit coal dust detection and recognition; A result output module configured to perform output of the detection result of the open-pit coal dust.

Images