A coal gangue image classification method based on a lightweight convolutional neural network

Abstract

The application discloses a coal gangue image classification method based on a light-weight convolutional neural network, which comprises collecting coal gangue image data sets, pre-processing the data set images and dividing the data sets; performing data enhancement operation on the divided data sets; proposing a more efficient light-weight convolutional module based on an original Ghost module, and building a light-weight convolutional neural network for coal gangue image recognition; inputting the enhanced training data set into the designed recognition model for training, obtaining the trained light-weight convolutional neural network and saving the model weight parameters; loading the saved model parameter file, recognizing the test data set, and obtaining the prediction result. The method effectively improves the recognition accuracy of the light-weight convolutional neural network model, can be well applied to the underground industrial scene to recognize coal and gangue, and solves the problem that the traditional convolutional neural network is difficult to deploy on the embedded device with limited computing resources due to high model complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of coal gangue classification, and particularly relates to a coal gangue image classification method based on a lightweight convolutional neural network. BACKGROUND

[0002] In the process of coal production, the gangue mined accounts for 15%-20% of the raw coal production. The gangue not only has the characteristics of large density and low calorific value, but also contains a large amount of heavy metal elements. In the process of coal mining, the gangue can cause damage to the transport belt and even tear, thereby increasing the industrial cost. In addition, the gangue mixed with coal for combustion can reduce the combustion efficiency of the coal and aggravate the pollution of the ecological environment. Therefore, the accurate sorting of coal and gangue is particularly important for the efficient use of coal energy.

[0003] The traditional methods for sorting coal and gangue mainly include manual gangue selection, heavy medium gangue selection, dynamic sieve jigging gangue selection, dual-energy gamma ray detection, X-ray detection, and laser detection. However, these methods have the disadvantages of high cost, low precision, low environmental protection, and harm to human health, which are not conducive to the long-term development of coal mining. At present, the development of computer technology and machine vision effectively solves these problems. Traditional image classification techniques rely on manually designed features, such as color features, texture features, and shape features. Since the manually designed features depend on engineering experience and cannot adapt to the huge changes in imaging conditions, their accuracy cannot meet the actual needs of people. In recent years, with the continuous upgrading of computing systems, some complex architectures based on deep convolutional neural networks have shown forward-looking results in the task of coal gangue classification. In order to achieve higher classification accuracy, one general trend is to build deeper and more complex models. However, due to the limited storage space and processor performance of mobile and embedded devices, the large amount of computation prevents these devices from further using complex networks. In order to meet the application requirements, it is necessary to explore network architectures that can achieve the best accuracy under very limited computational budget. SUMMARY

[0004] In view of the above technical problems, the present application provides a coal gangue image classification method based on a lightweight convolutional neural network.

[0005] The technical solution adopted by the present application to solve its technical problems is:

[0006] A coal gangue image classification method based on a lightweight convolutional neural network, the method comprising the following steps:

[0007] Step (1): Collecting visible light image data sets of coal and gangue, pre-processing the data sets, and dividing the data;

[0008] Step (2): Considering the small number of the training data set, data augmentation is performed on the divided data set;

[0009] Step (3): A more efficient lightweight convolution module is proposed on the basis of the original Ghost module, and a lightweight convolutional neural network for coal gangue image classification is built;

[0010] Step (4): The enhanced training data set is input into the designed lightweight convolutional neural network model for training, and the trained lightweight convolutional neural network model is obtained, and the model weight file is saved;

[0011] Step (5): Load the saved model parameter file, input the enhanced test data set into the model for prediction classification, and output the corresponding category and accuracy of each coal gangue image.

[0012] Preferably, the image of coal and gangue is collected by the collecting device, and the collected image is from 200 pieces of coal and 200 pieces of gangue, and a non-contact collection method is used to collect the image. The camera is installed at a fixed position for shooting, a total of 400 pictures are shot, the shot coal gangue pictures are preprocessed, including effective area interception, and resolution and inverse line are unified, which is more suitable for actual underground industrial scene application. The coal gangue image data set which needs to be classified is divided into training set and test set according to the proportion of 4:1.

[0013] Preferably, in order to reduce the influence of the number of data set on the classification rate of network model, a plurality of data augmentation methods are used to expand the divided data set, including multi-angle rotation (90 degrees, 180 degrees and 270 degrees), mirror image, random brightness, motion blur, salt and pepper and Gaussian noise. One picture is expanded to 10 pictures, and the expanded data set has a total of 4000 pictures.

[0014] Preferably, the lightweight convolutional neural network for coal gangue image classification comprises a first convolutional module, a plurality of GSAG bottleneck structures proposed by the application, an efficient channel attention (ECA) module, a second convolutional module, an average

[0015] pooling module, a third convolutional module and a fully connected layer connected in sequence,

[0016] The first convolutional module is used to receive the preprocessed coal gangue picture, and a convolution operation with a convolution kernel size of 3 and a step of 2 is used to extract the features in the picture, while reducing the size of the feature map;

[0017] The plurality of GSAG bottleneck structures proposed by the application are used to extract deep features of the picture after convolution operation with fewer parameters;

[0018] The ECA module is used to achieve local cross-channel interaction without dimensionality reduction, and to obtain semantic information with stronger expressive power;

[0019] The second convolution module is used to perform the first fusion of deep features using a convolution operation with a kernel size of 1 and a stride of 1;

[0020] The average pooling operation is used to reduce the feature size after the first fusion by using a pooling operation with a pooling kernel size of 7;

[0021] The third convolution module is used to perform a second fusion of the pooled features using a convolution operation with a kernel size of 1 and a stride of 1.

[0022] The fully connected layer is used to obtain the classification result based on the features after the second fusion.

[0023] Preferably, the GSAG bottleneck structure includes a first GSAG module, a first batch normalization module, a first non-linear activation function module, an SE attention module, a second GSAG module, a second batch normalization module, and a shortcut connection module connected in sequence.

[0024] The first GSAG module is used to extract features from the image after the convolution operation;

[0025] The first and second batch normalization modules are used to improve model performance and reduce the risk of overfitting.

[0026] The first non-linear activation function is used to avoid gradient vanishing and to train the model better;

[0027] The SE attention module is used to learn the features extracted by the first GSAG module;

[0028] The second GSAG module is used to extract the features learned by the SE attention module again;

[0029] The shortcut connection module is used to perform a shortcut connection between the feature map output by the second GSAG module and the feature map input by the first GSAG module to obtain deep features.

[0030] Preferably, when the stride is 2, the GSAG bottleneck structure further includes a first deep convolutional module, a second deep convolutional module, a second batch normalization module, and a third batch normalization module connected in parallel. The first deep convolutional module is positioned between the first GSAG module and the squeezing and activation attention module. The first deep convolutional module is used to change the size of the feature map extracted by the first GSAG module. The squeezing and activation attention module is used to learn the features in the feature map after the size has been changed. The second batch normalization module is connected to the first deep convolutional module, and the third batch normalization module is connected to the second deep convolutional module.

[0031] Preferably, the GSAG module includes grouped convolution, cheap linear transformation, channel concatenation, and channel shuffle:

[0032] The grouped convolution is used to generate the original feature map;

[0033] The cheap linear transformation is used to generate additional ghost feature maps with a small number of parameters;

[0034] The channel concatenation is used to concatenate the original feature map and the ghost feature map in the channel dimension;

[0035] The channel shuffle is used to facilitate the flow of information between the original feature map and the ghost feature map.

[0036] The above-mentioned coal gangue image classification method based on a lightweight convolutional neural network effectively improves the classification and recognition accuracy of the convolutional neural network model for coal and gangue, and greatly reduces the complexity of the convolutional neural network model with little change in the classification and recognition accuracy of the model, which is more suitable for deployment in embedded devices with limited memory and computing resources. The model proposed in the present application can be well applied to classify coal and gangue in coal mining, effectively reducing the existing hardware requirements. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A flow chart of a coal gangue image classification method based on a lightweight convolutional neural network in an embodiment of the present application;

[0038] Figure 2 An image schematic diagram of coal and gangue before and after enhancement in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of a lightweight convolutional neural network model in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a GSAG bottleneck structure in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a GSAG bottleneck structure in another embodiment of the present application;

[0042] Figure 6 A schematic diagram of an efficient channel attention (ECA) module structure in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of a GSAG module structure in an embodiment of the present application;

[0044] Figure 8 A comparison chart of ablation experiment accuracy in an embodiment of the present application;

[0045] Figure 9 Figure 2 is a precision curve diagram of a classical network comparison experiment in an embodiment of the present application; DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0047] In one embodiment, as shown in Figure 1 A coal and gangue image classification method based on a lightweight convolutional neural network, the method comprising the following steps:

[0048] Step (1): Collecting visible light image datasets of coal and gangue, pre-processing the datasets and dividing the data;

[0049] Specifically, the samples of coal and gangue in this study mainly come from a mining area in East China. We use the visible light mode of FLIRE50 infrared thermal imager of American FLIR Company to collect data. We randomly selected 200 pieces of coal and gangue for sampling, a total of 400 visible light pictures of coal and gangue were taken. The experimental data set of this study is mainly divided into training set and test set, the former contains 320 pictures, accounting for 80%, the latter contains 80 pictures, accounting for 20%.

[0050] Step (2): Considering that the number of training dataset is small, data augmentation is performed on the divided dataset.

[0051] In one embodiment, in order to improve the generalization ability of the coal and gangue classification model, based on Python software and its related image processing functions, multiple image enhancement methods are used for the images of the training set and the test set. The specific image enhancement methods include image multi-angle rotation (90 degrees, 180 degrees, 270 degrees), horizontal mirror image, vertical mirror image, random brightness, etc. In addition, considering the noise generated by the image acquisition device during image acquisition and the blurring of the collected images caused by device shaking. Gaussian noise and salt and pepper noise are added to the images respectively, and the images are motion blurred. Taking the image of coal in the training set as an example, the detailed distribution of the data is shown in Figure 2 .

[0052] Step (3): On the basis of the original Ghost module, a more efficient lightweight convolutional module is proposed, and a lightweight convolutional neural network for coal and gangue image classification is built;

[0053] In one embodiment, as shown in Figure 3 The lightweight convolutional neural network for coal and gangue image classification comprises a first convolutional module, a plurality of GSAG bottleneck structures proposed by the present application, an ECA module, a second convolutional module, an average pooling module, a third convolutional module and a fully connected layer connected in sequence.

[0054] The first convolutional module receives the preprocessed coal gangue image and extracts features from the image using a convolution operation with a kernel size of 3 and a stride of 2, while simultaneously reducing the size of the feature map.

[0055] Several GSAG bottleneck structures proposed in this application are used to extract deep features of images after convolution operations with fewer parameters;

[0056] The ECA module is used to achieve local cross-channel interaction without dimensionality reduction, and to obtain semantic information with stronger expressive power;

[0057] The second convolution module is used to perform the first fusion of deep features using a convolution operation with a kernel size of 1 and a stride of 1;

[0058] The average pooling operation is used to reduce the feature size after the first fusion by using a pooling operation with a pooling kernel size of 7;

[0059] The third convolution module is used to perform a second fusion of the pooled features using a convolution operation with a kernel size of 1 and a stride of 1.

[0060] The fully connected layer is used to obtain the classification result based on the features after the second fusion.

[0061] Specifically, in this embodiment, the number of GSAG bottleneck modules is 16, which can be set according to the actual situation.

[0062] In one embodiment, such as Figure 4 As shown, the GSAG bottleneck module includes, in sequence, a first GSAG module, a first batch normalization module, a first non-linear activation function module, an SE attention module, a second GSAG module, a second batch normalization module, and a shortcut connection module.

[0063] The first GSAG module is used to extract features from the image after the convolution operation;

[0064] The first and second batch normalization modules are used to improve model performance and reduce the risk of overfitting.

[0065] The first non-linear activation function is used to avoid gradient vanishing and to train the model better;

[0066] The SE attention module is used to learn the features extracted by the first GSAG module;

[0067] The second GSAG module is used to extract the features learned by the SE attention module again;

[0068] The shortcut connection module is configured to connect the feature map output by the second GSAG module with the feature map input by the first GSAG module in a shortcut manner, to obtain deep-level features.

[0069] In another embodiment, as shown in FIG. 2, when the step is 2, the GSAG bottleneck structure further includes two depthwise convolution (DWConv) modules, a fourth batch normalization module and a fifth batch normalization module connected in parallel, the first depthwise convolution module is arranged between the first GSAG module and the SE attention module, the first depthwise convolution module is configured to change the size of the feature map extracted by the first GSAG module, the SE module is configured to learn the features in the feature map with the changed size, the second batch normalization module is connected to the first depthwise convolution module, and the third batch normalization module is connected to the second depthwise convolution module. Figure 5

[0070] Specifically, the features of the image are first extracted by the first GSAG module, if the step is greater than 1, it indicates that the size of the feature map has changed, therefore the first depthwise convolution module is used to change the size of the feature map, then the SE module is used to learn the obtained features, so that the features can better express the image, then the second GSAG module is used to extract the features of the image, and finally the obtained feature map is connected with the input feature map of the GSAG bottleneck structure in a shortcut manner, to prevent the network from being too deep and the expression ability of the network from being reduced.

[0071] In one embodiment, as shown in FIG. 3, the ECA attention module includes a global average pooling module, a fourth convolution module, a first sigmoid function module and a channel-wise product module connected in sequence. Figure 6

[0072] Specifically, the ECA attention module uses the global average pooling module to perform spatial compression on the input feature map, then uses the fourth convolution module with a convolution kernel size of 1 to perform feature learning on the compressed feature map, obtains the corresponding weight through the first sigmoid function module, and finally uses the channel-wise product module to perform channel-wise multiplication of the learned channel features and the original feature map. The module has good cross-channel information acquisition capability and can realize local cross-channel interaction without dimension reduction. At the same time, the module can only involve a small amount of parameters, while it can bring obvious performance improvement.

[0073] In one embodiment, as shown in FIG. 4, the GSAG module proposed in the present application includes grouped convolution, cheap linear transformation, channel connection and channel shuffle: Figure 7

[0074] ​​​Specifically, although the use of the original Ghost module can greatly reduce the computational complexity of the model, the partial convolution operation in the module can only extract a small amount of key features, and cannot always guarantee high accuracy, resulting in poor stability. However, simply increasing the traditional convolution kernel input can improve the effect to a certain extent, but when the number of convolution kernels is too large, too many redundant parameters will cause overfitting. In order to avoid the occurrence of the above problems, we introduce a grouped convolution instead of the standard convolution in the Ghost module. Grouped convolution can increase the diagonal correlation between adjacent layers of filters, reduce the training parameters, so that the training is not easy to overfit, and the effect is similar to regularization. In addition, using grouped convolution can reduce the FLOPs required by the Ghost module, further realizing the lightweight of the model. At the same time, in the specific implementation process of the Ghost module, the cheap linear operation in it is generally equivalent to a depth convolution by default. However, the depth convolution only mixes the spatial dimension, and the information between each channel is not interacted, resulting in the loss of some information between channels in the subsequent flow of information. This will affect the representation ability and recognition accuracy of the model. In order to solve this problem, we introduce a channel shuffle operation in the Ghost module to help information flow. Without increasing the computational cost of the model, by disturbing and reorganizing the intrinsic feature maps and Ghost feature maps, we ensure that the input of the subsequent Ghost module contains more effective information. Therefore, information can flow between different channels, which helps to encode more information and improve the robustness of the model. The steps of the GSAG module proposed in the present application are as follows: first, use grouped convolution to generate part of the intrinsic feature map. Then generate the Ghost feature map by linear transformation based on the intrinsic feature map. Then the intrinsic feature map and the Ghost feature map are spliced. Finally, the channel of the spliced feature map is shuffled.

[0075] The enhanced training data set is input into the designed lightweight convolutional neural network model for training, and a trained lightweight convolutional neural network model is obtained, and the model weight file is saved. Load the saved model parameter file, input the enhanced test data set into the model for prediction classification, and output the corresponding category and accuracy of each coal gangue picture.

[0076] In one embodiment, the present application uses some classic convolutional neural networks to verify the superiority and effectiveness of the GSAG module proposed in the present application compared with the Ghost module on the self-made image data set, including AlexNet, VggNet and ResNet.

[0077] Table 1 Comparison results of GSAG module and Ghost module

[0078]

[0079] As can be seen from Table 1, both the Ghost module and the GSAG module significantly reduced the FLOPs of the base model. Meanwhile, for

[0080] The AlexNet and Vgg13 models still showed improved accuracy after embedding the Ghost and GSAG modules. Notably, compared to Ghost_AlexNet and Ghost_Vgg13, GSAG_AlexNet and GSAG_Vgg13 reduced the number of FLOPs by 31.97% and 47.43%, respectively, and improved accuracy by 0.27% and 1.24%. Compared to Ghost_ResNet50, GSAG_ResNet50 still achieved higher accuracy and had fewer FLOPs, indicating that the GSAG module is feasible.

[0081] In one embodiment, both the channel shuffling and ECA attention modules in the GSAG module proposed in this application contribute to improving the classification accuracy of coal gangue to a certain extent. Therefore, this application analyzes the effectiveness of these two components. Based on the addition scenarios, this application designed the following four experiments: (a) without adding any operation, (b) adding only channel shuffling, (c) adding only the ECA attention module, and (d) adding both channel shuffling and the ECA attention module simultaneously. Figure 8 This is the result of the ablation experiment.

[0082] To verify the superiority of the lightweight convolutional neural network model proposed in this application for coal gangue image classification, several currently popular lightweight convolutional neural network models were selected for comparison. To ensure fairness in the experiment, the FLOPs of all models were standardized at the same level. The results are as follows: Figure 9 As shown, the decimal after the network name represents a controllable width factor. Since there are only two types of test datasets, all methods achieve relatively high classification accuracy, exceeding 95%. For coal gangue image classification, the proposed GSAGNet achieves the best classification performance on the test set, with an accuracy of 97.50%. Compared to models such as MobileNetV2, ShuffleNetV2, GhostNet, and EfficientNet, the accuracy is improved by 1.78%, 0.79%, 1.01%, and 0.95%, respectively.

[0083] In addition, other index results are shown in Table 2. It can be seen that the classification accuracy of GSAGNet on coal and gangue is the highest compared with other models under the condition of similar model complexity. Specifically, the precision, recall and F1 score of GSAGNet reach 97.63%, 97.5% and 97.5% respectively. Among them, the classification accuracy of the classic lightweight model MobileNetV2 is the lowest. Compared with MobileNetV2, the precision, recall and F1 score of GSAGNet are increased by 1.41%, 1.78% and 1.8% respectively. Compared with the F1 scores of ShuffleNetV2, GhostNet and EfficientNetB0, the F1 score of GSAGNet is increased by 0.79%, 1.04% and 0.96% respectively. In general, under the condition of ensuring the model complexity at the same level, the strategy proposed in this study is more competitive when applied to the coal and gangue image dataset.

[0084] Table 2 Comparison of experimental results of different convolutional neural networks

[0085]

[0086] The above describes in detail a coal and gangue image classification method based on a lightweight convolutional neural network. In this paper, specific examples are used to illustrate the principles and implementation of the present application. The above examples are only used to help understand the core idea of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A coal gangue image classification method based on a lightweight convolutional neural network, the algorithm comprising the following steps: (1): Collect visible light image datasets of coal and gangue, preprocess the datasets and divide the data; (2): Considering the small number of training datasets, data augmentation is performed on the partitioned datasets; (3): Based on the original Ghost module, a more efficient lightweight convolution module is proposed, and a lightweight convolutional neural network for coal gangue image classification is built. (4): Input the enhanced training dataset into the designed lightweight convolutional neural network model for training, obtain the trained lightweight convolutional neural network model, and save the model weight file; (5): Load the saved model parameter file, input the enhanced test dataset into the model for prediction and classification, and output the category and accuracy corresponding to each coal gangue image; The lightweight convolutional neural network for coal gangue image classification includes a first convolutional module, multiple GSAG bottleneck structures, an efficient channel attention module, a second convolutional module, an average pooling module, a third convolutional module, and a fully connected layer connected in sequence. The GSAG bottleneck structure includes a first GSAG module, a first batch normalization module, a first nonlinear activation function module, a squeezing and excitation attention module, a second GSAG module, a second batch normalization module, and a shortcut connection module connected in sequence; and a first deep convolution module, a second deep convolution module, a second batch normalization module, and a third batch normalization module connected in parallel. The GSAG module includes grouped convolution, inexpensive linear transformation, channel concatenation, and channel shuffling.

2. A coal gangue image classification method based on a lightweight convolutional neural network according to claim 1, wherein in step (1): images of coal and gangue are collected using an acquisition device, the collected images are from 200 pieces of coal and 200 pieces of gangue respectively, and the images are collected using a non-contact acquisition method, the camera is installed in a fixed position to take pictures, a total of 400 pictures are taken, the collected coal gangue pictures are preprocessed, including the cropping of the effective area, and the resolution and inversion are unified, which is more suitable for the application of actual underground industrial scenarios, and the coal gangue image dataset that needs to be classified is divided into training set and test set in a ratio of 4:

1.

3. The coal gangue image classification method based on lightweight convolutional neural network according to claim 1, wherein in step (2), in order to reduce the impact of the number of datasets on the classification rate of the network model, a variety of data augmentation methods are used to expand the data of the divided dataset, including multi-angle rotation, mirroring, random brightness, motion blur, salt and pepper and Gaussian noise, expanding one photo into 10 photos, and the expanded dataset has a total of 4000 photos.

4. A coal gangue image classification method based on a lightweight convolutional neural network according to claim 1, comprising: the first convolutional module is used to receive a preprocessed coal gangue image, extract features from the image using a convolution operation with a kernel size of 3 and a stride of 2, and simultaneously reduce the size of the feature map; The multiple GSAG bottleneck structures are used to extract deep features from images after convolution operations using fewer parameters. The high-efficiency channel attention module is used to achieve local cross-channel interaction without dimensionality reduction and to obtain semantic information with stronger expressive power; The second convolution module is used to perform the first fusion of deep features using a convolution operation with a kernel size of 1 and a stride of 1. The average pooling operation is used to reduce the feature size after the first fusion by using a pooling operation with a pooling kernel size of 7. The third convolutional module is used to perform a second fusion of the pooled features using a convolutional operation with a kernel size of 1 and a stride of 1. The fully connected layer is used to obtain the classification result based on the features after the second fusion.

5. The method according to claim 1, wherein the first GSAG module is used to extract features from the image after the convolution operation; The first and second batch normalization modules are used to improve model performance and reduce the risk of overfitting. The first non-linear activation function is used to avoid gradient vanishing and to better train the model; The squeeze and stimulate attention module is used to learn the features extracted by the first GSAG module; The second GSAG module is used to extract the features learned by the squeeze and attention module again. The shortcut connection module is used to perform a shortcut connection between the feature map output by the second GSAG module and the feature map input by the first GSAG module to obtain deep features.

6. The method according to claim 1, wherein when the stride is 2, the first deep convolution module is disposed between the first GSAG module and the squeeze and stimulate attention module, the first deep convolution module is used to change the size of the feature map extracted by the first GSAG module, the squeeze and stimulate attention module is used to learn the features in the feature map after the size has been changed, the second batch normalization module is connected to the first deep convolution module, and the third batch normalization module is connected to the second deep convolution module.

7. The method according to claim 1, wherein the grouped convolution is used to generate the original feature map; The inexpensive linear transformation facilitates the generation of additional ghost feature maps with a small number of parameters; The channel connection is used to connect the original feature map and the ghost feature map according to the channel dimension; The channel shuffling is used to facilitate the flow of information between the original feature map and the ghost feature map.

Images