Lightweight coal gangue detection model and method based on parallel deep residual network

Abstract

The application discloses a lightweight coal gangue detection model and method based on a parallel deep residual network. The coal gangue detection model comprises an input layer, a backbone network, a head network and an output layer connected in sequence. The input layer is used for acquiring a coal gangue image. The backbone network is used for feature extraction of the coal gangue image and comprises a plurality of parallel deep residual blocks. The parallel deep residual block comprises a stacked nonlinear fitting layer F(x). The stacked nonlinear fitting layer is expressed as F(x) = H(x) - x, wherein H(x) represents a bottom layer mapping, and x represents an input image of the parallel deep residual block. The head network is used for multi-scale feature fusion of the features extracted by the backbone network, and outputs a corresponding coal gangue detection result through the output layer. The model can balance the detection speed and accuracy and is suitable for lightweight coal gangue detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a lightweight coal gangue detection model and method based on a parallel deep residual network. BACKGROUND

[0002] Coal preparation is an indispensable stage of coal production and an important part of clean coal technology, and is considered to be the most effective and economically valuable technology for reducing environmental pollution caused by coal. Manual picking is the most common method for coal gangue pre-separation, but it has low separation efficiency and poor safety, and mechanical separation methods such as heavy medium and flotation can easily cause pollution problems and have a huge impact on the environment. Automatic ore dressing technology based on sensors can reduce downstream costs and improve ore quality, but these methods are easily disturbed and have poor stability. In order to solve the above problems, coal gangue detection based on computer vision has become a new research direction. For coal gangue detection, the main research difficulty lies in that the recognition effect is greatly affected by the surface state of the gangue and the image environment acquisition environment.

[0003] In order to solve the problem that the target image is easily affected by light, coal dust, motion blur and other factors, many scholars have studied the coal gangue detection method based on image processing. In order to reduce the influence of color difference and motion difference between the gangue and coal transported by the conveyor belt, the literature "Machine vision gray pyramid convolution model of coal flow foreign matter identification [J], Coal Science and Technology, Yang Jianhui, Huang Ziyang, Wang Mei, etc." proposed a double-path pyramid convolution identification model based on machine vision gray. This model combines Lucas-Kanade optical flow algorithm and illumination compensation algorithm to realize the identification of abnormal color and abnormal motion pixels, but there are missed detection and false detection. In addition, the model does not study the training weight gray method under the condition of uneven illumination when it is designed. Dou et al. extracted the color and texture features of coal gangue image, and proposed a relief-SVM method based on image analysis for identifying the optimal features. However, this method only studies four properties of coal and does not consider the complex phenomenon of multiple situations occurring at the same time, which has certain application limitations. Hu et al. based on multispectral imaging and LBP feature extraction algorithm, studied the coal gangue detection method, but this research only considers the strategy of combining feature extraction and classifier to classify coal and coal gangue. However, different feature extraction methods and classifiers will have a certain impact on the results. The literature "Conveyor belt coal flow detection method based on TOF depth image repair [J], Industrial and Mining Automation, Wang Xin Yue, Qiao Tie Zhu, Pang Yu Song, etc." aims at the problem that the edge information of TOF depth image is not accurate due to the existence of flying pixel noise and multipath error noise at the edge. It proposes a depth image repair algorithm guided by intensity image and applies it to the conveyor belt coal flow detection. However, this method requires high equipment and does not consider the detection error caused by the shaking of the conveyor belt, which reduces the detection accuracy to a certain extent.

[0004] Coal gangue sorting using machine learning plays an important role in the field of coal and is a research hotspot in recent years. With the rise of deep learning, a large number of detectors based on deep neural network (DNN) have been proposed. For coal gangue images under multi-scale and half-occlusion conditions, Wang et al. proposed a coal gangue image semantic segmentation network based on pyramid scene interpretation network, but the experimental results showed that the loss was large when using this method for training. For fine-grained features, Lv et al. proposed a single-shot fine-grained object detector and applied it to coal gangue images in coal preparation plants, but this method sacrificed positioning accuracy while improving fine-grained target classification accuracy, and the classification accuracy of the method was low under general benchmarks. Zhang et al. used the YOLOv4 algorithm based on deep learning to detect coal gangue, but the experiments were all conducted in ideal environments without considering the influence of potential impurities in actual production environments, and the model robustness was poor. Lai et al. proposed an improved mask R-CNN combined with multispectral imaging for coal gangue instance segmentation, which lightened the classic mask R-CNN model, but the algorithm average precision was low without detailed description in subsequent experiments.

[0005] In related technologies, a Chinese patent application for invention with application publication number CN111346842A discloses a coal gangue sorting method, device, equipment and storage medium, which extracts the outline of the target object through the constructed deep learning model, so as to realize the sorting of coal blocks and coal gangue in the picture to be sorted. A Chinese patent application for invention with application publication number CN115410039A discloses a coal foreign matter detection system and method based on an improved YOLOv5 algorithm, which improves the backbone network, activation function and attention mechanism in YOLOv5, and uses heterogeneous convolution in the FPN structure to obtain an improved YOLOv5 network model for coal foreign matter detection.

[0006] A Chinese patent application for invention with application publication number CN112633075A discloses a SAR image classification method based on deep improved residual network, which establishes a deep residual network model, improves the residual module, and performs three times of jump connection on the basis of the traditional residual module to increase the number of connection branches. This scheme improves the identity mapping connection mode, but the essence is still the residual structure. SUMMARY

[0007] The technical problem to be solved by the present application is how to improve the recognition effect and lightweight degree of the coal gangue detection model.

[0008] The present application solves the above technical problems by the following technical means:

[0009] In one aspect, the present application provides a lightweight coal gangue detection model based on a parallel deep residual network, the coal gangue detection model comprising an input layer, a backbone network, a head network and an output layer connected in sequence;

[0010] The input layer is configured to obtain a coal gangue image;

[0011] The backbone network is configured to extract features from the coal gangue image, comprising a plurality of parallel deep residual blocks, each of the parallel deep residual blocks comprising a stacked nonlinear fitting layer F(x), represented as F(x) = H(x) - x, wherein H(x) represents a bottom layer mapping, and x represents an input image of the parallel deep residual block;

[0012] The head network is configured to perform multi-scale feature fusion on the features extracted by the backbone network, and output a corresponding coal gangue detection result through the output layer.

[0013] Further, the parallel deep residual block comprises a first stacked layer, a second stacked layer and a third stacked layer, each of the first stacked layer, the second stacked layer and the third stacked layer comprising a nonlinear fitting layer;

[0014] The output of the third stacked layer and the output of the second stacked layer are added through a first addition operation;

[0015] The output of the first stacked layer and the output of the first addition operation are added through a second addition operation.

[0016] Further, the parallel deep residual block comprises a first stacked layer and a second stacked layer, each of the first stacked layer and the second stacked layer comprising a nonlinear fitting layer;

[0017] The output of the first stacked layer, the output of the second stacked layer and the input image of the parallel deep residual block are added through a third addition operation.

[0018] Further, the backbone network further comprises a convolution layer for dimensionality increasing of an input image, and the output of the convolution layer is connected with the input of the parallel deep residual block.

[0019] Further, the head network comprises a feature pyramid network and a path aggregation network connected in sequence;

[0020] The feature pyramid network comprises a first convolution layer, a first upsampling operator, a first feature fusion layer, a first downsampling layer, a second convolution layer, a second upsampling operator and a second feature fusion layer;

[0021] The output of the first convolutional layer is connected to the input of the first feature fusion layer through a first upsampling operator, the output of the second convolutional layer is connected to the input of the second feature fusion layer through a second upsampling operator, and the output of the first feature fusion layer is connected to the input of the second convolutional layer through the first downsampling layer.

[0022] The first convolutional layer, the first feature fusion layer and the second feature fusion layer are all connected to the output of the backbone network.

[0023] Further, the first upsampling operator and the second upsampling operator both include an upsampling prediction module and a feature reorganization module, wherein the upsampling prediction module is used to predict an upsampling kernel and perform normalization processing, and the feature reorganization module is used to reassemble features in a local area and map positions in the output feature map back to the input feature map.

[0024] Further, the path aggregation network includes a second downsampling layer, a third downsampling layer, a fourth downsampling layer, a third convolutional layer, a fourth convolutional layer, a third feature fusion layer and a fourth feature fusion layer.

[0025] The input of the second downsampling layer is connected to the second feature fusion layer, and the output is connected to the third convolutional layer, the input of the third feature fusion layer is connected to the third convolutional layer and the second convolutional layer, and the output is connected to the third downsampling layer.

[0026] The output of the third downsampling layer is connected to the input of the fourth feature fusion layer through the fourth convolutional layer, the output of the first convolutional layer is connected to the input of the fourth feature fusion layer, and the output of the fourth feature fusion layer is connected to the fourth downsampling layer.

[0027] The outputs of the second downsampling layer, the third downsampling layer and the fourth downsampling layer are all connected to the output layer.

[0028] In addition, the present application also proposes a training method of a lightweight coal gangue detection model based on a parallel deep residual network, which is used to train the coal gangue detection model as described above, comprising:

[0029] Collecting coal gangue images and labeling the coal gangue images to construct a data set;

[0030] The data set is divided into a training set and a test set according to a proportion;

[0031] Based on the training set, the hyperparameters of the coal gangue detection model are optimized using a LookAhead optimizer to obtain a trained coal gangue detection model;

[0032] Select a model evaluation index, and based on the test set, the identification performance of the coal gangue detection model is evaluated.

[0033] Further, the model evaluation index includes average precision, precision and recall rate, wherein:

[0034] The calculation formula of the average precision is:

[0035]

[0036] The calculation formula of the precision is:

[0037]

[0038] The calculation formula of the recall rate is:

[0039]

[0040] In the formula, N is the total number of input images, Q is the number of categories, p(k) is the precision of the kth image, Δr(k) is the change of the recall rate from the k-1th to the kth image, TP represents that the predicted category and the real category are both coal gangue, FN represents that the predicted category and the real category are both non-coal gangue, FP represents that the predicted category is coal gangue and the real category is non-coal gangue, and TN represents that the predicted category is non-coal gangue and the real category is coal gangue.

[0041] Further, based on the training set, the hyperparameters of the coal gangue detection model are optimized using a LookAhead optimizer to obtain a trained coal gangue detection model, comprising:

[0042] A multi-branch network is constructed and trained, and when inferring, the weights of the multi-branch network are transferred to a single-branch network, the multi-branch network includes parallel identity mapping and convolution branches, and the parallel identity mapping and convolution branches are added with a BN layer, the single-branch network includes two convolution layers, and the two convolution layers are connected through an activation function.

[0043] The advantages of the present application are:

[0044] (1) The backbone network in the coal gangue detection model constructed by the present application is a basic feature extractor of the target detection task, based on a deep residual learning framework, so that the stacked layers can perform residual mapping, and the parallel deep residual block has the characteristics of high training performance based on the multi-branch model and fast inference speed based on the single-path model, so that the balance between detection speed and accuracy can be achieved, the backbone network can directly add the input of the convolution operation to the output of the convolution operation, so that no additional parameters and calculation complexity are added, and it is suitable for lightweight coal gangue detection.

[0045] (2) In order to increase the model complexity caused by a large number of residual calculations, the convolution layer is arranged in the backbone network, the convolution is used for dimension increasing, and the deep convolution is used for feature extraction to obtain more information, and since the deep convolution kernel only extracts features for one channel, the calculation amount is small.

[0046] (3) In order to solve the problems of mosaic blocks and edge sawtooth caused by the nearest neighbor interpolation upsampling method, a lightweight upsampling operator capable of guiding the reorganization process according to input features is used in the head network to enlarge high-level features; the operator first uses the input feature map to predict the upsampling kernel, and then performs feature reorganization based on the predicted upsampling kernel, and the use of the lightweight operator greatly improves the model performance while obtaining a small calculation amount.

[0047] (4) In order to solve the problems that the random gradient descent algorithm is easy to fall into a large number of local suboptimal solutions and saddle points in the error function optimization process, multiple experiments are conducted on the selection of the initial learning rate, and the LookAhead optimizer is used for parameter learning in the back propagation process. Through comparison experiments and testing on the verification set, the optimizer is more stable than the stochastic gradient descent (SGD).

[0048] Additional aspects and advantages of the application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a structure schematic diagram of a lightweight coal gangue detection model based on a parallel deep residual network proposed in the application;

[0050] Figure 2 is a structure schematic diagram of a ParallelBlock in the application, wherein (a) is a traditional residual structure, (b) is a ParallelBlock structure proposed in the application, and (c) is another ParallelBlock structure proposed in the application;

[0051] Figure 3 is a structure schematic diagram of a lightweight upsampling operator CARAFE in the application;

[0052] Figure 4 is a flowchart of a training method of a lightweight coal gangue detection model based on a parallel deep residual network proposed in the application;

[0053] Figure 5 is a flowchart of a core algorithm of a Lookahead optimizer in the application;

[0054] Figure 6 are the training curves of different optimizers in the application, wherein (a) is the mAP_0.5 curve when using the LookAhead optimizer, (b) is the mAP_0.5 curve when using the SGD optimizer, (c) is the mAP_0.5:0.95 curve when using the LookAhead optimizer, and (d) is the mAP_0.5:0.95 curve when using the SGD optimizer;

[0055] Figure 7 are the loss curves of different upsampling methods in the application, wherein (a) is the confidence loss curve when using the CARAFE operator, (b) is the confidence loss curve when using the nearest neighbor interpolation, (c) is the positioning loss curve when using the CARAFE operator, and (d) is the positioning loss curve when using the nearest neighbor interpolation;

[0056] Figure 8 are the experimental results of different upsampling methods in the application, wherein (a) is the mAP_0.5 curve, (b) is the mAP_0.5:0.95 curve, (c) is the accuracy curve, and (d) is the recall rate curve;

[0057] Figure 9 are the experimental results of different backbone networks in the application, wherein (a) is the mAP_0.5 curve, (b) is the mAP_0.5:0.95 curve, (c) is the confidence loss curve, and (d) is the positioning loss curve. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0059] As shown in Figure 1 , the first embodiment of the present application proposes a lightweight coal gangue detection model based on a parallel deep residual network, which comprises an input layer, a backbone network, a head network and an output layer connected in sequence.

[0060] The input layer is used to obtain a coal gangue image.

[0061] The backbone network is used for feature extraction of the coal gangue image, and includes a plurality of parallel deep residual blocks, the parallel deep residual block includes a stacked nonlinear fitting layer F(x), and the stacked nonlinear fitting layer is represented as F(x) = H(x)-x, wherein H(x) represents a bottom layer mapping, and x represents an input image of the parallel deep residual block;

[0062] The head network is used for multi-scale feature fusion of the features extracted by the backbone network, and outputs a corresponding coal gangue detection result through the output layer.

[0063] With the deepening of the network, more information can be obtained, and the features are also richer. However, experiments show that with the deepening of the network, the accuracy tends to be saturated and then rapidly degenerates, because the deepening of the network will cause the problems of gradient disappearance and gradient explosion, and increasing more layers in a model with a proper depth will lead to a higher training error. To solve the degradation problem, the lightweight backbone network in the coal gangue detection model constructed by the application is a basic feature extractor for a target detection task, which is used for extracting overall features of coal gangue, including color, texture and the like. The lightweight backbone network includes a plurality of parallel deep residual blocks, which are parallel inverted residual structures, based on a deep residual learning framework, so that the stacked layers can perform residual mapping. The parallel deep residual block is based on the characteristics of high training performance of a multi-branch model and fast inference speed of a single-path model, and can balance the detection speed and accuracy. The backbone network can directly add the input of a convolution operation to the output of the convolution operation, so that no additional parameters and computational complexity are added, and it is suitable for lightweight coal gangue detection.

[0064] In an embodiment, as shown in (b) of Figure 2 The parallel deep residual block includes a first stacked layer, a second stacked layer and a third stacked layer, and the first stacked layer, the second stacked layer and the third stacked layer each include a nonlinear fitting layer.

[0065] The output of the third stacked layer and the output of the second stacked layer are added through a first addition operation;

[0066] The output of the first stacked layer and the output of the first addition operation are added through a second addition operation.

[0067] Specifically, the first stacked layer includes a first nonlinear fitting layer Conv_1, a second nonlinear fitting layer Conv_2 and a third nonlinear fitting layer Conv_3 connected in sequence, wherein the first nonlinear fitting layer Conv_1 and the second nonlinear fitting layer Conv_2 each include a batch normalization layer, and an activation function is connected after the batch normalization layer, and the third nonlinear fitting layer Conv_3 is a convolution layer.

[0068] The second stack layer comprises a fourth nonlinear fitting layer Conv_4 and a fifth nonlinear fitting layer Conv_5 connected in sequence, and the fourth nonlinear fitting layer Conv_4 and the fifth nonlinear fitting layer Conv_5 both comprise a batch normalization layer, wherein the batch normalization layer in the fourth nonlinear fitting layer Conv_4 is followed by an activation function;

[0069] The third stack layer comprises a sixth nonlinear fitting layer Conv_6, and the sixth nonlinear fitting layer Conv_6 comprises a batch normalization layer followed by an activation function;

[0070] The output of the sixth nonlinear fitting layer Conv_6 and the output of the fifth nonlinear fitting layer Conv_5 are added through a first addition operation Add_1, and the output of the third nonlinear fitting layer Conv_3 and the output of the first addition operation Add_1 are added through a second addition operation Add_2.

[0071] In an embodiment, as shown in (c) of FIG. 1, the parallel deep residual block comprises a first stack layer and a second stack layer, and the first stack layer and the second stack layer both comprise a nonlinear fitting layer; Figure 2

[0072] The output of the first stack layer, the output of the second stack layer and an input image of the parallel deep residual block are added through a third addition operation.

[0073] Specifically, the first stack layer and the second stack layer both comprise a seventh nonlinear fitting layer Conv_7, an eighth nonlinear fitting layer Conv_8 and a ninth nonlinear fitting layer Conv_9 connected in sequence, and the seventh nonlinear fitting layer Conv_7, the eighth nonlinear fitting layer Conv_8 and the ninth nonlinear fitting layer Conv_9 all comprise a batch normalization layer, and the batch normalization layer in the seventh nonlinear fitting layer Conv_7 and the eighth nonlinear fitting layer Conv_8 are both followed by an activation function;

[0074] The outputs of the two ninth nonlinear fitting layers Conv_9 and an input of the current parallel deep residual block are added through a third addition operation Add_3.

[0075] It should be noted that the parallel deep residual block designed in the embodiment can reshape the original mapping to Y=F(x)+x, and Y can be realized by a feedforward neural network. Since the identity mapping operation can directly add the input of a convolution operation to the output of the convolution operation, it does not increase the additional parameters and the calculation complexity, and therefore the parallel deep residual block ParallelBlock structure designed and implemented is suitable for a lightweight coal gangue detection model.

[0076] ​Figure 2 The inverted residual structure shown in (a) is a traditional MobileNet V2, and the parallel multi-branch structure shown in (b) and (c) is beneficial to the transmission of context information, further improves the model performance, and has less impact on the model complexity. Figure 2

[0077] In an embodiment, to achieve a balance between detection speed and accuracy, inspired by RepVGG, a parallel identity mapping and convolution branch is constructed during training and added to the subsequent BN layer, so that a VGG-like architecture can be obtained as a multi-branch network. During inference, only the structure of convolution+ReLU+convolution is used as a single-branch network. The training and inference processes are independently designed in this embodiment. During training, a high-precision multi-branch network is used for weight learning, while during inference, a low-latency single-branch network is used. Finally, the weights of the multi-branch network are transferred to the single-branch network, so that the performance of the ParallelBlock reaches SOTA. Therefore, the ParallelBlock module can balance the detection speed and accuracy based on the characteristics of high training performance of the multi-branch model and fast inference speed of the single-path model.

[0078] In an embodiment, as shown in Figure 1 The backbone network further includes a convolution layer for dimensionality increasing of the input image, and the output of the convolution layer is connected to the input of the parallel deep residual block.

[0079] Specifically, the backbone network specifically includes a convolution layer Conv3BN and 11 ParallelBlock modules, wherein the input of the convolution layer is the coal gangue image, and the 11 ParallelBlock modules are connected in sequence and connected to the output of the convolution layer; wherein the output of the third ParallelBlock module, the eighth ParallelBlock module and the eleventh ParallelBlock module is connected to the head network.

[0080] It should be noted that experiments show that when extracting features of coal gangue images, 11 ParallelBlock modules have better effect in the coal gangue data set used, and those skilled in the art can set the specific number of ParallelBlock module structures in the backbone network according to actual conditions. The embodiment is not limited specifically, but the problem of gradient vanishing or explosion may occur as the network deepens.

[0081] In an embodiment, as shown in Figure 1 The head network includes a feature pyramid network and a path aggregation network connected in sequence.

[0082] ​The feature pyramid network includes a first convolutional layer Conv1, a first upsampling operator CARAFE_1, a first feature fusion layer Concat_1, a first downsampling layer C3_1, a second convolutional layer Conv2, a second upsampling operator CARAFE_2, and a second feature fusion layer Concat_2;

[0083] The output of the first convolutional layer Conv1 is connected to the input of the first feature fusion layer Concat_1 via the first upsampling operator CARAFE_1; the output of the second convolutional layer Conv2 is connected to the input of the second feature fusion layer Concat_2 via the second upsampling operator CARAFE_2; and the output of the first feature fusion layer Concat_1 is connected to the input of the second convolutional layer Conv2 via the first downsampling layer C3_1.

[0084] The first convolutional layer Conv1, the first feature fusion layer Concat_1, and the second feature fusion layer Concat_2 are all connected to the output of the backbone network.

[0085] like Figure 1 As shown, the path aggregation network includes a second downsampling layer C3_2, a third downsampling layer C3_3, a fourth downsampling layer C3_4, a third convolutional layer Conv3, a fourth convolutional layer Conv4, a third feature fusion layer Concat_3, and a fourth feature fusion layer Concat_4.

[0086] The input of the second downsampling layer C3_2 is connected to the second feature fusion layer Concat_2, and the output is connected to the third convolutional layer Conv3. The input of the third feature fusion layer Concat_3 is connected to the third convolutional layer Conv3 and the second convolutional layer Conv2, and the output is connected to the third downsampling layer C3_3.

[0087] The output of the third downsampling layer C3_3 is connected to the input of the fourth feature fusion layer Concat_4 via the fourth convolutional layer Conv4, the output of the first convolutional layer Conv1 is connected to the input of the fourth feature fusion layer Concat_4, and the output of the fourth feature fusion layer Concat_4 is connected to the fourth downsampling layer C3_4.

[0088] The outputs of the second downsampling layer C3_2, the third downsampling layer C3_3, and the fourth downsampling layer C3_4 are all connected to the output layer Detect.

[0089] It should be noted that, in order to better utilize the features extracted by the backbone network and make a prediction, the head network module in the embodiment uses an FPN+PAN structure. The FPN structure is designed using a top-down architecture, which can construct high-level semantic feature maps at all scales and transmit and fuse high-level information through up-sampling, thereby obtaining a prediction feature map. In the PAN structure, a bottom-up architecture is used, which enhances the feature hierarchy by using precise signals at lower levels. In addition, a full connection operation is used to connect the feature grid and all feature layers, so that the semantic information in the layers is directly transmitted to the sub-network.

[0090] In an embodiment, in the FPN structure, a lightweight up-sampling operator CARAFE is used, which can guide the reorganization process according to the input features. The CARAFE operator structure is as shown in Figure 3 The input of Conv2d_2 and Conv2d_1 comes from the previous convolution layer, and the input of Conv2d_2 is connected. The output of Conv2d_2 is connected to an Input layer, which is connected to the input of another Input after being operated on the tensor by a plurality of Kernel functions. Finally, the Input output is connected to the input of Conv2d_3, and the result is spliced with the ParallelBlock layer data in the feature fusion layer Concat. The first up-sampling operator and the second up-sampling operator both include an up-sampling prediction module and a feature reorganization module, wherein the up-sampling prediction module is used to predict the up-sampling kernel and perform normalization processing, and the feature reorganization module is used to reassemble the features in the local region and map the position in the output feature map back to the input feature map.

[0091] Specifically, CARAFE mainly consists of an up-sampling prediction module and a feature reorganization module. First, to reduce the amount of calculation, a 1x1 convolution layer is used to compress the feature map channel. Then, a convolution layer with a kernel size of k encoder is used to predict the up-sampling kernel and use softmax to normalize the k up x k up reorganization kernel in space. Wherein, k up is the size of the reorganization kernel, and k encoder =k up -2. Finally, in the feature reorganization module, a weighted sum operator φ is used to reassemble the features in the local region, and the position in the output feature map is mapped back to the input feature map. The k up x k upThe area is dotted with the predicted up-sampling kernel to obtain the output value.

[0092] The embodiment uses a lightweight sample operator capable of guiding the reorganization process according to input features to perform up-sampling in the model, thereby obtaining a predicted feature map; unlike the most common nearest neighbor interpolation up-sampling method, the operator only brings a small amount of additional parameters and computational load while improving the performance of the model, and can avoid problems such as mosaic blocks and edge jaggedness. The model realizes low-cost deployment of the model by fusing the inverse residual structure, the lightweight up-sampling operator CARAFE and the LookAhead optimizer, while the performance of the model reaches SOTA.

[0093] In addition, as Figures 4 to 5 shown, the second embodiment of the present application also proposes a training method of a lightweight coal gangue detection model based on a parallel deep residual network, which is used to train the coal gangue detection model as described in the first embodiment above, and includes the following steps:

[0094] S10, collect coal gangue images and label the coal gangue images to construct a data set;

[0095] S20, divide the data set into a training set and a test set according to a proportion;

[0096] S30, based on the training set, use the LookAhead optimizer to optimize the hyperparameters of the coal gangue detection model to obtain a trained coal gangue detection model;

[0097] S40, select a model evaluation index, and evaluate the recognition performance of the coal gangue detection model based on the test set.

[0098] It should be noted that the present embodiment aims at the shortcomings of the random gradient descent algorithm in the error function optimization process, such as being easily trapped in a large number of local suboptimal solutions and saddle points, and a plurality of experiments are carried out on the selection of the initial learning rate and the LookAhead optimizer is used for parameter learning in the back propagation process to improve the generalization ability of the model, so that the model can be widely applied to the field of coal gangue detection.

[0099] Specifically, the real-time exchange rate adopts the Pytorch deep learning framework to design a neural network model, and the programming language is Python3.9. In the experimental environment, this paper adopts Windows10 64-bit operating system, the processor is Intel(R) Xeon(R) E5-2670@2.60GHz, the memory is 32GB, and the GPU is NVIDIA GeForce RTX 2080Ti. In terms of model parameter configuration, the model depth is 0.33, the width is 0.50, the learning rate momentum is set to 0.937, the Epoch is set to 500, the error evaluation selects the cross-entropy loss function, and the optimization algorithm selects the LookAhead optimizer.

[0100] In an embodiment, in the step S10, a coal gangue image is collected and labeled, and a data set is constructed, including the following steps:

[0101] A coal gangue image is collected, and the coal gangue image is labeled as coal gangue or non-coal gangue;

[0102] The coal gangue image is preprocessed to obtain a clear coal gangue image;

[0103] After the clear coal gangue image is enhanced and expanded, the data set is constructed.

[0104] It should be noted that the data is collected from the monitoring video of some coal preparation plant, and the training and verification set data is composed of 1500 conveyor belt coal transportation images, and the images are labeled by LabelImg software. In view of the problems that the collected images are easy to be affected by light, coal dust and fast running speed of the conveyor, and the image is blurred, linear transformation and deblurring algorithm are used for data preprocessing in this embodiment to increase the image clarity and reduce noise. In order to improve the generalization ability of the model and enhance the robustness of the algorithm, data augmentation technology is used to expand the data, and 1000 images are randomly taken as training samples and 430 images are taken as test samples.

[0105] In an embodiment, the step SS30: based on the training set, using the LookAhead optimizer to optimize the hyperparameters of the coal gangue detection model, obtaining the trained coal gangue detection model, including:

[0106] A multi-branch network is constructed and trained, and the weight of the multi-branch network is transferred to a single-branch network during inference, the multi-branch network includes parallel identity mapping and convolution branches, the parallel identity mapping and convolution branches are added with a BN layer, the single-branch network includes two convolution layers, and the two convolution layers are connected through an activation function.

[0107] By independently designing the training and inference processes, a high-precision multi-branch network is used for weight learning during training, and a low-delay single-branch network is used during inference. Finally, the weights of the multi-branch network are transferred to the single-branch network, so that the performance of the ParallelBlock reaches the SOTA. Therefore, the ParallelBlock module can balance the detection speed and accuracy based on the characteristics of high training performance of the multi-branch model and fast inference speed of the single-path model.

[0108] In an embodiment, the model evaluation indicators include average precision, precision, and recall in the step S40, wherein:

[0109] The calculation formula of the average precision is:

[0110]

[0111] The calculation formula of the precision is:

[0112]

[0113] The calculation formula of the recall is:

[0114]

[0115] In the formula, N is the total number of input images, Q is the number of categories, p(k) is the precision of the kth image, Δr(k) is the change in recall from the k-1th to the kth image, TP represents that the predicted category and the real category are both coal gangue, FN represents that the predicted category and the real category are both non-coal gangue, FP represents that the predicted category is coal gangue and the real category is non-coal gangue, and TN represents that the predicted category is non-coal gangue and the real category is coal gangue.

[0116] It should be noted that the present embodiment will use positive samples and negative samples to mark coal gangue and non-coal gangue targets, respectively, and the classification is as shown in Table 1. Among them, true positive (True Positive) represents the number of coal gangue detection results and real results that are both coal gangue; true negative (True Negative) represents the number of coal gangue detection results and real results that are both non-coal gangue; false positive (False Positive) represents the number of non-coal gangue samples classified as coal gangue samples; and false negative (False Negative) represents the number of coal gangue samples classified as non-coal gangue samples.

[0117] Table 1 Classification of coal gangue detection model

[0118]

[0119] The parallel deep residual module designed and trained in this embodiment is applied to coal gangue detection, and a lightweight upsampling operator is fused. To verify the effectiveness of the coal gangue detection model proposed in this embodiment, comparative experiments are performed with other deep learning models. Among them, the deep learning models include YOLOv5s model, YOLOv3 model, YOLOv5-RepVGG model and YOLOv5-ghost model, and the above models are the optimal solutions of multiple experiments, and the experimental results are shown in Table 2.

[0120] Table 2 Comparison of experimental results of each model

[0121]

[0122] As can be seen from Table 2, the ParallelDetector model designed in this paper can better detect coal gangue, which is better than other models. On the data set constructed in this paper, the recall rates of each model are the same. Compared with the existing YOLOv5s, YOLOv3 and YOLOv5s-REPVGG models, the model size of the ParallelDetector model is reduced by 80.29%, 97.71% and 73.47% respectively. Although YOLOv5s-ghost performs better in model size, it is reduced by 12 percentage points compared with ParallelDetector. Through in-depth analysis, it is found that the smaller model size of ParallelDetector model comes from three aspects. First, in the use of the inverted residual structure, the convolution is first upgraded, and then the deep convolution is used for feature extraction to obtain more information. Since the deep convolution kernel only extracts features for one channel, the calculation amount is small; second, although parallel convolution modules are added at different positions to improve the robustness of the model, since the identity mapping operation can directly add the input of the convolution operation to the output of the convolution operation, it will not increase the additional parameters and computational complexity; third, in the upsampling process, compared with other methods, the lightweight up operator CARAFE can compress the feature map channel, which to a certain extent reduces the model size.

[0123] In summary, the specific advantages of the ParallelDetector model are: a good balance is achieved among various indicators, the model size is greatly compressed without loss of accuracy, precision and recall rate, so that the model can meet the lightweight demand; the ParallelBlock module adopts the decoupled training and inference method, which can effectively balance the detection speed and accuracy; the Lookahead optimizer can prevent the error function from falling into a large number of local suboptimal solutions and saddle points in the optimization process, and at the same time make the model parameters approach or reach the optimal value, thereby improving the performance of the entire model.

[0124] It is worth noting that although the ParallelDetector model has some similar metrics to other models, its model size is significantly compressed, with a compression rate of 97.71% (compared to YOLOv3) and a minimum of 73.47% (compared to YOLOv5s-REPVGG). To address the problems of existing coal gangue detection methods, this paper utilizes inverse residuals and parallel modules to effectively improve coal gangue detection performance. Furthermore, this real-time exchange rate model emphasizes lightweight design, which facilitates low-cost deployment and enhances its industrial application prospects. Subsequent experiments will further verify the effectiveness and necessity of the method presented in this embodiment through comparative experiments.

[0125] To verify the effectiveness of the LookAhead optimizer in the ParallelDetector model, this paper conducts a comparative experiment using the SGD optimizer and the LookAhead optimizer. The experimental results are shown in Table 3, and the training curves are as follows: Figure 6 As shown in Table 3, the LookAhead optimizer exhibits more stable performance across all parameters. Both the SGD optimizer and the LookAhead optimizer achieve their best results when the learning rate is 0.007.

[0126] Table 3 Comparison of detection results between SGD optimizer and LookAhead optimizer

[0127]

[0128] Depend on Figure 6 It can be seen that when using the LookAhead optimizer, the curve begins to converge around 50 epochs, exhibiting a faster convergence speed than the SGD optimizer. For example... Figure 6 In cases 6(a), 6(b), and 6(d), the convergence speed gradually slows down as the initial learning rate increases, but the stability also gradually deteriorates. Figure 6 As shown in (c), when the initial learning rate is 0.003, 0.005, 0.007 and 0.009, mAP first increases and then decreases, indicating severe overfitting; when the learning rate is set to 0.001, as the curve converges, mAP increases steadily and no overfitting occurs.

[0129] The LookAhead optimizer achieves better generalization and converges faster than the SGD optimizer during training. Furthermore, the LookAhead optimizer exhibits more stable performance and stronger robustness during training. This embodiment uses the LookAhead optimizer to dynamically adjust the learning rate. To prevent overfitting during training, the initial learning rate is set to 0.001 for subsequent experiments.

[0130] The embodiment carries out detailed comparative test on the selection of initial learning rate, and further verifies the effectiveness of each module through ablation experiment.

[0131] To verify the effectiveness of the light-weight up-sampling operator, nearest neighbor interpolation (nearest) and CARAFE are used as up-sampling methods in the FPN structure for comparative experiments, and the loss curves are obtained as shown in Figure 7 . It can be seen from Figure 7 that as the initial learning rate increases, the model loss gradually decreases and converges around 100 Epochs. It can be seen from Figure 7 (a) and (b) of 7 that when using CARAFE operator and nearest neighbor interpolation for up-sampling, the model confidence loss presents a trend of first rising and then falling, and in the 5 experiments with different learning rates, the loss degree using CARAFE operator is lower than that using nearest neighbor interpolation. From Figure 7 (c) and (d) of 7, it can be seen that in the 5 experiments with different learning rates, the positioning loss presents a downward trend with the increase of Epoch.

[0132] The learning rate is set to 0.001, and multiple comparative experiments are carried out using different up-sampling methods, and the optimal solution is taken, and the experimental results are shown in Figure 8 . It can be seen from Figure 8 (a) and (b) of 8 that using CARAFE as the up-sampling model, mAP_0.5 and mAP_0.5:0.95 are greatly improved than nearest neighbor interpolation. From Figure 8 (c) and (d) of 8, it can be seen that when using CARAFE operator, Precision and Recall begin to converge and tend to be stable around 100 Epochs, but when using nearest neighbor interpolation, the curve oscillates obviously and cannot converge, and the model has poor robustness.

[0133] In summary, the model using CARAFE operator is superior to the model using nearest neighbor interpolation in terms of loss degree, average precision, precision and recall, and has stronger generalization ability. The experiment fully verifies the effectiveness of the model using CARAFE operator in the FPN structure.

[0134] To further explore the effect of ParalellBlock model, InvertedBlock and ParallelBlock are used in the backbone network module for experiment, and the experimental results are obtained as shown in Figure 9 by analyzing the average precision, confidence loss and positioning loss. It can be seen from Figure 9 that the curve begins to converge around 100 Epochs, and the model using InvertedBlock and ParallelBlock_2 has faster convergence speed.

[0135] Further analysis shows that two different ParallelBlock structures have different effects on the model. Among them, ParallelBlock_1 is the parallel structure of inverted residual, convolution and identity mapping. From Figure 9 (b), it can be seen that the model using this structure has a stable rise and is higher than the other two structures. ParallelBlock_2 is a parallel structure of double inverted residual and identity mapping. From Figure 9 (c) and Figure 9 (d), it can be seen that the model using this structure has an advantage in confidence loss and positioning loss.

[0136] Since the traditional computer vision algorithm has problems such as complex image processing process and limited application scene, and the existing coal gangue detector cannot better balance speed and accuracy, an embodiment proposes a lightweight coal gangue detection model based on parallel deep residual network, called ParallelDetector. The model uses decoupled training and inference for coal gangue detection features, designs a parallel deep residual block for feature extraction, and then introduces a lightweight operator for upsampling to obtain a prediction feature map. Finally, the initial learning rate and the optimizer are adjusted to improve the generalization ability of the algorithm, which has important significance for promoting the intelligent development of coal mines and breaking through clean coal technology.

[0137] The experimental results show that the ParallelDetector model proposed in the embodiment can better balance performance and model size, and compared with existing models, the average accuracy, size and confidence loss are all improved. At the same time, the feasibility of the parallel deep residual method and the effectiveness of the lightweight upsampling operator applied in the FPN structure are verified in detail. In addition, the adjustment of the initial learning rate and the selection of the optimizer can prevent the model from falling into a large number of local suboptimal solutions and saddle points during the error function optimization process, thereby improving the robustness of the model.

[0138] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0139] Furthermore, the terms "first", "second", etc. are used only for descriptive purposes and do not connote or imply relative importance or a quantity of the indicated technical features. Thus, a feature defined with "first", "second", etc. can include at least one of the features implicitly or explicitly. In the description of the present application, the meaning of "a plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined.

[0140] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A lightweight coal gangue detection model based on parallel deep residual networks, characterized in that, The coal gangue detection model includes an input layer, a backbone network, a head network, and an output layer connected in sequence. The input layer is used to acquire coal gangue images; The backbone network is used to extract features from the coal gangue image, and includes multiple parallel deep residual blocks. Each parallel deep residual block includes a stacked nonlinear fitting layer F(x), which is represented as follows: ,in, This represents the underlying mapping, and x represents the input image of this parallel deep residual block; The head network is used to perform multi-scale feature fusion on the features extracted by the backbone network, and outputs the corresponding coal gangue detection results through the output layer; The parallel deep residual block includes a first stacked layer, a second stacked layer, and a third stacked layer, each of which includes a nonlinear fitting layer. The output of the third stacked layer is added to the output of the second stacked layer via a first addition operation. The output of the first stacked layer is added to the output of the first addition operation via a second addition operation. The head network comprises a feature pyramid network and a path aggregation network connected in sequence; the feature pyramid network comprises a first convolutional layer, a first upsampling operator, a first feature fusion layer, a first downsampling layer, a second convolutional layer, a second upsampling operator, and a second feature fusion layer; the output of the first convolutional layer is connected to the input of the first feature fusion layer via the first upsampling operator, the output of the second convolutional layer is connected to the input of the second feature fusion layer via the second upsampling operator, and the output of the first feature fusion layer is connected to the input of the second convolutional layer via the first downsampling layer; the first convolutional layer, the first feature fusion layer, and the second feature fusion layer are all connected to the output of the backbone network; The path aggregation network includes a second downsampling layer, a third downsampling layer, a fourth downsampling layer, a third convolutional layer, a fourth convolutional layer, a third feature fusion layer, and a fourth feature fusion layer. The input of the second downsampling layer is connected to the second feature fusion layer, and its output is connected to the third convolutional layer. The input of the third feature fusion layer is connected to the third convolutional layer and the second convolutional layer, and its output is connected to the third downsampling layer. The output of the third downsampling layer is connected to the input of the fourth feature fusion layer via the fourth convolutional layer. The output of the first convolutional layer is connected to the input of the fourth feature fusion layer, and the output of the fourth feature fusion layer is connected to the fourth downsampling layer. The outputs of the second downsampling layer, the third downsampling layer, and the fourth downsampling layer are all connected to the output layer.

2. The lightweight coal gangue detection model based on parallel deep residual networks as described in claim 1, characterized in that, The parallel deep residual block includes a first stacked layer and a second stacked layer, and both the first stacked layer and the second stacked layer include a nonlinear fitting layer. The outputs of the first stacked layer, the outputs of the second stacked layer, and the input image of the parallel depth residual block are added together by a third addition operation.

3. The lightweight coal gangue detection model based on parallel deep residual networks as described in claim 1, characterized in that, The backbone network also includes convolutional layers for upscaling the input image, the output of which is connected to the input of the parallel depth residual block.

4. The lightweight coal gangue detection model based on parallel deep residual networks as described in claim 1, characterized in that, Both the first upsampling operator and the second upsampling operator include an upsampling prediction module and a feature recombination module. The upsampling prediction module is used to predict the upsampling kernel and perform normalization processing. The feature recombination module is used to reassemble the features in the local region and map the position in the output feature map back to the input feature map.

5. A training method for a lightweight coal gangue detection model based on a parallel deep residual network, characterized in that, Training the coal gangue detection model as described in any one of claims 1 to 4 includes: Collect coal gangue images and annotate the coal gangue images to construct a dataset; The dataset is divided into a training set and a test set according to a certain ratio; Based on the training set, the hyperparameters of the coal gangue detection model are optimized using the LookAhead optimizer to obtain a trained coal gangue detection model. Model evaluation metrics were selected, and the recognition performance of the coal gangue detection model was evaluated based on the test set.

6. The training method for the lightweight coal gangue detection model based on parallel deep residual networks as described in claim 5, characterized in that, The model evaluation metrics include average precision, accuracy, and recall, where: The formula for calculating the average accuracy is: The formula for calculating the accuracy is: The formula for calculating the recall rate is: In the formula: The total number of input images, For the number of categories, For the first The accuracy of each image, For from the first To the Changes in image recall TP This indicates that both the predicted and actual categories are coal gangue. FN This indicates that both the predicted and actual categories are non-coal gangue. FP This indicates that the predicted category is coal gangue and the actual category is non-coal gangue. TN This indicates that the predicted category is non-coal gangue and the actual category is coal gangue.

7. The training method for the lightweight coal gangue detection model based on parallel deep residual networks as described in claim 5, characterized in that, Based on the training set, the hyperparameters of the coal gangue detection model are optimized using the LookAhead optimizer to obtain a trained coal gangue detection model, including: A multi-branch network is constructed and trained to transfer the weights of the multi-branch network to a single-branch network during inference. The multi-branch network includes parallel identity mapping and convolutional branches, which are added to a batch normalization (BN) layer. The single-branch network includes two convolutional layers connected by an activation function.

Images