Mine car carrying capacity detection system and method based on YOLOv4 target detection model

Abstract

The present invention belongs to the field of machine vision technology, specifically a mine car load detection system and method based on the YOLOv4 target detection model. The mine car load detection system of the present invention includes a backbone feature extraction module, an enhanced feature extraction module and a prediction module; the backbone feature extraction module adopts an inverse residual unit containing a depth-separable convolution to extract network features to reduce the amount of network parameters and calculations; the enhanced feature extraction module uses SPPNet to increase the receptive field, extract rich semantic information, and use the improved PANet to shorten the fusion path between feature layers; the prediction module converts the acquired features into prediction results. The system has high detection accuracy, fast detection speed, and small memory usage, which improves the detection effect of large target objects and can be well applied in the field of mine information management.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision, and in particular relates to a mine car load detection system and method based on a YOLOv4 target detection model. Background Art

[0002] Object detection is an important direction in the field of computer vision and has always been a key research topic in academia. It is different from classification tasks in that it not only needs to identify the category of the target object in the image, but also needs to determine their specific location. It is a combination of classification and positioning. Depending on whether it is necessary to extract candidate regions from the input image, it can be divided into two categories: two-stage algorithms and one-stage algorithms. Two-stage algorithms have high accuracy but low speed. For example, the R-CNN series includes three iterative versions: R-CNN[1], FastR-CNN[2], and Faster R-CNN[3]. The one-stage algorithm combines the determination of the target object category and the determination of the location into one step, and the detection speed is faster. Take the YOLO series as an example, including four versions: v1[4], v2[5], v3[6], and v4[7]. YOLOv4 uses the new network structure CSPDarknet53[8] and SPPNet[9] based on YOLOv3, which greatly improves the performance of object detection.

[0003] With the vigorous development of technologies such as big data and artificial intelligence, the concept of "smart mines" has been mentioned more and more. Nowadays, the load status of vehicles in earthwork stripping projects is counted manually by counting and detecting, which requires workers to observe the situation on site at all times. Not only is the work efficiency of workers low, but it also increases the operating expenses of coal mining companies. Therefore, using the target detection method to quickly detect the load status of mine cars can effectively reduce the danger of workers' work and improve the level of information management of mines. At the same time, in actual engineering application scenarios, the detection algorithm needs to meet the characteristics of high detection accuracy, fast detection speed, and small memory usage of the detection model in addition to high detection accuracy. Summary of the invention

[0004] The object of the present invention is to provide a mine car carrying capacity detection system and method with high detection accuracy, fast detection speed and small memory occupation of the detection model.

[0005] The mine car carrying capacity detection system provided by the present invention is based on the YOLOv4 target detection model, including a backbone feature extraction module, an enhanced feature extraction module and a prediction module, wherein:

[0006] The backbone feature extraction module uses an inverse residual unit with depthwise separable convolution to extract network features to reduce the amount of network parameters and calculations;

[0007] The enhanced feature extraction module uses SPPNet to increase the receptive field and extract rich semantic information; and uses the improved PANet to shorten the fusion path between feature layers;

[0008] The prediction module converts the acquired features into prediction results, including determining whether an object is contained in a priori box preset on the grid and the type of the object, and adjusting the center position, width and height of the priori box to finally obtain a prediction result.

[0009] Furthermore, in the backbone feature extraction module, an inverse residual unit with depthwise separable convolution is used to extract network features. By continuously downsampling, the width and height of the feature layer are continuously compressed, and the number of channels is continuously expanded to obtain three effective feature layers, namely, feature layer 1, feature layer 2, and feature layer 3. According to the generally large characteristics of the mine car models in the mine car dataset, finally two effective feature layers, feature layer 2 and feature layer 3, are selected.

[0010] Furthermore, in the enhanced feature extraction module, the SPPNet is used to increase the receptive field. Specifically, SPPNet first performs three convolutions on the feature layer 3, and then performs maximum pooling on the convolution results using pooling kernels of 3×3, 7×7, and 11×11. The pooling results are concatenated with the effective feature layer before pooling, and finally the output result of SPPNet is obtained.

[0011] Furthermore, in the enhanced feature extraction module, the improved PANet refers to fusing the output result of SPPNet with feature layer 2, and deleting the relevant processing units of PANet in YOLOv4 for feature layer 1 of the backbone feature extraction module, including convolution, splicing, and upsampling units, thereby shortening the fusion path between feature layers.

[0012] Furthermore, the prediction module clusters the labeled real boxes of the mine carts in the training set of the mine cart data set through the K-means++ algorithm

[10] to obtain the relationship between the average intersection-over-union ratio of the real boxes and the prior boxes and the number of prior boxes. By weighing the relationship between the average intersection-over-union ratio of the real boxes and the prior boxes and the number of prior boxes, 5-8 prior boxes are obtained. During the prediction, the center position, width and height of the prior box are adjusted to finally obtain the prediction result.

[0013] The present invention also provides a method for detecting the carrying capacity of a mine car based on a YOLOv4 target detection model, which is characterized by comprising the following steps:

[0014] Step S1: Obtain the original mine car dataset, fill it with data enhancement methods, and divide the final mine car dataset into training set A, validation set B and test set C;

[0015] Step S2: Establishing a mine car carrying capacity detection system, which consists of a main feature extraction module, an enhanced feature extraction module and a prediction module;

[0016] Step S3: The mine car load detection system is trained through the training set A, the verification set B is used to preliminarily evaluate the learning ability of the mine car load detection system and help adjust the model parameters, and the test set C is used to finally evaluate the ability of the mine car load detection system.

[0017] Furthermore, the original mine cart dataset contains two categories, namely, empty mine carts and full mine carts.

[0018] Furthermore, the data enhancement means include random flipping, random brightness, random rotation, and random cropping.

[0019] Furthermore, the division standard of the mine car data set is training set: validation set: test set = 5:2:3.

[0020] By adopting the technical scheme of the present invention, the beneficial effects of the present invention are as follows: the system and method adopt a deep learning framework based on PyTorch. First, the backbone feature extraction module uses an inverted residual structure containing deep separable convolution to enhance the learning ability of the network and reduce the amount of network calculation; secondly, the improved PANet is used to simplify the network structure and reduce the number of network parameters; finally, K-means++ is used to cluster the mine car data set, which can effectively ensure the coverage of the prior frame on the mine car and improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is an example diagram of data enhancement in an embodiment of the present invention.

[0022] Figure 2 This is a diagram of the depthwise separable convolution structure in an embodiment of the present invention.

[0023] Figure 3 2 is a structural diagram of an inverted residual unit in an embodiment of the present invention.

[0024] Figure 4 This is the visualization result of the samples on the test set in the embodiment of the present invention.

[0025] Figure 5 The figure is a flowchart of the steps of a mine car load detection method based on the YOLOv4 target detection model in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0027] Figure 1 This is an example diagram of data enhancement in an embodiment of the present invention, such as Figure 1 As shown, from left to right are the original image, random flip, random brightness, random rotation, and random cropping.

[0028] Figure 2 is a diagram of a depth-separable convolution structure in an embodiment of the present invention, such as Figure 2 As shown in the figure, the input is first filtered using a 3×3 convolution filter, followed by batch normalization and activation function, and then a 1×1 convolution kernel is used for feature fusion, and finally the final output is obtained through batch normalization and activation function.

[0029] Figure 3 is a structural diagram of an inverted residual unit in an embodiment of the present invention, such as Figure 3 As shown, the inverse residual unit uses a point-by-point convolution with a kernel size of 1×1 at the beginning and end, which respectively increases the number of channels by 6 times and reduces the dimension; in the middle, a deep convolution with a kernel size of 3×3 is used for feature extraction.

[0030] Figure 4 is the visualization result of the samples on the test set in the embodiment of the present invention, such as Figure 4 As shown in the figure, the model detects the input image, identifies and determines the empty and full trams in the target, locates the target, draws a frame around the specific location, and adds a label and the probability of judgment. Among them, the empty tramcar is represented by "empty_tramcar" and the full tramcar is represented by "full_tramcar".

[0031] The mine car carrying capacity detection method based on the YOLOv4 target detection model provided by the present invention is as follows: Figure 5 As shown, the following steps are included.

[0032] The method for detecting the carrying capacity of a mine car based on the YOLOv4 target detection model provided by the present invention comprises the following steps:

[0033] Step S1: Obtain the original mine car dataset, fill it with data enhancement methods, and divide the final mine car dataset into training set A, validation set B and test set C;

[0034] Step S2: Establishing a mine car carrying capacity detection system, which consists of a main feature extraction module, an enhanced feature extraction module and a prediction module;

[0035] Step S3: The mine car load detection system is trained through the training set A, the verification set B is used to preliminarily evaluate the learning ability of the mine car load detection system and help adjust the model parameters, and the test set C is used to finally evaluate the ability of the mine car load detection system.

[0036] The technical solution of the present invention is effective and can achieve a detection accuracy of more than 90% for the load of mine vehicles. Compared with the YOLOv4 target detection model before the improvement, the detection speed is increased by 150% and the model size is reduced by 80%. It has the characteristics of high detection accuracy, fast detection speed and small memory usage. It has a good detection effect on large target objects and can be well applied in the field of mine information management.

[0037] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

[0038] References

[0039] [1]Girshick R, Donahue J, Darrell T, et al. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[C] / / IEEE ComputerSociety. IEEE Computer Society, 2013.

[0040] [2]Girshick R. Fast R-CNN[J]. Computer Science, 2015.

[0041] [3]Ren S, He K, Girshick R, et al. Faster R-CNN: Towards Real-TimeObject Detection with Region Proposal Networks[J]. IEEE Transactions onPattern Analysis & Machine Intelligence, 2017, 39(6):1137-1149.

[0042] [4]Redmon J, Divvala S, Girshick R, et al. You Only Look Once:Unified, Real-Time Object Detection[J]. Computer Vision & PatternRecognition, 2016.

[0043] [5]Redmon J, Farhadi A. YOLO9000: Better, Faster, Stronger[C] / / IEEEConference on Computer Vision & Pattern Recognition. IEEE, 2017:6517-6525.

[0044] [6]Redmon J, Farhadi A. YOLOv3: An Incremental Improvement[J]. arXive-prints, 2018.

[0045] [7]Bochkovskiy A, Wang C Y, Liao H Y M. Yolov4: Optimal speed andaccuracy of object detection[J]. arXiv preprint arXiv:2004.10934, 2020.

[0046] [8]Wang C Y, Liao H, Yeh I H, et al. CSPNet: A New Backbone that canEnhance Learning Capability of CNN[J]. 2019.

[0047] [9]He K, Zhang X, Ren S, et al. Spatial Pyramid Pooling in DeepConvolutional Networks for Visual Recognition[C] / / IEEE Transactions onPattern Analysis & Machine Intelligence. 2014:1904-16.

[0048]

[10] Vassilvitskii. k-means++: The advantages of careful seeding[C] / / Proceedings of the eighteenth annual ACM-SIAM symposium on Discretealgorithms. 2006: 1027-1035。

Claims

1. A mine car load detection system based on the YOLOv4 target detection model, characterized in that: It includes a backbone feature extraction module, an enhanced feature extraction module and a prediction module, wherein: The backbone feature extraction module uses an inverse residual unit with depthwise separable convolution to extract network features to reduce the amount of network parameters and calculations; The enhanced feature extraction module uses SPPNet to increase the receptive field and extract rich semantic information; and uses the improved PANet to shorten the fusion path between feature layers; The prediction module converts the acquired features into prediction results, including determining whether an object is contained in a priori box preset on the grid and the type of the object, and adjusting the center position, width and height of the priori box to finally obtain a prediction result; In the backbone feature extraction module, an inverse residual unit with depthwise separable convolution is used to extract network features. The width and height of the feature layer are continuously compressed and the number of channels is continuously expanded through continuous downsampling, so as to obtain three effective feature layers: feature layer 1, feature layer 2, and feature layer 3. According to the characteristics that the mine car models in the mine car data set are generally large, finally two effective feature layers, feature layer 2 and feature layer 3, are selected; In the enhanced feature extraction module, the SPPNet is used to increase the receptive field. Specifically, SPPNet first performs three convolutions on the feature layer 3, and then performs maximum pooling on the convolution results using 3×3, 7×7, and 11×11 pooling kernels, and concatenates the pooling results with the effective feature layer before pooling, and finally obtains the output result of SPPNet; In the enhanced feature extraction module, the improved PANet refers to fusing the output result of SPPNet with feature layer 2, and deleting the relevant processing units of PANet in YOLOv4 for feature layer 1 of the backbone feature extraction module, including convolution, splicing, and upsampling units, thereby shortening the fusion path between feature layers; The prediction module clusters the labeled real frames of the mine carts in the training set in the mine cart data set through the K-means++ algorithm to obtain the relationship between the average intersection and union ratio of the real frame and the prior frame and the number of prior frames; by weighing the relationship between the average intersection and union ratio of the real frame and the prior frame and the number of prior frames, 5-8 prior frames are obtained; during prediction, the center position, width and height of the prior frame are adjusted to finally obtain the prediction result.

2. The mine vehicle carrying capacity detection system according to claim 1 is characterized in that: The structure of the depthwise separable convolution includes first filtering the input using a 3×3 convolution filter, then passing batch normalization and an activation function, then using a 1×1 convolution kernel for feature fusion, and finally obtaining the final output through batch normalization and an activation function.

3. The mine vehicle carrying capacity detection system according to claim 2 is characterized in that: The structure of the inverted residual unit includes: using a point-by-point convolution with a kernel size of 1×1 at the beginning and the end to expand the number of channels by 6 times and reduce the dimension respectively; and using a deep convolution with a kernel size of 3×3 to extract features in the middle.

4. A method for detecting the carrying capacity of a mine car based on the mine car carrying capacity detection system according to any one of claims 1 to 3, characterized in that: The specific steps are: Step S1: Obtain the original mine car dataset, fill it with data enhancement methods, and divide the final mine car dataset into training set A, validation set B and test set C; Step S2: Establishing a mine car carrying capacity detection system, including a main feature extraction module, an enhanced feature extraction module and a prediction module; Step S3: The mine car load detection system is trained through the training set A, the verification set B is used to preliminarily evaluate the learning ability of the mine car load detection system and help adjust the model parameters, and the test set C is used to finally evaluate the ability of the mine car load detection system.

5. The method for detecting the carrying capacity of a mine vehicle according to claim 4, characterized in that: The original mine cart dataset contains two categories, namely empty mine carts and full mine carts.

6. The method for detecting the carrying capacity of a mine vehicle according to claim 4, characterized in that: The data enhancement means include random flipping, random brightness, random rotation, and random cropping.

Images