A multi-target detection method for a mine excavation environment
By creating a custom dataset and improving the YOLOv5 network model, the difficulty of multi-target detection in mining environments was solved, enabling rapid and accurate identification of targets such as excavators and trucks, thus improving detection efficiency and adaptability.
Patent Information
- Application Number
- CN202211276786.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-18
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2042-10-18
AI Technical Summary
Mining environments are harsh, and existing target detection algorithms struggle to detect targets under conditions of poor visibility, such as dust, rain, and fog, which hinders the realization of automated and unmanned operations.
Using a self-made mining environment dataset, a YOLOv5 network model was built and improved. A lightweight ECA attention mechanism and SPPF layer were added to perform multi-object detection, including the identification of excavators, trucks and materials. An object detection model was generated through iterative training.
It enables rapid and accurate identification of multiple targets in mining environments, meets real-time requirements, improves detection performance and dataset adaptability, and is suitable for various harsh working environments.
Smart Images
Figure CN115661704B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of excavator target detection technology, and in particular to a multi-target detection method for mining excavation environments. Background Technology
[0002] Mining is one of the world's main methods of mineral extraction. Due to the harsh environment of mining sites, prolonged high-intensity work amidst large amounts of dust and noise severely harms the physical and mental health of excavation equipment operators. Furthermore, inefficiency and safety accidents frequently occur due to inexperienced operators and improper handling. Therefore, the demand for automation and unmanned operation of mining equipment is increasingly urgent. Multi-target detection methods in the mining environment are a crucial foundation for achieving automated and unmanned operations.
[0003] With the widespread application of deep learning in the field of vision, an increasing number of advanced algorithms are being used for multi-object detection. Deep learning-based object detection algorithms can be broadly categorized into two-stage algorithms based on candidate regions and one-stage algorithms based on regression. Classic methods for the former include R-CNN, Fast R-CNN, and Faster R-CNN, while classic methods for the latter include YOLO and SSD. This invention addresses the difficulties and low efficiency of object detection in harsh mining environments with poor visibility, such as those involving dust, rain, and fog. It proposes a YOLOv5-based object detection method for mining environments. YOLOv5 boasts a fast inference speed and a lightweight network structure, enabling it to quickly and accurately complete multi-object detection tasks in mining environments. Summary of the Invention
[0004] To address the shortcomings of existing technologies in target detection, such as difficulty and inefficiency, this invention proposes a multi-target detection method for mining environments, the technical solution of which is as follows:
[0005] A multi-target detection method for mining environments includes the following steps:
[0006] S1. Create a custom mining environment dataset and preprocess it;
[0007] S2. Use annotation tools to annotate each image and divide the dataset into training and testing sets;
[0008] S3. Build and modify the YOLOv5 network model;
[0009] The YOLOv5 network model includes an input module, a backbone module, a neck module, and an output module. The backbone module is structured as follows: a first convolutional layer, a second convolutional layer, a first C3 feature extraction layer, a third convolutional layer, a second C3 feature extraction layer, a fourth convolutional layer, a third C3 feature extraction layer, a fifth convolutional layer, a fourth C3 feature extraction layer, an ECA attention mechanism layer, and an SPPF layer are connected sequentially.
[0010] This module extracts features from a self-made mining environment dataset, resulting in feature maps of three sizes. These feature maps are output by the second C3 feature extraction layer, the third C3 feature extraction layer, and the SPPF layer, respectively, and then fed into the neck module for fusion.
[0011] The structure of the neck module is as follows: a first convolutional layer, a first upsampling layer, a first fusion layer, a first C3 feature extraction layer, a second convolutional layer, a second upsampling layer, a second fusion layer, a second C3 feature extraction layer, a third convolutional layer, a third fusion layer, a third C3 feature extraction layer, a fourth convolutional layer, a fourth fusion layer, and a fourth C3 feature extraction layer are connected in sequence.
[0012] The feature map of the SPPF layer in the backbone module is fed into the first convolutional layer of the neck module; the feature map of the first convolutional layer in the neck module is upsampled by the first upsampling layer and then fused with the feature map of the third C3 feature extraction layer in the backbone module in the first fusion layer, and then fed into the fourth fusion layer to fuse with the feature map of the fourth convolutional layer; the feature map of the second convolutional layer in the neck module is upsampled by the second upsampling layer and then fused with the feature map of the second C3 feature extraction layer in the backbone module in the second fusion layer, and then fed into the third fusion layer to fuse with the feature map of the third convolutional layer.
[0013] The input feature maps of three sizes are fused at multiple scales to output three predicted feature maps; the second C3 feature extraction layer outputs the third predicted feature map, the third C3 feature extraction layer outputs the second predicted feature map, and the fourth C3 feature extraction layer outputs the first predicted feature map.
[0014] The output module includes three convolutional layers, which convolve the three predicted feature maps output by the neck module to output three features.
[0015] S4. Feed the training set into the modified YOLOV5 network model for iterative training to obtain the target detection model in the mining environment, and output the corresponding target detection box, recognition type and pixel coordinates.
[0016] Furthermore, the specific content of step S1 is as follows:
[0017] The self-made mining environment dataset, created through web scraping, photography, and continuous video capture by cameras, includes images of mine excavation under various conditions such as lighting, nighttime, dust, and rain / fog.
[0018] The specific details of step S2 are as follows:
[0019] The labeling tool used is labelimg, which converts the labeled data into VOC format and generates files with the .xml extension. YOLOv5 training requires YOLO format files with the .txt extension, so the .xml label files are converted to .txt files, and then divided into training and test sets.
[0020] The specific details of step S3 are as follows:
[0021] In the backbone module, the original SPP layer is changed to an SPPF layer. The three large-size pooling cores of 5×5, 9×9, and 13×13 used in the SPP layer are replaced by multiple small-size pooling cores. The 9×9 pooling core is replaced by two 5×5 pooling cores, and the 13×13 pooling core is replaced by three 5×5 pooling cores.
[0022] The SPPF layer sequentially passes through a first convolutional layer, a MaxPool max pooling layer, a concat connection layer, and a second convolutional layer. The MaxPool max pooling layer contains four parallel channels, which are as follows: the first channel performs a max pooling operation with a 5×5 kernel, the second channel performs a max pooling operation with two 5×5 kernels, the third channel performs a max pooling operation with three 5×5 kernels, and the fourth channel does not perform any operation. The four channels are then concat-connected, and the output of the second convolutional layer is fed into the neck module.
[0023] The backbone module adds a lightweight ECA attention mechanism layer to the original network. It transforms the input feature map from a matrix of [h,w,c] into a vector of [1,1,c] through global average pooling. It calculates the adaptive one-dimensional convolution kernel size kernel_size based on the number of channels in the feature map. The kernel_size is used in the one-dimensional convolution to obtain the weight for each channel of the feature map. The normalized weights are multiplied with the original input feature map channel by channel to generate a weighted feature map.
[0024] The size of the convolution kernel is obtained through an adaptive function:
[0025]
[0026] kernel_size is the kernel size, c is the number of channels, ||odd indicates that kernel_size can only be an odd number, γ = 2, b = 1
[0027] The specific details of step S4 are as follows:
[0028] Input a 640-pixel x 640-pixel image, set the batch size to 4, and the number of training iterations to 100. Obtain the test results by inputting the optimal weights and the test image.
[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0030] 1. The multi-target detection method for mining excavation environment described in this invention can simultaneously identify multiple targets in the mining excavation environment, including excavators, trucks, materials, etc., and calculate the pixel coordinates of each target to obtain the relative positional relationship of each target.
[0031] 2. The multi-target detection method for mining excavation environment described in this invention uses a self-built dedicated mining excavation environment dataset, and is trained by manual classification and labeling. The trained model has a small memory footprint and fast detection speed, which can meet the real-time requirements for multi-target detection in mining excavation environment.
[0032] 3. The mining environment dataset collected by this invention includes various harsh working environments such as rain, fog, night, and dust. The dataset is rich in scope and improves the target detection capability under harsh working environments.
[0033] 4. The multi-target detection method for mining environments described in this invention improves the neural network by adding a lightweight ECA attention mechanism. The lightweight ECA attention mechanism adds only a small number of parameters but achieves significant performance gains, making the neural network more focused on the detection target and improving the detection effect. Attached Figure Description
[0034] Figure 1 This is a flowchart of the multi-target detection method for the mining environment described in this invention;
[0035] Figure 2 This refers to a portion of the dataset described in this invention;
[0036] Figure 3 This is a classification diagram of the dataset described in this invention;
[0037] Figure 4 This is a flowchart illustrating the overall training process of the YOLOv5 network model described in this invention.
[0038] Figure 5 This is a schematic diagram of the YOLOv5 network structure with added EAC attention mechanism as described in this invention;
[0039] Figure 6 The figure shows the test results of the method described in this invention;
[0040] Figure 7 This is a PR curve diagram of the method described in this invention;
[0041] Figure 8 This is a diagram of the SPPF layer structure in this invention. Detailed Implementation
[0042] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0043] like Figure 1 As shown, a multi-target detection method for mining environments includes the following steps:
[0044] S1. Create a custom mining environment dataset and preprocess it;
[0045] By using web crawlers, photography, and camera footage to capture continuous video, and then cutting the video into images every 10 frames, a total of 15,000 images of the mining environment were collected. Figure 2 As shown, the collected images include mining scenes under various conditions such as lighting, nighttime, dust, and rain / fog. Data processing techniques such as Mosaic data augmentation and adaptive scaling were employed. The images were randomly flipped horizontally or vertically, cropped, and scaled. Adaptive scaling involved first reducing the image size, then adaptively adding a small amount of black borders to the input image to create a square, and finally scaling the image uniformly to (640, 640) before feeding it into the YOLOv5 network for learning. Mosaic data augmentation involved randomly cropping and scaling four images, then randomly arranging and stitching them together to form a single image.
[0046] S2. Use annotation tools to annotate each image and divide the dataset into training and testing sets;
[0047] The aforementioned mining environment dataset adopts the VOC dataset format and uses the labelimg annotation tool; the dataset labels are as follows: Figure 3As shown, there are six types: excavation (excavator), truck (loader), sand (sand), soil (soil), stone (stone), and colliery (coal). After labeling, the Labelimg file becomes a VOC format, generating a .xml file. YOLOv5 training requires YOLO format with a .txt extension; therefore, the .xml label file is converted to a .txt file, and then divided into training and test sets. The labeled dataset is divided into training and test sets in an 8:2 ratio, with 12,000 images in the training set and 3,000 images in the test set.
[0048] S3. Build and modify the YOLOv5 network model;
[0049] S31. Building the YOLOv5 network model
[0050] The YOLOv5 network model includes an input module, a backbone module, a neck module, and an output module. The backbone module comprises five convolutional layers (conv), four C3 feature extraction layers, one attention layer, and one SPPF layer. It extracts features from a self-made mining environment dataset to obtain feature maps of three sizes, which are then fed into the neck module for fusion. The neck module comprises four convolutional layers (conv), three C3 feature extraction layers, two upsampling layers, and four fusion layers. It performs multi-scale fusion of the three input feature maps to output three predicted feature maps. The output module comprises three convolutional layers (conv), which convolve the three predicted feature maps output from the neck module.
[0051] The overall training flowchart of the YOLOv5 network model is as follows: Figure 4 As shown:
[0052] The input module takes an image of (640, 640, 3) as input.
[0053] The backbone module is structured as follows: a first convolutional layer, a second convolutional layer, a first C3 feature extraction layer, a third convolutional layer, a second C3 feature extraction layer, a fourth convolutional layer, a third C3 feature extraction layer, a fifth convolutional layer, a fourth C3 feature extraction layer, an ECA attention mechanism layer, and an SPPF layer are connected sequentially.
[0054] The feature maps of the three sizes are output by the second C3 feature extraction layer, the third C3 feature extraction layer and the SPPF layer respectively and are fed into the neck module for fusion; the feature map size output by the second C3 feature extraction layer is [80, 80, 128], the feature map size output by the third C3 feature extraction layer is [40, 40, 256], and the feature map size output by the SPPF layer is [20, 20, 512].
[0055] The neck module is structured as follows: a first convolutional layer, a first upsampling layer, a first fusion layer, a first C3 feature extraction layer, a second convolutional layer, a second upsampling layer, a second fusion layer, a second C3 feature extraction layer, a third convolutional layer, a third fusion layer, a third C3 feature extraction layer, a fourth convolutional layer, a fourth fusion layer, and a fourth C3 feature extraction layer are connected sequentially.
[0056] In the neck module, the feature maps of the first convolutional layer with size [20, 20, 256] and the fourth convolutional layer with size [20, 20, 256] are fused in the fourth fusion layer; the feature maps of the second convolutional layer with size [40, 40, 128] and the third convolutional layer with size [40, 40, 128] are fused in the third fusion layer.
[0057] In the backbone module, the SPPF layer feature map with size [20, 20, 512] is fed into the first convolutional layer of the neck module; the feature map of the second C3 feature extraction layer with size [80, 80, 128] in the backbone module and the feature map of the second convolutional layer of the neck module are upsampled by the second upsampling layer and then fused in the second fusion layer; the feature map of the third C3 feature extraction layer with size [40, 40, 256] in the backbone module and the feature map of the first convolutional layer of the neck module are upsampled by the first upsampling layer and then fused in the first fusion layer of the neck module.
[0058] The output of the second C3 feature extraction layer is a third predicted feature map with a size of [80, 80, 128], a second predicted feature map with a size of [40, 40, 256], and a first predicted feature map with a size of [40, 40, 512];
[0059] The output module consists of three independent convolutional layers, with each predicted feature map being output through a separate convolutional layer.
[0060] S32. Modify the YOLOv5 network model
[0061] First, the backbone module adds a lightweight ECA attention mechanism layer to the original network. As shown in the figure, the input feature map is transformed from a matrix of [20, 20, 512] into a vector of [1, 1, 512] through global average pooling. The adaptive one-dimensional convolution kernel size kernel_size is calculated based on the number of channels of the feature map. The kernel_size is used in the one-dimensional convolution to obtain the weight for each channel of the feature map. The normalized weights are multiplied with the original input feature map channel by channel to generate a weighted feature map.
[0062] The size of the convolution kernel is obtained through an adaptive function:
[0063]
[0064] kernel_size is the kernel size, c is the number of channels, ||odd means that kernel_size can only be an odd number, γ=2, b=1;
[0065] Second, the original SPP layer in the backbone module is changed to an SPPF layer. The three large-size pooling cores of 5×5, 9×9, and 13×13 used in the SPP layer are replaced by multiple small-size pooling cores. The 9×9 pooling core is replaced by two 5×5 pooling cores, and the 13×13 pooling core is replaced by three 5×5 pooling cores.
[0066] The SPPF layer sequentially passes through a first convolutional layer, a MaxPool max pooling layer, a concat connection layer, and a second convolutional layer. The MaxPool max pooling layer contains four parallel channels, which are as follows: the first channel performs a max pooling operation with a 5×5 kernel, the second channel performs a max pooling operation with two 5×5 kernels, the third channel performs a max pooling operation with three 5×5 kernels, and the fourth channel does not perform any operation. The four channels are then concat-connected, and the output of the second convolutional layer is fed into the neck module.
[0067] S33. Constructing the loss function
[0068] The YOLOv5 network loss function consists of three parts: classification loss, localization loss, and confidence loss. The localization loss is calculated using GIou_loss, which represents the error between the predicted and labeled bounding boxes. The confidence and classification losses are calculated using BCEWithLogitsLoss. The localization loss function GIou_loss is calculated as follows:
[0069]
[0070] Localization_Loss=1-GIoU_loss1
[0071] Where M is the predicted bounding box, N is the ground truth bounding box, let Q denote the smallest bounding rectangle containing M and N, P is the intersection of M and N, and R is the union of M and N;
[0072] The YOLOv5 network loss function formula is:
[0073] Loss 总 =θ1×Localization_Loss+θ2×Localization_Loss+θ3×Confidence_Loss, where θ1, θ2, and θ3 are weighting coefficients;
[0074] S34. Modify the data configuration file.
[0075] Modify the corresponding common.yaml file in the data directory, including changing the paths of the training set and test set, changing the number of categories to be detected to 6, and changing the names of the categories to be identified to excavator, truck, sand, soil, stone, and colliery.
[0076] S35. Modify the model configuration file.
[0077] Modify the yolov5s.yaml file in the models directory and change the number of categories to be detected to 6.
[0078] S4. Feed the training set into the YOLOV5 network for iterative training to obtain the target detection model in the mining environment, and output the corresponding target detection box, recognition type and pixel coordinates.
[0079] Input a 640 pixel × 640 pixel image, set the batch size to 4, and the number of training iterations to 100 epochs; obtain the test results by inputting the optimal weights and the test image, such as... Figure 6 As shown.
[0080] Evaluate the performance of the method of the present invention for multi-target detection in mining environments.
[0081] The model evaluation metrics used in this invention include: mean AP, precision, and recall; where precision represents the proportion of examples classified as positive that are actually positive, and can be expressed by the formula... Therefore, recall represents the proportion of all positive examples that are correctly identified, and can be expressed by the formula: The results show that TP is the number of samples that the model predicts as positive and whose true class is also positive, FP is the number of samples that the model predicts as positive but whose true class is negative, and FN is the number of samples that the model predicts as negative but whose true class is positive.
[0082] PR curve diagram as follows Figure 7 As shown in the PR curve graph, P stands for Precision and R stands for Recall, representing the relationship between precision and recall. Recall is set on the horizontal axis, and precision is set on the vertical axis. The area under the PR curve is AP, and the average AP across all categories is mAP. mAP@0.5&mAP@0.5:0.95: , where mAP is used to measure recognition accuracy. The number after @ represents the threshold for determining whether an IOU is positive or negative. @0.5:0.95 means taking the average of the thresholds of 0.5:0.05:0.95.
[0083] The highest values of Precision, Recall, mAP_0.5, and mAP_0.5:0.95 of the model trained by this invention can reach 0.98282, 0.96807, 0.98155, and 0.73653, respectively.
[0084] This invention provides a multi-target detection method for mining environments. There are many methods and approaches to implement this technical solution. The above description is only a preferred embodiment of this invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.
Claims
1. A multi-target detection method for mining environments, comprising the following steps: S1. Create a custom mining environment dataset and preprocess it; S2. Use annotation tools to annotate each image and divide the dataset into training and testing sets; S3. Build and modify the YOLOv5 network model; S4. Feed the training set into the modified YOLOV5 network model for iterative training to obtain the target detection model in the mining environment, and output the corresponding target detection box, recognition type and pixel coordinates. The YOLOv5 network model includes an input module, a backbone module, a neck module, and an output module. The backbone module is structured as follows: a first convolutional layer, a second convolutional layer, a first C3 feature extraction layer, a third convolutional layer, a second C3 feature extraction layer, a fourth convolutional layer, a third C3 feature extraction layer, a fifth convolutional layer, a fourth C3 feature extraction layer, an ECA attention mechanism layer, and an SPPF layer are connected sequentially. This module extracts features from a self-made mining environment dataset, resulting in feature maps of three sizes. These feature maps are output by the second C3 feature extraction layer, the third C3 feature extraction layer, and the SPPF layer, respectively, and then fed into the neck module for fusion. The structure of the neck module is as follows: a first convolutional layer, a first upsampling layer, a first fusion layer, a first C3 feature extraction layer, a second convolutional layer, a second upsampling layer, a second fusion layer, a second C3 feature extraction layer, a third convolutional layer, a third fusion layer, a third C3 feature extraction layer, a fourth convolutional layer, a fourth fusion layer, and a fourth C3 feature extraction layer are connected in sequence. The feature map of the SPPF layer in the backbone module is fed into the first convolutional layer of the neck module; the feature map of the first convolutional layer in the neck module is upsampled by the first upsampling layer and then fused with the feature map of the third C3 feature extraction layer in the backbone module in the first fusion layer, and then fed into the fourth fusion layer to fuse with the feature map of the fourth convolutional layer; the feature map of the second convolutional layer in the neck module is upsampled by the second upsampling layer and then fused with the feature map of the second C3 feature extraction layer in the backbone module in the second fusion layer, and then fed into the third fusion layer to fuse with the feature map of the third convolutional layer. The input feature maps of three sizes are fused at multiple scales to output three predicted feature maps; the second C3 feature extraction layer outputs the third predicted feature map, the third C3 feature extraction layer outputs the second predicted feature map, and the fourth C3 feature extraction layer outputs the first predicted feature map. The output module includes three convolutional layers, which convolve the three predicted feature maps output by the neck module respectively.
2. The multi-target detection method for mining environments according to claim 1, characterized in that, The specific details of step S1 are as follows: The self-made mining environment dataset, created through web scraping, photography, and continuous video capture by cameras, includes images of mine excavation under various conditions such as lighting, nighttime, dust, and rain / fog.
3. The multi-target detection method for mining environments according to claim 1, characterized in that, The specific details of step S2 are as follows: The labeling tool used is labelimg, which converts the labeled data into VOC format and generates files with the .xml extension. YOLOv5 training requires YOLO format files with the .txt extension, so the .xml label files are converted to .txt files, and then divided into training and test sets.
4. The multi-target detection method for mining environments according to claim 1, characterized in that, In the backbone module of the YOLOv5 network model, the SPPF layer is composed of a first convolutional layer, a max pooling layer, a concat connection layer, and a second convolutional layer connected in sequence. The max pooling layer contains four parallel channels, which are as follows: the first channel performs a max pooling operation with a 5×5 kernel, the second channel performs a max pooling operation with two 5×5 kernels, the third channel performs a max pooling operation with three 5×5 kernels, and the fourth channel does not perform any operation. The four channels are then concat connected, and the output of the second convolutional layer is fed into the neck module.
5. The multi-target detection method for mining environments according to claim 1, characterized in that, In the backbone module of the YOLOv5 network model, the ECA attention mechanism layer transforms the input feature map from a matrix of [h,w,c] into a vector of [1,1,c] through global average pooling. It calculates the adaptive one-dimensional convolution kernel size kernel_size based on the number of channels in the feature map, and uses kernel_size in the one-dimensional convolution to obtain the weight for each channel of the feature map. The normalized weights are multiplied with the original input feature map channel by channel to generate a weighted feature map. The size of the convolution kernel is obtained through an adaptive function: kernel_size is the kernel size, c is the number of channels, | |odd means that kernel_size can only be an odd number, γ = 2, b = 1.
6. The multi-target detection method for mining environments according to claim 1, characterized in that, The specific details of step S4 are as follows: Input a 640-pixel x 640-pixel image, set the batch size to 4, and the number of training iterations to 100. Obtain the test results by inputting the optimal weights and the test image.