Construction method and device of underground coal mine drill site object detection model

Abstract

The invention provides an underground coal mine drill site object detection model construction method and device, and the method comprises the steps: obtaining a training data set based on a coal mine data set; an improved YOLOv5 model is determined, the improved YOLOv5 model replaces a Focus structure in a backbone network with a common convolution structure, a Swish activation function is replaced with a Leaky ReLU activation function, and a self-adaptive anchor frame calculation module at an input end is replaced with a data self-adaptive anchor frame calculation module; training the improved YOLOv5 model through the training data set to obtain an initial detection model; and performing iterative processing including pruning, quantification and training on the initial detection model to obtain a coal mine underground drill site object detection model supporting DPU hardware deployment. A coal mine underground drill site object detection model with high accuracy and high recognition speed is obtained, and a target detection method supporting DPU hardware deployment and light weight is realized.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a method, device, electronic device and storage medium for constructing an object detection model for an underground drilling site in a coal mine. Background Art

[0002] Drilling in underground coal mines is an important measure to solve gas disasters, water disasters, and hidden geological disasters, and can significantly improve the level of disaster prevention and control in my country's coal mines. In order to monitor the drilling process in real time and improve drilling efficiency, it is necessary to detect targets in underground coal mines. Target detection in underground coal mines is to identify and locate important targets involved in the drilling site. Compared with traditional underground coal mine drilling site target detection methods, the underground coal mine drilling site target detection method based on deep learning can improve the accuracy, timeliness and stability of target detection.

[0003] Coal mine underground images lack detailed features due to the influence of the environment. To ensure underground safety, the model is required to have high detection accuracy and fast reasoning speed. In addition, the current underground hardware equipment with CPU as the computing core cannot provide sufficient computing resources and cannot meet the real-time requirements of target detection. Therefore, the hardware equipment uses DPU, and the existing model is not convenient for DPU deployment.

[0004] Compared with target detection models such as R-CNN and SSD, YOLOv5 uses adaptive anchor frame calculation and multi-scale fusion detection mechanism to quickly and effectively fuse rich high-level semantic information with low-level location information, thereby achieving rapid detection of targets. However, the existing structure of the YOLOV5 model is not supported in the current DPU core, which is not convenient for hardware deployment; the activation function of YOLOv5 is relatively complex, and it is not convenient to accelerate the nonlinear calculation operation through FPGA; in order to adapt to coal mine target detection data, the Anchor anchor frame can also be optimized to improve detection accuracy. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first purpose of this application is to propose a method for constructing an object detection model for an underground drilling site in a coal mine to realize DPU hardware deployment while ensuring the accuracy and speed of target detection and meeting lightweight requirements.

[0007] The second purpose of the present application is to propose a device for constructing an object detection model in a drilling site underground in a coal mine.

[0008] The third objective of the present application is to provide an electronic device.

[0009] A fourth objective of the present application is to provide a computer-readable storage medium.

[0010] A fifth object of the present application is to provide a computer program product.

[0011] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a method for constructing an object detection model for an underground coal mine drilling site, comprising:

[0012] Based on the coal mine data set, obtain a training data set;

[0013] Determine an improved YOLOv5 model, wherein the improved YOLOv5 model replaces the Focus structure in the backbone network with a common convolution structure, replaces the Swish activation function with a Leaky ReLU activation function, and replaces the adaptive anchor frame calculation module at the input end with a data adaptive anchor frame calculation module; wherein the data adaptive anchor frame calculation module determines the number of anchor frames through a clustering algorithm and determines the size of the anchor frame according to the pixel size of the original image in the training data set;

[0014] Training the improved YOLOv5 model through the training data set to obtain an initial detection model;

[0015] The initial detection model is iteratively processed including pruning, quantization and training to obtain a coal mine underground drilling site object detection model that supports DPU hardware deployment.

[0016] In some implementations, the iterative processing of the initial detection model including pruning, quantization, and training to obtain a coal mine underground drilling site object detection model supporting DPU hardware deployment includes:

[0017] The following process is iterated until the number of iterations meets the number threshold, and a coal mine underground drilling field object detection model that supports DPU hardware deployment is obtained;

[0018] Pruning the initial detection model through the Vitis AI sparsity pruner, wherein the pruning process includes generating a sparse model, fine-tuning the sparse model, and deriving the sparse model;

[0019] The pruned model is quantized by the Vitis AI quantizer to obtain a quantized initial detection model; the quantization process uses layer-by-layer data quantization, and quantizes weight data, inter-layer transmission data, intra-layer intermediate results, and activation data respectively;

[0020] The quantized initial detection model is trained using the training data set.

[0021] In some implementations, after obtaining the coal mine underground drilling site object detection model supporting DPU hardware deployment, the method includes:

[0022] Based on the coal mine dataset, obtain a test dataset;

[0023] Inputting the test data set into the coal mine underground drilling site object detection model to obtain a detection result;

[0024] According to the detection results, the model performance of the coal mine underground drilling site object detection model is evaluated.

[0025] In some implementations, after evaluating the model performance of the coal mine drilling site object detection model, the method further includes:

[0026] After determining that the model performance meets the performance index, compiling the coal mine underground drilling field object detection model supporting DPU hardware deployment into a model execution file recognizable by the DPU;

[0027] The model execution file is used as a coal mine underground drilling site object detection model for DPU hardware deployment.

[0028] In some implementations, the data adaptive anchor box calculation module determines the number of anchor boxes by a clustering algorithm, including:

[0029] The data adaptive anchor frame calculation module determines the number of anchor frames through the K-means clustering algorithm.

[0030] In some implementations, the step of training the improved YOLOv5 model using the training data set to obtain an initial detection model includes:

[0031] The improved YOLOv5 model is trained by the training data set using the PyTorch deep learning framework to obtain an initial detection model; wherein the hardware platform supporting the training of the improved YOLOv5 model includes an Intel Xeon Gold5118, 2.30GHz processor, 377GB memory and a GeForce RTX 2080Ti graphics processor.

[0032] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a device for constructing an object detection model for a drilling site in an underground coal mine, comprising:

[0033] A sample acquisition module is used to acquire a training data set based on a coal mine data set;

[0034] A model determination module, used to determine an improved YOLOv5 model, wherein the improved YOLOv5 model replaces the Focus structure in the backbone network with a common convolution structure, replaces the Swish activation function with a Leaky ReLU activation function, and replaces the adaptive anchor frame calculation module at the input end with a data adaptive anchor frame calculation module; wherein the data adaptive anchor frame calculation module determines the number of anchor frames through a clustering algorithm and determines the size of the anchor frame according to the pixel size of the original image in the training data set;

[0035] A model training module, used to train the improved YOLOv5 model through the training data set to obtain an initial detection model;

[0036] The model optimization module is used to perform iterative processing including pruning, quantization and training on the initial detection model to obtain a coal mine underground drilling site object detection model that supports DPU hardware deployment.

[0037] In some implementations, the model optimization module is specifically used to:

[0038] The following process is iterated until the number of iterations meets the number threshold, and a coal mine underground drilling field object detection model that supports DPU hardware deployment is obtained;

[0039] Pruning the initial detection model through the Vitis AI sparsity pruner, wherein the pruning process includes generating a sparse model, fine-tuning the sparse model, and deriving the sparse model;

[0040] The pruned model is quantized by the Vitis AI quantizer to obtain a quantized initial detection model; the quantization process uses layer-by-layer data quantization, and quantizes weight data, inter-layer transmission data, intra-layer intermediate results, and activation data respectively;

[0041] The quantized initial detection model is trained using the training data set.

[0042] In some implementations, the apparatus further includes a performance evaluation module, configured to:

[0043] Based on the coal mine dataset, obtain a test dataset;

[0044] Inputting the test data set into the coal mine underground drilling site object detection model to obtain a detection result;

[0045] According to the detection results, the model performance of the coal mine underground drilling site object detection model is evaluated.

[0046] In some implementations, the apparatus further includes a model compilation module, configured to:

[0047] After determining that the model performance meets the performance index, compiling the coal mine underground drilling field object detection model supporting DPU hardware deployment into a model execution file recognizable by the DPU;

[0048] The model execution file is used as a coal mine underground drilling site object detection model for DPU hardware deployment.

[0049] In some implementations, when the data adaptive anchor box calculation module of the model determination module determines the number of anchor boxes by a clustering algorithm, it is used to:

[0050] The data adaptive anchor frame calculation module determines the number of anchor frames through the K-means clustering algorithm.

[0051] In some implementations, the model training module is specifically used to:

[0052] The improved YOLOv5 model is trained by the training data set using the PyTorch deep learning framework to obtain an initial detection model; wherein the hardware platform supporting the training of the improved YOLOv5 model includes an Intel Xeon Gold5118, 2.30GHz processor, 377GB memory and a GeForce RTX 2080Ti graphics processor.

[0053] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0054] To achieve the above-mentioned purpose, the fourth aspect of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0055] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0056] The construction method, device, electronic device and storage medium of the underground mine drilling site object detection model provided by the present application propose an improved YOLOv5 model based on the hardware characteristics of the DPU, train the improved YOLOv5 model with the coal data set, and perform pruning and quantization and other optimization processing on the trained model to obtain an underground mine drilling site object detection model with high accuracy and high recognition speed, and realize a target detection method that supports DPU hardware deployment and lightweight. The underground mine drilling site object detection model is deployed in hardware to further improve the model reasoning speed. The deployed model meets the real-time reasoning requirements and has a significant acceleration effect.

[0057] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0059] Figure 1 A flow chart of a method for constructing an object detection model for an underground drilling site in a coal mine provided in an embodiment of the present application;

[0060] Figure 2 A structural diagram of the improved YOLOv5 model provided in an embodiment of the present application;

[0061] Figure 3 A structural block diagram of a device for constructing an object detection model for an underground drilling site in a coal mine provided in an embodiment of the present application;

[0062] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0064] Terminology explanation:

[0065] DPU (Data Processing Unit) is a dedicated processor with data as the center.

[0066] PyTorch is an open source Python machine learning library based on Torch for applications such as natural language processing. PyTorch can be seen as numpy with GPU support, but it can also be seen as a powerful deep neural network with automatic differentiation capabilities.

[0067] The following describes the method, device and equipment for constructing an object detection model for an underground drilling site in a coal mine according to an embodiment of the present application with reference to the accompanying drawings.

[0068] Figure 1 A flowchart of a method for constructing an object detection model for an underground drilling site in a coal mine provided in an embodiment of the present application.

[0069] It should be noted that the executor of the method for constructing an object detection model for an underground mine drilling site in the embodiment of the present application is the device for constructing an object detection model for an underground mine drilling site in the embodiment of the present application. The device for constructing an object detection model for an underground mine drilling site can be configured in an electronic device so that the electronic device can execute the function of constructing an object detection model for an underground mine drilling site.

[0070] like Figure 1 As shown, the method for constructing the object detection model of the underground drilling site of a coal mine includes the following steps:

[0071] Step 101: Obtain a training data set based on a coal mine data set.

[0072] In some embodiments, the coal mine dataset uses a dataset publicly available from China University of Mining and Technology. This dataset uses a coal mine underground safety law enforcement recorder to shoot the underground drilling site of the coal mine. Through data cleaning, data annotation, expert on-site inspection and other steps, a standardized target detection dataset for underground drilling sites in coal mines is established. The dataset contains 70,948 images from different drilling locations and environmental background conditions, covering five types of objects: clamps, chucks, coal miners, mine helmets, and drill rods. It can provide strong data support for the research on target detection at underground drilling sites in coal mines, and plays an important role in promoting intelligent monitoring and early warning in underground coal mines.

[0073] As an implementation method, feature data and corresponding recognition results for training an object detection model for an underground drilling site in a coal mine are obtained from a coal mine data set, and the feature data and recognition results are combined into training samples, thereby obtaining a training data set including multiple training samples.

[0074] Step 102, determine an improved YOLOv5 model, the improved YOLOv5 model replaces the Focus structure in the backbone network with a common convolution structure, replaces the Swish activation function with a Leaky ReLU activation function, and replaces the adaptive anchor frame calculation module at the input end with a data adaptive anchor frame calculation module; wherein the data adaptive anchor frame calculation module determines the number of anchor frames through a clustering algorithm and determines the size of the anchor frame according to the pixel size of the original image in the training data set.

[0075] In recent years, deep learning and computer vision related technologies have been widely used in many fields. The YOLO algorithm is a classic algorithm for single-stage target detection, which can achieve fast and high-precision image recognition and target detection. Researchers proposed an improved YOLOv4 network for underground personnel detection based on thermal infrared images, using super-resolution methods to improve the resolution of infrared images and enhancing the generalization ability of the YOLOv4 network through transfer learning. However, the YOLOv4 model has a large number of parameters and complex calculations, and the super-resolution preprocessing method introduces additional calculations, which cannot meet the real-time requirements of personnel detection. In order to improve the computational efficiency of the neural network model, a deep channel attention module was proposed to evaluate the importance of different computing channels of the neural network model, and based on this, the low-importance channels were pruned to reduce the number of model calculation parameters. However, the model compression technology based on channel pruning will reduce the accuracy of model reasoning, which is unacceptable for underground target detection with high precision requirements.

[0076] Based on YOLOV4, researchers proposed the YOLOv5 algorithm, which is the fifth version of the YOLO algorithm update iteration. It is an end-to-end deep learning model that can detect and locate targets directly from raw images. It can be applied to many different scenarios, including autonomous driving, robot perception, image analysis, etc. It uses convolutional neural networks (CNNs) to learn the features of objects in images and uses multi-scale prediction and grid segmentation to detect and locate targets.

[0077] YOLOv5 is mainly composed of input end, backbone network, multi-scale feature fusion network and detection head. The input end uses image preprocessing operations such as Mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling; the backbone network uses CSPDarknet53, and Focus image downsampling slices and CSP (Cross Stage Partial) structure are used in the backbone network to extract the feature information of the image; the multi-scale feature fusion network mainly adopts the feature pyramid structure of FPN+PAN, which effectively integrates the rich semantic information in the high-level feature map and the rich position information in the low-level feature map, and transmits the feature information of objects of different sizes, which can improve the network's prediction ability for objects of different sizes; finally, the detection head performs category and position regression calculations on the multi-scale output.

[0078] The FPGA unified deep learning processing unit DPU puts forward additional requirements and constraints on the deployment model. In order to fully utilize the capabilities of the DPU hardware core, the deployment model needs to be optimized according to the DPU design characteristics. The optimized model network structure is shown in the figure below: Figure 2 The main work points of this application to improve the YOLOv5 model include: replacing the Focus structure, changing the activation function of YOLOv5, optimizing the Anchor anchor box, hardware customization of the model, experimental process and performance analysis.

[0079] The backbone network of YOLOv5 contains a Focus module, whose main function is to divide the input original image into multiple feature map slices, so that the image can retain more and more complete information during the downsampling process. The calculation amount and parameter amount of the Focus slice operation are about 4 times that of ordinary convolution downsampling. Replacing the Focus structure with ordinary convolution will not have a big impact on the model, and this structure is not supported in the current DPU core. If the embedded CPU core is used for calculation, it will become the bottleneck of the end-to-end reasoning of the model. Therefore, the improved YOLOv5 model replaces the Focus structure in YOLOv5 with ordinary convolution.

[0080] The activation function plays a very important role. It can introduce nonlinear factors to the neural nodes in the neural network, allowing the neural network to better adapt to complex data and simulate more complex functions; it can determine whether the neuron is activated; and it can help optimize the gradient back propagation process. YOLOv5 uses Swish as the activation function. The expressions of Swish activation function and Sigmoid activation function are shown in formula (1) and formula (2).

[0081] f(x)=x*sigmoid(x) (1)

[0082]

[0083] The calculation of the Sigmoid function in the Swish function is relatively complex, and the activation functions currently supported by the DPU are mainly ReLU, ReLU6, and Leaky ReLU. After analysis, Leaky ReLU is computationally efficient and can solve the neuron "death" problem of the ReLU activation function, avoiding the corresponding weight and bias parameters from being unable to be updated during back propagation. Therefore, the improved YOLOv5 model uses the Leaky ReLU activation function to replace the Swish activation function. The expression of the Leaky ReLU activation function is shown in formula (3).

[0084]

[0085] In order to achieve efficient and rapid detection of targets of different scales, YOLOv5 uses anchors of different scales for different feature maps. For shallow feature maps, small-scale anchors are used to detect small targets; for deep feature maps, large-scale anchors are used to detect large targets. Since the scale selection of anchors is strongly correlated with the input data set, it is necessary to optimize the anchors to adapt to the data set of coal mine target detection. The improved YOLOv5 model uses the K-means clustering algorithm to determine the number of anchor boxes and calculate the corresponding size. Anchors contains a total of 9 numerical parameters, namely 3 rows and 6 columns. The width and height of the anchor are determined according to the pixel size of the original image for optimization.

[0086] It can be seen that the data adaptive anchor box calculation module of this embodiment determines the number of anchor boxes through the K-means clustering algorithm.

[0087] The main idea of ​​optimizing the model is to ensure that the end-to-end reasoning process of the model can be supported by the DPU hardware core, making full use of the FPGA parallel computing characteristics, so as to fully benefit from the FPGA hardware deployment acceleration.

[0088] Step 103, training the improved YOLOv5 model using the training data set to obtain an initial detection model.

[0089] In some embodiments, the PyTorch deep learning framework is used to train an improved YOLOv5 model through a training data set to obtain an initial detection model; wherein, the hardware platform supporting the training of the improved YOLOv5 model includes an Intel XeonGold 5118, 2.30GHz processor, 377GB of memory, and a GeForce RTX 2080Ti graphics processor.

[0090] The model training adopts the PyTorch deep learning framework, the computer processor is Intel Xeon Gold5118, 2.30GHz, the memory is 377GB, and the graphics processor is GeForce RTX 2080Ti, corresponding to the subsequent CPU and GPU platforms respectively.

[0091] Step 104, performing iterative processing including pruning, quantization and training on the initial detection model to obtain a coal mine underground drilling site object detection model that supports DPU hardware deployment.

[0092] As an implementation method, a method for obtaining an object detection model for a coal mine underground drilling site supporting DPU hardware deployment includes:

[0093] The following process is iterated until the number of iterations meets the number threshold, and a coal mine underground drilling field object detection model that supports DPU hardware deployment is obtained;

[0094] The initial detection model is pruned using the Vitis AI sparsity pruner. The pruning process includes generating a sparse model, fine-tuning the sparse model, and exporting the sparse model.

[0095] The pruned model is quantized through the Vitis AI quantizer to obtain the quantized initial detection model. The quantization process uses layer-by-layer data quantization, and quantizes the weight data, inter-layer transmission data, intra-layer intermediate results, and activation data respectively.

[0096] The quantized initial detection model is trained using the training dataset.

[0097] It can be understood that the storage and computation of neural network models are optimized through two aspects: data quantization and reducing weight parameters (i.e. model pruning). Inference is a computationally intensive process that requires a large amount of memory bandwidth to meet the low latency and high throughput requirements of edge applications. Quantization and pruning techniques can not only meet these challenges, but also achieve optimal performance and high energy efficiency, and minimize accuracy degradation.

[0098] In terms of data quantization, the entire process is completed based on the Vitis AI quantizer. Quantization can make integer computing units effective, and weights and activations can be presented with lower precision. Usually, 32-bit floating-point weights and activation values ​​are used when training neural networks. The Vitis AI quantizer can convert 32-bit floating-point weights and activations into 8-bit integer (INT8) format. Since the deployment of fixed-point network models requires less memory bandwidth, it is faster and more energy-efficient than floating-point models. This method can reduce computational complexity without losing prediction accuracy. The Vitis AI quantizer accepts floating-point models as input and performs preprocessing (folding batch-norm and removing nodes that are not needed for inference). Then, it quantizes weights / biases and activations to a given bit width. The Vitis AI quantizer requires multiple iterations of inference to calibrate activations to improve the accuracy of the quantized model and capture activation statistics. Take a calibration image dataset as input. Usually, since backpropagation is not required, the quantizer can properly handle 100 to 1000 calibration images, and unlabeled datasets can also work. After calibration, the quantized model is transformed into a DPU deployable format. For the pytorch framework, it is called model_name.xmodel, which is consistent with the data format of the DPU. Then, the Vitis AI compiler can compile the model and deploy it to the DPU. Furthermore, for different types of data such as weight data, inter-layer transmission data, intra-layer intermediate results, and activation data, the bit width of the data is reduced as much as possible while ensuring high accuracy, so as to improve the storage efficiency of the network model and reduce the computational complexity; at the same time, since different types of data have different requirements for decimal precision in actual model training, in order to ensure a balance between model accuracy and computational complexity, for example, the accuracy requirements for intra-layer intermediate results are lower than those for inter-layer transmission data, different integer and decimal ratios are used for the above different data types to ensure good accuracy; and since the weights and activation data of each layer have their own fluctuation range, data quantization is performed layer by layer.

[0099] In terms of model pruning, the model is optimized. Neural networks are often over-parameterized and have a lot of redundancy. Pruning is the process of removing redundant weights while keeping the loss of accuracy as low as possible. Vitis AIOptimizer supports the optimization of neural network models. Vitis AI optimizer is used to remove redundant kernels in neural networks and reduce the overall computational cost of inference. Vitis AI sparsity pruner can implement high-precision sparse pruning algorithms for multiple N:M sparsity modes within each continuous block of M values. The pruning algorithm prunes the weight values ​​along the input channel dimension. In groups of M weights, the pruner sets N weights (minimum value is 0). Typical values ​​of M are 4, 8, or 16, and N is equal to half of the M value, so that 50% high-precision sparsity can be achieved.

[0100] In some embodiments, after obtaining a coal mine underground drilling site object detection model that supports DPU hardware deployment, it includes: obtaining a test data set based on a coal mine data set; inputting the test data set into the coal mine underground drilling site object detection model to obtain a detection result; and evaluating the model performance of the coal mine underground drilling site object detection model based on the detection result.

[0101] In some embodiments, after evaluating the model performance of the coal mine underground drilling site object detection model, it includes: after determining that the model performance meets the performance indicators, compiling the coal mine underground drilling site object detection model that supports DPU hardware deployment into a model execution file that can be recognized by the DPU; using the model execution file as the coal mine underground drilling site object detection model for DPU hardware deployment.

[0102] This application also compares the performance of the underground mine drilling site object detection model obtained by training the YOLOv5 model and the improved YOLOv5 model. The implementation process is: train the YOLOv5 and improved YOLOv5 models to convergence, use two networks to detect the test data set images respectively, and evaluate the model performance based on the mAP (mean Average Precision) evaluation index. Secondly, after the model weights are obtained through training, the original YOLOv5 model and the improved YOLOv5 model reasoning are executed on the test set using the above-mentioned experimental server CPU and GPU, and the model reasoning throughput is counted. In order to give full play to the FPGA hardware capabilities, the improved YOLOv5 model is quantized to 8 bits based on the Vitis AI tool library. After compiling the quantized model into DPU instructions, the corresponding embedded test program is written, the test set and model are loaded, and after the hardware is deployed, 100 test images are continuously iterated for 3 rounds, and the hardware reasoning throughput is counted.

[0103] In order to ensure fairness, the performance of the trained original YOLOv5 model and the trained improved YOLOv5 model were compared in the coal mine dataset. The trained improved YOLOv5 model improved the accuracy by 1.1% and the mAP value by 0.2% compared with the trained original YOLOv5 model. The statistical model inference throughput, the GPU inference speed increased by 57% compared with the original model, and the CPU inference speed increased by 25%. The experimental results show that the proposed method has better performance in all evaluation indicators. After the model is quantified and deployed on hardware, the inference throughput on the DPU and CPU is compared. It can be seen that the DPU hardware inference throughput reaches 32.51 frames per second, which meets the real-time inference requirements and is 2.41 times the inference throughput of the CPU platform, with obvious acceleration effect.

[0104] By optimizing and compiling the model, the YOLOv5 model after software and hardware co-optimization was deployed, and experiments were established to fully evaluate the model's target detection effect, verifying the model's high precision and low-latency inference performance. Compared with the existing model, the coal mine underground drilling site object detection model based on the improved YOLOv5 model has improved the detection accuracy of all classes to varying degrees, and the inference throughput of the CPU and GPU has also been improved.

[0105] This application scheme is of great help to the field of intelligent coal mines: personnel-based intelligent coal mine target detection is of great significance to the safe and effective management of personnel and the safe production of coal mines. In addition, it has a wide range of application fields, such as autonomous driving, intelligent security, etc., and can provide more accurate and faster target detection technology for related applications in these fields, thereby improving the performance and effect of the application.

[0106] The method for constructing the underground mine drilling site object detection model of the embodiment of the present application proposes an improved YOLOv5 model based on the DPU hardware characteristics, trains the improved YOLOv5 model with a coal data set, and performs pruning and quantization and other optimization processing on the trained model to obtain an underground mine drilling site object detection model with high accuracy and high recognition speed, thereby realizing a target detection method that supports DPU hardware deployment and lightweight. The underground mine drilling site object detection model is deployed in hardware to further improve the model reasoning speed. The deployed model meets the real-time reasoning requirements and has a significant acceleration effect.

[0107] In order to implement the above-mentioned embodiment, the present application also proposes a device for constructing an object detection model for an underground drilling site in a coal mine. Figure 3 The present invention provides a schematic diagram of a device for constructing an object detection model for a coal mine drilling site. Figure 3As shown, the device for constructing an object detection model for an underground coal mine drilling site may include: a sample acquisition module 301 , a model determination module 302 , a model training module 303 and a model optimization module 304 .

[0108] The sample acquisition module 301 is used to acquire a training data set based on the coal mine data set;

[0109] A model determination module 302 is used to determine an improved YOLOv5 model, wherein the improved YOLOv5 model replaces the Focus structure in the backbone network with a common convolutional structure, replaces the Swish activation function with a Leaky ReLU activation function, and replaces the adaptive anchor frame calculation module at the input end with a data adaptive anchor frame calculation module; wherein the data adaptive anchor frame calculation module determines the number of anchor frames through a clustering algorithm and determines the size of the anchor frame according to the pixel size of the original image in the training data set;

[0110] A model training module 303 is used to train the improved YOLOv5 model through a training data set to obtain an initial detection model;

[0111] The model optimization module 304 is used to perform iterative processing including pruning, quantization and training on the initial detection model to obtain a coal mine underground drilling site object detection model that supports DPU hardware deployment.

[0112] Furthermore, in a possible implementation of the embodiment of the present application, the model optimization module 304 is specifically used to:

[0113] The following process is iterated until the number of iterations meets the number threshold, and a coal mine underground drilling field object detection model that supports DPU hardware deployment is obtained;

[0114] The initial detection model is pruned using the Vitis AI sparsity pruner. The pruning process includes generating a sparse model, fine-tuning the sparse model, and exporting the sparse model.

[0115] The pruned model is quantized through the Vitis AI quantizer to obtain the quantized initial detection model. The quantization process uses layer-by-layer data quantization, and quantizes the weight data, inter-layer transmission data, intra-layer intermediate results, and activation data respectively.

[0116] The quantized initial detection model is trained using the training dataset.

[0117] Furthermore, in a possible implementation of the embodiment of the present application, the apparatus further includes a performance evaluation module 305, which is used to:

[0118] Based on the coal mine dataset, obtain a test dataset;

[0119] The test data set is input into the object detection model of the underground drilling site of the coal mine to obtain the detection results;

[0120] Based on the detection results, the model performance of the object detection model in the underground coal mine drilling site is evaluated.

[0121] Furthermore, in a possible implementation of the embodiment of the present application, the apparatus further includes a model compilation module 306, which is used to:

[0122] After determining that the model performance meets the performance indicators, the coal mine underground drilling site object detection model that supports DPU hardware deployment is compiled into a model execution file that can be recognized by the DPU;

[0123] Use the model execution file as a coal mine drilling site object detection model for DPU hardware deployment.

[0124] Further, in a possible implementation of the embodiment of the present application, when the data adaptive anchor frame calculation module of the model determination module 302 determines the number of anchor frames by using a clustering algorithm, it is used to:

[0125] The data adaptive anchor box calculation module determines the number of anchor boxes through the K-means clustering algorithm.

[0126] Furthermore, in a possible implementation of the embodiment of the present application, the model training module 303 is specifically used to:

[0127] The PyTorch deep learning framework is used to train the improved YOLOv5 model through the training data set to obtain the initial detection model. The hardware platform supporting the training of the improved YOLOv5 model includes Intel Xeon Gold 5118, 2.30GHz processor, 377GB memory and GeForce RTX 2080Ti graphics processor.

[0128] It should be noted that the aforementioned explanation of the embodiment of the method for constructing an object detection model for an underground mine drilling site is also applicable to the device for constructing an object detection model for an underground mine drilling site in this embodiment, and will not be repeated here.

[0129] In order to implement the above embodiment, the present application also proposes an electronic device. Figure 4 , Figure 4 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 4 As shown, the electronic device 400 includes: a processor 401, and a memory 402 communicatively connected to the processor 401; the memory 402 stores computer-executable instructions; the processor 401 executes the computer-executable instructions stored in the memory to implement the method provided in the aforementioned embodiment.

[0130] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0131] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0132] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0133] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0134] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0135] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0136] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0137] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0139] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0140] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0142] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for constructing an object detection model for a coal mine drilling site, characterized in that: The method comprises the following steps: Based on the coal mine data set, obtain a training data set; Determine an improved YOLOv5 model, wherein the improved YOLOv5 model replaces the Focus structure in the backbone network with a common convolution structure, replaces the Swish activation function with a Leaky ReLU activation function, and replaces the adaptive anchor frame calculation module at the input end with a data adaptive anchor frame calculation module; wherein the data adaptive anchor frame calculation module determines the number of anchor frames through a clustering algorithm and determines the size of the anchor frame according to the pixel size of the original image in the training data set; Training the improved YOLOv5 model through the training data set to obtain an initial detection model; The initial detection model is iteratively processed including pruning, quantization and training to obtain a coal mine underground drilling site object detection model that supports DPU hardware deployment.

2. The method according to claim 1, characterized in that The initial detection model is iteratively processed including pruning, quantization and training to obtain a coal mine underground drilling field object detection model supporting DPU hardware deployment, including: The following process is iterated until the number of iterations meets the number threshold, and a coal mine underground drilling field object detection model that supports DPU hardware deployment is obtained; Pruning the initial detection model through the Vitis AI sparsity pruner, wherein the pruning process includes generating a sparse model, fine-tuning the sparse model, and deriving the sparse model; The pruned model is quantized by the Vitis AI quantizer to obtain a quantized initial detection model; the quantization process uses layer-by-layer data quantization, and quantizes weight data, inter-layer transmission data, intra-layer intermediate results, and activation data respectively; The quantized initial detection model is trained using the training data set.

3. The method according to claim 1, characterized in that After obtaining the coal mine underground drilling field object detection model supporting DPU hardware deployment, it includes: Based on the coal mine dataset, obtain a test dataset; Inputting the test data set into the coal mine underground drilling site object detection model to obtain a detection result; According to the detection results, the model performance of the coal mine underground drilling site object detection model is evaluated.

4. The method according to claim 3, characterized in that After evaluating the model performance of the coal mine underground drilling site object detection model, the method further comprises: After determining that the model performance meets the performance index, compiling the coal mine underground drilling field object detection model supporting DPU hardware deployment into a model execution file recognizable by the DPU; The model execution file is used as a coal mine underground drilling site object detection model for DPU hardware deployment.

5. The method according to claim 1, characterized in that The data adaptive anchor frame calculation module determines the number of anchor frames through a clustering algorithm, including: The data adaptive anchor frame calculation module determines the number of anchor frames through the K-means clustering algorithm.

6. The method according to claim 1, characterized in that The step of training the improved YOLOv5 model through the training data set to obtain an initial detection model comprises: The improved YOLOv5 model is trained by the training data set using the PyTorch deep learning framework to obtain an initial detection model; wherein the hardware platform supporting the training of the improved YOLOv5 model includes an Intel Xeon Gold5118, 2.30GHz processor, 377GB memory and a GeForce RTX 2080Ti graphics processor.

7. A device for constructing an object detection model for a coal mine drilling site, characterized in that: include: A sample acquisition module is used to acquire a training data set based on a coal mine data set; A model determination module, used to determine an improved YOLOv5 model, wherein the improved YOLOv5 model replaces the Focus structure in the backbone network with a common convolution structure, replaces the Swish activation function with a Leaky ReLU activation function, and replaces the adaptive anchor frame calculation module at the input end with a data adaptive anchor frame calculation module; wherein the data adaptive anchor frame calculation module determines the number of anchor frames through a clustering algorithm and determines the size of the anchor frame according to the pixel size of the original image in the training data set; A model training module, used to train the improved YOLOv5 model through the training data set to obtain an initial detection model; The model optimization module is used to perform iterative processing including pruning, quantization and training on the initial detection model to obtain a coal mine underground drilling site object detection model that supports DPU hardware deployment.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images