Underground personnel safety helmet wearing real-time detection method based on deep learning

Abstract

The invention provides an underground personnel safety helmet wearing real-time detection method based on deep learning, and the method comprises the steps: determining whether an underground personnel safety helmet wears a corresponding target image data set or not, and a YOLOX network model, a trunk deep convolutional neural network in the YOLOX network model adopts deformable convolution capable of learning offset, shape prior is introduced during sampling, and ECA-Net attention mechanism models are added to feature layers of different scales of a path aggregation network in a neck network and an up-sampling part of a feature pyramid network. The obj branch and the cls branch in the head network use a binary cross entropy loss function, the prediction reg branch uses a generalized intersection-parallel ratio loss function, an angle loss item is added, an initial safety helmet detection model is trained, and the real-time image is detected to determine whether the underground personnel wear the safety helmets in real time. Therefore, the small target detection of whether the underground personnel wear the safety helmet is realized, and the safety helmet detection precision and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of underground target detection, and in particular to a real-time detection method and device for underground personnel wearing safety helmets based on deep learning. Background Art

[0002] Coal mine workers should follow good dress code when performing production operations, such as wearing safety helmets, which can effectively prevent accidents. However, accidents caused by workers not wearing safety helmets are common, mainly because some workers lack safety awareness. In related technologies, the main way to check clothing is to manually patrol underground and check surveillance videos, but the above two methods are inefficient and cannot correct improper clothing in a timely manner. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a real-time detection method for underground personnel wearing safety helmets based on deep learning, so as to realize small target detection of whether underground personnel are wearing safety helmets and improve the precision and accuracy of safety helmet detection.

[0005] The second purpose of the present invention is to propose a real-time detection device for underground personnel wearing safety helmets based on deep learning.

[0006] A third objective of the present invention is to provide an electronic device.

[0007] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0008] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a real-time detection method for underground personnel wearing safety helmets based on deep learning, the method comprising:

[0009] Obtain an initial image of whether the underground personnel are wearing a helmet, taken by an 850-band infrared camera, and perform Gaussian blur operation, flip transformation, and rotation transformation on the initial image to obtain a target image data set after the operation, wherein the target image data set includes images of wearing helmets and images of not wearing helmets marked with helmet angles;

[0010] Construct a YOLOX network model, wherein the YOLOX network model includes a backbone deep convolutional neural network, a neck network and a head network, an ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, and the obj branch and the cls branch in the decoupled detection head of the head network use a binary cross entropy loss function, and the prediction reg branch uses a generalized intersection-over-union loss function, and an angle loss term is added to the generalized intersection-over-union loss function, and the backbone deep convolutional neural network uses a deformable convolution with a learnable offset, and a shape prior is introduced when the deformable convolution performs sampling of a target image data set;

[0011] Input the target image data set into the YOLOX network model to train an initial helmet detection model for distinguishing whether underground personnel are wearing helmets;

[0012] The helmet detection model after inference acceleration of the initial helmet detection model through TensorRT detects the real-time images of underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time.

[0013] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a real-time detection device for underground personnel wearing safety helmets based on deep learning, and the device comprises:

[0014] The data acquisition module is used to acquire the initial image taken by the 850-band infrared camera to show whether the underground personnel are wearing helmets, and perform Gaussian blur operation, flip transformation, and rotation transformation on the initial image to obtain a target image data set after the operation, wherein the target image data set includes images of wearing helmets and images of not wearing helmets marked with helmet angles;

[0015] A construction module for constructing a YOLOX network model, wherein the YOLOX network model includes a backbone deep convolutional neural network, a neck network and a head network, wherein an ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, and the obj branch and the cls branch in the decoupled detection head of the head network use a binary cross entropy loss function, and the prediction reg branch uses a generalized intersection-over-union loss function, and an angle loss term is added to the generalized intersection-over-union loss function, and the backbone deep convolutional neural network uses a deformable convolution with a learnable offset, and a shape prior is introduced when the deformable convolution performs sampling of a target image data set;

[0016] A training module, used to input the target image data set into the YOLOX network model to train an initial helmet detection model for distinguishing whether underground personnel are wearing helmets;

[0017] The detection module is used to detect the real-time images of underground personnel taken by the 850-band infrared camera using the safety helmet detection model after inference acceleration of the initial safety helmet detection model through TensorRT to determine whether the underground personnel are wearing safety helmets in real time.

[0018] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0019] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0020] The real-time detection method, device, electronic device and storage medium of the underground personnel wearing safety helmets based on deep learning of the embodiment of the present invention are based on the target image data set corresponding to whether the underground personnel wear safety helmets, and the YOLOX network model, wherein the backbone deep convolutional neural network in the YOLOX network model adopts deformable convolution with learnable offset and introduces shape prior when sampling, and the ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, the obj branch and the cls branch in the head network use the binary cross entropy loss function, the prediction reg branch uses the generalized intersection-over-union loss function and adds an angle loss term, and the initial safety helmet detection model is trained to detect the real-time image to determine whether the underground personnel wear safety helmets in real time. In this way, the small target detection of whether the underground personnel wear safety helmets is realized, and the precision and accuracy of safety helmet detection are improved.

[0021] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A schematic flow chart of a method for real-time detection of safety helmet wearing by underground personnel based on deep learning provided by an embodiment of the present invention;

[0024] Figure 2A flowchart of initial image augmentation provided by an embodiment of the present invention;

[0025] Figure 3 An ECA-Net network structure diagram is provided in an embodiment of the present invention;

[0026] Figure 4 A flow chart of a TensorRT reasoning acceleration YOLOX helmet detection model provided by an embodiment of the present invention;

[0027] Figure 5 A flowchart of a helmet detection model detection deployment provided by an embodiment of the present invention;

[0028] Figure 6 An application system diagram of a method for real-time detection of safety helmet wearing by underground personnel based on deep learning provided by an embodiment of the present invention;

[0029] Figure 7 A schematic structural diagram of a device for real-time detection of safety helmet wearing by underground personnel based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0031] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the present invention are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0032] The acquisition, transmission, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of relevant laws and regulations.

[0033] It should be noted that in the embodiments of the present disclosure, certain software, components, models, etc. existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present invention, but it does not mean that the inventor has or will necessarily use the solution.

[0034] The following describes a method and device for real-time detection of safety helmet wearing by underground personnel based on deep learning according to an embodiment of the present invention with reference to the accompanying drawings.

[0035] Figure 1 A flowchart of a method for real-time detection of safety helmet wearing by underground personnel based on deep learning provided in an embodiment of the present invention.

[0036] like Figure 1 As shown, the method comprises the following steps:

[0037] Step 101, obtain the initial image taken by the 850-band infrared camera to show whether the underground personnel are wearing safety helmets, and perform Gaussian blur operation, flip transformation, and rotation transformation on the initial image to obtain a target image data set after the operation, wherein the target image data set includes images of wearing safety helmets and images of not wearing safety helmets marked with safety helmet angles.

[0038] In some embodiments, an initial image of whether the underground personnel are wearing helmets is obtained by taking an 850-band infrared camera, and a Gaussian blur operation, flip transformation, and rotation transformation are performed on the initial image to obtain a target image data set after the operation. An implementation method can be to take video data of underground personnel based on an 850-band infrared camera, and convert the video data into an image in image format; annotate the image using a preset annotation tool to obtain an initial image of whether the underground personnel are wearing helmets; and perform a Gaussian blur operation, flip transformation, and rotation transformation on the initial image to obtain a target image data set after the operation. In this way, the data set can be enriched to ensure the integrity and reliability of the target image data set.

[0039] Specifically, when there are 5,800 images of people wearing helmets in the images converted from the video data to image format, the present invention can use the labeling tool labelimg to label the images as two types of initial images of underground personnel wearing helmets and not wearing helmets. In order to obtain more data sets, the initial images are amplified, enhanced, and other operations are performed to enhance the generalization ability of the model. Common data amplification and enhancement mainly include Gaussian blur operations, flip transformations, rotation transformations, and the like. The flip transformation mainly performs a reflection transformation at the center of y=x. It is assumed that the size of the initial image is n×n, and the size of the images in the initial image data set is the same. The initial image data set A is flipped to the target image data set A1, as shown in formula (1):

[0040]

[0041] In addition, since the underground environment is dim and complex, and dust and particles may be generated when the underground personnel are working, Gaussian blur is used to increase the generalization ability of the model. Gaussian blur is mainly used to reduce the initial image noise and to match the on-site environment during work. It uses the Gaussian normal distribution density function, such as formula (2):

[0042]

[0043] Taking the pixel value as the coordinate origin, weights are assigned to the surrounding pixel values ​​according to the density function, thus obtaining a weighted average value, as shown in formula (3):

[0044]

[0045] Here, σ is the standard deviation of x and μ is the mean of x.

[0046] Therefore, if Figure 2 As shown, the initial downhole image dataset was mainly subjected to Gaussian blur operation (image enhancement), flip transformation and rotation transformation, and a total of nearly 50,000 augmented datasets were finally obtained as the target image dataset.

[0047] Step 102, constructing a YOLOX network model, wherein the YOLOX network model includes a backbone deep convolutional neural network, a neck network and a head network, an ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, and the obj branch and the cls branch in the decoupled detection head of the head network use a binary cross entropy loss function, and the prediction reg branch uses a generalized intersection-over-union loss function, and an angle loss term is added to the generalized intersection-over-union loss function, and the backbone deep convolutional neural network uses a deformable convolution with a learnable offset, and a shape prior is introduced when the deformable convolution performs sampling of the target image data set.

[0048] Optionally, in the actual scene of the mine, the pictures taken by the 850-band infrared camera have a small proportion of the helmet and the human head in the whole image. When passing to the deep layer of the network, the available feature information is seriously lost, and it will be impossible to locate and lead to missed detection and wrong detection. In order to improve the detection accuracy of such small targets, an attention mechanism model (Efficient Channel Attention Network, ECA-Net) is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network. ECA-Net directly corresponds channels to attention weights, realizes cross-channel interaction through group convolution, enhances or suppresses feature information according to the size of attention weights, and improves the utilization of useful information by the YOLOX network model.

[0049] Optionally, by decoupling the head network, the obj branch and the cls branch in the detection head use a binary cross entropy loss function, and the reg branch is predicted using a generalized intersection-over-union loss function to improve the regression performance of the YOLOX network model and make the predicted value of the YOLOX network model gradually approach the true value.

[0050] Among them, obj represents the probability of existence of the target (safety helmet), cls represents the category prediction, and the main task of the reg branch is to predict the position of the safety helmet bounding box.

[0051] Alternatively, in special environments such as coal mines, workers have variable working postures, and the pitch angle of the head will also change, which causes the helmet to visually rotate. Therefore, the rotation angle is introduced into the definition of the target detection frame when sampling the deep convolutional neural network. Specifically, in addition to the center position and size of the helmet regression detection frame, an additional angle value is regressed. When annotating the target image dataset, we also need to annotate the rotation angle of each helmet. Then, in the loss function of the YOLOX network model, an angle loss term is added so that the YOLOX network model can learn the rotation angle information during training.

[0052] Alternatively, the helmet is generally elliptical, but the output of the YOLOX network model covers the rectangular detection box of the helmet. The rectangular detection box often contains some background information, which may contain complex textures or other shapes similar to the target, causing the YOLOX network model to make false detections. In order to effectively suppress the background feature responses that are not helmets, the backbone deep convolutional neural network of this scheme adopts deformable convolution with learnable offsets. Deformable convolution is an improved form of convolution that introduces a learnable offset so that the convolution kernel can adaptively adjust its shape to adapt to the local structure of the input image. Therefore, for the helmet detection task, we can train deformable convolution to make it more flexible in spatial sampling, learn the special shape and features of the helmet, and suppress the feature responses of the surrounding irrelevant background.

[0053] Optionally, since the shape of a helmet is usually half an ellipse, a shape prior is introduced in the center point sampling of the backbone deep convolutional neural network of the YOLOX network model, and an elliptical Gaussian distribution is generated with the midpoint of the bottom edge of the marked rectangular detection box as the center of the circle. These elliptical areas are used as the helmet sampling areas of the center points to improve the sampling effect of the helmet.

[0054] Step 103: Input the target image data set into the YOLOX network model to train an initial helmet detection model for distinguishing whether underground personnel are wearing helmets.

[0055] In some embodiments, the target image data set is input into the YOLOX network model to train an initial helmet detection model for distinguishing whether underground personnel are wearing helmets. One implementation method can be that when extracting features from the target image data set through the backbone deep convolutional neural network of the YOLOX network model, a shape prior is introduced to take the elliptical area as the helmet sampling area with the center point; the helmet sampling area is subjected to feature extraction by using a deep convolutional neural network composed of deformable convolutions with learnable offsets to extract a feature map of the helmet area; the feature map is subjected to a global average pooling operation by using the ECA-Net attention mechanism model added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, and the convolution operation with a kernel size of k is performed. The local cross-channel interaction is carried out by product, and the weight of each channel is obtained through the sigmoid activation function to recalibrate the feature map, and the helmet feature map of the specific area is weakened or strengthened; the helmet feature map is decoupled through the obj branch and cls branch of the binary cross entropy loss function in the decoupled detection head of the head network, the prediction reg branch using the generalized intersection-over-union loss function, and an angle loss term added to the generalized intersection-over-union loss function, and the predicted image of not wearing a helmet and the image of wearing a helmet with an angle are obtained; according to the difference between the predicted image of not wearing a helmet and the image of wearing a helmet with an angle, and the images of wearing a helmet and not wearing a helmet with helmet angles marked in the target image dataset, training is performed to obtain the initial helmet detection model. Thus, the initial helmet detection model is established quickly and accurately.

[0056] Furthermore, when local cross-channel interaction is performed through convolution with a kernel size of k, the banded convolution of cross-channel interaction in the channel neuron is as follows:

[0057]

[0058] Where k is the convolution kernel size and C is the number of channels. The mapping function between the convolution kernel and the number of channels is formula (4):

[0059]

[0060] Among them, γ is 2, b is 1, and k is the nearest odd number result.

[0061] Therefore, if Figure 3As shown, after adding the ECA-Net attention mechanism model to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, the structural diagram of the ECA-Net network added to the neck network is obtained, wherein the neck network includes feature fusion and feature segmentation (Concat+CSPLayer) and two-dimensional convolution Conv2D operations on feature layers of different scales, and two-dimensional convolution upsampling (Upsampling2D) and two-dimensional convolution downsampling (Downsample) operations of the feature pyramid network.

[0062] In addition, when the obj branch and cls branch in the decoupled detection head of the head network use the binary cross entropy loss function (BCE Loss) and the prediction reg branch uses the generalized intersection-over-union loss function (GIoU_Loss), the evaluation index (Generalized IoU, GIoU) ​​that measures the degree of overlap between the predicted rectangular box and the real rectangular box corresponding to the helmet is [-1,1]. Due to the introduction of the penalty term, in the case of non-overlap, the predicted rectangular box will move toward the real rectangular box. In addition to focusing on the differences in overlapping areas, GIoU also focuses on non-overlapping areas, which can better reflect the degree of overlap.

[0063] The GIoU_Loss calculation formula is as shown in formulas (5) and (6):

[0064]

[0065] Where: A is the predicted rectangle, B is the real rectangle, I is the intersection area of ​​A and B, and U is the union area of ​​A and B, which represents the minimum circumscribed rectangle area of ​​A and B. Only when the predicted rectangle and the real rectangle coincide, GIoU_Loss will be 0, and GIoU_Loss can be trained in any case.

[0066] In summary, we can set the number of training rounds to 500, the learning rate to 0.001, the number of training batches to 32, and train the YOLOX network model to obtain the initial helmet detection model.

[0067] Step 104, the helmet detection model after inference acceleration of the initial helmet detection model through TensorRT detects the real-time image of the underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time.

[0068] In some embodiments, the helmet detection model after inference acceleration of the initial helmet detection model through TensorRT detects the real-time image of the underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time. An implementation method can be as follows Figure 4As shown in the figure, the initial helmet detection model is input into the parser in TensorRT for parsing, the unused layers of the initial helmet detection model are deleted, and the convolutional layers, pooling layers and activation functions in the initial helmet detection model are fused to generate an engine; the network in the engine is horizontally combined based on the target parameters selected by the preset engine builder, and the layers with the same tensor and operation are fused together to obtain a serialized engine; the serialized engine is stored, and the serialized engine and the corresponding deserialized engine are used to generate a helmet detection model for model reasoning, so as to detect the real-time images of underground personnel taken by the 850-band infrared camera and determine whether the underground personnel are wearing helmets in real time. Therefore, the helmet detection model after the initial helmet detection model is accelerated by TensorRT can more quickly improve the detection speed of the helmet detection model.

[0069] In addition, the corresponding engine file when TensorRT performs inference acceleration on the initial helmet detection model is deployed on the computing platform Jetson TX2, and then Jetson TX2 is connected to the PLC module underground; when it is determined that the underground personnel are not wearing helmets in real time, the Programmable Logic Controller (PLC) module will alarm the underground personnel who are not wearing helmets in real time. In this way, the working safety of personnel in the mine is guaranteed.

[0070] The real-time detection method for underground personnel wearing safety helmets based on deep learning in the embodiment of the present invention constructs a distributed underground wireless transmission network architecture in coal mines through an underground bearer network linked to ground network equipment, multiple 5G base station controllers with frequency-shift transmission enhancement master modules, and 5G base stations and terminals with frequency-shift transmission enhancement slave modules; the transmission service type, transmission demand index, channel occupancy rate, channel quality and transmission area of ​​the target 5G base station corresponding to each terminal in any target area in the underground area are obtained through the frequency-shift transmission enhancement slave module, and transmitted to the corresponding target frequency-shift transmission enhancement master module, so that when it is determined that the target 5G base station needs to perform network transmission enhancement, the frequency-shift transmission enhancement slave module executes frequency shift technology to enhance network transmission. Therefore, by enhancing the design of 5G base station controllers and 5G base stations, it is possible to achieve hierarchical, classified, on-demand transmission enhancement, and basic coverage of the entire coal mine underground.

[0071] In other embodiments, the present invention also provides a safety helmet detection model detection deployment flow chart, such as Figure 5As shown, specifically, it includes four parts: data source, data processing, data modeling and evaluation, and model application. Among them, the data source is the initial image of whether the underground personnel are wearing helmets, which is taken by an 850-band infrared camera; data processing includes data cropping, data enhancement (Gaussian blur operation and flip transformation, rotation transformation), data normalization and data division to obtain the processed (data augmented) target image data set, and the target image data set after data augmentation is divided into a training set, a verification set and a test set; then in the data modeling and evaluation process, the target image data is input into the YOLOX network model as modeling data to train the initial helmet detection model, and then the accuracy evaluation of the initial helmet detection model is detected by P, R, and mAP to obtain the detection results of the initial helmet detection model and analyze them until the preset accuracy is reached; in the application of the model, the initial helmet detection model is accelerated and deployed for inference, wherein the acceleration and deployment of the initial helmet detection model includes accelerating the inference of the initial helmet detection model through TensorRT, and deploying the corresponding engine file when TensorRT accelerates the inference of the initial helmet detection model on the computing platform jetson TX2, jetson TX2 conducts downhole detection and application, and connects to the downhole PLC module for alarm prompts.

[0072] Among them, P, R, and mAP are respectively the accuracy (Precision, P), recall (Recall, R), and average accuracy of the helmet detection model. The preset accuracy can be mAP greater than or equal to 98%, but is not limited to this.

[0073] In addition, in order to better understand the present invention, the present invention also proposes an application system diagram of a real-time detection method for underground personnel wearing safety helmets based on deep learning, such as Figure 6As shown, the data acquisition unit, transmission unit (not shown in the figure), back-end processing unit, control unit, data storage and display unit, wherein the data acquisition unit (including infrared camera, protective shell, etc.) is used to use 850 band infrared camera installed in different environments, different angles and positions in the mine to collect data, so as to collect the initial image of whether the personnel in the mine wear the helmet; the transmission unit (including cable or optical cable) is used to transmit the initial image; the back-end processing unit (mainly composed of edge computing equipment, completing algorithm processing and linkage control signal output) is used for initial image augmentation processing, YOLOX network model training, and real-time image detection. The back-end processing unit is based on jeston tx2, auxiliary analog output input module, power input and output module, internal transplantation of relevant application algorithms and interactive interface, and algorithm processing and algorithm result expression. The algorithm result expression includes the relay output and analog output carried by itself, and is compatible with common network communication outputs such as communication protocol (Modbustcp, TCP) and database. The control unit is used to output the analog signal of the back-end processing unit to the control unit (PLC), and the PLC performs linkage control or alarm output, such as on-site sound and light alarm; the data storage and display unit (including management platform, server, disk, etc.) is connected to the back-end processing unit through the underground ring network, and is used to display the sound and light alarm results and video through the display, and at the same time store the historical sound and light alarm data on the server for optimization and upgrading of the safety helmet detection model.

[0074] In order to implement the above embodiments, the present invention also proposes a real-time detection device for underground personnel wearing safety helmets based on deep learning.

[0075] Figure 7 A schematic structural diagram of a device for real-time detection of safety helmet wearing by underground personnel based on deep learning provided in an embodiment of the present invention.

[0076] like Figure 7 As shown, the real-time detection device 70 for underground personnel wearing safety helmets based on deep learning includes: a data acquisition module 71, a construction module 72, a training module 73, and a detection module 74.

[0077] The data acquisition module 71 is used to acquire an initial image taken by an 850-band infrared camera to show whether the underground personnel are wearing helmets, and to perform Gaussian blur operation, flip transformation, and rotation transformation on the initial image to obtain a target image data set after the operation, wherein the target image data set includes images of wearing helmets and images of not wearing helmets marked with helmet angles;

[0078] A construction module 72 is used to construct a YOLOX network model, wherein the YOLOX network model includes a backbone deep convolutional neural network, a neck network and a head network, an ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, and the obj branch and the cls branch in the decoupled detection head of the head network use a binary cross entropy loss function, and the prediction reg branch uses a generalized intersection-over-union loss function, and an angle loss term is added to the generalized intersection-over-union loss function, and the backbone deep convolutional neural network uses a deformable convolution with a learnable offset, and a shape prior is introduced when the deformable convolution performs sampling of the target image data set;

[0079] A training module 73 is used to input the target image data set into the YOLOX network model to train an initial helmet detection model for distinguishing whether underground personnel are wearing helmets;

[0080] The detection module 74 is used to detect the real-time image of underground personnel taken by the 850-band infrared camera using the safety helmet detection model after inference acceleration of the initial safety helmet detection model through TensorRT to determine whether the underground personnel are wearing safety helmets in real time.

[0081] Furthermore, in a possible implementation of the embodiment of the present invention, the data acquisition module 71 is specifically used to:

[0082] Shooting video data of underground personnel based on an 850-band infrared camera, and converting the video data into images in image format;

[0083] The image is annotated by a preset annotation tool to obtain an initial image of whether the underground personnel are wearing safety helmets;

[0084] The initial image is subjected to Gaussian blur operation, flip transformation and rotation transformation to obtain a target image data set after the operation.

[0085] Furthermore, in a possible implementation of the embodiment of the present invention, the training module 73 is specifically used to:

[0086] When extracting features from the target image dataset using the backbone deep convolutional neural network of the YOLOX network model, a shape prior is introduced to take the elliptical area as the helmet sampling area at the center point;

[0087] By using a deep convolutional neural network composed of a deformable convolution with a learnable offset, feature extraction is performed on the helmet sampling area to extract a feature map of the helmet area;

[0088] Through the ECA-Net attention mechanism model added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, a global average pooling operation is performed on the feature map, local cross-channel interaction is performed through convolution with a kernel size of k, and the weight of each channel is obtained through a sigmoid activation function to recalibrate the feature map, thereby obtaining a helmet feature map that weakens or strengthens a specific area;

[0089] Through the decoupling detection head of the head network, the obj branch and cls branch using the binary cross entropy loss function, the prediction reg branch using the generalized intersection-over-union loss function, and an angle loss term added to the generalized intersection-over-union loss function, the helmet feature map is decoupled to obtain the predicted image without the helmet and the angled image of the helmet.

[0090] Based on the differences between the predicted images without helmets and the images with helmets at angles, and the images with helmets at angles and the images without helmets in the target image dataset, training is performed to obtain an initial helmet detection model.

[0091] Furthermore, in a possible implementation of the embodiment of the present invention, the detection module 74 is specifically configured to:

[0092] Input the initial helmet detection model into the parser in TensorRT for parsing, delete the unused layers of the initial helmet detection model, and fuse the convolutional layers, pooling layers, and activation functions in the initial helmet detection model to generate the engine;

[0093] Combine the networks in the engine horizontally based on the target parameters selected by the preset engine builder, and fuse the layers with the same tensors and operations together to obtain a serialized engine;

[0094] The serialization engine is stored, and the serialization engine and the corresponding deserialization engine generate a helmet detection model for model reasoning to detect the real-time images of underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time.

[0095] Furthermore, in a possible implementation of the embodiment of the present invention, the device further includes:

[0096] A deployment module is used to deploy the corresponding engine file when TensorRT performs reasoning acceleration on the initial helmet detection model on the computing platform Jetson TX2, and then connect Jetson TX2 to the PLC module underground;

[0097] The alarm module is used to give an alarm prompt to the underground personnel who are not wearing safety helmets in real time through the PLC module when it is determined that the underground personnel are not wearing safety helmets in real time.

[0098] The real-time detection device for underground personnel wearing helmets based on deep learning of the embodiment of the present invention is based on the target image data set corresponding to whether underground personnel wear helmets, and the YOLOX network model, wherein the main deep convolutional neural network in the YOLOX network model adopts deformable convolution with learnable offset and introduces shape prior when sampling, and the ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, the obj branch and the cls branch in the head network use the binary cross entropy loss function, the prediction reg branch uses the generalized intersection-over-union loss function and adds an angle loss term, and the initial helmet detection model is trained to detect the real-time image to determine whether the underground personnel wear helmets in real time. In this way, the small target detection of whether the underground personnel wear helmets is realized, and the precision and accuracy of the helmet detection are improved.

[0099] In order to implement the above embodiment, the present invention further provides an electronic device, including:

[0100] at least one processor; and

[0101] a memory communicatively connected to the at least one processor; wherein,

[0102] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the aforementioned method.

[0103] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0106] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0107] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0108] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0109] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0111] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A real-time detection method for underground personnel wearing safety helmets based on deep learning, characterized in that: The method comprises: Obtain an initial image of whether the underground personnel are wearing a helmet, taken by an 850-band infrared camera, and perform Gaussian blur operation, flip transformation, and rotation transformation on the initial image to obtain a target image data set after the operation, wherein the target image data set includes images of wearing helmets and images of not wearing helmets marked with helmet angles; Construct a YOLOX network model, wherein the YOLOX network model includes a backbone deep convolutional neural network, a neck network and a head network, an ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, and the obj branch and the cls branch in the decoupled detection head of the head network use a binary cross entropy loss function, and the prediction reg branch uses a generalized intersection-over-union loss function, and an angle loss term is added to the generalized intersection-over-union loss function, and the backbone deep convolutional neural network uses a deformable convolution with a learnable offset, and a shape prior is introduced when the deformable convolution performs sampling of a target image data set; Input the target image data set into the YOLOX network model to train an initial helmet detection model for distinguishing whether underground personnel are wearing helmets; The helmet detection model after inference acceleration of the initial helmet detection model through TensorRT detects the real-time images of underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time.

2. The method according to claim 1, characterized in that The initial image of whether the underground personnel are wearing safety helmets is obtained by taking an 850-band infrared camera, and Gaussian blur operation, flip transformation, and rotation transformation are performed on the initial image to obtain a target image data set after the operation, including: Shooting video data of underground personnel based on an 850-band infrared camera, and converting the video data into images in image format; The image is annotated by a preset annotation tool to obtain an initial image of whether the underground personnel are wearing safety helmets; The initial image is subjected to Gaussian blur operation, flip transformation and rotation transformation to obtain a target image data set after the operation.

3. The method according to claim 1, characterized in that: The target image data set is input into the YOLOX network model to obtain an initial helmet detection model for distinguishing whether underground personnel are wearing helmets, including: When extracting features from the target image dataset using the backbone deep convolutional neural network of the YOLOX network model, a shape prior is introduced to take the elliptical area as the helmet sampling area at the center point; By using a deep convolutional neural network composed of a deformable convolution with a learnable offset, feature extraction is performed on the helmet sampling area to extract a feature map of the helmet area; Through the ECA-Net attention mechanism model added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, a global average pooling operation is performed on the feature map, local cross-channel interaction is performed through convolution with a kernel size of k, and the weight of each channel is obtained through a sigmoid activation function to recalibrate the feature map, thereby obtaining a helmet feature map that weakens or strengthens a specific area; Through the decoupling detection head of the head network, the obj branch and cls branch using the binary cross entropy loss function, the prediction reg branch using the generalized intersection-over-union loss function, and an angle loss term added to the generalized intersection-over-union loss function, the helmet feature map is decoupled to obtain the predicted image without the helmet and the angled image of the helmet. Based on the differences between the predicted images without helmets and the images with helmets at angles, and the images with helmets at angles and the images without helmets in the target image dataset, training is performed to obtain an initial helmet detection model.

4. The method according to claim 1, characterized in that: The helmet detection model after inference acceleration of the initial helmet detection model by TensorRT detects the real-time image of the underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time, including: Input the initial helmet detection model into the parser in TensorRT for parsing, delete the unused layers of the initial helmet detection model, and fuse the convolutional layers, pooling layers, and activation functions in the initial helmet detection model to generate the engine; Combine the networks in the engine horizontally based on the target parameters selected by the preset engine builder, and fuse the layers with the same tensors and operations together to obtain a serialized engine; The serialization engine is stored, and the serialization engine and the corresponding deserialization engine generate a helmet detection model for model reasoning to detect the real-time images of underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time.

5. The method according to claim 1, characterized in that The method further comprises: Deploy the corresponding engine file when TensorRT performs reasoning acceleration on the initial helmet detection model on the computing platform Jetson TX2, and then connect Jetson TX2 to the PLC module underground; When it is determined that the underground personnel are not wearing safety helmets in real time, the PLC module will give an alarm to the underground personnel who are not wearing safety helmets in real time.

6. A real-time detection device for underground personnel wearing safety helmets based on deep learning, characterized in that: The device comprises: The data acquisition module is used to acquire the initial image taken by the 850-band infrared camera to show whether the underground personnel are wearing helmets, and perform Gaussian blur operation, flip transformation, and rotation transformation on the initial image to obtain a target image data set after the operation, wherein the target image data set includes images of wearing helmets and images of not wearing helmets marked with helmet angles; A construction module for constructing a YOLOX network model, wherein the YOLOX network model includes a backbone deep convolutional neural network, a neck network and a head network, wherein an ECA-Net attention mechanism model is added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, and the obj branch and the cls branch in the decoupled detection head of the head network use a binary cross entropy loss function, and the prediction reg branch uses a generalized intersection-over-union loss function, and an angle loss term is added to the generalized intersection-over-union loss function, and the backbone deep convolutional neural network uses a deformable convolution with a learnable offset, and a shape prior is introduced when the deformable convolution performs sampling of a target image data set; A training module, used to input the target image data set into the YOLOX network model to train an initial helmet detection model for distinguishing whether underground personnel are wearing helmets; The detection module is used to detect the real-time images of underground personnel taken by the 850-band infrared camera using the safety helmet detection model after inference acceleration of the initial safety helmet detection model through TensorRT to determine whether the underground personnel are wearing safety helmets in real time.

7. The device according to claim 6, characterized in that The data acquisition module is specifically used for: Shooting video data of underground personnel based on an 850-band infrared camera, and converting the video data into images in image format; The image is annotated by a preset annotation tool to obtain an initial image of whether the underground personnel are wearing safety helmets; The initial image is subjected to Gaussian blur operation, flip transformation and rotation transformation to obtain a target image data set after the operation.

8. The device according to claim 6, characterized in that The training module is specifically used for: When extracting features from the target image dataset using the backbone deep convolutional neural network of the YOLOX network model, a shape prior is introduced to take the elliptical area as the helmet sampling area at the center point; By using a deep convolutional neural network composed of a deformable convolution with a learnable offset, feature extraction is performed on the helmet sampling area to extract a feature map of the helmet area; Through the ECA-Net attention mechanism model added to the feature layers of different scales of the path aggregation network in the neck network and the upsampling part of the feature pyramid network, a global average pooling operation is performed on the feature map, local cross-channel interaction is performed through convolution with a kernel size of k, and the weight of each channel is obtained through a sigmoid activation function to recalibrate the feature map, thereby obtaining a helmet feature map that weakens or strengthens a specific area; Through the decoupling detection head of the head network, the obj branch and cls branch using the binary cross entropy loss function, the prediction reg branch using the generalized intersection-over-union loss function, and an angle loss term added to the generalized intersection-over-union loss function, the helmet feature map is decoupled to obtain the predicted image without the helmet and the angled image of the helmet. Based on the differences between the predicted images without helmets and the images with helmets at angles, and the images with helmets at angles and the images without helmets in the target image dataset, training is performed to obtain an initial helmet detection model.

9. The device according to claim 6, characterized in that The detection module is specifically used for: Input the initial helmet detection model into the parser in TensorRT for parsing, delete the unused layers of the initial helmet detection model, and fuse the convolutional layers, pooling layers, and activation functions in the initial helmet detection model to generate the engine; Combine the networks in the engine horizontally based on the target parameters selected by the preset engine builder, and fuse the layers with the same tensor and operation together to obtain a serialized engine; The serialization engine is stored, and the serialization engine and the corresponding deserialization engine generate a helmet detection model for model reasoning to detect the real-time images of underground personnel taken by the 850-band infrared camera to determine whether the underground personnel are wearing helmets in real time.

10. The device according to claim 6, characterized in that The device further comprises: A deployment module is used to deploy the corresponding engine file when TensorRT performs reasoning acceleration on the initial helmet detection model on the computing platform Jetson TX2, and then connect Jetson TX2 to the PLC module underground; The alarm module is used to give an alarm prompt to the underground personnel who are not wearing safety helmets in real time through the PLC module when it is determined that the underground personnel are not wearing safety helmets in real time.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5.

Images