A method for detecting personnel under a mine based on a deep learning algorithm

Abstract

The application discloses a kind of personnel detection methods under mine based on deep learning algorithm, it is related to target detection technical field.The present application includes the following steps: 1) obtain monitoring video under mine;2) according to the monitoring video under mine obtained in step 1) obtains personnel monitoring image under mine, the luminance of the personnel monitoring image under mine is enhanced;3) based on YOLOX network, obtain the detection result of personnel under mine.Step 2): frame cutting is carried out to the monitoring video under mine, obtains dark light image, personnel and equipment information in dark light image are marked, and personnel monitoring image under mine is obtained;Image is enhanced using the deep learning network based on U-Net structure in luminance.Step 3): YOLOX network includes Backbone component, Neck component and DecoupleHead component.The present application effectively combines image luminance enhancement algorithm on YOLOX framework, by dark light image enhancement pre-processing to input data, further improves the detection ability of target detection algorithm, to accurately and efficiently execute detection task.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, specifically to a method for detecting personnel underground based on deep learning algorithms. Background Technology

[0002] Object detection is a very important technique in deep learning. Since the introduction of R-CNN in 2014, deep learning-based object detection technology has developed rapidly. It is also a core algorithm of intelligent monitoring systems and an important branch of image processing and computer vision.

[0003] Object detection technology can be divided into two types based on the number of stages: single-stage and two-stage. The core idea of ​​two-stage networks is to propose a bounding box, and then use the first-stage network to regress the approximate location, size, and probability of the target box being foreground. The second stage uses another network to regress the location, size, and category of the target box. The core idea of ​​single-stage networks is to directly regress the size, location, and category of the target in the image. Single-stage networks are often more widely used in real-world production environments due to their faster detection speed and higher accuracy.

[0004] YOLOX is an improved version of the well-known single-stage object detection algorithm YOLO. It primarily enhances the data augmentation capabilities of the original YOLOv3 network, adds decoupling to the head stage, and improves the label allocation strategy. Despite these improvements, it still performs end-to-end detection.

[0005] Dark image enhancement algorithms are used to enhance useful information in images, with the aim of improving the visual effect of the image. Currently, commonly used neural network-based image enhancement algorithms include SRIE, LIME, RetinexNet, and EnlightenGAN, and these algorithms are constantly being improved with the development of technology.

[0006] Current algorithms for detecting personnel underground primarily rely on real-time video image data from underground coal mine cameras and intelligent image recognition technology. They mainly utilize higher-performance camera equipment to address the problem of poor visibility underground, but this approach is costly. While personnel detection can be considered a specific application of object detection, algorithms that use deep learning for end-to-end brightness enhancement and personnel detection have not yet been truly applied to this area. Therefore, this method could be applied to personnel detection in underground mines. Summary of the Invention

[0007] To address the problem that deep learning algorithms have not yet been applied to the detection of personnel underground in existing target detection technologies, this invention provides a method for detecting personnel underground based on deep learning algorithms.

[0008] This invention is achieved through the following technical solution: a method for detecting personnel underground based on deep learning algorithms, comprising the following steps:

[0009] 1) Obtain underground monitoring video;

[0010] 2) Obtain the underground personnel monitoring image based on the underground monitoring video obtained in step 1), and enhance the brightness of the underground personnel monitoring image;

[0011] ①The method for obtaining underground personnel monitoring images is as follows: the underground monitoring video is segmented into frames to obtain a dark light image, and the personnel and equipment information in the dark light image is labeled to obtain the underground personnel monitoring image.

[0012] ② A deep learning network based on the U-Net structure is used to enhance the brightness of the monitoring images of personnel underground. The main task of the low-light image enhancement algorithm is to process the image data collected in the harsh underground environment, improve the image quality and clarity, and assist the target detection algorithm in detection. The main task of the target detection algorithm is to calculate and analyze the personnel information in the current input image, delineate the danger zone, and warn personnel in the danger zone. After brightness enhancement, a training set based on the underground monitoring video is obtained. The training set includes a first public dataset, which includes images of underground personnel and labels. The training set also includes a second private dataset, which includes the enhanced images of underground personnel and labels labeled by professionals. The labels include personnel tags.

[0013] ③ The low-light image is enhanced in brightness using a semantic segmentation-based U-Net deep learning network. The upsampling layers of this deep learning network are replaced by deconvolution layers, and the convolutional layers use dilated convolutions. The network also includes a brightness evaluation layer. This layer groups the features obtained from the last convolutional layer and calculates spatiotemporal consistency and color consistency image parameters based on these feature groups. The formula for the image brightness curve evaluation function used in the brightness evaluation layer is as follows:

[0014] A = A + α(1 - A);

[0015] Where A represents the given input; α represents the pixel gradient, which is used to control the exposure level, and the value of α ranges from [-1, 1].

[0016] The low-light image enhancement algorithm of this invention is based on an improvement of U-Net, but the upsampling layer is replaced by a deconvolution layer, the convolution layer is replaced by a dilated convolution, and a brightness evaluation layer composed of a brightness curve evaluation function is added after the output.

[0017] 3) Obtain the detection results of personnel underground based on the YOLOX network:

[0018] ① The YOLOX network comprises a Backbone component, a Neck component, and a DecoupleHead component. The Backbone component includes a CBL module, a CSP module, and an SPP module connected in sequence for extracting image features. The Neck component includes a CBL module, a Cat module, and a DeConv module for providing features at different scales. The DecoupleHead component includes a CBL module and a Decouple module for dimensionality reduction of the extracted features, generating bounding boxes, and predicting categories. The Backbone component also includes convolutional layers, normalization layers, activation layers, upsampling layers, pooling layers, and residual connections. The Neck component also includes upsampling layers.

[0019] This invention optimizes the structure of the Neck layer. The main change is to use a deconvolution layer instead of an upsampling layer to upsample small-scale features. This modification can better predict smaller targets in the receptive field, thereby improving prediction accuracy. Furthermore, underground mining workspaces are characterized by low illumination, complex and variable lighting conditions, and high concentrations of coal dust and water mist during the excavation process, resulting in low visibility.

[0020] ② The YOLOX network is trained using a loss function with edge detection capabilities, where the formula is:

[0021]

[0022] Where L represents the loss function with edge detection capability; Indicates preliminary classification of losses; This indicates an initial assessment of the loss; This represents the classification loss between the boundary prediction result and the actual data. P represents the localization loss between the boundary prediction result and the actual data. B C represents the predicted boundary classification score; * Represents the true bounding box; This represents the average loss based on the number of positive samples.

[0023] Strengthening the loss function for edge detection can enhance the detection of object contours within the algorithm's field of view. In the low-light environment underground, contours created by light occlusion or reflected light are more easily captured by the camera. Furthermore, thermal imaging exhibits significant color gradient changes. Therefore, supplementing the edge detection capability can substantially improve the algorithm's detection accuracy. Testing showed that adding the edge detection loss significantly improved target detection for underground personnel, with the most pronounced improvement in accuracy for target detection tasks using a combination of thermal imaging and RGB image detection.

[0024] Steps 2) and 3) of this invention correspond to the low-light image enhancement algorithm and the target detection algorithm, respectively. The low-light image enhancement algorithm is an improved image enhancement algorithm based on U-Net. It internally uses deconvolution, dilated convolution, and a brightness curve evaluation function to enhance the brightness of the image while preserving the color consistency of the original image. It uses the image perception exponential function as the loss function to constrain the degree of image brightness enhancement. Finally, the weights of the low-light image enhancement algorithm are trained using downhole images. The target detection algorithm is an improved YOLOX algorithm, which consists of a Backbone component, a Neck component, and a DecoupleHead component. These components can respectively achieve feature extraction, feature scale enrichment, and dimensionality reduction prediction. The main improvement is in the upsampling part of the Neck component, which uses deconvolution instead of bilinear interpolation and other upsampling methods. In addition, the loss function is modified based on the downhole environment.

[0025] To achieve the above process, the present invention also provides a mine personnel detection system based on a deep learning algorithm, including a mine monitoring video acquisition module, an image brightness enhancement module, and a detection result acquisition module. The mine monitoring video acquisition module is used to acquire mine monitoring video, corresponding to step 1). The image brightness enhancement module obtains mine personnel monitoring images from the acquired mine monitoring video and enhances the brightness of the mine personnel monitoring images, corresponding to step 2). The detection result acquisition module is based on the YOLOX network to acquire the detection results of mine personnel, corresponding to step 3).

[0026] Corresponding to the aforementioned deep learning algorithm-based underground personnel detection system, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the aforementioned deep learning algorithm-based underground personnel detection method.

[0027] Preferably, in the above-mentioned electronic device, the memory is a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-mentioned method for detecting personnel in mines based on a deep learning algorithm.

[0028] Compared with the prior art, the present invention has the following beneficial effects: The deep learning algorithm-based method for detecting personnel in mines provided by the present invention effectively combines an image brightness enhancement algorithm on the YOLOX framework. By performing preprocessing of the input data to enhance the dark light image, the detection capability of the target detection algorithm is further improved, thereby performing the detection task accurately and efficiently. When personnel are detected in the image of personnel in mines, the information from the video source is used to determine whether the personnel are in a dangerous area or engaging in dangerous behavior. Attached Figure Description

[0029] Figure 1 This is a flowchart of a method for detecting personnel underground based on a deep learning algorithm, according to the present invention.

[0030] Figure 2 This is a schematic diagram of a U-Net deep learning network structure, which is a method for detecting personnel in mines based on deep learning algorithms according to the present invention.

[0031] Figure 3 This is a YOLOX network diagram of a deep learning-based method for detecting personnel underground, as described in this invention.

[0032] Figure 4 for Figure 2 A diagram illustrating the Chinese translation of the text.

[0033] Figure 5 Practical effect diagram of a specific embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to specific embodiments.

[0035] A method for detecting personnel underground based on deep learning algorithms, such as Figure 1 As shown, it includes the following steps:

[0036] 1) Obtain underground monitoring video;

[0037] 2) Obtain the underground personnel monitoring image based on the underground monitoring video obtained in step 1), and enhance the brightness of the underground personnel monitoring image;

[0038] ①The method for obtaining underground personnel monitoring images is as follows: the underground monitoring video is segmented into frames to obtain a dark light image, and the personnel and equipment information in the dark light image is labeled to obtain the underground personnel monitoring image.

[0039] ② Use a deep learning network based on the U-Net structure to enhance the brightness of monitoring images of personnel underground, such as... Figure 2 As shown, the main task of the low-light image enhancement algorithm is to process image data collected in the harsh underground environment, improve image quality and clarity, and assist the target detection algorithm in detection. The main task of the target detection algorithm is to calculate and analyze personnel information in the current input image, delineate dangerous areas, and warn personnel in dangerous areas. After brightness enhancement, a training set based on underground monitoring videos is obtained. The training set includes a first public dataset, which includes images of underground personnel and labels. The training set also includes a second private dataset, which includes images of underground personnel after brightness enhancement and is labeled by professionals, including personnel labels.

[0040] ③ The low-light image is enhanced in brightness using a semantic segmentation-based U-Net deep learning network. The upsampling layers of this deep learning network are replaced by deconvolution layers, and the convolutional layers use dilated convolutions. The network also includes a brightness evaluation layer. This layer groups the features obtained from the last convolutional layer and calculates spatiotemporal consistency and color consistency image parameters based on these feature groups. The formula for the image brightness curve evaluation function used in the brightness evaluation layer is as follows:

[0041] A = A + α(1 - A);

[0042] Where A represents the given input; α represents the pixel gradient, which is used to control the exposure level, and the value of α ranges from [-1, 1].

[0043] The low-light image enhancement algorithm of this invention is based on an improvement of U-Net, but the upsampling layer is replaced by a deconvolution layer, the convolution layer is replaced by a dilated convolution, and a brightness evaluation layer composed of a brightness curve evaluation function is added after the output.

[0044] 3) Obtain the detection results of personnel underground based on the YOLOX network:

[0045] ① The YOLOX network comprises a Backbone component, a Neck component, and a DecoupleHead component. The Backbone component includes a CBL module, a CSP module, and an SPP module connected in sequence for extracting image features. The Neck component includes a CBL module, a Cat module, and a DeConv module for providing features at different scales. The DecoupleHead component includes a CBL module and a Decouple module for dimensionality reduction of the extracted features, generating bounding boxes, and predicting categories. The Backbone component also includes convolutional layers, normalization layers, activation layers, upsampling layers, pooling layers, and residual connections. The Neck component also includes upsampling layers, such as... Figure 3 and Figure 4 As shown.

[0046] In this embodiment, the Neck layer structure was optimized. The main change is that a deconvolution layer is used instead of an upsampling layer to upsample small-scale features. This modification can better predict smaller targets in the receptive field, thereby improving prediction accuracy. Furthermore, the working space in underground mines has low illumination and complex and variable lighting conditions, and the concentration of coal dust and water mist is high during the tunneling process, resulting in low visibility in this working environment.

[0047] ② The YOLOX network is trained using a loss function with edge detection capabilities, where the formula is:

[0048]

[0049] Where L represents the loss function with edge detection capability; Indicates preliminary classification of losses; This indicates an initial assessment of the loss; This represents the classification loss between the boundary prediction result and the actual data. P represents the localization loss between the boundary prediction result and the actual data. B C represents the predicted boundary classification score; * Represents the true bounding box; This represents the average loss based on the number of positive samples.

[0050] Strengthening the loss function for edge detection can enhance the detection of object contours within the algorithm's field of view. In the low-light environment underground, contours created by light occlusion or reflected light are more easily captured by the camera. Furthermore, thermal imaging exhibits significant color gradient changes. Therefore, supplementing the edge detection capability can substantially improve the algorithm's detection accuracy. Testing showed that adding the edge detection loss significantly improved target detection for underground personnel, with the most pronounced improvement in accuracy for target detection tasks using a combination of thermal imaging and RGB image detection.

[0051] Figure 5 This is the result of actual testing of the present invention in images of underground working faces. The left image is the original image, the middle image shows the change in image brightness after enhancement, and the right image is the result obtained after enhancement and detection. The workers in the right image are marked to indicate the target category and prediction confidence.

[0052] The process involves determining whether the deep learning network model meets the requirements. If it does, training is terminated; otherwise, the network is modified and retrained. Specifically, this involves assessing whether the deep learning network model achieves the expected brightness enhancement effect, whether it accurately identifies personnel working in the mine, whether it accurately identifies the activity areas of these personnel, whether it precisely determines whether personnel have entered dangerous areas, and whether it provides timely real-time warnings and feedback regarding violations by these personnel.

[0053] To achieve the above process, this embodiment also provides a mine personnel detection system based on a deep learning algorithm, including a mine monitoring video acquisition module, an image brightness enhancement module, and a detection result acquisition module. The mine monitoring video acquisition module is used to acquire mine monitoring video, corresponding to step 1). The image brightness enhancement module obtains mine personnel monitoring images from the acquired mine monitoring video and enhances the brightness of the mine personnel monitoring images, corresponding to step 2). The detection result acquisition module is based on the YOLOX network to acquire the detection results of mine personnel, corresponding to step 3).

[0054] Corresponding to the aforementioned deep learning algorithm-based underground personnel detection system, this embodiment also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the aforementioned deep learning algorithm-based underground personnel detection method. In the aforementioned electronic device, the memory is a computer-readable storage medium, and the computer-readable storage medium stores the computer program. When the computer program is executed by the processor, it implements the steps in the aforementioned deep learning algorithm-based underground personnel detection method.

[0055] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0056] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0057] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0058] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0059] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0060] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the energy-saving signal transmission / reception methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0061] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0062] It should be understood that the sequence number of each step in the invention and embodiments of the present invention does not absolutely imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Claims

1. A method for detecting personnel underground based on deep learning algorithms, characterized in that: Includes the following steps: 1) Obtain underground monitoring video; 2) Obtain the underground personnel monitoring image based on the underground monitoring video obtained in step 1), and enhance the brightness of the underground personnel monitoring image; The method for obtaining underground personnel monitoring images is as follows: the underground monitoring video is segmented into frames to obtain a low-light image, and the personnel and equipment information in the low-light image is labeled to obtain the underground personnel monitoring image; A deep learning network based on the U-Net architecture is used to enhance the brightness of underground personnel monitoring images, resulting in a training set based on underground monitoring videos. The training set includes a first public dataset, which includes underground personnel images and labels. The training set also includes a second private dataset, which includes the enhanced underground personnel images and labels labeled by professionals, including personnel labels. The low-light image is enhanced using a semantic segmentation-based U-Net deep learning network. The upsampling layers of this deep learning network are replaced with deconvolution layers, and the convolutional layers use dilated convolutions. The network also includes a brightness evaluation layer. This layer groups the features obtained from the last convolutional layer and calculates spatiotemporal consistency and color consistency image parameters based on these feature groups. The formula for the image brightness curve evaluation function used in the brightness evaluation layer is as follows: A = A + α(1 - A); Where A represents the given input; α represents the pixel gradient, which is used to control the exposure level, and the value of α ranges from [−1, 1]; 3) Based on the YOLOX network, obtain the detection results of personnel underground.

2. The method for detecting personnel underground based on a deep learning algorithm according to claim 1, characterized in that: In step 3), the YOLOX network includes a Backbone component, a Neck component, and a DecoupleHead component; The Backbone component includes a CBL module, a CSP module, and an SPP module connected in sequence; the Neck component includes a CBL module, a Cat module, and a DeConv module; the DecoupleHead component includes a CBL module and a Decouple module; the Backbone component also includes convolutional layers, normalization layers, activation layers, upsampling layers, pooling layers, and residual connections; the Neck component also includes an upsampling layer.

3. The method for detecting personnel underground based on a deep learning algorithm according to claim 2, characterized in that: In step 3), the YOLOX network is trained using a loss function with edge detection capabilities, where the formula is: ； Where L represents the loss function with edge detection capability; Indicates preliminary classification of losses; This indicates an initial assessment of the loss; This represents the classification loss between the boundary prediction result and the actual data. P represents the localization loss between the boundary prediction result and the actual data. B C represents the predicted boundary classification score; * Represents the true bounding box; This represents the average loss based on the number of positive samples.

4. The method for detecting personnel underground based on a deep learning algorithm according to claim 1, characterized in that: When a person is detected in the image of personnel in the mine, the system determines whether the person is in a dangerous area or engaging in dangerous behavior based on information from the video source.

Images