Novel intelligent video identification method for typical three illegal behaviors of intelligent coal mine

Abstract

The invention discloses a novel intelligent video identification method for typical three illegal behaviors of a smart coal mine, and belongs to the field of computer vision and deep learning. The method specifically comprises the following steps: firstly, screening video data containing three illegal behaviors according to cameras distributed in a coal mine, preprocessing the video data, segmenting the video data into single image frames, generating a data set by using image enhancement, and dividing the data set into a training set, a test set and a verification set; then, a Little-SPD module is introduced behind a convolutional layer of a backbone network and a neck network of the YOLOv8 network, a neural network is constructed cooperatively, and training and testing are carried out; and finally, processing a real-time video stream from a coal mine site by using the tested neural network, carrying out target detection and three illegal behavior judgment stages, judging whether the local fan safety helmet is compliant or not according to number matching, and judging the running states of the vehicle, people and the belt according to the running people and the belt in combination with the neural network. The detection accuracy is high; by improving data preprocessing and algorithm fusion, the accuracy of coal mine three-illegal behavior recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a new intelligent video recognition method for typical three violations in smart coal mines. Background Art

[0002] In today's rapidly developing era of industrial automation and intelligence, deep learning technology has shown great potential and application value in many fields, especially in coal mine production environments. Due to complex working conditions and extremely high safety requirements, traditional monitoring methods can no longer meet the needs of real-time and accurate safety monitoring. Therefore, it is urgent to develop an algorithm that can adapt to the special environment of coal mines and achieve efficient target detection and tracking.

[0003] The coal mine production environment is characterized by high risk and high complexity, and has strict requirements on workers' helmet wearing, belt conveyor operation status monitoring, and vehicle safety behavior during operation. Traditional monitoring methods often rely on manual inspections, which are inefficient and prone to omissions. Therefore, the development of an intelligent monitoring system that can automatically identify the "three violations" is of great significance to improving the safety level of coal mine production.

[0004] Deep learning technology has powerful feature extraction and classification capabilities; target detection and tracking technology enables the algorithm to monitor and track target objects in the video in real time; image enhancement technology improves recognition accuracy in complex environments; background difference rules are used to detect moving objects in the video, which is particularly suitable for static camera monitoring scenarios.

[0005] In the existing technology, the target detection and tracking algorithms face the problems of limited small target detection ability, sensitivity to hyperparameter adjustment, and slightly low positioning accuracy when processing video data in coal mine production environments. In addition, due to the particularity of the coal mine environment, such as large light changes, serious interference from mining lamps, and various monitoring angles, the existing algorithms are often difficult to adapt to these complex conditions, resulting in low detection and tracking accuracy.

[0006] Although existing methods such as the YOLOv8 algorithm perform well in target detection, they still have some technical problems and defects when dealing with special situations in coal mine production environments: for example, the detection capability of small targets is insufficient, and it is difficult to accurately identify small-sized safety helmets in complex backgrounds; the interference processing of strong light sources such as mining lamps is not effective enough, which can easily lead to false detection or missed detection; the adaptability to changes in monitoring angles is not strong, and it is difficult to maintain stable detection performance at different angles. Summary of the invention

[0007] In view of the above problems, the present invention proposes a new intelligent video recognition method for typical three-violation behaviors in smart coal mines. By introducing the Little-SPD module and image enhancement technology, the small target detection capability is effectively improved, the robustness of the algorithm to the interference of mining lamps is enhanced, and the adaptability under different monitoring angles is improved. At the same time, combined with the background difference method and the target tracking algorithm, it can realize the real-time monitoring and accurate identification of the "three-violation" behaviors in the coal mine production environment, filling the application gap of the existing technology in the special environment of coal mines.

[0008] The specific steps of the intelligent video recognition method for typical three violations in the new smart coal mine are:

[0009] Step 1: Distribute cameras in key areas of the coal mine to monitor and collect video data in real time, screen video data containing the three violations, and perform pre-processing;

[0010] Preprocessing includes decoding of video streams, preliminary correction of images, and quality checks;

[0011] Step 2: Split the continuous preprocessed video stream into single image frames and enter the image enhancement module to generate a data set.

[0012] The image enhancement module processes the image generated by the frame extraction by using a random combination of brightness and contrast adjustment, color correction, sharpening and noise reduction, and adaptive filtering for specific light sources;

[0013] Step 3: Divide the data set generated by the image enhancement module into a training set, a test set, and a validation set;

[0014] Step 4: Introduce the Little-SPD module after the convolutional layer of the backbone network and the neck network of the YOLOv8 network to collaboratively build the neural network and perform training and testing.

[0015] The specific structure of the neural network constructed is:

[0016] 1) Add the Little-SPD module to the YOLOv8 network, reduce the stride of the convolutional layer in YOLOv8 to stride=1, keep the kernel size as 3, and transform the input C1 channel feature map into the C2 channel feature map, where C2 is smaller than C1.

[0017] 2) The Little-SPD module uses the SPD module to slice the feature map of the C2 channel to generate two sub-feature maps: x[1::2,::2], x[::2,1::2], while retaining the channel dimension information.

[0018] Specifically: First, based on the scale of 2, apply Python's slicing operation to the feature map of size (S×S×c2). From the (S×S) plane, by selecting odd rows and even columns and even rows and odd columns for slicing, generate two sub-feature maps: f 1,0 and f 0,1 ;

[0019] The calculation is as follows:

[0020] f 0,1 =X[0:S:scale,1:S:scale]

[0021] f 1,0 =X[1:S:scale,0:S:scale]

[0022] S is the step size of the slicing operation, scale is the length and width of the plane of the feature map except the channel dimension, 0:S:scale starts from 0, scale ends, and the elements in it are taken out;

[0023] X[0:S:scale,1:S:scale] uses Python slicing operations to extract the odd rows and even columns of the plane of the feature map except the channel dimension; X[1:S:scale,0:S:scale] uses Python slicing operations to extract the even rows and odd columns of the plane of the feature map except the channel dimension.

[0024] 3) Concatenate these two sub-feature maps along the channel dimension to obtain a downsampled feature map with a shape of (S2×S2×2c2, where S2=S / 2), and pass it to the next convolution layer. The number of input channels of the next convolution operation should be twice the original one.

[0025] Step 5: Use the trained neural network to process the video stream from the coal mine site in real time to perform target detection.

[0026] Quickly identify multiple targets in the video, including but not limited to workers, equipment, vehicles, etc., and provide precise location information of the targets.

[0027] Step 6. Based on the target detection results, enter the three violations judgment stage, determine whether the local fan helmet is compliant based on quantity matching, and determine the operating status of the vehicle, people and belts based on the driving person and the neural network.

[0028] The background difference method is used to monitor moving targets in a fixed background. For example, the operating status of the belt conveyor can be judged by detecting moving targets within the range of the belt conveyor, and the three violations judgment rules can be combined to determine whether any abnormal situation occurs.

[0029] The Botsort algorithm is used to focus on the movement trajectory of the target in continuous video frames. By analyzing the target's moving direction, speed and position changes, potential violations can be identified, such as workers entering dangerous areas without wearing safety helmets and improper equipment operation.

[0030] The advantages of the present invention are:

[0031] 1) The present invention provides a new type of intelligent video recognition method for typical three-violation behaviors in smart coal mines, which is widely applicable. By combining with target tracking and background difference algorithms, it can realize the detection of a variety of typical three-violation behaviors and has the potential to be promoted to more application scenarios.

[0032] 2) The present invention provides a new type of intelligent video recognition method for typical three violations in smart coal mines, which has high detection accuracy. Through the study of the coal mine production environment, the neural network structure is optimized, and improvements are made to data preprocessing and algorithm fusion based on the characteristics of the data set, thereby improving the accuracy of the video recognition method for the three violations in coal mines.

[0033] 3) The present invention provides a new type of intelligent video recognition method for typical three violations in smart coal mines. By monitoring and identifying the three violations in coal mines, safety hazards can be discovered in a timely manner, the possibility of accidents can be reduced, and a warning effect can be provided to employees. It helps to reduce human resource consumption and time costs and improve the overall management level of coal mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of a novel intelligent video recognition method of typical three violations in smart coal mines according to the present invention;

[0035] Figure 2 Schematic diagram of a YOLO grid unit of the present invention;

[0036] Figure 3 This is the YOLOv8 network structure diagram of the present invention;

[0037] Figure 4 This is a structural diagram of the Nostride-Conv-SPD module of the present invention;

[0038] Figure 5 This is a schematic diagram of the principle of the non-stride convolution operation adopted in the present invention;

[0039] Figure 6 The schematic diagram of the Little-SPD module of the present invention performing a slicing operation on a feature map;

[0040] Figure 7 This is the structure diagram of the YOLOv8-Little-SPD neural network of the present invention;

[0041] Figure 8It is a schematic diagram of detecting a person in a video according to an embodiment of the present invention;

[0042] Fig. 9 A schematic diagram of detecting the number of local fans and local fan lights according to an embodiment of the present invention;

[0043] Fig.10 A schematic diagram of detecting the running state of a belt and whether there is someone in a restricted area according to an embodiment of the present invention;

[0044] Fig.11 This is a schematic diagram of the running status of the tracking vehicle according to an embodiment of the present invention;

[0045] Fig.12 This is a schematic diagram of the running status of tracking a person according to an embodiment of the present invention;

[0046] Fig.13 Schematic diagram of detecting a safety helmet according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0048] The present invention proposes a new intelligent video recognition method for typical three violations in smart coal mines. Based on deep learning target detection and behavior analysis technology, it can identify and track workers, equipment and objects in the coal mine production environment in real time, and judge whether their status and behavior meet safety regulations. Through the YOLO network, combined with the Little-SPD module and the image enhancement module, the background difference method and the target tracking algorithm are used to significantly improve the recognition accuracy and recall rate of the three violations in coal mines.

[0049] The intelligent video recognition method of typical three violations in the new smart coal mine is as follows: Figure 1 As shown, the specific steps are:

[0050] Step 1: Distribute cameras in key areas of the coal mine to monitor and collect video data in real time, screen video data containing the three violations, and perform pre-processing;

[0051] In the complex environment of coal mines, real-time monitoring and data collection are crucial; cameras, as the source of data, are carefully placed in key areas of coal mines, such as wellheads, tunnels, and mining faces, to ensure full coverage. These cameras not only need to capture clear video streams, but must also be able to adapt to the special conditions of coal mines, such as low illumination and high dust.

[0052] The collected video data is then transmitted to the central processing unit for preliminary video data preprocessing; the preprocessing stage includes decoding of the video stream, preliminary correction and quality inspection of the image to eliminate possible noise and distortion, laying a solid foundation for subsequent image enhancement and target detection.

[0053] Step 2: Split the continuous preprocessed video stream into single image frames and enter the image enhancement module to generate a data set.

[0054] After the video data preprocessing is completed, data frame extraction and image enhancement are performed; frame extraction is the process of dividing the continuous video stream into individual image frames, which helps the subsequent deep learning model processing. The image data after frame extraction then enters the image enhancement module to adapt to the subsequent deep learning model;

[0055] The image enhancement module processes the image generated by the frame extraction by using a random combination of brightness and contrast adjustment, color correction, sharpening and noise reduction, and adaptive filtering for specific light sources;

[0056] Considering the changing light conditions in coal mine environments, the image enhancement module uses a series of technologies, such as color space-based image enhancement algorithms and image transformation-based image enhancement algorithms, to process the images generated by frame extraction to improve image quality, reduce interference from light sources such as mining lamps, and enhance the visibility of small targets. The enhanced images are divided into training sets, test sets, and validation sets in a ratio of 6:2:2 for training and evaluating models. This step provides high-quality input data for neural network training. Image enhancement greatly increases the visibility of small targets (such as helmets) in complex backgrounds, providing clearer and more accurate input data for subsequent deep learning models.

[0057] Step 3: Divide the data set generated by the image enhancement module into a training set, a test set, and a validation set;

[0058] Step 4: Introduce the Little-SPD module after the convolutional layer of the backbone network and the neck network of the YOLOv8 network to collaboratively build the neural network and perform training.

[0059] The YOLO algorithm is an algorithm that uses convolutional neural networks for target detection. Compared with the recognition algorithm, this detection algorithm not only predicts the category label, but also predicts the location of the detected object, and supports the simultaneous detection of multiple different objects in the image. Unlike the traditional R-CNN series of models, which need to extract image regions before feature extraction, the YOLO algorithm only needs to directly extract features from a picture to simultaneously achieve the task of predicting bounding boxes and object categories. Therefore, the YOLO algorithm is an algorithm suitable for fast recognition and is usually used to complete target detection tasks in real-time systems. YOLO is a single-stage target detection method that uses a separate CNN model to achieve end-to-end target detection. The YOLO series of algorithms has been updated and iterated to the v8 stage, and is constantly improving the accuracy and robustness of the algorithm.

[0060] like Figure 2 The following is a YOLO grid cell. Taking the PASCAL VOC dataset as an example, YOLOv1 uses a 7×7 grid, and each cell regresses 2 bounding boxes and 20 conditional category probabilities. The conditional category probability is the probability that the detected object belongs to a specific category (each grid cell has a probability for each category). The final prediction features consist of the location of the bounding box, the confidence score of the bounding box, and the category probability.

[0061] YOLOv8 is one of the most advanced target detection frameworks, known for its high speed and high accuracy, and is the core component of this neural network. Figure 3 As shown in the figure, the training process of the YOLOv8 network involves iterative learning of a large amount of image data, aiming to teach the model to identify and locate key targets in the coal mine environment. In order to further improve the performance of YOLOv8 in coal mine scenarios, this application introduces the Little-SPD module in the backbone layer and the neck layer. This module significantly enhances the model's detection ability for small-sized targets through streamlined convolution operations and optimized space-to-depth conversion strategies, while reducing computational complexity, allowing the model to run efficiently on devices with limited resources.

[0062] The training process involves not only the adjustment of model weights, but also the continuous optimization of training data to ensure that the model can be generalized to image data in various coal mining environments.

[0063] The module structure of Nostride-Conv-SPD (No-stride convolution-spatial pyramid pooling-spatial to depth layer) is as follows Figure 4As shown in the figure, it is a combination technology used in deep learning target detection algorithms, which has a good improvement effect on the YOLO series of algorithms. In convolutional neural networks, the stride refers to the interval at which the convolution kernel slides. Strideless convolution refers to changing the original stride of 2 in the backbone network convolution layer to 1 to maintain the consistency of the spatial dimensions of the input and output. Strideless convolution requires more convolution operations and can retain more spatial details. Some of the latest architectures have proposed designs that completely eliminate the convolution stride and pooling layer, retaining more fine-grained information and improving the learning efficiency of feature representation.

[0064] The Scale-Aware Distillation (SPD) module is the space_to_depth module after the convolution layer. After the convolution layer, a three-dimensional feature map is generated, and the feature map is sliced ​​in its non-channel dimension, which can be understood as a two-dimensional plane dimension. Then select the even rows and even columns of this two-dimensional one-sided tensor to form a tensor, select the odd rows and even columns of this two-dimensional plane tensor to form a tensor, select the even rows and odd columns of the tensor to form a tensor, select the odd rows and odd columns of the tensor to form a tensor, and then splice these four three-dimensional tensors in the channel dimension to form a feature map, and then enter Figure 4 The C2 module in .

[0065] In view of the problem of fine-grained feature information loss caused by the use of stride convolution (stride>=2) in most convolution layers in the YOLOv8 backbone network and the neck network, this example reduces the stride of most convolution layers in YOLOv8 to stride=1, thereby enhancing the feature extraction efficiency of the convolution layer. In order to solve the problems of extended training time and large number of parameters caused by directly adding the SPD module to the YOLOv8 network, this embodiment proposes a simplified SPD (Little-SPD) module. This module not only maintains the detection efficiency of small targets, but also improves the training speed, reduces the number of model parameters and computational complexity, and reduces the dependence on GPU configuration.

[0066] Initially, the convolution operation uses a stride of 2 and a kernel size of 3 to extract features. Now, a strideless convolution operation is used, such as Figure 5 As shown in the figure, the stride is reduced to 1 while keeping the kernel size at 3 to achieve more refined feature extraction. This process transforms the input C1 channel feature map into a C2 channel feature map, where C2 is smaller than C1. Using a stride of 1 during feature extraction can retain as much feature information as possible while reducing the number of channels.

[0067] like Figure 6As shown in the figure, the Little-SPD module uses the SPD module to slice the feature map while retaining the channel dimension information. First, based on the scale of 2, Python's slicing operation is applied to the feature map of size (S×S×c2). From the (S×S) plane, sub-feature maps are generated by selecting odd rows and even columns and even rows and odd columns for slicing, and f1,0 and f0,1 are obtained, resulting in a total of 2 sub-feature maps. Subsequently, these two sub-feature maps are concatenated along the channel dimension to obtain a downsampled feature map of shape (S2×S2×2c2, where S2=S / 2), which is passed to the next convolution layer. Of course, the number of input channels for the next convolution operation should be twice the original. The sub-feature maps are extracted from the odd rows and even columns and the even rows and odd columns of the original feature map, respectively, to ensure that the integrity of the feature information is preserved.

[0068] The calculation is as follows:

[0069] f 0,1 =X[0:S:scale,1:S:scale]

[0070] f 1,0 =X[1:S:scale,0:S:scale]

[0071] S is the step size of the slicing operation. In this article, the step size is 2. scale is the length and width of the plane of the feature map except the channel dimension. 0:S:scale starts from 0 and ends with scale. The step size is 2, and the elements are taken out.

[0072] X[0:S:scale,1:S:scale] uses Python slicing operations to extract the odd rows and even columns of the plane of the feature map except the channel dimension; X[1:S:scale,0:S:scale] uses Python slicing operations to extract the even rows and odd columns of the plane of the feature map except the channel dimension.

[0073] Compared to the original baseline network of YOLOv8, the YOLOv8-Little-SPD neural network completely removes the strided convolution operation. This change retains the fine-grained details of the image during the slicing process and improves the model's ability to detect small targets. In the YOLOv8-Little-SPD neural network, the channel dimension of the SPD module is reduced from 4 to 2 compared to the standard YOLOv8-nostride-SPD module. Therefore, the number of input channels of the convolutional layer after the SPD layer is also reduced from 4 to 2, resulting in a reduction in the number of parameters, a reduction in gradient descent, and a corresponding reduction in GFLOPS (billion floating-point operations per second), making the network more lightweight. The network structure is as follows Figure 7 shown.

[0074] Step 5: Use the trained neural network to process the video stream from the coal mine site in real time to perform target detection.

[0075] After the neural network training is completed, it enters the actual application stage to perform target detection; in this stage, the trained YOLOv8 network and Little-SPD module work together to process the video stream from the coal mine site in real time.

[0076] The neural network can quickly identify multiple targets in the video, including but not limited to workers, equipment, vehicles, etc., while providing accurate location information of the targets. This step is one of the most challenging parts of the entire system because it needs to achieve high processing speed while maintaining high accuracy to meet the real-time monitoring needs in coal mine environments.

[0077] Step 6. Based on the target detection results, enter the three violations judgment stage, determine whether the local fan helmet is compliant based on quantity matching, and determine the operating status of the vehicle, people and belts based on the driving person and the neural network.

[0078] The compliance of the local fan and safety helmet is determined based on the quantity match. Generally speaking, the number of local fans and switches detected by the YOLOv8-little-SPD neural network is twice the number, and the number of safety helmets and people matches. If they do not match, an alarm will be triggered.

[0079] In judging the operating status of the vehicle and the belt, it not only relies on the output of the neural network, but also combines the background difference method and target tracking algorithm to achieve accurate identification of violations in the coal mine production environment.

[0080] The background difference method is used to monitor moving targets under a fixed background, such as detecting the running status of a belt conveyor and judging whether any abnormal situation occurs. The background difference algorithm is used on the video frame to obtain a moving still mask map, and a box is selected for the running area of ​​the belt to obtain a polygonal mask. The two are combined to calculate the proportion of the moving area in the belt to determine the running status of the belt conveyor.

[0081] The target tracking algorithm focuses on the motion trajectory of the target in continuous video frames. By analyzing the change amplitude of the vehicle trajectory, the operating status of the vehicle can be identified.

[0082] According to the running status of the belt and the vehicle, whether the center point of the human motion prediction box detected by the YOLOv8-little-SPD neural network is within the belt frame and the vehicle detection frame, the two three violations of not allowing people in the restricted area during the operation of the belt and not allowing people to get off the vehicle while it is running are judged.

[0083] Through this series of technical combinations, all-round monitoring of the coal mine production environment can be achieved to ensure that safety regulations are strictly followed.

[0084] This paper proposes a deep learning-based target detection model (YOLOv8-Little-SPD) for detecting small targets in coal mines, such as safety helmets and local fan switches. Moreover, the motion state of different targets is evaluated: the background difference algorithm is introduced to detect the stable motion of the belt conveyor, while the Bot-SORT tracking algorithm is used to detect targets such as mine cars and workers, in which the center point of the target is constantly changing. Finally, a coal mine underground target detection dataset named Coal-Object-dataset was constructed, and a series of related experiments were conducted on it. The mAP50 of the YOLOv8-Little-SPD model reached 97.6%, an increase of 1.4% compared with the baseline model YOLOv8. The mAP50-95 of the YOLOv8-Little-SPD model reached 66.9%, an increase of 2.5 percentage points compared with the baseline YOLOv8, effectively realizing the detection of 5 common targets in coal mines.

[0085] Example:

[0086] The environment of the entire implementation platform of this embodiment is:

[0087] CPU: 12th Gen Intel(R)Core(TM)i7-12700H2.30 GHZ

[0088] GPU: NVIDIA GeForce RTX 3060

[0089] Operating system: Windows 11

[0090] Code environment: Python 3.9

[0091] CUDA version: 12.3

[0092] Deep learning framework: Pytorch1.12.0

[0093] The intelligent video recognition method for typical three violations in the new smart coal mine can achieve the following functions:

[0094] Function 1: Get the worker's working range.

[0095] Since workers work for a short time, it is necessary to obtain the workers' working intervals to facilitate subsequent manual or model detection of other "three violations" of workers. This function is applicable to all coal mine production monitoring environments and needs to cover all monitoring videos. Therefore, the monitoring consumes the most resources and requires the algorithm to have a high computing rate.

[0096] The difficulty in implementing this function lies in:

[0097] Monitoring angle problem: Due to different monitoring positions, there are many types of human postures in video images. To eliminate the influence of human posture on model detection, the data set must be diverse and the model must be highly applicable.

[0098] Impact of mining lamps: In video surveillance, mining lamps can easily cause the surrounding area to be blurred, making it impossible to fully display the human body posture, resulting in the inability of target recognition algorithms to correctly identify the target.

[0099] In the process of implementing this function, workers wearing helmet lights may cause the human body in the monitoring image to be blurred. To solve this interference, a corresponding image enhancement scheme is introduced for the recognition of workers' work. For the labeled images containing workers, local image enhancement based on color space transformation is performed on the worker area to repair the impact of the helmet light on the monitoring image and reduce the interference of the helmet light on the detection algorithm. In addition, other human detection datasets in non-coal mine production environments are introduced. Through image enhancement based on color space transformation, the situation of wearing helmet lights in coal mine production environments is simulated to increase the diversity of datasets and improve the generalization ability of the model. This function determines whether there are workers in the video image. If so, it can be determined that the video belongs to the worker's work area. It belongs to the target detection task in deep learning. It is implemented using the YOLOv8 model and the YOLOv8+Little-SPD model (mainly for targeted adjustment of the helmet light and the light in the coal mine production environment), and more than 30,000 datasets collected from the video surveillance system are used for training and verification.

[0100] Function 2: Determine the local fan monitoring status.

[0101] The status of the local fan is mainly determined by the status of the two switches on the front of the local fan. In order to ensure that the camera monitoring the status of the local fan can clearly reflect the status of the local fan, the camera can be judged whether it is placed correctly based on the overall local fan and the local fan switch status. In the absence of special interference, the local fan video monitoring within a certain period of time is relatively stable, and the camera placement and local fan placement are basically stable. Therefore, when applying this function, you can use a custom time interval to periodically judge the local fan monitoring status to reduce the consumption of invalid computing resources.

[0102] The difficulty in implementing this function lies in:

[0103] 1) Misdetection of local fan lights: Since there are not only local fan lights in the monitoring scene, but also similar detection lights of other devices, the target detection algorithm may misdetect the lights of other devices, resulting in errors in the judgment of the local fan monitoring status.

[0104] 2) Monitoring angle problem: Due to the poor placement of some monitoring devices, some local fan lights are only thin square strips in the image and can only be barely distinguished by the naked eye. The detection algorithm may miss detections, resulting in errors in the judgment of the local fan monitoring status.

[0105] 3) Object occlusion: In coal mine production environments, video visibility may be limited, and there may be pipes or other objects blocking the view and interference from mining lamps. These special conditions may cause errors in the detection algorithm and lead to misjudgment.

[0106] In the process of implementing this function, due to the inconsistency of monitoring perspectives, the size and proportion of local fan lights may vary greatly. In order to improve the generalization ability of the model, targeted image enhancement measures are used to perform geometric change enhancement operations such as stretching and rotating the local fan lights and local fan areas in the existing image data, and randomly simulate images taken from different angles. Through this image enhancement method, the trained model will have stronger adaptability and can better cope with the possible increase in local fan monitoring scenarios and other unlabeled scenarios.

[0107] For cameras that monitor the status of local fans, ensuring that the local fans under monitoring are completely visible and the status of the local fans can be accurately determined is a target detection task in deep learning. It is necessary to determine whether there are blocked local fans or blurred local fan lights in the video image. This is achieved using the YOLOv8 model and the YOLOv8+Little-SPD model (mainly for enhancing operations on changes in the size and proportion of local fan lights caused by the monitoring angle), and more than 30,000 data sets are used for training and verification. By detecting the overall number of local fans in video surveillance and the number of local fan lights within the local fan range, a mapping relationship between the overall local fans and the local fan lights can be formed. Based on the mapping relationship, it can be determined whether the local fans under monitoring are completely visible and the local fan monitoring status can be determined.

[0108] Function 3: Detect workers’ work when the belt conveyor is running.

[0109] During the operation of the belt conveyor, workers are prohibited from working inside the belt conveyor to avoid safety problems. In order to detect the "three violations", it is necessary to judge the belt operation status, determine the area inside the belt conveyor and monitor the workers.

[0110] The difficulty in implementing this function lies in:

[0111] 1) Selection of the inner area of ​​the belt conveyor: The shape of the inner area of ​​the belt conveyor is different in video surveillance. Due to the monitoring shooting angle, there are shapes such as rings and belts, which makes it difficult to label. As a result, it is impossible to automatically generate the inner area of ​​the belt conveyor based on the monitoring video. Therefore, it is necessary to first select the frame and then detect it to solve this problem.

[0112] 2) Empty belt conveyor running status detection: The running status of the belt conveyor needs to be judged based on the background difference method, and the basis for judgment is the change of the image in the belt area.

[0113] 3) Worker detection: The deep learning method used in the function of obtaining the worker's work area is used for detection. When it is detected that a worker has entered a restricted area and the belt is in motion, three violations are judged.

[0114] This function involves two deep learning tasks: motion state detection and target detection. The running state of the belt needs to first determine the coordinate position of the belt, and then analyze the motion state of the belt coordinate frame. This function is similar to the worker work interval detection function, both of which require workers to be detected. Therefore, in this task, corresponding local image enhancement based on color space transformation is also required to improve the generalization ability of the model. Grayscale and other data enhancement based on color space transformation are required for the belt conveyor area to meet the requirements of the background difference algorithm. For the worker work detection task when the belt conveyor is running, the background difference method and YOLOv8 network + Little-SPD model are combined with the image enhancement module (mainly for targeted adjustment of the safety helmet lamp and the light of the coal mine production environment, and grayscale operations on the belt conveyor), and more than 30,000 data sets are used for training and verification. Combined with all the detection and judgment results, the "three violations" behavior can be detected according to logical operations.

[0115] Function 4: Vehicle exit detection while driving.

[0116] During the operation of a manned vehicle, workers are prohibited from getting off the vehicle in advance to avoid causing safety problems. In order to detect the "three violations", it is necessary to first detect the target object for target tracking (vehicle, person), determine the object's motion state (vehicle, person), and then determine whether a person has gotten off the moving vehicle based on the position relationship between the target object detection frame and the object's state.

[0117] The difficulty in implementing this function lies in:

[0118] 1) Lack of real data: Since the "three violations" are serious violations, it is impossible to obtain real video data. It is necessary to simulate the "three violations" based on the existing video data to generate video data that can be used to verify the accuracy of the algorithm.

[0119] 2) Interference from workers on moving vehicles: According to the analysis of existing video data, workers in the rear of the vehicle can be seen by video surveillance. During the inspection, it is necessary to judge the people on the vehicle to avoid interference with the inspection results.

[0120] 3) Monitoring angle makes it difficult to determine parameters: The shapes of the vehicle detection frame and the human detection frame are very different at different monitoring angles, which makes it difficult to judge the position relationship between the vehicle and the worker, and requires specific confirmation of the judgment threshold.

[0121] This function involves three deep learning tasks: target detection, object motion state detection, and target tracking. This function is similar to the worker work zone detection function, both of which require the detection of workers. Therefore, in this function, corresponding local image enhancement based on color space transformation is also required to improve the generalization ability of the model. For the detection of vehicle operation status, it is necessary to use a target tracking algorithm to track the vehicle, and judge it according to the position change of the detection frame of the vehicle target. If the position change exceeds a certain threshold, it is judged that the vehicle is in motion. For the situation of workers getting off the vehicle, it is necessary to use a target tracking algorithm to number the workers detected in the video surveillance, and judge whether the workers get off the vehicle according to the worker number and the position relationship between the human body and the vehicle. More than 10,000 data sets are used for training and verification. Combined with all the detection and judgment results, the "three violations" behavior can be detected according to logical operations.

[0122] Function 5: Helmet detection.

[0123] Wearing a safety helmet is a basic requirement for workers in coal mine production environments. This function is applicable to all coal mine production monitoring environments and needs to cover all monitoring videos. Therefore, the amount of resources consumed by monitoring is the largest, requiring the algorithm to have a higher computing rate.

[0124] The difficulty in implementing this function lies in:

[0125] 1) Monitoring angle problem: Due to different monitoring positions and different impacts of helmet lights, the data set must be diverse and the model must be highly applicable.

[0126] 2) Impact of mining lamps: In video surveillance, mining lamps can easily cause the surrounding area to be blurred, making it difficult to determine the specific shape of the mining lamp, which may lead to errors in data labeling and deviations in the detection frame range of the target recognition algorithm.

[0127] In the process of implementing this function, the helmet light will cause the helmet in the monitoring image to be blurred. In order to solve this interference on the recognition of the helmet, the corresponding image enhancement scheme is introduced to perform local image enhancement on the helmet light. By changing the brightness and contrast, the image enhancement method based on the change of color space is used to reduce the impact of the helmet light on the detection. In addition, for image data with little impact from the helmet light or the helmet light is not turned on, the same method can be used to amplify the impact of the helmet light to simulate possible situations.

[0128] This function needs to be judged based on the correspondence between workers and helmets in video images. It is a target detection task in deep learning and is implemented using the YOLOv8 model and the YOLOv8+Little-SPD model (mainly for targeted adjustments to helmet lights and coal mine production environment light), and uses more than 30,000 data sets for training and verification.

[0129] The results show that:

[0130] The results of the algorithm's implementation are analyzed as follows:

[0131] 1) For the function of obtaining the worker's work interval, such as Figure 8 As shown in the figure, it is the detection of people in the video. In the target detection task, people can be identified in each frame of the video, and the video detection results of each frame are combined to form a coal mine working environment video containing only people. The YOLOv8 model and the YOLOv8+Little-SPD model were used for experiments. The results of the worker detection experiment are shown in Table 1:

[0132] Table 1

[0133] algorithm Precision Recall mAP (mean Average Precision) YOLOv8 0.935 0.928 0.971 YOLOv8+Little-SPD 0.950 0.955 0.983

[0134] From the table above, we can see that in the worker detection task, the YOLOv8+Little-SPD algorithm has a higher recall rate, but not as good as the YOLOv8 algorithm in terms of accuracy. The map value is the data of comprehensive accuracy and recall rate evaluation model effect. It can be seen that the comprehensive evaluation index map of the improved yolov8+Little-SPD has been significantly improved. The two algorithm models can be combined to process worker detection. Overall, by combining the two algorithm human detection tasks, a higher accuracy rate can be guaranteed and can be used in practical applications.

[0135] 2) To determine the local fan monitoring status function, such as Fig. 9 As shown in the figure, it is a schematic diagram of the number of local fans and local fan lights. If the number of local fan lights matches the number of local fans in the detection task, it is considered normal behavior; if the number of local fan lights does not match the number of local fans, it is considered three violations. The YOLOv8 model and the YOLOv8+Little-SPD model were used for experiments. The experimental results of the detection of the local fan as a whole and the local fan lights are shown in Table 2 and Table 3:

[0136] Table 2

[0137] algorithm Precision Recall mAP (mean Average Precision) YOLOv8 0.994 0.992 0.992 YOLOv8+Little-SPD 0.996 0.994 0.994

[0138] Table 3

[0139] algorithm Precision Recall mAP (mean Average Precision) YOLOv8 0.965 0.964 0.972 YOLOv8+Little-SPD 0.975 0.973 0.985

[0140] From the above table, we can see that in the local fan detection task, YOLOv8 has higher precision and recall than YOLOv8+Little-SPD algorithm, except that the map value is almost the same. Therefore, in the task of judging the number of local fans, YOLOv8 algorithm can be used for detection. In the local fan light detection task, YOLOv8+Little-SPD has higher recall and map value than YOLOv8 algorithm, except that the precision is not as good as YOLOv8. Therefore, in the task of judging the number of local fan lights, YOLOv8+Little-SPD algorithm can be used for detection, and the two algorithm models are combined to process the overall detection of local fans. Overall, by combining the two algorithms, the local fan detection task can ensure a high accuracy rate and can be used in practical applications.

[0141] 3) Detection of workers' work when the belt conveyor is running, such as Fig.10 As shown in the figure, it is a schematic diagram for detecting the running status of the belt and whether there are people in the restricted area. The belt running area and the restricted area are selected, and the running status is detected using the background difference method in the belt running area. The target detection method in the restricted area is used to detect whether there are people in the restricted area. If the belt is in running state and there are people in the restricted area, the video is deemed to be three violations.

[0142] The YOLOv8 model and the YOLOv8+Little-SPD model were used to conduct experiments to detect people in the process of video belt operation. The experimental results are shown in Table 4:

[0143] Table 4

[0144] algorithm Precision Recall mAP (mean Average Precision) YOLOv8 0.935 0.928 0.971 YOLOv8+Little-SPD 0.950 0.955 0.983

[0145] From the table above, we can see that in the local fan light detection task, YOLOv8+Little-SPD has almost the same accuracy, but the recall rate and map value are significantly higher than the YOLOv8 algorithm. Therefore, in the task of detecting people during the belt running process, YOLOv8 can be combined with the Little-SPD algorithm for detection, and the two algorithm models can be combined to process the overall local fan detection. By combining the two algorithms, the detection task of people can ensure a higher accuracy rate and can be used in practical applications.

[0146] 4) For vehicle detection function in driving state, such as Fig.11 As shown in Figure 1, it is a schematic diagram of the running status of the tracking vehicle. Fig.12 As shown in the figure, it is a schematic diagram of the running state of the tracked person. If the horizontal coordinate of the center point of the detection frame of the person in motion is within the horizontal coordinate range of the detection frame of the vehicle in motion, and the vertical coordinate of the center point of the detection frame of the person in motion is within the vertical coordinate range of the detection frame of the vehicle in motion, then this frame is judged as a three-violation video. If there are more than several frames in the video, then the video is judged as a three-violation video.

[0147] The YOLOv8 model and the YOLOv8+Little-SPD model were used to conduct experiments to detect people in the process of video belt running. The experimental results are shown in Table 5:

[0148] Table 5

[0149] algorithm Precision Recall mAP (mean Average Precision) YOLOv8 0.979 0.990 0.987 YOLOv8+Little-SPD 0.924 0.845 0.931

[0150] From the table above, we can see that in the task of detecting local fan lights, the recall rate and map value of YOLOv8+Little-SPD are almost the same as those of YOLOv8, and the accuracy is significantly higher than that of the YOLOv8 algorithm. Therefore, in the task of detecting people during the belt running process, YOLOv8 combined with the Little-SPD algorithm can be used for detection, and the two algorithm models can be combined to process the overall detection of local fans. By combining the two algorithms, the detection task of people can ensure a high accuracy rate and can be used in practical applications.

[0151] 5) For helmet detection function, such as Fig.13 The figure shows the detection diagram of the helmet. If the number of helmets in the figure does not match the number of people, it is judged that three violations have occurred. The YOLOv8 model and the YOLOv8+Little-SPD model were used to conduct experiments to detect the helmets in the video. The experimental results are shown in Table 6:

[0152] Table 6

[0153] algorithm Precision Recall mAP (mean Average Precision) YOLOv8 0.919 0.797 0.888 YOLOv8+Little-SPD 0.964 0.992 0.989

[0154] From the table above, we can see that the accuracy, recall rate and map value of YOLOv8+Little-SPD in the helmet detection task are significantly higher than those of the YOLOv8 algorithm. Therefore, in the helmet detection task, YOLOv8 can be combined with the Little-SPD algorithm for detection, and the two algorithm models can be combined to process the overall detection of the local fan. By combining the two algorithms, the helmet detection task can ensure a higher accuracy rate and can be used in practical applications.

Claims

1. A new intelligent video recognition method for typical three violations in smart coal mines, characterized by: The specific steps are as follows: Step 1: Distribute cameras in key areas of the coal mine to monitor and collect video data in real time, screen video data containing the three violations, and perform pre-processing; Step 2: Segment the continuous pre-processed video stream into single image frames and enter the image enhancement module to generate a data set; Step 3: Divide the data set generated by the image enhancement module into a training set, a test set, and a validation set; Step 4: Introduce the Little-SPD module after the convolutional layer of the backbone network and the neck network of the YOLOv8 network to collaboratively build the neural network and perform training and testing; The specific structure of the neural network constructed is: 1) Add the Little-SPD module to the YOLOv8 network, reduce the stride of the convolutional layer in YOLOv8 to stride=1, keep the kernel size as 3, and transform the input C1 channel feature map into the C2 channel feature map, where C2 is smaller than C1; 2) The Little-SPD module uses the SPD module to slice the feature map of the C2 channel to generate two sub-feature maps while retaining the channel dimension information; Specifically, first, based on the scale of 2, Python's slicing operation is applied to the feature map of size (S×S×C2). From the (S×S) plane, two sub-feature maps are generated by selecting odd rows and even columns and even rows and odd columns for slicing: f 1,0 and f 0,1 ; The calculation is as follows: f 0,1 =X[0:S:scale,1:S:scale] f 1,0 =X[1:S:scale,0:S:scale] S is the step size of the slicing operation, scale is the length and width of the plane of the feature map except the channel dimension, 0:S:scale starts from 0, scale ends, and the elements in it are taken out; 3) Concatenate these two sub-feature maps along the channel dimension to obtain a downsampled feature map of shape S2×S2×2c2, where S2=S / 2, and pass it to the next convolution layer. The number of input channels for the next convolution operation should be twice as many as the original one. Step 5: Use the trained neural network to process the video stream from the coal mine site in real time to perform target detection; Step 6. Based on the target detection results, enter the three violations judgment stage, determine whether the local fan helmet is compliant based on quantity matching, and determine the operating status of the vehicle, people and belts based on the driving person and the neural network.

2. A novel intelligent video recognition method for typical three violations in smart coal mines as claimed in claim 1, characterized in that: In the step 1, the preprocessing includes decoding of the video stream, preliminary correction and quality inspection of the image.

3. The intelligent video recognition method for typical three violations in a new smart coal mine as claimed in claim 1 is characterized in that: In the step 2, the image enhancement module processes the image generated by frame extraction by using: random combination of brightness and contrast adjustment, color correction, sharpening and noise reduction, and adaptive filtering for specific light sources.

4. The intelligent video recognition method for typical three violations in a new smart coal mine as claimed in claim 1 is characterized in that: In step 5, target detection refers to quickly identifying multiple targets in the video, including but not limited to workers, equipment, and vehicles, while providing accurate location information of the targets.

5. The intelligent video recognition method for typical three violations in a new smart coal mine as claimed in claim 1 is characterized in that: In step 6, a background difference method is used to monitor moving targets under a fixed background, and the running state of the belt conveyor is determined by detecting moving targets within the belt conveyor range, and whether an abnormal situation occurs is determined in combination with the three-violation judgment rule; The Botsort algorithm is used to focus on the movement trajectory of the target in continuous video frames. By analyzing the target's moving direction, speed and position changes, potential violations are identified: workers entering dangerous areas without wearing safety helmets and improper equipment operation.

Images