Underground coal mine unsafe behavior identification method

Abstract

The invention discloses a coal mine underground unsafe behavior identification method. The method comprises the following steps: acquiring coal mine underground real-time video stream data; extracting local features and global features from frame-by-frame continuous images of the real-time video stream data; performing fusion processing on the local features and the global features to generate a fusion feature map; analyzing motion space-time features in the fusion feature map to obtain a recognition result; according to the method, the local features and the global features are extracted, and then the local features and the global features are fused to obtain the fused feature map, so that redundant information existing in continuously stacked videos is reduced, and the accuracy of image recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method for identifying unsafe behaviors in underground coal mines. Background Art

[0002] Some of the safety accidents that occur in underground coal mines are caused by unsafe behaviors of absentees. The types of unsafe behaviors of absentees include: illegal operations: such as not wearing a safety helmet as required, using tools illegally, etc. Illegal movement: such as running in a dangerous area, crossing a cordon, etc. Illegal operations: such as smoking in a non-smoking area, operating mechanical equipment illegally, etc. For visual data with a high number of connected image frames, there is a large amount of redundant information in the continuously stacked videos, which makes many methods based only on local reasoning unable to effectively capture the correlation between behaviors that last for several seconds, thereby limiting their ability to understand and analyze complex behaviors. Summary of the invention

[0003] In order to overcome the shortcomings of existing technologies, there is a large amount of redundant information in continuously stacked videos, which makes many methods based solely on local reasoning unable to effectively capture the correlation between behaviors lasting several seconds, thereby limiting their ability to understand and analyze complex behaviors.

[0004] First aspect:

[0005] The present invention provides a method for identifying unsafe behaviors in underground coal mines, which is characterized by comprising:

[0006] Obtain real-time video stream data from underground coal mines, and obtain real-time video stream data through the video monitoring system underground in coal mines;

[0007] Extract local features and global features from continuous frame-by-frame images of real-time video stream data;

[0008] Fusing local features with global features to generate a fused feature map;

[0009] The recognition result is obtained by analyzing the spatiotemporal features of motion in the fused feature map.

[0010] Optional, including:

[0011] A target detection algorithm based on deep learning is used to process continuous images frame by frame, and the boundary of the target area is determined according to a preset human body detection threshold to generate a target area image. For the target area image, global features and local features in the target area image are extracted.

[0012] Optional, including:

[0013] A feature pyramid model is constructed, local features and global features are imported into the feature pyramid model, multi-scale feature data is generated, and the multi-scale feature data is fused to generate a fused feature map.

[0014] Optionally, local features and global features are extracted from continuous images frame by frame through a feature extraction module, and convolution and attention modules are introduced into the feature extraction module.

[0015] Optionally, a lightweight convolutional network is used to perform dimensionality reduction processing on the fused feature map to generate a reduced-dimensionality feature map, and the fused feature map is processed through a three-layer convolution operation to generate a reduced-dimensionality feature map.

[0016] Optionally, a three-layer cascade behavior classifier is constructed based on the dimensionality reduction feature map, wherein the first layer determines the static or moving state; the second layer determines the specific action type; and the third layer determines whether the action complies with safety regulations.

[0017] Optionally, the risk assessment of the detected unsafe behavior is performed based on the behavior recognition result and a preset safety rule library to calculate a risk score, including:

[0018] The risk score is calculated using three quantitative indicators: degree of violation, duration, and distance to the danger zone.

[0019] Second aspect:

[0020] A system for identifying unsafe behaviors in underground coal mines is provided, comprising:

[0021] The first module is used to obtain real-time video stream data from underground coal mines;

[0022] The second module is used to extract local features and global features;

[0023] The third module is used to generate a fusion feature map;

[0024] The fourth module is used to obtain the recognition result.

[0025] The third aspect:

[0026] A computer device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the method described in any one of claims 1 to 7 when executing the program stored in the memory.

[0027] The fourth aspect

[0028] A computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

[0029] The beneficial effect of the present invention is that redundant information existing in continuously stacked videos is reduced by extracting local features and global features and then fusing the local features with the global features to obtain a fused feature map, thereby improving the accuracy of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0031] Figure 1 is a flow chart of an embodiment;

[0032] Figure 2 It is the YOLOv8 network model flow chart;

[0033] Figure 3 This is the overall structure diagram of the network model of the YOLOv8 model;

[0034] Figure 4 It is the flow chart of convolution and attention fusion module;

[0035] Figure 5 It is a schematic diagram of the module structure of Efficient RepGFPN;

[0036] Figure 6 This is a diagram of the overall architecture of 2s-AGCN. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0038] The present invention provides an underground unsafe behavior identification method, which can be used in coal mine operations to determine whether workers in coal mine operations have unsafe behaviors.

[0039] like Figure 2The method for identifying unsafe behaviors in underground coal mines in this embodiment may specifically include:

[0040] Step S1, obtaining real-time video stream data of underground coal mine;

[0041] Step S2, extracting local features and global features from the frame-by-frame continuous images of the real-time video stream data;

[0042] Step S3, fusing the local features with the global features to generate a fused feature map;

[0043] Step S4, obtaining a recognition result based on analyzing the motion spatiotemporal features in the fused feature map.

[0044] By extracting local features and global features, and then fusing the local features with the global features to obtain a fused feature map, the redundant information in the continuously stacked videos can be reduced, thereby improving the accuracy of image recognition.

[0045] In the present invention, the above method is based on the unsafe behavior recognition completed by the improved YOLOv8 model, such as Figure 2 , 3 As shown, the identification of unsafe behaviors based on the improved YOLOv8 model includes the following modules: feature extraction module, feature fusion module, and behavior recognition module.

[0046] Step S2 specifically includes the following steps: using a deep learning-based target detection algorithm to process continuous images frame by frame, determining the target area boundary according to a preset human detection threshold, generating a target area image, and extracting global features and local features in the target area image.

[0047] The real-time video stream data is obtained through the video monitoring system in the coal mine. The video stream contains the activity information of the underground workers. The YOLOv8 target detection algorithm based on deep learning is used to process the video stream frame by frame. The algorithm can quickly and accurately detect the human target area in the video frame. After the backbone part of the YOLOv8 network is connected, Figure 4 The convolution and attention fusion module (CAFM) shown in Figure 2 enhances the network's ability to extract local and global features of the human body. Figure 5The Efficient RepGFPN module shown replaces the Neck part of YOLOv8, flexibly adjusts the number of channels under different scale features, balances the expressiveness of high- and low-level features under lightweight computational constraints, and improves detection accuracy and speed. According to the preset human detection confidence threshold, it is determined whether the target bounding box output by YOLOv8 contains a human target. The higher the threshold, the higher the accuracy of the detection result, but the recall rate may decrease. If the confidence score of the target bounding box is higher than the threshold, the bounding box is determined as the human target area, and the corresponding image area data is extracted. Figure 6 The two-stream adaptive graph convolutional network (2s-AGCN) shown in the figure performs action recognition on the extracted human target image, and analyzes the spatiotemporal characteristics of the key points of the human skeleton through convolution and graph attention mechanisms, including joint point branches, skeleton edge branches, and feature fusion. The B stream is the skeleton edge branch, and the J stream is the joint point branch. The 2s-AGCN outputs the classification label of the human action and determines whether the current human action belongs to an unsafe behavior based on the predefined unsafe behavior category. If it is identified as an unsafe behavior, an early warning prompt is issued in time, and the relevant information is reported to the safety supervision system. At the same time, the image area data of the unsafe behavior is saved and archived for subsequent analysis.

[0048] For example, a high-definition video surveillance system in a coal mine is used to obtain real-time video stream data at 30 frames per second, with a video resolution of 1920x1080. The optimized YOLOv8 algorithm is used to process the video frame by frame. A CAFM module consisting of 3x3 convolution and 8-head self-attention mechanism is connected after the backbone part, and the Efficient RepGFPN module is replaced in the Neck part. The channels are flexibly adjusted between [64, 128, 256] according to the input size. The confidence threshold of the human detection frame is set to 8, and the human image is extracted from the detection frame area that exceeds the threshold. The two-stream adaptive graph convolution network 2s-AGCN is input, and the spatial stream uses 8 layers of ST-GCN and the temporal stream uses 6 layers of adaptive graph convolution. The classification accuracy is 94% after training on the NTU-RGBD dataset. According to the predefined 8 types of unsafe behaviors, 2s-AGCN outputs classification labels. If it is an unsafe class, an early warning is issued and reported to the safety supervision system, and the image data of the area is archived and saved by timestamp for subsequent analysis.

[0049] Get the target area image and input it into the improved YOLOv8 network. Perform preliminary feature extraction based on the Backbone part of the YOLOv8 model. If the preliminary feature data contains global features and local features, input the preliminary feature data into the Convolution and Attention Fusion Module (CAFM). Use convolution operation to extract local features, and use the self-attention mechanism to extract global features. Fuse local features and global features to generate fused feature data. Input the fused feature data into the Efficient RepGFPN module. Use different numbers of channels under different scale features to control the expressive power of high-level features and low-level features, and generate multi-scale feature data. Input the multi-scale feature data into the two-stream adaptive graph convolutional network (2s-AGCN). Analyze the spatial structure and dynamic changes of the key points of the human skeleton in the time series to judge unsafe behaviors.

[0050] Exemplarily, first, an image of the target area, such as a 640x480 pixel construction site monitoring video frame, is input into the improved YOLOv8 network. The Backbone part of the network, such as an improved version of CSPDarknet53, includes multiple convolutional layers and residual connections for extracting preliminary features of the image. Assume that the feature map size output by Backbone is 16x12x256. Since the convolution operation can capture local features and the residual connection can retain global information, it can be considered that the preliminary feature data contains global features and local features. Next, the 16x12x256 preliminary feature data is input into the Convolution and Attention Fusion Module (CAFM). The CAFM module first uses a convolutional layer with a 3x3 convolution kernel, a stride of 1, and a padding of 1 to extract local features, and the output feature map size is kept at 16x12x256. Then, a spatial self-attention mechanism is used to calculate the relationship weight between each position on the feature map and all other positions, thereby extracting global features. Specifically, the feature map is converted into 16*12 vectors of length 256, the dot product between these vectors is calculated as the attention matrix, and then the attention matrix is ​​multiplied with the original feature map to obtain the global feature, and the dimension is still 16x12x256. Finally, the local features and global features are fused by element-wise addition to generate fused feature data, and the dimension is still 16x12x256. The 16x12x256 fused feature data is input into the Efficient RepGFPN module. This module adopts a structure similar to PANet, including top-down and bottom-up paths, as well as feature maps of different scales. Assume that Efficient RepGFPN outputs feature maps of three scales, namely 8x6x128, 16x12x256, and 32x24x512. By using convolutional layers with different numbers of channels on feature maps of different scales, the expressive power of high-level features and low-level features can be controlled, for example, using convolutional layers with fewer channels on feature maps of 8x6x128, and using convolutional layers with more channels on feature maps of 32x24x512. Multi-scale feature data is finally generated. Finally, the multi-scale feature data, such as feature maps of 8x6x128, 16x12x256, and 32x24x512, are input into a two-stream adaptive graph convolutional network (2s-AGCN). The network contains two branches, which respectively deal with the spatial structure and dynamic changes of human skeleton key points. The spatial structure branch uses a graph convolutional network (GCN) to model the spatial relationship of human skeleton key points, for example, using 17 nodes to represent 17 key points of the human body, and defining an adjacency matrix based on the human body structure. The dynamic change branch uses a temporal convolutional network (TCN) to model the changes of human skeleton key points in time series, for example, using a one-dimensional convolution kernel to convolve the coordinates of key points of consecutive frames.The outputs of the two branches are fused through an adaptive fusion mechanism to ultimately determine unsafe behavior. For example, if a worker is detected to be in a bent-over state for a long time or making a sudden and rapid movement, it is determined to be an unsafe behavior and an alarm is triggered.

[0051] Step S103, using an adaptive fusion module to process the multi-scale feature data, setting the convolution kernel sizes to three times three, five times five, and seven times seven, respectively, corresponding to feature data of different scales, to achieve multi-scale feature fusion and generate a fused feature map.

[0052] Obtain multi-scale feature data. Based on the YOLOv8 model, the convolution and attention fusion module (CAFM) in the HCANet network is placed after the backbone and before the detect to enhance the feature extraction capabilities of local and global features. The Efficient RepGFPN module in DAMO-YOLO is used to replace the Neck part in the yolov8 model. Different numbers of channels are used under different scale features to control the expression capabilities of high-level features and low-level features. If the feature data comes from the output of the EfficientRepGFPN module, feature maps of different scales from the backbone output are extracted. Convolution operations are performed on multi-scale feature data according to different convolution kernel sizes. The convolution kernel sizes are three by three, five by five, and seven by seven, corresponding to the different scale feature data output by the Efficient RepGFPN. The convolution and attention fusion module is used to process the convolution results of different convolution kernel sizes to capture long-range correlation and neighborhood spectral correlation. If the data dimensions processed by the convolution and attention fusion modules are inconsistent, the data dimensions are adjusted to keep them consistent. The processed multi-scale feature data is fused through the adaptive fusion module to adaptively adjust the weights of features at different scales. If the adaptive fusion module includes a channel attention mechanism, the channel weights are dynamically allocated according to the importance of features at different scales to highlight important features. The fused feature map is obtained and input into the subsequent detection network to realize the analysis and identification of unsafe behaviors in coal mines.

[0053] For example, assume that Efficient RepGFPN outputs feature maps of three different scales, with sizes of 80x80x128, 40x40x256, and 20x20x512. First, perform convolution operations on the feature maps of these three scales respectively. For the feature map of 80x80x128, a 3x3 convolution kernel is used, with a step size of 1 and a padding of 1. The size of the feature map after convolution remains unchanged, still 80x80x128. For the feature map of 40x40x256, a 5x5 convolution kernel is used, with a step size of 1 and a padding of 2. The size of the feature map after convolution remains unchanged, still 40x40x256. For the feature map of 20x20x512, a 7x7 convolution kernel is used, with a step size of 1 and a padding of 3. The size of the feature map after convolution remains unchanged, still 20x20x512. The convolution operation can be implemented using the Conv2d function in PyTorch, for example: conv1=torch.nn. Conv2d(128,128,kernel_size=3,stride=1,padding=1). Next, the convolutional feature map is processed using the Convolution and Attention Fusion Module (CAFM). Assume that CAFM contains a 3x3 convolution layer and a channel attention mechanism. The channel attention mechanism can use the SE module to obtain the global information of each channel through global average pooling, and then pass through two fully connected layers and the Sigmoid function to generate channel weights. The channel weights are multiplied by the feature map to obtain the attention-weighted feature map. Since CAFM does not change the size of the feature map, the sizes of the three feature maps after processing are still 80x80x128, 40x40x256, and 20x20x512. Then, the three feature maps are fused using the adaptive fusion module. In order to keep the dimensions consistent, upsampling is first used to upsample the 40x40x256 and 20x20x512 feature maps to 80x80x256 and 80x80x512, respectively. Upsampling can be implemented using bilinear interpolation or transposed convolution. For example: upsample = torch. nn. Upsample(scale_factor = 2, mode = 'bilinear'). Assume that the adaptive fusion module includes a channel attention mechanism. First, concatenate the three feature maps together to obtain a 80x80x896 feature map. Then, use global average pooling to obtain the global information of each channel, and generate channel weights after two fully connected layers and a Softmax function. Assume that the generated weight vector is [2,3,5], corresponding to the three feature maps respectively. Multiply the weight vector with the concatenated feature map to achieve adaptive weighted fusion. Finally, the fused 80x80x896 feature map is input into the subsequent detection network. The detection network can be the Head part of YOLOv8, which is used to analyze and identify unsafe behaviors in coal mines, such as not wearing a safety helmet, illegal operations, etc.Assume that the output of the detection network is the class probability and bounding box coordinates of each grid. By post-processing the output results, such as non-maximum suppression, the final detection results, including the target category and location, can be obtained. For example, a person without a helmet is detected with a confidence of 9 and a bounding box coordinate of (100, 100, 200, 300).

[0054] According to the fusion feature map, the heat map regression skeleton key point positioning algorithm is used to locate the coordinates of 17 key parts of the human body's trunk and limbs, and a skeleton key point coordinate set is generated. The joint angle and posture parameters are calculated based on the skeleton key point coordinate set, and the risk level is determined according to the preset value to generate three-level alarm information: high, medium and low.

[0055] Specifically, based on the extracted features, the heat map regression algorithm is used to locate the 17 key parts of the human body trunk and limbs. The heat map regression algorithm generates a heat map by predicting the probability distribution of the position of each key part in the image. Each pixel in the heat map represents the probability of a key part. Through mathematical methods such as Gaussian function, the coordinates of the key points are converted into high response value areas in the heat map. In this way, each key point has an obvious response peak in the heat map, which is convenient for subsequent positioning and extraction. According to the results of the heat map regression, the maximum probability position of each key part is extracted as the coordinate of the part. The coordinates of all key parts are combined to form a skeleton key point coordinate set. This coordinate set contains the complete posture information of the human body in the image. Using the skeleton key point coordinate set, the angles of each joint and the posture parameters such as the length and direction of the limbs are calculated through geometric calculations or machine learning algorithms. These parameters can more accurately describe the posture and movement of the human body and provide an important basis for the subsequent risk level determination. According to the preset risk determination criteria (such as the threshold of the joint angle, the normal range of the limb length, etc.), the calculated posture parameters are risk assessed. The results of risk assessment divide the risk level into three levels: high, medium and low. High risk means that the worker's actions may lead to serious safety accidents; medium risk means that the worker's actions have potential safety hazards; low risk means that the worker's actions are relatively safe, but still need attention. According to the risk level, the corresponding alarm information is generated. The alarm information can include risk level, risk description, recommended safety measures, etc., and real-time feedback is provided through the monitoring system or the terminal device worn by the worker. A lightweight convolutional network is used to reduce the dimension of the fused feature map. Through three-layer convolution operations, the number of convolution kernels is sixty-four, thirty-two, and sixteen respectively, to generate a reduced dimension feature map.

[0056] Based on the improved YOLOv8 network, the fused feature map is obtained. The network structure integrates the convolution and attention fusion module (CAFM) in the HCANet network and the Efficient RepGFPN module in DAMO-YOLO. The fused feature map is input to the first layer of lightweight convolutional network. The number of convolution kernels of the convolutional network is sixty-four. If the dimension of the feature map output by the first layer of convolutional network is greater than the preset threshold, the feature map is pooled to reduce the dimension. The output of the first layer of convolutional network is used as the input of the second layer of convolutional network. The number of convolution kernels of the second layer of convolutional network is thirty-two. If the dimension of the feature map output by the second layer of convolutional network is greater than the preset threshold, the feature map is pooled to reduce the dimension. The output of the second layer of convolutional network is used as the input of the third layer of convolutional network. The number of convolution kernels of the third layer of convolutional network is sixteen. If the dimension of the feature map output by the third layer of convolutional network is greater than the preset threshold, the feature map is pooled to reduce the dimension. The third layer of convolutional network outputs the feature map after dimensionality reduction. The reduced-dimensional feature map is input into a two-stream adaptive graph convolutional network (2s-AGCN) to analyze the key points of the human skeleton and identify unsafe behaviors.

[0057] For example, the improved YOLOv8 network fuses the CAFM of HCANet and the EfficientRepGFPN of DAMO-YOLO to obtain a fused feature map. Assume that the sizes of the three feature maps output by YOLOv8 are 80x80x256, 40x40x512, and 20x20x1024 respectively. The CAFM module first performs 1x1 convolution on the three feature maps to adjust the number of channels to 256, and then upsamples the 80x80 and 40x40 feature maps to 80x80, concatenates them with the 80x80 feature map, and obtains a feature map of 80x80x768 after the fusion of the channel attention mechanism and the spatial attention mechanism. The feature map and the 20x20x1024 feature map are then fused through the Efficient RepGFPN module. Efficient RepGFPN adopts the idea of ​​multi-scale feature fusion and re-parameterized convolution, and the final output fused feature map size is 80x80x512. The fused feature map is input to the first layer of lightweight convolutional network, with sixty-four convolution kernels, using 3x3 convolution, step size of 1, padding of 1, and obtaining a feature map of 80x80x64. The preset threshold is 10000. Since 80*80*64=409600>10000, the maximum pooling layer is used, with a pooling kernel size of 2x2 and a step size of 2, to reduce the feature map to 40x40x64. The output of the first layer of convolutional network is used as the input of the second layer of convolutional network. The number of convolutional kernels in the second layer of convolutional network is thirty-two, using 3x3 convolution, step size of 1, padding of 1, and obtaining a feature map of 40x40x32. Since 40*40*32=51200>10000, continue to use 2x2 maximum pooling with a step size of 2 to reduce the feature map to 20x20x32. The output of the second layer of convolutional network is used as the input of the third layer of convolutional network. The number of convolution kernels in the third layer of the convolutional network is sixteen. 3x3 convolution is used with a step size of 1 and a padding of 1 to obtain a feature map of 20x20x16. Since 20*20*16=6400>10000, 2x2 maximum pooling is used again with a step size of 2 to reduce the dimension of the feature map to 10x10x16. Finally, a reduced-dimensional feature map of 10x10x16 is obtained. The feature map is input into the 2s-AGCN for key point analysis of the human skeleton. Assuming that the human skeleton has 17 key points, a graph of 17 nodes is constructed. The 2s-AGCN network contains two streams, which process spatial information and temporal information respectively. The spatial stream uses a graph convolutional network (GCN) to extract features from the skeleton graph. For example, two layers of GCN are used, and the feature dimension of each node is 64, and the node features are updated. The temporal stream uses a temporal convolutional network (TCN) to model the time series information of key points, for example, using two layers of TCN, with a convolution kernel size of 3 and a number of channels of 64. The outputs of the two streams are fused, for example, concatenated and passed through a fully connected layer to output the coordinates of each key point.By analyzing the coordinate changes of key points, such as calculating joint angles, unsafe behaviors such as falls can be identified: angle thresholds for the hip, knee, and ankle joints are set, and if these angles are less than the preset thresholds, it is judged as a fall.

[0058] Step S105, construct a three-layer cascade behavior classifier based on the dimensionality reduction feature map, the first layer determines the static or moving state, the second layer determines the specific action type, and the third layer determines whether the action meets the safety specifications to generate a behavior recognition result.

[0059] The improved YOLOv8 network is used to extract the skeleton key point information of miners in the underground coal mine monitoring video, and the CAFM module in the HCANet network is combined to enhance the extraction ability of local and global features, so as to obtain more comprehensive miner behavior features. The extracted features are multi-scale fused by the Efficient RepGFPN module in DAMO-YOLO, and the number of channels is flexibly controlled under different scale features to obtain the feature map after dimensionality reduction. According to the change amplitude of the key points of the human posture in the dimensionality reduction feature map, it is judged whether the miner is in a static state or a moving state. If the miner is in a static state, the behavior recognition result is directly classified as a specific static behavior, such as standing, squatting, etc. If the miner is in a moving state, the two-stream adaptive graph convolutional network (2s-AGCN) is used to analyze the time series information of the skeleton key points to determine the specific action type, such as walking, bending, operating equipment, etc. The feature vector of the identified action type is obtained and compared with the feature vector in the pre-established safety specification database. By calculating the similarity between the feature vectors, it is judged whether the current miner's behavior meets the safety specifications. Based on the output of 2s-AGCN and the comparison results of safety regulations, behavior recognition results including action type and safety compliance judgment are generated. The behavior recognition results are superimposed on the monitoring video in a visual way to monitor the miners' behavior in real time and detect unsafe behavior in time.

[0060] Exemplarily, first, the key point information of the miner skeleton is extracted using the improved YOLOv8 network. The network is pre-trained on the COCO dataset, and then fine-tuned using 1,000 collected images of miners annotated with 15 key points. The input resolution is set to 640x640. Post-processing is performed with a confidence threshold of 0.5 and an intersection-over-union threshold of 0.6, and the coordinates (x, y) of the 15 key points of each miner are output. For example, the coordinates of the left shoulder key point of a miner are (234, 156). Then, the extracted key point information is input into the CAFM module in the HCANet network. The CAFM module consists of multiple convolutional layers and an attention mechanism to extract local and global features. The convolutional layer uses a 3x3 convolution kernel with a step size of 1 and a padding of 1. The attention mechanism uses a channel attention mechanism to enhance the feature expression capability by learning the weight relationship between different channels. For example, the output feature dimension of the CAFM module is 256. Then, the output features of the CAFM module are input into the Efficient RepGFPN module of DAMO-YOLO for multi-scale fusion. This module uses a learnable channel scaling factor to flexibly control the number of channels under different scale features. For example, the number of feature channels of the three scales P3, P4, and P5 is controlled to 128, 256, and 512 respectively, and these multi-scale features are weighted fused to finally obtain a 128-dimensional feature map after dimensionality reduction. The state of the miner is judged according to the change amplitude of the key points of the human posture in the dimensionality reduction feature map. The average value of the coordinate change amplitude of 15 key points in two consecutive frames of images is calculated. If the value is less than the preset threshold of 5, the miner is judged to be in a stationary state. For example, if the calculated change amplitude is 3, it is judged to be in a stationary state, and its behavior recognition result is classified as "standing". Otherwise, the miner is judged to be in a moving state. If the miner is in a moving state, the skeleton key point information of 24 consecutive frames (corresponding to 0.8 seconds, assuming the video frame rate is 30fps) is input into the 2s-AGCN network. The 2s-AGCN network contains two streams, which process the spatial information and temporal information of the skeleton key points respectively. Each stream contains three graph convolution layers, each of which is followed by a ReLU activation function and a batch normalization layer. The network outputs the probability distribution of the miner's action type. For example, if the output result is: walking: 0.8, bending: 0.1, operating equipment: 0.1, the miner's behavior is judged to be "walking". The feature vector corresponding to "walking" (for example, a 128-dimensional vector) is obtained and compared with the feature vector in the pre-established safety specification database (for example, a standard feature vector containing actions such as "walking", "bending", and "operating equipment"). The cosine similarity is used to calculate the similarity between feature vectors. For example, the similarity between the feature vector of "walking" and the feature vector of "safe walking" in the safety specification database is 0.9, which is higher than the preset threshold of 0.75, then the current miner's behavior is judged to meet the safety specifications.Finally, the behavior recognition results (e.g., “walking-safe”) are superimposed on the surveillance video to monitor the miners’ behavior in real time.

[0061] Step S106, based on the behavior recognition results and the preset safety rule library, a risk assessment is performed on the detected unsafe behavior, and a risk score is calculated using three quantitative indicators: degree of violation, duration, and distance to the danger zone.

[0062] The improved YOLOv8 network performs real-time analysis on underground coal mine surveillance videos to obtain the sequence data of the key points of the miner's skeleton and environmental information. The two-stream adaptive graph convolutional network (2s-AGCN) analyzes the spatial structure and dynamic changes of the key points of the human skeleton in the time series to identify the miner's behavior. If the recognition result matches the definition of the unsafe behavior in the preset safety rule library, the feature information of the unsafe behavior is extracted. According to the extracted feature information, combined with the environmental information obtained by the improved YOLOv8 network, the specific location and time of the unsafe behavior are determined. By querying the safety rule library, the violation degree baseline value corresponding to the unsafe behavior is obtained. According to the duration of the unsafe behavior, the violation degree weighted value is calculated, and the weighted value is proportional to the duration. The improved YOLOv8 network is used to obtain the distance information between the miner and the dangerous area, and the distance weighted value is calculated, and the weighted value is inversely proportional to the distance. The risk score is calculated by combining the violation degree baseline value, violation degree weighted value and distance weighted value. The risk score result is stored in the database, and the alarm system is triggered according to the preset threshold.

[0063] Exemplarily, the improved YOLOv8 network can detect miners and environmental information in real time by pre-training on the COCO dataset and then fine-tuning using mine monitoring video data. For example, the coordinates (x, y) of 17 skeleton key points of the miner are detected, such as the head coordinates (120, 85), the left wrist coordinates (55, 150), and so on. At the same time, the YOLOv8 network identifies equipment in the mine, such as a conveyor belt, and locates its position coordinates (300, 200). The skeleton key point data of consecutive frames are input into the 2s-AGCN network. The 2s-AGCN network contains a spatial stream and a temporal stream. The spatial stream uses graph convolution operations, such as representing the skeleton topology structure with an adjacency matrix, aggregating the features of each key point, and learning the spatial relationship between key points. The temporal stream uses, for example, an LSTM network to model the key point features on the time series and learn the dynamic changes of the action. Assume that the 2s-AGCN recognizes that the current behavior of the miner is "bending over to pick up items." This behavior matches the unsafe behavior definition of "bending over to get close to the conveyor belt" in the safety rule base. The safety rule base defines the violation level baseline value of this behavior as 5. Assume that the miner's "bending over to get close to the conveyor belt" behavior lasts for 5 seconds. The calculation formula for the violation level weighted value is: weighted value = duration * 2. Therefore, the violation level weighted value is 5 * 2 = 1. The YOLOv8 network detects that the distance between the miner and the conveyor belt is 1 meter. The calculation formula for the distance weighted value is: weighted value = 1 / distance. Therefore, the distance weighted value is 1 / 1 = 1. The calculation formula for the risk score is: risk score = baseline value + violation level weighted value + distance weighted value. Therefore, the risk score of the miner is 5 + 1 + 1 = 7. The risk score result is stored in the database. Since the risk score 7 exceeds the preset threshold 6, the system triggers an alarm to remind the miner to pay attention to safety. At the same time, the system records the time of occurrence of the event, such as 10:00:00 on January 1, 2024, and the location of occurrence, such as the coordinates near the conveyor belt (305, 210).

[0064] The invention discloses an unsafe behavior recognition system for underground coal mines, comprising: a first module for acquiring real-time video stream data of underground coal mines; a second module for extracting local features and global features; a third module for generating a fusion feature map; and a fourth module for obtaining recognition results.

[0065] The present invention discloses a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement the steps of the method described in the above embodiment when executing the program stored in the memory.

[0066] The present invention discloses a computer-readable storage medium on which a computer program is stored, characterized in that the steps of the method described are implemented when the computer program is executed by a processor.

[0067] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for identifying unsafe behaviors in underground coal mines, characterized in that: include: Obtain real-time video stream data from underground coal mines; Extract local features and global features from continuous frame-by-frame images of real-time video stream data; Fusing local features with global features to generate a fused feature map; The recognition result is obtained by analyzing the spatiotemporal features of motion in the fused feature map.

2. The method for identifying unsafe behaviors in underground coal mines according to claim 1, characterized in that: include: A target detection algorithm based on deep learning is used to process continuous images frame by frame, and the boundary of the target area is determined according to the preset human detection threshold to generate a target area image. For the target area image, the global features and local features in the target area image are extracted.

3. The method for identifying unsafe behaviors in underground coal mines according to claim 1, characterized in that: include: The local features and global features are fused to generate fused feature data, the fused feature data is imported into the feature pyramid network to generate multi-scale feature data, and the multi-scale feature data is processed using an adaptive fusion module to generate a fused feature map.

4. The method for identifying unsafe behaviors in underground coal mines according to claim 1, characterized in that: According to the fusion feature map, the heat map regression skeleton key point positioning algorithm is used to locate the coordinates of 17 key parts of the human trunk and limbs, and a skeleton key point coordinate set is generated. The joint angles and posture parameters are calculated based on the skeleton key point coordinate set, and the risk level is determined according to the preset value to generate three levels of alarm information: high, medium and low.

5. The method for identifying unsafe behaviors in underground coal mines according to claim 1, characterized in that: A lightweight convolutional network is used to reduce the dimension of the fused feature map to generate a reduced-dimensional feature map. The fused feature map is processed through three-layer convolution operations to generate a reduced-dimensional feature map.

6. The method for identifying unsafe behaviors in underground coal mines according to claim 5, characterized in that: A three-layer cascade behavior classifier is constructed based on the dimensionality reduction feature map, in which the first layer determines the static or moving state; the second layer determines the specific action type; and the third layer determines whether the action complies with safety regulations.

7. The method for identifying unsafe behaviors in underground coal mines according to claim 1, characterized in that: The method of performing risk assessment on the detected unsafe behavior based on the behavior recognition results and the preset safety rule base, and calculating the risk score, includes: The risk score is calculated using three quantitative indicators: degree of violation, duration, and distance to the danger zone.

8. A system for identifying unsafe behaviors in underground coal mines, characterized in that: include: The first module is used to obtain real-time video stream data from underground coal mines; The second module is used to extract local features and global features; The third module is used to generate a fusion feature map; The fourth module is used to obtain the recognition result.

9. A computer device, characterized in that: The invention comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images