A coal mine conveying belt area miner unsafe behavior identification method, device, equipment and medium

Abstract

This application discloses a method, apparatus, equipment, and medium for identifying unsafe behaviors of miners in coal mine conveyor belt areas, relating to the field of image recognition technology. The method includes acquiring video images of the coal mine conveyor belt area and extracting keyframes; inputting the keyframes into a pre-established and trained image super-resolution reconstruction model to obtain a high-resolution reconstructed image; inputting the high-resolution reconstructed image in parallel into a pre-established and trained unsafe behavior identification model and a conveyor belt area semantic segmentation model, outputting miner location bounding boxes, preliminary behavior category information, and pixel-level segmentation results; and finally, making a final judgment on the unsafe behaviors of miners. This application can effectively recover blurred details from the acquired images and achieve high recognition accuracy even with only a small amount of labeled data, realizing accurate judgment of behavioral safety attributes and improving the accuracy of unsafe behavior identification.

Description

Technical Field

[0001] This application relates to the field of image recognition, and in particular to a method, apparatus, equipment and medium for identifying unsafe behaviors of miners in coal mine conveyor belt areas. Background Technology

[0002] Coal is a vital basic energy source, and its safe production is of paramount importance. The conveyor belt area is the core link in underground material transportation in coal mines and is also a high-risk area for accidents. Statistics show that most accidents in this area are directly related to miners' unsafe practices such as violations of regulations. Traditional safety monitoring relies on manual review of video footage, which is inefficient, slow to respond, and unable to provide real-time early warnings.

[0003] With the development of artificial intelligence, deep learning-based target detection and behavior recognition technologies have made automated monitoring possible. However, the application of this technology in underground conveyor belt areas of coal mines faces three unique challenges: (1) Poor image quality. The underground environment is dark and dusty, and the collected images generally have problems such as low resolution, poor contrast, and blurred details, which seriously affect the model's accuracy in extracting features of miners and their behaviors. (2) Scarce labeled data. It is difficult to collect samples of miners' unsafe behaviors, and accurate labeling requires a lot of professional knowledge and manpower, resulting in a limited scale of labeled datasets that can be used to train deep learning models, insufficient model training, and weak generalization ability. (3) Behavior discrimination needs to be combined with the environment. The danger of many unsafe behaviors depends on the interaction between miners and specific environmental targets. For example, "leaning" itself is not necessarily dangerous, but "leaning against guardrails" is very likely to lead to a fall and is a high-risk unsafe behavior. Existing research focuses on identifying the miners' own postures and lacks detailed analysis of the interaction with the environment, resulting in insufficient discrimination reliability. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, equipment, and medium for identifying unsafe behaviors of miners in coal mine conveyor belt areas. This method can effectively recover blurred details from acquired images, providing a high-quality image foundation for subsequent identification tasks. It enables the model to achieve high recognition accuracy even with only a small amount of labeled data, greatly reducing the data labeling cost and cycle of system construction. It achieves accurate discrimination of behavioral safety attributes, solves the core pain point of lacking environmental context in simple pose recognition, and significantly reduces the false alarm rate and false negative rate.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a method for identifying unsafe behaviors of miners in coal mine conveyor belt areas, including:

[0007] Collect video images of the coal mine conveyor belt area and extract keyframes from them;

[0008] The keyframes are input into a pre-established and trained image super-resolution reconstruction model to obtain a high-resolution reconstructed image;

[0009] The high-resolution reconstructed image is input in parallel into a pre-established and trained model for identifying unsafe behaviors of miners and a semantic segmentation model for the conveyor belt region. The output of the unsafe behavior identification model includes at least the bounding box of the miner's location and preliminary behavior category information. The output of the semantic segmentation model for the conveyor belt region includes at least the pixel-level segmentation results of the miner, the guardrail, and the conveyor belt.

[0010] Based on the spatial interaction relationship between the preliminary behavior category information, the miner's location bounding box, and the pixel-level segmentation results, the miner's unsafe behavior is finally determined.

[0011] Optionally, the specific process of pre-establishing and training the image super-resolution reconstruction model includes:

[0012] Original images of the underground conveyor belt area in a coal mine are collected to construct a high-resolution image set, and the high-resolution images are downsampled to generate corresponding low-resolution images, thereby forming low-resolution-high-resolution image pairs as training samples.

[0013] A generative adversarial network (GAN) model is constructed, comprising a shallow feature extraction module, a multi-scale dense residual attention network, an image reconstruction module, and a discriminator. The shallow feature extraction module performs initial feature extraction on the input low-resolution image. Multi-scale feature extraction is performed using dense residual blocks with convolutional kernels of different sizes connected in parallel within the multi-scale dense residual attention network, and the fused features are adaptively weighted via a channel attention mechanism. The image reconstruction module upsamples the features output from the multi-scale dense residual attention network and reconstructs a high-resolution image through convolution. The discriminator determines whether the input image is a true high-resolution image or an image reconstructed by the generator.

[0014] Based on the low-resolution to high-resolution image pairs used as training samples, the generative adversarial network model is alternately optimized and trained using texture loss and adversarial loss as joint loss functions until the model converges.

[0015] Optionally, the specific process of performing multi-scale feature extraction through dense residual blocks with convolutional kernels of different sizes connected in parallel in a multi-scale dense residual attention network, and adaptively weighting the fused features via a channel attention mechanism, includes:

[0016] For the input feature map of the multi-scale dense residual channel attention module, it is simultaneously input into the first dense residual block and the second dense residual block in parallel.

[0017] Feature extraction is performed using a convolutional kernel of the first size from the first dense residual block, and a first feature map is output.

[0018] Feature extraction is performed using a second-size convolutional kernel that is different from the first size in the second dense residual block, and a second feature map is output.

[0019] The first feature map and the second feature map are concatenated along the channel dimension to obtain a fused feature map;

[0020] The fused feature map is input into the channel attention mechanism to generate a channel attention weight vector;

[0021] The channel attention weight vector is multiplied channel by channel with the fused feature map to output an adaptively weighted feature map.

[0022] Optionally, the specific process of pre-establishing and training the unsafe behavior identification model for miners includes:

[0023] A semi-supervised learning training architecture based on a teacher-student framework is constructed, in which the student model is the target detection network to be trained, and the teacher model has the same structure as the student model and its weights are updated from the student model through a dynamic exponential moving average.

[0024] An improved anchor-free object detection network is used as the basic structure of the student model. The improved anchor-free object detection network includes an efficient local attention pyramid structure.

[0025] Construct a feature alignment detection head, which includes a classification branch, a localization branch, and a feature alignment path that maps the features of the localization branch to the classification branch;

[0026] The unsafe behavior identification model for miners is trained using a mixed dataset containing a small amount of labeled data and a large amount of unlabeled data. The training process includes:

[0027] In the initial supervised training phase, the student model is pre-trained using only the labeled data;

[0028] In the semi-supervised mutual learning stage, the teacher model and student model are initialized with pre-trained weights, and the teacher model weights are updated with the dynamic exponential moving average strategy. The teacher model generates pseudo-labels for weakly enhanced unlabeled data, and the student model is jointly trained by combining the real labels of the labeled data with the supervision signals of the pseudo-labels.

[0029] The loss functions used in training include supervised loss based on real labels and consistency loss based on the relationship between the teacher model output and the student model output with strongly augmented data.

[0030] Optionally, the efficient local attention pyramid structure embeds an efficient local attention module before the upsampling path of the feature pyramid network. The efficient local attention module obtains feature vectors of the feature map in the horizontal and vertical directions of space through one-dimensional pooling operations, and generates spatial attention weights based on the feature vectors.

[0031] Optionally, the specific process of pre-establishing and training the semantic segmentation model for the conveyor belt region includes:

[0032] A semantic segmentation network based on an encoder-decoder structure is constructed. The encoder adopts a short-term dense cascaded network to extract multi-level features from the input image, and the decoder adopts a lightweight multi-level unified attention fusion decoding network.

[0033] After the encoder outputs high-level features, a feature attention pyramid pooling module is connected. This module captures multi-scale global context information through pooling operations at different scales and integrates a parameter-free average attention mechanism to enhance attention to detailed regions of the feature map.

[0034] The lightweight multi-level unified attention fusion decoding network extracts low-level spatial features from the encoder through convolutional layers, and adaptively fuses the upsampled high-level semantic features with the low-level spatial features through a unified attention fusion module. The unified attention fusion module generates fusion weights through a spatial attention mechanism.

[0035] The semantic segmentation model for the conveyor belt region is trained using a two-stage strategy based on transfer learning. The training process includes:

[0036] In the first stage, the encoder is pre-trained using a publicly available large-scale semantic segmentation dataset to obtain the pre-trained weights of the encoder part.

[0037] In the second stage, on the conveyor belt area image dataset, the encoder is initialized with pre-trained weights, and the entire segmentation network, including the feature attention pyramid pooling module and the lightweight multi-level unified attention fusion decoding network, is fine-tuned end-to-end.

[0038] Optionally, the specific process of making a final judgment on unsafe behaviors of miners based on the spatial interaction relationship between preliminary behavior category information, miner location bounding boxes, and pixel-level segmentation results includes:

[0039] When the initial behavior category is "falling", it is judged as an unsafe behavior;

[0040] When the initial behavior category is "leaning" or "climbing", perform the following steps:

[0041] Based on the miner's location bounding box, obtain the corresponding miner pixel region;

[0042] Calculate the spatial relationship measure between the miner pixel region and the guardrail pixel region and / or conveyor belt pixel region in the pixel-level segmentation result;

[0043] The spatial relationship measurement is judged based on a preset threshold. If the condition is met, it is judged as an unsafe behavior corresponding to "leaning against the guardrail", "climbing the guardrail" or "climbing the conveyor belt".

[0044] Secondly, this application provides a device for identifying unsafe behaviors of miners in coal mine conveyor belt areas, comprising:

[0045] The image acquisition and preprocessing module is used to acquire keyframes and call the image super-resolution reconstruction model for processing.

[0046] The behavior recognition module, which is built into the miner unsafe behavior recognition model, is used to output preliminary behavior recognition results;

[0047] An environment segmentation module, built into the semantic segmentation model of the conveyor belt region, is used to output the environmental target segmentation results;

[0048] The interactive analysis and discrimination module is used to discriminate unsafe behaviors based on the preliminary behavior recognition results and the environmental target segmentation results.

[0049] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the step of identifying unsafe behaviors of miners in the coal mine conveyor belt area as described in any of the above.

[0050] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of identifying unsafe behaviors of miners in the coal mine conveyor belt area as described above.

[0051] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0052] This application provides a method, apparatus, equipment, and medium for identifying unsafe behaviors of miners in coal mine conveyor belt areas. By inputting keyframes into a pre-established and trained image super-resolution reconstruction model, blurred details in the acquired images can be effectively recovered, providing a high-quality image foundation for subsequent recognition tasks. By inputting high-resolution reconstructed images in parallel into a pre-established and trained unsafe behavior identification model and a conveyor belt area semantic segmentation model, the model can achieve high recognition accuracy even with only a small amount of labeled data, greatly reducing the data labeling cost and cycle of system construction. Based on the spatial interaction relationship between preliminary behavior category information, miner location bounding boxes, and pixel-level segmentation results, the unsafe behaviors of miners are finally judged, achieving accurate judgment of behavioral safety attributes. This solves the core pain point of lacking environmental context in simple pose recognition, significantly reducing false alarm and false negative rates. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a method for identifying unsafe behaviors of miners in a coal mine conveyor belt area, provided as an embodiment of this application;

[0055] Figure 2 This is a schematic diagram illustrating the detailed process of pre-establishing and training an image super-resolution reconstruction model in one embodiment of this application.

[0056] Figure 3 This is a schematic diagram of the network structure of a generative adversarial network model constructed in one embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the process in one embodiment of the present application, which involves multi-scale feature extraction through dense residual blocks with convolutional kernels of different sizes connected in parallel in a multi-scale dense residual attention network, and adaptive weighting of the fused features through a channel attention mechanism.

[0058] Figure 5 This is a detailed flowchart illustrating the pre-establishment and training process of a miner unsafe behavior identification model in one embodiment of this application.

[0059] Figure 6 This is a schematic diagram of a semi-supervised learning training network structure based on a teacher-student framework constructed in one embodiment of this application;

[0060] Figure 7This is a detailed flowchart illustrating the pre-establishment and training of a semantic segmentation model for the conveyor belt region in one embodiment of this application.

[0061] Figure 8 This is a schematic diagram of a semantic segmentation network structure based on an encoder-decoder structure constructed in one embodiment of this application;

[0062] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] In one exemplary embodiment, such as Figure 1 As shown, a method for identifying unsafe behaviors of miners in coal mine conveyor belt areas is provided. This method is executed by a computer device and includes the following steps 101 to 104. Wherein:

[0066] Step 101: Acquire video images of the coal mine conveyor belt area and extract keyframes from them;

[0067] Step 102: Input the keyframe into the pre-established and trained image super-resolution reconstruction model to obtain a high-resolution reconstructed image;

[0068] Step 103: Input the high-resolution reconstructed image in parallel into the pre-established and trained miner unsafe behavior recognition model and conveyor belt region semantic segmentation model. The output of the miner unsafe behavior recognition model includes at least the miner's location bounding box and preliminary behavior category information. The output of the conveyor belt region semantic segmentation model includes at least the pixel-level segmentation results of the miner, guardrail and conveyor belt.

[0069] Step 104: Based on the spatial interaction relationship between the preliminary behavior category information, the miner's location bounding box, and the pixel-level segmentation results, make a final judgment on the miner's unsafe behavior.

[0070] By implementing steps 101 to 104 above, the image super-resolution reconstruction model can effectively restore the blurred details of the acquired images, providing a high-quality image foundation for subsequent recognition tasks. The unsafe behavior recognition model for miners and the semantic segmentation model for the conveyor belt region enable the model to achieve high recognition accuracy even with only a small amount of labeled data, greatly reducing the data labeling cost and cycle of system construction. Based on the spatial interaction relationship between the preliminary behavior category information, the bounding box of the miner's location, and the pixel-level segmentation results, the unsafe behavior of miners is finally judged, achieving accurate judgment of the safety attributes of the behavior. This solves the core pain point of the lack of environmental context in simple pose recognition, and significantly reduces the false alarm rate and false negative rate.

[0071] In one exemplary embodiment, such as Figure 2 As shown, the specific process of pre-establishing and training the image super-resolution reconstruction model includes the following steps 201 to 203:

[0072] Step 201: Collect original images of the underground conveyor belt area in the coal mine to construct a high-resolution image set, and perform downsampling processing on the high-resolution images to generate corresponding low-resolution images, thereby forming low-resolution-high-resolution image pairs as training samples.

[0073] Step 202: Construct a generative adversarial network (GAN) model. The GAN model includes a shallow feature extraction module, a multi-scale dense residual attention network, an image reconstruction module, and a discriminator. The shallow feature extraction module performs initial feature extraction on the input low-resolution image. The multi-scale feature extraction is performed using dense residual blocks with convolutional kernels of different sizes connected in parallel within the multi-scale dense residual attention network, and the fused features are adaptively weighted via a channel attention mechanism. The image reconstruction module upsamples the features output from the multi-scale dense residual attention network and reconstructs a high-resolution image through convolution. The discriminator determines whether the input image is a true high-resolution image or an image reconstructed by the generator.

[0074] Step 203: Based on the low-resolution-high-resolution image pairs of training samples, the generative adversarial network model is alternately optimized and trained using texture loss and adversarial loss as joint loss functions until the model converges.

[0075] In practice, raw surveillance videos were collected from fixed monitoring cameras in multiple underground conveyor belt areas of coal mines. Frame extraction or keyframe extraction based on motion detection was used to initially select relatively clear images containing targets such as miners, conveyor belts, and guardrails, forming an initial high-resolution image library (e.g., 600 raw images were collected). The raw images underwent standardized preprocessing. First, the shorter side of all images was uniformly scaled to 256 pixels while maintaining the aspect ratio. Then, the scaled images were center-cropped to obtain high-resolution (HR) images of 256×256 pixels. Finally, the pixel values ​​were normalized to the range [0,1]. To simulate the degradation process of underground images, each image in the HR set was downsampled by a factor of 4. Bicubic interpolation was used to downsample the 256×256 HR images to 64×64 pixels, obtaining the corresponding low-resolution (LR) images, forming one-to-one "LR-HR" image pairs as the basic data units for model training and testing. Six hundred pairs of images were randomly divided into a training set (480 pairs), a validation set (60 pairs), and a test set (60 pairs) in an 8:1:1 ratio. To improve the model's generalization ability, online data augmentation was performed on the image pairs in the training set. In each training iteration, the HR image and its corresponding LR image were randomly flipped horizontally and rotated randomly by 90 degrees, 180 degrees, and 270 degrees. After augmentation, the diversity of effective samples in the training set was significantly improved. To adapt to the model input and further increase the number of samples, during training, image patches of 88×88 pixels were randomly cropped from the augmented HR images, and corresponding image patches of 22×22 pixels were simultaneously cropped from the LR images. These image patches served as the actual input for model training.

[0076] Construct a generative adversarial network model, the model structure of which is as follows: Figure 3As shown, it includes a generator G and a discriminator D. The generator G consists of a shallow feature extraction module, a multi-scale dense residual attention network (MDRCAB), and an image reconstruction module. The shallow feature extraction module is composed of a convolutional layer with a kernel size of 9×9 and 64 output channels, followed by a PReLU activation function. This module extracts shallow features from the input 22×22×3 LR image patch, and the output feature map size is 22×22×64. The multi-scale dense residual attention network consists of five cascaded multi-scale dense residual channel attention modules. The workflow of each MDRCAB is as follows: a. Dual-path feature extraction: The input feature map is simultaneously fed into two parallel dense residual blocks. The first DRB uses a 3×3 depthwise separable convolution, and the second DRB uses a 5×5 depthwise separable convolution to extract... a. Features under different receptive fields; b. Feature fusion: The feature maps output by the two DRBs are concatenated along the channel dimension (Concat); c. Channel attention weighting: The concatenated feature map is fed into the efficient channel attention module. The ECA module first performs global average pooling on the feature map, then captures local cross-channel interaction information through a 1D convolution with adaptive kernel size, and then generates attention weights (between 0 and 1) for each channel through the Sigmoid function. Finally, the weights are multiplied with the original feature map channel by channel to highlight high-frequency detail channels that are important for reconstruction and suppress low-frequency background channels; d. Residual connection: The weighted feature map is added element-wise to the input of the MDRCAB module to form residual learning to stabilize the training of the deep network. The image reconstruction module uses a sub-pixel convolutional layer to upsample the deep feature map output by MDRCAN by 4 times, enlarging the spatial size from 22×22 to 88×88. Then, a convolutional layer with a kernel size of 9×9 maps the number of channels back to 3 to generate the final reconstructed HR image patch. The discriminator D uses a relatively simple CNN structure, consisting of 8 stacked convolutional blocks (each containing a convolutional layer, a batch normalization layer, and a LeakyReLU activation function). It downsamples progressively, flattens the feature map, and outputs a scalar representing the probability that the input image is a real HR image through a fully connected layer and a Sigmoid activation function.

[0077] In one exemplary embodiment, such as Figure 4 As shown, the specific process of multi-scale feature extraction through dense residual blocks with convolutional kernels of different sizes connected in parallel in a multi-scale dense residual attention network, and adaptive weighting of the fused features via a channel attention mechanism, includes the following steps 301 to 306:

[0078] Step 301: For the input feature map of the multi-scale dense residual channel attention module, input it simultaneously into the first dense residual block and the second dense residual block in parallel.

[0079] Step 302: Extract features using a first-size convolutional kernel from the first dense residual block and output a first feature map;

[0080] Step 303: Extract features using a second-size convolutional kernel of a different size than the first size from the second dense residual block, and output a second feature map;

[0081] Step 304: Concatenate the first feature map and the second feature map along the channel dimension to obtain a fused feature map;

[0082] Step 305: Input the fused feature map into the channel attention mechanism to generate a channel attention weight vector;

[0083] Step 306: Multiply the channel attention weight vector with the fused feature map channel by channel to output an adaptively weighted feature map.

[0084] In one exemplary embodiment, such as Figure 5 As shown, the specific process of pre-establishing and training the unsafe behavior identification model for miners includes the following steps 401 to 404:

[0085] Step 401: Construct a semi-supervised learning training architecture based on a teacher-student framework, where the student model is the target detection network to be trained, and the teacher model has the same structure as the student model and its weights are updated from the student model through a dynamic exponential moving average.

[0086] Step 402: An improved anchor-free object detection network is used as the basic structure of the student model. The improved anchor-free object detection network includes an efficient local attention pyramid structure.

[0087] Step 403: Construct a feature alignment detection head, which includes a classification branch, a localization branch, and a feature alignment path that maps the features of the localization branch to the classification branch;

[0088] Step 404: Train the unsafe behavior identification model for miners using a mixed dataset containing a small amount of labeled data and a large amount of unlabeled data. The training process includes:

[0089] In the initial supervised training phase, the student model is pre-trained using only the labeled data;

[0090] In the semi-supervised mutual learning stage, the teacher model and student model are initialized with pre-trained weights, and the teacher model weights are updated with the dynamic exponential moving average strategy. The teacher model generates pseudo-labels for weakly enhanced unlabeled data, and the student model is jointly trained by combining the real labels of the labeled data with the supervision signals of the pseudo-labels.

[0091] The loss functions used in training include supervised loss based on real labels and consistency loss based on the relationship between the teacher model output and the student model output with strongly augmented data.

[0092] In practical implementation, the semi-supervised learning and training architecture based on the teacher-student framework is constructed as follows: Figure 6 As shown, the framework comprises two identical object detection networks, serving as a student network model (trainable) and a teacher network model (updated via the student model's weights), respectively. The teacher model's weights are updated from the student model using a Dynamic Exponential Moving Average (Dynamic EMA) strategy, which can be expressed as:

[0093] , ;

[0094] In the formula, For dynamic weight parameters, The weighting parameter is fixed and set to a value of 0.999. This represents the current iteration number. The total number of iterations is used. Dynamic EMA uses exponential decay to change the weight parameters of EMA with the number of iterations, making the early training stages more efficient. With smaller values, the student model can update the teacher model to a greater extent, especially in the later stages of training. The values ​​are gradually increased, and conservative training is conducted in order to find the optimal parameters.

[0095] The student network model is an improvement upon the anchorless bounding box detection algorithm FCOS. The backbone network uses a ResNet-50 pre-trained on ImageNet as a feature extractor to extract multi-level features from the input image. An efficient local attention pyramid structure (ELA-FPN) serves as the neck network to fuse multi-scale features. In the traditional top-down path of FPN, an efficient local attention module is embedded before the upsampling operation of each feature map. This module performs one-dimensional global average pooling on the input feature map in both the horizontal and vertical directions to obtain feature vectors in two directions. Then, a spatial attention weight map is generated through one-dimensional convolution and a sigmoid activation function. This weight map effectively enhances the feature response of the miner target in the dark background of the conveyor belt and suppresses irrelevant background. Each feature scale corresponds to a detector head, which includes a shared feature layer: a shared module consisting of 3×3 convolutions, group normalization, and ReLU activation functions, which performs preliminary transformations on the input features; a localization branch: containing three consecutive shared modules (CGR3×3) and a final 3×3 convolutional layer, outputting the four distance parameters (l, t, r, b) and centerness of the bounding box; a classification branch: structured parallel to the localization branch, ultimately outputting the probability that each location belongs to different behavior categories (such as walking, sitting, falling, climbing, etc.); and a feature alignment path: a path derived from the last shared module of the localization branch, consisting of a single CGR3×3 module, which maps the features extracted by the localization branch and adds them element-wise to the features of the corresponding layer of the classification branch.

[0096] Monitoring videos of underground conveyor belt areas in coal mines were collected, and image frames containing miners' activities (especially unsafe behaviors such as falling, leaning, and climbing) were extracted. Some images were labeled, and the labeling information included the miners' bounding boxes and corresponding behavior category labels. All data were divided into training, validation, and test sets in an 8:1:1 ratio. To simulate a semi-supervised learning scenario, only 10% of the images in the training set were randomly selected as the labeled data subset, retaining their complete bounding boxes and category labels. The remaining 90% of the images in the training set were used as the unlabeled data subset, using only the images themselves without any labels. The validation and test sets used complete labels. In the initial supervised training phase, known as the "burning" phase, only a 10% subset of labeled data is used to train the student model in a fully supervised manner. The training loss is the FCOS standard loss function, which includes classification loss (Focal Loss), localization loss (IoU Loss), and centrality loss (cross-entropy loss). Training accounts for approximately 10% of the total number of iterations (e.g., 18k iterations) and aims to obtain a preliminary effective model weight. In the semi-supervised mutual learning phase, the student model weights obtained in the "burning" phase are simultaneously loaded into both the teacher and student models, serving as the starting point for the mutual learning phase. For each training iteration: a. Teacher generates pseudo-labels: A batch of unlabeled data undergoes weak augmentation (e.g., only size normalization), which is then input into the teacher model. The teacher model outputs predicted bounding boxes and class probabilities. Threshold filtering (e.g., classification score > 0.7) and non-maximum suppression are used to select high-quality prediction results, which serve as "pseudo-labels" for this batch of unlabeled data; b. Student model training: The same batch of unlabeled data undergoes strong augmentation (e.g., random color jitter, cropping, flipping, etc.), and is mixed with another batch of labeled data (with standard augmentation), and input into the student model; c. Loss calculation: For supervised loss: For the labeled data portion, the standard FCOS loss between its true label and the student model's prediction result is calculated; For unsupervised consistency loss: For the unlabeled data portion, the consistency loss between the student model's prediction result under strong augmentation and the pseudo-labels generated by the teacher model in step a is calculated; For classification tasks, Focal is used. Loss; For localization tasks, use the L1 loss or IoU loss between the teacher and student predicted boxes, and weight it according to the uncertainty of the teacher's prediction; d. Weight update: Calculate the total loss and backpropagate to update the weights of the student model; e. Dynamic EMA update for the teacher: Use the aforementioned dynamic EMA formula to update the weights of the teacher model with the weights of the student model that have just been updated. During training, the performance of the student model is evaluated periodically on the validation set (using full annotations). The main monitoring metric is the mean accuracy (mAP@0.5: 0.95 and mAP@0.5). After training, the student model weight with the highest mAP on the validation set is selected as the final miner unsafe behavior recognition model.

[0097] In an exemplary embodiment, the efficient local attention pyramid structure embeds an efficient local attention module before the upsampling path of the feature pyramid network. The efficient local attention module obtains feature vectors of the feature map in the horizontal and vertical directions of space through one-dimensional pooling operations, and generates spatial attention weights based on the feature vectors.

[0098] In one exemplary embodiment, such as Figure 7 As shown, the specific process of pre-establishing and training the semantic segmentation model for the conveyor belt region includes the following steps 501 to 504:

[0099] Step 501: Construct a semantic segmentation network based on an encoder-decoder structure. The encoder uses a short-term dense cascaded network to extract multi-level features from the input image, and the decoder uses a lightweight multi-level unified attention fusion decoding network.

[0100] Step 502: After the encoder outputs high-level features, a feature attention pyramid pooling module is connected. The feature attention pyramid pooling module captures multi-scale global context information through pooling operations at different scales and integrates a parameter-free average attention mechanism to enhance attention to the detailed regions of the feature map.

[0101] Step 503: Lightweight spatial features are extracted from low-level features in the encoder through the convolutional layer in the lightweight multi-level unified attention fusion decoding network, and the upsampled high-level semantic features are adaptively fused with the low-level spatial features through the unified attention fusion module. The unified attention fusion module generates fusion weights through a spatial attention mechanism.

[0102] Step 504: The semantic segmentation model for the conveyor belt region is trained using a two-stage strategy based on transfer learning. The training process includes:

[0103] In the first stage, the encoder is pre-trained using a publicly available large-scale semantic segmentation dataset to obtain the pre-trained weights of the encoder part.

[0104] In the second stage, on the conveyor belt area image dataset, the encoder is initialized with pre-trained weights, and the entire segmentation network, including the feature attention pyramid pooling module and the lightweight multi-level unified attention fusion decoding network, is fine-tuned end-to-end.

[0105] In specific implementation, the semantic segmentation network structure based on the encoder-decoder architecture is constructed as follows: Figure 8As shown, the Short-Term Dense Concatenated Network (STDC2) is used as the encoder. STDC2 contains five stages (Stage 1-Stage 5) for efficiently extracting multi-scale features from the input image. Stage 1 and Stage 2 each contain a 3×3 standard convolutional block (ConvX) with a stride of 2 for fast downsampling and shallow feature extraction. Stages 3-5 are the core modules, consisting of multiple short-term dense concatenated modules. This module processes the input feature map by performing a 1×1 convolution, then splits it into multiple 3×3 convolutional branches for parallel processing. Finally, all branch outputs are concatenated with the short-circuit concatenation at the channel dimension to achieve dense reuse and efficient extraction of features. The first STDCM of each Stage uses STDCM-B with a stride of 2 for downsampling, while the others use STDCM-A with a stride of 1. Finally, the encoder outputs three feature maps at different scales: a shallow high-resolution feature map (rich in spatial details) from Stage 3, a mid-level feature map from Stage 4, and a deep low-resolution feature map (rich in semantic information) from Stage 5. The Feature Attention Pyramid Pooling module is connected after the deep feature map (from Stage5) output by the encoder. It is used to capture multi-scale global context and enhance detail attention. It performs global average pooling at three scales (1×1, 2×2, and 4×4) in parallel on the input deep feature map to obtain context feature vectors under different receptive fields. Each pooled feature vector is reduced in dimensionality using a 1×1 convolution and then upsampled to the same spatial size as the input feature map through bilinear interpolation. The three upsampled feature maps are then added element-wise for preliminary fusion. Subsequently, the fused feature map is input into the Parametric Average Attention (PFAA) module. The PFAA module calculates the average value of the feature map along the spatial and channel dimensions, respectively, and generates a spatial attention weight map through expansion and the Sigmoid function. This weight map can automatically focus on regions in the feature map that contain important structural information (such as object edges), thereby enhancing the model's ability to perceive details. The decoder is used to progressively restore spatial resolution and fuse multi-level features. For the low-level feature maps extracted from encoder Stage 3 and Stage 4, partial convolutional layers are first used for processing. Only a portion of the channels of the input feature map are subjected to regular convolution calculations, while the remaining channels remain unchanged, thereby efficiently extracting spatial detail features at extremely low computational cost.The core of the decoder is the Unified Attention Fusion (UAFM) module. Taking the fusion of deep features (from the FAPP module, upsampled) and shallow features (from Stage3, processed by PConv) as an example: a. Bilinear interpolation upsampling is performed on the deep semantic features to make them consistent with the size of the shallow spatial features; b. The upsampled deep and shallow features are simultaneously input into the spatial attention submodule. This submodule generates four 1×1×C descriptors by calculating the mean and maximum value of the two feature maps in the channel dimension, and then concatenating them to generate a spatial attention weight map through convolution and sigmoid; c. This weight map is used to weight the deep and shallow features. The decoding process adopts a progressive fusion strategy, fusing deep, mid, and shallow features sequentially through UAFM. After each fusion, a 2x upsampling is performed, and finally the feature map is restored to the input image size. A convolutional layer and a softmax function are used to output a probability distribution map of each pixel belonging to three types of targets (miners, guardrails, conveyor belts) and the background.

[0106] In an exemplary embodiment, the specific process of making a final judgment on unsafe behaviors of miners based on the spatial interaction relationship between preliminary behavior category information, miner location bounding boxes, and pixel-level segmentation results includes:

[0107] When the initial behavior category is "falling", it is judged as an unsafe behavior;

[0108] When the initial behavior category is "leaning" or "climbing", perform the following steps:

[0109] Based on the miner's location bounding box, obtain the corresponding miner pixel region;

[0110] Calculate the spatial relationship measure between the miner pixel region and the guardrail pixel region and / or conveyor belt pixel region in the pixel-level segmentation result;

[0111] The spatial relationship measurement is judged based on a preset threshold. If the condition is met, it is judged as an unsafe behavior corresponding to "leaning against the guardrail", "climbing the guardrail" or "climbing the conveyor belt".

[0112] Based on the same inventive concept, this application also provides a device for identifying unsafe behaviors of miners in coal mine conveyor belt areas, as described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the device for identifying unsafe behaviors of miners in coal mine conveyor belt areas provided below can be found in the limitations of the method for identifying unsafe behaviors of miners in coal mine conveyor belt areas described above, and will not be repeated here.

[0113] In one exemplary embodiment, a device for identifying unsafe behaviors of miners in coal mine conveyor belt areas is provided, comprising:

[0114] The image acquisition and preprocessing module is used to acquire keyframes and call the image super-resolution reconstruction model for processing.

[0115] The behavior recognition module, which is built into the miner unsafe behavior recognition model, is used to output preliminary behavior recognition results;

[0116] An environment segmentation module, built into the semantic segmentation model of the conveyor belt region, is used to output the environmental target segmentation results;

[0117] The interactive analysis and discrimination module is used to discriminate unsafe behaviors based on the preliminary behavior recognition results and the environmental target segmentation results.

[0118] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores video tag processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for identifying unsafe behaviors of miners in coal mine conveyor belt areas.

[0119] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0120] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0121] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0124] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0125] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for identifying unsafe behaviors of miners in coal mine conveyor belt areas, characterized in that, The method includes: Collect video images of the coal mine conveyor belt area and extract keyframes from them; The keyframes are input into a pre-established and trained image super-resolution reconstruction model to obtain a high-resolution reconstructed image; The high-resolution reconstructed image is input in parallel into a pre-established and trained model for identifying unsafe behaviors of miners and a semantic segmentation model for the conveyor belt region. The output of the unsafe behavior identification model includes at least the bounding box of the miner's location and preliminary behavior category information. The output of the semantic segmentation model for the conveyor belt region includes at least the pixel-level segmentation results of the miner, the guardrail, and the conveyor belt. Based on the spatial interaction between the preliminary behavior category information, the miner's location bounding box, and the pixel-level segmentation results, the miner's unsafe behavior is finally identified.

2. The method for identifying unsafe behaviors of miners in coal mine conveyor belt areas according to claim 1, characterized in that, The specific process of pre-establishing and training the image super-resolution reconstruction model includes: Original images of the underground conveyor belt area in a coal mine are collected to construct a high-resolution image set, and the high-resolution images are downsampled to generate corresponding low-resolution images, thereby forming low-resolution-high-resolution image pairs as training samples. A generative adversarial network (GAN) model is constructed, comprising a shallow feature extraction module, a multi-scale dense residual attention network, an image reconstruction module, and a discriminator. The shallow feature extraction module performs initial feature extraction on the input low-resolution image. Multi-scale feature extraction is performed using dense residual blocks with convolutional kernels of different sizes connected in parallel within the multi-scale dense residual attention network, and the fused features are adaptively weighted via a channel attention mechanism. The image reconstruction module upsamples the features output from the multi-scale dense residual attention network and reconstructs a high-resolution image through convolution. The discriminator determines whether the input image is a true high-resolution image or an image reconstructed by the generator. Based on the low-resolution to high-resolution image pairs used as training samples, the generative adversarial network model is alternately optimized and trained using texture loss and adversarial loss as joint loss functions until the model converges.

3. The method for identifying unsafe behaviors of miners in coal mine conveyor belt areas according to claim 2, characterized in that, The specific process of extracting multi-scale features through dense residual blocks with convolutional kernels of different sizes connected in parallel in a multi-scale dense residual attention network, and adaptively weighting the fused features via a channel attention mechanism, includes: For the input feature map of the multi-scale dense residual channel attention module, it is simultaneously input into the first dense residual block and the second dense residual block in parallel. Feature extraction is performed using a convolutional kernel of the first size from the first dense residual block, and a first feature map is output. Feature extraction is performed using a second-size convolutional kernel that is different from the first size in the second dense residual block, and a second feature map is output. The first feature map and the second feature map are concatenated along the channel dimension to obtain a fused feature map; The fused feature map is input into the channel attention mechanism to generate a channel attention weight vector; The channel attention weight vector is multiplied channel by channel with the fused feature map to output an adaptively weighted feature map.

4. The method for identifying unsafe behaviors of miners in coal mine conveyor belt areas according to claim 1, characterized in that, The specific process of pre-establishing and training the unsafe behavior identification model for miners includes: A semi-supervised learning training architecture based on a teacher-student framework is constructed, in which the student model is the target detection network to be trained, and the teacher model has the same structure as the student model and its weights are updated from the student model through a dynamic exponential moving average. An improved anchor-free object detection network is used as the basic structure of the student model. The improved anchor-free object detection network includes an efficient local attention pyramid structure. Construct a feature alignment detection head, which includes a classification branch, a localization branch, and a feature alignment path that maps the features of the localization branch to the classification branch; The unsafe behavior identification model for miners is trained using a mixed dataset containing a small amount of labeled data and a large amount of unlabeled data. The training process includes: In the initial supervised training phase, the student model is pre-trained using only the labeled data; In the semi-supervised mutual learning stage, the teacher model and student model are initialized with pre-trained weights, and the teacher model weights are updated with the dynamic exponential moving average strategy. The teacher model generates pseudo-labels for weakly enhanced unlabeled data, and the student model is jointly trained by combining the real labels of the labeled data with the supervision signals of the pseudo-labels. The loss functions used in training include supervised loss based on real labels and consistency loss based on the relationship between the teacher model output and the student model output with strongly augmented data.

5. The method for identifying unsafe behaviors of miners in coal mine conveyor belt areas according to claim 4, characterized in that, The efficient local attention pyramid structure embeds an efficient local attention module before the upsampling path of the feature pyramid network. The efficient local attention module obtains the feature vectors of the feature map in the horizontal and vertical directions of space through one-dimensional pooling operation, and generates spatial attention weights based on the feature vectors.

6. The method for identifying unsafe behaviors of miners in coal mine conveyor belt areas according to claim 1, characterized in that, The specific process of pre-establishing and training the semantic segmentation model for the conveyor belt region includes: A semantic segmentation network based on an encoder-decoder structure is constructed. The encoder adopts a short-term dense cascaded network to extract multi-level features from the input image, and the decoder adopts a lightweight multi-level unified attention fusion decoding network. After the encoder outputs high-level features, a feature attention pyramid pooling module is connected. This module captures multi-scale global context information through pooling operations at different scales and integrates a parameter-free average attention mechanism to enhance attention to detailed regions of the feature map. The lightweight multi-level unified attention fusion decoding network extracts low-level spatial features from the encoder through convolutional layers, and adaptively fuses the upsampled high-level semantic features with the low-level spatial features through a unified attention fusion module. The unified attention fusion module generates fusion weights through a spatial attention mechanism. The semantic segmentation model for the conveyor belt region is trained using a two-stage strategy based on transfer learning. The training process includes: In the first stage, the encoder is pre-trained using a publicly available large-scale semantic segmentation dataset to obtain the pre-trained weights of the encoder part. In the second stage, on the conveyor belt area image dataset, the encoder is initialized with pre-trained weights, and the entire segmentation network, including the feature attention pyramid pooling module and the lightweight multi-level unified attention fusion decoding network, is fine-tuned end-to-end.

7. The method for identifying unsafe behaviors of miners in coal mine conveyor belt areas according to claim 1, characterized in that, The specific process for making a final judgment on unsafe behaviors of miners based on the spatial interaction relationship between preliminary behavior category information, miner location bounding boxes, and pixel-level segmentation results includes: When the initial behavior category is "falling", it is judged as an unsafe behavior; When the initial behavior category is "leaning" or "climbing", perform the following steps: Based on the miner's location bounding box, obtain the corresponding miner pixel region; Calculate the spatial relationship measure between the miner pixel region and the guardrail pixel region and / or conveyor belt pixel region in the pixel-level segmentation result; The spatial relationship measurement is judged based on a preset threshold. If the condition is met, it is judged as an unsafe behavior corresponding to "leaning against the guardrail", "climbing the guardrail" or "climbing the conveyor belt".

8. A device for identifying unsafe behaviors of miners in coal mine conveyor belt areas, characterized in that, The apparatus for performing the method as described in any one of claims 1-7 comprises: The image acquisition and preprocessing module is used to acquire keyframes and call the image super-resolution reconstruction model for processing. The behavior recognition module, which is built into the miner unsafe behavior recognition model, is used to output preliminary behavior recognition results; An environment segmentation module, built into the semantic segmentation model of the conveyor belt region, is used to output the environmental target segmentation results; The interactive analysis and discrimination module is used to discriminate unsafe behaviors based on the preliminary behavior recognition results and the environmental target segmentation results.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method for identifying unsafe behaviors of miners in coal mine conveyor belt areas according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method for identifying unsafe behaviors of miners in the coal mine conveyor belt area as described in any one of claims 1-7.

Images