Intelligent monitoring method for coal mine underground based on data fusion

Abstract

The application provides a coal mine underground intelligent supervision method based on data fusion, which is used to solve the problem that the existing coal mine monitoring system personnel violation behavior identification is easily affected by environmental parameters. The method steps are: obtaining environmental monitoring data and coal mine personnel operation images; setting an environmental monitoring data threshold, judging whether the current environment is abnormal according to the environmental monitoring data and the environmental monitoring data threshold, and recording the environmental abnormal data; extracting features from the coal mine personnel operation images, identifying the violation behavior of the coal mine personnel in combination with the environmental monitoring data, and recording the personnel violation data; transmitting the environmental abnormal data and the personnel violation data to a remote platform, and the remote platform sends corresponding decision control signals and alarm signals. The application realizes the monitoring of the underground environment and the personnel violation behavior, and adjusts the parameters of the personnel violation behavior identification model through the environmental monitoring data, thereby improving the identification accuracy.

Description

Technical Field

[0001] This invention relates to the field of underground safety supervision in coal mines, and in particular to an intelligent supervision method for underground coal mines based on data fusion. Background Technology

[0002] Coal mine safety is the lifeline of the coal industry, and real-time monitoring of the underground working environment and standardized management of personnel behavior are key links in ensuring safe production. With the continuous advancement of intelligent coal mine construction, the demand for refined and precise underground safety supervision is becoming increasingly urgent. Currently, underground safety supervision in coal mines mainly relies on environmental monitoring systems and video surveillance systems. Environmental monitoring systems typically collect environmental parameters in real time through various sensors deployed underground and issue alarms when parameters exceed limits; video surveillance systems acquire operational footage through cameras installed at key locations underground.

[0003] Currently, the characterization of violations by coal mine workers mainly employs traditional image recognition combined with single feature extraction methods. However, these methods have significant drawbacks in the complex underground environment: First, the complex underground environment, with factors such as high dust levels, low light levels, electromagnetic interference, and personnel occlusion, results in poor image quality and blurred personnel target features. Traditional single feature extraction methods cannot comprehensively capture the characteristics of personnel violations, easily leading to feature confusion and low accuracy in violation identification. Second, they have weak anti-interference capabilities. Existing methods have not been specifically optimized for interference such as underground dust, fog, uneven lighting, and occlusion, resulting in poor feature extraction stability. Third, the characterization is incomplete. Existing methods only achieve simple identification of violations without constructing a comprehensive violation characterization model, failing to provide key information such as violation characteristics, confidence levels, and location. Fourth, they have poor adaptability. Existing methods are mostly designed for surface scenarios and are not adapted to the diverse working postures of underground personnel, the complexity of violation types, and frequent occlusion.

[0004] In existing technologies, violation recognition is only used as an auxiliary function, without employing a skeleton and contour feature fusion approach. Feature extraction is incomplete, and there is no specific optimization for the complex underground environment. Furthermore, existing skeleton extraction and contour feature extraction technologies are mostly applied to surface personnel behavior recognition scenarios, without optimizing models for the environmental characteristics of coal mines, such as high dust, low light, and frequent occlusion, making them difficult to directly apply to the characterization of coal mine personnel violations. Summary of the Invention

[0005] The purpose of this invention is to provide a data fusion-based intelligent monitoring method for underground coal mines. This method addresses the technical problem that existing coal mine monitoring systems are susceptible to environmental parameter interference in identifying personnel violations.

[0006] This application provides a data fusion-based intelligent monitoring method for underground coal mines, the specific steps of which are as follows:

[0007] S1: Acquire environmental monitoring data and images of coal mine workers at work, wherein the environmental monitoring data includes dust concentration and light intensity data;

[0008] S2: Set environmental monitoring data thresholds, determine whether the current environment is abnormal based on environmental monitoring data and environmental monitoring data thresholds, and record abnormal environmental data;

[0009] S3: Extract features from images of coal mine workers at work, combine them with environmental monitoring data to identify violations by coal mine workers, and record the data on violations.

[0010] S4: Transmit abnormal environmental data and personnel violation data to the remote platform, and the remote platform issues corresponding decision control signals and alarm signals.

[0011] Optionally, the specific steps in step S3 are as follows:

[0012] S3.1: Sort the images of coal mine workers in operation according to time to form a continuous sequence of images of workers in operation, and perform multi-step preprocessing on the image sequence;

[0013] S3.2: Extract skeleton features from the preprocessed personnel image to obtain the coordinates of key skeletal nodes and skeleton connection information;

[0014] S3.3: Extract contour features from the preprocessed personnel images to obtain complete personnel contour information;

[0015] S3.4: Construct a feature fusion module based on an adaptive weighted fusion algorithm, adjust the weights according to environmental monitoring data, and fuse the skeleton features and contour features of personnel through the adjusted feature fusion module to obtain a fused feature vector;

[0016] S3.5: Construct a deep learning-based model for identifying violations, and train the model using a set of image samples of common violations by coal mine workers;

[0017] S3.6: Input the fused feature vector into the trained recognition model, output the type of violation, the confidence level of the violation, and the location information of the violation, and record the violation data.

[0018] Optionally, the specific steps in step S3.2 are as follows:

[0019] S3.2.1: Embed the CBAM spatial attention module into the OpenPose network to form a skeleton extraction network based on the improved OpenPose. The skeleton extraction network includes an input layer, a feature extraction layer, an attention optimization layer, a skeleton node prediction layer, and a skeleton generation layer.

[0020] S3.2.2: Input the preprocessed personnel image into the improved OpenPose network to obtain the coordinates of key skeletal nodes and skeleton connection information of the personnel;

[0021] S3.2.3: The Kalman filter algorithm is used to smooth the coordinates of key skeletal nodes and remove abnormal skeletal nodes that deviate from the normal range; and the broken skeleton structure is repaired by verifying the connection relationship of skeletal nodes.

[0022] Optionally, the specific steps in step S3.3 are as follows:

[0023] S3.3.1: The Canny edge detection algorithm is adopted, and high and low thresholds are set to detect the real edges of the person image, remove false edges, and retain the contour edges of the person target;

[0024] S3.3.2: A chain code-based contour tracking algorithm is used to track the detected contour edges, extract the closed contour of the person, and remove small burrs in the contour.

[0025] S3.3.3: Calculate the core feature parameters of the contour and integrate the core feature parameters to form a personnel contour feature vector; the core feature parameters include contour perimeter, contour area, number of contour concave and convex points, contour rectangularity, contour circularity, and Hu invariant moments.

[0026] Optionally, the specific steps in step S3.4 are as follows:

[0027] S3.4.1: Standardize the skeleton features and contour features to eliminate the influence of different feature dimensions;

[0028] S3.4.2: Dynamically adjust the weighting coefficients of skeleton features and contour features based on environmental monitoring data;

[0029] S3.4.3: Using a weighted summation method, the standardized skeleton features and contour features are fused to obtain a fused feature vector, which includes personnel limb posture information and body shape information.

[0030] Optionally, the specific steps in step S3.5 are as follows:

[0031] S3.5.1: Using ResNet18 as the backbone network, the backbone network is pruned, and a global average pooling layer, a fully connected layer, and a softmax classification layer are added to the last layer of the backbone network to form a violation behavior recognition model.

[0032] S3.5.2: The model is trained using a set of image samples of common violations by coal mine workers. The training process uses the cross-entropy loss function and the focus loss function to obtain the trained recognition model.

[0033] The image sample set of common violations includes images of not wearing a safety helmet, not wearing a safety belt, climbing heights without permission, entering dangerous areas without authorization, leaving one's post, operating equipment in violation of regulations, working under the influence of alcohol, and working while fatigued.

[0034] Optionally, the specific steps in step S3.6 are as follows:

[0035] S3.6.1: Input the fused feature vector into the trained recognition model, and output the type of violation, the confidence level of the violation, and the location information of the violation.

[0036] S3.6.2: Set a confidence threshold to eliminate false identification results with a confidence level below the threshold;

[0037] S3.6.3: Construct a characterization model of coal mine personnel violations, encapsulate the output of the characterization model into a standardized JSON format, and transmit the personnel violation data in JSON format to a remote platform.

[0038] Optionally, the representation model includes a behavior type representation represented by a violation behavior code and a text description, a feature representation represented by a fused feature vector, a confidence representation represented by violation confidence and feature matching degree, and a location representation represented by the bounding box coordinates and center coordinates of the violator.

[0039] The standardized JSON format includes a timestamp, camera device number, bounding box coordinates of the violator, center coordinates, violation type, behavior code, violation confidence, feature matching degree, fused feature vector, and environmental monitoring data.

[0040] Optionally, step S3 further includes step S3.7: collecting image samples of new types and new scenarios of violations in the mine on a monthly basis, incorporating the new image samples of violations using an incremental learning method, and fine-tuning the violation recognition model and feature fusion module.

[0041] Optionally, in step S4, the remote platform may send corresponding decision control signals and alarm signals, including:

[0042] When the dust concentration exceeds the preset upper threshold, the remote platform sends a control signal to start the backup fan and begins sampling in the next cycle. When the dust concentration falls below the preset lower threshold, the backup fan stops. When the light intensity falls below the preset lower threshold, the remote platform sends a control signal to start the backup lighting equipment and begins sampling in the next cycle. When the light intensity rises above the preset upper threshold, the backup lighting equipment is turned off. When a violation is detected, an early warning is issued and the on-site management personnel are notified.

[0043] Because of the adoption of the above technical solution, the present invention has the following advantages:

[0044] 1. This application achieves the monitoring of the underground environment and personnel violations by collecting environmental monitoring data and images of personnel operations, and improves the accuracy of identification by adjusting the parameters of the personnel violation identification model through environmental monitoring data.

[0045] 2. This application constructs a skeleton extraction network based on an improved OpenPose by embedding the CBAM attention module in OpenPose, achieving a skeleton node extraction accuracy of greater than or equal to 97%; it uses the Canny edge detection algorithm and contour tracking algorithm to extract contour features; and it fuses the two features through a feature fusion module, so that the fused feature vector has both pose and morphological information, significantly improving the comprehensiveness of feature representation.

[0046] 3. This application adopts an adaptive weight fusion strategy for underground environments with high dust, low light, and obstruction. It dynamically adjusts feature weights based on environmental monitoring data to ensure a high recognition rate even in harsh environments.

[0047] 4. This application adopts a pruned and optimized ResNet18 model, combined with a focus loss function to solve the sample imbalance problem. The accuracy of violation behavior recognition is greater than or equal to 98%, and the inference latency is less than or equal to 30ms, which meets the real-time requirements.

[0048] 5. This application constructs a four-dimensional representation model, outputs standardized JSON data, and deeply collaborates with existing regulatory systems to support the entire process of violation supervision, hidden danger investigation, and emergency response.

[0049] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0050] The accompanying drawings of this invention are described below.

[0051] Figure 1 This is a flowchart of the intelligent monitoring method for underground coal mines based on data fusion, as described in this invention.

[0052] Figure 2 This is a flowchart of the method for feature extraction and identification of personnel violations according to the present invention.

[0053] Figure 3 This is a flowchart of the method for preprocessing images of coal mine workers according to the present invention.

[0054] Figure 4This is a schematic diagram of the improved OpenPose skeleton extraction network structure of the present invention.

[0055] Figure 5 This is a flowchart of the method for extracting the skeleton of personnel according to the present invention.

[0056] Figure 6 This is a flowchart of the method for extracting human contour features according to the present invention.

[0057] Figure 7 This is a flowchart of the method for fusing skeleton features and contour features according to the present invention.

[0058] Figure 8 This is a schematic diagram of the violation behavior recognition model of the present invention.

[0059] Figure 9 This is a schematic diagram illustrating the collaborative linkage between the violation behavior representation model of this invention and the existing regulatory system. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0061] Example 1:

[0062] like Figures 1-9 The method for intelligent monitoring of underground coal mines based on data fusion, as shown, includes the following steps:

[0063] S1: Acquire environmental monitoring data and images of coal mine workers at work, wherein the environmental monitoring data includes dust concentration and light intensity data;

[0064] In this embodiment, images of coal mine workers and environmental monitoring data are collected in key monitoring areas such as underground mining faces, transport roadways, chambers, and transfer points. The environmental monitoring data is collected in real-time using dust concentration sensors and light sensors; the images of coal mine workers are collected in real-time using intrinsically safe mining cameras. These intrinsically safe cameras meet underground requirements for explosion-proof, dust-proof, waterproof, and electromagnetic interference resistance, with image resolution no lower than 1080P and a frame rate no lower than 25fps, ensuring image clarity and real-time performance, and adapting to the complex working environment of high dust, low light, strong interference, dense personnel, and frequent obstructions.

[0065] S2: Set environmental monitoring data thresholds, determine whether the current environment is abnormal based on environmental monitoring data and environmental monitoring data thresholds, and record abnormal environmental data;

[0066] In this embodiment, the upper limit threshold for dust concentration is set to 50 mg / m³, and the lower limit threshold is set to 15 mg / m³; the upper limit threshold for light intensity is set to 200 lux, and the lower limit threshold is set to 50 lux. When the dust concentration exceeds its upper limit threshold or the light intensity is lower than its lower limit threshold, the environment is judged to be abnormal.

[0067] S3: As Figure 2 As shown, feature extraction is performed on images of coal mine workers at work, and violations by coal mine workers are identified and recorded in conjunction with environmental monitoring data. The specific steps are as follows:

[0068] S3.1: Sort the images of coal mine workers in operation according to time to form a continuous sequence of images of workers in operation, and perform multi-step preprocessing on the image sequence;

[0069] In this embodiment, to address the common problems in downhole images such as dust interference, fog blurring, uneven lighting, significant noise, and poor image quality caused by personnel obstruction, such as... Figure 3 As shown, a multi-step dedicated preprocessing is performed on the acquired continuous image sequence:

[0070] S3.1.1: Dehazing based on Retinex enhancement algorithm: Separates the reflection component and illumination component of the image, effectively removes underground fog interference, and improves image clarity;

[0071] S3.1.2: Noise suppression based on median filtering and Gaussian filtering cascade: First, median filtering is used to remove salt-and-pepper noise from the image, and then Gaussian filtering is used to suppress random noise, balancing the denoising effect and the preservation of personnel target details;

[0072] S3.1.3: Illumination adjustment based on adaptive histogram equalization algorithm: Adjust the grayscale distribution of the image to solve the difference in image brightness caused by uneven illumination in the well and improve image contrast;

[0073] S3.1.4: Dust False Target Removal Based on Morphological Opening Operation: Removes small false targets caused by dust adhesion in the image while preserving the detailed features of the personnel target;

[0074] S3.1.5: Occlusion region repair based on generative adversarial networks: fill in the missing pixels in the occluded area of ​​a person, and restore the complete outline and pose information of the person.

[0075] Through the above multi-step preprocessing, the signal-to-noise ratio of the image is ensured to be ≥38dB, laying a high-quality data foundation for subsequent processing.

[0076] S3.2: As Figure 4 and Figure 5As shown, skeleton feature extraction is performed on the preprocessed personnel image to obtain the coordinates of key skeletal nodes and skeleton connection information; the specific steps are as follows:

[0077] S3.2.1: Embed the CBAM spatial attention module into the OpenPose network to form a skeleton extraction network based on the improved OpenPose. The skeleton extraction network includes an input layer, a feature extraction layer, an attention optimization layer, a skeleton node prediction layer, and a skeleton generation layer.

[0078] In this embodiment, the CBAM spatial attention module adopts a parallel channel-space dual attention structure, the channel attention submodule has a compression ratio of 16, and the spatial attention submodule uses a 7×7 convolution kernel.

[0079] S3.2.2: Input the preprocessed personnel image into the improved OpenPose network to obtain the coordinates of key skeletal nodes and skeleton connection information of the personnel;

[0080] In this embodiment, the feature extraction layer extracts multi-scale features of the person image; the attention optimization layer focuses on the area where the person's skeletal nodes are located through spatial weight allocation, weakening the influence of background interference and occlusion areas; the skeletal node prediction layer predicts the coordinates of 18 core skeletal nodes of the person (head, neck, shoulder, elbow, wrist, hip, knee and ankle); and the skeleton generation layer generates a complete person skeleton structure through the connection relationship of skeletal nodes.

[0081] S3.2.3: The Kalman filter algorithm is used to smooth the coordinates of key skeletal nodes and remove abnormal skeletal nodes that deviate from the normal range; and the broken skeleton structure is repaired by verifying the connection relationship of skeletal nodes.

[0082] In this embodiment, the integrity and accuracy of the skeleton structure are ensured through the above steps. The skeleton extraction accuracy rate is ≥97%, and the skeleton structure repair accuracy rate is ≥96%.

[0083] S3.3: For example Figure 6 As shown, contour feature extraction is performed on the preprocessed personnel image to obtain complete personnel contour information; the specific steps are as follows:

[0084] S3.3.1: The Canny edge detection algorithm is adopted, and high and low thresholds are set to detect the real edges of the person image, remove false edges, and retain the contour edges of the person target;

[0085] S3.3.2: A chain code-based contour tracking algorithm is used to track the detected contour edges, extract the closed contour of the person, remove small burrs in the contour, and ensure the integrity and smoothness of the contour.

[0086] S3.3.3: Calculate the core feature parameters of the contour and integrate the core feature parameters to form a personnel contour feature vector; the core feature parameters include contour perimeter, contour area, number of contour concave and convex points, contour rectangularity, contour circularity and 7 Hu invariant moments.

[0087] In this embodiment, Hu invariant moments are used to characterize the shape features of the contour, unaffected by image scaling, rotation, or translation, ensuring the stability of contour feature extraction. All feature parameters are integrated to form a person contour feature vector.

[0088] S3.4: For example Figure 7 As shown, a feature fusion module based on an adaptive weighted fusion algorithm is constructed. The weights are adjusted according to environmental monitoring data. The adjusted feature fusion module then fuses the skeleton features and contour features of the personnel to obtain a fused feature vector. The specific steps are as follows:

[0089] S3.4.1: Standardize the skeleton features and contour features to eliminate the influence of different feature dimensions;

[0090] S3.4.2: Dynamically adjust the weighting coefficients of skeleton features and contour features based on environmental monitoring data;

[0091] In this embodiment, the weighting coefficient of the entire skeleton feature ranges from 0.55 to 0.65, and the weighting coefficient of the contour feature ranges from 0.35 to 0.45. When the dust concentration is >50 mg / m³ or the light intensity is <50 lux, the weight of the contour feature is increased to 0.45, and the weight of the skeleton feature is decreased to 0.55; when the light intensity is ≥200 lux and the dust concentration is ≤15 mg / m³, the weight of the skeleton feature is increased to 0.65, and the weight of the contour feature is decreased to 0.35.

[0092] S3.4.3: Using a weighted summation method, the standardized skeleton features and contour features are fused to obtain a fused feature vector, which includes personnel limb posture information and body shape information.

[0093] S3.5: Construct a deep learning-based model for identifying violations, and train the model using a sample set of images depicting common violations by coal mine workers; the specific steps are as follows:

[0094] S3.5.1: Using ResNet18 as the backbone network, the backbone network is pruned, and a global average pooling layer, a fully connected layer, and a softmax classification layer are added to the last layer of the backbone network to form a violation behavior recognition model.

[0095] In this embodiment, the pruning rate of the backbone network is 30%, and the model volume is compressed to 40% of the original size after pruning.

[0096] S3.5.2: The model is trained using a set of image samples of common violations by coal mine workers. The training process uses the cross-entropy loss function and the focus loss function to obtain the trained recognition model.

[0097] The image sample set of common violations includes images of not wearing a safety helmet, not wearing a safety belt, climbing heights without permission, entering dangerous areas without authorization, leaving one's post, operating equipment in violation of regulations, working under the influence of alcohol, and working while fatigued.

[0098] In this embodiment, each violation corresponds to a unique behavior code; the number of samples for each violation is no less than 1000; and the cross-entropy loss function and the focal loss function are used during training to solve the problem of imbalanced violation samples.

[0099] S3.6: For example Figure 8 As shown, the fused feature vectors are input into the trained recognition model, which outputs the type of violation, the confidence level of the violation, and the location information of the violation, and records the violation data. The specific steps are as follows:

[0100] S3.6.1: Input the fused feature vector into the trained recognition model, and output the type of violation, the confidence level of the violation, and the location information of the violation.

[0101] S3.6.2: Set a confidence threshold to eliminate false identification results with a confidence level below the threshold;

[0102] In this embodiment, a confidence threshold of 0.6 is set to eliminate false identification results with a confidence level below the threshold. The inference delay is ≤30ms to ensure a violation identification accuracy rate of ≥98%.

[0103] S3.6.3: Construct a characterization model of coal mine personnel violations, encapsulate the output of the characterization model into a standardized JSON format, and transmit the personnel violation data in JSON format to a remote platform.

[0104] In this embodiment, as Figure 9 As shown, the representation model includes a behavior type representation represented by violation behavior encoding and text description, a feature representation represented by fused feature vectors, a confidence representation represented by violation confidence and feature matching degree, and a location representation represented by the bounding box coordinates and center coordinates of the violator.

[0105] The standardized JSON format includes a timestamp, camera device number, bounding box coordinates of the violator, center coordinates, violation type, behavior code, violation confidence, feature matching degree, fused feature vector, and environmental monitoring data.

[0106] The remote platform includes an AI-based coal mine safety hazard identification system and a safety supervision platform.

[0107] S3.7: Collect image samples of new types and scenarios of violations in the mine on a monthly basis, and use incremental learning methods to incorporate the new image samples of violations to fine-tune the violation recognition model and feature fusion module.

[0108] In this embodiment, by fine-tuning the violation behavior recognition model and feature fusion module, the adaptability of the method in complex underground scenarios and new violation types is improved, ensuring that the violation behavior recognition accuracy remains stable at ≥98% over a long period of time.

[0109] S4: Transmits abnormal environmental data and personnel violation data to a remote platform, which then issues corresponding decision control signals and alarm signals, including:

[0110] When the dust concentration exceeds the preset upper threshold, the remote platform sends a control signal to start the backup fan and begins sampling in the next cycle. When the dust concentration falls below the preset lower threshold, the backup fan stops. When the light intensity falls below the preset lower threshold, the remote platform sends a control signal to start the backup lighting equipment and begins sampling in the next cycle. When the light intensity rises above the preset upper threshold, the backup lighting equipment is turned off. When a violation is detected, an early warning is issued and the on-site management personnel are notified.

[0111] S5: Experimental Verification:

[0112] S5.1: Violation Detection Experiment at Underground Mining Face:

[0113] S5.1.1: Experimental environment configuration: Dust concentration sensor: CGWZ-100 type; Light sensor: GY-30 type; Intrinsically safe camera for mining: KBA12 type, resolution 1920×1080, frame rate 25fps; Edge computing node: NVIDIA Jetson AGX Xavier, explosion-proof packaging for mining, computing power 32 TOPS; Software environment: Python 3.8, PyTorch 1.12, CUDA 11.4, OpenCV 4.5.

[0114] Dataset: 120 hours of video footage from an underground coal mine's working face and transport roadways were collected, and eight types of violations were labeled: not wearing a safety helmet, not wearing a safety belt, unauthorized climbing, unauthorized entry into dangerous areas, leaving one's post, improper operation of equipment, working under the influence of alcohol, and working while fatigued. A total of 15,600 images were collected, with a resolution of 1920×1080, and were divided into training, validation, and test sets in a 7:2:1 ratio.

[0115] S5.1.2: Image Acquisition and Preprocessing: The acquired images were preprocessed using the Retinex algorithm for dehazing, followed by cascaded median filtering (3×3 kernel) and Gaussian filtering (5×5 kernel, σ = 1.0) for denoising, adaptive histogram equalization (amplitude 2.0, cell size 8×8), morphological opening (3×3 kernel) for dust removal, and a GAN network based on Partial Conv for repairing occluded regions. The average signal-to-noise ratio of the preprocessed images reached 40.2 dB, meeting the requirements for subsequent processing.

[0116] S5.1.3: Skeleton Extraction and Optimization: An improved OpenPose network was constructed. Taking the preprocessed image as input, the coordinates of 18 skeletal nodes were extracted. Kalman filtering (process noise covariance Q = 0.01, observation noise covariance R = 0.1) was used to smooth the node coordinates. Testing showed that the accuracy of skeletal node extraction reached 97.3%, and the accuracy of skeleton structure repair reached 96.5%.

[0117] S5.1.4: Contour feature extraction: The Canny algorithm (low threshold of 50, high threshold of 150) is used to detect edges, the contour is tracked based on the chain code algorithm, and the core feature parameters are calculated.

[0118] S5.1.5: Feature Fusion and Weight Adjustment: When environmental parameters exceed a threshold, the weights are automatically adjusted. Feature vectors are weighted and fused after Z-score standardization.

[0119] S5.1.6: Model Training and Optimization: After pruning the ResNet18 backbone network channels, the model size was compressed from 44.6MB to 17.8MB, and the inference latency was reduced from 45ms to 28ms. During training, the batch size was set to 32, the initial learning rate was 0.001, and a cosine annealing strategy was used, with a total of 150 training epochs. The loss function adopted was a weighted combination of cross-entropy loss (CE Loss) and focus loss (Focal Loss) (weights 0.7:0.3), effectively addressing the imbalanced sample problem.

[0120] S5.1.7: Ablation Experiment Verification: To verify the effectiveness of each module, control groups A, B, C and D were designed to conduct a comparative experiment with the group in this application (test set of 3120 images): The ablation experiment results are shown in Table 1.

[0121] Table 1 Ablation Experiment Results

[0122]

[0123] As shown in Table 1, 1. The CBAM attention module improves the skeleton extraction accuracy by about 4.3%; 2. Feature fusion improves the accuracy by about 4.8% compared to single features; 3. Adaptive weights improve the accuracy by about 3.2% compared to fixed weights in occluded scenarios; 4. Model pruning significantly reduces latency and storage requirements while maintaining an accuracy loss of <0.5%.

[0124] S5.1.8: Verification of the adaptive adjustment effect of feature weights:

[0125] In a high-dust-coverage scenario (dust concentration 65 mg / m³, illumination 30 lux), 100 sets of samples were tested. The fixed-weight group (0.6:0.4) achieved a recognition accuracy of 92.3%; the adaptive-weight group (automatically adjusted to 0.55:0.45) achieved a recognition accuracy of 97.8%.

[0126] 100 sets of samples were selected for testing in a normal lighting, unobstructed scene (dust concentration 15mg / m³, light intensity 200lux). The recognition accuracy was 97.5% for the fixed weight (0.6:0.4) group and 99.1% for the adaptive weight (automatically adjusted to 0.65:0.35) group.

[0127] S5.2: Stress test for multi-user concurrent scenarios:

[0128] S5.2.1: Experimental Environment Configuration: Four cameras are deployed in a transfer point scenario (space 15m×8m) to monitor 35 workers simultaneously. The system configuration is as follows: 1. Edge computing node: Parallel processing of 4 video streams; 2. Inference batch: Processing 8 people per batch, using batch inference optimization; 3. Communication: 5G private network, uplink bandwidth 100Mbps, latency <10ms.

[0129] S5.2.2: Test Results: Single frame processing time: 48ms (including acquisition, preprocessing, inference, and representation); Concurrent processing capability: 35 people can monitor simultaneously without frame loss or latency accumulation; System response time: ≤50ms, meeting real-time monitoring requirements; False detection rate: <1.2%; Missed detection rate: <0.8%.

[0130] S5.3: Model Adaptive Update Validation:

[0131] S5.3.1: Experimental environment configuration: run continuously for 3 months, collect 500 new samples each month (including the new violation type "illegally crossing the belt"), use incremental learning to fine-tune the model, set the learning rate to 0.0001, freeze the first 3 layers of the backbone network, train only the high-level feature extraction layer and classification layer, and fine-tune for 5 rounds per round.

[0132] S5.3.2: Update Results: Initial accuracy: 98.7%, After the first month update: 98.9% (96.2% accuracy for new type recognition), After the second month update: 99.0%, After the third month update: 98.8%.

[0133] Validation results show that incremental learning methods can effectively incorporate new knowledge while maintaining the memory of old knowledge, avoiding catastrophic forgetting, and ensuring the long-term stable operation of the system.

[0134] S5.4: Industrial Applicability Verification:

[0135] This system was deployed underground in a coal mine with an annual output of 5 million tons, covering eight key areas including the mining face, main haulage roadway, and central substation, and operated continuously for six months. Statistical results show that: a total of 12,450 violations were identified, with 12,280 accurately identified, achieving an accuracy rate of 98.6%; the average response time was 46ms, meeting real-time alarm requirements; it seamlessly integrated with the safety supervision platform, achieving a 100% success rate in pushing violation data; it reduced manual inspection workload by approximately 70% and improved the efficiency of safety hazard handling by approximately 65%.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for intelligent monitoring of underground coal mines based on data fusion, characterized in that, The specific steps are as follows: S1: Acquire environmental monitoring data and images of coal mine workers at work, wherein the environmental monitoring data includes dust concentration and light intensity data; S2: Set environmental monitoring data thresholds, determine whether the current environment is abnormal based on environmental monitoring data and environmental monitoring data thresholds, and record abnormal environmental data; S3: Extract features from images of coal mine workers at work, combine them with environmental monitoring data to identify violations by coal mine workers, and record the data on violations. S4: Transmit abnormal environmental data and personnel violation data to the remote platform, and the remote platform issues corresponding decision control signals and alarm signals; The specific steps in step S3 are as follows: S3.1: Sort the images of coal mine workers in operation according to time to form a continuous sequence of images of workers in operation, and perform multi-step preprocessing on the image sequence; S3.2: Extract skeleton features from the preprocessed personnel image to obtain the coordinates of key skeletal nodes and skeleton connection information; S3.3: Extract contour features from the preprocessed personnel images to obtain complete personnel contour information; S3.4: Construct a feature fusion module based on an adaptive weighted fusion algorithm, adjust the weights according to environmental monitoring data, and fuse the skeleton features and contour features of personnel through the adjusted feature fusion module to obtain a fused feature vector; S3.5: Construct a deep learning-based model for identifying violations, and train the model using a set of image samples of common violations by coal mine workers; S3.6: Input the fused feature vector into the trained recognition model, output the type of violation, the confidence level of the violation, and the location information of the violation, and record the violation data. The specific steps in step S3.4 are as follows: S3.4.1: Standardize the skeleton features and contour features to eliminate the influence of different feature dimensions; S3.4.2: Dynamically adjust the weighting coefficients of skeleton features and contour features based on environmental monitoring data; The weighting coefficient of the entire skeleton feature ranges from 0.55 to 0.65, and the weighting coefficient of the contour feature ranges from 0.35 to 0.

45. When the dust concentration is >50 mg / m³ or the light intensity is <50 lux, the weight of the contour feature is increased to 0.45, and the weight of the skeleton feature is decreased to 0.

55. When the light intensity is ≥200 lux and the dust concentration is ≤15 mg / m³, the weight of the skeleton feature is increased to 0.65, and the weight of the contour feature is decreased to 0.

35. S3.4.3: Using a weighted summation method, the standardized skeleton features and contour features are fused to obtain a fused feature vector, which includes personnel limb posture information and body shape information.

2. The intelligent monitoring method for underground coal mines based on data fusion according to claim 1, characterized in that, The specific steps in step S3.2 are as follows: S3.2.1: Embed the CBAM spatial attention module into the OpenPose network to form a skeleton extraction network based on the improved OpenPose. The skeleton extraction network includes an input layer, a feature extraction layer, an attention optimization layer, a skeleton node prediction layer, and a skeleton generation layer. S3.2.2: Input the preprocessed personnel image into the improved OpenPose network to obtain the coordinates of key skeletal nodes and skeleton connection information of the personnel; S3.2.3: The Kalman filter algorithm is used to smooth the coordinates of key skeletal nodes and remove abnormal skeletal nodes that deviate from the normal range; and the broken skeleton structure is repaired by verifying the connection relationship of skeletal nodes.

3. The intelligent monitoring method for underground coal mines based on data fusion according to claim 1, characterized in that, The specific steps in step S3.3 are as follows: S3.3.1: The Canny edge detection algorithm is adopted, and high and low thresholds are set to detect the real edges of the person image, remove false edges, and retain the contour edges of the person target; S3.3.2: A chain code-based contour tracking algorithm is used to track the detected contour edges, extract the closed contour of the person, and remove small burrs in the contour. S3.3.3: Calculate the core feature parameters of the contour and integrate the core feature parameters to form a personnel contour feature vector; the core feature parameters include contour perimeter, contour area, number of contour concave and convex points, contour rectangularity, contour circularity, and Hu invariant moments.

4. The intelligent monitoring method for underground coal mines based on data fusion according to claim 3, characterized in that, The specific steps in step S3.5 are as follows: S3.5.1: Using ResNet18 as the backbone network, the backbone network is pruned, and a global average pooling layer, a fully connected layer, and a softmax classification layer are added to the last layer of the backbone network to form a violation behavior recognition model. S3.5.2: The model is trained using a set of image samples of common violations by coal mine workers. The training process uses the cross-entropy loss function and the focus loss function to obtain the trained recognition model. The image sample set of common violations includes images of not wearing a safety helmet, not wearing a safety belt, climbing heights without permission, entering dangerous areas without authorization, leaving one's post, operating equipment in violation of regulations, working under the influence of alcohol, and working while fatigued.

5. A method for intelligent monitoring of underground coal mines based on data fusion according to claim 1 or 4, characterized in that, The specific steps in step S3.6 are as follows: S3.6.1: Input the fused feature vector into the trained recognition model, and output the type of violation, the confidence level of the violation, and the location information of the violation. S3.6.2: Set a confidence threshold to eliminate false identification results with a confidence level below the threshold; S3.6.3: Construct a characterization model of coal mine personnel violations, encapsulate the output of the characterization model into a standardized JSON format, and transmit the personnel violation data in JSON format to a remote platform.

6. The intelligent monitoring method for underground coal mines based on data fusion according to claim 5, characterized in that, The representation model includes a behavior type representation represented by violation behavior encoding and text description, a feature representation represented by fused feature vectors, a confidence representation represented by violation confidence and feature matching degree, and a location representation represented by the bounding box coordinates and center coordinates of the violator. The standardized JSON format includes a timestamp, camera device number, bounding box coordinates of the violator, center coordinates, violation type, behavior code, violation confidence, feature matching degree, fused feature vector, and environmental monitoring data.

7. The intelligent monitoring method for underground coal mines based on data fusion according to claim 1, characterized in that, Step S3 further includes step S3.7: On a monthly basis, collect image samples of new types and new scenarios of violations in the mine, incorporate the new image samples of violations using an incremental learning method, and fine-tune the violation recognition model and feature fusion module.

8. The intelligent monitoring method for underground coal mines based on data fusion according to claim 1, characterized in that, In step S4, the remote platform sends corresponding decision control signals and alarm signals, including: When the dust concentration exceeds the preset upper threshold, the remote platform sends a control signal to start the backup fan and begins sampling in the next cycle. When the dust concentration falls below the preset lower threshold, the backup fan stops. When the light intensity falls below the preset lower threshold, the remote platform sends a control signal to start the backup lighting equipment and begins sampling in the next cycle. When the light intensity rises above the preset upper threshold, the backup lighting equipment is turned off. When a violation is detected, an early warning is issued and the on-site management personnel are notified.

Images