Underground mine operation state analysis system and method based on video monitoring data

Abstract

The invention discloses an underground mine operation state analysis system and method based on video monitoring data, and the system comprises a data collection module which is used for collecting mine video and environment parameter data through a distributed sensor network, and generating a multi-dimensional data fusion set based on a space-time label technology; the edge analysis module is used for extracting feature parameters through a convolutional neural network algorithm based on the multi-dimensional data fusion set and generating a mine operation state recognition result; the fence construction module is used for constructing a three-dimensional digital model and a dynamic safety boundary based on the mine operation state recognition result to form a real-time monitoring reference framework; and the decision execution module is used for performing hierarchical risk assessment on the monitoring data in the security boundary based on the real-time monitoring reference framework, and generating a security early warning and disposal scheme with a tracing identifier. Each piece of early warning and disposal information is attached with a unique tracing identification code, so that follow-up event backtracking analysis is facilitated, and the risk management and control capability is continuously improved.

Description

Technical Field

[0001] This invention relates to the field of video surveillance technology, and in particular to an underground mine operation status analysis system and method based on video surveillance data. Background Technology

[0002] With the continuous expansion of mining scale and the increase in mining depth, underground mine safety supervision faces increasingly severe challenges. Traditional mine monitoring mainly relies on manual inspections and single sensor monitoring, which suffers from problems such as incomplete monitoring coverage, poor real-time performance, and isolated data.

[0003] In existing technologies, most mine video surveillance systems employ simple image acquisition and storage methods, lacking intelligent analysis capabilities. They cannot automatically identify operational status and abnormal behavior, requiring extensive manual monitoring, resulting in low efficiency and a high risk of missing critical information. While personnel positioning systems can provide location information, they operate independently of video surveillance and environmental monitoring systems, exhibiting low data integration and hindering comprehensive safety situational awareness. Traditional electronic fence systems, primarily based on two-dimensional planar designs, are ill-suited to the complex three-dimensional spatial structures of underground mines, failing to accurately reflect the boundary constraints of tunnels, mining areas, and other three-dimensional zones. Existing early warning mechanisms often rely on fixed threshold triggers, lacking dynamic risk assessment capabilities based on operational status and environmental changes, leading to false alarms or missed alarms. Furthermore, current mine monitoring systems primarily depend on cloud computing for data processing, with network latency and bandwidth limitations impacting real-time response capabilities. The lack of effective collaboration mechanisms between subsystems hinders unified safety management and emergency response.

[0004] Therefore, there is an urgent need for an underground mine operation status analysis system and method based on video surveillance data. Summary of the Invention

[0005] This invention provides a system and method for analyzing the operational status of underground mines based on video surveillance data, in order to solve the aforementioned problems existing in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A system for analyzing the operational status of underground mines based on video surveillance data includes:

[0008] The data acquisition module is used to collect mine video and environmental parameter data through a distributed sensor network, and generate a multi-dimensional data fusion set based on spatiotemporal tagging technology;

[0009] The edge analysis module is used to extract feature parameters based on multi-dimensional data fusion sets and generate mine operation status identification results through convolutional neural network algorithms;

[0010] The fence construction module is used to build a three-dimensional digital model and dynamic safety boundary based on the mining operation status identification results, forming a real-time monitoring reference framework;

[0011] The decision execution module is used to perform hierarchical risk assessment on monitoring data within the security boundary based on a real-time monitoring reference framework, and generate security early warning and response plans with traceability identifiers.

[0012] The data acquisition module includes:

[0013] The video acquisition submodule is used to acquire video stream data of the mining area under full illumination conditions through wide dynamic range camera equipment and infrared sensors;

[0014] The environmental monitoring submodule is used to collect data on changes in human vital signs, environmental gas concentrations, and electromagnetic field strength.

[0015] The data transmission submodule is used to transmit mining area video stream data and environmental monitoring data to edge computing nodes for storage via a software-defined heterogeneous communication network.

[0016] The edge analysis module includes:

[0017] The image enhancement submodule is used to apply physically based lighting compensation and contrast adjustment algorithms to video data to improve image recognizability.

[0018] The target recognition submodule is used to identify and track the equipment location and work behavior characteristics of workers based on image-enhanced video data using a convolutional neural network architecture.

[0019] The status analysis submodule is used to determine the operating status of equipment and the working status of personnel based on the target recognition results and combined with environmental parameter data, and generate mining operation status recognition results.

[0020] The fence construction module includes:

[0021] The 3D modeling submodule is used to integrate point cloud data collected by lidar, vibration waveform data collected by geological monitoring equipment, and engineering design drawings to construct a 3D model of the mine with geological structural attributes.

[0022] The fence configuration submodule is used to receive the 3D model and generate a multi-level nested safety zone boundary based on the geological stability index calculation and the national mine safety regulations.

[0023] The spatial relationship analysis submodule is used to calculate the spatial positional relationship between the target and the safety boundary based on the safety zone boundary and the mine operation status identification results, using graph theory algorithms to generate target safety status data, and integrate it with safety boundary information to form a real-time monitoring reference framework.

[0024] The video capture submodule includes:

[0025] The exposure control unit is used to dynamically adjust the exposure parameters of the camera device based on ambient light intensity information and the scene brightness distribution histogram to ensure image clarity under both low light and strong light conditions.

[0026] The multispectral imaging unit is used to simultaneously acquire image data in the visible light band, near-infrared band, and thermal imaging band to identify the material properties and temperature distribution characteristics of different objects.

[0027] The image fusion unit is used to perform image registration based on multi-band image data through a feature point matching algorithm, and to synthesize multi-source images using a weighted averaging method to generate information-enhanced video stream data.

[0028] The state analysis submodule includes:

[0029] The feature extraction unit is used to extract personnel action sequences and equipment motion parameters from the target recognition results;

[0030] The state recognition unit is used to identify the type of work behavior based on the sequence of personnel actions and the motion parameters of equipment through a pre-trained long short-term memory network model;

[0031] The anomaly detection unit is used to compare the identified work behavior with the standard operating procedure template, calculate the degree of deviation, determine whether the work complies with safety regulations, and integrate the judgment results into the mine operation status identification results.

[0032] The spatial relationship analysis submodule includes:

[0033] The hybrid positioning unit is used to fuse wireless signal positioning data, inertial measurement unit data, and visual feature point reference data, and generates interference-resistant personnel and equipment position data through a particle filtering algorithm.

[0034] The path planning unit is used to calculate the optimal evacuation path considering dynamic obstacles based on a 3D model and safety zone boundaries, using an improved random tree optimization algorithm.

[0035] The behavior prediction unit is used to analyze the movement trend based on the target's historical trajectory data through a long short-term memory network, predict the future location, calculate the probability of approaching dangerous areas, generate a behavior risk score, and trigger graded early warning signals.

[0036] The decision execution module includes:

[0037] The situation construction submodule is used to integrate personnel location, equipment status and safety boundary data onto the timeline based on a real-time monitoring reference framework, and build a digital twin model of the mining operation environment that supports time backtracking and extrapolation.

[0038] The risk assessment submodule is used to analyze the probability and severity of various security incidents based on digital twin models and Bayesian network technology, and generate a risk level distribution heatmap.

[0039] The early warning and response submodule is used to generate graded early warning information based on the risk level distribution heatmap and preset thresholds, and to extract corresponding response plans from the emergency plan database to form safety early warning and response instructions that include execution steps and responsible persons.

[0040] The early warning and response submodule includes:

[0041] The pattern recognition unit is used to match the current risk status with the historical case database and identify dangerous patterns within a preset range;

[0042] The plan generation unit is used to extract corresponding handling steps from the contingency plan library based on the identified hazard patterns, and to optimize and adjust them in combination with the current personnel location and equipment status;

[0043] The execution tracking unit is used to distribute the response plan to relevant terminal devices, monitor the execution process, and record the timestamps and execution status of key actions to form a traceable emergency response record.

[0044] One method for analyzing the operational status of underground mines based on video surveillance data includes:

[0045] S1: Collect mine video and environmental parameter data through a distributed sensor network, and generate a multi-dimensional data fusion set based on spatiotemporal tagging technology;

[0046] S2: Based on a multi-dimensional data fusion set, feature parameters are extracted using a convolutional neural network algorithm to generate mine operation status identification results;

[0047] S3: Based on the results of mine operation status identification, construct a three-dimensional digital model and dynamic safety boundary to form a real-time monitoring reference framework;

[0048] S4: Based on the real-time monitoring reference framework, perform hierarchical risk assessment on monitoring data within the security boundary, and generate security early warning and response plans with traceability identification.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] A system for analyzing the operational status of underground mines based on video surveillance data includes: a data acquisition module for collecting mine video and environmental parameter data through a distributed sensor network and generating a multi-dimensional data fusion set based on spatiotemporal tagging technology; an edge analysis module for extracting feature parameters from the multi-dimensional data fusion set using a convolutional neural network algorithm to generate mine operational status identification results; a fence construction module for constructing a three-dimensional digital model and dynamic safety boundary based on the mine operational status identification results, forming a real-time monitoring reference framework; and a decision execution module for performing hierarchical risk assessment on monitoring data within the safety boundary based on the real-time monitoring reference framework, generating safety warnings and response plans with traceability identifiers. Each warning and response message is accompanied by a unique traceability identifier code, facilitating subsequent event retrospective analysis and continuous improvement of risk management capabilities.

[0051] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention.

[0052] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a structural diagram of an underground mine operation status analysis system based on video surveillance data, as described in an embodiment of the present invention.

[0055] Figure 2 This is a structural diagram of the data acquisition module in an embodiment of the present invention;

[0056] Figure 3 This is a flowchart of a method for analyzing the operational status of underground mines based on video surveillance data, as described in an embodiment of the present invention. Detailed Implementation

[0057] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0058] This invention provides an underground mine operation status analysis system based on video surveillance data, comprising:

[0059] The data acquisition module is used to collect mine video and environmental parameter data through a distributed sensor network, and generate a multi-dimensional data fusion set based on spatiotemporal tagging technology;

[0060] The edge analysis module is used to extract feature parameters based on multi-dimensional data fusion sets and generate mine operation status identification results through convolutional neural network algorithms;

[0061] The fence construction module is used to build a three-dimensional digital model and dynamic safety boundary based on the mining operation status identification results, forming a real-time monitoring reference framework;

[0062] The decision execution module is used to perform hierarchical risk assessment on monitoring data within the security boundary based on a real-time monitoring reference framework, and generate security early warning and response plans with traceability identifiers.

[0063] The working principle and beneficial effects of the above technical solution are as follows: The data acquisition module employs a distributed sensor network deployed at key locations throughout the mine, including roadways, working faces, and electromechanical equipment areas. The data collected by the sensors includes high-definition video streams, personnel location information, ambient gas concentrations, temperature, humidity, and other parameters. The system uses spatiotemporal tagging technology to add precise timestamps and spatial coordinates to each data point, achieving spatiotemporal correlation of the data. For example, when an anomaly occurs at the working face, the system can correlate video data from the same time and location with environmental parameter data, forming a multi-dimensional data fusion set, providing a comprehensive information foundation for subsequent analysis.

[0064] The edge analysis module is deployed on edge computing nodes close to the data source, employing improved convolutional neural network algorithms (such as ResNet-50 or YOLOv5) to process the multidimensional data fusion set. First, image preprocessing is performed, including noise removal and illumination equalization. Then, feature parameters are extracted, including personnel posture features, equipment motion features, and environmental change features. Finally, a trained deep learning model is used for state classification, identifying different states such as normal operation, equipment malfunction, and personnel violations. The model uses transfer learning, pre-training on a public dataset and then fine-tuning with mining-specific scenario data, achieving an accuracy rate of over 95%.

[0065] The fence construction module, based on the mine operation status identification results and combined with mine engineering drawings and geological exploration data, uses spatial information technology to construct a three-dimensional digital model of the mine. On this basis, according to national mine safety regulations and real-time monitoring data, it dynamically divides areas into different safety levels, forming a multi-level nested safety boundary. For example, areas with excessive gas concentrations are marked as red restricted areas, areas with unstable roof support are marked as yellow warning areas, and normal operating areas are marked as green safe areas. The system uses 3D visualization technologies such as WebGL to present these safety boundaries in real time on the monitoring interface, forming an intuitive monitoring reference framework.

[0066] The decision-making and execution module, based on a real-time monitoring reference framework, performs hierarchical risk assessments on monitoring data within the safety boundary. The risk assessment employs a combination of the Analytic Hierarchy Process (AHP) and fuzzy comprehensive evaluation, comprehensively considering multiple dimensions such as environmental risk, personnel risk, and equipment risk. When a risk exceeds a preset threshold, the system automatically generates a safety warning and, based on the risk type and level, invokes a pre-set response plan library to generate a specific response plan including response steps, responsible personnel, and time requirements. Each warning and response message is accompanied by a unique traceability identifier, facilitating subsequent event retrospective analysis and continuous improvement of risk management capabilities.

[0067] In another embodiment, the data acquisition module includes:

[0068] The video acquisition submodule is used to acquire video stream data of the mining area under full illumination conditions through wide dynamic range camera equipment and infrared sensors;

[0069] The environmental monitoring submodule is used to collect data on changes in human vital signs, environmental gas concentrations, and electromagnetic field strength.

[0070] The data transmission submodule is used to transmit mining area video stream data and environmental monitoring data to edge computing nodes for storage via a software-defined heterogeneous communication network.

[0071] The working principle and beneficial effects of the above technical solution are as follows: The video acquisition submodule deploys wide dynamic range (WDR) cameras (dynamic range ≥ 120dB) and infrared sensors at key locations in the mining area. The WDR cameras employ dual-exposure technology, enabling simultaneous capture of details in both bright and dark areas, thus solving the problem of uneven mine lighting; the infrared sensors (operating wavelength 8-14μm) can capture thermal imaging data in completely dark environments. The system is configured with different equipment combinations based on the characteristics of the mining area's location. For example, high-definition cameras and infrared thermal imagers are installed simultaneously at the coal face to form a complementary system; WDR cameras are mainly deployed in personnel passages; and low-light cameras (minimum illumination 0.001 lux) are installed near key equipment. The video data acquisition resolution is 1080p, with a frame rate of no less than 25fps to ensure image clarity and the ability to capture moving targets.

[0072] The environmental monitoring submodule deploys various sensors to collect environmental and personnel parameters. Personnel vital signs are collected via wearable devices, including physiological parameters such as heart rate, blood oxygen, and body temperature, and transmitted using Bluetooth Low Energy (BLE) technology. Environmental gas concentrations are collected using specialized mining sensors such as catalytic combustion sensors (methane), electrochemical sensors (carbon monoxide, hydrogen sulfide), and infrared absorption sensors (carbon dioxide), with detection accuracy conforming to the AQ1029-2019 standard. Electromagnetic field intensity changes are monitored using electric field strength meters and magnetic field strength meters, recording the electromagnetic radiation level of the working environment in real time. The monitoring frequency range covers 10Hz-10GHz, and the data recording frequency is set to once every 10 seconds, automatically increasing to once per second in abnormal situations.

[0073] The data transmission submodule is based on a software-defined heterogeneous communication network architecture, integrating multiple communication methods including wired (industrial Ethernet) and wireless (WiFi, ZigBee, 4G / 5G private networks). The system dynamically selects the optimal transmission path and protocol based on data importance and network conditions to ensure data transmission reliability. For example, video stream data is preferentially transmitted via industrial Ethernet or 5G networks; environmental parameter data is transmitted via a ZigBee network in a mesh topology, ensuring network operation even when some nodes fail. Data transmission employs a tiered caching strategy, with local storage (SSD, capacity ≥1TB) at edge nodes, supporting temporary data storage (≥72 hours) during network outages, and automatic synchronization upon network recovery. All transmitted data is protected by AES-256 encryption to ensure data security.

[0074] In another embodiment, the edge analysis module includes:

[0075] The image enhancement submodule is used to apply physically based lighting compensation and contrast adjustment algorithms to video data to improve image recognizability.

[0076] The target recognition submodule is used to identify and track the equipment location and work behavior characteristics of workers based on image-enhanced video data using a convolutional neural network architecture.

[0077] The status analysis submodule is used to determine the operating status of equipment and the working status of personnel based on the target recognition results and combined with environmental parameter data, and generate mining operation status recognition results.

[0078] The working principle and beneficial effects of the above technical solution are as follows: The image enhancement submodule addresses video quality issues such as low illumination and high contrast in mining areas by implementing image processing based on a physical model. First, a multi-scale algorithm based on Retinex theory is applied to separate the illumination and reflection components, adaptively enhancing low-illumination areas. Then, a contrast adjustment algorithm based on the statistical characteristics of natural scenes is used to remap the image histogram, enhancing detail visibility. Finally, bilateral filtering technology is combined to reduce noise introduced during enhancement, maintaining edge clarity. The algorithm dynamically adjusts parameters according to the characteristics of image regions; for example, it uses a gradual enhancement coefficient in illumination transition areas to avoid obvious processing artifacts. The system's processing time for each frame is controlled within 20ms, meeting real-time processing requirements. The signal-to-noise ratio of the enhanced image is improved by more than 30%, providing high-quality input for subsequent target recognition.

[0079] The target recognition submodule employs an improved convolutional neural network architecture for the detection and tracking of specific targets in the mining area. The network backbone uses the MobileNetV3 structure to balance computational efficiency and recognition accuracy; the detection head uses the SSD algorithm to support multi-scale target detection. First, the target detection algorithm identifies targets such as miners, mining equipment, and transport vehicles, outputting target categories and location coordinates. Then, the DeepSORT algorithm is used for target tracking, assigning a unique ID to each target and constructing temporal associations. Finally, a pose estimation network (such as OpenPose) analyzes personnel poses, and a spatiotemporal behavior template library is used to determine work behavior characteristics, such as normal operation, violations, and emergency avoidance. The system achieves real-time processing capability of 30fps by deploying NVIDIA T4 or higher-performance GPUs on edge servers, with a personnel recognition accuracy of 97% and a behavior classification accuracy of 92%.

[0080] The status analysis submodule determines the operating status of equipment and the operational status of personnel based on target identification results and environmental parameter data. The system employs a hierarchical Bayesian network model to integrate multi-source information and construct a framework for analyzing the correlation between equipment, personnel, and the environment. Equipment status assessment identifies normal operation, abnormal vibration, and overheating by analyzing parameters such as equipment vibration characteristics, temperature distribution, and motion trajectory, combined with preset equipment health index thresholds. Personnel operational status assessment identifies standardized operation, violation of regulations, and fatigued operation by analyzing operational behavior sequences, position changes, and vital sign data, combined with operational procedure requirements. The system uses a time window sliding analysis method, generating a status assessment result every 5 seconds, and filters short-term fluctuations using a time-series smoothing algorithm to improve judgment stability. The final generated mine operation status identification result includes information such as object ID, location coordinates, current status, and status confidence level, providing data support for safety management decisions.

[0081] In another embodiment, the fence construction module includes:

[0082] The 3D modeling submodule is used to integrate point cloud data collected by lidar, vibration waveform data collected by geological monitoring equipment, and engineering design drawings to construct a 3D model of the mine with geological structural attributes.

[0083] The fence configuration submodule is used to receive the 3D model and generate a multi-level nested safety zone boundary based on the geological stability index calculation and the national mine safety regulations.

[0084] The spatial relationship analysis submodule is used to calculate the spatial positional relationship between the target and the safety boundary based on the safety zone boundary and the mine operation status identification results, using graph theory algorithms to generate target safety status data, and integrate it with safety boundary information to form a real-time monitoring reference framework.

[0085] The working principle and beneficial effects of the above technical solution are as follows: The 3D modeling submodule integrates multi-source data to construct a 3D model of the mine. First, a vehicle-mounted or handheld LiDAR (point cloud density ≥ 100 points / m²) is used. 2 The system collects spatial structure data of mine roadways and stopes; secondly, it collects rock strata vibration waveform data using geological monitoring equipment such as seismographs and microseismic monitoring systems, covering a frequency range of 1-100Hz; finally, it imports mine engineering design drawings in CAD format and geological data obtained from drilling. The system uses point cloud registration technology to stitch together multiple scans of data into a complete model, and optimizes the model surface through NURBS surface fitting; then, it analyzes geological structures based on seismic wave inversion algorithms, identifying weak zones such as faults and fissures; finally, it maps geological attributes (such as lithology, strength, and water content) onto the 3D model, forming a 3D mine model with geological structural attributes. The model update cycle is dynamically adjusted according to the mining progress, with rapid advancement areas updated weekly and stable areas updated monthly.

[0086] The fence configuration submodule constructs safety zone boundaries based on a 3D model. First, it calculates the geological stability index, comprehensively considering factors such as roof integrity, fault distribution, and stress concentration, and uses an improved Rock Mass Quality (RMR) method to quantitatively assess the stability of each area. Second, it determines the safety parameter requirements for different types of areas by combining national standards such as the "Coal Mine Safety Regulations" and the "Metal and Non-metal Mine Safety Regulations." Finally, based on the stability assessment results and safety regulations requirements, a multi-level nested safety boundary is constructed. The system typically divides safety zones into three levels: red areas are prohibited areas (e.g., roof fall hazard zones, high-gas zones), yellow areas are restricted work areas (e.g., support and reinforcement zones, poorly ventilated zones), and green areas are normal work areas. The safety boundaries are stored in a 3D grid format, with each grid cell (0.5m side length) containing attribute information such as location coordinates, safety level, and restrictions, supporting real-time updates and queries.

[0087] The spatial relationship analysis submodule analyzes the spatial positional relationship between targets and safety boundaries based on the identification results of safety zone boundaries and mine operation status. The system uses an improved Dijkstra algorithm to construct a mine spatial topology map, where nodes represent key locations, edges represent passable paths, and edge weights represent the difficulty of passage. For each identified target (such as personnel or equipment), the system calculates in real time its relative position to safety boundaries at all levels and the shortest evacuation path. Spatial relationship determination rules include: a target completely within the safety zone is considered "safe"; a target entering the warning zone is considered "warning"; and a target entering a restricted area or within a safe distance of the danger boundary (personnel ≤ 5m, equipment ≤ 10m) is considered "dangerous". The system further applies graph theory algorithms to analyze the spatial distribution characteristics of multi-target groups, identify group behavior patterns such as personnel gathering and dispersal, and integrates these with safety boundaries to form a real-time monitoring reference framework that includes information such as target status and surrounding environmental risks.

[0088] In another embodiment, the video acquisition submodule includes:

[0089] The exposure control unit is used to dynamically adjust the exposure parameters of the camera device based on ambient light intensity information and the scene brightness distribution histogram to ensure image clarity under both low light and strong light conditions.

[0090] The multispectral imaging unit is used to simultaneously acquire image data in the visible light band, near-infrared band, and thermal imaging band to identify the material properties and temperature distribution characteristics of different objects.

[0091] The image fusion unit is used to perform image registration based on multi-band image data through a feature point matching algorithm, and to synthesize multi-source images using a weighted averaging method to generate information-enhanced video stream data.

[0092] The working principle and beneficial effects of the above technical solution are as follows: The exposure control unit first obtains the current ambient light intensity value through a photosensitive sensor, and simultaneously extracts a brightness distribution histogram from the raw image captured by the camera device. The system analyzes the histogram, calculating the image brightness mean, standard deviation, and the proportion of overexposed and underexposed areas. When underexposure is detected (the peak on the left side of the histogram is too high), the system appropriately increases the exposure time and ISO sensitivity, and implements regional gamma correction; when overexposure is detected (the peak on the right side of the histogram is too high), the system shortens the exposure time and activates high dynamic range (HDR) mode. Exposure parameter adjustment uses a PID control algorithm to ensure smooth parameter changes and avoid image flicker. In areas with alternating strong and weak light, such as alleyways, the system employs zone exposure technology, applying different exposure parameters to different areas, and finally synthesizes a balanced image through software algorithms.

[0093] The multispectral imaging unit consists of three image sensors: a visible light band (400-700nm) sensor using CMOS technology with a resolution of 1920×1080, primarily used to capture the appearance features of personnel and equipment; a near-infrared band (780-1100nm) sensor using enhanced InGaAs material, which has unique absorption characteristics for minerals and moisture, allowing it to penetrate dust and light smoke, providing clearer object outlines; and a thermal imaging band (8-14μm) sensor using an uncooled microbolometer array with a resolution of 640×480 and a temperature resolution of 0.05℃, capable of identifying overheated areas of equipment, personnel locations, and areas of abnormal rock temperature. The multispectral imaging unit employs a coaxial optical design to ensure consistent field of view for the three bands, and a synchronous trigger controller ensures time synchronization of the three image sets, with a sampling rate of 25 frames per second.

[0094] The image fusion unit first preprocesses the multi-band images, including lens distortion correction, brightness normalization, and noise suppression. Then, it uses the SIFT (Scale Invariant Eigentransform) algorithm to extract feature points from each band image and employs the RANSAC (Random Sample Consensus) algorithm to remove mismatched points, establishing accurate inter-band correspondences. Based on these correspondences, the system uses a projection transformation matrix to register each band image to a unified coordinate system. The fusion process uses a weighted average method, with weight coefficients adaptively adjusted based on scene type: in high-dust environments, near-infrared image weights are increased; in equipment monitoring areas, thermal imaging weights are increased; and in areas with personnel activity, visible light image weights are increased. Finally, the system uses a wavelet transform domain fusion algorithm to retain high-frequency detail information from each band image, generating an information-enhanced video stream that simultaneously contains rich color information, material properties, and temperature distribution characteristics.

[0095] In another embodiment, the state analysis submodule includes:

[0096] The feature extraction unit is used to extract personnel action sequences and equipment motion parameters from the target recognition results;

[0097] The state recognition unit is used to identify the type of work behavior based on the sequence of personnel actions and the motion parameters of equipment through a pre-trained long short-term memory network model;

[0098] The anomaly detection unit is used to compare the identified work behavior with the standard operating procedure template, calculate the degree of deviation, determine whether the work complies with safety regulations, and integrate the judgment results into the mine operation status identification results.

[0099] The working principle and beneficial effects of the above technical solution are as follows: The feature extraction unit processes the target recognition results and extracts key point coordinates from the detected personnel targets, including 17 feature points such as the head, torso, and joints of the limbs. The system uses these feature points to construct a skeleton model and calculates the displacement vectors and angular changes of each joint point between adjacent frames to form a temporal feature sequence. For personnel actions, the system records basic action elements such as raising the arm, bending over, and squatting, as well as their duration and transition states. For equipment targets, the system extracts the coordinate changes of the bounding box center point, the rate of area change, and the contour deformation parameters, and calculates the motion speed, acceleration, and angular velocity of each component of the equipment using optical flow. During the feature extraction process, the system uses a Kalman filter to eliminate tracking jitter and uses a sliding window method (window size of 30 frames, step size of 10 frames) to segment continuous actions into basic action units for subsequent analysis.

[0100] The state recognition unit employs a bidirectional long short-term memory (Bi-LSTM) network model to process temporal feature data. This network consists of an input layer, two bidirectional LSTM layers (each with 128 hidden units), and a fully connected output layer. Input features include the personnel joint angle sequence, torso posture changes, and equipment motion parameters extracted in the previous step, which are normalized before being fed into the network. The model is pre-trained on a labeled dataset containing 10 categories of standard operational behaviors, including mining, support, maintenance, and loading / unloading, with approximately 500 sample sequences for each category. The network identifies the current behavior type and its confidence level by analyzing the temporal features and contextual relationships of the action sequences. To adapt to individual differences, the system uses transfer learning and few-shot fine-tuning techniques to update the model online, achieving a recognition accuracy of over 92%. The recognition results include the job type, duration, key action frame timestamps, and operation sequence.

[0101] The anomaly detection unit maintains a standard operating procedure knowledge base, containing standard action sequences, temporal relationships, spatial constraints, and safety requirements for various operations. The system performs Dynamic Time Warping (DTW) distance calculations between identified operational behaviors and their corresponding standard procedure templates, quantifying the degree of deviation between the operational behavior and the standard procedure. Specifically, the system checks whether the action sequence conforms to regulations (e.g., support before operation), whether the action posture is standardized (e.g., back straight when lifting heavy objects), whether the operation time is reasonable (e.g., insufficient drilling depth due to short drilling time), and whether human-machine collaboration is safe (e.g., whether personnel are within the safe operating area of ​​the equipment). The system sets thresholds for each indicator; when a deviation exceeds the threshold, it is marked as an anomaly and categorized into three levels based on severity: alert, warning, and severe. All analysis results are integrated into a mine operation status identification report, including anomaly timestamps, type descriptions, risk ratings, and improvement suggestions, supporting supervisors in quickly locating problems.

[0102] In another embodiment, the spatial relationship analysis submodule includes:

[0103] The hybrid positioning unit is used to fuse wireless signal positioning data, inertial measurement unit data, and visual feature point reference data, and generates interference-resistant personnel and equipment position data through a particle filtering algorithm.

[0104] The path planning unit is used to calculate the optimal evacuation path considering dynamic obstacles based on a 3D model and safety zone boundaries, using an improved random tree optimization algorithm.

[0105] The behavior prediction unit is used to analyze the movement trend based on the target's historical trajectory data through a long short-term memory network, predict the future location, calculate the probability of approaching dangerous areas, generate a behavior risk score, and trigger graded early warning signals.

[0106] The working principle and beneficial effects of the above technical solution are as follows: The hybrid positioning unit first collects multi-source positioning data, including: wireless signal positioning data provided by UWB (Ultra-Wideband) base stations deployed in the tunnel, with an accuracy of approximately ±0.3 meters; acceleration, angular velocity, and orientation data provided by IMUs (Inertial Measurement Units) carried by personnel and equipment; and visual feature point reference data extracted by the video surveillance system, including corner points, texture features, and pre-calibrated reference markers. The system uses a particle filtering algorithm to fuse these heterogeneous data, initializing 1000 weighted particles distributed in the possible location areas. After receiving new measurement data each time, the particle weights are updated according to the observation model, and particle degradation is avoided through resampling. This algorithm is specifically optimized for robustness against electromagnetic interference and metal reflection environments in mines. By setting an adaptive noise model and an outlier detection mechanism, sensor data with severe interference is eliminated. The system's positioning accuracy reaches ±0.5 meters in complex tunnel environments, with an update frequency of 10Hz, meeting the real-time positioning needs of personnel and equipment.

[0107] The path planning unit operates based on a pre-built 3D mine model and real-time updated safety zone boundary data. The 3D model contains static information such as the tunnel network topology, slope, and width, while the safety zone boundary data reflects dynamic factors such as ventilation conditions and roof status. The system employs an improved RRT* (Fast Random Search Tree) algorithm for path planning, which adds the following improvements to the standard RRT*: first, it introduces a heuristic sampling strategy to increase the sampling probability towards the exit direction; second, it adds path smoothing to reduce sharp-angle turns; and third, it incorporates a dynamic obstacle avoidance mechanism to reserve a safe distance for moving equipment. During the planning process, the system assigns a safety weight to each node on the path, with the weight influenced by factors such as ventilation conditions, lighting conditions, support quality, and water accumulation. When the system detects changes in the status of a path node (such as the addition of an obstacle), it triggers incremental replanning, updating only the affected path segments to avoid computational delays caused by global recalculation. The final generated evacuation path is characterized by high safety, smoothness, and strong adaptability.

[0108] The behavior prediction unit continuously tracks the historical trajectory data of the target, maintaining a timestamped location sequence for each target. The system uses a sliding window method to extract the trajectory data of the most recent 30 seconds and inputs it into a pre-trained LSTM network. This network contains two LSTM layers (each with 64 hidden units) and a fully connected output layer, outputting location predictions for the next 5, 10, and 15 seconds. The prediction results are spatially analyzed with known hazardous areas (such as tunnel collapse risk areas, equipment operation hazard areas, gas accumulation areas, etc.), calculating the nearest distance between the target and each hazardous area and its changing trend. Based on distance thresholds and approach speed, the system assesses the probability of a hazardous event and generates a behavior risk score of 0-100. When the score exceeds the set threshold, the system triggers a three-level early warning mechanism: a score of 60-75 is a mild warning, with an audible alert issued through the device worn by the target; 75-90 is a moderate warning, with a warning issued through the broadcast system and display screen; and a score of 90 or above is a severe warning, simultaneously notifying safety supervisors and initiating emergency response procedures. The early warning signal includes the target ID, location coordinates, risk type, and estimated risk occurrence time.

[0109] In another embodiment, the decision execution module includes:

[0110] The situation construction submodule is used to integrate personnel location, equipment status and safety boundary data onto the timeline based on a real-time monitoring reference framework, and build a digital twin model of the mining operation environment that supports time backtracking and extrapolation.

[0111] The risk assessment submodule is used to analyze the probability and severity of various security incidents based on digital twin models and Bayesian network technology, and generate a risk level distribution heatmap.

[0112] The early warning and response submodule is used to generate graded early warning information based on the risk level distribution heatmap and preset thresholds, and to extract corresponding response plans from the emergency plan database to form safety early warning and response instructions that include execution steps and responsible persons.

[0113] The working principle and beneficial effects of the above technical solution are as follows: The situation construction submodule first establishes a time-series data storage structure, using a time-series database to store personnel location coordinates (accuracy 0.5 meters), equipment operating parameters (including speed, load, temperature, etc.), and safety boundary change records. The data is indexed according to Unix timestamps, supporting query and playback with millisecond-level time precision.

[0114] The digital twin model uses the Unity3D engine for visualization and rendering, mapping real-time monitoring data onto a 3D scene. Personnel are represented by dynamic cylinders, with colors changing dynamically according to safety status (green - safe, yellow - warning, red - danger). Equipment is represented by detailed 3D models, displaying its operating status and key parameters in real time. The system updates the scene every 100 milliseconds and retains historical data from the last 72 hours for retrospective analysis.

[0115] The risk assessment submodule constructs a Bayesian network with 15 nodes, including: personnel location risk, equipment failure probability, geological disaster likelihood, gas concentration, dust concentration, and ventilation status. The network structure is determined based on expert knowledge and historical accident data, and the conditional probability table is learned from 5000 historical records using the maximum likelihood estimation method.

[0116] Risk calculation employs a variable elimination algorithm for Bayesian inference. Inputting current observed variables (such as measured methane concentration of 0.8% and personnel density of 3 people / 100 square meters), the posterior probability of various accidents is calculated. The calculation results are then used to divide the mining area into 50m × 50m grids. The comprehensive risk value of each grid is rendered using color, creating an intuitive heatmap. Risk values ​​of 0-0.2 are displayed in blue, 0.2-0.4 in green, 0.4-0.6 in yellow, 0.6-0.8 in orange, and 0.8-1.0 in red.

[0117] The early warning and response submodule sets four levels of warning thresholds: blue warning (risk value 0.3-0.5), yellow warning (0.5-0.7), orange warning (0.7-0.9), and red warning (>0.9). When the risk value of any grid exceeds the threshold, the system generates a warning message within 200 milliseconds, including the risk type, location coordinates, impact range, and recommended measures. Response plans are extracted from a contingency plan library containing standard response procedures for 12 types of accidents, including gas exceeding limits, roof collapse, flooding, and fire. Each procedure contains 5-8 execution steps, clearly defining responsible personnel and completion deadlines. Based on the current personnel distribution and equipment status, the system uses the A* path planning algorithm to calculate the optimal evacuation route and simultaneously issues instructions via SMS, voice broadcast, and the underground communication system.

[0118] In another embodiment, the early warning and handling submodule includes:

[0119] The pattern recognition unit is used to match the current risk status with the historical case database and identify dangerous patterns within a preset range;

[0120] The plan generation unit is used to extract corresponding handling steps from the contingency plan library based on the identified hazard patterns, and to optimize and adjust them in combination with the current personnel location and equipment status;

[0121] The execution tracking unit is used to distribute the response plan to relevant terminal devices, monitor the execution process, and record the timestamps and execution status of key actions to form a traceable emergency response record.

[0122] The working principle and beneficial effects of the above technical solution are as follows: The pattern recognition unit maintains a historical accident database containing 500 typical cases. Each case includes a multi-dimensional feature vector containing environmental parameter sequences, personnel distribution, equipment status, etc., from the 30 minutes prior to the accident. The system uses the Dynamic Time Warping (DTW) algorithm to calculate the similarity between the current state sequence and historical cases. The similarity calculation considers eight key dimensions, including the rate of change of gas concentration, personnel concentration, and abnormal equipment indicators. When the similarity exceeds 0.85, the system determines a successful match and extracts the corresponding hazard mode label, including gas accumulation type, roof instability type, and electromechanical failure type. For cases with similarity between 0.7 and 0.85, the system uses the K-nearest neighbor algorithm to select the three most similar cases and determines the most likely hazard mode through weighted voting.

[0123] The scheme generation unit retrieves corresponding schemes from a database containing 200 standard contingency plans based on the identified hazard patterns. The standard contingency plans are stored in XML format and include four parts: preconditions, execution steps, resource requirements, and expected results. The system first verifies whether the preconditions are met, such as "gas concentration > 1%" and "number of personnel > 10 people." The optimization process considers actual constraints: the shortest path from each person to the safety exit is calculated using Dijkstra's algorithm, with a path length not exceeding 500 meters; the arrival time at the designated location is estimated based on the current location and speed of the equipment; and channel capacity limitations are considered, ensuring that the number of people passing through each tunnel simultaneously does not exceed 80% of the design capacity. The optimized scheme includes specific personnel evacuation sequences, equipment shutdown sequences, and ventilation adjustment parameters.

[0124] The execution tracking unit pushes handling instructions to various underground terminal devices, including personnel positioning cards, equipment controllers, and environmental monitoring instruments, via the WebSocket protocol. Each instruction includes a unique identifier, execution content, target equipment ID, and timeout period. Upon receiving an instruction, the terminal device returns confirmation information within 500 milliseconds. The system monitors the execution progress of each measure in real time, recording timestamps of key nodes: instruction issuance time, equipment response time, personnel start moving time, and arrival time in the safe area. All execution records are stored in a blockchain ledger to ensure data immutability. When the execution delay of a measure exceeds 20% of the predetermined time, the system automatically activates a backup plan. A standardized report is generated upon completion of execution, including four parts: event details, handling measures, execution results, and improvement suggestions.

[0125] In another embodiment, a method for analyzing the operational status of underground mines based on video surveillance data includes:

[0126] S1: Collect mine video and environmental parameter data through a distributed sensor network, and generate a multi-dimensional data fusion set based on spatiotemporal tagging technology;

[0127] S2: Based on a multi-dimensional data fusion set, feature parameters are extracted using a convolutional neural network algorithm to generate mine operation status identification results;

[0128] S3: Based on the results of mine operation status identification, construct a three-dimensional digital model and dynamic safety boundary to form a real-time monitoring reference framework;

[0129] S4: Based on the real-time monitoring reference framework, perform hierarchical risk assessment on monitoring data within the security boundary, and generate security early warning and response plans with traceability identification.

[0130] The working principle and beneficial effects of the above technical solution are as follows: S1: The data acquisition module enables the acquisition and fusion of multi-dimensional mine environmental data. A distributed sensor network is deployed at key locations throughout the mine, including roadways, working faces, and electromechanical equipment areas. The data collected by the sensors includes high-definition video streams, personnel location information, ambient gas concentrations, temperature, humidity, and other parameters. The system employs spatiotemporal tagging technology to add precise timestamps and spatial coordinates to each data point, achieving spatiotemporal correlation of the data. For example, when an anomaly occurs at the working face, the system can correlate video data from the same time and location with environmental parameter data to form a multi-dimensional data fusion set, providing a comprehensive information foundation for subsequent analysis.

[0131] S2: Real-time identification and judgment of mine operation status is achieved through an edge analysis module. Deployed on edge computing nodes close to the data source, an improved convolutional neural network algorithm (such as ResNet-50 or YOLOv5) is used to process the multi-dimensional data fusion set. First, image preprocessing is performed, including noise removal and illumination equalization; then, feature parameters are extracted, including personnel posture features, equipment motion features, and environmental change features; finally, a trained deep learning model is used for status classification to identify different states such as normal operation, equipment malfunction, and personnel violation. The model adopts a transfer learning approach, first pre-trained on a public dataset and then fine-tuned with mine-specific scenario data, achieving an accuracy rate of over 95%.

[0132] S3: A three-dimensional safety monitoring boundary system for the mine is established through a fence construction module. Based on the mine operation status identification results, combined with mine engineering drawings and geological exploration data, a three-dimensional digital model of the mine is constructed using spatial information technology. On this basis, according to national mine safety regulations and real-time monitoring data, different safety level areas are dynamically divided, forming a multi-level nested safety boundary. For example, areas with excessive gas concentration are marked as red restricted areas, areas with unstable roof support are marked as yellow warning areas, and normal operation areas are marked as green safe areas. The system uses 3D visualization technologies such as WebGL to present these safety boundaries on the monitoring interface in real time, forming an intuitive monitoring reference framework.

[0133] S4: Risk assessment and emergency response are implemented through the decision execution module. Based on a real-time monitoring reference framework, hierarchical risk assessment is performed on monitoring data within the safety boundary. Risk assessment employs a combination of the Analytic Hierarchy Process (AHP) and fuzzy comprehensive evaluation, comprehensively considering multiple dimensions such as environmental risk, personnel risk, and equipment risk. When a risk exceeds a preset threshold, the system automatically generates a safety warning and, based on the risk type and level, invokes a preset response plan library to generate a specific response plan including response steps, responsible personnel, and time requirements. Each warning and response message is accompanied by a unique traceability identifier, facilitating subsequent event retrospective analysis and continuous improvement of risk management capabilities.

[0134] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from the spirit and scope of this invention.

Claims

1. An underground mine operation state analysis system based on video monitoring data, characterized by, Comprise: a data acquisition module for acquiring mine video and environmental parameter data through a distributed sensor network, and generating a multi-dimensional data fusion set based on a space-time tag technology; an edge analysis module for extracting feature parameters based on the multi-dimensional data fusion set through a convolutional neural network algorithm, and generating a mine operation state recognition result; a fence construction module for constructing a three-dimensional digital model and a dynamic safety boundary based on the mine operation state recognition result, and forming a real-time monitoring reference framework; a decision execution module for performing hierarchical risk assessment on monitoring data within the safety boundary based on the real-time monitoring reference framework, and generating a safety warning and disposal scheme with traceability identification.

2. The video surveillance data based underground mine operation state analysis system of claim 1, wherein, The data acquisition module comprises: a video acquisition sub-module for acquiring mine area video stream data under full light conditions through a wide dynamic range camera device and an infrared sensor; an environmental monitoring sub-module for acquiring personnel vital sign signals, environmental gas concentration and electromagnetic field intensity change data; a data transmission sub-module for transmitting mine area video stream data and environmental monitoring data to an edge computing node through a software-defined heterogeneous communication network and storing them.

3. The video surveillance data based underground mine operation state analysis system of claim 1, wherein, The edge analysis module comprises: an image enhancement sub-module for applying a physical model-based illumination compensation and contrast adjustment algorithm to video data to improve image recognizability; a target recognition sub-module for recognizing and tracking the device position and work behavior characteristics of workers based on the image-enhanced video data through a convolutional neural network architecture; a state analysis sub-module for judging the device operation state and personnel operation state based on the target recognition result in combination with environmental parameter data, and generating a mine operation state recognition result.

4. The video surveillance data based underground mine operation state analysis system of claim 1, wherein, The fence construction module comprises: a three-dimensional modeling sub-module for integrating point cloud data collected by a laser radar, vibration waveform data collected by a geological monitoring device and engineering design drawings to construct a mine three-dimensional model with geological structure attributes; a fence configuration sub-module for receiving a three-dimensional model, generating a multi-level nested safety region boundary based on geological stability index calculation and national mine safety regulations; a spatial relationship analysis sub-module for calculating the spatial position relationship between a target and a safety boundary based on the safety region boundary and the mine operation state recognition result, generating target safety state data, and integrating with safety boundary information to form a real-time monitoring reference framework.

5. The video surveillance data based underground mine operation state analysis system of claim 2, wherein, The video acquisition sub-module comprises: an exposure control unit for dynamically adjusting the exposure parameters of the camera device according to the scene brightness distribution histogram based on the ambient light intensity information, to ensure image clarity under low light and strong light conditions; a multispectral imaging unit for simultaneously acquiring image data in the visible light band, near-infrared band and thermal imaging band to identify the material characteristics and temperature distribution characteristics of different objects; an image fusion unit for image registration based on multi-band image data through a feature point matching algorithm, and synthesizing multi-source images using a weighted average method to generate information-enhanced video stream data.

6. The video surveillance data based underground mine operation state analysis system of claim 3, wherein, The state analysis sub-module comprises: a feature extraction unit for extracting personnel action sequences and device motion parameters from the target recognition result; The state recognition unit is configured to recognize the work behavior type by a pre-trained long short-term memory network model based on the personnel action sequence and the equipment motion parameter; The anomaly detection unit is configured to compare the recognized work behavior with a standard work procedure template, calculate a deviation degree, judge whether the work conforms to a safety specification, and integrate the judgment result into the mine work state recognition result.

7. The video surveillance data based underground mine operation state analysis system of claim 4, wherein, The spatial relationship analysis submodule includes: The hybrid positioning unit is configured to fuse wireless signal positioning data, inertial measurement unit data, and visual feature point reference data, and generate anti-interference personnel and equipment position data by a particle filtering algorithm; The path planning unit is configured to calculate an optimal evacuation path considering dynamic obstacles based on an improved random tree optimization algorithm based on the three-dimensional model and the safety area boundary; The behavior prediction unit is configured to analyze a moving trend by a long short-term memory network based on target historical trajectory data, predict a future position, and calculate a proximity probability to a dangerous area, generate a behavior risk score, and trigger a graded early warning signal.

8. The video surveillance data based underground mine operation state analysis system of claim 1, wherein, The decision execution module includes: The situation construction submodule is configured to integrate personnel position, equipment state, and safety boundary data on a time axis based on a real-time monitoring reference framework, and construct a mine work environment digital twin model supporting time backtracking and deduction; The risk assessment submodule is configured to analyze the occurrence probability and severity of various safety events based on the digital twin model by a Bayesian network technology, and generate a risk level distribution heat map; The early warning and disposal submodule is configured to generate graded early warning information according to a preset threshold value based on the risk level distribution heat map, and extract a corresponding disposal scheme from an emergency plan library, and form safety early warning and disposal instructions including execution steps and responsible persons.

9. The video surveillance data based underground mine operation state analysis system of claim 8, wherein, The early warning and disposal submodule includes: The pattern recognition unit is configured to match the current risk state with a historical case library, and identify a dangerous pattern within a preset range; The scheme generation unit is configured to extract corresponding disposal steps from the plan library according to the identified dangerous pattern, and optimize and adjust in combination with the current personnel position and equipment state; The execution tracking unit is configured to issue the disposal scheme to related terminal devices, monitor the execution process, and record the timestamp and execution state of key actions, and form traceable emergency disposal records.

10. A method of analyzing a state of underground mine work based on video monitoring data, characterized by, The method includes: S1: collecting mine video and environmental parameter data through a distributed sensor network, and generating a multi-dimensional data fusion set based on a space-time label technology; S2: extracting feature parameters by a convolutional neural network algorithm based on the multi-dimensional data fusion set, and generating a mine work state recognition result; S3: constructing a three-dimensional digital model and a dynamic safety boundary based on the mine work state recognition result, and forming a real-time monitoring reference framework; S4: performing layered risk assessment on the monitoring data within the safety boundary based on the real-time monitoring reference framework, and generating safety early warning and disposal schemes with traceable identifiers.

Images