Local fan area personnel identification method and device based on multi-modal data fusion

Abstract

The invention discloses a local fan area personnel identification method and device based on multi-modal data fusion, and belongs to the technical field of coal mine safety monitoring. Infrared and visible light video signals of a key area of a local ventilator are comprehensively utilized, and data of a coal mine personnel positioning system are linked; monitoring boundaries of key equipment such as a local fan, a power distribution cabinet and a frequency converter are dynamically calibrated through a key point detection algorithm, and an association rule base of personnel behaviors and equipment states is constructed; personnel identification is carried out by fusing the infrared heat radiation contour and the safety helmet reflective stripe features, and identity verification and entrance recording are carried out by linkage with a positioning system ID. And based on the operation continuity of the spatio-temporal trajectory analysis personnel, returning after short-time leaving is regarded as continuous operation, the operation compliance is predicted according to the rule base, and early warning is triggered in time. According to the invention, accurate identification and automatic safety monitoring of local fan area personnel are realized, and the coal mine safety management efficiency and the risk early warning capability are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of coal mine safety monitoring technology, and in particular to a method, device, equipment and storage medium for local sector personnel identification based on multimodal data fusion. Background Technology

[0002] In the field of coal mine safety production, local ventilation fans (hereinafter referred to as "local fans") are core equipment for ensuring ventilation safety in underground tunneling faces. The areas surrounding these fans, including the fan body itself, the power distribution cabinet, and frequency converters, are key areas for safety monitoring. Accurate identification and recording of personnel activities in this area are crucial for standardizing work practices, clarifying safety responsibilities, and preventing accidents.

[0003] Currently, personnel monitoring in local sector areas in the industry generally relies on visible light surveillance cameras deployed in key locations. The typical working mode is as follows: the personnel on duty at the monitoring center observe the video footage, manually identify and determine whether personnel have entered the local sector area, approached what equipment, and performed what operations, and manually record information such as the personnel's arrival time, operation content, and departure time.

[0004] However, this existing technological solution, which relies primarily on manual visual recognition and recording, has many inherent drawbacks: First, the operation is complex and labor costs are high. Relying on manual monitoring and recording throughout the process is not only inefficient, but also requires a continuous investment of dedicated personnel. When multiple monitoring points exist, visual fatigue can easily lead to oversights and omissions.

[0005] Secondly, the recording is highly subjective, resulting in poor accuracy and consistency. The identification of personnel and the judgment of their behavior rely on the personal experience of the personnel on duty, lacking objective and unified judgment standards. This leads to highly subjective and erroneous recorded information, making it difficult to form accurate data that can be traced and analyzed.

[0006] Secondly, reliability is severely constrained by the environment and signal conditions. The lighting conditions in the underground environment are complex, and the imaging quality of visible light cameras is easily affected by dust, fog, insufficient lighting, or equipment malfunctions, leading to video signal interruption or unclear images, resulting in personnel identification failure or recording interruption.

[0007] Finally, information is isolated, lacking intelligent correlation and early warning capabilities. Existing methods can only record basic arrival and departure times, and cannot perform intelligent spatiotemporal trajectory analysis of continuous personnel operations, let alone correlate personnel behavior with equipment status in real time, thus failing to achieve proactive early warning of violations or abnormal states.

[0008] Therefore, there is an urgent need for a new technical solution that can automatically and accurately identify personnel in a local sector area and achieve intelligent analysis and early warning, so as to overcome the limitations of the existing monitoring methods based on manual identification. Summary of the Invention

[0009] The present invention aims to at least partially solve one of the technical problems in the related art.

[0010] To address this, the present invention proposes a method for identifying personnel in a local sector area based on multimodal data fusion. By comprehensively utilizing infrared thermal imaging, visible light video, and personnel positioning data, the monitoring area is dynamically calibrated using key point detection and a behavioral rule base is established. The method integrates thermal radiation profiles and safety helmet features to achieve accurate identification and recording. Furthermore, it analyzes the continuity and compliance of operations based on spatiotemporal trajectory to trigger early warnings, thereby achieving automated intelligent monitoring and safety management of personnel in a local sector area.

[0011] Another objective of this invention is to propose a local sector personnel identification device based on multimodal data fusion.

[0012] The third objective of this invention is to provide a computer device.

[0013] The fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0014] To achieve the above objectives, this invention proposes a method for identifying people in a local sector based on multimodal data fusion, comprising: S1 calls up the infrared thermal imaging video signal and visible light video signal of the key area of ​​the local ventilation fan, and obtains the real-time personnel data of the coal mine personnel positioning system; S2 uses a key point detection algorithm to dynamically define the boundaries of the monitoring areas of the local fan body, distribution cabinet and frequency converter, and establishes a rule base for the association between personnel actions and equipment status; S3 integrates infrared thermal radiation profile features with visible light safety helmet reflective strip features for personnel identification, verifies identity by combining coal mine personnel positioning system ID, and records personnel information and time of entry into the monitoring area; S4 determines the continuity of personnel operations based on spatiotemporal trajectory analysis. If a person returns within 1 minute after leaving, it is considered a continuous operation. At the same time, it predicts the compliance of the operation and triggers an early warning by associating the rule base.

[0015] The method for identifying people in a local sector based on multimodal data fusion according to an embodiment of the present invention may also have the following additional technical features: In one embodiment of the present invention, the step of calling the infrared thermal imaging video signal and visible light video signal of the key area of ​​the local ventilation fan and obtaining real-time personnel data from the coal mine personnel positioning system includes: S11 uses an infrared thermal imaging camera to collect the thermal radiation contour features of the human body and uses a Gaussian mixture model for background separation. S12 performs histogram equalization on the visible light video signal and extracts the HSV color features of the reflective strips on the safety helmet.

[0016] In one embodiment of the present invention, the step of dynamically calibrating the boundaries of the monitoring areas of the local fan body, distribution cabinet, and frequency converter through a key point detection algorithm, and establishing a rule base for the association between personnel actions and equipment status, includes: S21, Based on the YOLOv5 improved model, key points of the device body are identified, and dynamic monitoring boundaries are generated by the minimum bounding rectangle algorithm; S22, construct a rule base containing rules that trigger compliance judgments when the device is near the frequency converter and the dwell time is greater than 30 seconds, and use a decision tree for behavior classification.

[0017] In one embodiment of the present invention, the method of fusing infrared thermal radiation contour features and visible light safety helmet reflective strip features for personnel identification, verifying identity in conjunction with the coal mine personnel positioning system ID, and recording personnel information and time of entry into the monitoring area includes: S31, Establish an illumination-invariant feature extraction model, and fuse thermal imaging contour features through feature stitching. With visible light color characteristics ; S32, uses the Euclidean distance algorithm to compare the positioning system ID with the recognition result, when the matching degree... Secondary verification is triggered at that time.

[0018] In one embodiment of the present invention, the step of determining the continuity of personnel operations based on spatiotemporal trajectory analysis, and considering a person's return within one minute of leaving as a continuous operation, while simultaneously predicting operation compliance and triggering an early warning through an associated rule base, includes: S41, Calculate the time personnel spend in the monitored area. ; S42, when If the standard operating procedure is not followed within seconds, a level three warning will be triggered via an audible and visual alarm device.

[0019] In one embodiment of the present invention, it further includes: S5, according to the preset statistical period Generate multi-dimensional analysis reports; S6, for the frequency of personnel entry included in the report content. Average length of stay and the number of violations Perform statistical analysis.

[0020] To achieve the above objectives, another aspect of the present invention proposes a local sector area personnel identification device based on multimodal data fusion, comprising: The multimodal data acquisition module is used to call up infrared thermal imaging video signals and visible light video signals of key areas of local ventilation fans, and to acquire real-time personnel data from the coal mine personnel positioning system; The equipment boundary calibration module is used to dynamically calibrate the boundaries of the monitoring areas of the local fan body, power distribution cabinet and frequency converter through key point detection algorithm, and establish a rule library for the association between personnel actions and equipment status; The feature fusion and recognition module is used to fuse infrared thermal radiation contour features and visible light safety helmet reflective strip features for personnel identification, verify identity by combining the coal mine personnel positioning system ID, and record personnel information and time of entry into the monitoring area; The spatiotemporal trajectory analysis module is used to determine the continuity of personnel operations based on spatiotemporal trajectory analysis. If a person returns within 1 minute after leaving, it is considered a continuous operation. At the same time, the module predicts the compliance of the operation and triggers an alert by associating the rule base.

[0021] In one embodiment of the present invention, it further includes: The statistical report generation module is used to generate reports based on preset statistical periods. Generate multi-dimensional analysis reports; The report analysis module is used to analyze the frequency of personnel entry in the report content. Average length of stay and the number of violations Perform statistical analysis.

[0022] This invention discloses a method and apparatus for collaborative prediction of multi-level health status and lifespan of a battery pack. By constructing a multi-level graph model that integrates spatial topology and time-varying dynamics, and employing attribute decoupling and cross-layer bidirectional message passing mechanisms, it effectively solves the problems of insufficient prediction accuracy and poor stability caused by hierarchical isolation, feature aliasing, and separate modeling in existing technologies. It achieves integrated modeling from multi-level feature extraction and cross-layer information fusion to collaborative prediction of health status and lifespan, significantly improving the accuracy and reliability of overall battery pack status assessment, and enhancing the generalization ability and engineering applicability of the prediction model under complex operating conditions.

[0023] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing a method for identifying personnel in a local sector area based on multimodal data fusion as described in the first aspect embodiment.

[0024] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for identifying personnel in a local sector based on multimodal data fusion as described in the first aspect embodiment.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a method for identifying people in a local sector based on multimodal data fusion according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the process for identifying personnel in a sector area according to an embodiment of the present invention, which is based on multimodal data fusion. Figure 3 This is a schematic diagram of the layout of a local sector personnel identification device in a local sector personnel identification system based on multimodal data fusion according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a local sector personnel identification device based on multimodal data fusion according to an embodiment of the present invention; Figure 5 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] The following description, with reference to the accompanying drawings, describes a method, apparatus, device, and storage medium for local sector personnel identification based on multimodal data fusion according to an embodiment of the present invention.

[0030] The core idea of ​​this invention is to construct a multimodal perception and decision-making system that integrates infrared thermal imaging, visible light video, and personnel positioning data to achieve accurate identification and intelligent analysis of personnel identity and behavior in a local area. First, it collaboratively calls upon multi-source heterogeneous data and uses key point detection technology to dynamically calibrate the monitoring boundaries of key equipment, thereby establishing a rule base for the association between personnel actions and equipment status. Based on this, it identifies personnel by fusing infrared thermal radiation contour features and visible light safety helmet reflective strip features, and performs identity cross-verification and entry recording by combining the positioning system ID. Finally, based on the continuous analysis of personnel spatiotemporal trajectories, the system can intelligently judge the integrity of operation sequences and predict and warn of potential violations. This transforms the traditional passive security mode relying on manual monitoring into a closed-loop safety management paradigm integrating automatic identification, accurate tracking, intelligent analysis, and proactive early warning, significantly improving the safety monitoring efficiency and automation level of key areas in underground coal mines.

[0031] Example 1 To achieve the above invention, embodiments of the present invention provide a method for identifying people in a local sector based on multimodal data fusion, such as... Figure 1 As shown, it includes: S1 calls up the infrared thermal imaging video signal and visible light video signal of the key area of ​​the local ventilation fan, and obtains real-time personnel data from the coal mine personnel positioning system.

[0032] Specifically, the core of this step lies in achieving synchronous acquisition and spatiotemporal alignment of multi-source heterogeneous data, thereby providing high-confidence input for subsequent personnel identification and behavior analysis.

[0033] Specifically, infrared thermal imaging images are acquired using uncooled infrared detectors (e.g., 640×480 resolution, 30fps), whose thermal sensitivity (NETD) is typically less than 50mK, enabling stable capture of human thermal radiation contours in low-light, high-dust environments. Visible light video images are acquired by industrial-grade high-definition cameras (e.g., 1080P resolution, 25fps), supporting H.264 or H.265 encoding, and possessing an IP67 protection rating, adapting to complex underground environments such as humidity and vibration. Personnel positioning system data is acquired through UWB (Ultra-Wideband) or RFID technology, achieving positioning accuracy of 0.1m to 0.5m, supporting real-time location update frequencies of 1Hz to 5Hz, ensuring high timeliness of personnel identification and location information.

[0034] Furthermore, infrared and visible light images need to be synchronized in terms of timestamps and spatial coordinates. This is typically achieved using the RTSP protocol for real-time video stream transmission, with NTP time synchronization ensuring temporal consistency across multiple data sources. For spatial alignment, a rigid registration algorithm based on coordinate system transformation can be optionally employed to map the infrared and visible light image coordinates to a unified three-dimensional spatial coordinate system, with the error controlled within ±5cm. Personnel positioning data is then bound to the image system via an API interface to match personnel IDs with human body contours in the images.

[0035] Specifically, this step is applicable to all-weather monitoring of local ventilation fan areas in coal mines, especially at key operation points such as ventilation equipment, power distribution cabinets, and frequency converters. Through multimodal data fusion, the identity, job type, and operation behavior of personnel entering the premises can be effectively identified, providing real-time and accurate data support for safety management.

[0036] Specifically, the multimodal perception data source constructed in this step significantly improves the system's robustness and recognition accuracy in complex environments, laying a solid foundation for subsequent personnel behavior analysis, operational compliance judgment, and report generation.

[0037] Furthermore, S1 includes: S11 uses an infrared thermal imaging camera to collect the thermal radiation contour features of the human body and employs a Gaussian mixture model for background separation.

[0038] Specifically, this step involves continuously collecting thermal radiation data of the monitored area at a specific sampling frequency using uncooled infrared focal plane detectors deployed in key areas of the local fan. The core of this process lies in the stable extraction of human targets from complex backgrounds. To achieve this, the system employs a background modeling algorithm based on Gaussian mixture models. By establishing multiple Gaussian distribution models for each pixel in the scene, it uses probabilistic statistics to distinguish moving foreground targets from static or slowly changing backgrounds. This method effectively adapts to background disturbances caused by sudden changes in downhole lighting and interference from equipment heating elements, thus robustly separating complete human thermal radiation contour features, laying the foundation for subsequent accurate identification.

[0039] Specifically, this step directly addresses the harsh visual environment of coal mines, characterized by low visibility and severe dust interference. Infrared thermal imaging technology does not rely on visible light illumination and can penetrate some smoke and dust, ensuring reliable detection of personnel even in complete darkness or low-light conditions. Combined with the dynamic background separation capability of the Gaussian mixture model, the system can effectively filter out continuous interference from non-human heat sources (such as operating equipment and heat dissipation from tunnel walls), significantly reducing the false alarm rate under complex thermal backgrounds and improving the practical reliability of personnel detection.

[0040] Furthermore, the infrared thermal imaging camera preferably has a pixel resolution of no less than 320x240, and its thermal sensitivity (NETD) should be better than 50mK to ensure that it can clearly distinguish the subtle temperature differences between the human body outline and the background. In specific implementations, the Gaussian mixture model typically establishes 3 to 5 Gaussian distributions for each pixel, with its learning rate parameter α set to 0.005. This strikes a balance between model update speed and background stability, thereby ensuring the ability to adapt to changes in lighting and periodic interference while avoiding misclassifying people who have been stationary for a long time as background.

[0041] Furthermore, this step constitutes the foundational perception layer of the entire multimodal personnel identification system. The extracted human thermal radiation contour, separated from the background, is one of the key information sources for subsequent fusion identification with visible light features and cross-validation with personnel positioning system data. This capability is particularly suitable for detecting the presence of personnel in blind spots such as behind and to the sides of key equipment like power distribution cabinets and frequency converters, compensating for the limited field of view of single visible light video surveillance.

[0042] Specifically, the system achieves stable and robust detection of human targets within the monitoring area. Compared to traditional visible light solutions, it fundamentally overcomes the effects of insufficient underground lighting and dust obscuring the view; and compared to simple infrared alarms without background separation, it significantly improves recognition accuracy through intelligent algorithms and effectively suppresses false alarms caused by environmental heat source interference. This provides the primary and crucial technical guarantee for building a highly reliable and automated local sector area security monitoring system.

[0043] S12 performs histogram equalization on the visible light video signal and extracts the HSV color features of the reflective strips on the safety helmet.

[0044] Specifically, this step involves acquiring video streams using high-definition visible light cameras deployed on-site, and first performing histogram equalization on each frame. This preprocessing operation aims to redistribute the grayscale values ​​of image pixels and expand the image contrast range, thereby enhancing the detail information in both dark and bright areas of the image under uneven lighting conditions underground. Subsequently, the system converts the enhanced RGB color space image to the HSV color space. In this space, considering the unique high saturation and brightness characteristics of the reflective strips on safety helmets, specific threshold ranges are set for their H, S, and V components to accurately separate and extract the significant color features of the reflective strips, providing a reliable visual basis for subsequent feature fusion and personnel identification.

[0045] Specifically, this step directly addresses the challenges of complex and variable lighting conditions in underground coal mines (such as dim lighting, localized strong light, and shadow occlusion). Histogram equalization effectively improves the overall visibility of the image, enabling the reflective strips on the safety helmet to maintain high color saturation and brightness even in low-light or backlit environments, significantly reducing the feature extraction failure rate caused by insufficient lighting. By utilizing the HSV color model's ability to separate luminance and chromaticity, this extraction method exhibits stronger robustness to changes in light intensity, ensuring the stability and consistency of feature recognition under different ambient light conditions.

[0046] Furthermore, the histogram equalization process preferably employs a contrast-limited adaptive histogram equalization algorithm, with its clipping limit parameter typically set between 2.0 and 3.0, and the tile grid size set to 8x8, to achieve a balance between enhancing contrast and suppressing background noise. For HSV color feature extraction, the threshold range is set based on the colorimetric characteristics of the reflective material of a standard safety helmet. For example, the hue component H can be set within a specific angular range, the lower limit of the saturation component S can be set above 0.4, and the lower limit of the lightness component V can be set above 0.6, thereby ensuring the accuracy and specificity of feature extraction.

[0047] Specifically, the HSV color feature of the safety helmet reflective strip extracted in this step, together with the aforementioned infrared thermal radiation contour feature, constitutes the core visible light modality data in the multimodal recognition system. This feature, as a strong visual identifier of personnel (especially workers wearing standard safety helmets), effectively complements the thermal imaging feature in the feature fusion recognition module, and together they are used for subsequent cross-verification with the personnel positioning system ID. This is an indispensable part of achieving accurate identity association and compliance judgment.

[0048] Specifically, the system significantly improves the visual recognition capability of underground workers. Histogram equalization improves image quality, creating favorable conditions for feature extraction; while the thresholding extraction method based on the HSV color space directly and efficiently captures the key identifier of the safety helmet. Compared to processing directly in the RGB space, this method is less affected by lighting fluctuations and has higher feature discrimination, thus improving the overall accuracy, robustness, and practical value of the personnel recognition system in complex underground environments.

[0049] S2 uses a key point detection algorithm to dynamically define the boundaries of the monitoring areas of the local fan body, distribution cabinet, and frequency converter, and establishes a rule library for the association between personnel actions and equipment status.

[0050] Specifically, this step integrates key point detection technology from computer vision with illumination invariance feature modeling methods, aiming to address complex working conditions in the mining environment, such as large changes in illumination, strong dust interference, and possible equipment position shifts.

[0051] Specifically, this step first invokes video surveillance signals installed at key locations such as the main unit of the power distribution unit, the distribution cabinet, and the frequency converter. The signal source can be a color or monochrome camera. The system employs deep learning-based key point detection algorithms (such as OpenPose and HRNet) to identify and locate the geometric contours of the devices in the video frames, extracting corner or edge features to dynamically delineate the boundary of the monitoring area. This boundary can adaptively adjust as the device position changes, avoiding the monitoring failure problems caused by traditional fixed electronic fences due to device movement or installation deviations.

[0052] Furthermore, based on the defined boundary region, the system constructs an illumination-invariant feature extraction model. This model employs a multi-scale feature fusion strategy, combining infrared thermal imaging and visible light image information to extract the human body's thermal radiation contour and visible light features (such as safety helmets and reflective strips) to enhance recognition robustness in low-light and high-dust environments. During feature extraction, image preprocessing techniques such as histogram equalization and illumination normalization are used to ensure that the model maintains a 95% recognition accuracy under different illumination conditions.

[0053] Specifically, the confidence threshold of keypoint detection algorithms is typically set to... The above measures are to ensure the accuracy of device boundary calibration. The input image size for the illumination invariance model is [size missing]. The feature extraction layer employs lightweight network structures such as ResNet-50 or MobileNetV3 to meet the real-time requirements of mine monitoring systems. The recognition frequency is set to... Frames per second, meeting the real-time response requirements for personnel entering and leaving.

[0054] Specifically, this step is widely used in local ventilation systems in underground coal mines, especially for personnel management in key equipment areas such as the local fan body, power distribution cabinet, and frequency converter. The system can be deployed in the mine monitoring center to analyze personnel entry in real time through video streams and combine it with personnel positioning system data for identity verification and behavior recording, achieving a technological upgrade from "area intrusion" to "behavioral intent prediction".

[0055] Specifically, it enables automatic identification and adaptive boundary adjustment of personnel in local fan areas, effectively improving identification accuracy and system robustness. This provides a reliable data foundation for subsequent personnel behavior analysis, operational compliance judgment, and report generation, significantly enhancing the intelligent management level of the mine ventilation system.

[0056] Furthermore, S2 includes: S21, based on the YOLOv5 improved model, identifies key points of the device body and generates dynamic monitoring boundaries through the minimum bounding rectangle algorithm.

[0057] Specifically, this step first involves targeted improvements to the YOLOv5 target detection architecture. By introducing an attention mechanism and increasing the anchor frame size to suit the shape of downhole equipment, a dedicated model is constructed to accurately identify the body contours and key structural points of critical equipment such as the main fan, distribution cabinet, and frequency converter. After successfully locating the key components of the equipment, the system employs the minimum bounding rectangle algorithm from computational geometry to calculate and generate the minimum area rectangular region that can completely enclose the target equipment based on the spatial distribution of the identified key points. This rectangular boundary serves as a dynamic monitoring area, and its position and angle can adaptively adjust with changes in the equipment's viewing angle or slight displacement, providing an accurate spatial reference for subsequent personnel behavior analysis.

[0058] Specifically, this step directly addresses practical issues such as potential temporary changes in equipment layout due to maintenance and repair in the underground environment, as well as vibrations or shifts in the viewing angle of monitoring cameras. By dynamically calibrating rather than pre-setting fixed areas, the system avoids inaccurate monitoring areas caused by these factors, ensuring the long-term effectiveness of personnel positioning and equipment association. This adaptive capability significantly improves the system's deployment flexibility and environmental adaptability in complex actual working conditions, reduces stringent requirements on camera installation positions and angles, and decreases system maintenance costs due to environmental changes.

[0059] Furthermore, during the training phase, the improved YOLOv5 model preferably uses an input image resolution of 640x640 pixels and a confidence threshold of no less than 0.85 to ensure the accuracy of device recognition. During inference, the IoU threshold for non-maximum suppression is set to 0.5 to balance recall and precision. For generating the minimum bounding rectangle, the algorithm calculates the convex hull of all keypoints and uses the rotating caliper method to find the bounding rectangle with the smallest area. Its boundary coordinates are output with pixel precision as a benchmark for subsequent spatial judgment.

[0060] Specifically, the dynamic monitoring boundary generated in this step forms the spatial basis for the accurate application of the entire association rule base. It provides a precise spatial definition, synchronized with the actual location of the equipment, for determining whether personnel have "entered" or "approached" a specific piece of equipment. This boundary information is a prerequisite for triggering subsequent rules (such as "approaching the inverter and staying for more than a set threshold"), ensuring the accuracy and reliability of the spatial dimension in the correlation analysis between personnel behavior and equipment status.

[0061] Specifically, the system achieves automated, high-precision, and adaptive calibration of the monitoring area for key equipment. Compared to the method of manually pre-delineating static monitoring areas, this technology significantly improves the accuracy and reliability of spatial calibration, effectively overcoming monitoring blind spots or misjudgments caused by equipment displacement or changes in viewing angle. This dynamic boundary generation capability lays a solid technical foundation for building an accurate human-equipment interaction behavior analysis model and is one of the core components of the entire system's intelligent monitoring capabilities.

[0062] S22, construct a rule base containing rules that trigger compliance judgments when the device is near the frequency converter and the dwell time is greater than 30 seconds, and use a decision tree for behavior classification.

[0063] Specifically, the core of this step lies in constructing a structured association rule base and utilizing machine learning models to automate the classification of personnel behavior. Specifically, the rule base is logically encoded based on equipment operation safety procedures, including "personnel entering the inverter's dynamic monitoring boundary and remaining there continuously for more than 30 seconds" as one of the key conditions triggering compliance judgment. In the behavior classification stage, the system employs a decision tree algorithm, using personnel location coordinates, dwell time, the type of nearby equipment, and their historical operation records as feature attributes. Node splitting is performed using criteria such as information gain or Gini impurity to construct a highly interpretable and efficient classification model, thereby enabling automatic differentiation of different behavior categories such as "compliant operation," "unauthorized stay," and "unauthorized access."

[0064] Specifically, this step aims to address the problems of untimely supervision and inconsistent standards caused by reliance on manual inspections and subjective judgment in traditional safety management. By transforming clear safety procedures into executable digital rules and combining them with a decision tree model to comprehensively analyze multi-dimensional behavioral characteristics, the system can achieve standardized and automated supervision of personnel operations. This significantly reduces reliance on the personal experience of monitoring personnel, ensures the objectivity and consistency of safety judgments, and enables 24 / 7 uninterrupted real-time identification and early warning of high-risk operational behaviors.

[0065] Furthermore, the dwell time threshold in the rule base is set to 30 seconds, a value optimized based on statistical analysis of standard operating procedures and typical violation patterns. In constructing the decision tree model, the CART algorithm is preferred, with its maximum depth typically limited to 5 to 8 layers to prevent overfitting; the minimum number of samples required for node splitting is set to 10; and the minimum number of samples for leaf nodes is set to 5. After model training, its classification accuracy on the test set should be no less than 90%, and both recall and precision should reach over 85% to ensure the reliability of behavior determination.

[0066] Specifically, the rule base and decision tree classification model constitute the core logical unit for the entire system's intelligent analysis and early warning decision-making. It directly relies on the dynamic monitoring boundaries generated in the preceding steps and the real-time spatiotemporal data of personnel. Once a rule condition is triggered, the system will immediately initiate the decision tree classification process to quickly assess the compliance of the current behavior. Its output will be directly linked to the subsequent early warning triggering module, making it a key link in realizing the intelligent closed loop from "perception" to "cognition" to "decision".

[0067] Specifically, the system achieves a leap from passive recording of personnel operations to proactive intelligent analysis. The rule base ensures that safety supervision is based on established rules, while the decision tree model empowers the system to simulate complex judgments by human experts. The combination of these two elements not only significantly improves the efficiency and response speed of safety monitoring, but more importantly, through precise and automated compliance assessments, it identifies potential violations in advance. This elevates safety management from post-event traceability to in-process intervention and even pre-event prevention, significantly enhancing the safety assurance capabilities of critical equipment areas in underground coal mines.

[0068] S3 integrates infrared thermal radiation profile features with visible light safety helmet reflective strip features for personnel identification, verifies identity using the coal mine personnel positioning system ID, and records personnel information and time of entry into the monitoring area.

[0069] Specifically, the system performs real-time identity verification and behavior recording of identified personnel by binding them to the coal mine personnel positioning system information. This step is technically based on a multi-source data fusion mechanism, specifically including data synchronization and comparison between the image recognition module and the personnel positioning system. In some implementations, the system acquires personnel images using infrared and visible light cameras and extracts their illumination-invariant features, such as the human body's thermal radiation contour and reflective strips on safety helmets, to improve the robustness of identification. Simultaneously, the personnel positioning system uploads personnel ID, location coordinates, and timestamps in real time via wearable positioning tags (such as UWB or RFID technology). Upon receiving the personnel contour or action information output by the image recognition module, the system immediately calls upon the positioning system database for ID comparison and verification.

[0070] Furthermore, when the image recognition module detects personnel entering a preset key area (such as the main fan, distribution cabinet, frequency converter, etc.), it matches the personnel's image features with the personnel ID stored in the positioning system. The matching process uses a feature vector-based similarity calculation method to ensure an accuracy rate of no less than 95%. The system records the entry time, exit time, and operational behavior (such as switch operations, dwell time, etc.), and performs compliance judgments based on a preset personnel action-equipment status association rule base. For example, if a person approaches and operates the frequency converter without authorization, the system will trigger an alarm mechanism and record the violation.

[0071] Specifically, this step is widely used in underground ventilation systems in coal mines, particularly in monitoring personnel behavior in local fan areas. Through real-time comparison and recording, the system can effectively prevent unauthorized personnel from operating critical equipment, ensuring the safe operation of the ventilation system. The technical value of this step lies in constructing a closed-loop management mechanism for identity verification and behavior analysis, significantly improving the automation level and response efficiency of mine safety management.

[0072] Furthermore, S3 includes: S31, Establish an illumination-invariant feature extraction model, and fuse thermal imaging contour features through feature stitching. With visible light color characteristics .

[0073] Specifically, this model typically employs a convolutional neural network pre-trained on a large amount of data as its foundation, extracting abstract feature representations insensitive to changes in illumination through its deep nonlinear transformations. Specifically, the system extracts human thermal radiation contour feature vectors from infrared images. Extracting color and texture feature vectors of the reflective stripes on a safety helmet from a preprocessed visible light image. Subsequently, at the feature level, a concatenation method is used for fusion, that is, the two feature vectors are connected in a dimension to form a more discriminative joint feature representation, providing a data foundation for subsequent accurate identification.

[0074] Furthermore, the illumination-invariant feature extraction model effectively suppresses the interference of shadows, strong light reflection, and low illumination on feature stability by learning features of the same target under different illumination conditions. The feature stitching fusion strategy fully utilizes the complementary advantages of infrared mode being unaffected by illumination and visible light mode having rich color and texture information. This allows the system to maintain high-precision recognition capabilities even when personnel are wearing reflective identification clothing and in complex lighting environments, significantly improving the system's practical value and reliability in real-world industrial scenarios.

[0075] Furthermore, the feature extraction model preferably uses a network structure of ResNet-50 or similar depth, with the fully connected layer output feature dimension typically set to 512. For infrared features (F_IR) and visible light features (F_RGB), L2 normalization is performed separately before concatenation to ensure that the features of each modality have similar magnitudes and distributions during fusion. The total dimension of the concatenated joint feature vector is 1024. During model training, triplet loss or center loss is used as a metric for learning loss functions to optimize the feature space, minimizing the feature distance between similar samples and maximizing the feature distance between dissimilar samples.

[0076] Specifically, this step is the core of achieving synergistic effects from multimodal data. The generated illumination-invariant joint features directly serve the subsequent module for accurate personnel identification and verification. As a crucial bridge connecting front-end sensing data and back-end identification decisions, it ensures that even when the quality of visible light images temporarily deteriorates due to environmental factors, the system can still maintain reliable identification performance by relying on the fused robust features, thereby guaranteeing the continuous and effective operation of the entire monitoring system in the complex underground environment.

[0077] Specifically, the illumination invariance model significantly improves the system's adaptability and stability under dynamic lighting conditions; the feature splicing fusion strategy preserves the unique identification information of each modality to the greatest extent. The combination of these two approaches enables the final personnel identification model to exhibit higher accuracy, stronger robustness, and lower false recognition rate when facing complex and ever-changing underground working conditions, providing crucial technical support for achieving precise and unmanned safety monitoring.

[0078] S32, uses the Euclidean distance algorithm to compare the positioning system ID with the recognition result, when the matching degree... Secondary verification is triggered at that time.

[0079] Specifically, the system first maps the target personnel information identified based on visual features to the real-time ID data transmitted from the coal mine personnel positioning system, respectively, into comparable feature vectors. Then, it uses the Euclidean distance algorithm to calculate the straight-line distance between these two feature vectors in multi-dimensional space, thereby quantifying their similarity. This distance value is then normalized and converted into a matching degree index. The system presets a confidence threshold; when the calculated matching degree is lower than this threshold, it is determined that there is uncertainty in the first identification, and a preset secondary verification mechanism is automatically triggered. This mechanism may include calling historical behavioral data, initiating multimodal data re-evaluation, or marking the system as awaiting manual review.

[0080] Specifically, this step aims to address the potential for missed or false recognitions in complex downhole environments when using a single recognition modality. By introducing the positioning system ID for cross-validation, a redundant identity verification channel is constructed. When visual recognition lacks confidence due to occlusion, image blurring, or special poses, the system does not directly adopt the potentially biased result but instead activates a secondary verification process to seek a more reliable judgment criterion.

[0081] Furthermore, the matching threshold was validated through extensive experimental data and optimally set at 0.8. This value balances the conflict between false recognition and false negative rates, aiming to ensure the accuracy of the main recognition channel with high confidence. The Euclidean distance is calculated based on the L2 norm of the feature vectors, and its formula is the square root of the sum of the squares of the differences in each dimension. The system has specific requirements for the response time of secondary verification; the delay from the initial matching failure to the start of the verification process should be controlled within milliseconds to ensure real-time monitoring. The algorithm complexity of the entire verification process has been optimized to be completed within limited computing resources.

[0082] Specifically, this step constitutes a key decision-making node and reliability assurance link in the entire identification process. Located after feature fusion identification and before final recording and early warning decisions, it acts as a "quality check" and a "safety valve." Especially in densely populated areas, during periods of drastic light changes, or during critical equipment operation phases, this cross-validation mechanism can effectively filter out unreliable identification results, ensuring the high accuracy and reliability of the personnel identity information entered into the system and subsequent behavioral analysis data.

[0083] Specifically, the system constructs an intelligent recognition closed loop with self-questioning and corrective capabilities. This not only improves the accuracy of single-identification but, more importantly, significantly enhances the robustness of the entire system in the face of abnormal situations by introducing a secondary verification mechanism.

[0084] S4 determines the continuity of personnel operations based on spatiotemporal trajectory analysis. If a person returns within 1 minute after leaving, it is considered a continuous operation. At the same time, it predicts the compliance of the operation and triggers an early warning by associating the rule base.

[0085] Specifically, the system integrates an image recognition module with a coal mine personnel positioning system to obtain real-time information on the identity of personnel entering key areas of the mine's ventilation system, their entry and exit times, and their operational behavior records. This data is then structured and aggregated in multiple dimensions according to timestamps, job categories (such as ventilation workers, electricians, and safety officers) and their respective work teams (such as the No. 1 mining area and the No. 2 tunneling team).

[0086] Specifically, the system automatically triggers a data acquisition process when it detects personnel entering preset key areas (such as frequency converters, distribution cabinets, and local fan units). Entry time is extracted using video frame timestamps, achieving millisecond-level accuracy; personnel identification information is compared in real-time with a unique ID bound to the coal mine personnel positioning system, ensuring an accuracy rate of no less than 95%. Furthermore, the system supports tracking the duration of personnel stay within the area; if personnel leave... If the user returns within a few minutes, the system will determine it as a continuous operation and will not consider it a new entry event, thus avoiding misjudgment.

[0087] Furthermore, the system employs a hierarchical structured data model, supporting statistical analysis at time granularities such as hourly, daily, weekly, and monthly. Report content includes, but is not limited to: the number of personnel entering the system, personnel identification information, operation duration, and operation compliance judgment results. Reports can be exported in standard formats such as Excel and PDF, facilitating data archiving and decision analysis for management personnel.

[0088] Specifically, this step is widely applicable in practical coal mine ventilation system management scenarios, especially in complex roadway environments where multiple local fans operate in tandem, enabling closed-loop management and traceability of personnel operations. Its technological value lies in significantly improving management efficiency and reducing manual intervention through automated data integration and report generation, providing reliable data support for the safe operation of ventilation equipment and the assessment of personnel compliance.

[0089] Furthermore, S4 includes: S41, Calculate the time personnel spend in the monitored area. .

[0090] Specifically, this process involves capturing, storing, and performing basic arithmetic operations on time data, providing core time-series quantitative data for subsequent behavioral analysis. This calculation directly serves the intelligent assessment of the continuity and standardization of personnel operations. By objectively and automatically measuring dwell time, the system replaces the traditional manual estimation and recording mode, providing accurate data support for judging whether operations are complete and whether there are abnormal delays or hurried operations.

[0091] Furthermore, the timestamp collection accuracy is optimized to the second level to ensure the accuracy of the calculation results. The key indicator of dwell time will be directly applied to the preset association rule base, such as comparing it with preset thresholds, to automatically trigger corresponding compliance judgments or early warning processes.

[0092] Specifically, this step forms the basis of spatiotemporal trajectory analysis. The calculated dwell time is a key input parameter for determining whether personnel are performing continuous operations and assessing whether their operational behavior conforms to work procedures, forming the core bridge connecting personnel location data and intelligent behavioral analysis. By implementing this step, the system achieves refined and digital measurement of personnel operational behavior. This effectively eliminates the subjectivity and errors of manual recording, significantly improves the objectivity and accuracy of judging operational compliance, and provides crucial data support for achieving automated safety monitoring and early warning.

[0093] S42, when If the standard operating procedure is not followed within seconds, a level three warning will be triggered via an audible and visual alarm device.

[0094] Specifically, this step uses a logic judgment module to compare the calculated dwell time with a preset threshold in real time, and combines this with the monitoring results of the standard operating procedure execution status from the behavior recognition module. When the system detects that the dwell time exceeds sixty seconds and fails to capture the preset key action sequence that conforms to the standard operating procedure, it is determined to be a potential violation of the rules or an operational anomaly. Subsequently, the system generates a specific warning command, driving the audible and visual alarm devices deployed on-site to issue preset three-level alarm signals, thus completing the closed-loop control from data analysis to physical warning.

[0095] Specifically, this step directly addresses the safety hazards caused by prolonged lingering in critical equipment areas without effective operation due to negligence, violations, or unclear circumstances during downhole operations. Through automated condition judgment and real-time alarms, the system proactively intervenes in such abnormal behaviors, promptly reminding on-site personnel to correct inappropriate actions or alerting surrounding personnel to potential risks, effectively compensating for the shortcomings of traditional manual supervision in terms of timeliness and continuity.

[0096] Furthermore, the threshold for determining the dwell time is set at sixty seconds, a value optimized based on statistical analysis of normal equipment operation cycles. The triggered "Level 3 Warning" indicates that the severity of the event falls within a specific preset level. Its corresponding audible and visual alarm signals (such as specific flashing and sound frequencies) are distinct from other alarm levels to achieve a differentiated warning effect. The overall response time of the system from condition determination to signal issuance must be controlled within seconds.

[0097] Specifically, this step is the direct interface between the rule base and the early warning execution agency, and it is located at the end of the entire monitoring process. It relies on the accurate dwell time data and behavior recognition results provided by the preceding steps, and is a key step in transforming the analysis conclusions into actual security intervention actions. It is mainly used in high-risk monitoring areas around critical equipment such as local fans and power distribution cabinets.

[0098] Specifically, this not only significantly improves the response speed and efficiency of handling safety hazards, but also forcibly introduces safety intervention through technical means, greatly reducing the operational accidents or equipment risks that may be caused by personnel staying in violation of regulations for a long time. This effectively promotes the safety management model from post-event traceability to in-event real-time intervention, enhancing the initiative and reliability of the overall safety assurance system.

[0099] S5, according to the preset statistical period Generate multi-dimensional analysis reports.

[0100] Specifically, this step involves a background data processing engine that automatically performs periodic aggregation calculations on the raw monitoring records stored in the database according to multiple preset fixed time scales. Based on the set statistical period parameters, the system extracts key indicators from personnel entry and exit records, behavior recognition results, and early warning logs in 1-hour, 24-hour, and 720-hour time windows, respectively. These data are then integrated through a report generation engine to finally output structured, multi-dimensional analysis reports.

[0101] Furthermore, this function aims to transform the large amount of low-level perception data generated during system operation into decision support information that can be directly used by managers. By providing statistical analysis views at different time granularities, it not only meets the monitoring needs of work teams for real-time operations, but also provides direct data support for daily work arrangements in work areas and monthly safety management summaries in the mine, realizing the effective connection of safety monitoring data from the operational level to the management level.

[0102] Specifically, the statistical period P takes values ​​of {1 hour, 24 hours, 720 hours}, corresponding to short-, medium-, and long-term analysis needs, respectively. The reports cover at least the following dimensions: time, space (e.g., different device regions), personnel, and behavioral events. The system typically executes this batch processing task at a specific time after the end of each statistical period to avoid peak business hours and ensure real-time system performance.

[0103] Specifically, this step is a crucial step in realizing the value of the system's data. The generated reports directly serve the coal mine's daily safety meetings, operational procedure optimization, and periodic safety audits. These data-driven analytical conclusions provide objective evidence for assessing the safety status of local sectors, personnel behavior trends, and the effectiveness of prevention and control measures.

[0104] Specifically, it elevates fragmented real-time alerts and identification records into systematic operational insights, helping managers grasp the security dynamics of a sector from both macro-trend and micro-detail levels. This supports them in making more forward-looking and scientific management decisions, ultimately driving a systematic shift in security management from passive response to proactive prevention.

[0105] S6, for the frequency of personnel entry included in the report content. Average length of stay and the number of violations Perform statistical analysis.

[0106] Specifically, the system uses Structured Query Language to count and statistically analyze personnel entry and exit records to obtain the frequency of personnel entry. The arithmetic mean of the single stay duration series is used to obtain the average stay duration. The number of violations is counted by querying the total number of confirmed violations in the warning log. These calculations are performed automatically through predefined stored procedures and statistical functions, ensuring consistency and efficiency in data processing.

[0107] Specifically, this statistical analysis function transforms raw monitoring data into key performance indicators (KPIs) that can be directly used for safety management decisions. By tracking entry frequency, managers can understand the activity level of personnel in different areas; average dwell time provides a basis for assessing operational efficiency; and the number of violations directly reflects the safety compliance status within the area. This quantitative assessment method shifts safety management from qualitative judgment to quantitative management, significantly improving the level of management precision.

[0108] Furthermore, the three statistical indicators have clear mathematical definitions: frequency of entry. This refers to the total number of times people entered the monitoring area within the statistical period; and the average length of stay. The number of violations is obtained by summing up the duration of all individual stays and dividing by the total number of entries. This represents the cumulative number of violations confirmed and recorded by the system. The system controls the statistical error of these indicators within an acceptable range to ensure the accuracy and reliability of the data.

[0109] Specifically, the statistical analysis results generated in this step primarily serve multiple aspects of coal mine safety management. These quantified indicators provide a basis for discussion in safety meetings, offer data support for optimizing operating procedures, and provide an objective basis for safety performance evaluation. Managers can promptly identify abnormal trends and take targeted measures based on comparative analysis of indicators across different periods and regions.

[0110] Specifically, the system enables in-depth value mining of monitoring data. These statistically processed indicators not only intuitively reflect the safe operation status of local sectors, but more importantly, lay the foundation for establishing a data-driven safety management model. This system allows safety management to make decisions based on objective data, realizing a shift from experience-based management to scientific management, and significantly improving the professionalism and decision-making efficiency of coal mine safety management.

[0111] This invention discloses a method for personnel identification in local sector areas based on multimodal data fusion. By constructing a multimodal perception system integrating infrared sensing, visual recognition, and positioning verification, it achieves accurate identification and intelligent judgment of personnel identity and operational behavior in key underground areas. This method effectively overcomes the core problems of existing technologies, such as low identification efficiency, large subjective errors, and delayed early warnings due to reliance on manual monitoring. It realizes intelligent processing throughout the entire process, from collaborative data collection, dynamic boundary calibration, feature fusion recognition to automatic judgment of behavioral compliance. This significantly improves the automation level, identification accuracy, and real-time early warning capabilities of safety monitoring in local sector areas of coal mines, providing reliable technical support for building an active safety protection system for high-risk underground areas.

[0112] Example 2 To achieve the above invention, embodiments of the present invention also provide a local sector area personnel identification system based on multimodal data fusion, such as... Figure 2 As shown, it includes: In embodiments of this invention, image recognition and area management are integrated: traditional underground personnel identification often relies on a single visible light camera, which is greatly affected by dust and low light conditions. This invention fuses infrared and visible light camera data, combined with information from the coal mine personnel positioning system, to construct an illumination-invariant feature extraction model. Compared to existing recognition methods based on geometric features or template matching, this invention maintains a 95% recognition rate even in low light conditions, and achieves closed-loop management of identity verification and behavior recording by binding the positioning system ID.

[0113] In embodiments of the present invention, rapid personnel detection and identification (such as...) Figure 3 As shown in the diagram: Existing area monitoring methods mostly use fixed electronic fences, while this method innovatively uses image analysis to automatically calibrate the boundaries of key equipment areas such as the main unit of the inverter and the distribution cabinet, and establishes a rule base for the association between personnel actions and equipment status. When personnel are detected approaching the inverter, an operation compliance judgment is automatically triggered, realizing a technological leap from "area intrusion detection" to "behavioral intent prediction". This dynamic calibration algorithm adaptively adjusts the monitoring range through key point detection, solving the monitoring blind spot problem caused by equipment movement or installation offset.

[0114] In embodiments of this invention, online analysis of personnel positioning system data is implemented: Based on personnel identification, this invention further integrates personnel positioning system data to achieve online analysis of personnel entering the local sector area. By binding coal mine personnel positioning system information, the system can automatically compare the number of personnel entering and record detailed information such as the personnel's name, team, and job type. This integration not only improves the accuracy and completeness of the data but also provides managers with a more comprehensive view of personnel management. Through online analysis, managers can understand the personnel composition and activities in the local sector area in real time, providing strong data support for decision-making.

[0115] In embodiments of this invention, the system features automated management and report export: By setting monitoring areas and personnel identification algorithms, the system can achieve 24 / 7, fully automated personnel monitoring and management of local ventilation fans. Simultaneously, the system also has a report export function, capable of automatically generating daily, weekly, and monthly analytical reports to analyze the personnel management status and duration of multiple local ventilation fans underground at different times. This automated management and data analysis function not only significantly reduces the workload of personnel but also improves management efficiency and accuracy.

[0116] In embodiments of this invention, online automatic management of local ventilation fan areas is achieved: By employing image recognition technology and personnel positioning system data, this invention realizes online automatic management of the area, which not only improves management efficiency and accuracy but also provides strong protection for the safe operation of local ventilation fans. Furthermore, the application of this invention promotes the development of local fan area management towards intelligence and automation, providing valuable insights for technological advancements and application promotion in related fields.

[0117] This invention discloses a personnel identification system for local sector areas based on multimodal data fusion. By constructing a multimodal perception system integrating infrared sensing, visual recognition, and positioning verification, it achieves accurate identification and intelligent judgment of personnel identity and operational behavior in key underground areas. This system effectively overcomes the core problems of existing technologies, such as low identification efficiency, large subjective errors, and delayed early warnings due to reliance on manual monitoring. It achieves intelligent processing throughout the entire process, from collaborative data collection, dynamic boundary calibration, feature fusion recognition to automatic judgment of behavioral compliance. This significantly improves the automation level, identification accuracy, and real-time early warning capabilities of safety monitoring in local sector areas of coal mines, providing reliable technical support for building an active safety protection system for high-risk underground areas.

[0118] Example 3 To achieve the above invention, embodiments of the present invention also provide specific steps for a method for identifying people in a local sector based on multimodal data fusion, including: By acquiring video signals from local ventilation fans in the mine, image processing methods are used to mark the monitoring area. A personnel recognition algorithm is employed to identify in real time whether personnel have entered the marked area. When personnel enter the marked area, the entry time is automatically recorded, and by linking this data to the personnel positioning system, the system analyzes the personnel's information in real time, including name, job type, and team affiliation. When personnel leave the area, the departure time is automatically recorded, automatically creating an area entry record. This system aims to solve the problems of insufficient manual recording for long-term monitoring of local ventilation fans and the inability to accurately record personnel entering the monitored area, achieving 24 / 7, fully automated personnel monitoring and management of key areas of local ventilation fans. The specific steps are as follows: S101: Call the video monitoring signal of the key monitoring scene of the local ventilation fan (such as...) Figure 3 As shown in the image, the video includes the location of the local ventilation fan body, power distribution cabinet, frequency converter, etc. The video can be a color video signal or a black and white video signal.

[0119] S102: Perform image analysis on the monitoring video signal, and delineate the monitoring range of key areas by setting key points in the video image. When personnel enter the monitoring range, the system will automatically prompt the personnel to enter.

[0120] Furthermore, image analysis collects images of personnel and their actions through surveillance video, establishes a database of personnel's work actions, and the image analysis unit compares and analyzes the received real-time image information with the database in the image storage unit and provides prompts.

[0121] S103: Establish a personnel recognition algorithm that can identify human information and personnel movements. When a person appears in the video, it will automatically alert the system. When a person enters a key monitoring area, it can identify that there is a person in the area and record the entry time.

[0122] S104: By binding the coal mine personnel positioning system information, view the current personnel in this area, compare the number of personnel entering, and record the name, team, job type, and other information of the personnel entering this area.

[0123] S105: When personnel leave, record the departure time. If the personnel return to the area within 1 minute, it is considered that they are still moving in the area. Record personnel entry information continuously.

[0124] S106: Statistically track personnel entry into the area hourly, daily, and monthly, and automatically generate daily and weekly reports on local ventilation fan maintenance.

[0125] This invention discloses a specific step-by-step method for personnel identification in local sector areas based on multimodal data fusion. By constructing an intelligent identification system integrating video surveillance and personnel positioning, it achieves automated personnel monitoring and management in key areas of local sectors. This effectively solves the core problems of traditional manual supervision, such as low efficiency, inaccurate recording, and inability to operate continuously, realizing intelligent management throughout the entire process from area calibration, personnel identification, identity verification to behavior recording. Through the organic combination of dynamic monitoring and data analysis, it significantly improves the automation level and monitoring accuracy of local sector area safety management, providing reliable technical support for the safe operation of key equipment areas in underground coal mines.

[0126] Example 4 To achieve the above invention, such as Figure 4 As shown, this embodiment also provides a local sector area personnel identification device 10 based on multimodal data fusion, the device 10 including: The multimodal data acquisition module 100 is used to call up infrared thermal imaging video signals and visible light video signals of key areas of local ventilation fans, and to acquire real-time personnel data from the coal mine personnel positioning system. The equipment boundary calibration module 200 is used to dynamically calibrate the boundaries of the monitoring areas of the local fan body, power distribution cabinet and frequency converter through key point detection algorithm, and establish a rule library for the association between personnel actions and equipment status; The feature fusion and recognition module 300 is used to fuse infrared thermal radiation contour features and visible light safety helmet reflective strip features for personnel identification, verify identity by combining the coal mine personnel positioning system ID, and record personnel information and time of entry into the monitoring area. The spatiotemporal trajectory analysis module 400 is used to determine the continuity of personnel operations based on spatiotemporal trajectory analysis. If a person returns within 1 minute after leaving, it is considered a continuous operation. At the same time, the module predicts the compliance of the operation and triggers an early warning by associating the rule base.

[0127] In one embodiment of the present invention, it further includes: a statistical report generation module, used to generate reports according to a preset statistical period. Generate multi-dimensional analysis reports; the report analysis module is used to analyze the frequency of personnel entry included in the report content. Average length of stay and the number of violations Perform statistical analysis.

[0128] This invention discloses a personnel identification device for local sector areas based on multimodal data fusion. By constructing a collaborative architecture of a multimodal perception system and an intelligent analysis module, it achieves accurate identification and closed-loop management of personnel identity and operational behavior in key underground areas. This device effectively overcomes the core shortcomings of traditional single-monitoring methods, such as large blind spots, poor environmental adaptability, and insufficient early warning capabilities. It achieves fully automated processing from data acquisition, boundary calibration, feature fusion to behavior analysis. This significantly improves the system integration and intelligent decision-making level of safety monitoring in local sector areas of coal mines, providing reliable equipment support for building an active protection system for high-risk underground areas.

[0129] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 5 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads the executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the local sector area personnel identification method based on multimodal data fusion described above.

[0130] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for identifying personnel in a local sector area based on multimodal data fusion as described in the foregoing embodiments.

[0131] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0132] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for identifying people in a local sector based on multimodal data fusion, characterized in that, include: S1 calls up the infrared thermal imaging video signal and visible light video signal of the key area of ​​the local ventilation fan, and obtains the real-time personnel data of the coal mine personnel positioning system; S2 uses a key point detection algorithm to dynamically define the boundaries of the monitoring areas of the local fan body, distribution cabinet and frequency converter, and establishes a rule base for the association between personnel actions and equipment status; S3 integrates infrared thermal radiation profile features with visible light safety helmet reflective strip features for personnel identification, verifies identity by combining coal mine personnel positioning system ID, and records personnel information and time of entry into the monitoring area; S4 determines the continuity of personnel operations based on spatiotemporal trajectory analysis. If a person returns within 1 minute after leaving, it is considered a continuous operation. At the same time, it predicts the compliance of the operation and triggers an early warning by associating the rule base.

2. The method as described in claim 1, characterized in that, The process of calling up infrared thermal imaging video signals and visible light video signals of key areas of local ventilation fans, and acquiring real-time personnel data from the coal mine personnel positioning system, includes: S11 uses an infrared thermal imaging camera to collect the thermal radiation contour features of the human body and uses a Gaussian mixture model for background separation. S12 performs histogram equalization on the visible light video signal and extracts the HSV color features of the reflective strips on the safety helmet.

3. The method as described in claim 1, characterized in that, The boundary of the monitoring area of ​​the local fan body, distribution cabinet, and frequency converter is dynamically calibrated through a key point detection algorithm, and a rule base for the association between personnel actions and equipment status is established, including: S21, Based on the YOLOv5 improved model, key points of the device body are identified, and dynamic monitoring boundaries are generated by the minimum bounding rectangle algorithm; S22, construct a rule base containing rules that trigger compliance judgments when the device is near the frequency converter and the dwell time is greater than 30 seconds, and use a decision tree for behavior classification.

4. The method as described in claim 1, characterized in that, The method integrates infrared thermal radiation profile features with visible light safety helmet reflective strip features for personnel identification, verifies identity using the coal mine personnel positioning system ID, and records personnel information and time of entry into the monitoring area, including: S31, Establish an illumination-invariant feature extraction model, and fuse thermal imaging contour features through feature stitching. With visible light color characteristics ; S32, uses the Euclidean distance algorithm to compare the positioning system ID with the recognition result, when the matching degree... Secondary verification is triggered at that time.

5. The method as described in claim 1, characterized in that, The method of determining the continuity of personnel operations based on spatiotemporal trajectory analysis considers a person's return within one minute of leaving as a continuous operation. Simultaneously, it uses a rule base to predict operational compliance and trigger warnings, including: S41, Calculate the time personnel spend in the monitored area. ; S42, when If the standard operating procedure is not followed within seconds, a level three warning will be triggered via an audible and visual alarm device.

6. The method as described in claim 1, characterized in that, Also includes: S5, according to the preset statistical period Generate multi-dimensional analysis reports; S6, for the frequency of personnel entry included in the report content. Average length of stay and the number of violations Perform statistical analysis.

7. A local sector area personnel identification device based on multimodal data fusion, characterized in that, include: The multimodal data acquisition module is used to call up infrared thermal imaging video signals and visible light video signals of key areas of local ventilation fans, and to acquire real-time personnel data from the coal mine personnel positioning system; The equipment boundary calibration module is used to dynamically calibrate the boundaries of the monitoring areas of the local fan body, power distribution cabinet and frequency converter through key point detection algorithm, and establish a rule library for the association between personnel actions and equipment status; The feature fusion and recognition module is used to fuse infrared thermal radiation contour features and visible light safety helmet reflective strip features for personnel identification, verify identity by combining the coal mine personnel positioning system ID, and record personnel information and time of entry into the monitoring area; The spatiotemporal trajectory analysis module is used to determine the continuity of personnel operations based on spatiotemporal trajectory analysis. If a person returns within 1 minute after leaving, it is considered a continuous operation. At the same time, the module predicts the compliance of the operation and triggers an alert by associating the rule base.

8. The apparatus as claimed in claim 7, characterized in that, Also includes: The statistical report generation module is used to generate reports based on preset statistical periods. Generate multi-dimensional analysis reports; The report analysis module is used to analyze the frequency of personnel entry in the report content. Average length of stay and the number of violations Perform statistical analysis.

9. An electronic device, comprising: processor; The memory stores executable instructions; when the processor executes the instructions, it implements the method for identifying personnel in a local sector area based on multimodal data fusion as described in any one of claims 1-6.

10. A computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for identifying personnel in a local sector area based on multimodal data fusion as claimed in any one of claims 1-6.

Images