Non-coal mine multi-source video data fusion and intelligent behavior recognition management system

Abstract

The invention belongs to the technical field of non-coal mine safety production supervision, and provides a non-coal mine multi-source video data fusion and intelligent behavior recognition management system which comprises a data access module, a video structured processing module, a data fusion module, a behavior recognition module, an archive construction module, an early warning module and a dynamic risk assessment module. Multi-source heterogeneous video data are converged through a standard interface, real-time structured processing is performed by using a target detection model based on a Transform architecture and fused with a non-coal mine scene pre-training weight, and accurate detection, file construction, intelligent early warning and dynamic risk assessment of safety production violation behaviors are realized through data fusion and behavior recognition in a specific scene. According to the invention, full-process intelligent supervision of non-coal mine safety production data fusion-intelligent identification-accurate early warning-dynamic evaluation can be realized, and the risk prevention and control capability and management efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of safety production supervision technology in non-coal mines, specifically to a multi-source video data fusion and intelligent behavior recognition management system for non-coal mines. Background Technology

[0002] Non-coal mines, as the core scenario for mineral resource development, have complex production environments (encompassing confined underground spaces, blasting areas, and transport ramps), with dense interactions between personnel, vehicles, and equipment. They are also affected by natural environmental factors (heavy rain, dense fog) and fluctuations in work intensity, resulting in numerous safety risks and significant management challenges. Currently, safety supervision in non-coal mines mainly relies on traditional video surveillance systems and manual inspections, but existing technologies have the following prominent problems: First, the internal and external monitoring and sensing equipment (such as ordinary cameras, infrared thermal imaging equipment, and millimeter-wave radar) of non-coal mining enterprises often come from different manufacturers and have inconsistent interfaces (lacking compatibility with standards such as GB / T28181 and GA / T669-2008).

[0003] Secondly, traditional video surveillance relies heavily on real-time human monitoring, which leads to high labor costs, susceptibility to missed or incorrect judgments due to fatigue, and response delays. For example, the timeliness of manual identification and intervention is difficult to guarantee for violations requiring rapid response, such as exceeding the limit for the number of personnel in low-light underground environments or illegal intrusion into blasting operation areas.

[0004] Third, the environmental characteristics of different operating scenarios in non-coal mines (such as low light in confined underground spaces and high dynamic environments on transport ramps) vary significantly. Existing target detection models are mostly trained for general scenarios and do not integrate parameters specific to non-coal mine scenarios, resulting in low recognition accuracy in specific scenarios (such as difficulty in accurately capturing behaviors such as people standing still or vehicles driving illegally).

[0005] Fourth, existing risk assessments are mostly based on single violations (such as judging risk solely based on "not wearing a safety helmet"), without taking into account dynamic factors such as real-time environmental data (weather, lighting) and work intensity data (personnel / vehicle distribution density, work duration), resulting in one-sided risk assessment results that fail to reflect the actual safety situation.

[0006] Fifth, the records of violations are mostly scattered logs, lacking hierarchical management of the enterprise's main database and target sub-databases and association with risk levels. The early warning push relies on a single rule and does not combine historical violation data (such as multiple violations by the same person) and scenario complexity (such as high-priority response required in highly dynamic scenarios), resulting in low accuracy of early warnings and poor response efficiency.

[0007] The aforementioned problems make it difficult to achieve closed-loop management of non-coal mine safety production supervision, which includes pre-event prevention, in-event intervention, and post-event traceability. There is an urgent need for an integrated management system that can achieve multi-source data fusion, intelligent behavior recognition, and dynamic risk assessment to improve the intelligence and precision of safety production supervision. Summary of the Invention

[0008] This invention provides a multi-source video data fusion and intelligent behavior recognition management system for non-coal mines, aiming to solve the regulatory challenges in existing non-coal mine operation scenarios, such as fragmented video data, lagging behavior recognition, and static risk assessment. By constructing a full-process technical architecture of data fusion, intelligent recognition, dynamic assessment, and closed-loop supervision, it achieves intelligent safety management and control of all scenarios, all elements, and all time periods of non-coal mine production.

[0009] This invention provides a multi-source video data fusion and intelligent behavior recognition management system for non-coal mines, comprising: The data access module is used to access and aggregate multi-source heterogeneous video data from inside and outside non-coal mining enterprises through a standard interface; The video structuring module is used to perform real-time structuring processing on people, vehicles, and objects in multi-source heterogeneous video data using a target detection model based on the Transformer architecture and incorporating pre-trained weights and dynamic attention mechanisms for non-coal mine production operation scenarios, and to extract key information about people, vehicles, and objects. The data fusion module is used to correlate and fuse key information to generate fused data; The behavior recognition module is used to perform in-depth mining and correlation analysis on the fused data to identify at least one safety production violation among not wearing a safety helmet, illegal intrusion into key areas, exceeding the limit for the number of workers in confined spaces, and workers leaving their posts. The archive construction module is used to build an archive of violations of safety production regulations based on such violations. The early warning module is used to generate early warning information based on violations of safety production regulations, combined with a database of violations of safety production regulations, and push it to management personnel. The dynamic risk assessment module is used to access weather data and work intensity data from non-coal mine sites, adjust the selection of behavior recognition models based on weather data and work intensity data, and dynamically assess the comprehensive risk level of safety production violations. When the comprehensive risk level exceeds a preset threshold, dynamic risk information is generated and pushed to management personnel.

[0010] Furthermore, the data access module supports GB / T28181 and GA / T669-2008 standard interfaces, enabling seamless integration with video surveillance systems deployed internally and externally in non-coal mining enterprises.

[0011] Furthermore, the video structuring module includes a multimodal data association layer. This layer is used to perform cross-modal fusion annotation of the structured target attributes of people, vehicles, and objects with the temperature features in infrared thermal imaging videos and the distance information from millimeter-wave radar in multi-source heterogeneous video data. This generates a multidimensional target profile that includes the spatiotemporal coordinates and temperature status of personnel targets, the behavioral trajectory and temperature distribution of vehicle targets, and the spatiotemporal coordinates and temperature / distance features of object targets. The multidimensional target profile is used to extract key information about people, vehicles, and objects.

[0012] Furthermore, when the data fusion module performs correlation and fusion of key information, it is also used to: obtain the timestamps and geographical location information of video sources in multi-source heterogeneous video data, perform spatiotemporal correlation analysis on the key information of the same person, vehicle, or object target at different timestamps and / or different geographical locations, so as to construct the motion trajectory and behavior sequence of the person, vehicle, or object target, and use the motion trajectory and behavior sequence as fused data.

[0013] Furthermore, the behavior recognition module includes a specific scenario enhanced recognition unit; the specific scenario enhanced recognition unit includes a scenario perception subunit, an enhanced recognition model selection subunit, and a specific scenario safety production violation recognition subunit; The scene perception subunit is used to identify the production operation scene type of non-coal mine corresponding to multi-source heterogeneous video data; The enhanced recognition model selection subunit is used to select a specific behavior recognition model that matches the production operation scenario type from a set of preset enhanced recognition models based on the identified production operation scenario type. The enhanced identification models include a confined space behavior identification model, a blasting operation area behavior identification model, and a transport ramp behavior identification model. The confined space behavior identification model is used to identify safety violations such as exceeding the limit in the number of personnel, vehicles, and objects in low-light environments within confined underground spaces. The blasting operation area behavior identification model is used to identify safety violations such as personnel and vehicles illegally entering key areas. The transport ramp behavior identification model is used to identify safety violations such as personnel entering transport ramps and vehicles not driving according to regulations. The specific scenario safety production violation identification subunit is used to analyze the fused data using a specific behavior identification model to identify safety production violations in production operation scenarios.

[0014] Furthermore, the archive construction module includes an enterprise master archive construction unit, target sub-archive construction units, and hierarchical association units; The enterprise master database construction unit is used to build a summary archive of safety production violations of non-coal mining enterprises based on safety production violations. The target database construction unit is used to build a classified archive database that distinguishes between personnel targets, vehicle targets, and object targets based on violations of safety production regulations. The graded association unit is used to associate and map safety production violations with preset risk levels, match corresponding risk level labels for each safety production violation entry in the summary archive and the classification archive, and establish a dynamic association index between safety production violations and risk levels.

[0015] Furthermore, the early warning module is used to perform the following operations: Based on violations of safe production regulations and their corresponding risk levels, the system retrieves the corresponding early warning rules from a pre-set early warning strategy library. The early warning rules define the content generation method, push priority, push channel combination, and push target hierarchy of the early warning information. Early warning information is generated based on early warning rules, and then pushed to the corresponding level of management personnel according to the push priority and push channel combination.

[0016] Furthermore, before retrieving warning rules from the preset warning strategy library, the warning module also performs the following operations: The complexity of production operation scenarios in non-coal mines is assessed by using target detection and feature extraction algorithms, combined with classification or regression models, to calculate the complexity score of the scenario. If the complexity score is higher than a set threshold, the warning rule with the highest emergency handling priority is selected when the warning rule is retrieved. The complexity of production operation scenarios in non-coal mines is defined as the number of targets, the speed change of moving targets, and the complexity of the scenario structure within the production operation scenario. Based on the constructed safety production violation association model, combinations of violations with potential correlation within the same time period are identified. When a combination of related violations is identified, the warning rules for that combination are retrieved. The safety production violation association model is built based on historical safety production violation data and domain expert experience. Through association rule mining algorithms, the co-occurrence frequency and temporal relationship between different violations are analyzed, and confidence and support evaluation indicators between items of safety production violations are established.

[0017] Furthermore, before generating warning information based on warning rules, the warning module also performs the following operations: The system can identify the identity of the person detected in the current video stream in real time, and retrieve the person's historical violation records confirmed by the system in the past 30 days from the background personnel classification archive based on the identity. Based on timestamps, violations in historical violation records are sorted and categorized chronologically to construct an individual behavior trajectory map that includes the time, location, type, duration, and handling status of the violation. Unsupervised behavioral clustering algorithms are used to analyze individual behavioral trajectory maps to identify high-frequency violation trends or recurring specific violation types; When the cumulative number of violations by an individual exceeds a preset threshold, or the frequency of the same type of violation reaches a set number, a deep warning mechanism is triggered. Generate structured explanatory text and embed it into the body of the warning message. The structured explanatory text includes: personnel identity information, total number of violations in the past 30 days, distribution ratio of each major type of violation, description of repetitive behavior patterns, and comprehensive risk level assessment results. The individual risk weight coefficient of the person is dynamically adjusted based on the structured explanatory text, and the sensitivity threshold for judging the behavior of the person in subsequent intelligent video analysis is adjusted based on the updated risk weight coefficient.

[0018] Furthermore, the dynamic risk assessment module includes an environmental data access unit, a work intensity analysis unit, a behavior identification and adjustment unit, and a dynamic risk assessment unit; The environmental data access unit is used to access real-time weather data from non-coal mine on-site meteorological stations or IoT sensors. The real-time weather data includes at least information on heavy rain, fog, visibility, and temperature. The task intensity analysis unit is used to statistically analyze the task intensity data of personnel and vehicle targets in the task area based on the structured processing results of multi-source heterogeneous video data; the task intensity data includes distribution density and task duration. The behavior recognition adjustment unit is used to identify safety production violations based on weather data and work intensity data, utilizing the severe weather work behavior recognition model and the high work intensity behavior recognition model from multiple preset enhanced recognition models. Specifically, when the weather data is heavy rain or fog, the severe weather work behavior recognition model is used to identify running and slipping behavior of personnel targets; when the distribution density of personnel / vehicle targets in the work intensity data exceeds the distribution density threshold within a set period, or the work duration exceeds the preset duration, the high work intensity behavior recognition model is used to identify personnel who are stationary or have abnormal postures. The dynamic risk assessment unit is used to dynamically calculate the comprehensive risk level based on the safety production violations identified by the behavior recognition and adjustment unit, combined with real-time weather data and work intensity data, through a weighted risk fusion algorithm. When the comprehensive risk level exceeds a preset threshold, dynamic risk information including the causes of the risk and disposal suggestions is generated and pushed to the management personnel.

[0019] Compared with existing technologies, this invention has the following advantages and beneficial effects: First, by constructing a unified data access standard and intelligent processing framework, it effectively breaks down the barriers of data dispersion and heterogeneous formats in traditional video surveillance systems for non-coal mines, achieving full-domain aggregation and efficient integration of internal and external multi-source video data, providing a complete data foundation for subsequent intelligent analysis; Second, by combining the Transformer architecture with pre-trained weights and dynamic attention mechanisms for non-coal mine scenarios, it improves the accuracy and real-time performance of target detection in complex operating environments, making the extraction of key information about people, vehicles, and objects more accurate and reliable; Third, through multi-dimensional data association and fusion and deep behavior mining, it can accurately identify various typical safety production violations, overcoming the limitations of traditional monitoring relying on manual inspections, and realizing the automated and intelligent capture of violations; Fourth, relying on the construction of a systematic archive and multi-dimensional early warning strategies, it forms a regulatory closed loop from violation identification and record-keeping to accurate early warning, improving the standardization and initiative of safety production management; Fifth, through dynamic risk assessment mechanisms and intelligent management methods, it not only reduces the cost of manual supervision but also predicts potential risks in advance, effectively reducing the probability of accidents and providing strong support for the intelligent upgrading of safety production in non-coal mines.

[0020] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 A schematic diagram of the structure of a multi-source video data fusion and intelligent behavior recognition management system for non-coal mines; Figure 2 A schematic diagram of the structure of the enhanced recognition unit for specific scenarios; Figure 3 A schematic diagram of the module structure for building the archive repository. Detailed Implementation

[0023] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0024] This invention provides a multi-source video data fusion and intelligent behavior recognition management system for non-coal mines, such as... Figure 1 As shown, it includes: The data access module is used to access and aggregate multi-source heterogeneous video data from inside and outside non-coal mining enterprises through a standard interface; The video structuring module is used to perform real-time structuring processing on people, vehicles, and objects in multi-source heterogeneous video data using a target detection model based on the Transformer architecture and incorporating pre-trained weights and dynamic attention mechanisms for non-coal mine production operation scenarios, and to extract key information about people, vehicles, and objects. The data fusion module is used to correlate and fuse key information to generate fused data; The behavior recognition module is used to perform in-depth mining and correlation analysis on the fused data to identify at least one safety production violation among not wearing a safety helmet, illegal intrusion into key areas, exceeding the limit for the number of workers in confined spaces, and workers leaving their posts. The archive construction module is used to build an archive of violations of safety production regulations based on such violations. The early warning module is used to generate early warning information based on violations of safety production regulations, combined with a database of violations of safety production regulations, and push it to management personnel. The dynamic risk assessment module is used to access weather data and work intensity data from non-coal mine sites, adjust the selection of behavior recognition models based on weather data and work intensity data, and dynamically assess the comprehensive risk level of safety production violations. When the comprehensive risk level exceeds a preset threshold, dynamic risk information is generated and pushed to management personnel.

[0025] The working principle of the above technical solution is as follows: In order to realize the multi-source video data fusion and intelligent behavior recognition management system for non-coal mines, this invention proposes a data access module to aggregate multi-source heterogeneous video data from inside and outside non-coal mining enterprises. Multi-source heterogeneous video data refers to data from various sources (such as mine monitoring cameras, drone aerial photography, vehicle cameras, etc.) and different data formats / protocols (such as H.264 / H.265 encoding, video streams of different resolutions, RTSP protocol, GB / T28181 standard interface protocol, etc.). Standard interface refers to data access interface that conforms to unified technical specifications (such as ONVIF protocol interface, GB / T28181 standard compatible interface), ensuring that data from different types of devices can be interoperable. For example, a non-coal mine can access underground fixed cameras (real-time video stream), mining vehicle DVRs (stored video), and drone inspection videos (periodic uploads) through this module, and uniformly aggregate them to the system platform. A video structuring module is proposed to perform real-time structuring of people, vehicles, and objects in multi-source heterogeneous video data, extracting key information (such as personnel location, vehicle type, and equipment status). The Transformer architecture is a deep learning model based on a self-attention mechanism, which excels at capturing long-distance dependencies in data and is used for object detection. Pre-trained weights for non-coal mine production operation scenarios refer to model parameters pre-trained for non-coal mine scenarios (such as the dim underground environment, the appearance of mining equipment, and the characteristics of workers' safety helmets / work clothes), which can improve object detection accuracy (avoiding misidentification of rocks as people). The dynamic attention mechanism refers to the model's ability to dynamically adjust its attention based on the scene during detection. Adjusting the focus area (e.g., in underground scenes, focusing on whether personnel are wearing safety helmets, rather than background rocks); structured processing refers to converting unstructured video into structured data (e.g., "Personnel ID: P001, Time: 10:05, Location: No. 3 tunneling face, Action: Not wearing a safety helmet"). For example, the system processes underground video streams, improves the ability to recognize workers wearing reflective clothing through pre-trained weights (to reduce the impact of background interference), and focuses on the analysis of the head area through a dynamic attention mechanism to determine whether a safety helmet is worn, outputting a structured result: "Personnel A, Coordinates (x=120, y=150), Status: Not wearing a safety helmet"; A data fusion module is proposed to correlate and fuse structured key information from different sources, eliminating redundancy, supplementing missing information, and generating unified fused data. Correlation fusion refers to integrating multi-source data based on time, space, and semantic relationships (such as stitching together the trajectory of the same person under different cameras, and the interaction relationship between personnel and equipment). For example, in terms of time correlation, if personnel A is identified as "not wearing a safety helmet" by camera A at 10:00 and "entering a restricted area" by camera B at 10:02, the fusion will generate "personnel P001 did not wear a safety helmet and entered a restricted area between 10:00 and 10:02". In terms of spatial correlation, the data "abnormality of No. 2 mine pile" in the drone inspection video is fused with the data "shovel truck approaching No. 2 mine pile" in the ground camera to determine "the shovel truck may be operating in violation of regulations". A behavior recognition module is proposed to perform in-depth mining of fused data and identify violations of safety production regulations (such as not wearing a safety helmet, illegal intrusion, etc.). In-depth mining and correlation analysis refers to using rule engines and machine learning models to combine multi-dimensional data to determine whether a behavior is in violation (such as judging whether a person is off duty by combining "personnel location + time + historical behavior"). A database construction module is proposed. Based on the identified violations, a database of safety production violations is built to record historical violation information. The database content includes: violation type, time, location, personnel / equipment involved, and handling result (e.g., "2023-10-01, No. 3 tunneling face, personnel P001 not wearing a safety helmet, warning issued"). An early warning module is proposed, which combines real-time violations with historical records to generate early warning information and push it to management personnel; A dynamic risk assessment module is proposed. By accessing weather data (such as rainfall, wind speed, and visibility) collected by meteorological stations deployed in non-coal mines and work intensity data (such as the number of people working underground during the shift, blasting frequency, and heavy equipment operating time) obtained through the work scheduling system and equipment operation logs, the selection strategy of the behavior recognition model is dynamically adjusted. For example, in scenarios with heavy rain (rainfall ≥ 50 mm / h) and high underground tunneling work intensity (more than 3 tunneling faces under construction at the same time), the system automatically prioritizes the improved target detection model, which is more adaptable to low light and water mist environments, to improve the accuracy of personnel positioning and safety helmet wearing recognition. In contrast, in sunny weather and low work intensity scenarios where equipment maintenance is the main focus, a lightweight model is switched to improve processing efficiency. Meanwhile, this module quantifies the overall risk level of safety production violations based on weather data and work intensity data. When the calculated overall risk level exceeds a preset threshold (e.g., the threshold is set to 3.0), the system immediately generates dynamic risk information, which includes the current weather conditions, work intensity, specific violation type, overall risk level value, and risk warning (e.g., "Current heavy rain (coefficient 1.5), high-intensity work at tunnel face No. 3 (coefficient 1.3), 2 instances of not wearing safety helmets detected (severity coefficient 1.8), overall risk level 4.23, far exceeding the threshold 3.0, posing a risk of fall from height, it is recommended to immediately suspend work and investigate safety hazards"). This information is then pushed to relevant management personnel through multiple channels such as the system platform, mobile APP, and SMS.

[0026] The beneficial effects of the above technical solution are as follows: By adopting the solution provided in this embodiment, and through a target detection model based on the Transformer architecture, combined with pre-trained weights and a dynamic attention mechanism for non-coal mine scenarios, the accuracy and real-time performance of extracting key information about people, vehicles, and objects in complex production environments are significantly improved. This effectively solves the problems of low recognition accuracy and low processing efficiency caused by special scenarios and heterogeneous data in traditional monitoring. The data fusion module breaks down information barriers between different video data sources through multi-dimensional correlation and fusion, forming complete and coherent fused data, providing a high-quality data foundation for subsequent behavior recognition. The behavior recognition module accurately identifies typical safety production violations in non-coal mines, realizing a shift from passive monitoring to proactive early warning, while ensuring safety... The establishment of a production violation archive provides enterprises with traceable and analyzable historical data support for safety production management, helping them to summarize violation patterns and optimize management strategies. The early warning module can promptly push violation information to management personnel, buying valuable time for rapid response and handling of safety hazards, thereby effectively reducing the risk of non-coal mine production safety accidents and improving the overall safety production management level of enterprises. The dynamic risk assessment module integrates real-time violation data, historical accident cases, equipment operating status, and environmental monitoring information to build a dynamically updated risk assessment model, which can calculate and visualize the safety risk level of each work area in the mine in real time, providing a scientific basis for managers to formulate differentiated inspection plans and key points for hazard investigation.

[0027] In one embodiment, the data access module supports GB / T28181 and GA / T669-2008 standard interfaces to achieve seamless integration with video surveillance systems deployed internally and externally in non-coal mining enterprises.

[0028] The working principle of the above technical solution is as follows: The data access module receives connection requests from internal and external video surveillance systems of non-coal mining enterprises through a built-in interface protocol stack conforming to GB / T28181 and GA / T669-2008 standards. Both parties complete identity authentication, device information exchange, and media stream parameter negotiation (such as encoding format, resolution, transmission protocol, etc.) based on standard protocols. After reaching an agreement, the video surveillance system encapsulates real-time video streams or historical video data according to standard formats and transmits them to the data access module via the network. The data access module parses, verifies, and converts the data, ultimately achieving seamless reception and subsequent processing (such as storage and analysis) of the video data. GB / T28181, "Technical Requirements for Information Transmission, Exchange, and Control of Security Video Surveillance Network Systems," specifies network communication protocols, device encoding, and media stream transmission standards between video surveillance devices, ensuring interoperability between surveillance devices from different manufacturers (such as networked surveillance in public security, transportation, and enterprise scenarios). GA / T669-2008, officially titled "Technical Standard for Urban Monitoring and Alarm Network Systems," is a public safety industry standard. This system focuses on the technical requirements of city-level surveillance systems, including device interfaces, data formats, and transmission protocols, and is commonly used in urban security and park monitoring scenarios. The data access module's workflow is as follows: After startup, the module simultaneously listens on the default ports of both GB / T28181 and GA / T669-2008 protocols (e.g., SIP protocol port 5060). Internal systems send registration requests to the module via the GB / T28181 protocol. After verifying the device ID and password, the module negotiates the use of H.264 encoding and RTSP transmission protocol for the video stream. External systems access the system via... The module initiates a connection using the GA / T669-2008 protocol. It parses the device code according to the standard (e.g., "37020000001320000001" represents the region and device type). After confirming permissions, it receives the H.265 encoded video stream (the module automatically converts it to the H.264 format consistent with the internal system, which is transparent to subsequent platforms). Finally, the module transmits the video streams from both systems to the backend management platform. Users can simultaneously view the real-time images of the workshop and entrances / exits on the monitoring screen without manually switching protocols or devices, achieving seamless integration.

[0029] The beneficial effects of the above technical solution are as follows: By adopting the solution provided in this embodiment, the protocol barriers between different video surveillance systems can be effectively broken down, and standardized access to multi-source video data inside and outside non-coal mining enterprises can be realized. This provides high-quality data support for subsequent business functions such as intelligent behavior recognition and risk warning, and improves the overall coordination and intelligence level of safety monitoring in non-coal mines.

[0030] In one embodiment, the video structuring processing module includes a multimodal data association layer. The multimodal data association layer is used to perform cross-modal fusion annotation of the target attributes of people, vehicles, and objects after structuring processing with the temperature features in infrared thermal imaging video and the distance information of millimeter-wave radar in multi-source heterogeneous video data. This generates a multidimensional target profile that includes the spatiotemporal coordinates and temperature status of personnel targets, the behavioral trajectory and temperature distribution of vehicle targets, and the spatiotemporal coordinates and temperature / distance features of object targets. The multidimensional target profile is used to extract key information of people, vehicles, and objects.

[0031] The working principle of the above technical solution is as follows: The multimodal data association layer is responsible for associating and fusing multimodal data of different types and sources. First, it extracts features from the structured human, vehicle, and object target attributes to clarify the basic information of each target, such as the identity of the person, the model and license plate of the vehicle, and the type of the object. Next, this layer acquires temperature feature data from infrared thermal imaging video and distance information data from millimeter-wave radar, and preprocesses this data, including data cleaning, format conversion, and spatiotemporal calibration, to ensure that the different modal data remain consistent in time and space. Then, through a preset fusion algorithm, the target attributes are deeply integrated with the temperature features and distance information. The system employs various correlation methods. For example, when a person is detected, their location's infrared temperature data is simultaneously correlated to determine if there is an abnormal body temperature. This is combined with distance information from millimeter-wave radar to determine the person's relative position to surrounding objects or other individuals. For vehicles, based on their movement trajectory, temperature distribution characteristics of different parts of the vehicle are fused with infrared thermal imaging video to analyze potential overheating or other malfunctions. Millimeter-wave radar distance information can further refine the accuracy of the vehicle's trajectory, particularly in complex environments. For objects, their type, state, and other attributes are combined with temperature characteristics and distance information to assess their safety. This process then determines the risk level. For example, for stored flammable and explosive materials, temperature characteristics are monitored to detect abnormal temperature rises, and distance information is used to determine whether the safe distance between the materials and fire sources or high-temperature equipment meets the standards. Finally, all the processed information is integrated to generate a multi-dimensional target profile, including the spatiotemporal coordinates of personnel targets, the behavioral trajectories of vehicle targets, and the risk level of object targets. This lays the foundation for the rapid extraction and efficient application of key information on people, vehicles, and objects. Structured processing refers to organizing, classifying, and standardizing information related to people, vehicles, and objects, making it an organized and understandable data format. For example, it clarifies target attributes such as personnel identification, vehicle model and license plate, and object type. Target attributes refer to… The characteristic information presented by people, vehicles, and objects after structured processing, such as people's identification, vehicle models and license plates, and object types and states, serves as the foundational information for subsequent fusion with other modal data. Infrared thermal imaging video refers to generating video images by detecting the infrared radiation emitted by objects. Temperature features can reflect the temperature distribution of objects and can be used to determine conditions such as abnormal body temperature in people and overheating of vehicle parts. Cross-modal fusion annotation refers to fusing data from different modalities (such as structured data modality, infrared thermal imaging modality, and millimeter-wave radar modality) and adding labeling information to the fused data to generate multi-dimensional target profiles, thereby achieving deep association and comprehensive utilization of data from different modalities.A multidimensional target profile is a file that integrates multidimensional information such as the spatiotemporal coordinates of personnel targets, the behavioral trajectory of vehicle targets, and the risk level of object targets. This provides a foundation for the subsequent extraction of key information about people, vehicles, and objects. Spatiotemporal coordinates represent the specific location information of personnel targets in time and space. The time dimension records the moment the target appears or its behavior occurs, while the spatial dimension determines the target's geographical location, used to accurately describe personnel activities. Behavioral trajectories refer to the movement path information of vehicles over a certain period. Combined with millimeter-wave radar distance information, this information can correct the positioning accuracy in complex environments, assisting in the analysis of vehicle driving status and potential risks. Risk levels are determined by assessing the safety of an object's state based on its type and state, combined with the temperature characteristics of infrared thermal imaging and the distance information of millimeter-wave radar. This level is used to determine whether an object is in a safe state, such as whether flammable or explosive materials pose a safety hazard.

[0032] The beneficial effects of the above technical solution are as follows: By adopting the solution provided in this embodiment, the deep fusion of the attributes of people, vehicles and objects with infrared temperature features and millimeter-wave radar distance information is achieved through the multimodal data association layer, breaking down the information barriers between different modal data, and making the originally independent structured data and unstructured perception data form an organic whole.

[0033] In one embodiment, when the data fusion module performs correlation fusion on key information, it is also used to: obtain the timestamps and geographical location information of video sources in multi-source heterogeneous video data, perform spatiotemporal correlation analysis on the key information of the same person target, vehicle target, or object target at different timestamps and / or different geographical locations, so as to construct the motion trajectory and behavior sequence of the person target, vehicle target, or object target, and use the motion trajectory and behavior sequence as fused data.

[0034] The working principle of the above technical solution is as follows: The data fusion module first performs time synchronization and spatial calibration on each video source in the multi-source heterogeneous video data. By aligning timestamps, it ensures that the same target data collected by different devices are in a unified time dimension. Based on geographical location information, a spatial coordinate system is established to realize the spatial association of video sources in different locations. Next, for personnel targets, vehicle targets, and object targets, key features are extracted at different timestamps and geographical locations, such as the outline of a person's body and the color of their clothing, the model and license plate number of a vehicle, and the shape and size of an object. Then, a spatiotemporal correlation algorithm is used to match and connect these scattered key information. By calculating the similarity of target features and the continuity of motion trajectory, it is determined whether the targets appearing in different video sources are the same entity, thereby constructing the complete motion trajectory of the target, including the starting point, waypoints, ending point, and the time information corresponding to each point. At the same time, combined with the target's behavior during the motion process, such as people walking, stopping, and operating equipment, vehicles driving, turning, and parking, and objects moving and changing states, an ordered sequence of behaviors is formed. Finally, the constructed motion trajectories and behavior sequences are integrated into structured fused data, providing comprehensive and coherent data support for subsequent upper-level applications such as intelligent behavior recognition and risk warning. Among these, timestamps refer to the time markers of an event recorded in the video data, used to synchronize the time of the same target data collected from different video sources, ensuring they are on the same time dimension, facilitating subsequent analysis of key information about the target at different time points. Geographical location information represents the location of the video source; based on this information, a spatial coordinate system can be established to achieve spatial correlation between video sources at different locations, enabling information about the same target from different video sources to correspond spatially. Spatiotemporal correlation analysis refers to analyzing key information about the same person, vehicle, or object target at different timestamps and / or different geographical locations, determining the similarity of target features and the continuity of motion trajectories by calculating similarity of target features, etc. To determine whether targets appearing in different video sources belong to the same entity, we can construct their motion trajectory and behavior sequence. Motion trajectory refers to the target's movement path in space, including the starting point, intermediate points, ending point, and corresponding time information. This is constructed by linking relevant information about the same target in different video sources through spatiotemporal correlation analysis. Behavior sequence refers to an ordered sequence formed by combining the target's actions during movement, such as walking, driving, or moving objects, according to chronological order. Time synchronization ensures that data on the same target collected from different video sources are aligned to a unified time dimension through timestamp alignment, which is the foundation for effective analysis of target information at different time points. Spatial calibration establishes a spatial coordinate system based on geographical location information, spatially correlating data from video sources at different locations to ensure accurate correspondence of spatial information about the same target across different video sources.

[0035] The beneficial effects of the above technical solution are as follows: By using the solution provided in this embodiment to perform time synchronization and spatial calibration on multi-source heterogeneous video data, the time deviation and spatial misalignment between different video sources can be effectively eliminated, laying the foundation for the spatiotemporal correlation analysis of the target.

[0036] In one embodiment, the behavior recognition module includes a scene-specific enhanced recognition unit; such as Figure 2 As shown, the specific scenario enhanced recognition unit includes a scenario perception subunit, an enhanced recognition model selection subunit, and a specific scenario safety production violation recognition subunit; The scene perception subunit is used to identify the production operation scene type of non-coal mine corresponding to multi-source heterogeneous video data; The enhanced recognition model selection subunit is used to select a specific behavior recognition model that matches the production operation scenario type from a set of preset enhanced recognition models based on the identified production operation scenario type. The enhanced identification models include a confined space behavior identification model, a blasting operation area behavior identification model, and a transport ramp behavior identification model. The confined space behavior identification model is used to identify safety violations such as exceeding the limit in the number of personnel, vehicles, and objects in low-light environments within confined underground spaces. The blasting operation area behavior identification model is used to identify safety violations such as personnel and vehicles illegally entering key areas. The transport ramp behavior identification model is used to identify safety violations such as personnel entering transport ramps and vehicles not driving according to regulations. The specific scenario safety production violation identification subunit is used to analyze the fused data using a specific behavior identification model to identify safety production violations in production operation scenarios.

[0037] The working principle of the above technical solution is as follows: The scene perception subunit first analyzes the input multi-source heterogeneous video data, extracting environmental features (such as light intensity, spatial structure, equipment type, etc.) and dynamic features (such as personnel activity trajectory, equipment operating status, etc.) from the video frames. Combined with a preset non-coal mine production operation scene feature library, it accurately determines the scene type corresponding to the current video data, such as an underground tunneling face, an open-pit blasting area, or an ore transport ramp. The enhanced recognition model selection subunit performs matching and retrieval in a preset model library based on the scene type result output by the scene perception subunit. If the scene is identified as an underground confined space, the underground confined space behavior recognition model is automatically retrieved; if it is a blasting operation area, the blasting operation area behavior recognition model is activated; if it is a transport ramp, the model is activated. If the specific scenario safety production violation identification subunit is selected, the transportation ramp behavior recognition model will be used. Subsequently, the fusion data processed by the data fusion module will be input into the selected specific behavior recognition model. This model will conduct a special analysis on the typical violation characteristics of the corresponding scenario. For example, the underground confined space behavior recognition model will focus on detecting whether the number of personnel, vehicles and objects exceeds the safety carrying capacity threshold of the space under low light conditions. The blasting operation area behavior recognition model will identify whether there are unauthorized personnel or vehicles entering the warning area through the target detection algorithm. The transportation ramp behavior recognition model will monitor in real time whether personnel enter the ramp without authorization and whether vehicles have speeding, driving against the flow of traffic and other violations. Finally, it will output the specific safety production violation categories and related parameter information in the production operation scenario.

[0038] The beneficial effects of the above technical solution are as follows: By adopting the solution provided in this embodiment, the scene perception subunit and the enhanced recognition model selection subunit work together to achieve accurate adaptation to different production operation scenarios in non-coal mines, avoiding the problem of insufficient recognition accuracy of a single model in complex scenarios.

[0039] In one embodiment, such as Figure 3 As shown, the archive construction module includes an enterprise master archive construction unit, a target sub-archive construction unit, and a hierarchical association unit; The enterprise master database construction unit is used to build a summary archive of safety production violations of non-coal mining enterprises based on safety production violations. The target database construction unit is used to build a classified archive database that distinguishes between personnel targets, vehicle targets, and object targets based on violations of safety production regulations. The graded association unit is used to associate and map safety production violations with preset risk levels, match corresponding risk level labels for each safety production violation entry in the summary archive and the classification archive, and establish a dynamic association index between safety production violations and risk levels.

[0040] The working principle of the above technical solution is as follows: The archive construction module systematically integrates safety production violation data across the entire mine through the enterprise master archive construction unit, establishing a structured storage framework according to multiple dimensions such as time, scenario, and violation type, forming a comprehensive archive covering the entire enterprise, realizing centralized management and traceability of various violations; the target sub-archive construction unit performs fine classification for different subjects. The personnel target archive records data such as the identity information of violators, violation frequency, and historical violations; the vehicle target archive associates vehicle number, affiliated unit, and violation driving records; and the object target archive focuses on equipment misuse. Information related to violations, such as improper stacking of materials, is categorized and stored to improve the relevance of subsequent data queries and analyses. The risk level association unit matches each safety production violation with its corresponding risk level based on a preset risk assessment index system. For example, "unauthorized personnel entering the blasting warning zone" is marked as an extremely high risk level, and "vehicle speeding within 10%" is marked as a general risk level. The risk level labels are updated in real time through dynamic association indexes. When the frequency of a violation or the consequences it causes reach the upgrade conditions, the system will automatically adjust its risk level to ensure that the risk level information in the archive is always consistent with the actual safety situation.

[0041] The beneficial effects of the above technical solution are as follows: The solution provided in this embodiment enables systematic and refined management of non-coal mine safety production violations through the construction of a multi-dimensional archive; the establishment of the aggregated archive breaks down the barriers of traditional scattered storage of violation data, providing enterprise management with a macro-level perspective on safety violations across the entire mine, facilitating a rapid grasp of the overall safety situation; the categorized archive achieves precise data accumulation for different target objects, making specialized safety management for personnel, vehicles, and objects more operational; and the dynamic association and indexing mechanism of risk levels allows safety management personnel to intuitively identify high-risk violations, prioritize resource allocation for key rectification, while dynamically updated risk level labels ensure the timeliness and accuracy of safety management, effectively avoiding the problem of missed safety hazard assessments due to fixed risk levels, thereby comprehensively improving the intelligence and precision of non-coal mine safety production management.

[0042] In one embodiment, the early warning module is used to perform the following operations: Based on violations of safe production regulations and their corresponding risk levels, the system retrieves the corresponding early warning rules from a pre-set early warning strategy library. The early warning rules define the content generation method, push priority, push channel combination, and push target hierarchy of the early warning information. Early warning information is generated based on early warning rules, and then pushed to the corresponding level of management personnel according to the push priority and push channel combination.

[0043] The working principle of the above technical solution is as follows: When the early warning module receives a violation of safety production regulations and its matching risk level, it first compares the specific type, time, and location of the violation with the preset early warning strategy library. Based on the risk level, the system accurately locates the matching early warning rule in the early warning strategy library. These rules are pre-set based on the actual needs of non-coal mine safety production management and cover information generation specifications under different risk scenarios. For example, for high-risk violations, the early warning rule may stipulate that the early warning information must include detailed content such as real-time video footage of the violation, the basis for risk level assessment, historical similar violation cases, and rectification suggestions to ensure that management personnel can fully grasp the situation. For medium- and low-risk violations, the content of the early warning information can be appropriately simplified, highlighting only the core violation elements. After determining the early warning rule, the system sorts the early warning information according to the push priority defined in the rule. High-priority early warning information will occupy push resources first, ensuring that emergencies can be handled as soon as possible. The push notification channels are configured based on the responsibilities and commonly used contact methods of management personnel. These channels may include system platform pop-ups, SMS messages, dedicated management app push notifications, emails, and other methods to ensure that warning information reaches management personnel through multiple channels. Simultaneously, the push notification hierarchy strictly adheres to the management authority system. High-risk warning information is directly pushed to the mine-level safety production manager and relevant supervisors, medium-risk warning information is pushed to workshop-level management personnel, and low-risk warning information is pushed to team-level leaders. This achieves precise, tiered delivery of warning information, ensuring that management personnel at all levels can promptly obtain safety warnings relevant to their responsibilities, enabling rapid response and effective measures, thus forming a complete closed-loop management process from violation identification to warning notification.

[0044] The beneficial effects of the above technical solution are as follows: by adopting the solution provided in this embodiment, differentiated early warning management of safety production violations of different risk levels can be achieved through a preset early warning strategy library, which not only ensures the comprehensiveness and timeliness of high-risk early warning information, but also avoids information redundancy caused to management personnel by medium and low-risk early warnings.

[0045] In one embodiment, before retrieving warning rules from a preset warning strategy library, the warning module is also used to perform the following operations: The complexity of production operation scenarios in non-coal mines is assessed by using target detection and feature extraction algorithms, combined with classification or regression models, to calculate the complexity score of the scenario. If the complexity score is higher than a set threshold, the warning rule with the highest emergency handling priority is selected when the warning rule is retrieved. The complexity of production operation scenarios in non-coal mines is defined as the number of targets, the speed change of moving targets, and the complexity of the scenario structure within the production operation scenario. Based on the constructed safety production violation association model, combinations of violations with potential correlation within the same time period are identified. When a combination of related violations is identified, the warning rules for that combination are retrieved. The safety production violation association model is built based on historical safety production violation data and domain expert experience. Through association rule mining algorithms, the co-occurrence frequency and temporal relationship between different violations are analyzed, and confidence and support evaluation indicators between items of safety production violations are established.

[0046] The working principle of the above technical solution is as follows: First, the Faster R-CNN object detection algorithm is used to identify and count various targets in the production operation scene. At the same time, the decision tree algorithm is combined to quantitatively analyze the speed change trend of moving targets and the spatial distribution characteristics of scene structure. Multi-dimensional indicators such as the number of targets, speed change parameters, and structural complexity parameters are input into a preset scoring model to calculate the comprehensive complexity score of the scene. When the score exceeds a set threshold, the system determines that the current scene is in a high-risk state. At this time, the early warning rule retrieval mechanism will automatically prioritize rules with high emergency handling priority to shorten the early warning response time. In the process of identifying combinations of related violations, the safety production violation association model first encodes the violation entries in the historical data. The Apriori algorithm in the association rule mining algorithm is used to mine the co-occurrence patterns of different violations in the time series. The support and confidence between pairs of violations are calculated. When the support and confidence simultaneously meet the preset conditions, it is determined that there is a potentially related combination of violations. The system will monitor the sequence of violations in the current work scenario in real time. Once a combination of related violations is matched, the corresponding special early warning rules will be retrieved immediately to achieve accurate early warning of complex violation risks. Among them, the Faster R-CNN object detection algorithm is an advanced object detection algorithm based on a convolutional neural network (CNN). It mainly consists of a Region Proposal Network (RPN) and a Fast R-CNN detector. The RPN is responsible for generating candidate regions that may contain objects, while Fast R-CNN classifies these candidate regions and performs accurate bounding box regression, thereby realizing the recognition and localization of various objects in the image, which can then be used for counting. The decision tree algorithm is an algorithm that makes decisions based on a tree structure. Each internal node represents a test on an attribute, each branch represents a test output, and each leaf node represents a category or a value. In the text, a decision tree algorithm is used to quantify the velocity change trends of moving targets (such as whether the velocity is accelerating, decelerating, or constant) and the spatial distribution characteristics of scene structure (such as the density of objects in the scene), transforming these complex features into measurable numerical indicators. The scoring model is a pre-defined model that takes multiple dimensions, such as the number of targets, velocity change parameters, and structural complexity parameters, as input. Following predetermined calculation rules, it outputs a score representing the overall complexity of the scene. This score quantifies the overall complexity of the scene and can be used to subsequently assess the scene's risk status. The early warning rule retrieval mechanism activates when the overall scene complexity score exceeds a set threshold, indicating a high-risk state. This mechanism automatically prioritizes rules with high urgency, aiming to trigger corresponding early warnings more quickly in high-risk situations, shortening the early warning response time and enabling relevant personnel to take timely measures. Regarding risk; the association model for safety production violations aims to uncover potential associations between different violations during the safety production process. First, violation entries in historical data are encoded, transforming various types of violations into computer-processable codes. Then, the Apriori algorithm is used to mine co-occurrence patterns of different violations over time. The Apriori algorithm is a classic algorithm for association rule mining, a recursive algorithm based on frequent itemset theory. It scans the dataset to find frequent itemsets that meet the minimum support requirement, and then generates association rules that meet the minimum confidence requirement based on these frequent itemsets. This algorithm is used to mine co-occurrence patterns of different violations over time, calculating the support and confidence between pairs of violations to determine if a potential association exists. Support refers to the proportion of records containing both violations in the entire dataset, measuring the frequency of two violations occurring simultaneously. Higher support indicates a higher frequency of co-occurrence of the two violations in historical data. Confidence refers to the proportion of records containing one violation that also contain the other violation, compared to the total number of records containing the first violation.It represents the probability of one violation occurring given that the other has already occurred, reflecting the strength of the association between the two violations. A associated violation combination refers to a combination of violations where the support and confidence scores calculated using the Apriori algorithm simultaneously meet preset conditions. The system monitors violation sequences in the current work scenario in real time, and once an established associated violation combination is matched, it immediately retrieves the corresponding specific early warning rule. This specific early warning rule is specifically designed for identified associated violation combinations. When the system detects a matching associated violation combination in the work scenario, it immediately retrieves this rule to achieve accurate early warning of complex violation risks, enabling timely measures to prevent risks.

[0047] The beneficial effects of the above technical solution are as follows: by adopting the solution provided in this embodiment, it is possible to achieve intelligent and precise retrieval of early warning rules through dynamic assessment of the complexity of production operation scenarios and in-depth mining of the correlation between violations.

[0048] In one embodiment, before generating warning information based on warning rules, the warning module is also used to perform the following operations: The system can identify the identity of the person detected in the current video stream in real time, and retrieve the person's historical violation records confirmed by the system in the past 30 days from the background personnel classification archive based on the identity. Based on timestamps, violations in historical violation records are sorted and categorized chronologically to construct an individual behavior trajectory map that includes the time, location, type, duration, and handling status of the violation. Unsupervised behavioral clustering algorithms are used to analyze individual behavioral trajectory maps to identify high-frequency violation trends or recurring specific violation types; When the cumulative number of violations by an individual exceeds a preset threshold, or the frequency of the same type of violation reaches a set number, a deep warning mechanism is triggered. Generate structured explanatory text and embed it into the body of the warning message. The structured explanatory text includes: personnel identity information, total number of violations in the past 30 days, distribution ratio of each major type of violation, description of repetitive behavior patterns, and comprehensive risk level assessment results. The individual risk weight coefficient of the person is dynamically adjusted based on the structured explanatory text, and the sensitivity threshold for judging the behavior of the person in subsequent intelligent video analysis is adjusted based on the updated risk weight coefficient.

[0049] The working principle of the above technical solution is as follows: Before generating early warning information based on early warning rules, the early warning module further performs the following refined operations: real-time identification of the identity markers (such as employee ID card, unique facial recognition code) of the detected personnel in the current video stream, and immediate linkage with the personnel classification archive maintained in the background to accurately retrieve all historical violation records of the individual confirmed by the system in the past 30 days; sorting and classifying the violation events based on the timestamp sequence to construct a personal behavior trajectory map with time-series characteristics, covering dimensions such as the time, location, type, duration, and handling status of the violation; and using unsupervised behavior clustering algorithms (such as DBSCAN or hierarchical clustering). The system analyzes violation patterns in the data map to identify high-frequency violation trends or recurring patterns of specific behavior types, such as not wearing a safety helmet, entering unauthorized areas, and prolonged solitary stays in high-risk work areas. When statistics show that the individual's cumulative number of violations exceeds the system's preset threshold (default 3 times) or the frequency of the same type of violation reaches 2 or more times, the system automatically activates a deep warning mechanism. At this time, a structured explanatory text is generated, including but not limited to: the individual's name, employee ID, work group, total number of violations in the past 30 days, and the distribution ratio of each major violation type (e.g., crossing boundaries accounts for 60%, not wearing a safety helmet is 40%). The system includes 40% of the data, descriptions of repetitive behavioral patterns (such as "entering the blasting warning zone around 4 PM for three consecutive days"), and comprehensive risk level assessment results (low / medium / high / extremely high). This text is embedded as enhanced information in the final warning message to improve the contextual awareness of the alarm information. Simultaneously, the system dynamically updates the individual risk weight coefficient of the person based on the analysis results—for example, gradually increasing it from a baseline value of 1.0 to 1.5 or higher. This influences the sensitivity threshold for judging their behavior during subsequent intelligent video analysis, such as reducing the time tolerance window for boundary crossing detection or increasing the confidence priority for abnormal posture recognition, thus shifting from single-event triggering to baseline risk assessment. The system evolves from a proactive early warning strategy driven by historical profiles of individuals. The structured explanatory text is simultaneously pushed to the mobile terminal APP of managers and the large screen display interface of the command center in the form of visual summary cards. It supports clicking to expand the complete timeline-style violation log details, showing the specific time, location screenshots, related video clip links and processing records of each violation. In addition, it provides a "one-click generation of warning notification" function button, which allows managers to quickly initiate a standardized electronic notification process, including violation summary data and rectification requirements, and automatically push it to team leaders, safety supervisors and relevant responsible parties, forming a closed-loop management process from identification, early warning, assessment to closed-loop disposal.

[0050] The beneficial effects of the above technical solution are as follows: By using the solution provided in this embodiment, and through in-depth correlation analysis between the current violation and the personnel's historical violation records over the past 30 days, an individual behavior trajectory map is constructed, and an unsupervised behavior clustering algorithm is used to identify high-frequency violation trends. This can upgrade isolated event alarms to continuous tracking and evaluation of personnel behavior patterns. The individual risk assessment and early warning mechanism based on historical data not only enhances the decision support value of early warning information and promotes the transformation of safety management from passive response to proactive prevention, but also helps to form more targeted safety education and management strategies, thereby fundamentally reducing the incidence of safety accidents caused by human factors and improving the overall safety production management level of the mine.

[0051] In one embodiment, the dynamic risk assessment module includes an environmental data access unit, a work intensity analysis unit, a behavior identification and adjustment unit, and a dynamic risk assessment unit. The environmental data access unit is used to access real-time weather data from non-coal mine on-site meteorological stations or IoT sensors. The real-time weather data includes at least information on heavy rain, fog, visibility, and temperature. The task intensity analysis unit is used to statistically analyze the task intensity data of personnel and vehicle targets in the task area based on the structured processing results of multi-source heterogeneous video data; the task intensity data includes distribution density and task duration. The behavior recognition adjustment unit is used to identify safety production violations based on weather data and work intensity data, utilizing the severe weather work behavior recognition model and the high work intensity behavior recognition model from multiple preset enhanced recognition models. Specifically, when the weather data is heavy rain or fog, the severe weather work behavior recognition model is used to identify running and slipping behavior of personnel targets; when the distribution density of personnel / vehicle targets in the work intensity data exceeds the distribution density threshold within a set period, or the work duration exceeds the preset duration, the high work intensity behavior recognition model is used to identify personnel who are stationary or have abnormal postures. The dynamic risk assessment unit is used to dynamically calculate the comprehensive risk level based on the safety production violations identified by the behavior recognition and adjustment unit, combined with real-time weather data and work intensity data, through a weighted risk fusion algorithm. When the comprehensive risk level exceeds a preset threshold, dynamic risk information including the causes of the risk and disposal suggestions is generated and pushed to the management personnel.

[0052] The working principle of the above technical solution is as follows: The dynamic risk assessment module works in concert with four functional units to achieve real-time perception, intelligent identification and dynamic assessment of safety risks at non-coal mine operation sites. First, the environmental data access unit collects weather information in real time from the on-site meteorological station and IoT sensors, including key parameters such as rainstorms, fog, visibility and temperature, to provide external environmental basis for subsequent risk judgment. These data reflect the degree of impact of natural conditions on operation safety. For example, rainstorms may cause landslides or electrical equipment short circuits, while fog reduces visibility and increases the probability of mechanical operation accidents. Secondly, the work intensity analysis unit performs structured processing on multi-source heterogeneous video data (such as surveillance cameras from different angles, resolutions, or types) to extract target information of personnel and vehicles. Based on this, it statistically analyzes two types of work intensity data: distribution density and work duration. Distribution density refers to the number of personnel or vehicles per unit area, reflecting the degree of spatial congestion; work duration refers to the length of continuous work, used to determine whether fatigue work occurs. This unit transforms the raw visual data into quantifiable work load indicators, providing contextual support for the identification of abnormal behavior. Next, the behavior recognition and adjustment unit dynamically calls the corresponding enhanced recognition model based on the weather data and work intensity data output by the first two units to accurately identify potential violations. When severe weather conditions such as heavy rain or fog are detected, the system automatically activates the "severe weather work behavior recognition model" to focus on analyzing whether personnel are running, slipping, or engaging in other high-risk actions. Such behaviors are very likely to cause falls or collisions in wet or low-visibility environments. When the system determines that the density of personnel or vehicles in a certain area exceeds a preset threshold within a set period, or that the individual work duration exceeds the safety limit, the "high work intensity behavior recognition model" is triggered to detect whether personnel are still for a long time or have abnormal postures (such as leaning against equipment or squatting on the ground). These may be signs of fatigue, fainting, or other health problems and are considered potential safety hazards in production. Finally, the dynamic risk assessment unit integrates all the above information, namely the identified safety production violations, real-time weather conditions, and work intensity levels, and uses a weighted risk fusion algorithm to calculate the comprehensive risk level. This algorithm assigns corresponding weights based on the risk contribution of different factors. For example, the probability of slipping under heavy rain conditions has a higher weight, while individual disability in high-density crowds may trigger a chain reaction, so its impact coefficient is also increased. The system continuously updates the weights of each factor and dynamically sums them to obtain the overall risk score for the current scenario. Once the score exceeds the preset safety threshold, a dynamic risk information is immediately generated, including the risk cause (such as "multiple people running in heavy rain" or "personnel found stationary for a long time in high-density areas") and targeted handling suggestions (such as "suspending outdoor operations" or "dispatching patrol personnel to confirm the status"), and pushed to the management personnel terminal so that timely intervention measures can be taken.

[0053] The beneficial effects of the above technical solution are as follows: By adopting the solution provided in this embodiment, the dynamic risk assessment module can realize precise and dynamic control of safety production risks in non-coal mines, realize closed-loop management from environmental perception to behavioral understanding to risk decision-making, have strong adaptability and real-time response capabilities, and can effectively improve the level of intelligent safety management in complex scenarios of non-coal mines.

[0054] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A non-coal mine multi-source video data fusion and intelligent behavior recognition management system, characterized in that, The application relates to a non-coal mine safety production behavior monitoring system. The system comprises: a data access module for accessing and converging multi-source heterogeneous video data inside and outside a non-coal mine enterprise through a standard interface; a video structured processing module for performing real-time structured processing on people, vehicles and objects in multi-source heterogeneous video data by adopting a target detection model based on a Transformer architecture and integrating non-coal mine production operation scene pre-training weights and a dynamic attention mechanism, and extracting key information of the people, vehicles and objects; a data fusion module for correlatively fusing the key information to generate fusion data; a behavior recognition module for deeply mining and correlatively analyzing the fusion data to recognize at least one safety production violation behavior of not wearing a safety helmet, illegal intrusion in a key area, exceeding the number of people in a limited space operation and staff leaving the post; an archive construction module for constructing a safety production violation behavior archive based on the safety production violation behavior; a warning module for generating warning information based on the safety production violation behavior and the safety production violation behavior archive and pushing the warning information to managers; 2. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 1, characterized in that, a dynamic risk assessment module for accessing weather data and operation intensity data of a non-coal mine site, adjusting selection of the behavior recognition model according to the weather data and the operation intensity data, and dynamically assessing a comprehensive risk level of the safety production violation behavior, and generating and pushing dynamic risk information to managers when the comprehensive risk level exceeds a preset threshold.

3. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 1, characterized in that, The data access module supports GB / T28181 and GA / T669-2008 standard interfaces, realizes seamless connection with a video monitoring system arranged inside and outside a non-coal mine enterprise, and realizes seamless connection with a video monitoring system arranged inside and outside a non-coal mine enterprise.

4. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 1, characterized in that, The video structured processing module comprises a multi-modal data correlation layer, the multi-modal data correlation layer is used for cross-modal fusion labeling of target attributes of people, vehicles and objects after structured processing and temperature characteristics in infrared thermal imaging video and distance information of a millimeter wave radar in multi-source heterogeneous video data, and generates multi-dimensional target archives including time and space coordinates and temperature states of personnel targets, behavior trajectories and temperature distributions of vehicle targets and time and space coordinates and temperature / distance characteristics of object targets, and the multi-dimensional target archives are used for extraction of key information of the people, vehicles and objects.

5. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 1, characterized in that, When the data fusion module correlatively fuses the key information, the data fusion module is also used for acquiring time stamps and geographic position information of video sources in multi-source heterogeneous video data, performing time and space correlation analysis on the extracted key information of the same personnel target or vehicle target or object target at different time stamps and / or different geographic positions, constructing motion trajectories and behavior sequences of the personnel target or the vehicle target or the object target, and taking the motion trajectories and the behavior sequences as the fusion data. The behavior recognition module comprises a specific scene reinforcement identification unit; The specific scene reinforcement identification unit comprises a scene perception subunit, a reinforcement identification model selection subunit and a specific scene safety production violation behavior recognition subunit; The scene perception subunit is used for identifying a production operation scene type of a non-coal mine corresponding to multi-source heterogeneous video data. The reinforced identification model selection subunit is configured to select a specific behavior identification model matching the production operation scene type from a plurality of preset reinforced identification models according to the identified production operation scene type; The reinforced identification model includes a downhole limited space behavior identification model, a blasting operation zone behavior identification model, and a transportation ramp behavior identification model. The downhole limited space behavior identification model is configured to identify a safety production violation behavior of a number of personnel targets, vehicle targets, and object targets exceeding a limit in a low-illumination environment in a downhole limited space. The blasting operation zone behavior identification model is configured to identify a safety production violation behavior of a personnel target or a vehicle target illegally invading a key area. The transportation ramp behavior identification model is configured to identify a safety production violation behavior of a personnel target invading a transportation ramp or a vehicle target not driving as specified. The specific scene safety production violation behavior identification subunit is configured to analyze the fused data by using the specific behavior identification model to identify a safety production violation behavior in a production operation scene.

6. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 1, characterized in that, The archive library construction module includes an enterprise general library construction unit, a target sub-library construction unit, and a level association unit. The enterprise general library construction unit is configured to construct a summary archive library of safety production violation behaviors of non-coal mine enterprises based on the safety production violation behaviors. The target sub-library construction unit is configured to construct a classified archive library distinguishing personnel targets, vehicle targets, and object targets based on the safety production violation behaviors. The level association unit is configured to associate and map the safety production violation behaviors with preset risk levels, match corresponding risk level tags for entries of the safety production violation behaviors in the summary archive library and the classified archive library, and establish a dynamic association index between the safety production violation behaviors and the risk levels.

7. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 6, characterized in that, The early warning module is configured to perform the following operations: Based on the safety production violation behaviors and the matched corresponding risk levels, the early warning module is configured to retrieve corresponding early warning rules from a preset early warning strategy library, the early warning rules defining a content generation manner, a push priority, a push channel combination, and a push object level of early warning information; Based on the early warning rules, the early warning module is configured to generate early warning information and push the early warning information to corresponding levels of management personnel according to the push priority and the push channel combination.

8. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 7, characterized in that, Before retrieving the early warning rules from the preset early warning strategy library, the early warning module is further configured to perform the following operations: The complexity of the production operation scene of the non-coal mine is evaluated, a complexity score of the scene is calculated by a target detection and feature extraction algorithm combined with a classification or regression model, and if the complexity score is higher than a set threshold, high-emergency processing priority early warning rules are preferentially selected when the early warning rules are retrieved; the complexity of the production operation scene of the non-coal mine is defined as a number of targets in the production operation scene of the non-coal mine, a speed change of a moving target, and a complexity of a scene structure. Based on the constructed safety production violation behavior association model, combinations of violations with potential association in the same period are identified. When an associated violation combination is identified, the pre-warning rule for the associated violation combination is called. The safety production violation behavior association model is constructed based on historical safety production violation behavior data and domain expert experience. Through association rule mining algorithm, the co-occurrence frequency and time sequence relationship between different violation behaviors are analyzed, and the confidence and support evaluation indicators between the entries of safety production violation behaviors are established.

9. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 7, characterized in that, Before generating the pre-warning information based on the pre-warning rule, the pre-warning module is further configured to perform the following operations: Real-time identification of the identity of the detected personnel target in the current video stream, and retrieval of the personnel's historical violation records confirmed by the system in the past 30 days from the background personnel classification archive according to the identity; Time stamp-based time sequence sorting and classification of the violation events in the historical violation records, construction of a personal behavior trajectory graph including violation occurrence time, location, type, duration and disposal status; Using an unsupervised behavior clustering algorithm to analyze the personal behavior trajectory graph, identify high-frequency violation trends or repeated occurrence of specific violation behavior types; When the cumulative number of violations of the personnel exceeds a preset threshold, or the frequency of the same type of violation behavior reaches a set number of times, a deep pre-warning mechanism is triggered; Generating a structured explanatory text and embedding the structured explanatory text into the pre-warning message body, wherein the structured explanatory text includes personnel identity information, total number of violations in the past 30 days, distribution ratio of each major violation type, repetitive behavior pattern description and comprehensive risk level evaluation result; Dynamically adjusting the individual risk weight coefficient of the personnel according to the structured explanatory text, and adjusting the sensitivity threshold for behavior judgment of the personnel in subsequent video intelligent analysis based on the updated risk weight coefficient.

10. The non-coal mine multi-source video data fusion and intelligent behavior recognition management system according to claim 1, characterized in that, The dynamic risk assessment module includes an environment data access unit, a work intensity analysis unit, a behavior recognition adjustment unit, and a dynamic risk assessment unit; The environment data access unit is configured to access real-time weather data from a non-coal mine site weather station or an Internet of Things sensor, and the real-time weather data at least includes heavy rain, fog visibility and temperature information; The work intensity analysis unit is configured to statistically analyze the work intensity data of personnel targets and vehicle targets in the work area based on the structured processing results of multi-source heterogeneous video data; The work intensity data includes distribution density and work duration; The behavior recognition adjustment unit is configured to use a preset plurality of reinforcement identification models, including a bad weather work behavior identification model and a high work intensity behavior identification model, to identify safety production violation behaviors according to weather data and work intensity data. Specifically, when the weather data is heavy rain or fog, the bad weather work behavior identification model is used to identify the running and slipping of personnel targets; when the distribution density of personnel targets / vehicle targets in a set period in the work intensity data exceeds the distribution density threshold, or the work duration exceeds the preset duration, the high work intensity behavior identification model is used to identify the stillness or abnormal posture of personnel targets. The dynamic risk assessment unit is configured to dynamically calculate a comprehensive risk level based on the safety production violation behavior identified by the behavior recognition adjustment unit, in combination with real-time weather data and work intensity data, through a weighted risk fusion algorithm. When the comprehensive risk level exceeds a preset threshold, the dynamic risk information including risk causes and disposal suggestions is generated and pushed to the management personnel.

Images