Model-based coal mine safety monitoring method, device and safety monitoring system

Abstract

The application provides a model-based coal mine safety monitoring method, device and safety monitoring system, the method comprising: acquiring multi-source heterogeneous data in a coal mine working environment, the multi-source heterogeneous data comprising monitoring video data, equipment operation data, personnel behavior data and environment monitoring data; performing feature extraction on the multi-source heterogeneous data to obtain heterogeneous feature data, and using a neural network to fuse the heterogeneous feature data to obtain fused feature data; processing the fused feature data using a safety large model to obtain a risk identification result, wherein the safety large model is trained based on coal mine safety rules and a semantic knowledge base; generating an alarm signal according to the risk identification result and sending the alarm signal to a preset terminal. The method solves the problem of insufficient accuracy of automatic safety detection caused by the fact that safety monitoring devices in the prior art belong to different systems and form data islands.

Description

Technical Field

[0001] This invention relates to the field of coal mine production safety technology, and more specifically, to a model-based coal mine safety monitoring method, device, computer-readable storage medium, and safety monitoring system. Background Technology

[0002] As a high-risk industry, coal mines have widely deployed various safety monitoring equipment, such as video surveillance systems, gas concentration sensors, dust concentration sensors, temperature and humidity sensors, personnel positioning systems, and equipment PLC data acquisition devices, forming a multi-source data acquisition network covering the underground working environment.

[0003] However, currently, each system operates independently, resulting in fragmented data silos. Monitoring personnel must switch between multiple terminals, relying on manual observation of video footage, comparison of sensor thresholds, verification of personnel location information, and personal experience to determine the existence of safety hazards. While some systems have introduced intelligent analysis technologies based on single data sources, such as identifying whether safety helmets are worn through video or triggering alarms based on excessive gas concentrations, these methods can only identify explicit risks with simple logic and clear conditions. They cannot effectively link the semantic and spatiotemporal relationships between different modalities of data.

[0004] In summary, due to the lack of deep integration of data from various monitoring systems, risk identification is limited to single-point, single-modal, and static threshold judgments, making it difficult to cope with the multi-condition, multi-logic, and dynamically coupled safety risks in complex underground coal mine operation scenarios, resulting in insufficient detection accuracy. Summary of the Invention

[0005] The main objective of this application is to provide a model-based coal mine safety monitoring method, device, computer-readable storage medium, and safety monitoring system, so as to at least solve the problem that in the prior art, safety monitoring equipment belongs to different systems, forming data silos, which leads to insufficient accuracy of automated safety detection.

[0006] To achieve the above objectives, according to one aspect of this application, a model-based coal mine safety monitoring method is provided, comprising: acquiring multi-source heterogeneous data within the coal mine operating environment, the multi-source heterogeneous data including monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data; extracting features from the multi-source heterogeneous data to obtain heterogeneous feature data; fusing the heterogeneous feature data using a neural network to obtain fused feature data; processing the fused feature data using a large safety model to obtain risk identification results, wherein the large safety model is trained based on coal mine safety rules and a semantic knowledge base; generating an alarm signal based on the risk identification results, and sending the alarm signal to a preset terminal.

[0007] Optionally, feature extraction is performed on the multi-source heterogeneous data to obtain heterogeneous feature data, including: extracting the spatiotemporal features of the surveillance video data to obtain the first feature data; extracting the temporal features of the equipment operation data to obtain the second feature data; performing location encoding on the personnel behavior data and extracting the temporal features of the encoded personnel behavior data to obtain the third feature data; extracting the temporal features of the environmental monitoring data to obtain the fourth feature data; and aligning the first feature data, the second feature data, the third feature data, and the fourth feature data with timestamps to obtain the heterogeneous feature data.

[0008] Optionally, acquiring multi-source heterogeneous data within the coal mine operating environment includes: extracting video streams including equipment and / or personnel from videos captured by surveillance cameras to obtain surveillance video data; capturing equipment operation data from PLC interfaces to obtain equipment operation data; receiving location data sent by UWPN342412 positioning tags worn by personnel to obtain personnel behavior data; and receiving monitoring signals from sensor groups, extracting one or more of the following from the monitoring signals: gas concentration, temperature, humidity, and dust concentration in the operating environment to obtain environmental monitoring data.

[0009] Optionally, before processing the fused feature data using the safety big model to obtain the risk identification result, the method further includes: acquiring the safety specification text of the coal mine operation environment and processing the safety specification text into a semantic knowledge base; processing the samples in the semantic knowledge base using a mask and determining the logical relationship type between each sample, the mask including rule mask, causal mask and prohibition mask; pre-training the big language model based on the processed semantic knowledge base until the loss function value of the big language model is less than a first preset value, wherein the loss function includes mask language modeling loss, logical relationship classification cross-entropy loss and terminology consistency loss; acquiring historical accident logs, extracting risk events from the historical accident logs, and using the timestamp corresponding to the risk event as the end time, extracting the corresponding multi-source heterogeneous data according to a preset duration to obtain historical monitoring data; using the historical monitoring data as input data and the risk event as a label, training the pre-trained big language model until the error rate of the big language model is less than a second preset value, and determining the big language model as the safety big model.

[0010] Optionally, the spatiotemporal features of the surveillance video data are extracted to obtain the first feature data, including: processing the surveillance video data using a three-dimensional convolutional neural network to obtain the first feature data.

[0011] Optionally, extracting the temporal features of the equipment operation data to obtain the second feature data includes: processing the equipment operation data using a long short-term memory network to obtain the second feature data.

[0012] Optionally, a three-dimensional convolutional neural network is used to process the surveillance video data to obtain the first feature data. The method further includes: using a target monitoring model to process the surveillance video data to identify whether a target object exists in the surveillance video data; and if a target object exists in the surveillance video data, using a three-dimensional convolutional neural network to process the surveillance video data to obtain the first feature data.

[0013] According to another aspect of this application, a model-based coal mine safety monitoring device is provided. The device includes: an acquisition unit for acquiring multi-source heterogeneous data within the coal mine operating environment, including monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data; a first processing unit for extracting features from the multi-source heterogeneous data to obtain heterogeneous feature data, and fusing the heterogeneous feature data using a neural network to obtain fused feature data; a second processing unit for processing the fused feature data using a large safety model to obtain risk identification results, wherein the large safety model is trained based on coal mine safety rules and a semantic knowledge base; and a generation unit for generating an alarm signal based on the risk identification results and sending the alarm signal to a preset terminal.

[0014] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0015] According to another aspect of this application, a security monitoring system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any one of the methods described.

[0016] Applying the technical solution of this application, in the above-mentioned model-based coal mine safety monitoring method, firstly, multi-source heterogeneous data within the coal mine operating environment is acquired, including monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data; then, features are extracted from the multi-source heterogeneous data to obtain heterogeneous feature data, and a neural network is used to fuse the heterogeneous feature data to obtain fused feature data; subsequently, a large safety model is used to process the fused feature data to obtain risk identification results, wherein the large safety model is trained based on coal mine safety rules and a semantic knowledge base; finally, an alarm signal is generated based on the risk identification results and sent to a preset terminal. By acquiring multi-source heterogeneous information such as monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data in the coal mine operation environment, features are extracted from various types of data to form heterogeneous feature data. A neural network is then used to achieve deep fusion of multimodal features, generating fused feature data representing the overall operation status. Furthermore, a large-scale safety model trained based on coal mine safety rules and a semantic knowledge base performs semantic understanding and logical reasoning on the fused feature data, accurately identifying complex risk scenarios composed of multiple overlapping safety conditions. This overcomes the limitations of traditional methods that rely on human experience and struggle to automatically identify risks with multiple logical couplings. Finally, alarm signals are automatically generated based on the identified risk results and pushed to preset terminals, achieving real-time, proactive, and intelligent early warning of complex safety risks with high concealment and strong correlation in coal mine operations. This solves the problem in existing technologies where safety monitoring equipment belongs to different systems, forming data silos and resulting in insufficient accuracy of automated safety detection. Attached Figure Description

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for a model-based coal mine safety monitoring method provided in an embodiment of this application is shown.

[0018] Figure 2 A schematic flowchart of a model-based coal mine safety monitoring method according to an embodiment of this application is shown.

[0019] Figure 3 A schematic diagram of the architecture of a security monitoring system provided according to an embodiment of this application is shown;

[0020] Figure 4 A flowchart illustrating a specific model-based coal mine safety monitoring method according to another embodiment of this application is shown.

[0021] Figure 5 A schematic diagram of the structure of a multimodal processing module according to an embodiment of this application is shown;

[0022] Figure 6A structural block diagram of a model-based coal mine safety monitoring device provided according to an embodiment of this application is shown. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] As described in the background section, the data from various monitoring systems in the prior art have not been deeply integrated, and risk identification is limited to single-point, single-modal, and static threshold judgments. This makes it difficult to cope with the multi-condition, multi-logic, and dynamically coupled safety risks in complex underground coal mine operation scenarios, resulting in insufficient detection accuracy. To solve the problem that safety monitoring equipment in the prior art belongs to different systems, forming data silos and resulting in insufficient accuracy of automated safety detection, the embodiments of this application provide a model-based coal mine safety monitoring method, device, computer-readable storage medium, and safety monitoring system.

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a model-based coal mine safety monitoring method according to an embodiment of the present invention. Figure 1As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the model-based coal mine safety monitoring method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] This embodiment provides a model-based coal mine safety monitoring method that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] Figure 2 This is a flowchart of a model-based coal mine safety monitoring method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0032] Step S201: Obtain multi-source heterogeneous data in the coal mine operating environment. The multi-source heterogeneous data includes monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data.

[0033] Specifically, it acquires multi-source heterogeneous data within the coal mine operating environment, including surveillance video data, equipment operation data, personnel behavior data, and environmental monitoring data, aiming to comprehensively cover all types of safety-related information sources in underground operating scenarios.

[0034] Understandably, the system directly establishes a multi-dimensional data input foundation for the coal mine operating environment. Monitoring video data reflects the visual spatial state, equipment operation data characterizes the real-time working conditions of machinery, personnel behavior data reflects the activity trajectories and operational intentions of workers, and environmental monitoring data records physical environmental parameters such as gas, dust, temperature, and humidity. Together, these constitute the raw input set for subsequent system analysis. The acquisition process directly achieves the synchronous collection of diverse information related to coal mine safety, providing the necessary and complete data prerequisites for subsequent risk identification based on multi-modal feature fusion.

[0035] Step S202: Extract features from multi-source heterogeneous data to obtain heterogeneous feature data, and use a neural network to fuse the heterogeneous feature data to obtain fused feature data;

[0036] Specifically, feature extraction is performed on multi-source heterogeneous data such as monitoring video data, production equipment data, and personnel operation behavior data acquired in the coal mine operation environment. By using computational methods adapted to different data types, original feature information that can characterize their essential attributes is extracted from each data type, forming heterogeneous feature data with modal differences. Then, neural networks are used to perform structured processing on these heterogeneous feature data. Through the weight connection and nonlinear transformation mechanism within the network, features from different data sources are associated and integrated in a unified expression space, thereby outputting a fused feature data that can comprehensively reflect the inherent relationship between multi-source information.

[0037] Step S203: The fused feature data is processed using a safety big model to obtain the risk identification result. The safety big model is trained based on coal mine safety rules and a semantic knowledge base.

[0038] Specifically, this process directly applies the safety big model to the fused feature data, using its embedded coal mine safety rules and semantic knowledge base to make logical judgments on the scenarios represented by the features, thereby outputting risk identification results. The safety big model is trained based on coal mine safety rules and semantic knowledge base, and its processing capabilities are directly defined by these rules and knowledge bases. It does not rely on external rule bases or human intervention, but only matches and infers the inherent semantic associations of the fused features through the model to achieve direct judgment of risk status.

[0039] Step S204: Generate an alarm signal based on the risk identification result and send the alarm signal to a preset terminal.

[0040] Specifically, once the system has completed the assessment of safety risks in the coal mine operating environment, it directly triggers the generation of alarm signals based on the assessment results, and promptly handles risk events.

[0041] In this embodiment, firstly, multi-source heterogeneous data within the coal mine operating environment is acquired, including surveillance video data, equipment operation data, personnel behavior data, and environmental monitoring data. Then, features are extracted from the multi-source heterogeneous data to obtain heterogeneous feature data. A neural network is then used to fuse the heterogeneous feature data to obtain fused feature data. Next, a large-scale safety model is used to process the fused feature data to obtain risk identification results. This large-scale safety model is trained based on coal mine safety rules and a semantic knowledge base. Finally, an alarm signal is generated based on the risk identification results and sent to a preset terminal. By acquiring multi-source heterogeneous information such as monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data in the coal mine operation environment, features are extracted from various types of data to form heterogeneous feature data. A neural network is then used to achieve deep fusion of multimodal features, generating fused feature data representing the overall operation status. Furthermore, a large-scale safety model trained based on coal mine safety rules and a semantic knowledge base performs semantic understanding and logical reasoning on the fused feature data, accurately identifying complex risk scenarios composed of multiple overlapping safety conditions. This overcomes the limitations of traditional methods that rely on human experience and struggle to automatically identify risks with multiple logical couplings. Finally, alarm signals are automatically generated based on the identified risk results and pushed to preset terminals, achieving real-time, proactive, and intelligent early warning of complex safety risks with high concealment and strong correlation in coal mine operations. This solves the problem in existing technologies where safety monitoring equipment belongs to different systems, forming data silos and resulting in insufficient accuracy of automated safety detection.

[0042] In order to extract the internal relationships between different types of data, in one optional implementation, step S202 above includes:

[0043] Step S2021: Extract the spatiotemporal features of the surveillance video data to obtain the first feature data;

[0044] Step S2022: Extract the temporal features of the equipment operation data to obtain the second feature data;

[0045] Step S2023: The personnel behavior data is location-encoded and the temporal features of the encoded personnel behavior data are extracted to obtain the third feature data;

[0046] Step S2024: Extract the time-series features of the environmental monitoring data to obtain the fourth feature data;

[0047] Step S2025: Timestamp alignment is performed on the first feature data, the second feature data, the third feature data, and the fourth feature data to obtain heterogeneous feature data.

[0048] In the above embodiments, by extracting the spatiotemporal features of surveillance video data, the temporal features of equipment operation data, the temporal features of personnel behavior data after location encoding, and the temporal features of environmental monitoring data, four types of basic feature data with modality specificity are formed. Based on a unified timestamp, the above four types of feature data are precisely aligned to ensure that heterogeneous information from different sensors and data sources remains synchronized in the time dimension.

[0049] The above embodiments eliminate the problems of feature misalignment and fusion inaccuracy caused by differences in data acquisition frequency, transmission delay, or modal heterogeneity, providing a temporally consistent and semantically clear input foundation for subsequent feature fusion of neural networks. This ensures that the security big model can accurately associate multi-dimensional risk association rules and semantic knowledge on the fused feature data, significantly improving the reliability and accuracy of risk identification results.

[0050] To achieve synchronous acquisition of multi-source information in coal mines, in one optional implementation, step S201 includes:

[0051] Step S2011: Extract video streams including equipment and / or personnel from the video captured by the surveillance camera to obtain surveillance video data;

[0052] Step S2012: Capture the equipment's operating data from the PLC interface to obtain the equipment operating data;

[0053] Step S2013: Receive location data sent by the UWB positioning tag worn by the operator to obtain personnel behavior data;

[0054] Step S2014: Receive monitoring signals from the sensor group, and extract one or more of the following environmental monitoring data from the monitoring signals: gas concentration, temperature, humidity, and dust concentration of the working environment, to obtain environmental monitoring data.

[0055] In the above embodiments, structured monitoring video data is obtained by extracting video streams containing equipment and / or workers from the video captured by the monitoring camera, and equipment operating parameters are captured in real time from the PLC interface to form accurate equipment operating data. High-precision positioning information sent by the UWPN342412 positioning tag worn by the workers is received to construct dynamic personnel behavior data, and environmental indicators such as gas concentration, temperature, humidity and dust concentration are collected through the sensor group to generate multi-dimensional environmental monitoring data. This achieves accurate source capture and semantic standardization output of four types of heterogeneous data in the coal mine operation environment.

[0056] The above embodiments ensure the integrity, consistency, and real-world mapping of the input data in the subsequent feature extraction stage, enabling the neural network to effectively fuse high-quality heterogeneous features. This provides a clear scene semantics as the input foundation for the safety big model trained on coal mine safety rules and semantic knowledge base, ultimately improving the accuracy and reliability of risk identification results.

[0057] In order to select the aforementioned security big model, in one optional implementation, before processing the fused feature data using the security big model to obtain the risk identification result, the method further includes:

[0058] Step S301: Obtain the safety specification text for the coal mine operation environment and process the safety specification text into a semantic knowledge base;

[0059] Step S302: Process the samples in the semantic knowledge base using masks and determine the logical relationship type between each sample. The masks include rule masks, causal masks, and prohibition masks.

[0060] Step S303: Pre-train the large language model based on the processed semantic knowledge base until the loss function value of the large language model is less than the first preset value. The loss function includes masked language modeling loss, logical relation classification cross-entropy loss and term consistency loss.

[0061] Step S304: Obtain historical accident logs, extract risk events from the historical accident logs, and use the timestamp corresponding to the risk event as the end time to extract the corresponding multi-source heterogeneous data according to the preset duration to obtain historical monitoring data.

[0062] Step S305: Using historical monitoring data as input data and risk events as labels, train the pre-trained large language model until the error rate of the large language model is less than the second preset value, and then determine the large language model as the safety large model.

[0063] In the above embodiments, a structured semantic knowledge base is constructed by parsing safety regulations text in the coal mine operation environment. Rule masks, causal masks, and prohibition masks are introduced to semantically and logically annotate the knowledge base samples, enabling the large language model to learn the implicit logical relationships and semantic constraints between safety rules during the pre-training stage. Combined with a multi-objective optimization mechanism of masked language modeling loss, logical relationship classification cross-entropy loss, and terminology consistency loss, the model's ability to understand safety semantics in a structured way is improved. Subsequently, using risk events in historical accident logs as endpoints, multi-source heterogeneous data within the corresponding time period are extracted as training samples. The pre-trained model is fine-tuned using risk events as labels, enabling the model to establish a precise mapping relationship from real-time environmental data representation to real risk events. Finally, a large safety model with semantic reasoning and multi-logical superposition risk identification capabilities is formed.

[0064] Through the above embodiments, the security big model is implemented from professional semantic knowledge base and security rules to real-world security events, realizing high-precision semantic reasoning and proactive identification of multi-dimensional and multi-level security risks. This effectively solves the problem of inaccurate identification of complex risk scenarios and frequent false alarms and missed alarms caused by the lack of joint modeling of the logical structure of security rules and the correlation with historical accidents in existing technologies.

[0065] In order to preserve the temporal and spatial characteristics of the video data, in an optional implementation, step S2021 above includes:

[0066] Step S20211: The surveillance video data is processed using a three-dimensional convolutional neural network to obtain the first feature data.

[0067] In the above embodiments, by using a three-dimensional convolutional neural network to process the monitoring video data in the coal mine operation environment, the spatiotemporal correlation features of personnel behavior and equipment movement in the video sequence are directly extracted to form structured first feature data with time-series dynamic perception capabilities.

[0068] Through the above embodiments, it is ensured that video modal data can be matched and fused with equipment operation data, personnel behavior data and environmental monitoring data, providing fusion feature input with real scene semantic support for the safety big model, and improving the recognition accuracy and response reliability of complex risk scenarios with multiple logic superpositions.

[0069] To ensure compatibility between the device and its environment, in one optional implementation, step S2022 includes:

[0070] Step S20221: The device operation data is processed using a long short-term memory network to obtain the second feature data.

[0071] In the above embodiments, by employing a long short-term memory network to extract temporal features from equipment operation data in the coal mine operating environment, the dynamic dependency relationship of equipment operating status evolving over time is accurately modeled, generating second feature data that can characterize the abnormal evolution trend of equipment.

[0072] Through the above embodiments, the equipment status information is input into the neural network along with other heterogeneous data features for fusion, which enhances the temporal expression capability of the fused feature data. This enables the subsequent safety big data model to capture the complex temporal coupling relationship between equipment operation and multi-dimensional factors such as personnel behavior and environmental changes when combining coal mine safety rules and semantic knowledge base for risk identification. This allows for proactive early warning and precise prevention and control of potential chain risks caused by the evolution of equipment status in the coal mine operating environment.

[0073] To ensure the accuracy of video processing, in one optional implementation, step S20211 includes:

[0074] A target monitoring model is used to process surveillance video data in order to identify whether target objects exist in the surveillance video data;

[0075] When a target object exists in the surveillance video data, a three-dimensional convolutional neural network is used to process the surveillance video data to obtain the first feature data.

[0076] In the above embodiments, by first using a target monitoring model to pre-screen the surveillance video data to identify whether a target object exists, and only when the existence of a target object is confirmed, the computationally intensive three-dimensional convolutional neural network is activated to extract spatiotemporal features, thus avoiding redundant calculations for video frames without targets.

[0077] Through the above embodiments, the overall computational load and processing latency of the system are reduced, while ensuring the accuracy and real-time performance of feature extraction in key risk information areas. Thus, in the complex coal mine safety monitoring scenario of multi-source heterogeneous data fusion, efficient and accurate identification of high-risk behaviors is achieved.

[0078] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the model-based coal mine safety monitoring method of this application will be described in detail below with reference to specific embodiments.

[0079] In another embodiment of this application, Figure 3 This is a schematic diagram of the architecture of the aforementioned security monitoring system, as shown below. Figure 3 As shown, the aforementioned security monitoring system includes:

[0080] The coal mine safety early warning system based on a multimodal large model is deployed on the ground data center or edge computing node of the coal mine, and mainly includes a data acquisition module, a multimodal processing module, and an early warning output module.

[0081] The data acquisition module is responsible for comprehensively collecting various types of data from the underground coal mine operating environment. Specifically, it collects real-time monitoring video data through high-definition or infrared video cameras deployed in roadways and working faces. Through connections to the control systems of various equipment, such as the programmable logic controllers (PLCs) of coal mining machines, tunneling machines, and belt conveyors, the PLCs acquire production equipment data such as operating status, current, power, and speed. It also uses precise personnel positioning systems, such as Ultra-Wide Band (UWB) positioning systems, to obtain the real-time location coordinates and movement trajectory of each worker underground, serving as personnel operation behavior data. This module also collects environmental sensor data such as methane, carbon monoxide, temperature, humidity, and dust concentration by connecting to environmental monitoring substations. All collected data is transmitted in real-time to the multimodal processing module via the underground industrial ring network.

[0082] After receiving the data, the multimodal processing module extracts features from the received raw multi-source data and fuses them. Then, based on the fused features, it identifies and outputs early warning results to the early warning output module.

[0083] The early warning output module generates specific early warning signals based on the received risk identification results. These signals can be output in various ways, such as displaying a highlighted alarm window on the central monitoring room screen and emitting an audible and visual alarm. Simultaneously, alarm information can be pushed to the mobile devices of relevant safety management personnel and on-site workers via SMS or a mobile app. The early warning information is detailed, not only indicating the type of risk but also including the location of the risk, a timestamp, and related data snippets, such as a video recording of the incident, to facilitate rapid verification and response.

[0084] This embodiment relates to a specific model-based coal mine safety monitoring method, such as... Figure 4 As shown, it includes the following steps:

[0085] Step S1: Data acquisition, continuously acquiring multi-source heterogeneous data from downhole.

[0086] Step S2: Feature extraction and fusion processing. Different network models are used to process data of different modalities, and the processed features are further fused.

[0087] Step S3: Real-time analysis and risk identification can be performed by using a classifier consisting of one or more fully connected layers to process the fused features and obtain the risk identification results.

[0088] Step S4: Early Warning Output. Based on the risk identification results from the model, execute the next action. If the result indicates no risk, the system continues to monitor in a loop. If a safety risk is identified, the risk identification results, including the risk type and confidence level, are immediately output.

[0089] Step S5: Generate an early warning signal and output the risk identification results after the above judgment to the staff in a preset manner so as to deal with potential risks.

[0090] In another embodiment, the structure of the above-described multimodal processing module is as follows: Figure 5 As shown, the feature extraction unit uses different network models to process data of different modalities. For example, for video data, a 3D convolutional neural network (3D-CNN) is used to extract dynamic visual features containing temporal and spatial information. For time-series data such as production equipment data and environmental sensor data, a Long Short-Term Memory (LSTM) network is used to extract its time-series features. For personnel location data, coordinate encoding is performed directly. The feature fusion unit fuses the features from each modality. Fusion can be achieved in various ways, such as simple feature vector concatenation or more complex attention mechanisms (AM). The risk identification unit typically consists of a classifier composed of one or more fully connected layers, used to determine whether there is a safety risk in the current scene and obtain the risk identification result.

[0091] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0092] This application also provides a model-based coal mine safety monitoring device. It should be noted that the model-based coal mine safety monitoring device of this application can be used to execute the model-based coal mine safety monitoring method provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0093] The following describes the model-based coal mine safety monitoring device provided in the embodiments of this application.

[0094] Figure 6This is a structural block diagram of a model-based coal mine safety monitoring device according to an embodiment of this application. Figure 6 As shown, the device includes:

[0095] The "Single Na Yuan 10" is used to acquire multi-source heterogeneous data within the coal mine operating environment. This multi-source heterogeneous data includes surveillance video data, equipment operation data, personnel behavior data, and environmental monitoring data.

[0096] Specifically, it acquires multi-source heterogeneous data within the coal mine operating environment, including surveillance video data, equipment operation data, personnel behavior data, and environmental monitoring data, aiming to comprehensively cover all types of safety-related information sources in underground operating scenarios.

[0097] Understandably, the system directly establishes a multi-dimensional data input foundation for the coal mine operating environment. Monitoring video data reflects the visual spatial state, equipment operation data characterizes the real-time working conditions of machinery, personnel behavior data reflects the activity trajectories and operational intentions of workers, and environmental monitoring data records physical environmental parameters such as gas, dust, temperature, and humidity. Together, these constitute the raw input set for subsequent system analysis. The acquisition process directly achieves the synchronous collection of diverse information related to coal mine safety, providing the necessary and complete data prerequisites for subsequent risk identification based on multi-modal feature fusion.

[0098] The first processing unit 20 is used to extract features from multi-source heterogeneous data to obtain heterogeneous feature data, and to fuse the heterogeneous feature data using a neural network to obtain fused feature data.

[0099] Specifically, feature extraction is performed on multi-source heterogeneous data such as monitoring video data, production equipment data, and personnel operation behavior data acquired in the coal mine operation environment. By using computational methods adapted to different data types, original feature information that can characterize their essential attributes is extracted from each data type, forming heterogeneous feature data with modal differences. Then, neural networks are used to perform structured processing on these heterogeneous feature data. Through the weight connection and nonlinear transformation mechanism within the network, features from different data sources are associated and integrated in a unified expression space, thereby outputting a fused feature data that can comprehensively reflect the inherent relationship between multi-source information.

[0100] The second processing unit 30 is used to process the fused feature data using a safety big model to obtain risk identification results, wherein the safety big model is trained based on coal mine safety rules and semantic knowledge base;

[0101] Specifically, this process directly applies the safety big model to the fused feature data, using its embedded coal mine safety rules and semantic knowledge base to make logical judgments on the scenarios represented by the features, thereby outputting risk identification results. The safety big model is trained based on coal mine safety rules and semantic knowledge base, and its processing capabilities are directly defined by these rules and knowledge bases. It does not rely on external rule bases or human intervention, but only matches and infers the inherent semantic associations of the fused features through the model to achieve direct judgment of risk status.

[0102] The generation unit 40 is used to generate an alarm signal based on the risk identification result and send the alarm signal to a preset terminal.

[0103] Specifically, once the system has completed the assessment of safety risks in the coal mine operating environment, it directly triggers the generation of alarm signals based on the assessment results, and promptly handles risk events.

[0104] In this embodiment, the acquisition unit acquires multi-source heterogeneous data within the coal mine operating environment, including monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data. The first processing unit extracts features from the multi-source heterogeneous data to obtain heterogeneous feature data, and then uses a neural network to fuse the heterogeneous feature data to obtain fused feature data. The second processing unit processes the fused feature data using a large safety model to obtain risk identification results, wherein the large safety model is trained based on coal mine safety rules and a semantic knowledge base. The generation unit generates an alarm signal based on the risk identification results and sends the alarm signal to a preset terminal. By acquiring multi-source heterogeneous information such as monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data in the coal mine operation environment, features are extracted from various types of data to form heterogeneous feature data. A neural network is then used to achieve deep fusion of multimodal features, generating fused feature data representing the overall operation status. Furthermore, a large-scale safety model trained based on coal mine safety rules and a semantic knowledge base performs semantic understanding and logical reasoning on the fused feature data, accurately identifying complex risk scenarios composed of multiple overlapping safety conditions. This overcomes the limitations of traditional methods that rely on human experience and struggle to automatically identify risks with multiple logical couplings. Finally, alarm signals are automatically generated based on the identified risk results and pushed to preset terminals, achieving real-time, proactive, and intelligent early warning of complex safety risks with high concealment and strong correlation in coal mine operations. This solves the problem in existing technologies where safety monitoring equipment belongs to different systems, forming data silos and resulting in insufficient accuracy of automated safety detection.

[0105] In order to extract the internal relationships between different types of data, in one optional implementation, the first processing unit includes:

[0106] The first processing module is used to extract the spatiotemporal features of the surveillance video data to obtain the first feature data.

[0107] The second processing module is used to extract the temporal features of the equipment operation data to obtain the second feature data.

[0108] The third processing module is used to encode the location of personnel behavior data and extract the temporal features of the encoded personnel behavior data to obtain the third feature data.

[0109] The fourth processing module is used to extract the time-series features of environmental monitoring data to obtain the fourth feature data;

[0110] The fifth processing module is used to align the first feature data, second feature data, third feature data and fourth feature data with timestamps to obtain heterogeneous feature data.

[0111] In the above embodiments, by extracting the spatiotemporal features of surveillance video data, the temporal features of equipment operation data, the temporal features of personnel behavior data after location encoding, and the temporal features of environmental monitoring data, four types of basic feature data with modality specificity are formed. Based on a unified timestamp, the above four types of feature data are precisely aligned to ensure that heterogeneous information from different sensors and data sources remains synchronized in the time dimension.

[0112] The above embodiments eliminate the problems of feature misalignment and fusion inaccuracy caused by differences in data acquisition frequency, transmission delay, or modal heterogeneity, providing a temporally consistent and semantically clear input foundation for subsequent feature fusion of neural networks. This ensures that the security big model can accurately associate multi-dimensional risk association rules and semantic knowledge on the fused feature data, significantly improving the reliability and accuracy of risk identification results.

[0113] To achieve synchronous acquisition of multi-source information in coal mines, in one optional implementation, the acquisition unit includes:

[0114] The first acquisition module is used to extract video streams, including equipment and / or personnel, from the video captured by the surveillance camera to obtain surveillance video data;

[0115] The second acquisition module is used to capture the equipment's operating data from the PLC interface and obtain the equipment's operating data.

[0116] The third acquisition module is used to receive location data sent by the UWB positioning tag worn by the workers and obtain personnel behavior data.

[0117] The fourth acquisition module is used to receive monitoring signals from the sensor group, and extract one or more of the following parameters from the monitoring signals: gas concentration, temperature, humidity, and dust concentration of the working environment, to obtain environmental monitoring data.

[0118] In the above embodiments, structured monitoring video data is obtained by extracting video streams containing equipment and / or workers from the video captured by the monitoring camera, and equipment operating parameters are captured in real time from the PLC interface to form accurate equipment operating data. High-precision positioning information sent by the UWPN342412 positioning tag worn by the workers is received to construct dynamic personnel behavior data, and environmental indicators such as gas concentration, temperature, humidity and dust concentration are collected through the sensor group to generate multi-dimensional environmental monitoring data. This achieves accurate source capture and semantic standardization output of four types of heterogeneous data in the coal mine operation environment.

[0119] The above embodiments ensure the integrity, consistency, and real-world mapping of the input data in the subsequent feature extraction stage, enabling the neural network to effectively fuse high-quality heterogeneous features. This provides a clear scene semantics as the input foundation for the safety big model trained on coal mine safety rules and semantic knowledge base, ultimately improving the accuracy and reliability of risk identification results.

[0120] In order to select the aforementioned security model, in one optional embodiment, the device further includes:

[0121] The third processing unit is used to obtain the safety specification text of the coal mine operation environment and process the safety specification text into a semantic knowledge base before processing the fused feature data using the safety big model to obtain the risk identification result.

[0122] The fourth processing unit is used to process the samples in the semantic knowledge base using masks and determine the logical relationship type between each sample. The masks include rule masks, causal masks and prohibition masks.

[0123] The first training unit is used to pre-train the large language model based on the processed semantic knowledge base until the loss function value of the large language model is less than the first preset value. The loss function includes masked language modeling loss, logical relation classification cross-entropy loss and term consistency loss.

[0124] The fifth processing unit is used to acquire historical accident logs, extract risk events from the historical accident logs, and extract corresponding multi-source heterogeneous data according to a preset time period with the timestamp corresponding to the risk event as the end time to obtain historical monitoring data.

[0125] The second training unit is used to train the pre-trained large language model with historical monitoring data as input data and risk events as labels, until the error rate of the large language model is less than the second preset value, and the large language model is determined as the safety large model.

[0126] In the above embodiments, a structured semantic knowledge base is constructed by parsing safety regulations text in the coal mine operation environment. Rule masks, causal masks, and prohibition masks are introduced to semantically and logically annotate the knowledge base samples, enabling the large language model to learn the implicit logical relationships and semantic constraints between safety rules during the pre-training stage. Combined with a multi-objective optimization mechanism of masked language modeling loss, logical relationship classification cross-entropy loss, and terminology consistency loss, the model's ability to understand safety semantics in a structured way is improved. Subsequently, using risk events in historical accident logs as endpoints, multi-source heterogeneous data within the corresponding time period are extracted as training samples. The pre-trained model is fine-tuned using risk events as labels, enabling the model to establish a precise mapping relationship from real-time environmental data representation to real risk events. Finally, a large safety model with semantic reasoning and multi-logical superposition risk identification capabilities is formed.

[0127] Through the above embodiments, the security big model is implemented from professional semantic knowledge base and security rules to real-world security events, realizing high-precision semantic reasoning and proactive identification of multi-dimensional and multi-level security risks. This effectively solves the problem of inaccurate identification of complex risk scenarios and frequent false alarms and missed alarms caused by the lack of joint modeling of the logical structure of security rules and the correlation with historical accidents in existing technologies.

[0128] In order to preserve the temporal and spatial characteristics of the video data, in one optional implementation, the first processing module includes:

[0129] The first processing submodule is used to process the surveillance video data using a three-dimensional convolutional neural network to obtain the first feature data.

[0130] In the above embodiments, by using a three-dimensional convolutional neural network to process the monitoring video data in the coal mine operation environment, the spatiotemporal correlation features of personnel behavior and equipment movement in the video sequence are directly extracted to form structured first feature data with time-series dynamic perception capabilities.

[0131] Through the above embodiments, it is ensured that video modal data can be matched and fused with equipment operation data, personnel behavior data and environmental monitoring data, providing fusion feature input with real scene semantic support for the safety big model, and improving the recognition accuracy and response reliability of complex risk scenarios with multiple logic superpositions.

[0132] To ensure compatibility between the device and its environment, in one optional implementation, the second processing module includes:

[0133] The second processing submodule is used to process the device operation data using a long short-term memory network to obtain the second feature data.

[0134] In the above embodiments, by employing a long short-term memory network to extract temporal features from equipment operation data in the coal mine operating environment, the dynamic dependency relationship of equipment operating status evolving over time is accurately modeled, generating second feature data that can characterize the abnormal evolution trend of equipment.

[0135] Through the above embodiments, the equipment status information is input into the neural network along with other heterogeneous data features for fusion, which enhances the temporal expression capability of the fused feature data. This enables the subsequent safety big data model to capture the complex temporal coupling relationship between equipment operation and multi-dimensional factors such as personnel behavior and environmental changes when combining coal mine safety rules and semantic knowledge base for risk identification. This allows for proactive early warning and precise prevention and control of potential chain risks caused by the evolution of equipment status in the coal mine operating environment.

[0136] To ensure the accuracy of video processing, in one optional implementation, the first processing submodule includes:

[0137] The third processing submodule is used to process the surveillance video data using a target monitoring model in order to identify whether there is a target object in the surveillance video data.

[0138] The fourth processing submodule is used to process the surveillance video data using a three-dimensional convolutional neural network to obtain the first feature data when the target object exists in the surveillance video data.

[0139] In the above embodiments, by first using a target monitoring model to pre-screen the surveillance video data to identify whether a target object exists, and only when the existence of a target object is confirmed, the computationally intensive three-dimensional convolutional neural network is activated to extract spatiotemporal features, thus avoiding redundant calculations for video frames without targets.

[0140] Through the above embodiments, the overall computational load and processing latency of the system are reduced, while ensuring the accuracy and real-time performance of feature extraction in key risk information areas. Thus, in the complex coal mine safety monitoring scenario of multi-source heterogeneous data fusion, efficient and accurate identification of high-risk behaviors is achieved.

[0141] The aforementioned model-based coal mine safety monitoring device includes a processor and a memory. The acquisition unit, first processing unit, second processing unit, and generation unit are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0142] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the accuracy of security monitoring.

[0143] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0144] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to execute the model-based coal mine safety monitoring method.

[0145] Specifically, model-based coal mine safety monitoring methods include:

[0146] Step S201: Obtain multi-source heterogeneous data in the coal mine operating environment. The multi-source heterogeneous data includes monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data.

[0147] Step S202: Extract features from multi-source heterogeneous data to obtain heterogeneous feature data, and use a neural network to fuse the heterogeneous feature data to obtain fused feature data;

[0148] Step S203: The fused feature data is processed using a safety big model to obtain the risk identification result. The safety big model is trained based on coal mine safety rules and a semantic knowledge base.

[0149] Step S204: Generate an alarm signal based on the risk identification result and send the alarm signal to a preset terminal.

[0150] This invention provides a processor for running a program, wherein the program executes the model-based coal mine safety monitoring method described above.

[0151] Specifically, model-based coal mine safety monitoring methods include:

[0152] Step S201: Obtain multi-source heterogeneous data in the coal mine operating environment. The multi-source heterogeneous data includes monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data.

[0153] Step S202: Extract features from multi-source heterogeneous data to obtain heterogeneous feature data, and use a neural network to fuse the heterogeneous feature data to obtain fused feature data;

[0154] Step S203: The fused feature data is processed using a safety big model to obtain the risk identification result. The safety big model is trained based on coal mine safety rules and a semantic knowledge base.

[0155] Step S204: Generate an alarm signal based on the risk identification result and send the alarm signal to a preset terminal.

[0156] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0157] Step S201: Obtain multi-source heterogeneous data in the coal mine operating environment. The multi-source heterogeneous data includes monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data.

[0158] Step S202: Extract features from multi-source heterogeneous data to obtain heterogeneous feature data, and use a neural network to fuse the heterogeneous feature data to obtain fused feature data;

[0159] Step S203: The fused feature data is processed using a safety big model to obtain the risk identification result. The safety big model is trained based on coal mine safety rules and a semantic knowledge base.

[0160] Step S204: Generate an alarm signal based on the risk identification result and send the alarm signal to a preset terminal.

[0161] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0162] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0167] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0168] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0171] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0172] 1) The model-based coal mine safety monitoring method of this application firstly acquires multi-source heterogeneous data in the coal mine operating environment, including monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data; then, it extracts features from the multi-source heterogeneous data to obtain heterogeneous feature data, and uses a neural network to fuse the heterogeneous feature data to obtain fused feature data; then, it uses a large safety model to process the fused feature data to obtain risk identification results, wherein the large safety model is trained based on coal mine safety rules and a semantic knowledge base; finally, it generates an alarm signal based on the risk identification results and sends the alarm signal to a preset terminal. By acquiring multi-source heterogeneous information such as monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data in the coal mine operation environment, features are extracted from various types of data to form heterogeneous feature data. A neural network is then used to achieve deep fusion of multimodal features, generating fused feature data representing the overall operation status. Furthermore, a large-scale safety model trained based on coal mine safety rules and a semantic knowledge base performs semantic understanding and logical reasoning on the fused feature data, accurately identifying complex risk scenarios composed of multiple overlapping safety conditions. This overcomes the limitations of traditional methods that rely on human experience and struggle to automatically identify risks with multiple logical couplings. Finally, alarm signals are automatically generated based on the identified risk results and pushed to preset terminals, achieving real-time, proactive, and intelligent early warning of complex safety risks with high concealment and strong correlation in coal mine operations. This solves the problem in existing technologies where safety monitoring equipment belongs to different systems, forming data silos and resulting in insufficient accuracy of automated safety detection.

[0173] 2) The model-based coal mine safety monitoring device of this application acquires multi-source heterogeneous data in the coal mine operating environment, including monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data; the first processing unit extracts features from the multi-source heterogeneous data to obtain heterogeneous feature data, and uses a neural network to fuse the heterogeneous feature data to obtain fused feature data; the second processing unit processes the fused feature data using a safety big model to obtain risk identification results, wherein the safety big model is trained based on coal mine safety rules and a semantic knowledge base; the generation unit generates an alarm signal based on the risk identification results and sends the alarm signal to a preset terminal. By acquiring multi-source heterogeneous information such as monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data in the coal mine operation environment, features are extracted from various types of data to form heterogeneous feature data. A neural network is then used to achieve deep fusion of multimodal features, generating fused feature data representing the overall operation status. Furthermore, a large-scale safety model trained based on coal mine safety rules and a semantic knowledge base performs semantic understanding and logical reasoning on the fused feature data, accurately identifying complex risk scenarios composed of multiple overlapping safety conditions. This overcomes the limitations of traditional methods that rely on human experience and struggle to automatically identify risks with multiple logical couplings. Finally, alarm signals are automatically generated based on the identified risk results and pushed to preset terminals, achieving real-time, proactive, and intelligent early warning of complex safety risks with high concealment and strong correlation in coal mine operations. This solves the problem in existing technologies where safety monitoring equipment belongs to different systems, forming data silos and resulting in insufficient accuracy of automated safety detection.

[0174] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model-based coal mine safety monitoring method, characterized in that, include: Acquire multi-source heterogeneous data within the coal mine operating environment, including surveillance video data, equipment operation data, personnel behavior data, and environmental monitoring data; Feature extraction is performed on the multi-source heterogeneous data to obtain heterogeneous feature data, and the heterogeneous feature data is fused using a neural network to obtain fused feature data; The fused feature data is processed using a large safety model to obtain risk identification results, wherein the large safety model is trained based on coal mine safety rules and a semantic knowledge base; An alarm signal is generated based on the risk identification result and sent to a preset terminal.

2. The method according to claim 1, characterized in that, Feature extraction is performed on the multi-source heterogeneous data to obtain heterogeneous feature data, including: Extract the spatiotemporal features of the surveillance video data to obtain the first feature data; Extract the temporal features of the device operation data to obtain the second feature data; The personnel behavior data is location-encoded, and the temporal features of the encoded personnel behavior data are extracted to obtain third feature data; The temporal features of the environmental monitoring data are extracted to obtain the fourth feature data; The first feature data, the second feature data, the third feature data, and the fourth feature data are timestamped to obtain the heterogeneous feature data.

3. The method according to claim 1, characterized in that, Acquire multi-source heterogeneous data within the coal mine operating environment, including: The surveillance video data is obtained by extracting video streams including equipment and / or personnel from the videos captured by the surveillance cameras; The device's operating data is obtained by capturing the device's operating data from the PLC interface; The system receives location data from the UWPN342412 positioning tag worn by the operator to obtain the operator's behavior data. The system receives monitoring signals from a group of sensors and extracts one or more of the following environmental monitoring data from the monitoring signals: gas concentration, temperature, humidity, and dust concentration of the working environment.

4. The method according to claim 1, characterized in that, Before processing the fused feature data using a large security model to obtain the risk identification result, the method further includes: Obtain the safety regulations text for the coal mine operating environment, and process the safety regulations text into the semantic knowledge base; The samples in the semantic knowledge base are processed using masks to determine the logical relationship types between the samples. The masks include rule masks, causal masks, and prohibition masks. The large language model is pre-trained based on the processed semantic knowledge base until the loss function value of the large language model is less than a first preset value. The loss function includes masked language modeling loss, logical relation classification cross-entropy loss, and term consistency loss. Obtain historical incident logs, extract risk events from the historical incident logs, and use the timestamp corresponding to the risk event as the end time to extract the corresponding multi-source heterogeneous data according to a preset duration to obtain historical monitoring data. Using the historical monitoring data as input data and the risk events as labels, the pre-trained large language model is trained until the error rate of the large language model is less than a second preset value, and then the large language model is determined as the security large model.

5. The method according to claim 2, characterized in that, Extracting the spatiotemporal features of the surveillance video data to obtain first feature data includes: The surveillance video data is processed using a three-dimensional convolutional neural network to obtain the first feature data.

6. The method according to claim 2, characterized in that, Extracting the temporal features of the device operation data to obtain second feature data includes: The device operation data is processed using a long short-term memory network to obtain the second feature data.

7. The method according to claim 5, characterized in that, The method further includes processing the surveillance video data using a three-dimensional convolutional neural network to obtain the first feature data. The surveillance video data is processed using a target monitoring model to identify whether a target object exists in the surveillance video data; If the target object exists in the surveillance video data, the surveillance video data is processed using the three-dimensional convolutional neural network to obtain the first feature data.

8. A model-based coal mine safety monitoring device, characterized in that, The device includes: The acquisition unit is used to acquire multi-source heterogeneous data in the coal mine operating environment, including monitoring video data, equipment operation data, personnel behavior data, and environmental monitoring data. The first processing unit is used to extract features from the multi-source heterogeneous data to obtain heterogeneous feature data, and to fuse the heterogeneous feature data using a neural network to obtain fused feature data. The second processing unit is used to process the fused feature data using a safety big model to obtain risk identification results, wherein the safety big model is trained based on coal mine safety rules and a semantic knowledge base. The generation unit is used to generate an alarm signal based on the risk identification result and send the alarm signal to a preset terminal.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

10. A security monitoring system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 7.

Images