A method for constructing and optimizing a multimodal knowledge base for coal mine fire and gas disasters

Abstract

The present invention relates to the technical field of intelligent prevention and control of typical disasters in coal mines, and discloses a method for constructing and optimizing a multimodal knowledge base for coal mine fire and gas disasters, comprising: constructing a unified ontology structure based on typical disasters of coal mine fire and gas; extracting entities, attributes and relationships between entities from a multimodal data source based on the unified ontology structure; integrating the extracted entities, attributes and relationships using a graph neural network model to form a structured knowledge graph; identifying and linking multimodal data entities based on the structured knowledge graph to build a multimodal knowledge base; acquiring historical data and real-time monitoring data of an area to be evaluated, and performing dynamic risk assessment based on entities and relationships in the multimodal knowledge base using a constructed risk assessment hybrid model; wherein the risk assessment hybrid model integrates GNN, DBN and LSTM; and updating attribute values ​​and relationships of entities in the multimodal knowledge base according to real-time monitoring data and risk assessment results.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent prevention and control of typical coal mine disasters, and relates to but is not limited to a method for constructing and optimizing a multimodal knowledge base of coal mine fire and gas disasters. Background Art

[0002] With the continuous advancement of coal mining technology and the increasing attention paid to coal mine safety production, the prevention and control of disasters such as coal mine fires and gas explosions has become an important part of coal mine safety management. However, the complexity and variability of coal mine fire and gas disasters make it extremely challenging to effectively prevent and control them. Coal mine fires and gas explosions usually involve multiple factors, such as the geological conditions of the coal seam, the design and operation of the mine ventilation system, and the monitoring and control of gas concentration.

[0003] Traditional disaster management methods mainly rely on experience and manual operations, lacking systematic and intelligent data support, which often leads to untimely responses and inadequate measures when dealing with sudden disasters. In particular, traditional methods are unable to cope with complex multimodal data, such as text records, image monitoring and table data.

[0004] In recent years, with the development of artificial intelligence technology, coal mine disaster management has gradually shifted towards intelligence and informatization. Through the collection, analysis and processing of multimodal data of coal mine fire and gas disasters, a comprehensive assessment and accurate prediction of disaster risks can be achieved. However, how to integrate scattered multi-source data and build a systematic knowledge base is still a difficult problem to be solved. Although the existing knowledge graph construction methods in the field of coal mine safety production have solved the problems of data integration and knowledge management to a certain extent, they still have many shortcomings when facing multimodal data. For example, the extraction and association technology of text, picture and table data is not mature enough, the efficiency and accuracy of data integration need to be improved, and the updating and maintenance of the knowledge base are also cumbersome. At the same time, the existing methods lack the risk assessment and early warning functions of coal mine fire and gas disasters, and cannot provide strong technical support for disaster prevention and control.

[0005] Therefore, there is an urgent need for a new method that can build a comprehensive knowledge base based on the multimodal data of coal mine fire and gas disasters, which can not only efficiently integrate multi-source data, but also realize real-time assessment and early warning of disaster risks. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method for constructing and optimizing a multimodal knowledge base of coal mine fire and gas disasters, which at least solves the problem that the existing technology cannot effectively integrate multimodal data of multiple disaster sources in coal mines, and manual processing of these data consumes a lot of time and manpower, and has a high error rate, resulting in inaccurate risk assessment, insufficient intelligent early warning capabilities, and incomplete disaster prevention measures.

[0007] The technical solutions of the embodiments of the present invention are as follows:

[0008] The embodiment of the present invention provides a method for constructing and optimizing a multimodal knowledge base of coal mine fire and gas disasters, the method comprising:

[0009] Construct a unified ontology structure based on typical disasters of coal mine fire and gas; based on the unified ontology structure, extract entities, attributes and relationships between entities from multimodal data sources; wherein the multimodal data sources include at least text, images and tables; use a graph neural network model to integrate the extracted entities, attributes and relationships to form a structured knowledge graph; based on the structured knowledge graph, identify and link multimodal data entities to build a multimodal knowledge base; obtain historical data and real-time monitoring data of the area to be evaluated, and based on the entities and relationships in the multimodal knowledge base, use the constructed risk assessment hybrid model to perform dynamic risk assessment; wherein the risk assessment hybrid model integrates a graph neural network GNN, a dynamic Bayesian network DBN and a long short-term memory network LSTM; according to the real-time monitoring data and the risk assessment results, update the attribute values ​​and relationships of the entities in the multimodal knowledge base.

[0010] Furthermore, the construction of a unified ontology structure based on typical coal mine fire and gas disasters includes: collecting literature, accident investigation reports, and technical specifications on coal mine fire and gas disasters, and extracting key concepts therefrom; wherein the key concepts include: disaster type, spatiotemporal attributes, disaster influencing factors, physical and chemical parameters, equipment and facilities, and emergency measures, and the disaster types include mine fires, mine gas disasters, and fire and gas coupled disasters; based on the extracted key concepts, determining the main entities in the field and the relationships between the entities to form a preliminary ontology concept model; selecting an ontology language to define classes and attributes, establishing relationships and clarifying object attributes, and constructing a unified ontology structure; formulating naming conventions for terms in the unified ontology structure, and determining the definitions of key entities in the unified ontology structure in combination with the characteristics of the coal mine fire and gas disaster field.

[0011] Furthermore, based on the unified ontology structure, entities, attribute properties and relationships between entities are extracted from multimodal data sources, including: collecting multimodal data sets of typical fire and gas disasters and performing data preprocessing; for the preprocessed data sets, building a joint extraction model to extract entities; wherein, the joint extraction model is based on the LEBERT model, superimposed with a Lattice-LSTM model to perform named entity recognition tasks, and constructed using a CRF conditional random field model on each hidden layer of the LSTM model; the LEBERT model is used to perform Chinese sequence labeling tasks; a rule-based template method is used to take the relationship between classes in the unified ontology structure as a relationship word in a relationship triplet, and extract the relationship between entities from the preprocessed data set.

[0012] Furthermore, the graph neural network model is used to integrate the extracted entities, attributes and relationships to form a structured knowledge graph, including: using the extracted entities as nodes, the relationships between the extracted entities as edges, assigning corresponding feature attributes to each of the nodes with the extracted attributes, and organizing all nodes and all edges according to a unified ontology structure to form an initial knowledge graph; respectively extracting the feature vectors corresponding to different modalities in each node of the initial knowledge graph to obtain a unified node feature representation; constructing a graph neural network model, performing message transmission and node update on the unified node feature representation of each node, so as to perform deep learning and optimization of node features; wherein, a multimodal attention mechanism is introduced in the message transmission process to perform weighted fusion of node features of different modalities.

[0013] Furthermore, the feature vectors corresponding to different modalities in each node of the initial knowledge graph are extracted respectively to obtain a unified node feature representation, including: using a pre-trained BERT model to map the text information in the node into a text feature vector; using a pre-trained ResNet-50 model to extract the image feature vector of the image information in the node; normalizing the numerical attributes of the table information in the node to form a numerical feature vector; one-hot encoding the categorical attributes to obtain a categorical feature vector; and concatenating the text feature vector, the image feature vector, the numerical feature vector and the categorical feature vector to form the unified node feature representation.

[0014] Furthermore, node update is achieved through the following formula:

[0015] ;

[0016] In the formula, For Node i The updated feature representation is, σ is the activation function, For Node i The weight of the self-loop edge, node i The weight coefficient of is a learnable weight matrix used to adjust the node i Its own characteristics h i Perform a linear transformation, h i For Node i The initial feature representation vector of contains the information after multimodal fusion. For Node i The set of neighbor nodes of For edge ( i,j ), Neighbor node j The weight coefficient of is a learnable weight matrix used to adjust the neighbor node features h j Perform a linear transformation, h j Neighbor node j The feature representation vector of For Node i The normalization coefficient of .

[0017] Furthermore, based on the structured knowledge graph, multimodal data entities are identified and linked to build a multimodal knowledge base, including: determining the text entity set, image entity set and audio entity set in the structured knowledge graph; taking the text entity set as a candidate entity set, and using the STELM model to link the image entity set and the audio entity set to the candidate entity set; defining entity storage rules, adopting the storage method of a graph database, and realizing the storage of a multimodal knowledge base composed of entity nodes and edges.

[0018] Furthermore, the system architecture of the multimodal knowledge base includes: a data storage layer, which is used to store multimodal data using a hybrid storage solution, including: a graph database for storing nodes and edges of the knowledge graph, supporting efficient graph query and analysis; a relational database for storing structured data; a file storage system for storing unstructured data; a knowledge representation layer, which is used to construct a structured knowledge graph using a resource description framework and a Web ontology language, and to standardize the representation of entities, attributes and relationships in the field of coal mine fires and gas disasters; a knowledge access layer: providing a unified access interface for users and applications, supporting at least two query languages; an application service layer, which includes at least three functional modules: risk assessment, intelligent early warning, and data analysis, and is used to perform real-time dynamic risk assessment, accident cause analysis, and generate corresponding early warning prompts and emergency measures based on the multimodal data provided by the multimodal knowledge base; a user interaction layer, which provides users with a friendly visualization interface and human-computer interaction functions, and supports the visualization of fire and gas knowledge graphs and the graphical output of data analysis results.

[0019] Furthermore, the GNN module in the risk assessment hybrid model is used to utilize the structured knowledge graph to capture the complex topological structure and association relationships between risk factors, and generate a high-dimensional representation of nodes; the DBN module is used to model the dynamic changes of risk factors, capture the random processes in the time series, and reflect the evolution of the risk status; the LSTM module is used to process time series data, capture long-term and short-term dependencies, and predict future trends of risk factors; the risk assessment hybrid model combines the node representation extracted by the GNN module with the time series features output by the LSTM module, and inputs them into the DBN module to achieve a comprehensive assessment of coupled disaster risks.

[0020] Furthermore, the constructed risk assessment hybrid model is used to perform dynamic risk assessment, including: combining the inference results of the DBN module and the predicted output of the LSTM module to determine the posterior probability of the target event; based on the posterior probability, comprehensively considering various risk factors to determine a comprehensive risk score; according to the size of the comprehensive risk score and a preset threshold, the risk assessment results are divided into four levels: low risk, medium risk, high risk and extremely high risk; and corresponding early warning and emergency measures are formulated for different risk levels.

[0021] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0022] 1) This paper provides an entity extraction method based on a joint extraction model, which combines the vocabulary-enhanced BERT model and the Lattice-LSTM structure, and uses professional dictionaries in the field of coal mine fires and gas disasters to improve the accuracy of entity recognition. At the same time, a rule-based relationship extraction method is used to apply the relationship rules in the domain ontology to entity pairs, which improves the efficiency and accuracy of relationship extraction.

[0023] 2) The graph neural network model was introduced, and a multimodal feature fusion and dynamic weighting mechanism were designed to achieve efficient integration of multi-source heterogeneous data. By weighting nodes and edges, key risk factors and important correlations were highlighted, and a high-quality coal mine fire and gas disaster knowledge graph was constructed.

[0024] 3) A hybrid model integrating graph neural network, dynamic Bayesian network and long short-term memory network was constructed to realize multimodal risk dynamic assessment of coal mine fire and gas disasters. This model fully utilizes the advantages of multimodal data, captures the complex topological structure and temporal dynamic characteristics between risk factors, and improves the accuracy and real-time performance of risk assessment.

[0025] 4) Through the dynamic risk assessment method, the knowledge base is optimized through feedback while realizing the quantitative assessment of risk factors, thus realizing the dynamic update and self-improvement of the knowledge base. This method combines historical data and real-time monitoring data to further express the causal and temporal complex relationship in the field of coal mine fires and gas disasters, solves the problem of multimodal data fusion, improves the integrity and accuracy of the knowledge base, and provides support for the comprehensive knowledge management of coal mine fire and gas disasters and the tracing prediction of subsequent disaster chains. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0027] Figure 1 A schematic diagram of a process for constructing and optimizing a multimodal knowledge base for coal mine fire and gas disasters provided by an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a knowledge base dynamic update and feedback closed loop provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the structure of the coal mine fire and gas disaster entity provided by an embodiment of the present invention;

[0030] Figure 4A schematic diagram of the processing flow of the joint extraction model provided by an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of a multi-source data integration process based on a graph neural network provided in an embodiment of the present invention;

[0032] Figure 6 A schematic diagram of the structure of a multimodal feature fusion and graph neural network model provided in an embodiment of the present invention;

[0033] Figure 7 A diagram of the architecture of a multimodal knowledge base system provided by an embodiment of the present invention;

[0034] Figure 8 A schematic diagram of the processing flow of the entity linking model provided by an embodiment of the present invention;

[0035] Fig. 9 A schematic diagram of a knowledge graph with a relational hierarchical structure provided by an embodiment of the present invention;

[0036] Fig.10 A schematic diagram of the overall process of multimodal risk dynamic assessment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described here can be implemented in an order other than that illustrated or described here.

[0040] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the embodiments of the present invention belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0041] Figure 1 A schematic diagram of a process flow of a coal mine fire and gas disaster multimodal knowledge base construction and optimization method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises at least the following steps:

[0042] Step S110, constructing a unified ontology structure based on typical disasters of coal mine fire and gas.

[0043] Here, a unified ontology structure suitable for the field of coal mine fire and gas disasters is constructed to support the construction of a multimodal knowledge base and the application of risk assessment models. By combining professional knowledge and knowledge graph technology in the field of coal mine fire and gas disasters, it is ensured that the unified ontology structure is both in line with industry reality and has good expansion compatibility.

[0044] In implementation, a unified ontology structure can be constructed through the following process: first, collect literature, accident investigation reports, technical specifications and other materials related to coal mine fires and gas disasters to analyze domain knowledge; second, select an ontology language to define classes and attributes, and establish relationships to clarify object attributes; finally, formulate naming conventions and clarify the definition of special entities based on the characteristics of the coal mine fire and gas disaster field.

[0045] Step S120: extracting entities, attributes, and relationships between entities from the multimodal data source based on the unified ontology structure.

[0046] Here, the multimodal data source includes at least text, images and tables. Text data includes coal mine fire and gas disaster accident investigation documents, emergency plans and other document materials, image data includes atlases, fire monitoring images, and table data includes worksheets.

[0047] In the implementation, we first collected multimodal datasets of typical fire and gas disasters through experimental simulation, cameras and sound monitoring devices placed at fixed locations underground, etc. Then we extracted multimodal entity-relationships of typical coal mine fire and gas disasters.

[0048] Step S130, using the graph neural network model to integrate the extracted entities, attributes and relationships to form a structured knowledge graph.

[0049] Here, entities include "gas concentration", "fire incident", "mine", etc., and attributes describe the characteristics of these entities (such as the value and unit of gas concentration). Through relationships (such as "occurred in", "caused by", "monitored"), these entities are connected to form a structured knowledge network.

[0050] The graph neural network model is introduced to establish the association relationship between multimodal data, so as to realize the efficient integration of multi-source heterogeneous data such as text, images, and tables, and build a high-quality knowledge graph. For example, data such as "gas concentration" and "temperature change" from different data sources are integrated into a comprehensive description of disaster events. At the same time, the integrated initial knowledge graph is imported into the relational database (MySQL) and the graph database (Neo4j) to realize the structured storage of data, establish indexes of entities, attributes, and relationships, and support efficient query and analysis. The key is to design a multimodal fusion graph neural network model to make full use of the characteristics of each modal data, capture complex entity relationships, and improve the completeness and accuracy of the knowledge base.

[0051] Step S140, based on the structured knowledge graph, identify and link multimodal data entities to build a multimodal knowledge base.

[0052] Here, the Neo4j graph database is used to build a multimodal knowledge base, into which the structured knowledge graph is imported, covering the characteristics of disaster risk factors, physical and chemical parameters, and spatial information, providing a basis for the systematic storage and utilization of knowledge. At the same time, the database management system can also provide efficient data query and analysis functions.

[0053] The multimodal knowledge base provides a flexible query interface and supports multiple query methods, including keyword query, entity query, relationship query, etc. It provides data analysis tools to support data visualization and statistical analysis. It supports visual browsing of knowledge graphs, which facilitates users to understand the overall picture of coal mine fire and gas disasters. It can provide knowledge support for coal mine safety management, including accident analysis, emergency plan formulation, etc. It supports intelligent early warning and prevention measures for coal mine fire and gas disasters, and conducts real-time monitoring and prediction through risk assessment models. It provides comprehensive knowledge resources for researchers and decision makers to support scientific research and decision-making related to coal mine safety.

[0054] Step S150, obtaining historical data and real-time monitoring data of the area to be assessed, and performing dynamic risk assessment based on the entities and relationships in the multimodal knowledge base using the constructed risk assessment hybrid model.

[0055] Here, the risk assessment hybrid model combines the Graph Neural Network (GNN), Dynamic Bayesian Network (DBN) and Long Short-Term Memory (LSTM). The model fully utilizes the advantages of multimodal data, captures the complex topological structure and temporal dynamic characteristics between risk factors, and improves the accuracy and real-time performance of risk assessment. Based on the entities and relationships in the multimodal knowledge base, a DBN structure diagram is constructed, where nodes represent risk factors and edges represent causal relationships. GNN combines knowledge graphs to capture complex associations in multi-source data and improve the accuracy of risk assessment. LSTM processes time series data, captures long-term dependencies, and predicts the trends of risk factors.

[0056] Risk assessment mainly evaluates certain specific entities (accident entities, hidden danger entities) in the knowledge base, including the evaluation of key causes and triggering conditions of coal mine fire and gas disasters, and determines the mapping relationship and weights between them, so as to achieve a comprehensive prevention assessment before mine fire and gas disasters, so as to support intelligent early warning and prevention of coal mine fire and gas disasters. In some embodiments, the mapping relationship and weights in the risk assessment hybrid model are determined, and a risk assessment report is generated to provide a comprehensive prevention assessment before the disaster. In some embodiments, dynamic monitoring and early warning of coal mine fire and gas disasters can also be carried out based on the risk assessment value output by the risk assessment hybrid model.

[0057] The implementation of the risk assessment model includes: establishing and training a machine learning model, optimizing model parameters through historical data and expert knowledge; combining multimodal data to conduct a comprehensive analysis of disaster risk factors; and using the risk assessment values ​​output by the model to dynamically monitor and warn of coal mine fire and gas disasters.

[0058] Step S160: updating the attribute values ​​and relationships of entities in the multimodal knowledge base according to the real-time monitoring data and the risk assessment result.

[0059] Here, we use new monitoring data and adaptive learning mechanisms to dynamically adjust model parameters and improve assessment accuracy. We build a closed-loop system from data collection, risk assessment, early warning decision-making to knowledge updating to achieve self-optimization and continuous improvement of the model. Figure 2 As shown, it includes implementing feedback data, feeding back and optimizing the parameters of the risk assessment hybrid model, and updating the inference rules.

[0060] Through the dynamic risk assessment method, the knowledge base is optimized through feedback while achieving quantitative assessment of risk factors, realizing dynamic updating and self-improvement of the knowledge base, and maintaining the timeliness and accuracy of knowledge. Combining historical data and real-time monitoring data, it further expresses the causal and temporal complex relationship in the field of coal mine fires and gas disasters, solves the problem of multimodal data fusion, improves the integrity and accuracy of the knowledge base, and provides strong support for the comprehensive knowledge management of coal mine fires and gas disasters and the tracing prediction of subsequent disaster chains.

[0061] The coal mine fire and gas disaster multimodal knowledge base construction and optimization method provided by the embodiment of the present invention can build a comprehensive knowledge base based on the multimodal data of coal mine fire and gas disasters, which can not only efficiently integrate multi-source data, but also realize real-time assessment and early warning of disaster risks. By introducing advanced natural language processing, image recognition and machine learning technologies, the construction efficiency and intelligence level of the knowledge base can be greatly improved, providing scientific and comprehensive decision-making support for coal mine safety management.

[0062] In some embodiments, the above step S110 is implemented through the following steps: collecting literature, accident investigation reports, and technical specifications on coal mine fires and gas disasters, and extracting key concepts therefrom; wherein the key concepts include: disaster types, spatiotemporal attributes, disaster influencing factors, physical and chemical parameters, equipment and facilities, and emergency measures, and the disaster types include mine fires, mine gas disasters, and fire and gas coupled disasters; based on the extracted key concepts, determining the main entities in the field and the relationships between entities to form a preliminary ontology concept model; selecting an ontology language to define classes and attributes, establishing relationships and clarifying object attributes, and constructing a unified ontology structure; formulating naming conventions for terms in the unified ontology structure, and determining the definitions of key entities in the unified ontology structure in combination with the characteristics of the field of coal mine fires and gas disasters.

[0063] Here, the literature materials include but are not limited to the "Coal Mine Safety Regulations", "Coal Mine Fire Prevention and Extinguishing Rules", "Coal Mine Spontaneous Ignition Marker Gas Chromatographic Analysis and Index Gas Optimization Method", "Mine Closed Fire Prevention and Fire Extinguishing Technical Specifications", "Coal Mine Design Fire Protection Specifications" accident investigation reports include "Coal Mine Fire Accident Investigation Reports over the Years", "Gas Explosion Accident Analysis Report", etc.; in terms of academic papers and research results, Citespace literature intelligence analysis software is used to organize research papers on coal mine safety management, fire and gas disaster mechanisms, simulation and typical scenario analysis at home and abroad in the past 15 years; technical manuals and textbooks include "Coal Mine Safety Technical Manual", gas prevention and control and monitoring technology textbooks, etc.

[0064] Extract key concepts from the collected data, including: ① Clarify the disaster type and define typical coal mine fire and gas disasters: mine fires include internal fires (coal spontaneous combustion fires) and external fires (electrical fires, mechanical friction fires, etc.). Mine gas disasters include: gas outburst, gas explosion, gas combustion, etc. Fire and gas coupling disasters (coal spontaneous combustion-gas explosion, tunnel fire-gas explosion, working face fire-gas explosion, gas combustion thermal radiation-coal spontaneous combustion-gas explosion, gas combustion thermal radiation-gas explosion, etc.) ② Determine the spatiotemporal attributes: Time attributes include the date of the accident, time period, duration, etc. Spatial attributes include mine name, working face number, geographic coordinates, etc. ③ Clarify the influencing factors of the disaster: geological factors include coal seam occurrence conditions, geological structure, etc., and internal environmental factors of the mine: temperature, humidity, ventilation conditions. Human factors include operational errors and management omissions. ④ Physical and chemical parameters: gas concentration: unit is percentage (%); temperature: unit is Celsius (℃); pressure: unit is Pascal (Pa). ⑤ Equipment and facilities include: ventilation equipment (fans, dampers, air ducts), monitoring instruments (gas detectors, temperature sensors, etc.). ⑥ Emergency measures include prevention strategies (gas extraction, isolation and closure) and rescue actions (personnel evacuation, fire extinguishing plans).

[0065] According to the extracted key concepts, the main entities in the domain and the relationships between them are determined to form a preliminary ontology concept model. ① Entity definitions include disaster event, location, time, equipment, personnel, physical parameter, influencing factor, and emergency measure. ② Relationship definitions include occurrence (occursAt): disaster event-location. Example: "Gas explosion" occurred at "No. 3 working face of East Mining Area 1"; occurred (occursOn): disaster event-time. Example: "Fire" occurred on "May 12, 2023"; causes (causes): entity-entity. Example: "Excessive gas concentration" causes "gas explosion"; affects (affects): entity-entity. Example: "Poor ventilation" affects "gas accumulation"; uses (uses): personnel / measures-equipment. Example: "Rescuers" use "gas detectors"; monitors (monitors): equipment-physical parameters. Example: "Temperature sensor" monitors "temperature"; executes: personnel-emergency measures. Example: "Rescue team" performs "personnel evacuation"; has attribute: entity-attribute. Example: "Gas concentration" has the attribute "3%".

[0066] Select the ontology language: Use OWL (Web Ontology Language) as the ontology description language. Further define classes and attributes, such as Figure 3 The figure shows a unified ontology structure diagram in the field of coal mine fire and gas disasters, where the top-level classes include entity, event, and attribute; entity subclasses include location, equipment, personnel, physical parameters, and influencing factors; event subclasses include disaster events: fire, gas disaster, and fire and gas coupling disaster. Attribute subclasses include time attributes and numerical attributes. Establish relationships and clarify object properties, including "occursAt": connect events and locations; "occursOn": connect events and time; "causes": connect risk factors and events; "affects": connect different risk factors; and "monitors": connect equipment and physical parameters.

[0067] Use a unified naming convention to avoid ambiguity and duplication. At the same time, ensure that the terms use a unified language and expression throughout the ontology structure. For example, use CamelCase: capitalize the first letter of the class name and attribute name, and capitalize the first letter of each word in a compound word. Examples: DisasterEvent, GasConcentration, occursAt. At the same time, to avoid ambiguity and duplication, perform the following processing:

[0068] Synonym handling: Specify a preferred name for different names of the same concept, and record other names as synonyms. Example: "Gas", "Coalbed Methane", and "Methane" are unified as Gas.

[0069] Polysemous word distinction: For polysemous words, different concepts are divided according to context or field. Example: "Pressure" can be divided into "Gas Pressure" and "Formation Pressure".

[0070] The entire unified ontology structure adopts the unified Chinese language and the terms commonly used in the coal mining industry, avoiding self-made words or non-standard expressions. A glossary is established to record all the terms used and their definitions for subsequent reference. The naming examples are as follows: class name (such as DisasterEvent, Fire, GasDisaster, Equipment, Location), attribute name (such as hasValue, hasUnit, hasName, occursAt, causes, affects), instance name (such as fireEvent20230512, mainVentilator01, gasSensorA1, mineLocationEastWing).

[0071] Finally, combined with the characteristics of the field of coal mine fire and gas disasters, the definition of special entities is clarified: Coal Seam, that is, a geological entity unique to coal mines, needs to be clearly defined in the ontology. Ventilation System, that is, an important facility for coal mine safety, including fans, air ducts, dampers and other sub-equipment. Gas Drainage System, a system used to reduce the gas content of coal seams. Description of specific relationships: The relationship between gas concentration and ventilation volume, the gas concentration changes with the ventilation volume, and the relationship influencedBy can be defined. The relationship between fire and spontaneous combustion tendency, the coal seam has a high spontaneous combustion tendency and the fire risk increases, and the relationship hasRiskFactor can be defined. Classification of emergency measures: Preventive measures (PreventiveMeasure): such as strengthening ventilation and gas extraction. Control measures (ControlMeasure): such as closed isolation and nitrogen injection fire extinguishing. Rescue measures (RescueMeasure): such as personnel evacuation and emergency rescue.

[0072] Finally, the ontology reasoning engine is used to infer new knowledge based on the defined classes and relationships. Complex queries are performed on the ontology structure and knowledge base through query languages ​​such as SPARQL. In order to achieve cross-domain data sharing, mapping and alignment with other ontologies are considered to obtain a unified ontology structure, which lays the foundation for multi-source data integration based on graph neural networks in subsequent steps.

[0073] In some embodiments, the above step S120 is further implemented through the following process: collecting a multimodal dataset of typical fire and gas disasters and performing data preprocessing; for the preprocessed dataset, constructing a joint extraction model for entity extraction; wherein, the joint extraction model is based on the LEBERT model, superimposing a Lattice-LSTM model to perform a named entity recognition task, and using a CRF conditional random field model on each hidden layer of the LSTM model to construct the model; the LEBERT model is used to perform Chinese sequence labeling tasks; a rule-based template method is used to take the relationship between classes in the unified ontology structure as a relationship word in a relationship triplet, and extract the relationship between entities from the preprocessed dataset.

[0074] Here, the LEBERT model (Lexicon Enhanced BERT) is a model for Chinese sequence labeling tasks, which improves performance by incorporating vocabulary information into the underlying encoding process of BERT. The LEBERT model can be seen as a combination of Lexicon Adapter and BERT, where the Lexicon Adapter is applied between a certain Transformer layer of BERT.

[0075] Extracting entities, attributes and their relationships from multimodal data sources mainly includes three stages: multimodal knowledge data collection, entity extraction based on the joint extraction model (LEBERT-Lattice-LSTM-CRF) model, and rule-based relationship extraction. The structure of the joint extraction model is as follows: Figure 4 As shown, where R1, R2, ... R n Represents entities or tags in the input text, which will be processed to extract relations; label0, label1 refer to different labels assigned to entities to indicate their role in the relationship (for example, entity type or specific role in the structure); B-disaster, IB-disaster, EB-disaster, these labels follow the B / I / E labeling scheme commonly used in named entity recognition, where B represents the "beginning" of the entity, I represents the "inside" or continuation of the entity, and E represents the "end" of the entity; CRF (Conditional Random Field) is a probabilistic model for structured prediction, where CRF may be used to optimize sequence labeling to help predict the most likely label for each tag, such as entity type or relationship; U usually represents the input vector or matrix of the model, which is the initial input embedding of tags or characters in the text; h, h0, h1, ..., h nRepresents the hidden states of the LSTM network, which captures the sequence information of the input. Each state (h0, h1, etc.) represents the information before that point in the sequence. TRM (Token Representation Module) is used to generate the representation of each token in the text, which may combine character-level and word-level information. E0, E1, ..., E n Represents the embeddings of each character or token in the input sequence. These embeddings provide the underlying representations for the model to learn entity and relation patterns.

[0076] First, a multimodal dataset of typical fire and gas disasters is collected through experimental simulation, cameras and sound monitoring devices are deployed at fixed positions underground. Text data preprocessing mainly includes corpus cleaning, Chinese word segmentation, part-of-speech tagging and other steps. Corpus cleaning includes: deleting irrelevant data and duplicate data and removing garbled and redundant symbols. After the text data is cleaned, the LTP platform is used to perform Chinese word segmentation and construct a label table. Among them, "B" means "entity start (Begin)", "I" means "entity inside (Inside)", "O" means "words or punctuation marks outside the entity (Outside)", "E" means "entity end (End)", and "S" means "entity word (Single)". The semantically clear text labels of multimodal data, that is, the entity names of multimodal data, provide the necessary data basis for the subsequent addition of multimodal attributes of entities.

[0077] Secondly, a joint extraction model is constructed for entity extraction. The specific process is as follows: by obtaining typical fire and gas disaster data in coal mines, professional terms with field specificity are extracted as feature words, and then a feature word dictionary is constructed. Through the word vector model, its feature words are converted into word vectors and stored in the form of prefix trees, denoted as T. On this basis, using T and a sentence containing n characters Traverse all character subsequences in the sentence and match them with the prefix tree to obtain all potential paired words. For example, the input sentence "Gas explosion is preventable" can be matched with three feature words, namely "gas", "explosion" and "gas explosion". Use the above three feature words to match the sentence characters containing the word respectively. When the character matches the feature word successfully, the information at each position is transformed from a single character to a character and its corresponding feature phrase. Then, a nonlinear transformation is performed on the feature phrase vector, and a bilinear attention mechanism is introduced to determine the importance of each feature word in the feature phrase. After that, on the basis of the LEBERT deep neural network model, the Lattice-LSTM model is superimposed for the named entity recognition task, and the output sequence of the LEBERT model and the forward vocabulary information required by the pre-constructed vocabulary table are selected and integrated into the character information. Finally, the CRF model is used on the hidden layer of each layer to calculate the probability of entity sequence generation.

[0078] Finally, rule-based relationship extraction: Since the data collected in this project contain relatively clear associations, a rule-based template method is used to take the relationship between classes in the coal mine fire and gas disaster ontology as the relationship words in the relationship triple, thereby realizing the relationship extraction between entities. If a pair of entity pairs with corresponding rules appears in a sentence, the entity pairs and the corresponding relationships form a relationship triple and are stored in a database based on Neo4j driver objects. If several entities appear in a sentence, an array is created and named after the entity type to temporarily store the entities; then all entity types are combined in pairs. If two entity types have corresponding rules, the entities in the corresponding arrays are combined; otherwise, they are not combined.

[0079] This method combines the vocabulary-enhanced BERT model and the Lattice-LSTM structure, and uses the professional dictionary in the field of coal mine fire and gas disasters to improve the accuracy of entity recognition. At the same time, a rule-based relationship extraction method is adopted to apply the relationship rules in the domain ontology to entity pairs, which improves the efficiency and accuracy of relationship extraction.

[0080] In some embodiments, the above step S130 is implemented through the following steps: using the extracted entities as nodes, the relationships between the extracted entities as edges, assigning corresponding feature attributes to each of the nodes using the extracted attributes, and organizing all nodes and all edges according to a unified ontology structure to form an initial knowledge graph; respectively extracting the feature vectors corresponding to different modalities in each node of the initial knowledge graph to obtain a unified node feature representation; constructing a graph neural network model, performing message passing and node update on the unified node feature representation of each node, so as to perform deep learning and optimization of the node features; wherein, a multimodal attention mechanism is introduced in the message passing process to perform weighted fusion of node features of different modalities.

[0081] Here, if Figure 5 The figure shows the multi-source data integration process based on graph neural network. First, multimodal extraction is performed from text, images, and tables. Then, entities and relationships in the text, image, and table data are constructed into an initial knowledge graph based on a unified ontology structure. Multimodal feature fusion is then performed at the input layer of the graph neural network model. At the same time, the domain knowledge in the knowledge base is used to guide the weighting of nodes and edges in the graph neural network model. For example, according to the coal mine safety regulations, a safety threshold for gas concentration is set, and a higher weight is given to the "gas concentration" node that exceeds the standard. After the model training is completed, the knowledge base is updated, and the node representation and relationship prediction results output by the model are used to update the knowledge base, improve the initial knowledge graph, and add new high-risk factors or important relationships identified by the model to the knowledge base.

[0082] Firstly, nodes and edges are determined based on the extracted entities, attributes and relationships, and organized according to a unified ontology structure to form a knowledge graph for coal mine fire and gas disasters.

[0083] Vertices use entities extracted from multi-source data as nodes, including: risk factor nodes (gas concentration, temperature, pressure, etc.), which represent key parameters affecting coal mine safety; event nodes (gas explosion accidents, fire accidents, fire and gas coupling accidents), which represent actual disaster events; equipment nodes (such as gas sensor A1, ventilation equipment B1, etc.), which represent equipment used for monitoring and control; location nodes (such as the No. 3 working face of the East 1 mining area, a coal mine, etc.), which represent specific geographical locations; personnel nodes (rescue teams, miners, etc.), which represent related personnel entities.

[0084] Edges use the relationship between entities as edges, including causal relationships (causes) used to connect risk factor nodes and event nodes, indicating that a certain risk factor leads to the occurrence of a certain event, such as excessive gas concentration leading to a gas explosion accident; monitoring relationships (monitoredBy) connect risk factor nodes and equipment nodes, indicating that a certain risk factor is monitored by a certain device, such as gas concentration monitoredBy gas sensor A1; spatial relationships (occursAt): connecting event nodes and location nodes, indicating that an event occurs at a certain location, such as a gas explosion accident occurred at the No. 3 working face of the East 1 mining area; time relationships (occursOn) connect event nodes and time nodes, indicating that an event occurs at a certain time, such as a fire and gas coupling accident occurred on May 12, 2023); impact relationships (affects) connect risk factor nodes and risk factor nodes, indicating that one risk factor affects another risk factor, such as poor ventilation affects increased gas concentration.

[0085] Attributes are used to assign characteristic attributes to each node. For example, the attributes of gas concentration nodes include value (6%), unit (%), and measurement time; the attributes of event nodes include event type, severity, casualties, etc. The attributes of equipment nodes include equipment model, accuracy, and operating status.

[0086] Then, the graph neural network model is introduced to use graph structure data to learn the representation (embedding) of nodes and capture the complex associations in the graph, thereby improving the quality of the knowledge graph. Figure 6 As shown in the figure, the model features of text, image, numerical value and category are fused in the model input layer, and then the neighbor node messages are aggregated in the stacked GNN layer, and the weighted processing of nodes and edges is added. Finally, the updated node representation and relationship prediction results are output.

[0087] In order for the model to accurately predict risks and identify important relationships, it is necessary to train and optimize the model. For labeled risk nodes, the cross entropy loss function is used:

[0088] ;

[0089] In the formula, is the cross entropy loss, is a set of labeled nodes, Display Node i The true label (such as high risk, low risk) is 0 or 1. represents the node label probability predicted by the model, Representation Node iThe loss weight is given to high-risk nodes with greater weight.

[0090] Adaptive optimization algorithm (AdamW) is used to minimize the loss function and update the model parameters. To prevent overfitting, L2 regularization term and Dropout mechanism are introduced, and cosine annealing strategy is used to optimize the model convergence speed and effect. The model training process includes dividing the training set, validation set and test set. Initialize node features and model parameters. According to the message passing formula, calculate the updated features of the node. Calculate the loss function L. Calculate the gradient, update the model parameters, evaluate the model performance on the validation set, and adjust the parameters.

[0091] In some embodiments, the feature vectors corresponding to different modalities in each node of the initial knowledge graph are extracted respectively to obtain a unified node feature representation, including: using a pre-trained BERT model to map the text information in the node into a text feature vector; using a pre-trained ResNet-50 model to extract the image feature vector of the image information in the node; normalizing the numerical attributes of the table information in the node to form a numerical feature vector; one-hot encoding the categorical attributes to obtain a categorical feature vector; and concatenating the text feature vector, the image feature vector, the numerical feature vector, and the categorical feature vector to form the unified node feature representation.

[0092] Here, the feature vectors of different modalities: text feature vector h text (contains semantic information, such as accident, risk factor names), image feature vector h image (contains visual information), numerical feature vector h numeric (containing numerical information) and the category feature vector h categorical (including category attributes such as device status) to form a unified node feature representation h i : , where “;” represents the concatenation operation of vectors.

[0093] In some embodiments, node updating is achieved by the following formula:

[0094] ;

[0095] In the formula, For Node i The updated feature representation is, σ is the activation function, For Node i The weight of the self-loop edge, For Node i The weight coefficient of is a learnable weight matrix used to adjust the node i Its own characteristics h i Perform a linear transformation, h i For Node i The initial feature representation vector of contains the information after multimodal fusion. For Node i The set of neighbor nodes of For edge ( i,j ), Neighbor node j The weight coefficient of is a learnable weight matrix used to adjust the neighbor node features h j Perform a linear transformation, h j Neighbor node j The feature representation vector of For Node i The normalization coefficient of .

[0096] In order to highlight key risk factors and important relationships, the nodes and / or edges are weighted in the graph neural network model. In some embodiments, entities in the initial knowledge graph are used as nodes and assigned different weights. ω i (node i The weight coefficient reflects the importance of the node). It mainly includes: risk factor nodes (gas concentration, temperature, pressure...); event nodes (gas explosion accidents, fire events); equipment nodes (such as gas sensor A1, ventilation equipment B1); location nodes (such as "No. 3 working face of East 1 mining area", and the specific weight settings are shown in Table 1:

[0097] Table 1 Example of node weight setting

[0098]

[0099] In some embodiments, the relationship between entities is used as an edge, and different weights are assigned to the edges. α ij (side( i,j ) weight coefficient, reflecting the importance of the relationship), mainly including: ① causal relationship (causes) connects risk factor nodes and event nodes; ② monitoring relationship (monitoredBy) connects risk factor nodes and equipment nodes; ③ spatial relationship (occursAt) connects event nodes and location nodes; ④ temporal relationship (occursOn) connects event nodes and time nodes.

[0100] On this basis, key nodes and relationships are weighted. For high-risk nodes, such as nodes with excessive gas concentration, higher risk weights are assigned; for important relationship edges, such as the "causes" relationship, higher relationship weights are assigned to emphasize their importance in risk propagation.

[0101] Table 2 Example of edge weight setting

[0102]

[0103] In order to solve the differences in scale and information content of different modal features, a weighted fusion mechanism is designed to assign different weights to each modal feature: ;in , , , are text feature vectors , image feature vector , table feature vector and the category feature vector The weight coefficient is set according to the importance of the mode in a specific case. Taking the coupled disaster of coal spontaneous combustion and gas explosion as an example, the weight setting is shown in Table 3:

[0104] Table 3 Modal weight setting example

[0105]

[0106] In some embodiments, the above step S140 "based on the structured knowledge graph, identify and link multimodal data entities, and build a multimodal knowledge base" further includes the following steps: determine the text entity set, image entity set and audio entity set in the structured knowledge graph; use the text entity set as a candidate entity set, and use the STELM model to link the image entity set and the audio entity set to the candidate entity set; define entity storage rules, adopt the storage method of the graph database, and realize the storage of the multimodal knowledge base composed of entity nodes and edges.

[0107] In the implementation, the text entity set related to typical disasters of coal mine fire and gas is used as the candidate entity set, and the multimodal entity set (reference set) such as image and audio is linked to the candidate entity set. The entity linking task is performed by using the STELM model. Each BERT model in the STELM model processes a pair of references and candidate entities respectively, and multiple BERT models are processed in parallel, and then the vectors output by each model are spliced ​​together and input into a multi-classifier. First, each reference and its corresponding candidate entity name and description are spliced ​​to construct the candidate entity context. The input form is; then, each is input into each BERT feature extractor respectively to obtain the output vector. The Dropout function is used here to optimize the model. The vectors output by the Dense layer are spliced ​​together and input into a multi-classification layer. The number of units in the multi-classification layer is the number of candidate entities. Finally, the Softmax function is used to select the candidate entity with the highest score as the final linked entity. During the model training process, the classification cross entropy loss function is used for model training. The overall process of the above entity linking model is as follows. Figure 8 shown.

[0108] Define knowledge storage rules. Labels are used to indicate the risk level of coal mine fire and gas disasters, the attribute names of disaster-causing factors, the categories of source information, etc.; nodes indicate the attribute values ​​between disaster description entities and disaster monitoring threshold description entities, and relationships indicate the relationships between the above entities and between entity attribute values. Entities are derived from labels extracted from multimodal knowledge; different entities are associated through the above entity links to realize the construction of a knowledge base for coal mine fire and gas disasters; according to the above knowledge storage rules, the storage method of a graph database is adopted to realize the storage of a multimodal knowledge base composed of entity nodes and edges; by establishing a Neo4j driver object instance, operations such as querying node information and association relationships in the multimodal knowledge base are realized.

[0109] In some embodiments, the system architecture of the multimodal knowledge base includes: a data storage layer, which is used to store multimodal data using a hybrid storage solution, including: a graph database for storing nodes and edges of the knowledge graph, supporting efficient graph query and analysis; a relational database for storing structured data; a file storage system for storing unstructured data; a knowledge representation layer, which is used to construct a structured knowledge graph using a resource description framework and a Web ontology language, and to standardize the entities, attributes and relationships in the field of coal mine fires and gas disasters; a knowledge access layer: providing a unified access interface for users and applications, supporting at least two query languages; an application service layer, which includes at least three functional modules: risk assessment, intelligent early warning, and data analysis, and is used to perform real-time dynamic risk assessment, accident cause analysis, and generate corresponding early warning prompts and emergency measures based on the multimodal data provided by the multimodal knowledge base; a user interaction layer, which provides users with a friendly visualization interface and human-computer interaction functions, supports the visualization of fire and gas knowledge graphs and the graphical output of data analysis results, and is easy to intuitively understand and operate.

[0110] Here, the system architecture of the multimodal knowledge base is as follows Figure 7 As shown in the figure, the data storage layer adopts a hybrid storage solution to meet the storage requirements of multimodal data, including graph databases (such as Neo4j) to store nodes and edges of knowledge graphs, relational databases (such as MySQL, PostgreSQL) to store structured data such as table data and metadata, and file storage systems (such as HDFS, FastDFS) to store large-capacity images, videos and other unstructured data. The knowledge access layer supports query languages ​​such as Cypher and SPARQL to meet different query requirements. By querying entities, relationships and path analysis, users can quickly retrieve disaster events and key risk factors to achieve efficient use of knowledge and reasoning expansion.

[0111] The above-mentioned multi-level architecture design can effectively integrate multimodal data, ensure the efficiency of data visualization and storage, and provide comprehensive support for the early warning and analysis of coal mine fire and gas disasters.

[0112] The implementation method of the multimodal knowledge base is as follows: Use the Protégé ontology editing tool to build an ontology model in the field of coal mine fires and gas disasters. The ontology content defines entity classes (Class), data attributes (Datatype Property) and object properties (Object Property), and clarifies the concept hierarchy and relationship. Import the entities, attributes and relationships after integrating multi-source data into the knowledge graph according to the specifications of the domain ontology. Use RDF and OWL to map entities and relationships into triples (Subject-Predicate-Object). Visualize part of the knowledge graph containing the relationship hierarchy. According to the relationship hierarchy, it is helpful to deduce potential triples in the knowledge graph, such as Fig. 9 As shown, the circle represents the entity and the rounded sentence represents the relationship. Indicates the existing relationship between the source entity and the relationship, bold Represents the hierarchical structure between relationships, and the dotted lines represent the potential relationships between entities. ,relation The embedding of and hierarchical information embedding Combination of two parts. When the relationship hierarchy is modeled as multiple layers, the influence between relationships will weaken as the distance increases. On this basis, the knowledge of domain experts is used to define logical reasoning rules, and reasoning engines such as Apache Jena are used to execute the rules and generate new knowledge, so that the knowledge base has the ability to reason and expand.

[0113] In some embodiments, the GNN module in the risk assessment hybrid model is used to use the structured knowledge graph to capture the complex topological structure and association relationships between risk factors, and generate a high-dimensional representation of nodes; the DBN module is used to model the dynamic changes of risk factors, capture random processes in time series, and reflect the evolution of risk status; the LSTM module is used to process time series data, capture long-term and short-term dependencies, and predict future trends of risk factors; the risk assessment hybrid model combines the node representation extracted by the GNN module with the time series features output by the LSTM module, and inputs them into the DBN module to achieve a comprehensive assessment of coupled disaster risks.

[0114] Here, based on the structured knowledge graph, GNN is used to learn the representation of nodes. Based on the node representation extracted by GNN, a DBN model is built to capture the temporal dynamic characteristics of risk factors. The node representation output by GNN is used as the input of LSTM to predict the future state of risk factors.

[0115] The embodiment of the present invention realizes the multimodal risk dynamic assessment of coal mine fire and gas disasters by constructing a risk assessment hybrid model that integrates graph neural network, dynamic Bayesian network and long short-term memory network. The model fully utilizes the advantages of multimodal data, captures the complex topological structure and time dynamic characteristics between risk factors, and improves the accuracy and real-time performance of risk assessment.

[0116] In some embodiments, the "dynamic risk assessment using the constructed risk assessment hybrid model" in the above step S150 is implemented through the following process: combining the inference results of the DBN module and the predicted output of the LSTM module to determine the posterior probability of the target event; based on the posterior probability, comprehensively considering various risk factors, determining a comprehensive risk score; according to the size of the comprehensive risk score and a preset threshold, dividing the risk assessment results into four levels: low risk, medium risk, high risk and extremely high risk; and formulating corresponding early warning and emergency measures for different risk levels.

[0117] Here, the overall process of multimodal risk dynamic assessment is as follows: Fig.10 Assume that the target event is a gas explosion or a gas explosion caused by spontaneous combustion of coal. H , the observed data is E 1:t , then the posterior probability is:

[0118] ;

[0119] In the formula, For in time t Given observation data E 1:t Next, the target event H t The posterior probability of occurrence; H t For time t The target events include: coal spontaneous combustion-gas explosion, tunnel fire-gas explosion, working face fire-gas explosion, gas combustion heat radiation-coal spontaneous combustion-gas explosion, gas combustion heat radiation-gas explosion and other compound disaster events; E 1:t From time 1 to time t The observation data set includes: various sensor data of coal mines, such as gas concentration, temperature, air flow rate, etc., which are used to analyze and evaluate the current risk status; P( E 1:t ∣ H t ) is the target event H t In the event of a E 1:tLikelihood probability; P( H t ) is the target event H t The prior probability of is estimated based on historical data or expert knowledge; P( E 1:t ) is the observed data E 1:t The marginal probability of E 1:t The total probability of .

[0120] Taking all risk factors into consideration, calculate the comprehensive risk score R t :

[0121] ;

[0122] In the formula, R t For time t A comprehensive risk score to quantify the current overall risk level; n is the number of risk factors, i.e., the total number of different risk factors considered; For the i The dynamic weight of each risk factor satisfies: , weight It reflects the relative importance of each risk factor in the overall risk and can be adjusted based on real-time monitoring data and expert opinions; For the i Risk factors in time t The posterior probability of , which indicates the probability that the risk factor is in a dangerous state; For the i Risk factors in time t The state variables may include: excessive gas concentration, abnormal temperature increase, increased carbon monoxide concentration, poor ventilation, equipment failure, etc.

[0123] Based on risk score R t The risk is divided into four levels: low risk (0< R t <0.25), medium risk (0.25 < R t <0.5), high risk (0.5 < R t <0.75), very high risk (0.75 < R t <1) Develop corresponding early warning and emergency measures for different risk levels.

[0124] Example 1: Dynamic assessment of gas explosion risk caused by spontaneous combustion of coal.

[0125] At the No. 3 working face of a coal mine, the following abnormalities were detected: the temperature continued to rise from 25°C to 80°C, the carbon monoxide concentration rose from 10 ppm to 60 ppm, and the gas concentration rose from 1.5% to 4.8%.

[0126] Use GNN to extract key risk factors: coal seam temperature (X1), carbon monoxide concentration (X2), gas concentration (X3), ventilation conditions (X4). Dynamically set weights based on real-time conditions w 1=0.3, w 2=0.25, w 3=0.35, w 4=0.1.

[0127] Calculation of posterior probability of risk factors: Probability of coal seam temperature exceeding the standard P(X1) = 1 (severely exceeding the standard). Probability of carbon monoxide concentration exceeding the standard: P(X2) = 1 (exceeding the standard). Gas concentration close to the lower explosion limit P(X3) = 0.96. Probability of poor ventilation conditions P(X4) = 0.7.

[0128] Comprehensive risk score calculation: R t = w 1P(X1)+ w 2P(X2)+ w 3P(X3)+ w 4P(X4)=0.956. R t =0.956, which is "extremely high risk". The model infers that coal spontaneous combustion may cause gas explosion, and emergency measures must be taken immediately. For example, an "emergency warning" is issued: There is an extremely high risk of coal spontaneous combustion causing gas explosion at the No. 3 working face. Please evacuate personnel immediately and take cooling and gas emission measures! ".

[0129] Emergency measures recommended: Personnel evacuation: quickly organize the evacuation of personnel from the working face and nearby areas. Strengthen ventilation: increase ventilation volume to reduce gas concentration and temperature. Nitrogen injection cooling: inject nitrogen into high-temperature areas to suppress coal spontaneous combustion reactions. Real-time monitoring: increase monitoring frequency and pay close attention to changes in key parameters.

[0130] Based on the above content, the new monitoring data and risk assessment results are updated to the knowledge base to strengthen the correlation between "spontaneous combustion of coal" and "increased gas concentration". According to the actual situation, the weights and conditional probabilities of risk factors in the model are optimized. New inference rules are added, such as "When the coal seam temperature exceeds 75°C and the gas concentration exceeds 4.5%, the risk of explosion is extremely high."

[0131] Example 2: Coupling risk assessment of tunnel fire and gas explosion

[0132] In the No. 5 tunnel of a mine, a fire caused by a cable short circuit occurred, and the gas concentration rose to 4.2%. GNN was used to identify risk factors: the presence of a fire source (Y1), gas concentration (Y2), and ventilation conditions (Y3). Dynamic weight setting w 1=0.4, w 2=0.4, w 3=0.2.

[0133] Calculation of posterior probability of risk factors: Probability of fire source existence P(Y1)=1 (fire has occurred). Probability of gas concentration exceeding the standard P(Y2)=0.84 (close to the lower explosion limit). Probability of poor ventilation: P(Y3)=0.6 (smoke affects ventilation), calculation of comprehensive risk score R t =0.856.

[0134] The risk level is determined to be "extremely high risk". There is a high risk of a gas explosion caused by a tunnel fire. Early warning measures were issued: "Emergency warning: A fire has occurred in the No. 5 tunnel, the gas concentration is high, and there is a risk of explosion. Please activate the emergency plan immediately!"

[0135] The emergency measures are as follows: Fire extinguishing measures: Use explosion-proof fire extinguishing equipment to extinguish the fire. Gas emission: Increase ventilation to quickly reduce gas concentration. Evacuation: Ensure the safe evacuation of personnel at the site and nearby areas. Monitoring upgrade: Real-time monitoring of gas and fire conditions to prevent secondary disasters. Follow-up records of this accident will be made to enrich the case library and improve the knowledge of fire and gas coupling risks.

[0136] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0137] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0138] In the several embodiments provided by the present invention, it should be understood that the disclosed methods can be implemented in other ways. The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0139] The above is only an embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for constructing and optimizing a multimodal knowledge base of coal mine fire and gas disasters, characterized in that: include: Construct a unified ontology structure based on typical disasters of coal mine fire and gas; Based on the unified ontology structure, entities, attributes and relationships between entities are extracted from a multimodal data source; wherein the multimodal data source includes at least text, images and tables; Use graph neural network models to integrate extracted entities, attributes, and relationships to form a structured knowledge graph; Based on the structured knowledge graph, identify and link multimodal data entities to build a multimodal knowledge base; Obtain historical data and real-time monitoring data of the area to be assessed, and perform dynamic risk assessment using the constructed risk assessment hybrid model based on the entities and relationships in the multimodal knowledge base; wherein the risk assessment hybrid model is used to combine the node representation extracted by the graph neural network GNN module with the temporal characteristics of the long short-term memory network LSTM module, and input them into the dynamic Bayesian network DBN module to achieve a comprehensive assessment of coupled disaster risks; According to the real-time monitoring data and risk assessment results, updating the attribute values ​​and relationships of entities in the multimodal knowledge base; The method of using a graph neural network model to integrate the extracted entities, attributes and relationships to form a structured knowledge graph includes: using the extracted entities as nodes, the relationships between the extracted entities as edges, assigning corresponding feature attributes to each of the nodes using the extracted attributes, and organizing all nodes and all edges according to a unified ontology structure to form an initial knowledge graph; extracting feature vectors corresponding to different modalities in each node of the initial knowledge graph to obtain a unified node feature representation; constructing a graph neural network model to perform message transmission and node update on the unified node feature representation of each node to perform deep learning and optimization on the node features; wherein a multimodal attention mechanism is introduced in the message transmission process to perform weighted fusion on node features of different modalities; Node update is achieved through the following formula: ； In the formula, For Node i The updated feature representation is, σ is the activation function, For Node i The weight of the self-loop edge, For Node i The weight coefficient of is a learnable weight matrix used to adjust the node i Its own characteristics h i Perform a linear transformation, h i For Node i The initial feature representation vector of contains the information after multimodal fusion. For Node i The set of neighbor nodes of For edge ( i,j ), Neighbor node j The weight coefficient of is a learnable weight matrix used to adjust the neighbor node features h j Perform a linear transformation, h j Neighbor node j The feature representation vector of For Node i The normalization coefficient of .

2. The method according to claim 1, characterized in that The construction is based on a unified ontology structure of typical coal mine fire and gas disasters, including: Collect documents, accident investigation reports, and technical specifications on coal mine fire and gas disasters, and extract key concepts from them; the key concepts include: disaster type, spatiotemporal attributes, disaster influencing factors, physical and chemical parameters, equipment and facilities, and emergency measures. The disaster types include mine fire, mine gas disaster, and fire and gas coupling disaster; Based on the extracted key concepts, determine the main entities in the field and the relationships between entities to form a preliminary ontology concept model; Select an ontology language to define classes and attributes, establish relationships and clarify object attributes, and build a unified ontology structure; A naming standard for terms in the unified ontology structure is formulated, and the definition of key entities in the unified ontology structure is determined in combination with the characteristics of the field of coal mine fire and gas disasters.

3. The method according to claim 1, characterized in that The step of extracting entities, attributes and their relationships from a multimodal data source based on the unified ontology structure includes: Collect multimodal datasets of typical fire and gas disasters and perform data preprocessing; For the preprocessed data set, a joint extraction model is constructed to extract entities; wherein, the joint extraction model is constructed by superimposing the Lattice-LSTM model on the LEBERT model to perform the named entity recognition task, and using the CRF conditional random field model on each hidden layer of the LSTM model; the LEBERT model is used to perform the Chinese sequence labeling task; By adopting a rule-based template method, the relationships between classes in the unified ontology structure are taken as relationship words in relationship triples, and the relationships between entities are extracted from the preprocessed data set.

4. The method according to claim 1, characterized in that: The extracting feature vectors corresponding to different modes in each node of the initial knowledge graph to obtain a unified node feature representation includes: Use the pre-trained BERT model to map the text information in the node into a text feature vector; Use the pre-trained ResNet-50 model to extract the image feature vector of the image information in the node; Normalize the numerical attributes of the table information in the node to form a numerical feature vector; Perform one-hot encoding on the categorical attributes to obtain the categorical feature vector; The text feature vector, the image feature vector, the numerical feature vector and the category feature vector are concatenated to form the unified node feature representation.

5. The method according to any one of claims 1 to 4, characterized in that: Based on the structured knowledge graph, identifying and linking multimodal data entities and building a multimodal knowledge base include: Determine a text entity set, an image entity set, and an audio entity set in the structured knowledge graph; Taking the text entity set as a candidate entity set, and linking the image entity set and the audio entity set to the candidate entity set using a STELM model; Define entity storage rules and adopt the storage method of graph database to realize multimodal knowledge base storage consisting of entity nodes and edges.

6. The method according to claim 5, characterized in that The system architecture of the multimodal knowledge base includes: The data storage layer is used to store multimodal data using a hybrid storage solution, including: graph databases for storing nodes and edges of knowledge graphs, supporting efficient graph query and analysis; relational databases for storing structured data; and file storage systems for storing unstructured data. The knowledge representation layer is used to construct a structured knowledge graph using the resource description framework and Web ontology language to standardize the entities, attributes and relationships in the field of coal mine fire and gas disasters; Knowledge access layer: provides a unified access interface for users and applications, and supports at least two query languages; The application service layer includes at least three functional modules: risk assessment, intelligent early warning, and data analysis, which are used to perform real-time dynamic risk assessment and accident cause analysis based on the multimodal data provided by the multimodal knowledge base, and generate corresponding early warning prompts and emergency measures suggestions; The user interaction layer provides users with a friendly visualization interface and human-computer interaction functions, and supports the visualization of fire and gas knowledge graphs and the graphical output of data analysis results.

7. The method according to any one of claims 1 to 4, characterized in that: The GNN module in the risk assessment hybrid model is used to capture the complex topological structure and association relationship between risk factors using the structured knowledge graph to generate a high-dimensional representation of the nodes; The DBN module is used to model the dynamic changes of risk factors, capture the random process in the time series, and reflect the evolution of risk status; The LSTM module is used to process time series data, capture long-term and short-term dependencies, and predict future trends of risk factors.

8. The method according to any one of claims 1 to 4, characterized in that: The method of using the constructed risk assessment hybrid model to perform dynamic risk assessment includes: Combining the inference result of the DBN module and the prediction output of the LSTM module, determining the posterior probability of the target event; Based on the posterior probability, various risk factors are comprehensively considered to determine a comprehensive risk score; According to the size of the comprehensive risk score and the preset threshold, the risk assessment results are divided into four levels: low risk, medium risk, high risk and extremely high risk; Formulate corresponding early warning and emergency measures for different risk levels.

Images