A coal mine safety risk multi-modal knowledge graph construction and intelligent reasoning method

Abstract

The application relates to the technical field of coal mine safety risk construction reasoning, and discloses a coal mine safety risk multi-modal knowledge graph construction and intelligent reasoning method, which comprises the following steps: constructing an ontology and coding a concept vector by using a pre-training model to establish a benchmark semantic space; extracting a time-space feature and projecting the time-space feature to the benchmark space to calculate a semantic similarity with the concept vector; synthesizing conflict evidence by using evidence theory to instantiate a node, generating a real-time dynamic graph, eliminating a redundant node to construct a Bayesian network, mapping a weight to execute propagation and output a posterior probability, quantifying an entropy value to measure uncertainty, and feeding back a control device to trigger reasoning iteration when the value exceeds a threshold. The application projects heterogeneous sensor time sequence data and monitoring video data into a unified dimension feature vector by constructing a benchmark semantic space aligned with a static ontology, introduces evidence theory to process semantic conflicts among multi-source evidence, and effectively solves the problem that different modal physical signals cannot be directly fused at a feature level.

Description

Technical Field

[0001] This invention relates to the field of coal mine safety risk construction and reasoning technology, specifically a method for constructing and intelligently reasoning about a multimodal knowledge graph of coal mine safety risks. Background Technology

[0002] The underground coal mine production environment is highly complex and dynamic, and safety monitoring has gradually evolved from single gas sensor monitoring to a multi-dimensional sensing model. In order to achieve early warning of potential disasters, the industry has begun to try to introduce knowledge graph technology, aiming to connect scattered monitoring data and uncover hidden risk patterns.

[0003] Existing coal mine safety monitoring applications typically employ discrete or shallowly fused data processing methods. Various sensors deployed underground continuously upload time-series values, while monitoring cameras independently transmit video streams. The system generally uses pre-built expert rule bases or static ontology models to match the data, triggering alarms, for example, when values ​​exceed thresholds or visual algorithms detect specific objects. In terms of data processing flow, it mostly follows a unidirectional linear path; front-end devices are usually set to fixed operating parameters, while the back-end computing platform passively receives these data streams and performs logical reasoning or status determination based on the received information.

[0004] However, existing coal mine safety technologies suffer from completely heterogeneous data structures between sensor readings and video images, lacking a unified benchmark semantic space to align these two types of data. When different modalities represent the same event in conflict—for example, sensor readings are normal but video analysis shows anomalies—the system often cannot resolve semantic contradictions mathematically, leading to frequent false alarms. Risk monitoring is a continuous process; without dynamic maintenance of the knowledge graph, a large number of redundant nodes that no longer have timeliness will accumulate over time. This disordered expansion of the topology causes an exponential increase in the computational load of subsequent inference algorithms, making it difficult for the system to meet the millisecond-level real-time requirements of disaster early warning. The existing one-way perception mode leaves the cognitive and perception layers disconnected. When the backend inference results are highly uncertain, the system struggles to drive the frontend equipment to adjust parameters to obtain more accurate data for secondary confirmation. Therefore, this invention provides a method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent inference to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent reasoning. This method solves the problems in existing coal mine safety monitoring systems, such as the difficulty in integrating heterogeneous multimodal data semantics, the high false alarm rate caused by conflicts between sensor and video data, and the difficulty of static reasoning models adapting to dynamic risk evolution and lacking proactive perception capabilities.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of this invention provides a method for constructing a multimodal knowledge graph of coal mine safety risks and for intelligent reasoning, comprising the following steps: S1. Construct a static ontology model containing a set of concepts, relationships, and attributes based on coal mine safety regulations, and use a pre-trained language model to encode the concept set to construct a high-dimensional concept vector in order to determine the baseline semantic space. S2. Receive the downhole sensor time-series data stream and monitoring video stream and perform spatiotemporal feature extraction. Project the extracted feature vectors onto the reference semantic space. By calculating the semantic similarity between the feature vectors and the high-dimensional concept vectors, obtain the semantic similarity value of the multimodal data belonging to the corresponding concept. S3. Transform the semantic similarity value into the basic probability allocation function of evidence theory, use evidence theory to resolve conflicts and synthesize multi-source evidence pointing to the same concept, instantiate dynamic event nodes, and attach the dynamic event nodes to the static ontology model to generate a real-time dynamic knowledge graph. S4. Based on the information gain index, redundant nodes in the real-time dynamic knowledge graph are removed to maintain the graph topology and construct an isomorphic Bayesian network. The confidence weights of the dynamic event nodes are mapped to the observation evidence execution probability propagation, and the posterior probability of the target risk node is output. S5. Quantify the information entropy of the posterior probability to measure the inference uncertainty. When the information entropy exceeds a threshold, adjust the sampling frequency or generate control commands to collect high-precision data to trigger a new round of inference iteration.

[0007] Preferably, in step S1, the step of encoding the concept set using a pre-trained language model to construct a high-dimensional concept vector further includes: A general pre-trained language model is selected as the backbone network, and the backbone network is trained in an intra-domain adaptive manner using a professional text corpus in the coal mining field. The intra-domain adaptive training adopts the masked language model task, and updates the model parameters by minimizing the negative log-likelihood loss of the masked labels. The trained model is used to vectorize each concept in the static ontology model, extract the semantic representation output by the encoder, and perform average pooling to generate a set of concept embedding vectors, which constitute the baseline semantic space.

[0008] Preferably, in step S2, the step of receiving the downhole sensor time-series data stream and monitoring video stream, performing spatiotemporal feature extraction, and projecting the extracted feature vectors onto the reference semantic space further includes: A sliding window slice is performed on the sensor time-series data stream, the sliced ​​sequence is input into a long short-term memory network, the final hidden state of the long short-term memory network is extracted, and the time-series feature vector is output through a fully connected mapping layer; Convolutional operations are performed on the frame images of the monitoring video stream to extract feature maps, global average pooling is performed on the feature maps, and visual feature vectors are output through a visual fully connected mapping layer. The cosine similarity between the temporal feature vector and the high-dimensional concept vector, and the cosine similarity between the visual feature vector and the high-dimensional concept vector are calculated respectively, and used as semantic similarity values.

[0009] Preferably, in step S3, the step of converting the semantic similarity value into a basic probability allocation function of evidence theory, and using evidence theory to resolve conflicts and synthesize multi-source evidence pointing to the same concept, further includes: A recognition framework is established that includes the occurrence and non-occurrence of risk events. The semantic similarity value is converted into the degree of support for the propositions in the recognition framework using a mapping function. Basic probability allocation functions for sensor evidence and visual evidence are constructed respectively. The orthogonality sum of the sensor evidence and the visual evidence is calculated using the Dempster synthesis rule. The conflict coefficient between the two evidence sources is calculated, and the probability quality after fusion is normalized using the conflict coefficient to obtain the confidence weight after fusion.

[0010] Preferably, in step S3, the step of instantiating the dynamic event node further includes: Set an instantiation threshold; when the confidence weight after fusion is greater than the instantiation threshold, create a dynamic event node. The attribute data of the dynamic event node includes: the node type corresponding to the static ontology model, the confidence weight, the timestamp recording the specific moment the event was generated, and the original data index associated with the triggering node.

[0011] Preferably, in step S4, the step of removing redundant nodes from the real-time dynamic knowledge graph based on the information gain index further includes: Calculate the prior entropy of the target risk node and the conditional entropy under the known occurrence of the dynamic event node, and define the difference between the prior entropy and the conditional entropy as the information gain of the dynamic event node on the target risk node. Determine whether the information gain is less than the pruning threshold or whether the difference between the current system time and the timestamp of the dynamic event node exceeds the lifecycle threshold; If any of the above conditions are met, the corresponding node is determined to be a redundant node and a physical deletion operation is performed to remove it from the real-time dynamic knowledge graph.

[0012] Preferably, in step S4, the step of maintaining the graph topology and constructing an isomorphic Bayesian network, and mapping the confidence weights of the dynamic event nodes to perform probability propagation based on observed evidence, further includes: The maintained real-time dynamic knowledge graph is mapped to a Bayesian network structure, entity nodes in the dynamic knowledge graph are mapped to random variable nodes, and causal relationship edges are mapped to directed edges. The confidence weights of the dynamic event nodes are converted into the observation probability inputs of the corresponding evidence variables in the Bayesian network; The Bayesian network is transformed into a joint tree structure using the joint tree algorithm. A message passing mechanism is then executed on the joint tree to update the probability distribution of the entire network and to calculate the posterior probability of the target risk node under the current set of observed evidence.

[0013] Preferably, in step S5, the information entropy of quantifying the posterior probability specifically includes: Obtain the state space of the target risk node and the posterior probability corresponding to each state; Calculate the negative of the sum of the products of the posterior probabilities of each state and their logarithms, and use this as an information entropy indicator to measure the uncertainty of the current risk assessment; The larger the information entropy value, the more uniform the posterior probability distribution, and the higher the uncertainty of risk assessment.

[0014] Preferably, in step S5, the step of adjusting the sampling frequency or generating control commands further includes: Query the relationship links in the static ontology model and trace back the set of underlying sensing device nodes that have a causal relationship with the target risk node; When the underlying sensing device is a scalar sensor, the improved sampling frequency is calculated using a linear mapping function based on the degree to which the information entropy exceeds the threshold and then sent to the sensor. When the underlying sensing device is a visual sensor, it generates gimbal control protocol commands to drive the camera to perform zoom operations or to generate commands to activate multi-frame super-resolution reconstruction algorithms.

[0015] The first aspect of this invention provides a multimodal knowledge graph construction and intelligent reasoning system for coal mine safety risks, comprising: The ontology library management module is used to construct a static ontology model containing a set of concepts, relations and attributes based on coal mine safety regulations, and to encode the concept set using a pre-trained language model to construct high-dimensional concept vectors in order to determine the baseline semantic space. The feature mapping module is used to receive the time-series data stream from downhole sensors and the monitoring video stream and perform spatiotemporal feature extraction. It projects the extracted feature vectors onto the reference semantic space and obtains the semantic similarity value of multimodal data belonging to the corresponding concept by calculating the semantic similarity between the feature vectors and the high-dimensional concept vectors. The graph generation module is used to convert the semantic similarity value into the basic probability allocation function of evidence theory, use evidence theory to resolve conflicts and synthesize multi-source evidence pointing to the same concept, instantiate dynamic event nodes, and attach the dynamic event nodes to the static ontology model to generate a real-time dynamic knowledge graph. The risk reasoning module is used to remove redundant nodes in the real-time dynamic knowledge graph based on the information gain index, so as to maintain the graph topology and construct an isomorphic Bayesian network, and map the confidence weight of the dynamic event node to the observation evidence to perform probability propagation, and output the posterior probability of the target risk node. The feedback control module is used to quantify the information entropy of the posterior probability to measure the inference uncertainty. When the information entropy exceeds a threshold, it adjusts the sampling frequency or generates control commands to collect high-precision data and trigger a new round of inference iteration.

[0016] This invention provides a method for constructing a multimodal knowledge graph and intelligent reasoning for coal mine safety risks. It has the following beneficial effects: 1. This invention constructs a benchmark semantic space aligned with a static ontology, projecting heterogeneous sensor time-series data and monitoring video data into feature vectors of a unified dimension, and introduces evidence theory to handle semantic conflicts between multi-source evidence. This effectively solves the problem of direct fusion of different modal physical signals at the feature level, and utilizes an evidence synthesis mechanism to eliminate false alarms from single sensors or interference from environmental noise, improving the accuracy and robustness of risk event identification in complex coal mine environments.

[0017] 2. This invention employs a dynamic knowledge graph pruning strategy based on information entropy gain. It filters the effective information of target risk nodes according to dynamic event nodes, promptly removing redundant or expired nodes. This avoids the exponential expansion of the dynamic knowledge graph over monitoring time, significantly reducing the topological complexity and computational load of subsequent Bayesian networks while preserving key inference evidence. This ensures the real-time performance and response speed of the inference process when handling massive amounts of real-time data.

[0018] 3. This invention establishes a reverse active sensing adjustment mechanism driven by inference uncertainty. It utilizes information entropy to quantify the credibility of Bayesian inference results and, when uncertainty is high, reversely controls the front-end equipment to adjust the sampling frequency or observation focal length. This closed-loop control logic changes the traditional system's passive reception of fixed parameter data. It can automatically acquire high spatiotemporal resolution data for secondary confirmation when risk assessment is ambiguous, achieving dynamic coordination between computing resources and physical sensing resources, and improving the system's decision-making reliability under conditions of sparse or low-quality data. Attached Figure Description

[0019] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart of the method steps of the present invention; Figure 3 This is a flowchart of the multi-source evidence fusion and node instantiation process of the present invention; Figure 4 This is a flowchart of the uncertainty-driven reverse perception control of the present invention.

[0020] Among them, 100 is the ontology library management module; 200 is the feature mapping module; 300 is the graph generation module; 400 is the risk reasoning module; and 500 is the feedback control module. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 , Figure 1 This is a system architecture diagram according to an embodiment of the present invention. The present invention provides a coal mine safety risk multimodal knowledge graph construction and intelligent reasoning system, including an ontology management module 100, a feature mapping module 200, a graph generation module 300, a risk reasoning module 400, and a feedback control module 500.

[0023] The ontology library management module 100 is used to construct and store static ontology models in the field of coal mine safety. The ontology library management module 100 defines a set of concepts, a set of relations, and a set of attributes, and transforms the discrete symbols in the set of concepts into concept embedding vectors in a high-dimensional vector space.

[0024] The feature mapping module 200 is connected to the downhole physical sensing device and is used to receive real-time sensor time-series data streams and monitoring video streams. The feature mapping module 200 is equipped with a feature extraction network, which is responsible for converting continuous time-series data and unstructured video frame data into feature vectors respectively, and calculating the semantic similarity between the feature vectors and the concept embedding vectors in the ontology library management module 100, thus completing the mapping from physical signals to semantic space.

[0025] The knowledge graph generation module 300 is used to detect the semantic consistency of different modal data when mapped to the same concept. When a semantic conflict is detected, the knowledge graph generation module 300 performs fusion calculation on multi-source evidence based on evidence theory, generates dynamic event nodes according to the credibility weights of the fused evidence, and attaches the nodes to the static ontology model to form a real-time dynamic knowledge graph.

[0026] The risk reasoning module 400 includes a graph topology maintenance unit and a Bayesian inference unit. The graph topology maintenance unit calculates the dynamic node value based on information entropy gain and performs graph pruning operations to maintain computational efficiency in the topology structure. The Bayesian inference unit maps the pruned dynamic knowledge graph into a Bayesian network, uses the joint tree algorithm for probability propagation, and calculates the posterior probability of the target risk node.

[0027] The feedback control module 500 is used to measure the uncertainty of the risk assessment results. When the uncertainty index exceeds the preset threshold, it generates control commands for the sensor sampling frequency or camera focal length and sends the commands to the physical sensing device to form a closed-loop control circuit.

[0028] See attached document Figure 2 , Figure 2 This is a flowchart of method steps according to an embodiment of the present invention. The present invention provides a method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent reasoning, comprising the following steps: S1. Construct a semantic baseline space. Based on coal mine safety regulations and historical accident data, define a static ontology model that includes a concept set, a relation set, and an attribute set. Encode the entity concepts in the concept set using a pre-trained language model to construct high-dimensional concept vectors and establish a baseline semantic space for measuring the semantic distance of heterogeneous data. S2, Semantic Projection of Physical Signals. LSTM feature extraction based on a sliding window is performed on the time-series data stream acquired by downhole sensors to generate time-series feature vectors. Simultaneously, visual feature extraction based on a CNN is performed on the monitoring video stream to generate visual feature vectors. By calculating the cosine similarity between each feature vector and the concept vectors in the reference semantic space, the continuous physical signal is projected into discrete candidate semantic events. S3, Multi-source evidence fusion and dynamic graph generation. The cosine similarity obtained in S2 is transformed into the basic probability assignment function in evidence theory. Conflict detection and Dempster-Shafer evidence synthesis are performed on heterogeneous modal data pointing to the same concept. Dynamic event nodes with lifecycle attributes are generated based on the confidence weights of the synthesized data, and these nodes are attached to the corresponding topological positions of the static ontology model to generate a real-time dynamic knowledge graph. S4, Topology Optimization and Probabilistic Inference. Calculate the conditional entropy gain of the dynamic event nodes generated in S3 on the target risk nodes, remove redundant nodes with gain values ​​below a preset threshold to optimize the graph topology, then establish a Bayesian network isomorphic to the optimized graph topology, map the confidence weights of the retained dynamic event nodes to the observation probabilities of the evidence variables in the Bayesian network, and execute the joint tree algorithm to calculate the posterior probability of the target risk event occurring; S5, uncertainty-driven reverse perception control. The information entropy of the posterior probability distribution output by S4 is quantified. When the inference uncertainty represented by the information entropy exceeds the warning threshold, the front-end acquisition device is locked based on the entity mapping relationship in the ontology model. Hardware control commands are generated to increase the sampling frequency or adjust the observation focal length, driving the front-end device to acquire high-precision data and re-enter it into step S2. Uncertainty in the inference process is eliminated through iteration.

[0029] In step S1, the ontology library management module 100 performs the initialization operation of the coal mine safety static ontology library and the benchmark semantic space. By establishing the bijective relationship between physical world entities and digital space vectors, a unified semantic benchmark is provided for the subsequent fusion of heterogeneous data.

[0030] Based on coal mine safety regulations, equipment operation manuals, and historical disaster case data, a static ontology model for the field of coal mine safety is constructed. This static ontology model is formally defined as a triplet structure. .in, This represents a set of concepts, encompassing entities and events related to underground coal mines. Specifically, this includes, but is not limited to, the following subsets: environmental parameter concepts (such as gas concentration, carbon monoxide concentration, wind speed, and temperature), equipment and facility concepts (such as coal mining machines, hydraulic supports, local ventilation fans, and belt conveyors), personnel behavior concepts (such as personnel positioning, unauthorized entry, and failure to wear protective equipment), and disaster type concepts (such as gas explosion, roof water inrush, and coal dust combustion). Represents a set of relationships that describe the logical connections between concept nodes, specifically including spatial topological relationships (such as "located in", "adjacent to"), causal logical relationships (such as "caused", "caused"), and attribute subordinate relationships (such as "belongs to", "has"). It represents a set of attributes used to describe the numerical characteristics or state constraints of a concept, including sensor threshold limits, device rated power, and area safety level parameters.

[0031] After constructing the static ontology model, it is necessary to transform the discrete symbolic concepts into computer-computable continuous vector representations. In this process, a pre-trained language model built using deep neural networks is employed as a feature extractor to establish a high-dimensional baseline semantic space. Although general-purpose pre-trained language models (such as BERT, RoBERTa, and other Transformer-based models) are existing technologies in this field, directly using general-purpose models makes it difficult to accurately capture the specific semantics of coal mining terminology (e.g., the semantic differences between "working face" in general contexts and coal mining contexts). Therefore, to ensure the construction of the baseline semantic space, this embodiment provides a detailed description of the model construction and domain adaptation process.

[0032] In its implementation, the system selects a general pre-trained language model as its backbone network, which consists of a multi-layer bidirectional Transformer encoder. The system also collects specialized text corpora from the coal mining sector. This corpus contains the "Coal Mine Safety Regulations," standardized operating procedure documents, and accident investigation reports from the past ten years. Utilizing the corpus... The backbone network is trained adaptively within the domain. The training task adopts the Masked Language Model (MLM) task, which randomly masks part of the tokens in the input text sequence and requires the model to predict the masked tokens based on the context.

[0033] Suppose the input is a text sequence from the coal mining sector. ,in The tokens after word segmentation. The objective function for optimizing the pre-trained language model. Defined as minimizing the negative log-likelihood loss of the masked label, it is calculated as follows: ; in, Represents the set of occluded markers. This represents the input text sequence related to the coal mining industry. Indicates belonging to a set The specific obscured markers in the text, This represents the unmasked context sequence. For pre-trained language model parameters, These are the conditional probabilities output by the pre-trained language model. The parameters of the pre-trained language model are updated through iterative training on a coal mine-specific corpus. This enables pre-trained language models to learn the contextual dependencies and potential semantic associations of coal mining terminology.

[0034] After completing the domain-specific training, the pre-trained language model is used to process each concept in the static ontology model. Perform vectorized encoding. Translate the concepts... Text description sequence The input is fed into a trained encoder, and the output of the encoder's last hidden layer is extracted as the semantic representation of the concept. To obtain a fixed-dimensional global semantic vector, average pooling is performed on the output sequence. Concept embedding vector The calculation process is as follows: ; in, For the first in the text description sequence One tag, This represents the mapping function of the Transformer encoder. For sequence length, Representing concepts The text description sequence, Represents the index variable in the sequence. Represents a text description sequence The first in One tag, This represents the set of parameters of the encoder model. This indicates the average pooling operation.

[0035] By analyzing the set Perform the above encoding operation on all concepts to generate a corresponding set of concept embedding vectors. The vector space formed by all concept embedding vectors constitutes the baseline semantic space described in this invention. In this space, semantically related concepts (such as "gas exceeding limits" and "ventilator malfunction") are geometrically close to each other, while semantically unrelated concepts are far apart. This is the baseline semantic space. Dimensions Maintaining the same hidden layer dimension as the encoder (e.g., 768-dimensional or 1024-dimensional) provides a unified metric coordinate system for mapping real-time sensor data and video data to the same space in subsequent steps.

[0036] In step S2, the feature mapping module 200 performs spatiotemporal feature extraction and semantic mapping of multimodal streaming data. By establishing a mapping mechanism from the low-level physical signal space to the high-level semantic vector space, continuously changing sensor readings and unstructured video images are converted into the reference semantic space in step S1. The same-dimensional vector representation enables a computable comparison between heterogeneous data and ontological concepts.

[0037] The feature mapping module 200 connects to an underground industrial Ethernet or fiber optic ring network to acquire two types of data streams: A type of time-series data stream from sensors such as gas, carbon monoxide, temperature, and wind speed. ; Type II video surveillance streams from surveillance cameras in key areas To address the differences in the physical characteristics of these two types of data, the system employs different deep neural network architectures for feature encoding.

[0038] For sensor time-series data streams Considering that numerical values ​​at a single moment are insufficient to represent dynamic risks such as "gas accumulation trends" or "sudden changes in wind speed," the system employs a sliding window mechanism combined with a long short-term memory network for temporal feature extraction, setting the length of the sliding window to [value missing]. (e.g., including data points from the past 60 seconds) and sliding step size For the current moment Extracting sensor data sequences ,in This is the normalized sensor reading vector.

[0039] will sequence The data is input into an LSTM network, which uses gating mechanisms (forget gate, input gate, output gate) to capture long-range dependencies and evolutionary trends in the sequence. The system extracts the final hidden state of the LSTM network after processing the last data point within the window. The final hidden state contains the temporal evolution features within the current time window. To ensure that the generated feature vector is consistent with the baseline semantic space dimension in step S1, the following steps are taken: The input is fed into a fully connected mapping layer, and the output is the final temporal feature vector. : ; in, For mapping weights, This represents the feature dimension of the hidden layer of an LSTM network. This represents the final hidden state vector output by the LSTM network after processing the current time window sequence. This represents the bias vector of the fully connected mapping layer. Dimensions With concept embedding vector They have the same dimensions.

[0040] For video surveillance stream The system employs a convolutional neural network (CNN) to extract visual morphological features. Using mature architectures such as ResNet or VGG as the backbone network, the original fully connected layers used for image classification are removed, retaining only convolutional and pooling layers as feature extractors. For the video frame image at the current moment... The system preprocesses the data (such as resizing and pixel normalization) before inputting it into the CNN backbone network.

[0041] Convolution operations extract edges, textures, and high-level semantic patterns by sliding multiple convolution kernels across an image. The feature map output by the backbone network is subjected to global average pooling to compress the spatial dimension, obtaining an intermediate vector representation of the visual features. To achieve cross-modal alignment, this intermediate vector is passed through a fully connected visual mapping layer to output the visual feature vector. : ; in, This represents the final visual feature vector output after mapping by the fully connected visual layer. This represents the projection weight matrix of the fully connected visual mapping layer. This represents the global average pooling operation function. This represents the feature extraction operation of a convolutional neural network (backbone network). Indicates the current moment Input frame image captured from a surveillance video stream.

[0042] In obtaining time-series feature vectors With visual feature vectors Subsequently, the system performs a cross-modal semantic mapping operation. This operation aims to determine whether the current physical signal contains a risk meaning defined in the ontology library. The system iterates through the set of concept embedding vectors generated in step S1. The cosine similarity between the real-time feature vector and each concept vector is calculated separately.

[0043] For time series feature vectors With the Concept vectors The similarity is calculated as follows: ; For visual feature vectors With the Concept vectors The similarity is calculated as follows: ; in This represents the L2 norm of the vector. The system sets a matching threshold. (For example, 0.75). When the calculated similarity... At that time, it is determined that the current physical data stream semantically activates the corresponding ontology concept. For example, when the feature vector of sensor data... When the similarity to the concept vector of "gas exceeding the limit" is extremely high, even if the sensor does not directly output a text alarm, the system can semantically identify that the current physical state corresponds to the risk event of "gas exceeding the limit." This mapping method based on geometric distance in vector space allows continuous analog signals to be transformed into discrete spectral event nodes, completing a symbolic leap from the perception layer to the cognition layer. For matching results that do not reach the threshold, the system treats them as background noise or a normal state and does not trigger the subsequent node instantiation process.

[0044] See attached document Figure 3 In step S3, the graph generation module 300 performs cross-modal conflict resolution and node instantiation based on Dempster-Shafer (DS) evidence theory to solve the problem of inconsistency or conflict in the representation of the same environmental state by different physical perception modalities in complex downhole environments. Through a mathematical evidence synthesis mechanism, a fusion conclusion with high robustness is obtained and the conclusion is transformed into entity nodes in the knowledge graph.

[0045] After feature mapping is completed in step S2, the system will obtain the feature mapping results for the same concept to be detected. Multiple independent sources of evidence (e.g., "belt conveyor friction causing fire"), i.e., time-series feature vectors Matching results and visual feature vectors The matching results are then processed. The atlas generation module 300 first executes a consistency detection sub-step. This involves calculating the confidence difference between two modal feature vectors mapped to the same concept, or directly calculating the cosine of the angle between the two feature vectors in the vector space. When the consistency index of the two is lower than a preset conflict threshold (e.g., flame sensor data is normal, but the video analysis model indicates smoke features), the system determines that there is a cross-modal semantic conflict at the current moment and initiates a fusion calculation process based on DS evidence theory.

[0046] This fusion computing process establishes an identification framework, denoted as... For monitoring specific risk events in coal mines, the identification framework is defined as a mutually exclusive set. ,in This indicates that a risk event has occurred (e.g., "a fire has occurred"). This indicates that no risk event has occurred or the status is normal.

[0047] The system constructs the basic probability assignment function (BPA), also known as the quality function. The BPA function maps the similarity between feature vectors and concept vectors to the similarity of propositions. The degree of support. For sensor modes, define their probability assignment function. For visual modalities, define their probability assignment function. The probability assignment function satisfies and .

[0048] In practice, the cosine similarity calculated in step S2 is used. To construct The mapping function is set to convert the normalized similarity values ​​into basic probability mass, and the remaining probability mass is assigned to the entire set. Or negation of proposition Two independent sources of evidence were calculated using Dempster's rules of composition. and The orthogonal sum, i.e., the probability mass function after fusion. .

[0049] For any assumption in the identification framework The fusion calculation formula is as follows: ; in, and These represent the focal elements in sensor evidence and visual evidence, respectively. The basic probability assignment function representing the sensor evidence source is assigned to the focal element. The probability mass value; The basic probability assignment function representing the source of visual evidence is assigned to the focal element. The probability mass value; Indicates the condition for summation; The basic probability assignment function after multi-source evidence fusion represents the hypothesis proposition. Assigning a value; is the conflict coefficient, used to quantify the degree of conflict between two sources of evidence. The formula for calculating it is defined as the sum of the intersections of all empty sets: ; Conflict coefficient The range of values ​​is .when When the value is close to 1, it indicates a high degree of conflict between the evidence from the two modalities (e.g., one is extremely certain that it occurred, and the other is extremely certain that it did not). In this case, the denominator... This process serves a normalization function, redistributing the probability losses caused by conflicts. Through the above calculations, the system obtains the event occurrence confidence after fusion. This value integrates the information from multimodal data and eliminates the interference from false alarms caused by single modality.

[0050] Execute the dynamic node instantiation sub-step and set an instantiation threshold. If the confidence level after fusion The map generation module 300 will create dynamic event nodes in memory. This dynamic event node includes the following attribute data: Node type: corresponds to the concept in the static ontology library. ; Confidence weight: assigned a value This weight will serve as the initial strength of evidence for subsequent Bayesian inference; Timestamp: Records the moment an event was generated. ; Raw data index: Associates the original sensor ID and camera ID that triggered this node.

[0051] Generated dynamic event nodes By connecting relational edges to corresponding concept nodes in the static ontology, real-time updates from "multi-source data flow" to "dynamic knowledge graph" are achieved. Regarding the conflict coefficient... If the data is so highly suspicious that it cannot be effectively integrated, the system will mark the data in that time slice as high-suspicion data, will not generate a specific event node, and will record a log for the subsequent reverse control module to refer to.

[0052] In step S4, the risk reasoning module 400 performs graph dynamic pruning and Bayesian inference based on information entropy gain. This addresses the problem of decreased computational efficiency caused by the exponential growth of the dynamic knowledge graph over time, and utilizes a probabilistic graphical model to handle uncertain correlations in the risk evolution process.

[0053] After node instantiation is completed in step S3, the dynamic knowledge graph contains a large number of real-time generated event nodes. To ensure the real-time performance of subsequent reasoning, a graph topology maintenance sub-step is initiated, employing an information theory-based pruning strategy to optimize the graph structure.

[0054] The system defines the target risk node to be assessed as follows: (For example, a "gas explosion accident"), this risk node belongs to a high-level concept in the static ontology. For each newly added dynamic event node in the graph... (For example, "abnormal gas concentration at time T=t"), the system calculates the information gain of the dynamic event node to the target risk node to quantify the contribution of the information gain data to reducing the system entropy.

[0055] The calculation of information gain depends on the measurement of information entropy. First, the target risk node is calculated. Prior entropy This prior entropy value represents the uncertainty of whether the target risk will occur in the absence of any new evidence, calculated at known dynamic event nodes. Conditional entropy under occurrence conditions Information gain It is defined as the difference between prior entropy and conditional entropy.

[0056] The formulas involved in the calculation are as follows: ; ; in, Represents the state space of the target risk node (whether it has occurred or not). Representing state The prior probability. The system sets the pruning threshold. and lifecycle threshold The risk reasoning module iterates through the dynamic nodes in the graph 400 times. If a node meets any of the following conditions, it performs a physical deletion operation and removes it from the graph topology: Calculated information gain Less than the pruning threshold This indicates that the data provides insufficient effective information for current risk assessment; The difference between the current system time and the generation timestamp of this node exceeds the lifecycle threshold. This indicates that the data is no longer timely.

[0057] After the graph pruning is completed, the system executes the Bayesian risk inference sub-step. The risk inference module 400 maps the maintained dynamic knowledge graph to a Bayesian network structure. The mapping rules are as follows: entity nodes in the graph are mapped to random variable nodes in the Bayesian network, causal relationship edges in the graph are mapped to directed edges, and the conditional probability tables between nodes are pre-set based on historical statistical data or expert knowledge.

[0058] The system will synthesize the node credibility weights obtained from DS evidence theory in step S3. This is transformed into the observation probability input of the corresponding evidence variable in the Bayesian network. Since the Bayesian network structure is relatively complex, this embodiment uses the joint tree algorithm for accurate reasoning.

[0059] This joint tree algorithm transforms a Bayesian network into a moral graph, constructs a chord graph through triangulation, and then generates a joint tree structure composed of cliques. A message-passing mechanism, i.e., the process of collecting and distributing evidence, is executed on the joint tree to update the probability distribution of the entire network. The system then calculates the target risk nodes. Based on the current set of observational evidence Posterior probability The calculation of the posterior probability follows Bayes' theorem: ; in, Let be the likelihood probability. For prior probability, This is a normalization constant. The system outputs the calculated posterior probability value, which intuitively reflects the likelihood of a specific security risk occurring under the current multimodal sensing data-driven approach. The specific implementation details of the joint tree algorithm, such as the summation-product operation in message passing, are standard algorithm implementations in the field of probabilistic graphical models and will not be elaborated upon here.

[0060] See attached document Figure 4 In step S5, the feedback control module 500 performs reverse active sensing adjustment based on inference uncertainty. A closed-loop feedback mechanism is established from the inference layer to the physical sensing layer. By measuring the credibility of the risk assessment results, the acquisition parameters of the front-end device are dynamically adjusted in reverse to solve the problem of decision ambiguity caused by data sparsity or low quality.

[0061] Output the target risk node in step S4 After obtaining the posterior probability distribution, an uncertainty measurement operation is performed. Bayesian inference provides the probability value of the risk occurring, but a single probability value cannot directly reflect the system's degree of confidence in the judgment. For example, in binary classification, a probability of 0.5 means that the system is in a state of ignorance, and the uncertainty is the highest. The feedback control module 500 uses information entropy as a quantitative indicator to evaluate the uncertainty of the current judgment result.

[0062] Set target risk nodes The state space is The posterior probability distribution of the Bayesian inference output is Calculate the information entropy of the posterior probability distribution. The calculation logic is as follows: ; in, Indicates the current set of evidence Next state The posterior probability of occurrence, Represents the state space of the target risk node The total number of states in the game. This represents the loop index variable in the summation operation. Represents the first in the state space A specific state, This represents the set of observed evidence at the current moment. The calculated entropy value. The larger the entropy value, the more uniform the probability distribution and the more ambiguous the system's judgment; the smaller the entropy value, the more concentrated the probability distribution and the more certain the system's judgment.

[0063] Set uncertainty warning threshold When the calculated information entropy If the current data is insufficient to make a reliable risk assessment, the system triggers a reverse adjustment mechanism. The feedback control module 500 queries the relationship links in the static ontology library to trace back to the target risk node. A set of underlying sensing device nodes with causal relationships .

[0064] For sets The system generates differentiated control commands for different types of devices. .

[0065] For scalar sensors (such as gas and wind speed sensors), control commands aim to improve the temporal resolution of the data to capture transient characteristics. Based on the current level of uncertainty, an adjusted sampling frequency is calculated using a linear mapping function. : ; in, This is the reference sampling frequency of the sensor. The maximum sampling frequency supported by the hardware. This function adjusts the gain coefficient. It ensures that the higher the uncertainty, the greater the increase in sampling frequency, thus acquiring a denser data sequence within the subsequent time window.

[0066] For visual sensors (such as surveillance cameras), control commands aim to improve the spatial resolution of the data or focus on a specific area. The system generates PTZ control protocol (such as the ONVIF protocol) commands to drive the camera to perform zoom operations or turn towards the region of interest. If the hardware does not support physical zoom, the system activates a software-level multi-frame super-resolution reconstruction algorithm to enhance the image stream.

[0067] Generated control commands The command is transmitted to the front-end physical devices via the industrial ring network. The high-precision data stream collected by the devices after responding to the command serves as new input, re-entering step S2 and subsequent processes to initiate a new round of inference iteration. This iterative process continues until one of the following termination conditions is met: the calculated information entropy... Drop to threshold The following indicates that the risk assessment has sufficient credibility; or the number of iterations has reached the preset limit. This is to prevent the system from getting stuck in an infinite loop.

[0068] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a multimodal knowledge graph and intelligent reasoning for coal mine safety risks, characterized in that, Includes the following steps: S1. Construct a static ontology model containing a set of concepts, relationships, and attributes based on coal mine safety regulations, and use a pre-trained language model to encode the concept set to construct a high-dimensional concept vector in order to determine the baseline semantic space. S2. Receive the downhole sensor time-series data stream and monitoring video stream and perform spatiotemporal feature extraction. Project the extracted feature vectors onto the reference semantic space. By calculating the semantic similarity between the feature vectors and the high-dimensional concept vectors, obtain the semantic similarity value of the multimodal data belonging to the corresponding concept. S3. Transform the semantic similarity value into the basic probability allocation function of evidence theory, use evidence theory to resolve conflicts and synthesize multi-source evidence pointing to the same concept, instantiate dynamic event nodes, and attach the dynamic event nodes to the static ontology model to generate a real-time dynamic knowledge graph. S4. Based on the information gain index, redundant nodes in the real-time dynamic knowledge graph are removed to maintain the graph topology and construct an isomorphic Bayesian network. The confidence weights of the dynamic event nodes are mapped to the observation evidence execution probability propagation, and the posterior probability of the target risk node is output. S5. Quantify the information entropy of the posterior probability to measure the inference uncertainty. When the information entropy exceeds a threshold, adjust the sampling frequency or generate control commands to collect high-precision data to trigger a new round of inference iteration.

2. The method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent reasoning according to claim 1, characterized in that, In step S1, the step of encoding the concept set using a pre-trained language model to construct a high-dimensional concept vector further includes: A general pre-trained language model is selected as the backbone network, and the backbone network is trained in an intra-domain adaptive manner using a professional text corpus in the coal mining field. The intra-domain adaptive training adopts the masked language model task, and updates the model parameters by minimizing the negative log-likelihood loss of the masked labels. The trained model is used to vectorize each concept in the static ontology model, extract the semantic representation output by the encoder, and perform average pooling to generate a set of concept embedding vectors, which constitute the baseline semantic space.

3. The method for constructing a multimodal knowledge graph and intelligent reasoning for coal mine safety risks according to claim 1, characterized in that, In step S2, the step of receiving the downhole sensor time-series data stream and monitoring video stream, performing spatiotemporal feature extraction, and projecting the extracted feature vectors onto the reference semantic space further includes: A sliding window slice is performed on the sensor time-series data stream, the sliced ​​sequence is input into a long short-term memory network, the final hidden state of the long short-term memory network is extracted, and the time-series feature vector is output through a fully connected mapping layer; Convolutional operations are performed on the frame images of the monitoring video stream to extract feature maps, global average pooling is performed on the feature maps, and visual feature vectors are output through a visual fully connected mapping layer. The cosine similarity between the temporal feature vector and the high-dimensional concept vector, and the cosine similarity between the visual feature vector and the high-dimensional concept vector are calculated respectively, and used as semantic similarity values.

4. The method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent reasoning according to claim 1, characterized in that, In step S3, the step of converting semantic similarity values ​​into the basic probability allocation function of evidence theory, and using evidence theory to resolve conflicts and synthesize multi-source evidence pointing to the same concept, further includes: A recognition framework is established that includes the occurrence and non-occurrence of risk events. The semantic similarity value is converted into the degree of support for the propositions in the recognition framework using a mapping function. Basic probability allocation functions for sensor evidence and visual evidence are constructed respectively. The orthogonality sum of the sensor evidence and the visual evidence is calculated using the Dempster synthesis rule. The conflict coefficient between the two evidence sources is calculated, and the probability quality after fusion is normalized using the conflict coefficient to obtain the confidence weight after fusion.

5. The method for constructing a multimodal knowledge graph and intelligent reasoning for coal mine safety risks according to claim 1, characterized in that, In step S3, the step of instantiating the dynamic event node further includes: Set an instantiation threshold; when the confidence weight after fusion is greater than the instantiation threshold, create a dynamic event node. The attribute data of the dynamic event node includes: the node type corresponding to the static ontology model, the confidence weight, the timestamp recording the specific moment the event was generated, and the original data index associated with the triggering node.

6. The method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent reasoning according to claim 1, characterized in that, In step S4, the step of removing redundant nodes from the real-time dynamic knowledge graph based on the information gain index further includes: Calculate the prior entropy of the target risk node and the conditional entropy under the known occurrence of the dynamic event node, and define the difference between the prior entropy and the conditional entropy as the information gain of the dynamic event node on the target risk node. Determine whether the information gain is less than the pruning threshold or whether the difference between the current system time and the timestamp of the dynamic event node exceeds the lifecycle threshold; If any of the above conditions are met, the corresponding node is determined to be a redundant node and a physical deletion operation is performed to remove it from the real-time dynamic knowledge graph.

7. The method for constructing a multimodal knowledge graph and intelligent reasoning for coal mine safety risks according to claim 1, characterized in that, In step S4, the step of maintaining the graph topology and constructing an isomorphic Bayesian network, and mapping the confidence weights of the dynamic event nodes to perform probability propagation based on observed evidence, further includes: The maintained real-time dynamic knowledge graph is mapped to a Bayesian network structure, entity nodes in the dynamic knowledge graph are mapped to random variable nodes, and causal relationship edges are mapped to directed edges. The confidence weights of the dynamic event nodes are converted into the observation probability inputs of the corresponding evidence variables in the Bayesian network; The Bayesian network is transformed into a joint tree structure using the joint tree algorithm. A message passing mechanism is then executed on the joint tree to update the probability distribution of the entire network and to calculate the posterior probability of the target risk node under the current set of observed evidence.

8. The method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent reasoning according to claim 1, characterized in that, In step S5, the quantification of the information entropy of the posterior probability specifically includes: Obtain the state space of the target risk node and the posterior probability corresponding to each state; Calculate the negative of the sum of the products of the posterior probabilities of each state and their logarithms, and use this as an information entropy indicator to measure the uncertainty of the current risk assessment; The larger the information entropy value, the more uniform the posterior probability distribution, and the higher the uncertainty of risk assessment.

9. The method for constructing a multimodal knowledge graph of coal mine safety risks and intelligent reasoning according to claim 1, characterized in that, In step S5, the step of adjusting the sampling frequency or generating control commands further includes: Query the relationship links in the static ontology model and trace back the set of underlying sensing device nodes that have a causal relationship with the target risk node; When the underlying sensing device is a scalar sensor, the improved sampling frequency is calculated using a linear mapping function based on the degree to which the information entropy exceeds the threshold and then sent to the sensor. When the underlying sensing device is a visual sensor, it generates gimbal control protocol commands to drive the camera to perform zoom operations or to generate commands to activate multi-frame super-resolution reconstruction algorithms.

10. A coal mine safety risk multimodal knowledge graph construction and intelligent reasoning system, applied to the coal mine safety risk multimodal knowledge graph construction and intelligent reasoning method described in any one of claims 1-9, characterized in that, include: The ontology library management module is used to construct a static ontology model containing a set of concepts, relations and attributes based on coal mine safety regulations, and to encode the concept set using a pre-trained language model to construct high-dimensional concept vectors in order to determine the baseline semantic space. The feature mapping module is used to receive the time-series data stream from downhole sensors and the monitoring video stream and perform spatiotemporal feature extraction. It projects the extracted feature vectors onto the reference semantic space and obtains the semantic similarity value of multimodal data belonging to the corresponding concept by calculating the semantic similarity between the feature vectors and the high-dimensional concept vectors. The graph generation module is used to convert the semantic similarity value into the basic probability allocation function of evidence theory, use evidence theory to resolve conflicts and synthesize multi-source evidence pointing to the same concept, instantiate dynamic event nodes, and attach the dynamic event nodes to the static ontology model to generate a real-time dynamic knowledge graph. The risk reasoning module is used to remove redundant nodes in the real-time dynamic knowledge graph based on the information gain index, so as to maintain the graph topology and construct an isomorphic Bayesian network, and map the confidence weight of the dynamic event node to the observation evidence to perform probability propagation, and output the posterior probability of the target risk node. The feedback control module is used to quantify the information entropy of the posterior probability to measure the inference uncertainty. When the information entropy exceeds a threshold, it adjusts the sampling frequency or generates control commands to collect high-precision data and trigger a new round of inference iteration.

Images