A job safety risk identification method based on a multi-modal knowledge graph

Abstract

The application discloses a kind of based on multi-modal knowledge graph's job safety risk identification method, comprising: S1, collection multi-source data extraction feature, obtain modal feature set;S2, alignment history feature set calculates dynamic benchmark and tolerance interval, obtains space-time baseline;S3, compare feature set with baseline, deviation is input CVAE-GAN model and obtains abnormal confidence;S4, confidence is distributed to pre-safety topology node, and dynamic knowledge graph is obtained;S5, improved MPNN network is constructed, with confidence iterative transmission message, introduce baseline and calculate risk aggregation energy with dynamic damping update state;S6, energy is over threshold value then trigger alarm, and backtrack and obtain causal evidence chain;S7, output alarm and evidence chain, and optimize parameter according to review result.The application deeply fuses multi-source data, suppresses fluctuation interference, improves the accuracy and explainability of risk identification.

Description

Technical Field

[0001] This invention relates to the field of intelligent safety monitoring and risk warning based on multimodal perception and knowledge graphs, and particularly to a method for identifying operational safety risks based on multimodal knowledge graphs. Background Technology

[0002] Multimodal knowledge graphs and graph neural networks, due to their powerful relationship modeling and feature propagation capabilities in complex interconnected scenarios, have been widely applied in recent years in fields such as industrial monitoring, disaster early warning, and safe production, becoming an important development direction for intelligent risk identification. However, in practical applications, operational safety risk identification faces many challenges, such as strong heterogeneity of multi-source data, hidden risk evolution paths, and large randomness of state fluctuations. The deployment effectiveness of existing graph-based and graph neural network-based methods is still constrained by many factors.

[0003] Currently, most operational safety risk identification methods rely on single-modal data perception, making it difficult to fully integrate cross-dimensional heterogeneous information such as personnel micro-expressions and movements, equipment voiceprints, and environmental concentrations. This results in a lack of comprehensiveness in characterizing the risk status at the work site. Some systems use static thresholds to assess abnormal states, ignoring the dynamic changes in the normal fluctuation range of the same process at different micro-movement stages. This leads to minor anomalies being masked by normal fluctuations or normal operations being misjudged as abnormal, limiting the adaptive capability of risk identification. Furthermore, existing graph neural networks lack damping adjustment mechanisms during message transmission, making it easy for abnormal signals from local nodes to be amplified and spread infinitely along associated edges, causing risk "panic" and a proliferation of false alarms across all nodes in the graph, severely impacting the model's stability in real-world scenarios.

[0004] Furthermore, the existing risk identification logic lacks a transparent causal backtracking path, only outputting global or local risk probability scores. This makes it difficult to provide safety managers with key nodes that trigger alarms and the causal transmission chain between entities, affecting the credibility of the warning results and the usability of decision-making. Moreover, the system has failed to establish a human-machine closed-loop parameter self-optimization mechanism, and cannot adaptively evolve the transmission weight of related edges and dynamic alarm thresholds based on on-site verification results. This makes it difficult to adapt to the continuous evolution of the operating environment and risk patterns, severely restricting the practical value and long-term robustness of the risk identification system.

[0005] Therefore, how to provide a method for identifying operational safety risks based on multimodal knowledge graphs is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a method for identifying operational safety risks based on a multimodal knowledge graph. This invention fully integrates key steps such as multimodal spatiotemporal baseline construction, CVAE-GAN model anomaly quantification, dynamic instantiation of knowledge graph evolution, and causal evidence chain backtracking. It constructs an intelligent identification and response process with accurate reconstruction of multimodal deviations, dynamic damping message transmission, and interpretable display of risk causality. This invention utilizes an improved MPNN neural network, introducing a multimodal spatiotemporal baseline and fluctuation tolerance range as dynamic damping coefficients. This effectively suppresses false alarms across the entire graph caused by the infinite propagation of local anomaly signals along associated edges, achieving stable propagation and aggregation of risk states. Simultaneously, it combines a CVAE-GAN model to extract high-precision anomaly confidence and supports closed-loop feedback adaptive optimization of graph weights and alarm thresholds. This invention possesses advantages such as comprehensive multimodal feature fusion, strong anti-interference capability of damping adjustment, and traceable early warning logic. It can significantly improve the accuracy of risk identification and the credibility of decision-making in complex operational scenarios, thereby effectively solving problems such as single state expression, uncontrolled anomaly propagation, and opaque early warning processes in existing methods.

[0007] A method for identifying operational safety risks based on a multimodal knowledge graph according to an embodiment of the present invention includes the following steps:

[0008] S1. Synchronously collect video streams, audio streams, and environmental sensor data from the production site, identify and extract facial micro-expression features, arm movement features, spatial distance features, equipment acoustic features, and dust concentration features to obtain a multimodal feature set;

[0009] S2. Obtain the historical multimodal feature set of the same worker in the same process within the historical period, align the time axis according to the micro-action stage of the process, calculate the dynamic benchmark value and fluctuation tolerance range respectively, and obtain the multimodal spatiotemporal baseline.

[0010] S3. Compare the multimodal feature set with the multimodal spatiotemporal baseline, input the deviation value into the CVAE-GAN model, and obtain the anomaly confidence of each modality. The improved CVAE-GAN model includes a multimodal deviation coding module, a conditional feature reconstruction module, a real distribution adversarial discrimination module, and an anomaly confidence fusion calculation module.

[0011] S4. Pre-safe topology structure: The anomaly confidence of each modality is dynamically assigned to the corresponding entity nodes as an attribute, resulting in a dynamically instantiated knowledge graph.

[0012] S5. Construct an improved MPNN neural network. In the dynamically instantiated knowledge graph, based on the causal relationship between entities, the abnormal confidence of each node is used as the initial message feature and iteratively passed along the association edge to the adjacent nodes. Multimodal spatiotemporal baseline and fluctuation tolerance range are introduced as dynamic damping coefficients, and the risk state features of each node are updated. The risk aggregation energy value of the whole graph or local subgraph is calculated.

[0013] S6. Determine whether the risk aggregation energy value of the whole graph or a local subgraph exceeds the preset dynamic alarm threshold. If it does, trigger a risk alarm, backtrack to extract the key nodes and related edges that triggered the alarm, and obtain a multimodal risk causal evidence chain.

[0014] S7. Output the risk alarm and multimodal risk causal evidence chain to the audit terminal to assist manual judgment, and receive the confirmation result of the auditor's on-site review. Adjust the transmission weight of causal relationship and dynamic alarm threshold to obtain the optimized risk identification parameters.

[0015] Optionally, S1 specifically includes:

[0016] S11. Simultaneously trigger the visible light camera, audio pickup and dust concentration sensor deployed on the production site through synchronous control signal to collect video stream, audio stream and environmental sensor data respectively.

[0017] S12. The video stream is processed by extracting frames at a preset frame rate to obtain video image frames. The face detection algorithm is used to locate the worker's facial region. The local contour coordinates of eyebrows, eyes, nose, mouth and cheeks are extracted sequentially from the facial region, and the relative displacement and deformation amplitude between each coordinate point are calculated. The relative displacement and deformation amplitude are classified and statistically analyzed according to the facial muscle movement unit to obtain the worker's facial micro-expression features.

[0018] S13. Use the skeletal key point detection algorithm to locate the worker's shoulder, elbow and wrist joints in the video image frame, calculate the motion vector between adjacent joints and count the change frequency per unit time, and extract arm movement features.

[0019] S14. Use an object detection algorithm to identify the worker's body bounding box and the equipment bounding box in the video image frame, calculate the Euclidean distance between the center point of the worker's body bounding box and the center point of the equipment bounding box, and extract the spatial distance features.

[0020] S15. Perform frame-by-frame windowing processing on the audio stream, apply Fast Fourier Transform to the windowed audio frames to calculate the frequency domain amplitude, extract the frequency values ​​and amplitudes corresponding to the top N largest energy peaks in the frequency domain amplitude to obtain the device's acoustic signature characteristics, perform moving average filtering on the dust concentration sensing signal in the environmental sensing data, calculate the average amplitude of the filtered signal according to the preset time window to obtain the dust concentration characteristics.

[0021] S16. The facial micro-expression features, arm movement features, and spatial distance features of workers at the same timestamp are classified into the personnel dimension, the voiceprint features of equipment are classified into the equipment dimension, and the dust concentration features are classified into the environment dimension, and combined to obtain a multimodal feature set.

[0022] Optionally, S2 specifically includes:

[0023] S21. Obtain the historical multimodal feature set of the same worker in the same process within the historical period. Divide the process into multiple micro-action stages according to the operation sequence of the process. Match and compare the timestamp of each modality feature in the historical multimodal feature set with the start and end times of each micro-action stage. Assign each modality feature to the corresponding micro-action stage and complete the time axis alignment by the micro-action stage of the process.

[0024] S22. In each micro-movement stage after alignment, extract the data of worker's facial micro-expression features, arm movement features, spatial distance features, equipment voiceprint features and dust concentration features under normal operating conditions, calculate the arithmetic mean of all values ​​of each modal feature under normal operating conditions, and use the arithmetic mean as the dynamic benchmark value of each modal feature under normal operating conditions.

[0025] S23. Calculate the absolute difference between all values ​​of each modal feature under normal operating conditions and the corresponding dynamic reference value. Calculate the standard deviation of all absolute differences. Use the standard deviation of the dynamic reference value plus a preset multiple as the upper limit and the standard deviation of the dynamic reference value minus a preset multiple as the lower limit. The upper and lower limits constitute the fluctuation tolerance range of each modal feature.

[0026] S24. Combine the dynamic baseline values ​​and fluctuation tolerance ranges of each modal feature corresponding to each micro-action stage to obtain the multimodal spatiotemporal baseline.

[0027] Optionally, S3 specifically includes:

[0028] S31. Subtract the values ​​of each modal feature in the real-time multimodal feature set from the dynamic reference value in the multimodal spatiotemporal baseline of the corresponding micro-action stage, and use the difference obtained by subtraction as the deviation value generated by comparison.

[0029] S32. Input the bias value generated by the comparison into the multimodal bias encoding module in the CVAE-GAN model. Map the bias value to the latent variable space through the linear transformation of the fully connected layer and output the mean vector and variance vector of the latent variables. Randomly sample from the standard normal distribution to generate a random noise vector with the same dimension as the variance vector. Multiply the random noise vector and the variance vector element by element and add the mean vector to obtain the latent space feature vector.

[0030] S33. The latent space feature vector is concatenated with the one-hot encoded label of the current micro-action stage. The concatenated vector is input into the conditional feature reconstruction module. The concatenated vector is upsampled and deconvolved through the transposed convolutional layer. The reconstruction deviation value is output after the calculation result is flattened.

[0031] S34. Subtract the corresponding position of the input deviation value from the reconstructed deviation value and calculate the sum of squares. Divide the sum by the number of features contained in the deviation value to obtain the reconstruction error. At the same time, input the reconstructed deviation value and the real normal deviation value extracted from the historical normal operation state into the real distribution adversarial discrimination module. The module will then pass through the convolutional layer to extract features and the fully connected layer to calculate the prediction probability value. Subtract the prediction probability value from 1 to obtain the discrimination mismatch probability.

[0032] S35. Input the reconstruction error and the discrimination mismatch probability into the anomaly confidence calculation module, multiply the reconstruction error by a preset first weight coefficient, multiply the discrimination mismatch probability by a preset second weight coefficient, add the two product results, and obtain the anomaly confidence of each mode.

[0033] Optionally, S4 specifically includes:

[0034] S41. In the safety topology of the production site, create worker nodes representing workers, work nodes representing production equipment, and area nodes representing the working environment, and treat worker nodes, work nodes, and area nodes as personnel-equipment-environment entities.

[0035] S42. Create equipment operation association edges between worker nodes and work nodes, create environmental impact association edges between work nodes and area nodes, and create personnel exposure association edges between area nodes and worker nodes. Use the equipment operation association edges, environmental impact association edges, and personnel exposure association edges as corresponding causal relationships to construct a safety topology.

[0036] S43. Assign the abnormal confidence scores of worker facial micro-expression features, arm movement features, and spatial distance features to worker nodes as dynamic attributes, assign the abnormal confidence scores of equipment voiceprint features to work nodes as dynamic attributes, and assign the abnormal confidence scores of dust concentration features to region nodes as dynamic attributes, thereby obtaining a dynamic instantiated knowledge graph.

[0037] Optionally, S5 specifically includes:

[0038] S51. Construct an improved MPNN neural network, initialize the weight matrix and bias vector of the message passing layer in the improved MPNN neural network, and in the dynamically instantiated knowledge graph, based on the causal relationship between entities, use the abnormal confidence of each node as the initial message feature and iteratively pass it to the adjacent nodes along the corresponding association edge.

[0039] S52. In the dynamically instantiated knowledge graph, the equipment operation association edges between worker nodes and work nodes, the environmental impact association edges between work nodes and region nodes, and the personnel exposure association edges between region nodes and worker nodes are all extracted. For each association edge, the current assigned value is set to 1, and the value 1 is used as the first initial weight value, the second initial weight value, and the third initial weight value of the corresponding association edge. The first initial weight value, the second initial weight value, and the third initial weight value are used as the transmission weights of the corresponding equipment operation association edge, environmental impact association edge, and personnel exposure association edge, respectively.

[0040] S53. In each iteration of the transmission process, for any target node in the dynamically instantiated knowledge graph, obtain all neighboring nodes directly connected to the target node through the associated edges, extract the abnormal confidence of each neighboring node, multiply the abnormal confidence of each neighboring node with the transmission weight of the corresponding associated edge, sum the product results obtained from all neighboring nodes, and obtain the aggregated message features of the target node.

[0041] S54. The aggregated message features of the target node are concatenated with the current abnormal confidence of the target node and input into the improved MPNN neural network to calculate the linear mapping features. The fluctuation tolerance range of the modal features corresponding to the target node in the multimodal spatiotemporal baseline under the current micro-action stage is read, the deviation ratio is calculated, the dynamic damping coefficient is obtained, and the risk state features are updated to obtain the updated risk state features.

[0042] S55. Repeat steps S52 to S54 until the risk state features of each node in the dynamically instantiated knowledge graph have been updated and calculated. Increment the current iteration number by 1 and proceed to the next iteration.

[0043] S56. When the iteration reaches the preset number of iterations, stop the iteration. Based on the geographical region identifier or process collaboration relationship corresponding to the node in the dynamically instantiated knowledge graph, extract all worker nodes, work nodes and region nodes and their corresponding associated edges within the target monitoring area to form a local subgraph. If no target monitoring area is set, the entire dynamically instantiated knowledge graph will be used as the full graph.

[0044] S57. Read the risk status characteristics of all nodes in the full map or local subgraph, square the risk status characteristics of all nodes and sum them to obtain the feature square sum, divide the feature square sum by the total number of nodes in the full map or local subgraph to calculate the average risk energy value, and use it as the risk aggregate energy value of the full map or local subgraph.

[0045] Optionally, S54 specifically includes:

[0046] S541. The aggregated message features of the target node are concatenated with the current abnormal confidence of the target node to obtain the concatenated feature vector. The concatenated feature vector is input into the message passing layer of the improved MPNN neural network. The concatenated feature vector is multiplied with the weight matrix of the message passing layer. The result of the matrix multiplication is added to the bias vector of the message passing layer to obtain the linear mapping feature.

[0047] S542. Read the fluctuation tolerance range of the modal features corresponding to the target node in the multimodal spatiotemporal baseline under the current micro-action stage, extract the upper limit and lower limit values ​​from the fluctuation tolerance range, subtract the upper limit value from the current abnormal confidence of the target node to obtain the first difference value, subtract the lower limit value from the upper limit value to obtain the second difference value, divide the first difference value by the second difference value to perform a division calculation, and use the quotient value obtained by the division calculation as the deviation ratio value.

[0048] S543. Calculate the negative deviation of the natural constant e to the power of the proportional value to obtain the exponential result. Add the constant 1 to the exponential result as the denominator and use the constant 1 as the numerator. Calculate the ratio of the numerator to the denominator and use the ratio as the dynamic damping coefficient.

[0049] S544. Calculate the damping adjustment feature by multiplying the linear mapping feature and the dynamic damping coefficient element by element. Input the damping adjustment feature into the ReLU activation function and use the output value as the updated risk state feature.

[0050] Optionally, S6 specifically includes:

[0051] S61. Compare the calculated risk aggregation energy value of the whole map or local sub-map with the preset dynamic alarm threshold. If the risk aggregation energy value of the whole map or local sub-map is less than or equal to the preset dynamic alarm threshold, no operation is performed and the next real-time collection and comparison is continued. If the risk aggregation energy value of the whole map or local sub-map is greater than the preset dynamic alarm threshold, it is determined that the preset dynamic alarm threshold is exceeded and a risk alarm is triggered.

[0052] S62. After triggering a risk alarm, find the node with the largest risk status feature value from the dynamically instantiated knowledge graph and use it as the starting backtracking node. Obtain the aggregated message features of all adjacent nodes that the starting backtracking node has received during the iterative transmission process.

[0053] S63. Multiply the aggregated message features transmitted by each adjacent node received by the starting backtracking node with the transmission weight of the corresponding associated edge, and select the adjacent node with the largest product as the first preceding node, and take the associated edge between the starting backtracking node and the first preceding node as the first key associated edge.

[0054] S64. Obtain the aggregated message features of all adjacent nodes that the first preceding node has received during the iterative transmission process, repeatedly perform product calculation and filtering operations, find the adjacent node with the largest product result as the second preceding node, and take the associated edge between the first preceding node and the second preceding node as the second key associated edge.

[0055] S65. Repeat the backtracking extraction operation until the current backtracked node has no adjacent nodes that pass the aggregated message feature. Arrange all the backtracked nodes in the backtracking order to obtain the key node sequence. Extract the associated edges between adjacent nodes in the key node sequence to obtain the key associated edge sequence.

[0056] S66. Read the modal features and anomaly confidence of the dynamic attributes of each key node in the key node sequence, read each key association edge in the key association edge sequence, determine whether the key association edge is a device operation association edge, an environmental impact association edge or a personnel exposure association edge, mark the device operation association edge as a device operation causal type, mark the environmental impact association edge as an environmental impact causal type, and mark the personnel exposure association edge as a personnel exposure causal type.

[0057] S67. The type obtained from the labeling is taken as the causal relationship type of the corresponding key associated edge. The modal features corresponding to the key node sequence and the abnormal confidence and the causal relationship type corresponding to the key associated edge sequence are spliced ​​and combined in the backtracking order to obtain the multimodal risk causal evidence chain.

[0058] Optionally, S7 specifically includes:

[0059] S71. The risk warning and multimodal risk causal evidence chain are sent to the display interface of the audit terminal through the communication network for display, to assist the auditor in making manual judgments;

[0060] S72. Receive the confirmation result entered by the auditor through the audit terminal after on-site review, and determine whether the confirmation result is a true risk or a false risk report.

[0061] S73. When the confirmed result is a false alarm, the transmission weight of the key association edge sequence contained in the corresponding multimodal risk causal evidence chain in the dynamically instantiated knowledge graph is multiplied by the preset first attenuation coefficient, the preset dynamic alarm threshold is added to the preset first increment value, and the adjusted transmission weight and dynamic alarm threshold are used as the optimized risk identification parameters.

[0062] S74. When the confirmed result is that the risk is real-time, the transmission weight of the key related edge sequence contained in the corresponding multimodal risk causal evidence chain in the dynamically instantiated knowledge graph is multiplied by the preset second amplification coefficient, the preset dynamic alarm threshold is subtracted from the preset second reduction value, and the adjusted transmission weight and dynamic alarm threshold are used as the optimized risk identification parameters.

[0063] The beneficial effects of this invention are:

[0064] This invention addresses the challenges of strong heterogeneity in multi-source data from work sites, high randomness in normal operational fluctuations, and the concealment of minor anomalies by constructing a multimodal spatiotemporal baseline and deploying a CVAE-GAN model. It employs micro-action stage alignment to calculate dynamic baseline values ​​and fluctuation tolerance ranges, combined with the CVAE-GAN model to perform bias encoding, conditional reconstruction, and adversarial discrimination, outputting anomaly confidence scores that fuse reconstruction errors and discrimination mismatch probabilities. By dynamically assigning each modal anomaly confidence score as an attribute to corresponding entity nodes, a dynamically instantiated knowledge graph is constructed. This dynamically instantiated knowledge graph is then input into an improved MPN. An N-neural network is used, introducing a multimodal spatiotemporal baseline and fluctuation tolerance range as dynamic damping coefficients. Based on the deviation ratio, the dynamic damping coefficient is calculated and multiplicatively adjusted on the linear mapping features. Iterative message propagation and damping truncation are performed to calculate the risk aggregation energy value of the entire graph or a local subgraph. During the risk alarm phase, an alarm is triggered by combining the risk aggregation energy value. Key node sequences and key related edge sequences are extracted by backtracking along the anomaly confidence propagation path, generating a multimodal risk causal evidence chain. Furthermore, an adaptive correction adjustment is performed on the propagation weights of related edges and the dynamic alarm threshold through a human-machine closed-loop feedback mechanism. Ultimately, this achieves comprehensive perception of operational safety risks, dynamic damping propagation suppression, and closed-loop intelligent identification with interpretable causal logic, effectively improving the anomaly quantification accuracy of multimodal feature fusion, the anti-interference capability of risk propagation, and the credibility of early warning decisions. Attached Figure Description

[0065] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0066] Figure 1 This is a flowchart of a job safety risk identification method based on multimodal knowledge graph proposed in this invention;

[0067] Figure 2 This is a flowchart of the multimodal bias quantization and anomaly confidence calculation based on the CVAE-GAN model proposed in this invention;

[0068] Figure 3 This is a flowchart of the risk state propagation and aggregated energy calculation based on the improved MPNN neural network and dynamic damping mechanism proposed in this invention. Detailed Implementation

[0069] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0070] refer to Figures 1-3 A method for identifying operational safety risks based on multimodal knowledge graphs includes the following steps:

[0071] S1. Simultaneously collect video streams, audio streams, and environmental sensor data from the production site. Identify and extract facial micro-expression features, arm movement features, and spatial distance features from the video stream. Extract equipment voiceprint features from the audio stream. Extract dust concentration features from the environmental sensor data to obtain a multimodal feature set containing three dimensions: personnel, equipment, and environment.

[0072] S2. Obtain the historical multimodal feature set of the same worker in the same process within the historical period, align the time axis according to the micro-motion stage of the process, calculate the dynamic benchmark value and fluctuation tolerance range of each modal feature under normal working conditions, and obtain the multimodal spatiotemporal baseline corresponding to each micro-motion stage.

[0073] S3. The real-time acquired multimodal feature set is compared with the multimodal spatiotemporal baseline. The deviation value generated by the comparison is input into the CVAE-GAN model to obtain the anomaly confidence of each modality. The improved CVAE-GAN model includes a multimodal deviation coding module, a conditional feature reconstruction module, a real distribution adversarial discrimination module, and an anomaly confidence fusion calculation module.

[0074] S4. A safety topology structure containing personnel-equipment-environment entities and corresponding causal relationships is preset. The anomaly confidence of each modality is dynamically assigned to the corresponding entity nodes to obtain a dynamically instantiated knowledge graph.

[0075] S5. Construct an improved MPNN neural network. In the dynamically instantiated knowledge graph, based on the causal relationship between entities, the abnormal confidence of each node is used as the initial message feature and iteratively passed along the association edge to the adjacent nodes. Multimodal spatiotemporal baseline and fluctuation tolerance range are introduced as dynamic damping coefficients, and the risk state features of each node are updated. The risk aggregation energy value of the whole graph or local subgraph is calculated.

[0076] S6. Determine whether the risk aggregation energy value of the whole graph or a local subgraph exceeds the preset dynamic alarm threshold. If it does, trigger a risk alarm. Based on the path of abnormal confidence transmission, backtrack to extract the key nodes and related edges that triggered the alarm, and obtain the multimodal risk causal evidence chain.

[0077] S7. Output the risk alarm and multimodal risk causal evidence chain to the audit terminal to assist manual judgment, and receive the confirmation result of the auditor's on-site review. Adjust the transmission weight of causal relationship and dynamic alarm threshold according to the confirmation result to obtain the optimized risk identification parameters.

[0078] This invention significantly improves the accuracy and interpretability of operational safety risk identification. By constructing a multimodal spatiotemporal baseline, it achieves unified modeling of heterogeneous data such as personnel micro-expressions, actions, equipment voiceprints, and environmental dust, accurately depicting normal operational patterns at different micro-action stages. When risk characteristics appear, the CVAE-GAN model is used to deeply quantify real-time deviations, effectively capturing weak yet authentic abnormal signals. After constructing a dynamically instantiated knowledge graph, isolated anomaly confidence levels are transformed into tightly interconnected graph structures, greatly enhancing the ability to uncover potential complex risk relationships. In particular, the introduction of a dynamic damping coefficient in the improved MPNN neural network message passing precisely suppresses the spread of abnormal signals caused by fluctuations in normal operations, fundamentally overcoming the problem of high false alarm rates in traditional methods. When the aggregated risk energy exceeds a threshold and triggers an alarm, the system can automatically trace back the transmission path, generating a multimodal risk causal evidence chain, making the warning process completely transparent and interpretable. Meanwhile, by adaptively adjusting the transmission weight and alarm threshold based on the results of manual review, a closed-loop self-evolution mechanism is formed, which demonstrates stronger robustness in complex and high-risk operating environments and significantly improves the level of intelligence in dynamic risk discovery, causal tracing and proactive intervention.

[0079] In this embodiment, S1 specifically includes:

[0080] S11. Simultaneously trigger the visible light camera, audio pickup and dust concentration sensor deployed on the production site through synchronous control signal to collect video stream, audio stream and environmental sensor data respectively.

[0081] S12. The video stream is processed by extracting frames at a preset frame rate to obtain video image frames. The face detection algorithm is used to locate the worker's facial region. The local contour coordinates of eyebrows, eyes, nose, mouth and cheeks are extracted sequentially from the facial region, and the relative displacement and deformation amplitude between each coordinate point are calculated. The relative displacement and deformation amplitude are classified and statistically analyzed according to facial muscle movement units to obtain the worker's facial micro-expression features. The preset frame rate is 30 frames / second.

[0082] S13. Use the skeletal key point detection algorithm to locate the worker's shoulder, elbow and wrist joints in the video image frame, calculate the motion vector between adjacent joints and count the change frequency per unit time, and extract arm movement features.

[0083] S14. Use an object detection algorithm to identify the worker's body bounding box and the equipment bounding box in the video image frame, calculate the Euclidean distance between the center point of the worker's body bounding box and the center point of the equipment bounding box, and extract the spatial distance features.

[0084] S15. Perform frame-by-frame windowing processing on the audio stream, apply Fast Fourier Transform to the windowed audio frames to calculate the frequency domain amplitude, extract the frequency values ​​and amplitudes corresponding to the top N largest energy peaks in the frequency domain amplitude to obtain the device's acoustic signature characteristics, perform moving average filtering processing on the dust concentration sensing signal in the environmental sensing data, calculate the average amplitude of the filtered signal according to the preset time window to obtain the dust concentration characteristics, where N is 3 and the preset time window is 5 seconds;

[0085] S16. The facial micro-expression features, arm movement features, and spatial distance features of workers at the same timestamp are classified into the personnel dimension, the voiceprint features of equipment are classified into the equipment dimension, and the dust concentration features are classified into the environment dimension, and combined to obtain a multimodal feature set.

[0086] In this embodiment, S2 specifically includes:

[0087] S21. Obtain the historical multimodal feature set of the same worker in the same process within the historical period. Divide the process into multiple micro-action stages according to the operation sequence of the process. Match and compare the timestamp of each modality feature in the historical multimodal feature set with the start and end times of each micro-action stage. Assign each modality feature to the corresponding micro-action stage and complete the time axis alignment by the micro-action stage of the process.

[0088] S22. In each micro-movement stage after alignment, extract the data of worker's facial micro-expression features, arm movement features, spatial distance features, equipment voiceprint features and dust concentration features under normal operating conditions, calculate the arithmetic mean of all values ​​of each modal feature under normal operating conditions, and use the arithmetic mean as the dynamic benchmark value of each modal feature under normal operating conditions.

[0089] S23. Calculate the absolute difference between all values ​​of each modal feature under normal operating conditions and the corresponding dynamic reference value. Calculate the standard deviation of all absolute differences. Use the standard deviation of the dynamic reference value plus a preset multiple as the upper limit and the standard deviation of the dynamic reference value minus a preset multiple as the lower limit. The upper and lower limits constitute the fluctuation tolerance range of each modal feature. The preset multiple is 3.

[0090] S24. Combine the dynamic baseline values ​​and fluctuation tolerance ranges of each modal feature corresponding to each micro-motion stage to obtain the multimodal spatiotemporal baseline corresponding to each micro-motion stage.

[0091] In this embodiment, S3 specifically includes:

[0092] S31. Subtract the values ​​of each modal feature in the real-time multimodal feature set from the dynamic reference value in the multimodal spatiotemporal baseline of the corresponding micro-action stage, and use the difference obtained by subtraction as the deviation value generated by comparison.

[0093] S32. Input the bias value generated by the comparison into the multimodal bias encoding module in the CVAE-GAN model. Map the bias value to the latent variable space through the linear transformation of the fully connected layer and output the mean vector and variance vector of the latent variables. Randomly sample from the standard normal distribution to generate a random noise vector with the same dimension as the variance vector. Multiply the random noise vector and the variance vector element by element and add the mean vector to obtain the latent space feature vector.

[0094] S33. The latent space feature vector is concatenated with the one-hot encoded label of the current micro-action stage. The concatenated vector is input into the conditional feature reconstruction module. The concatenated vector is upsampled and deconvolved through the transposed convolutional layer. The calculation result is flattened and the reconstruction deviation value with the same dimension as the input deviation value is output.

[0095] S34. Subtract the corresponding position of the input deviation value from the reconstructed deviation value and calculate the sum of squares. Divide the sum by the number of features contained in the deviation value to obtain the reconstruction error. At the same time, input the reconstructed deviation value and the true normal deviation value extracted from the historical normal operation state into the true distribution adversarial discrimination module. The module will then pass through the convolutional layer to extract features and the fully connected layer to calculate the output. The predicted probability value of the reconstructed deviation value belonging to the true normal distribution will be output. Subtract the predicted probability value from 1 to obtain the discrimination mismatch probability.

[0096] S35. Input the reconstruction error and the discrimination mismatch probability into the anomaly confidence calculation module, multiply the reconstruction error by a preset first weight coefficient, multiply the discrimination mismatch probability by a preset second weight coefficient, add the two product results, and obtain the anomaly confidence of each mode. The preset first weight coefficient is 0.6 and the preset second weight coefficient is 0.4.

[0097] This implementation introduces an improved CVAE-GAN model as an innovative technology, which has significant differences and advantages compared to traditional autoencoder (AE), variational autoencoder (VAE), and standard GAN anomaly detection models. Traditional AE and VAE, when processing complex multimodal industrial data, often suffer from over-reconstruction, leading to the misrepresentation of anomalous features and resulting in extremely low differentiation between normal and anomalous reconstruction errors, making it prone to missing weak anomalies. Standard GAN, on the other hand, suffers from training instability and mode collapse, making it difficult to accurately characterize the boundaries of normal data distribution, resulting in a high false positive rate.

[0098] This invention cleverly integrates the conditional reconstruction capability of variational inference with the distribution discrimination capability of adversarial learning through an improved CVAE-GAN model. In S32, reparameterization techniques are used to map to the latent variable space, enhancing the diversity of feature representation. S33 introduces one-hot encoded labels for the micro-action stage as conditional constraints, ensuring strict alignment of the reconstruction process with the current process features and effectively suppressing cross-stage feature confusion. S34 outputs the discrimination mismatch probability through the real distribution adversarial discrimination module, accurately identifying minute anomalies deviating from the normal at the fundamental level of data distribution, completely overcoming the deficiency of a single reconstruction error in capturing high-dimensional distribution shifts. S35 fuses the reconstruction error and discrimination mismatch probability with weights of 0.6 and 0.4, taking into account both feature-level and distribution-level bias measurements. This mechanism can accurately capture subtle evolutionary patterns such as micro-expression anomalies and voiceprint micro-distortions under complex operational interference, significantly reducing false positives and false negatives while significantly enhancing the model's accuracy and robustness in extracting multimodal anomaly features.

[0099] In this embodiment, S4 specifically includes:

[0100] S41. In the safety topology of the production site, create worker nodes representing workers, work nodes representing production equipment, and area nodes representing the working environment, and treat worker nodes, work nodes, and area nodes as personnel-equipment-environment entities.

[0101] S42. Create equipment operation association edges between worker nodes and work nodes, create environmental impact association edges between work nodes and area nodes, and create personnel exposure association edges between area nodes and worker nodes. Use the equipment operation association edges, environmental impact association edges, and personnel exposure association edges as corresponding causal relationships to construct a security topology structure that includes personnel-equipment-environment entities and corresponding causal relationships.

[0102] S43. Assign the abnormal confidence scores of worker facial micro-expression features, arm movement features, and spatial distance features to worker nodes as dynamic attributes, assign the abnormal confidence scores of equipment voiceprint features to work nodes as dynamic attributes, and assign the abnormal confidence scores of dust concentration features to region nodes as dynamic attributes, thereby obtaining a dynamic instantiated knowledge graph.

[0103] In this embodiment, S5 specifically includes:

[0104] S51. Construct an improved MPNN neural network, initialize the weight matrix and bias vector of the message passing layer in the improved MPNN neural network, and in the dynamically instantiated knowledge graph, based on the causal relationship between entities, use the abnormal confidence of each node as the initial message feature and iteratively pass it to the adjacent nodes along the corresponding association edge.

[0105] S52. In the dynamically instantiated knowledge graph, the equipment operation association edges between worker nodes and work nodes, the environmental impact association edges between work nodes and region nodes, and the personnel exposure association edges between region nodes and worker nodes are all extracted. For each association edge, the current assigned value is set to 1, and the value 1 is used as the first initial weight value, the second initial weight value, and the third initial weight value of the corresponding association edge. The first initial weight value, the second initial weight value, and the third initial weight value are used as the transmission weights of the corresponding equipment operation association edge, environmental impact association edge, and personnel exposure association edge, respectively.

[0106] S53. In each iteration of the transmission process, for any target node in the dynamically instantiated knowledge graph, obtain all neighboring nodes directly connected to the target node through the associated edges, extract the abnormal confidence of each neighboring node, multiply the abnormal confidence of each neighboring node with the transmission weight of the corresponding associated edge, sum the product results obtained from all neighboring nodes, and obtain the aggregated message features of the target node.

[0107] S54. The aggregated message features of the target node are concatenated with the current abnormal confidence of the target node and input into the improved MPNN neural network to calculate the linear mapping features. The fluctuation tolerance range of the modal features corresponding to the target node in the multimodal spatiotemporal baseline under the current micro-action stage is read, the deviation ratio is calculated, the dynamic damping coefficient is obtained, and the risk state features are updated to obtain the updated risk state features.

[0108] S55. Repeat steps S52 to S54 until the risk state features of each node in the dynamically instantiated knowledge graph have been updated and calculated. Increment the current iteration number by 1 and proceed to the next iteration.

[0109] S56. When the iteration reaches the preset number of iterations, the iteration stops. Based on the geographical region identifier or process collaboration relationship corresponding to the node in the dynamic instantiated knowledge graph, all worker nodes, work nodes and region nodes and their corresponding associated edges within the target monitoring area are extracted to form a local subgraph. If no target monitoring area is set, the entire dynamic instantiated knowledge graph is used as the full graph. The preset number of iterations is 3.

[0110] S57. Read the risk status characteristics of all nodes in the full map or local subgraph, square the risk status characteristics of all nodes and sum them to obtain the feature square sum, divide the feature square sum by the total number of nodes in the full map or local subgraph to calculate the average risk energy value, and use it as the risk aggregate energy value of the full map or local subgraph.

[0111] In this embodiment, S54 specifically includes:

[0112] S541. The aggregated message features of the target node are concatenated with the current abnormal confidence of the target node to obtain the concatenated feature vector. The concatenated feature vector is input into the message passing layer of the improved MPNN neural network. The concatenated feature vector is multiplied with the weight matrix of the message passing layer. The result of the matrix multiplication is added to the bias vector of the message passing layer to obtain the linear mapping feature.

[0113] S542. Read the fluctuation tolerance range of the modal features corresponding to the target node in the multimodal spatiotemporal baseline under the current micro-action stage, extract the upper limit and lower limit values ​​from the fluctuation tolerance range, subtract the upper limit value from the current abnormal confidence of the target node to obtain the first difference value, subtract the lower limit value from the upper limit value to obtain the second difference value, divide the first difference value by the second difference value to perform a division calculation, and use the quotient value obtained by the division calculation as the deviation ratio value.

[0114] S543. Calculate the negative deviation of the natural constant e to the power of the proportional value to obtain the exponential result. Add the constant 1 to the exponential result as the denominator and use the constant 1 as the numerator. Calculate the ratio of the numerator to the denominator and use the ratio as the dynamic damping coefficient.

[0115] S544. Calculate the damping adjustment feature by multiplying the linear mapping feature and the dynamic damping coefficient element by element. Input the damping adjustment feature into the ReLU activation function and use the output value as the updated risk state feature.

[0116] This invention utilizes an improved MPNN neural network combined with a dynamic damping mechanism to achieve graph-structured propagation of multimodal anomaly signals and global risk energy aggregation. The confidence levels of each modality's anomaly are used as initial message features. Iterative propagation and weighted summation based on causal relationships between personnel, equipment, and the environment are used to construct a dynamic message aggregation structure. During propagation, the deviation ratio is calculated using the fluctuation tolerance range of the multimodal spatiotemporal baseline. A dynamic damping coefficient is adaptively generated using the Sigmoid function, and the linear mapping features are adjusted element-wise and updated with ReLU activation to update the risk state. This invention can accurately suppress the unlimited propagation of redundant signals under normal operational fluctuation interference or weak distortion of a single feature, effectively preventing the spread of false alarms. The node features, optimized through three iterations, are used to calculate the risk aggregation energy value using the feature squared mean, significantly enhancing the sensitivity to the evolution trend of complex risks and the robustness of global assessment.

[0117] The improved MPNN neural network of this invention is similar to the original MPNN neural network in that both retain the core architecture of the message-passing neural network, namely, the topological association based on the graph structure, which transmits the features of neighboring nodes as messages along the association edges, and aggregates neighbor information to update the features of the central node by weighted summation. Both also support a multi-round iterative transmission mechanism to capture higher-order association information.

[0118] The difference lies in that this invention breaks the limitations of the original MPNN model, which uses fixed linear transformation and indiscriminate aggregation in the message passing update stage. It introduces a dynamic damping adjustment mechanism based on the spatiotemporal baseline and constructs an adaptive risk propagation attenuation system. Building upon the original model's direct linear mapping and activation output of aggregated features, this invention innovatively embeds steps S542 and S543 after step S541. First, the fluctuation tolerance range of the modal features corresponding to the target node in the multimodal spatiotemporal baseline of the current micro-action stage is read, and the upper and lower limits are extracted. The upper limit is subtracted from the current anomaly confidence to obtain the first difference, and the lower limit is subtracted from the upper limit to obtain the second difference. The two are divided to calculate the deviation ratio. Subsequently, the negative deviation ratio power of the natural constant e is calculated, and the dynamic damping coefficient is calculated using the Sigmoid formula. Finally, in step S544, the linear mapping features and the dynamic damping coefficient are multiplied element-wise to obtain the damping adjustment features, which are then input into the ReLU activation function to update the risk state.

[0119] Based on the aforementioned improvements, the beneficial effect of this invention lies in the fact that by introducing a dynamic damping coefficient, the improved MPNN can adaptively adjust the message propagation intensity according to the degree to which the current feature deviates from the normal tolerance range. When node fluctuations are within the tolerance range, the damping coefficient is larger, which can effectively suppress the infinite spread of weak abnormal signals caused by normal operational fluctuations in the graph and the resulting false alarms across the entire graph; when deviating from the tolerance range, the damping coefficient decreases, ensuring the rapid propagation and aggregation of true risk signals. This design breaks through the limitations of error accumulation and false alarm spread caused by the indiscriminate transmission of the original MPNN, significantly enhancing the model's resistance to complex operational interference, and greatly improving the robustness of dynamic risk identification while improving the accuracy of composite risk evolution perception.

[0120] In this embodiment, S6 specifically includes:

[0121] S61. The calculated risk aggregation energy value of the whole map or a local sub-map is compared with a preset dynamic alarm threshold. If the risk aggregation energy value of the whole map or a local sub-map is less than or equal to the preset dynamic alarm threshold, no operation is performed and the next real-time acquisition and comparison is continued. If the risk aggregation energy value of the whole map or a local sub-map is greater than the preset dynamic alarm threshold, it is determined that the preset dynamic alarm threshold is exceeded and a risk alarm is triggered. The preset dynamic alarm threshold is 0.75.

[0122] S62. After triggering a risk alarm, find the node with the largest risk status feature value from the dynamically instantiated knowledge graph and use it as the starting backtracking node. Obtain the aggregated message features of all adjacent nodes that the starting backtracking node has received during the iterative transmission process.

[0123] S63. Multiply the aggregated message features transmitted by each adjacent node received by the starting backtracking node with the transmission weight of the corresponding associated edge, and select the adjacent node with the largest product as the first preceding node, and take the associated edge between the starting backtracking node and the first preceding node as the first key associated edge.

[0124] S64. Obtain the aggregated message features of all adjacent nodes that the first preceding node has received during the iterative transmission process, repeatedly perform product calculation and filtering operations, find the adjacent node with the largest product result as the second preceding node, and take the associated edge between the first preceding node and the second preceding node as the second key associated edge.

[0125] S65. Repeat the backtracking extraction operation until the current backtracked node has no adjacent nodes that pass the aggregated message feature. Arrange all the backtracked nodes in the backtracking order to obtain the key node sequence. Extract the associated edges between adjacent nodes in the key node sequence to obtain the key associated edge sequence.

[0126] S66. Read the modal features and anomaly confidence of the dynamic attributes of each key node in the key node sequence, read each key association edge in the key association edge sequence, determine whether the key association edge is a device operation association edge, an environmental impact association edge or a personnel exposure association edge, mark the device operation association edge as a device operation causal type, mark the environmental impact association edge as an environmental impact causal type, and mark the personnel exposure association edge as a personnel exposure causal type.

[0127] S67. The type obtained from the labeling is taken as the causal relationship type of the corresponding key associated edge. The modal features corresponding to the key node sequence and the abnormal confidence and the causal relationship type corresponding to the key associated edge sequence are spliced ​​and combined in the backtracking order to obtain the multimodal risk causal evidence chain.

[0128] In this embodiment, S7 specifically includes:

[0129] S71. The risk warning and multimodal risk causal evidence chain are sent to the display interface of the audit terminal through the communication network for display, to assist the auditor in making manual judgments;

[0130] S72. Receive the confirmation result entered by the auditor through the audit terminal after on-site review, and determine whether the confirmation result is a true risk or a false risk report.

[0131] S73. When the confirmed result is a false positive, the transmission weight of the key association edge sequence contained in the corresponding multimodal risk causal evidence chain in the dynamically instantiated knowledge graph is multiplied by a preset first attenuation coefficient, the preset dynamic alarm threshold is added to a preset first increment value, and the adjusted transmission weight and dynamic alarm threshold are used as optimized risk identification parameters. The preset first attenuation coefficient is 0.9, and the preset first increment value is 0.05.

[0132] S74. When the result confirms that the risk is real-time, the transmission weight of the key association edge sequence contained in the corresponding multimodal risk causal evidence chain in the dynamically instantiated knowledge graph is multiplied by a preset second amplification coefficient, the preset dynamic alarm threshold is subtracted from the preset second reduction value, and the adjusted transmission weight and dynamic alarm threshold are used as the optimized risk identification parameters. The preset second amplification coefficient is 1.1, and the preset second reduction value is 0.05.

[0133] Example 1: To verify the feasibility of this invention in the field of operational safety risk identification, it was applied to the intelligent mining safety monitoring platform of a large ferrous metal mining group in a certain province. The mine operates in an extremely harsh environment, constantly facing challenges of high temperature, high humidity, and high dust levels. The underground working area is widely distributed, including 3 main mining areas, 12 main transport roadways, and 8 core tunneling faces. On average, more than 650 people work underground daily, and there are over 140 large mining and transport equipment. Risk events in mining operations are complex and evolve rapidly. Typical risk events include: mechanical injuries caused by a combination of personnel inattention and violations of operating procedures; production stoppage risks caused by equipment tool wear or malfunction combined with voiceprint distortion; and suffocation and explosion risks caused by sudden changes in environmental dust concentration combined with exceeding safe distance limits.

[0134] The platform receives over 2.4TB of video streams from underground explosion-proof cameras, audio streams from microphones, and various environmental sensor data daily. Traditional mine safety monitoring systems rely primarily on single-threshold sensor over-limit alarms and manual video inspections, resulting in severe response lag and high false alarm rates, failing to dynamically adapt to the complex changes in the underground working environment. Particularly during the micro-movement phase of blasting and loading / unloading operations, drastic fluctuations in dust concentration and equipment load often lead to frequent false alarms in traditional systems due to their inability to distinguish between normal operational fluctuations and genuine anomalies. Furthermore, when personnel exhibit early abnormal blinking frequency, missed arm movements, or slight abnormal noises from equipment, these subtle and isolated characteristics lead to missed detections, seriously threatening safe production in the mine.

[0135] In practical deployment, the method of this invention transforms the aforementioned multi-source heterogeneous data into structured input, constructing a dynamic perception and identification process for operational safety. First, video streams, audio streams, and environmental sensor data are collected simultaneously. From the video stream, features such as the number of blinks on a worker's face, the number of times an arm is raised, and the spatial distance to hazards are precisely extracted. From the audio stream, equipment acoustic waveform features are extracted. From the environmental data, dust concentration contrast features are extracted, forming a multimodal feature set encompassing personnel, equipment, and environment. Historical data of the same worker performing the same process within a historical period is acquired, and timelines are aligned according to micro-movement stages. For example, a baseline of worker alertness is derived from a month's worth of video recordings of average blinks; an operational standard baseline is derived from recording the average number of arm raises; the distance between the head or arm and the target object is recorded to determine the normal operating distance; the daily waveform range is derived from machine acoustic analysis over 30 days; and environmental patterns are derived from recording the average dust concentration over a certain period. Based on this, the dynamic baseline values ​​and fluctuation tolerance ranges for each modality under normal operating conditions are calculated, resulting in a multimodal spatiotemporal baseline. The deviation values ​​generated by comparing real-time features with the spatiotemporal baseline are input into the CVAE-GAN model. Through multimodal deviation encoding, conditional feature reconstruction, and adversarial discrimination of the true distribution, the anomaly confidence of each modality is accurately quantified. For example, excessive blinking may indicate danger from dust and insects, while excessive blinking indicates lack of concentration; an abnormal arm raising indicates missed procedures or physical discomfort; exceeding distance limits indicates collision danger; and abnormal voiceprint waveforms indicate tool wear or machine malfunction. Subsequently, each anomaly confidence is dynamically assigned as an attribute to a preset safety topology entity node, constructing a dynamically instantiated knowledge graph. An improved MPNN neural network is introduced into the graph for message passing. In particular, a multimodal spatiotemporal baseline and fluctuation tolerance range are introduced as dynamic damping coefficients. When node fluctuations are within the tolerance range, the damping coefficient is larger, effectively suppressing the unlimited spread of abnormal signals and false alarms across the entire graph caused by fluctuations in normal operations; when deviating from the tolerance range, the damping coefficient decreases, ensuring the rapid propagation and aggregation of real risk signals. The risk aggregation energy value of the entire graph or a local subgraph is calculated. Once it exceeds the dynamic alarm threshold, an alarm is triggered, and key nodes and related edges are extracted by backtracking along the transmission path to generate a multimodal risk causal evidence chain. Finally, the alarm and evidence chain are output to the scheduling terminal to assist manual judgment. Based on the confirmation results of manual review, the transmission weight of related edges and the dynamic alarm threshold are adaptively adjusted to achieve closed-loop self-evolution. Table 1 below shows the comparison data between the method of this invention and the existing traditional single threshold monitoring method in the task of identifying operational safety risks during a three-month industrial trial operation:

[0136] Table 1. Multidimensional performance comparison data between the present invention and traditional methods

[0137]

[0138] Based on the comparative data shown in Table 1, it can be seen that the operation safety risk identification method based on multimodal knowledge graph proposed in this invention shows significant performance advantages over the traditional single threshold monitoring method in complex mining scenarios, especially in key indicators such as early warning accuracy, response speed, false alarm and missed alarm control, and causal tracing.

[0139] Regarding early warning accuracy, this invention maintains a high level of over 94% in all four typical operational risk scenarios, far exceeding the average accuracy of traditional systems (approximately 66%). For example, in the scenario of "sudden changes and exceeding limits in environmental dust," traditional systems rely on fixed dust concentration thresholds and are highly susceptible to misjudgments due to normal operational fluctuations such as the natural settling of dust after blasting, achieving only an accuracy of 62.7%. In contrast, this invention constructs a multimodal spatiotemporal baseline and utilizes a CVAE-GAN model to accurately quantify deviations, combined with a dynamic damping mechanism to suppress the propagation of normal fluctuations, achieving an accuracy of 94.9%, effectively improving the risk identification capability in complex environments.

[0140] Regarding response timeliness, this invention achieves efficient aggregation of risk propagation links through an improved MPNN neural network, significantly shortening response time. In four types of risk scenarios, the average response time is reduced from nearly 40 seconds using traditional methods to approximately 10 seconds, a nearly fourfold increase in response speed. For example, the traditional response to "equipment acoustic distortion and tool failure" takes an average of 38.9 seconds, while this invention only takes 9.7 seconds, greatly improving the real-time handling capability for sudden anomalies in mines.

[0141] It also demonstrates significant advantages in controlling false alarm and false negative rates. The average false alarm rate of this invention is controlled at around 4%, and the false negative rate is as low as around 2%. Compared with the traditional method's average false alarm rate of 17% and false negative rate of nearly 11%, it significantly reduces redundant warnings and omissions, lowers the cost of manual intervention, and improves system stability.

[0142] In terms of causal attribution, traditional methods are completely unable to provide a basis for decision-making, while this invention can backtrack along the abnormal confidence propagation path to generate a multimodal risk causal evidence chain, with an attribution accuracy rate consistently above 93%. For example, it reaches 96.1% in the scenario of "dangerous overstepping of human-machine spatial distance," providing transparent and explainable decision support for regulatory personnel.

[0143] Overall, this invention effectively solves the problems of limited state expression and uncontrolled anomaly propagation in traditional methods in key operational scenarios such as distracted personnel, distorted equipment voiceprints, sudden changes in environmental dust, and exceeding human-machine space limits. It achieves efficient, accurate, and interpretable risk identification and early warning, filling a technical gap in dynamic risk control.

[0144] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying operational safety risks based on multimodal knowledge graphs, characterized in that, Includes the following steps: S1. Synchronously collect video streams, audio streams, and environmental sensor data from the production site, identify and extract facial micro-expression features, arm movement features, spatial distance features, equipment acoustic features, and dust concentration features to obtain a multimodal feature set; S2. Obtain the historical multimodal feature set of the same worker in the same process within the historical period, align the time axis according to the micro-action stage of the process, calculate the dynamic benchmark value and fluctuation tolerance range respectively, and obtain the multimodal spatiotemporal baseline. S3. Compare the multimodal feature set with the multimodal spatiotemporal baseline, input the deviation value into the CVAE-GAN model, and obtain the anomaly confidence of each modality. The improved CVAE-GAN model includes a multimodal deviation coding module, a conditional feature reconstruction module, a real distribution adversarial discrimination module, and an anomaly confidence fusion calculation module. S4. Pre-safe topology structure: The anomaly confidence of each modality is dynamically assigned to the corresponding entity nodes as an attribute, resulting in a dynamically instantiated knowledge graph. S5. Construct an improved MPNN neural network. In the dynamically instantiated knowledge graph, based on the causal relationship between entities, the abnormal confidence of each node is used as the initial message feature and iteratively passed along the association edge to the adjacent nodes. Multimodal spatiotemporal baseline and fluctuation tolerance range are introduced as dynamic damping coefficients, and the risk state features of each node are updated. The risk aggregation energy value of the whole graph or local subgraph is calculated. S6. Determine whether the risk aggregation energy value of the whole graph or a local subgraph exceeds the preset dynamic alarm threshold. If it does, trigger a risk alarm, backtrack to extract the key nodes and related edges that triggered the alarm, and obtain a multimodal risk causal evidence chain. S7. Output the risk alarm and multimodal risk causal evidence chain to the audit terminal to assist manual judgment, and receive the confirmation result of the auditor's on-site review. Adjust the transmission weight of causal relationship and dynamic alarm threshold to obtain the optimized risk identification parameters.

2. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 1, characterized in that, S1 specifically includes: S11. Simultaneously trigger the visible light camera, audio pickup and dust concentration sensor deployed on the production site through synchronous control signal to collect video stream, audio stream and environmental sensor data respectively. S12. The video stream is processed by extracting frames at a preset frame rate to obtain video image frames. The face detection algorithm is used to locate the worker's facial region. The local contour coordinates of eyebrows, eyes, nose, mouth and cheeks are extracted sequentially from the facial region, and the relative displacement and deformation amplitude between each coordinate point are calculated. The relative displacement and deformation amplitude are classified and statistically analyzed according to the facial muscle movement unit to obtain the worker's facial micro-expression features. S13. Use the skeletal key point detection algorithm to locate the worker's shoulder, elbow and wrist joints in the video image frame, calculate the motion vector between adjacent joints and count the change frequency per unit time, and extract arm movement features. S14. Use an object detection algorithm to identify the worker's body bounding box and the equipment bounding box in the video image frame, calculate the Euclidean distance between the center point of the worker's body bounding box and the center point of the equipment bounding box, and extract the spatial distance features. S15. Perform frame-by-frame windowing processing on the audio stream, apply Fast Fourier Transform to the windowed audio frames to calculate the frequency domain amplitude, extract the frequency values ​​and amplitudes corresponding to the top N largest energy peaks in the frequency domain amplitude to obtain the device's acoustic signature characteristics, perform moving average filtering on the dust concentration sensing signal in the environmental sensing data, calculate the average amplitude of the filtered signal according to the preset time window to obtain the dust concentration characteristics. S16. The facial micro-expression features, arm movement features, and spatial distance features of workers at the same timestamp are classified into the personnel dimension, the voiceprint features of equipment are classified into the equipment dimension, and the dust concentration features are classified into the environment dimension, and combined to obtain a multimodal feature set.

3. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 1, characterized in that, S2 specifically includes: S21. Obtain the historical multimodal feature set of the same worker in the same process within the historical period. Divide the process into multiple micro-action stages according to the operation sequence of the process. Match and compare the timestamp of each modality feature in the historical multimodal feature set with the start and end times of each micro-action stage. Assign each modality feature to the corresponding micro-action stage and complete the time axis alignment by the micro-action stage of the process. S22. In each micro-movement stage after alignment, extract the data of worker's facial micro-expression features, arm movement features, spatial distance features, equipment voiceprint features and dust concentration features under normal operating conditions, calculate the arithmetic mean of all values ​​of each modal feature under normal operating conditions, and use the arithmetic mean as the dynamic benchmark value of each modal feature under normal operating conditions. S23. Calculate the absolute difference between all values ​​of each modal feature under normal operating conditions and the corresponding dynamic reference value. Calculate the standard deviation of all absolute differences. Use the standard deviation of the dynamic reference value plus a preset multiple as the upper limit and the standard deviation of the dynamic reference value minus a preset multiple as the lower limit. The upper and lower limits constitute the fluctuation tolerance range of each modal feature. S24. Combine the dynamic baseline values ​​and fluctuation tolerance ranges of each modal feature corresponding to each micro-action stage to obtain the multimodal spatiotemporal baseline.

4. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 1, characterized in that, S3 specifically includes: S31. Subtract the values ​​of each modal feature in the real-time multimodal feature set from the dynamic reference value in the multimodal spatiotemporal baseline of the corresponding micro-action stage, and use the difference obtained by subtraction as the deviation value generated by comparison. S32. Input the bias value generated by the comparison into the multimodal bias encoding module in the CVAE-GAN model. Map the bias value to the latent variable space through the linear transformation of the fully connected layer and output the mean vector and variance vector of the latent variables. Randomly sample from the standard normal distribution to generate a random noise vector with the same dimension as the variance vector. Multiply the random noise vector and the variance vector element by element and add the mean vector to obtain the latent space feature vector. S33. The latent space feature vector is concatenated with the one-hot encoded label of the current micro-action stage. The concatenated vector is input into the conditional feature reconstruction module. The concatenated vector is upsampled and deconvolved through the transposed convolutional layer. The reconstruction deviation value is output after the calculation result is flattened. S34. Subtract the corresponding position of the input deviation value from the reconstructed deviation value and calculate the sum of squares. Divide the sum by the number of features contained in the deviation value to obtain the reconstruction error. At the same time, input the reconstructed deviation value and the real normal deviation value extracted from the historical normal operation state into the real distribution adversarial discrimination module. The module will then pass through the convolutional layer to extract features and the fully connected layer to calculate the prediction probability value. Subtract the prediction probability value from 1 to obtain the discrimination mismatch probability. S35. Input the reconstruction error and the discrimination mismatch probability into the anomaly confidence calculation module, multiply the reconstruction error by a preset first weight coefficient, multiply the discrimination mismatch probability by a preset second weight coefficient, add the two product results, and obtain the anomaly confidence of each mode.

5. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 1, characterized in that, S4 specifically includes: S41. In the safety topology of the production site, create worker nodes representing workers, work nodes representing production equipment, and area nodes representing the working environment, and treat worker nodes, work nodes, and area nodes as personnel-equipment-environment entities. S42. Create equipment operation association edges between worker nodes and work nodes, create environmental impact association edges between work nodes and area nodes, and create personnel exposure association edges between area nodes and worker nodes. Use the equipment operation association edges, environmental impact association edges, and personnel exposure association edges as corresponding causal relationships to construct a safety topology. S43. Assign the abnormal confidence scores of worker facial micro-expression features, arm movement features, and spatial distance features to worker nodes as dynamic attributes, assign the abnormal confidence scores of equipment voiceprint features to work nodes as dynamic attributes, and assign the abnormal confidence scores of dust concentration features to region nodes as dynamic attributes, thereby obtaining a dynamic instantiated knowledge graph.

6. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 1, characterized in that, S5 specifically includes: S51. Construct an improved MPNN neural network, initialize the weight matrix and bias vector of the message passing layer in the improved MPNN neural network, and in the dynamically instantiated knowledge graph, based on the causal relationship between entities, use the abnormal confidence of each node as the initial message feature and iteratively pass it to the adjacent nodes along the corresponding association edge. S52. In the dynamically instantiated knowledge graph, the equipment operation association edges between worker nodes and work nodes, the environmental impact association edges between work nodes and region nodes, and the personnel exposure association edges between region nodes and worker nodes are all extracted. For each association edge, the current assigned value is set to 1, and the value 1 is used as the first initial weight value, the second initial weight value, and the third initial weight value of the corresponding association edge. The first initial weight value, the second initial weight value, and the third initial weight value are used as the transmission weights of the corresponding equipment operation association edge, environmental impact association edge, and personnel exposure association edge, respectively. S53. In each iteration of the transmission process, for any target node in the dynamically instantiated knowledge graph, obtain all neighboring nodes directly connected to the target node through the associated edges, extract the abnormal confidence of each neighboring node, multiply the abnormal confidence of each neighboring node with the transmission weight of the corresponding associated edge, sum the product results obtained from all neighboring nodes, and obtain the aggregated message features of the target node. S54. The aggregated message features of the target node are concatenated with the current abnormal confidence of the target node and input into the improved MPNN neural network to calculate the linear mapping features. The fluctuation tolerance range of the modal features corresponding to the target node in the multimodal spatiotemporal baseline under the current micro-action stage is read, the deviation ratio is calculated, the dynamic damping coefficient is obtained, and the risk state features are updated to obtain the updated risk state features. S55. Repeat steps S52 to S54 until the risk state features of each node in the dynamically instantiated knowledge graph have been updated and calculated. Increment the current iteration number by 1 and proceed to the next iteration. S56. When the iteration reaches the preset number of iterations, stop the iteration. Based on the geographical region identifier or process collaboration relationship corresponding to the node in the dynamically instantiated knowledge graph, extract all worker nodes, work nodes and region nodes and their corresponding associated edges within the target monitoring area to form a local subgraph. If no target monitoring area is set, the entire dynamically instantiated knowledge graph will be used as the full graph. S57. Read the risk status characteristics of all nodes in the full map or local subgraph, square the risk status characteristics of all nodes and sum them to obtain the feature square sum, divide the feature square sum by the total number of nodes in the full map or local subgraph to calculate the average risk energy value, and use it as the risk aggregate energy value of the full map or local subgraph.

7. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 6, characterized in that, Specifically, S54 includes: S541. The aggregated message features of the target node are concatenated with the current abnormal confidence of the target node to obtain the concatenated feature vector. The concatenated feature vector is input into the message passing layer of the improved MPNN neural network. The concatenated feature vector is multiplied with the weight matrix of the message passing layer. The result of the matrix multiplication is added to the bias vector of the message passing layer to obtain the linear mapping feature. S542. Read the fluctuation tolerance range of the modal features corresponding to the target node in the multimodal spatiotemporal baseline under the current micro-action stage, extract the upper limit and lower limit values ​​from the fluctuation tolerance range, subtract the upper limit value from the current abnormal confidence of the target node to obtain the first difference value, subtract the lower limit value from the upper limit value to obtain the second difference value, divide the first difference value by the second difference value to perform a division calculation, and use the quotient value obtained by the division calculation as the deviation ratio value. S543. Calculate the negative deviation of the natural constant e to the power of the proportional value to obtain the exponential result. Add the constant 1 to the exponential result as the denominator and use the constant 1 as the numerator. Calculate the ratio of the numerator to the denominator and use the ratio as the dynamic damping coefficient. S544. Calculate the damping adjustment feature by multiplying the linear mapping feature and the dynamic damping coefficient element by element. Input the damping adjustment feature into the ReLU activation function and use the output value as the updated risk state feature.

8. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 1, characterized in that, S6 specifically includes: S61. Compare the calculated risk aggregation energy value of the whole map or local sub-map with the preset dynamic alarm threshold. If the risk aggregation energy value of the whole map or local sub-map is less than or equal to the preset dynamic alarm threshold, no operation is performed and the next real-time collection and comparison is continued. If the risk aggregation energy value of the whole map or local sub-map is greater than the preset dynamic alarm threshold, it is determined that the preset dynamic alarm threshold is exceeded and a risk alarm is triggered. S62. After triggering a risk alarm, find the node with the largest risk status feature value from the dynamically instantiated knowledge graph and use it as the starting backtracking node. Obtain the aggregated message features of all adjacent nodes that the starting backtracking node has received during the iterative transmission process. S63. Multiply the aggregated message features transmitted by each adjacent node received by the starting backtracking node with the transmission weight of the corresponding associated edge, and select the adjacent node with the largest product as the first preceding node, and take the associated edge between the starting backtracking node and the first preceding node as the first key associated edge. S64. Obtain the aggregated message features of all adjacent nodes that the first preceding node has received during the iterative transmission process, repeatedly perform product calculation and filtering operations, find the adjacent node with the largest product result as the second preceding node, and take the associated edge between the first preceding node and the second preceding node as the second key associated edge. S65. Repeat the backtracking extraction operation until the current backtracked node has no adjacent nodes that pass the aggregated message feature. Arrange all the backtracked nodes in the backtracking order to obtain the key node sequence. Extract the associated edges between adjacent nodes in the key node sequence to obtain the key associated edge sequence. S66. Read the modal features and anomaly confidence of the dynamic attributes of each key node in the key node sequence, read each key association edge in the key association edge sequence, determine whether the key association edge is a device operation association edge, an environmental impact association edge or a personnel exposure association edge, mark the device operation association edge as a device operation causal type, mark the environmental impact association edge as an environmental impact causal type, and mark the personnel exposure association edge as a personnel exposure causal type. S67. The type obtained from the labeling is taken as the causal relationship type of the corresponding key associated edge. The modal features corresponding to the key node sequence and the abnormal confidence and the causal relationship type corresponding to the key associated edge sequence are spliced ​​and combined in the backtracking order to obtain the multimodal risk causal evidence chain.

9. The method for identifying operational safety risks based on multimodal knowledge graphs according to claim 1, characterized in that, Specifically, S7 includes: S71. The risk warning and multimodal risk causal evidence chain are sent to the display interface of the audit terminal through the communication network for display, to assist the auditor in making manual judgments; S72. Receive the confirmation result entered by the auditor through the audit terminal after on-site review, and determine whether the confirmation result is a true risk or a false risk report. S73. When the confirmed result is a false alarm, the transmission weight of the key association edge sequence contained in the corresponding multimodal risk causal evidence chain in the dynamically instantiated knowledge graph is multiplied by the preset first attenuation coefficient, the preset dynamic alarm threshold is added to the preset first increment value, and the adjusted transmission weight and dynamic alarm threshold are used as the optimized risk identification parameters. S74. When the confirmed result is that the risk is real-time, the transmission weight of the key related edge sequence contained in the corresponding multimodal risk causal evidence chain in the dynamically instantiated knowledge graph is multiplied by the preset second amplification coefficient, the preset dynamic alarm threshold is subtracted from the preset second reduction value, and the adjusted transmission weight and dynamic alarm threshold are used as the optimized risk identification parameters.

Images