A large model-based dangerous work site safety monitoring system

Abstract

The application discloses a dangerous operation site safety monitoring system based on a large model and relates to the technical field of safety monitoring, comprising: a multi-modal recognition module, which is used for fusing monitoring video frames, operation ticket text data and gas sensor data through a multi-modal large model with visual-language joint understanding capability to recognize a set of irregular behaviors in a dangerous operation site, wherein the irregular behaviors at least include not wearing protective equipment, irregular ignition, guardian off-duty and insufficient safety distance; and a coupling gain quantization module. The application introduces a coupling gain coefficient to quantify the nonlinear coupling effect between irregular behaviors. When multiple irregular behaviors occur simultaneously in work, the accident probability is often far more than the simple superposition of the risk of each single irregular behavior. The coupling gain coefficient is used to statistically calculate the conditional probability of accidents caused by irregular combinations from historical data, to quantify the nonlinear coupling gain between irregular behaviors, so that the risk assessment result is closer to the actual risk condition of a chemical site.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring technology, specifically to a safety monitoring system for hazardous workplaces based on a large model. Background Technology

[0002] Safety monitoring of chemical production sites has always been a core issue in the field of industrial safety. These sites are classified as hazardous workplaces, as chemical production processes involve flammable, explosive, toxic, and harmful substances, and the working environment is complex. Accidents at such sites often result in serious casualties and property damage. With the rapid development of technologies such as computer vision, IoT sensing, and natural language processing, safety management systems based on intelligent monitoring methods are gradually becoming an important technological direction for the safety management of hazardous workplaces.

[0003] Currently, safety monitoring in hazardous work areas primarily relies on a combination of video surveillance and manual inspections. In terms of intelligent technologies, existing methods can utilize computer vision models to identify individual violations such as not wearing protective equipment, unauthorized hot work, supervisors leaving their posts, and insufficient safe distances, triggering alarms by setting thresholds. Some solutions incorporate multimodal data fusion, inputting video frames, work permit text, and gas sensor readings into the model to improve the coverage of violation identification. Furthermore, some technical solutions focus on the physical transmission relationships between equipment, attempting to predict accident outcomes from abnormal linkages in equipment operating parameters.

[0004] Current technologies have the following shortcomings: First, the assessment of violations is conducted in an isolated manner, item by item. In actual chemical operations, multiple violations often occur simultaneously or sequentially, exhibiting a coupling effect—the accident risk resulting from the superposition of two violations is not simply additive but may exhibit a non-linear amplification. For example, the probability of a fire or explosion caused by "unauthorized hot work" combined with "absence of a supervisor" is far higher than the sum of the probabilities of the two violations existing individually. Current technologies assess each violation independently and then weight and summarize them, failing to capture the coupling gain between violations and easily underestimating the true risk level of combined violations.

[0005] Secondly, there is a lack of reasoning ability regarding the transmission path from violations to accident consequences. Violations do not cause accidents instantaneously; rather, they evolve gradually along a certain transmission chain: violation patterns under specific work types trigger intermediate events, which further transmit to accident consequences of varying severity. Most existing technologies remain at a superficial level of "detecting violations → triggering alarms," ​​unable to deduce the transmission paths that combinations of violations might trigger, let alone reveal which nodes in the transmission chain are the key hubs in the evolution of risk. A few solutions involving inter-equipment transmission only focus on the linkage of physical parameters, failing to incorporate human-caused violations into the transmission reasoning framework.

[0006] Third, intervention strategies are generalized and imprecise. Existing systems, upon detecting violations, typically use uniform alarm pushes or general handling suggestions, failing to consider the specific location and degree of intervention capability of the violation in the transmission chain. The intervention time window varies for different violation nodes, and the losses that can be avoided after blocking also differ. A one-size-fits-all intervention strategy may delay handling at nodes with extremely narrow time windows, or waste emergency resources at nodes with minimal blocking benefits.

[0007] In summary, the field of safety monitoring in hazardous workplaces urgently needs a technical solution that can quantify the risks of combined violations, deduce risk transmission paths, identify key nodes in the transmission chain, and recommend precise intervention strategies. Summary of the Invention

[0008] To address the aforementioned technical issues, a safety monitoring system for hazardous workplaces based on a large model is provided. This technical solution solves the problems of low accuracy in linear risk superposition when violations occur, lack of reasoning ability for the transmission path from violations to accident consequences, and generalization and inaccuracy of intervention strategies.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a safety monitoring system for hazardous work sites based on a large model, characterized in that it includes: a multimodal recognition module, used to identify a set of violations in hazardous work sites by fusing monitoring video frames, work order text data and gas sensor data through a multimodal large model with vision-language joint understanding capabilities, wherein the violations include at least failure to wear protective equipment, unauthorized hot work, absence of supervisors from their posts and insufficient safety distance; The coupling gain quantization module is used to mine combination patterns of violations from historical violation records using association rule mining algorithms, and introduces a coupling gain coefficient to quantify the nonlinear coupling gain between violations, generating a comprehensive coupling risk score. The risk transmission reasoning module is used to infer the risk transmission path that the current combination of violations may trigger based on a pre-built knowledge graph of the relationship between violation patterns and accident consequences. The key node identification module is used to calculate the quantitative indicators of each node in the risk transmission path and identify the key nodes in the transmission chain. The quantitative indicators are the product of the operability and the blocking benefit. The intervention strategy recommendation module is used to match and recommend precise intervention strategies from a pre-built intervention strategy library based on the type and location of key nodes. The closed-loop feedback module is used to obtain the accident feature vector and the severity of the accident consequences after the accident occurs, and to update the system parameters using an incremental learning method. Communication connections between various modules.

[0010] Preferably, the coupling gain quantization module generates the comprehensive coupling risk score in the following ways: Define integrated coupling risk score The calculation formula is: in, The currently detected number The feature vector of a violation, For the first time in the historical database Typical feature vectors of violation patterns For the first Risk weighting for each violation pattern The similarity function is... Cosine similarity is used, and the calculation formula is as follows: ,in This is the feature vector of the current detection behavior. The feature vector of the sample in the violation sample library; For violations and The coupling gain coefficient between them.

[0011] Preferably, the determination of the coupling gain coefficient The methods of determination include, Calculate the probability of an incident caused by a combination of violations based on Bayesian conditional probability: in, and This data was compiled based on the company's historical accident records, near miss reports, and daily safety inspection records. is the Laplace smoothing coefficient, with a value ranging from 1 to 10, and is dynamically adjusted according to the amount of historical data. The coupling gain coefficient According to the violation and The increase in accident probability when coexisting with an individual is determined by the following formula: when At that time, it indicates a violation. and There is positive coupling gain between them.

[0012] Preferably, the knowledge graph of violation pattern-accident consequence association in the risk transmission reasoning module includes a four-layer structure: operation type layer, violation pattern layer, accident consequence layer, and severity level layer. Each layer is associated with another layer through a conditional probability table. The probability values ​​in the conditional probability table are calculated based on the co-occurrence frequency between nodes in each layer in historical samples and are obtained through maximum likelihood estimation optimization.

[0013] Preferably, in the key node identification module: Interventionability is defined as the ratio of the average time window between the occurrence of a violation and the implementation of an intervention measure to the time required to implement the intervention measure, with a value ranging from 0 to 1. The blocking benefit is the ratio of the expected accident loss that can be avoided by blocking the node to the expected total loss. The expected accident loss is the weighted average of the number of casualties and the amount of property damage after normalization.

[0014] Preferably, the intervention strategy recommendation module retrieves corresponding intervention measures from a pre-built intervention strategy library based on the location of key nodes in the risk transmission path and their corresponding violation pattern types, using vector-based similarity matching or rule-based reasoning matching, to determine the timing and method of intervention and generate precise intervention strategies.

[0015] Preferably, the multimodal recognition module further includes a behavior phase prediction unit, used to identify behavior phase states based on the heart rate variability and electromyography signals collected in real time by the wearable device worn by the worker. The behavior phase states include alertness, fatigue, and exhaustion. The behavior phase states are encoded into one-hot vectors, concatenated with visual feature vectors, and input into the multimodal large model as auxiliary features for identifying violations.

[0016] Preferably, the multimodal recognition module further includes a generative enhancement unit, which is used to expand the training samples by using a diffusion generation model and combining controllable generation conditions for less common violation scenarios, so as to increase the number of samples for less common scenarios.

[0017] Preferably, the diffusion generation model includes a stable diffusion model or a denoised diffusion probability model, and the controllable generation conditions include ControlNet condition control, which is used to control the scene layout and violation type of the generated samples.

[0018] Preferably, the closed-loop feedback module employs an experience-based replay incremental learning approach, adding the feature vectors and severity of consequences after the accident as new samples to the historical sample set, and incrementally fine-tuning the parameters of the coupled gain quantization module and the risk transmission inference module. The risk weight in the coupling gain quantization module and coupling gain coefficient The conditional probability table in the risk transmission inference module is updated based on the frequency statistics of the new samples after recalculation based on the new samples.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention quantifies the nonlinear coupling effect between violations by introducing a coupling gain coefficient. The system's risk assessment accuracy for high-risk violation combinations is significantly higher than that of the linear weighting method. When multiple violations occur simultaneously in chemical operations, the probability of an accident often far exceeds the simple sum of the risks of each individual violation. Traditional linear weighting schemes cannot capture this coupling amplification effect and easily underestimate the true risk level of combined violations. The coupling gain coefficient statistically analyzes the conditional probability of accidents caused by violation combinations from historical data, quantifies the nonlinear coupling gain between violations, and makes the risk assessment results closer to the actual risk situation at the chemical site.

[0020] The system of this invention uses a four-layer graph structure—operation type layer, violation mode layer, accident consequence layer, and severity level layer—combined with conditional probability tables to reason layer by layer, assisting safety managers in understanding the evolution path and transmission mechanism of risks, rather than simply providing isolated alarm signals.

[0021] The key node identification of this invention comprehensively considers both interveneability and the benefits of blocking, focusing limited intervention resources on the most worthwhile key links. Some nodes have extremely narrow time windows and cannot be intervened in time, while some nodes do not yield significant benefits even if blocked. The quantitative indicators unify the two dimensions of interveneability and the benefits of blocking, enabling the system to recommend high-value intervention points that have sufficient time windows for intervention and can minimize accident losses after blocking, thereby improving the actual effectiveness of intervention.

[0022] Behavioral phase prediction and generative augmentation enhance the robustness of multimodal recognition from the two dimensions of personnel state perception and data scarcity, respectively. Behavioral phase prediction captures the physiological state changes of workers from alertness to fatigue in real time through wearable devices, enabling the model to automatically increase its sensitivity to violations related to decreased attention during the fatigue period. Generative augmentation addresses the problem of insufficient training samples in less common violation scenarios by using diffusion models and controllable generation conditions to expand the sample size, enabling the system to maintain a high recognition accuracy even in less common violation scenarios. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a schematic diagram of the system architecture; Figure 3 The coupling gain matrix for multiple violations; Figure 4 This is a bar chart comparing the coupled score and the general linear score of this invention. Figure 5 This is a scatter plot of the nodes' intervention and blocking benefits in this invention. Figure 6 This is a ranking chart of the key quantitative indicators of the present invention. Figure 7 This is a schematic diagram illustrating the transmission path of the risks associated with this invention. Figure 8 This is a joint probability distribution diagram of the transmission path of the present invention. Detailed Implementation

[0024] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0025] Reference Figure 1 and Figure 2 As shown, a safety monitoring system for hazardous workplaces based on a large model includes: The multimodal recognition module serves as the system's perception entry point. This module deploys a large-scale multimodal model with joint vision-language understanding capabilities, such as a pre-trained vision-language model based on the Transformer architecture. It fuses video frames captured by surveillance cameras, electronic work permit text, and real-time gas sensor readings as three input channels. The video frames are processed by a visual encoder to extract spatial features, the work permit text by a text encoder to extract semantic features, and the gas sensor data is mapped into feature vectors through a numerical encoding layer. These three types of features are aligned and fused at a cross-modal attention layer. The fused multimodal representation is then fed into a classification head, which outputs a set of violations present in the current scene. Violations include at least four categories: not wearing protective equipment, unauthorized hot work, supervisor absenteeism, and insufficient safe distance. Furthermore, this module contains two sub-units: a behavior phase prediction unit and a generative enhancement unit. These are used to introduce auxiliary features related to the worker's physiological state and to expand training samples for less common scenarios, respectively. These will be discussed in detail in subsequent sections.

[0026] The coupling gain quantization module receives the set of violations output by the multimodal recognition module. It then uses association rule mining algorithms (such as the Apriori algorithm or FP-Growth algorithm) to mine frequently co-occurring violation combination patterns from historical violation records. A coupling gain coefficient is introduced to quantify the nonlinear coupling gain between different violations, ultimately generating a comprehensive coupling risk score R. This score not only reflects the inherent risk of a single type of violation but also characterizes the additional risk beyond linear superposition brought about by combinations of violations.

[0027] The risk transmission reasoning module uses the highly coupled violation combination output by the coupling gain quantification module as the trigger condition. Based on a pre-built knowledge graph of violation patterns and accident consequences, it infers the risk transmission path that the current violation combination may trigger. In other words, this module answers the question: if these violations exist simultaneously, along what path will the accident evolve and what consequences might it ultimately cause?

[0028] The key node identification module performs "breakpoint analysis" on the transmission path output by the risk transmission reasoning module, calculates the quantitative indicators of each node in the path (=interventionability × blocking benefit), and identifies those key nodes that "have the opportunity to intervene and are the most cost-effective to intervene in". These key nodes are the intervention focus points that the system should prioritize in the future.

[0029] The intervention strategy recommendation module retrieves matching intervention measures from a pre-built intervention strategy library based on the type of key nodes output by the key node identification module and their position in the transmission path, determines the timing and method of intervention, generates precise intervention strategies, and pushes them to on-site safety management personnel for execution.

[0030] After an accident (including near misses) occurs, the closed-loop feedback module automatically collects the accident feature vector and the severity of the accident consequences. It adopts an experience replay incremental learning method to integrate new samples into the historical sample set and update the parameters of the coupled gain quantization module and the risk transmission inference module, so that the system can continuously self-calibrate over time.

[0031] The six modules communicate with each other via a message bus. Data flows between the modules in a pipeline sequence of "identification → quantification → reasoning → positioning → intervention → feedback". At the same time, the output of the closed-loop feedback module is sent back to the coupling gain quantization module and the risk transmission reasoning module, forming a complete adaptive optimization loop.

[0032] like Figure 3 As shown, the coupling gain quantization module generates a comprehensive coupling risk score as follows: First, a similarity function is used to measure the degree of matching between the currently detected violation and typical patterns in the historical violation pattern library. Then, risk weights for individual violations and coupling gain coefficients between violation pairs are assigned respectively. The linear term and the coupling term are superimposed to obtain the total score. The calculation formula is as follows: in, The currently detected number The feature vector of each violation is 512-dimensional and is extracted from the output layer of the multimodal large model in the multimodal recognition module. For the first time in the historical database The typical feature vector of a violation pattern is obtained by taking the centroid (i.e. the mean vector) of the feature vectors of the same type of violation behavior in historical samples, and it is also 512-dimensional. For the first The risk weight for each violation pattern is determined based on the frequency of incidents caused by each violation type in historical data, obtained through frequency normalization. The calculation formula is as follows: ;in Indicates that a violation has occurred The probability of an accident occurring under certain conditions is obtained from historical statistics. All The sum is 1. In actual deployment, if there is insufficient historical data for a certain type of violation, a priori value can be given based on the experience of domain experts, and then gradually corrected as data accumulates; For similarity functions, similarity function Cosine similarity is used, and the calculation formula is: ,in This is the feature vector of the current detection behavior. This represents the feature vector of a sample in the violation sample library. The range of cosine similarity values ​​is... In the embodiments of the present invention, since the feature vectors are all extracted from the same multimodal large model, the feature space has good consistency, and the similarity usually falls within the range of [missing information]. Within the range.

[0033] For violations and The coupling gain coefficient between them.

[0034] For example, in the current scenario, two violations are detected: not wearing protective equipment ( ) and unauthorized hot work ( ),and , , , , (This indicates that the probability of an accident is significantly higher when both violations occur simultaneously than when they occur individually.) Therefore, the comprehensive coupled risk score is: The coupling term contributed 1.119, far exceeding the value of any linear term, indicating that the risk brought about by the combination of "not wearing protective equipment + illegal hot work" is far more than a simple superposition of two risks, but rather a nonlinear amplification effect of capturing the combination of violations.

[0035] In the embodiments of this application, the determination of the coupling gain coefficient The methods of determination include, Calculate the probability of an incident caused by a combination of violations based on Bayesian conditional probability: in, This represents the number of times this combination of violations and incidents has occurred simultaneously in historical records. The total number of times this violation combination occurred in the historical record (regardless of whether an accident occurred). These two statistics are based on the company's historical accident records, near miss reports, and daily safety inspection records. The Laplace smoothing coefficient ranges from 1 to 10 and is dynamically adjusted based on the amount of historical data. When the number of historical samples for a particular violation combination... hour, Take 1; when hour, Take 5; when hour, We set the value to 10. The purpose of this dynamic adjustment strategy is to reduce the impact of smoothing on estimation when data is plentiful, and to increase the smoothing force to prevent overfitting when data is sparse.

[0036] Coupling gain coefficient According to the violation and The increase in accident probability when coexisting with an individual is determined by the following formula: when At that time, it indicates a violation. and There is a positive coupling gain, meaning that the probability of an accident occurring when both violations occur simultaneously is higher than the product of their individual contributions, indicating a nonlinear amplification effect between the two. When this condition is met, it indicates that there is no coupling gain between the two, and the risks can be considered as independent superpositions. When the two are negatively coupled (mutually exclusive), it indicates that there is a negative coupling (mutual exclusion) between them. This is relatively rare in hazardous operation scenarios, but it may still occur in some special scenarios.

[0037] Let's continue with "not wearing protective equipment" (violation 1) and "unauthorized hot work" (violation 2) as examples. Assume the historical data is as follows: , , .but: A value much greater than 0 indicates that the combination of "not wearing protective equipment + unauthorized hot work" significantly increases the probability of an accident from the level under the independent assumption. Soaring to actual The coupling effect is extremely strong. This value is directly incorporated into the coupling term of the comprehensive coupling risk score R, enabling the system to generate alarm levels for high-risk violation combinations that are far higher than those for single-type violations.

[0038] In a real system, all violations (total) Yes, among them (total number of violation types) Pre-compute offline and store as a coupling gain matrix Once the system is online, the corresponding gain coefficient is directly retrieved from a table based on the detected illegal combinations in real time, eliminating the need for online calculations and ensuring real-time performance.

[0039] like Figure 7 As shown, this knowledge graph adopts a four-level hierarchical structure: The first layer is the job type layer. The nodes in this layer represent different types of hazardous jobs, such as hot work, confined space work, work at height, and temporary electrical work. Each node stores the basic attributes of the job type (such as job name, typical risk factors, and applicable regulations).

[0040] The second layer is the violation pattern layer. Nodes in this layer represent various violation patterns, such as not wearing protective equipment, supervisors leaving their posts, and insufficient safe distance. Each violation pattern node is connected to its possible job type nodes via directed edges, with the conditional probability of the violation occurring in the corresponding job type marked on the edges.

[0041] The third layer: the accident consequence layer. Nodes in this layer represent specific accident consequences that violations may cause, such as fires and explosions, poisoning and suffocation, falls from heights, and electric shocks. Each accident consequence node is connected to the violation pattern node that may cause that consequence via a directed edge.

[0042] The fourth layer is the severity level layer. The nodes in this layer represent the severity level of the accident consequences, which are divided into four levels: general, major, serious, and extremely serious. Each accident consequence node is connected to the corresponding severity level node through a directed edge.

[0043] The different layers are linked by a conditional probability table, where each entry in the conditional probability table... This represents the probability that the child node will occur given that the parent node has occurred. The calculation process for these probability values ​​is as follows: First, the co-occurrence frequency among nodes at each level is calculated from historical samples. Let the parent node be... The child node is The co-occurrence frequency is The frequency of the parent node is The initial probability estimate is then... .

[0044] Then, the conditional probability is optimized using maximum likelihood estimation, given a historical observation dataset. The log-likelihood function is: in, Let be all the parameters in the conditional probability table. Maximizing the log-likelihood function is equivalent to making the parameters in the conditional probability table best match the historical data. Without additional constraints, the solution of the maximum likelihood estimate is the frequency estimate itself. When the historical data is sparse, introducing the Dirichlet prior for Bayesian smoothing is equivalent to adding a prior pseudo-count to the frequency count.

[0045] The risk propagation inference process is as follows: Given a currently detected combination of violations, propagation starts from the corresponding node in the violation pattern layer and proceeds down the directed edges, calculating the joint probability of reaching each consequence node and severity level node. Only paths with a joint probability exceeding the propagation path screening threshold of 0.05 (i.e., 5%) are retained, forming a risk propagation path set. The selection of this threshold of 0.05 balances recall and precision: a threshold that is too high will miss low-probability but serious propagation paths, while a threshold that is too low will introduce a large number of noisy paths and increase the computational burden. In actual deployment, this can be adjusted according to the risk tolerance of the enterprise's management.

[0046] like Figure 8 As shown, interveneability is defined as the ratio of the average time window between the occurrence of a violation and the implementation of an intervention measure to the time required to implement the intervention measure, with a value ranging from 0 to 1. Let the average time window of a certain violation be... (i.e., the average time between a violation and the potential for an accident), the time required to implement intervention measures is Intervention is possible The calculation formula is: when hour, This means there's no time to intervene, and the situation at that point is unmanageable. Taking "guardian absenteeism" as an example, the average time window for detecting such absences... The time required to notify the guardian to return to their post is approximately 15 minutes. It takes about 3 minutes. The intervention is relatively high. However, for scenarios like "an instantaneous explosion caused by unauthorized hot work," the time window may only be a few seconds. It is close to zero and almost impossible to intervene in.

[0047] The blocking benefit is defined as the ratio of the expected accident loss that could be avoided by blocking the node to the expected total loss. The expected accident loss is the weighted average of the number of casualties and the amount of property damage after normalization. Let the expected accident loss corresponding to a certain node be... The expected total loss is Then block the profit The calculation formula is: Expected Losses from Accidents The calculation formula is: in The normalized number of casualties (divided by the historical maximum number of casualties, mapped to) ), The normalized amount of property loss (divided by the historical maximum loss amount, mapped to) ), and These are the weighting coefficients. , The weight given to personal injury or death is higher than that given to property damage, reflecting the principle of "life first".

[0048] The formula for calculating quantitative indicators is: Nodes with higher quantitative indicators are considered "effective in intervention" and are thus key nodes. The system sorts all nodes along the transmission path in descending order of their quantitative indicators and selects the top K (default). As a key node, it is output to the intervention strategy recommendation module.

[0049] In this embodiment, the intervention strategy recommendation module receives the key node list output by the key node identification module, retrieves matching intervention measures from the pre-built intervention strategy library for each key node, determines the intervention timing and intervention method, and generates a precise intervention strategy.

[0050] The intervention strategy library is a structured knowledge base. Each strategy record contains the following fields: applicable violation type, applicable work type, description of intervention measures, intervention method (immediate work stoppage / rectification within a time limit / on-site correction / equipment interlocking), intervention timing (immediate / within 5 minutes / within 30 minutes), description of expected effect, and vector encoding. The vector encoding is a 768-dimensional semantic vector encoded from the strategy text by a pre-trained language model (such as BERT), which is used for subsequent similarity retrieval.

[0051] The matching process employs a strategy of parallel execution in two ways and result fusion: Method 1: Vector-based similarity matching. The violation type and task type of the key node are concatenated into the query text, which is then encoded into a 768-dimensional query vector using the same pre-trained language model. The cosine similarity between the query vector and the vector encoding of each policy in the policy library is calculated. The similarity matching threshold is set to 0.85, returning only policy candidate sets with similarity exceeding 0.85. If there are more than 5 policies with similarity exceeding 0.85, the 5 with the highest similarity are selected; if there are fewer than 3, the threshold is lowered to 0.75 to supplement the search.

[0052] Method 2: Rule-based reasoning and matching. A set of reasoning rules is predefined, with the rule format "IF Violation Type = X AND Operation Type = Y AND Severity Level ≥ Z THEN Recommended Strategy = K". For example: "IF Violation Type = Unauthorized Hot Work AND Operation Type = Hot Work AND Severity Level ≥ Major THEN Recommended Strategy = Immediately stop hot work and activate the emergency fire-fighting plan". The rule base is compiled by safety experts based on national safety production regulations and internal enterprise safety operating procedures. The initial number of rules is approximately 200, and it can be expanded later.

[0053] The results from the two methods are combined and then sorted according to the following priorities: immediate shutdown strategies take precedence over time-limited rectification strategies; personnel safety-related strategies take precedence over equipment safety-related strategies; and strategies with higher similarity take precedence over those with lower similarity. The sorted list of strategies is the precise intervention strategy output by the system.

[0054] In this embodiment, the multimodal recognition module includes a behavior phase prediction unit, which is used to introduce the physiological state of the operator as an auxiliary feature for identifying violations.

[0055] The design is based on the fact that the fatigue level of the workers directly affects their probability of violating regulations. Workers in the burnout and fatigue period are more likely to be distracted, make operational errors and violate regulations. Multimodal large models cannot fully capture this factor with visual and textual information alone.

[0056] The workflow of the behavioral phase prediction unit includes four steps: signal acquisition, feature extraction, phase classification, and feature fusion.

[0057] For signal acquisition, wearable devices worn by workers (such as smart helmets or smart bracelets) collect heart rate variability (HRV) and electromyography (EMG) signals in real time. The HRV signal is sampled at a frequency of 250Hz, and the EMG signal is sampled at a frequency of 1000Hz. The signals are processed in a sliding window with a window size of 5 seconds and a step size of 2.5 seconds to achieve near real-time prediction.

[0058] For feature extraction, time-domain and frequency-domain features were extracted from the HRV signal for each time window. Time-domain features included: standard deviation of adjacent RR intervals (SDNN), root mean square of the difference between adjacent RR intervals (RMSSD), and the proportion of adjacent RR interval differences exceeding 50ms (pNN50). Frequency-domain features included: low-frequency power (LF, 0.04-0.15Hz), high-frequency power (HF, 0.15-0.40Hz), and the LF / HF ratio. A total of 6 HRV features were extracted. For the EMG signal, root mean square value (RMS), mean absolute value (MAV), zero-crossing rate (ZCR), and median frequency of the power spectral density (MDF) were extracted, resulting in a total of 4 EMG features. These two sets of features combined yielded a total of 10 original feature vectors.

[0059] In phase classification, a 10-dimensional feature vector is input into a pre-trained behavior phase classification model, which outputs the behavior phase state. The classification model adopts a three-layer fully connected network structure: the input layer has a dimension of 10 (6 HRV feature dimensions + 4 EMG feature dimensions), the hidden layer contains 64 neurons using the ReLU activation function, and the output layer contains 3 neurons corresponding to three phase states (alertness, lethargy, and fatigue). After Softmax normalization, the probability distribution of each phase is output, and the phase with the highest probability is taken as the current prediction result. The classification model is trained on a wearable device dataset labeled with behavior phases, with a training set size of no less than 5000 window samples. The cross-entropy loss function and Adam optimizer are used, and the learning rate is [not specified]. Batch size 64, training for 100 epochs.

[0060] In terms of feature fusion, the predicted behavioral phase state is encoded as a 3D one-hot vector: the alert period is... The burnout period is The fatigue period is The one-hot vector is concatenated with the visual feature vector (512-dimensional) output by the multimodal large-scale visual encoder in the channel dimension to obtain a 515-dimensional fused feature vector, which is then input into the cross-modal attention layer of the multimodal large-scale model. In this way, behavioral phase information is integrated as an auxiliary feature into the entire process of violation behavior recognition. This means that when the system detects that a worker is fatigued, even if the visual evidence is insufficient, it will give higher attention to potential violations and lower the alarm threshold.

[0061] In this embodiment, the multimodal recognition module further includes a generative augmentation unit to address the problem of insufficient training samples for less common violation scenarios. In actual hazardous work scenarios, samples of most violations (such as not wearing a safety helmet) are relatively abundant, but historical samples of some less common violation scenarios (such as the unauthorized use of non-explosion-proof tools in confined spaces) are extremely scarce, resulting in low recognition accuracy of the multimodal large model in these scenarios. The generative augmentation unit compensates for this data gap by diffusing generative models to synthesize realistic training samples.

[0062] The criterion for determining unpopular scenarios is: the number of samples of this violation type in the historical training set is less than the minimum sample size threshold. The minimum sample size threshold is set to 200 images, meaning that when the number of labeled samples for a certain violation scenario is less than 200, the generative augmentation process is triggered.

[0063] The specific steps of generative enhancement are as follows: Step 1: Count the number of training samples for each type of violation, and select less common violation types with fewer than 200 samples to form a set to be enhanced; Step 2: For each less common violation type, extract the scene layout description and violation behavior type label from existing real samples. The scene layout is extracted from a real and general sample library, while the violation type label points to less common violations and serves as input for controllable generation conditions. Step 3: Input the controllable generation conditions into the diffusion generation model to generate synthetic images of the specified scene and violation type. The number of synthetic images is the difference between the target sample size and the existing sample size, ensuring that the number of samples for each violation type reaches at least 200 after enhancement. However, the proportion of synthetic samples to the total training samples is capped at 30%. That is, if the number of synthetic samples exceeds 30% of the total training samples, the amount of synthetic samples will be truncated to the 30% cap.

[0064] Step 4: Quality screening of the generated synthetic images: A pre-trained image quality assessment model (such as NIQE, Natural Image Quality Evaluator) is used to calculate a no-reference quality score. Only synthetic samples with quality scores below the threshold of 8 (the lower the NIQE score, the better the quality) are retained. The filtered synthetic samples and their corresponding violation labels are added to the training set for incremental fine-tuning of the multimodal large model.

[0065] This process of "first screening for less common scenarios, then generating targeted samples, and finally filtering for quality" ensures the relevance and quality of the synthesized samples, and avoids low-quality samples contaminating the training set.

[0066] The generative enhancement unit employs either a stable diffusion model (Stable Diffusion) or a denoising diffusion probability model (DDPM). The choice between these two models is based on the following criteria: when the generation task emphasizes the semantic fidelity and diversity of the image, Stable Diffusion is chosen, as its latent space-based diffusion process achieves a good balance between computational efficiency and generation quality; when the generation task requires higher pixel-level detail fidelity, DDPM is chosen, as it directly performs diffusion and denoising in pixel space, resulting in more refined generation results but with higher computational overhead. In a typical deployment scenario of this invention, the resolution requirement for the violation scene image is... Stable Diffusion is recommended as the default model.

[0067] The controllable generation conditions adopt the ControlNet condition control mechanism. ControlNet is a controllable generation method that injects spatial condition constraints into a pre-trained diffusion model. Its core idea is to integrate condition signals into the generation process through a trainable "condition copy" network while freezing the weights of the original diffusion model.

[0068] In this invention, the conditional input of ControlNet includes two items: One is the scene layout condition, which extracts the scene's depth map or edge map (Canny Edge Map) from existing real samples as spatial layout constraints. The depth map is estimated from real images using a pre-trained depth estimation model (such as MiDaS), and the edge map is extracted using the Canny edge detection operator. The scene layout condition ensures that the spatial structure of the generated image is consistent with the real working scene—such as the pipeline layout of a tank area or narrow passages in a confined space.

[0069] Secondly, there are violation type labels, which encode violation types into text prompts, such as "worker not wearing safety helmet in chemical plant" or "unauthorized hot work near flammable storage". After being encoded by the CLIP text encoder, the text prompts are injected into the denoising network of the diffusion model through a cross-attention mechanism.

[0070] ControlNet is trained by fine-tuning its conditional copy network using real-world samples from less common violation scenarios (in limited quantities), while freezing the weights of the main network in Stable Diffusion. The learning rate during fine-tuning is set to... The training steps do not exceed 5000. Because the number of parameters in the conditional copy network is much smaller than that of the main network (about 15% of the main network), it can achieve good conditional control even with a small number of training samples.

[0071] During the generation process, the diffusion steps were set to 50 (DDIM sampling), and the classifier free guided scale (CFG Scale) was set to 7.5 to achieve a balance between generation diversity and condition compliance.

[0072] In this embodiment, the closed-loop feedback module employs an experience-based replay incremental learning approach, adding the feature vectors and severity of consequences after the accident as new samples to the historical sample set, and incrementally fine-tuning the parameters of the coupled gain quantization module and the risk transmission inference module. The risk weight in the coupling gain quantization module and coupling gain coefficient The conditional probability table in the risk transmission inference module is updated based on the frequency statistics of the new samples after recalculation using the new samples.

[0073] The closed-loop feedback module adopts an incremental learning approach based on experience replay, and the specific process is as follows: Step 1: Accident Information Collection. After an accident occurs, the system automatically extracts the accident feature vector (including features of the combination of violations at the time of the accident, work type features, environmental condition features, etc., with dimensions aligned with the output of the multimodal recognition module), and records the severity of the accident consequences. The severity of the accident consequences is represented by a weighted composite value, the same as the blocking benefit. ,in and These are the normalized numbers of casualties and the amount of property damage, respectively.

[0074] Step 2: Experience Replay Sample Construction. New samples are created by combining accident feature vectors with the severity of consequences. Old samples (i.e., past accident records) are randomly selected from the historical sample set to construct the experience replay batch. The replay sample ratio is new samples:old samples = 1:3, and the number of old samples selected is 50-100. The purpose of introducing old samples is to prevent catastrophic forgetting during incremental learning—if only new samples are used to update the model, the model may "forget" previously learned knowledge. The selection of old samples adopts a feature similarity-based priority sampling: priority is given to old samples with a high cosine similarity to the current accident feature vector, ensuring that the replay content is relevant to the current learning task.

[0075] Step 3: Parameter update. The parameter update is triggered by either accumulating 5 incidents or once per quarter, whichever comes first. After triggering, the following parameter update operations are performed: (1) Parameter update of the coupled gain quantization module, risk weight The calculation is recalculated based on the newly added samples, still using the frequency normalization formula, but the probability statistics in the denominator are expanded to include the entire historical record containing the new samples. Coupling gain coefficient. Similarly, based on the expanded historical records, the co-occurrence frequency of violation combinations and incidents is recalculated, and the Bayesian conditional probability and gain coefficient are recalculated. Updated Store the coupling gain matrix Replace the old value.

[0076] (2) The parameters of the risk transmission reasoning module are updated, and the conditional probability table in the knowledge graph is updated based on the frequency statistics of the new samples: the co-occurrence relationships of nodes observed in the new samples are accumulated into the frequency count, and the conditional probabilities are recalculated. Then, perform maximum likelihood estimation optimization again. If a violation-consequence association that does not exist in the knowledge graph appears in the new sample (e.g., a new type of incident caused by a violation combination that has never been recorded before), then add the corresponding node and directed edge to the knowledge graph and initialize the conditional probability value.

[0077] Step 4: Effectiveness Verification. After the parameters are updated, evaluate the key indicators before and after the update (including the accuracy of violation identification, the actual correlation coefficient between risk score and accident, and the effectiveness of intervention strategies) on the retained validation set. If any indicator decreases by more than 5% after the update, roll back the update and temporarily store the new sample in the review queue. The safety management personnel will then manually review the sample and decide whether to adopt it.

[0078] The above closed-loop feedback mechanism ensures that the system can continuously learn from actual accidents, while avoiding the risk of becoming worse with each update through two lines of defense: experience replay and effect verification.

[0079] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A safety monitoring system for hazardous workplaces based on a large model, characterized in that, include: The multimodal recognition module is used to identify a set of violations in hazardous work sites by fusing monitoring video frames, work order text data and gas sensor data through a multimodal large model with vision-language joint understanding capabilities. The violations include at least failure to wear protective equipment, unauthorized hot work, absence of supervisors from their posts and insufficient safe distance. The coupling gain quantization module is used to mine combination patterns of violations from historical violation records using association rule mining algorithms, and introduces a coupling gain coefficient to quantify the nonlinear coupling gain between violations, generating a comprehensive coupling risk score. The risk transmission reasoning module is used to infer the risk transmission path that the current combination of violations may trigger based on a pre-built knowledge graph of the relationship between violation patterns and accident consequences. The key node identification module is used to calculate the quantitative indicators of each node in the risk transmission path and identify the key nodes in the transmission chain. The quantitative indicators are the product of the operability and the blocking benefit. The intervention strategy recommendation module is used to match and recommend precise intervention strategies from a pre-built intervention strategy library based on the type and location of key nodes. The closed-loop feedback module is used to obtain the accident feature vector and the severity of the accident consequences after the accident occurs, and to update the system parameters using an incremental learning method. Communication connections between various modules.

2. The safety monitoring system for hazardous workplaces based on a large model according to claim 1, characterized in that: The coupling gain quantization module generates the comprehensive coupling risk score in the following ways: Define integrated coupling risk score The calculation formula is: in, The currently detected number The feature vector of a violation, For the first time in the historical database Typical feature vectors of violation patterns For the first Risk weighting for each violation pattern The similarity function is... Cosine similarity is used, and the calculation formula is as follows: ,in This is the feature vector of the current detection behavior. The feature vector of the sample in the violation sample library; For violations and The coupling gain coefficient between them.

3. The safety monitoring system for hazardous workplaces based on a large model according to claim 2, characterized in that: Determination of the coupling gain coefficient The methods of determination include, Calculate the probability of an incident caused by a combination of violations based on Bayesian conditional probability: in, and This data was compiled based on the company's historical accident records, near miss reports, and daily safety inspection records. is the Laplace smoothing coefficient, with a value ranging from 1 to 10, and is dynamically adjusted according to the amount of historical data. The coupling gain coefficient According to the violation and The increase in accident probability when coexisting with an individual is determined by the following formula: when At that time, it indicates a violation. and There is positive coupling gain between them.

4. A safety monitoring system for hazardous workplaces based on a large model according to claim 3, characterized in that: The knowledge graph of violation patterns and accident consequences in the risk transmission reasoning module includes a four-layer structure: operation type layer, violation pattern layer, accident consequence layer, and severity level layer. Each layer is associated with another layer through a conditional probability table. The probability values ​​in the conditional probability table are calculated based on the co-occurrence frequency between nodes in each layer in historical samples and are obtained through maximum likelihood estimation optimization.

5. A safety monitoring system for hazardous workplaces based on a large model according to claim 4, characterized in that: In the key node identification module: Interventionability is defined as the ratio of the average time window between the occurrence of a violation and the implementation of an intervention measure to the time required to implement the intervention measure, with a value ranging from 0 to 1. The blocking benefit is the ratio of the expected accident loss that can be avoided by blocking the node to the expected total loss. The expected accident loss is the weighted average of the number of casualties and the amount of property damage after normalization.

6. A safety monitoring system for hazardous workplaces based on a large model according to claim 5, characterized in that: The intervention strategy recommendation module, based on the location of key nodes in the risk transmission path and their corresponding violation patterns, uses vector-based similarity matching or rule-based reasoning matching to retrieve corresponding intervention measures from a pre-built intervention strategy library, determine the timing and method of intervention, and generate precise intervention strategies.

7. A safety monitoring system for hazardous workplaces based on a large model according to claim 1, characterized in that: The multimodal recognition module also includes a behavior phase prediction unit, which is used to identify the behavior phase state based on the heart rate variability and electromyography signals collected in real time by the wearable device worn by the worker. The behavior phase state includes the alert period, the fatigue period and the tired period. The behavior phase state is encoded into a one-hot vector, concatenated with the visual feature vector and input into the multimodal large model as an auxiliary feature for the identification of violations.

8. A safety monitoring system for hazardous workplaces based on a large model according to claim 1, characterized in that: The multimodal recognition module also includes a generative enhancement unit, which is used to expand the training samples by using a diffusion generation model and combining controllable generation conditions for less common violation scenarios, so as to increase the number of samples for less common scenarios.

9. A safety monitoring system for hazardous workplaces based on a large model according to claim 8, characterized in that: The diffusion generation model includes a stable diffusion model or a denoised diffusion probability model, and the controllable generation conditions include ControlNet condition control, which is used to control the scene layout and violation type of the generated samples.

10. A safety monitoring system for hazardous workplaces based on a large model according to claim 1, characterized in that: The closed-loop feedback module employs an empirical replay incremental learning approach, adding the feature vectors and severity of consequences following the accident as new samples to the historical sample set, and incrementally fine-tuning the parameters of the coupled gain quantization module and the risk transmission inference module. The risk weight in the coupling gain quantization module and coupling gain coefficient The conditional probability table in the risk transmission inference module is updated based on the frequency statistics of the new samples after recalculation based on the new samples.

Images