Industrial safety behavior monitoring method based on vision and language model fusion

Abstract

The invention discloses an industrial safety behavior monitoring method based on vision and language model fusion, and belongs to the technical field of industrial safety monitoring and artificial intelligence. According to the method, chemical plant field personnel and protective equipment thereof are detected through a visual model, and structured semantic description is generated; image features and text features are extracted, and joint coding is carried out through a multi-modal fusion module; semantic reasoning and risk assessment are carried out in combination with the large language model and the safety knowledge base, violation risk values are calculated, and a safety event report is automatically generated. Meanwhile, cross-modal feature fusion and loss function optimization are used, accurate recognition of protection behaviors in a complex environment can be effectively met, the method can be used for personnel safety protection detection of chemical plants and similar industrial scenes, intelligent analysis from visual detection to semantic understanding is achieved, and automation and accuracy of safety supervision are improved.

Description

Technical Field

[0001] This invention belongs to the field of industrial safety monitoring and artificial intelligence technology, and more specifically, relates to an industrial safety behavior monitoring method based on the fusion of visual and language models. Background Technology

[0002] In high-risk industries such as chemical plants, mines, and power plants, safety management requirements are extremely stringent. Whether workers wear safety helmets and protective equipment correctly and in accordance with regulations directly affects their lives and the safety of the enterprise's production. In recent years, computer vision-based target detection methods (such as the YOLO series models) have been widely used in scenarios such as safety helmet detection and personnel identification, realizing automated identification of protective equipment wearing status and providing a certain degree of technical assistance to traditional manual inspections.

[0003] However, with the increasing demand for intelligent safety management, the limitations of existing technological systems in terms of practicality and intelligence are becoming increasingly apparent. Most current methods remain at the level of single-target recognition, lacking the ability to deeply understand the semantics of complex scenarios. Specifically, the semantic understanding capabilities of the systems are insufficient; existing methods can only output basic detection results such as "whether a safety helmet is worn," but cannot further determine advanced semantic information such as "whether a worker has entered a dangerous restricted area," "the correlation between multiple violations," or "the degree of violation." They remain at the perception level of single-target recognition, failing to achieve a comprehensive understanding of the safety scenario. Furthermore, due to the lack of support from natural language processing and logical reasoning capabilities, existing models struggle to link visual recognition results with specific corporate safety regulations, cannot automatically generate explanatory violation analysis reports, and even suffer from black-box judgment and hallucination misjudgments, resulting in insufficient basis for safety management decisions.

[0004] A search revealed several publicly available patents related to the combination of machine vision and semantic networks. For example, Chinese Patent Publication No. CN115410335A, published on November 29, 2022, is titled: "A Construction Engineering Safety Monitoring and Early Warning System Based on Machine Vision and Semantic Networks." This system includes: a semantic network construction module for building a construction engineering safety knowledge semantic network based on construction engineering safety documents; a data acquisition module for collecting construction site and work image data and environmental data; a feature recognition module for automatically identifying safety-related visual features from construction site images based on machine vision and deep learning algorithms; an analysis and reasoning module for analyzing and reasoning about hidden safety accident risks at the site based on several visual features and the construction engineering safety knowledge semantic network; and an early warning push module for locating and pushing the information of high-risk work points and personnel to relevant responsible persons based on the results of the on-site safety analysis and reasoning.

[0005] The aforementioned patent uses machine vision to identify visual features at construction sites and combines semantic network analysis to infer safety risks, thereby achieving safety monitoring and early warning at construction sites. However, its solution is mainly used at construction sites, where the environment is relatively open. When facing complex industrial environments, the above solution can only achieve a simple association between visual features and semantic networks, without cross-modal feature fusion and loss function optimization, making it difficult to meet the requirements for accurate identification of protective behaviors in complex environments. Summary of the Invention

[0006] The purpose of this invention is to address the lack of semantic understanding or weak semantic interpretation capabilities in existing technologies for visual inspection of industrial safety behaviors. It provides a method for monitoring industrial safety behaviors based on the fusion of visual and language models, particularly for safety protection inspections of personnel in chemical plants. This invention utilizes cross-modal fusion of visual and language models to achieve semantic alignment and reasoning between visual features and natural language. Simultaneously, it defines a joint loss function of detection loss + semantic consistency loss, effectively ensuring the consistency between visual inspection and the semantics of safety regulations. Furthermore, it can automatically generate interpretable reports containing rectification suggestions, thereby achieving automated safety supervision.

[0007] To address the above problems, the technical solution proposed by this invention is as follows: The present invention provides an industrial safety behavior monitoring method based on the fusion of visual and language models, comprising the following steps: Step S1: Acquire real-time video streams from the industrial site and extract frames from the images to obtain the original images as the input data source. After preprocessing, a dataset is formed. The improved YOLOv8 model is used to perform visual inspection on the dataset. The improved YOLOv8 model is to embed a coordinate attention mechanism into the backbone network of the original YOLOv8 model. Step S2: Perform structured integration of the visual detection results from Step S1 and generate semantic description text. T ; Step S3: Extract features from the original image obtained in Step S1 and the semantic description text obtained in Step S2 to obtain visual feature vectors. V With text feature vectors L Visual feature vectors V With text feature vectors L Multimodal fusion is performed to obtain multimodal features. F ; Step S4: Combine the multimodal features fused in step S3. F The input is fed into the language large model reasoning module, which then utilizes the language large model reasoning module in conjunction with the security rule knowledge base. K To coordinate and calculate the risk value of violations. R ; Step S5: Define and calculate the joint loss function to optimize the model, and automatically output a decision report. The joint loss function is the detection loss. L det With semantic loss L sem The weighted sum.

[0008] As one possible implementation of the present invention, in step S1, the acquired video stream or image is decoded and time-synchronized, key frames are extracted according to a set frame rate, and the extracted key frames are preprocessed. The image preprocessing includes noise reduction, brightness correction, motion compensation, and resolution adaptive adjustment.

[0009] In one possible implementation of the present invention, in step S2, the visual detection results are structured and integrated, and then output in the form of a structured set:

[0010] in, D A collection of detection results containing multiple detection targets; For the first Coordinates of each detection box For category labels and tooling labels, To determine the confidence level, N represents the total number of targets detected in the current image frame. i It is the target index, from 1 to N.

[0011] As one possible implementation of the present invention, in step S2, the scene environment metadata is combined. M Semantic description text generated using a large language model T as follows: .

[0012] In one possible implementation of the present invention, in step S3, multimodal features are calculated. F The methods include: (1) Calculate the dynamic gating vector ;

[0013] in,[ V , L The symbol ] represents the concatenation of extracted visual features and text features. , These are the parameters of the fully connected layer of the gated network. Use the Sigmoid activation function; The value changes dynamically between [0, 1]. (2) Generate fused multimodal features F ;

[0014] Here, ⊙ represents element-wise multiplication.

[0015] As one possible implementation of the present invention, in step S4, the calculated violation risk value R as follows:

[0016] in, Indicates the risk value of violation. For the Sigmoid function, These are trainable inference parameters. This is the regular attention vector.

[0017] As one possible implementation of the present invention, based on the calculated... R To set a threshold for comprehensively judging the level of violation, the details are as follows: when R If the value is less than 0.3, then it is not in violation of the rules; When 0.3≤ R A value less than 0.7 indicates a minor violation. when R A value of ≥0.7 indicates a serious violation.

[0018] As one possible implementation of the present invention, in step S5, the joint loss function L The details are as follows:

[0019] in, L det To detect loss, it is used to measure the deviation between the detection result and the true label; L sem Semantic loss is used to constrain the consistency between the model's language reasoning and the knowledge base rules. This is the balance coefficient.

[0020] As one possible implementation of the present invention, in step S5, the detection loss is calculated using binary cross-entropy. L det :

[0021] in, To predict the output, For real labels, These are the sample weights.

[0022] As one possible implementation of the present invention, in step S5, semantic loss L sem The calculation formula is as follows:

[0023] in, The semantic score for language reasoning outputs matching with the knowledge base, with a value range of [0,1]. R This indicates the risk value for violations.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The method of the present invention breaks through the technical limitation that traditional pure visual detection can only identify a single target. Through the deep integration of visual and language models, it constructs an integrated framework of "visual detection-semantic description-rule reasoning" for the first time. It can not only accurately identify the wearing status of protective equipment, but also realize advanced semantic reasoning such as "whether personnel have entered dangerous restricted areas", "association of multiple violations", and "grading of violation degree", such as judging entry into dangerous areas, association of multiple violations, and grading of violation degree, providing sufficient basis for safety decision-making and promoting the transformation of safety management from "passive response" to "proactive early warning".

[0025] (2) This invention uses the improved YOLOv8-Attention model to effectively solve the problem of detecting small targets and occluded targets in chemical plants through the coordinate attention mechanism. At the same time, it creatively proposes an adaptive cross-modal gating fusion mechanism, which can greatly improve the robustness of the model in complex environments such as low light, smoke, and reflection, and significantly reduce the false alarm rate.

[0026] (3) By forcibly binding the fusion features with the security knowledge base rule vector, this invention ensures that the risk assessment strictly follows the enterprise security specifications and completely solves the core pain points of the existing technology of “black box judgment”, “illusion false alarm” and inability to associate with security rules. It provides interpretable and evidence-based support for security management decisions, and the automatically generated structured report contains complete violation information and rectification suggestions, which improves the traceability and operability of security management.

[0027] (4) The method of this invention forms a complete closed loop of "visual detection - semantic description - multimodal fusion - rule reasoning - report generation - early warning linkage", which supports continuous identification of violations across perspectives and time periods. A single GPU can process 12 channels of 1080p video simultaneously, with an average frame latency of less than 280ms, meeting the real-time monitoring needs of large-scale industrial scenarios. Moreover, the model of this invention has an accuracy rate of 92.8% for the "not wearing a safety helmet" scenario and 91.6% for the "not wearing work clothes" scenario. The overall average risk assessment error is less than 1.2%, and the F1-score is 0.94, which can provide an efficient and reliable intelligent safety protection and monitoring solution for high-risk industries such as chemical industry. Attached Figure Description

[0028] Figure 1This is an overall flowchart of an industrial safety behavior monitoring method based on the fusion of visual and language models according to the present invention. Figure 2 This is a diagram of the backbone network structure after embedding the coordinate attention mechanism in the visual detection model (YOLOv8-Attention) of this invention; Figure 3 This is a schematic diagram of the process of visual and linguistic feature extraction and fusion in this invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0030] With the increasing demand for intelligent safety management, current vision-based detection methods have not yet achieved the technological leap from perception to cognition. There is an urgent need to build a new intelligent analysis method that can integrate visual recognition and language reasoning capabilities. By organically combining the detection capabilities of computer vision with the reasoning capabilities of natural language processing, it is possible not only to improve the accuracy of violation identification, but also to achieve a deep understanding of enterprise safety regulations and explainable decision-making. Ultimately, this will promote the transformation of industrial safety management from passive response to proactive early warning, and from single detection to the coordinated development of semantic understanding and decision-making.

[0031] To achieve the above objectives, combined with Figure 1 The present invention provides an industrial safety behavior monitoring method based on the fusion of visual and language models, which specifically includes the following steps: Step S1: Input and visual inspection of video data; First, real-time video streams or images are collected from industrial sites, such as chemical plants, using industrial cameras or fixed monitoring equipment as the input data source. Then, the acquired multi-channel video signals or images are decoded, frame rate adjusted, and time synchronized to ensure the temporal consistency of the data. Then, the acquired video frames are preprocessed, including noise reduction, brightness correction, motion compensation, and resolution adaptive adjustment, to improve the detection accuracy under low light, smoke, or complex background conditions.

[0032] Subsequently, the preprocessed image frames are input into the improved visual detection model (YOLOv8-Attention) for further processing. It should be noted that since the methods of this invention are mostly used in industrial scenarios, such as chemical plants, the traditional YOLO model has significant limitations. On the one hand, the long deployment distance of cameras results in small targets of workers and protective equipment, which are easily obstructed by equipment, creating blind spots. On the other hand, complex environments such as low light, smoke, and reflections can cause visual feature distortion, affecting detection accuracy. To address the complex detection environment of chemical plants, this invention proposes embedding a coordinate attention mechanism into the backbone network of the original YOLOv8 model. The improved model backbone network structure is shown in the figure below. Figure 2 As shown, by adding a Coordinate Attention (CA) module after each C2F module in the backbone of the original YOLOv8 model, the model's ability to locate key features is enhanced, thereby accurately capturing detection targets and automatically identifying categories such as workers, head protective equipment (such as safety helmets), and work clothes in the image. The detection results are output in the form of a structured set.

[0033] in, D A collection of detection results containing multiple detection targets; For the first Coordinates of each detection box For category labels and tooling labels, To determine the confidence level, N represents the total number of targets detected in the current image frame. i It is the target index, from 1 to N.

[0034] Through the above steps, real-time identification of on-site personnel and their protective status was achieved, providing a visual input basis for subsequent semantic analysis and risk assessment.

[0035] Step S2: Semantic description construction; After completing the visual detection in step S1, the output detection result set is semantically processed to construct the semantic description text T.

[0036] First, by performing time-series generation and spatial calibration of the detection box coordinates, category labels, and detection confidence, structured data is formed. Then, by combining scene environment metadata M (such as workshop number, camera number, shift time, and geographic coordinates), target identity, location, and environment matching are achieved, generating semantic context information. Then, using an algorithm combining template generation and language model generation, the structured information is transformed into readable semantic description text, providing a unified data interface for subsequent violation identification and voice broadcasting. T as follows:

[0037] Generated semantic description text T It not only includes the detected object and its status (e.g., "Employee A is not wearing a safety helmet"), but also embeds time and space information. This module uses a lightweight language generation model to ensure the accuracy and naturalness of the description, providing a text benchmark for multimodal fusion.

[0038] For example, if the detection result indicates that "Personnel A was not wearing a safety helmet in Workshop 3," the corresponding semantic sentence is automatically generated: "At 10:32 AM on October 21, 2025, Workshop 3 detected that Personnel A violated the rule of not wearing a safety helmet." Through this fusion of structured and linguistic methods, semantic description text is generated. T It is not only readable, but also provides a unified data interface for subsequent modules such as violation type identification, behavior statistics and voice broadcasting.

[0039] Step S3: Visual and linguistic feature extraction and fusion; First, the video images preprocessed in step S1 and the semantic description text generated in step S2 are compared. T Extract the feature vectors separately to obtain the visual feature vectors. V With text feature vectors L Specifically:

[0040] in, x For preprocessing images, For visual encoders (CLIP or ViT). This is a language encoder (such as BERT). After the semantic description text is processed by the text encoder to extract language features, the two types of features are aligned in a shared multimodal space to ensure that the visual content and the language description are semantically consistent, providing a basis for subsequent similarity calculation and violation matching.

[0041] It should be noted that during the experiment, the applicant discovered that using fixed coefficients in the testing resulted in all extracted visual features being incorrect if the acquired images contained smoke or reflections, yet the model continued to use them, leading to false positives. To address the problem that traditional linear weighting cannot adapt to complex and changing scenarios, this invention abandons fixed weights and designs an adaptive cross-modal gating fusion mechanism. This mechanism allows the self-learning gating network to automatically evaluate the importance of visual and textual features in the current context. The specific process can be found in [reference needed]. Figure 3 , specifically including: (1) Calculate the dynamic gating vector :

[0042] Among them, V , L represents the concatenation of the extracted visual features and text features, , are the parameters of the fully connected layer of the gating network, is the Sigmoid activation function; The value of changes dynamically between [0, 1], which is completely automatically learned by the model according to the feature distribution of the current input data, rather than being artificially fixed.

[0043] (2) Generate the fused multi-modal features F :

[0044] Among them, ⊙ represents element-wise multiplication (Hadamard Product).

[0045] Through the above mechanism, when the visual environment is harsh (such as smoke occlusion), the model will automatically increase the weight (1 - g) of the text features, and vice versa, it depends on the visual features, with stronger robustness.

[0046] Step S4, semantic reasoning and risk assessment; Input the fused multi-modal features F in Step S3 into the language large model reasoning module, and use the language large model reasoning module to interact with the safety rule knowledge base K At the same time, in order to make the reasoning process interpretable and strictly follow the chemical safety specifications, the present invention designs a violation risk value based on rule constraints:

[0047] Among them, represents the violation risk value, is the Sigmoid function, are the trainable reasoning parameters; is the rule attention vector, which is obtained by using the multi-modal features F in the knowledge base KThe embedded representation is obtained by retrieving and aggregating relevant security entries. This invention does not directly use a large language model, as this can create illusions and easily classify non-violation entries as violations without sufficient evidence. Therefore, instead of directly outputting the data, it forcibly binds the fused multimodal features and rule vectors from the knowledge base together for calculation. This design differs from traditional black-box classifiers; it forces the model to calculate violation risk values. R At this time, regular vectors must be fused. This ensures that risk assessment is based not only on "whether it resembles a violation" (visual) but also on "whether a specific clause is violated" (rules), thereby significantly reducing the false alarm rate. This invention combines the reasoning capabilities of a language model with the rule constraints of a knowledge base to achieve intelligent security analysis from the perception layer to the cognitive layer.

[0048] Specifically, based on calculations R To set a threshold for comprehensively judging the level of violation, the details are as follows: when R If the value is less than 0.3, then it is not in violation of the rules; When 0.3≤ R A value less than 0.7 indicates a minor violation. when R A value of ≥0.7 indicates a serious violation.

[0049] Step S5: Optimize the loss function and model, and automatically generate reports and output alarms; To achieve synergistic optimization of visual perception and semantic cognition, this invention defines a multi-task joint loss function. L The details are as follows:

[0050] in, L det To detect loss, it is used to measure the deviation between the detection result and the true label; L sem Semantic loss is used to constrain the consistency between the model's language reasoning and the knowledge base rules. As the balance coefficient, through By coupling detection loss and semantic loss, and through joint optimization, convergence can be achieved simultaneously in both visual detection and semantic reasoning, thereby improving the accuracy of violation identification and semantic interpretability. Furthermore, this invention determines the appropriate validation set through network search. The optimal value is 0.5.

[0051] The detection loss of the present invention L det Using binary cross-entropy:

[0052] in, To predict the output, For real labels, The sample weights are used to balance the ratio of positive to negative samples during training, thereby improving the detection sensitivity of minority classes (such as rare violations) and thus enhancing the overall detection performance and generalization ability.

[0053] Semantic loss L sem The calculation formula is as follows:

[0054] in, The semantic score for matching the language reasoning output with the knowledge base is calculated, with a value ranging from [0,1], and is used to evaluate the correctness and reasonableness of the reasoning. R This indicates the risk value for violations.

[0055] This invention achieves joint optimization L det and L sem The model converges simultaneously in both visual detection and semantic reasoning, improving the overall accuracy and stability of the system.

[0056] Final output report content as follows:

[0057] in, For semantic description text, This represents the risk value for violations. The risk level is represented by the "Suggestion," which is an automatically generated corrective action recommendation. The output report includes the violation type, time, location, personnel number, risk level, and corrective action recommendations. For example, it can automatically generate action suggestions based on the violation category, such as "Immediately remind personnel to wear safety helmets" or "Suspend operations and report to the safety administrator." Furthermore, the system can be integrated with audible and visual alarms, SMS notifications, or management platforms to achieve real-time early warning and automated safety management. This mechanism realizes a closed-loop safety monitoring process from detection to reasoning to decision-making to feedback, significantly improving the intelligence and traceability of personnel safety protection in chemical plants.

[0058] The method of the present invention will be described below with reference to specific implementation examples.

[0059] Example 1 like Figure 1 As shown in the figure, an industrial safety behavior monitoring method based on the fusion of vision and language models in this embodiment includes the following steps: Step S1: Video input and visual inspection; Fixed high-definition industrial cameras (1920×1080 resolution, 30fps) are deployed in chemical plant production areas, tank areas, loading and unloading areas and control rooms, and video data is transmitted to the analysis server in real time via industrial Ethernet. The video input module performs frame sampling (5 frames per second) and temporal synchronization on the video stream, and performs preprocessing, including Gaussian denoising, histogram equalization, and Gamma correction, to adapt to the complex optical environment commonly found in chemical plants, such as low light, smoke, and reflections, forming a dataset containing approximately 5,000 images covering three target categories: "personnel," "safety helmets," and "work clothes." The data is divided into three parts: 80% training set, 10% validation set, and 10% test set.

[0060] By embedding a coordinate attention mechanism into the YOLOv8 backbone network, the challenge of detecting small targets and occlusions in chemical plant scenarios is addressed. The improved visual detection model (YOLOv8-Attention) uses pre-processed video frame input and is trained on a local safety dataset within the chemical plant before outputting a set of detection results. D .

[0061] Step S2: Set the detection results output in step S1. D It is converted into natural language semantic description text so that language models can understand and reason about it.

[0062] This step will aggregate the detection results. D With scene metadata M This includes workshop number, camera ID, timestamp, shift number, and geographic coordinates. The generated semantic text is dynamically generated using a template combined with context. In this embodiment, confidence level was specifically detected. When the target number of personnel with a safety helmet value greater than 0.8 is not wearing a safety helmet, the following message will be automatically generated: "In the ethylene reaction zone C3, at 10:42 on October 21, 2025, employee A was detected not wearing a safety helmet." The language template in this embodiment adopts a filler NLP structure, combined with the BERT model for syntactic correction and entity recognition, to ensure that the text expression is standardized and the information is complete. The text data is then input into the language model for semantic encoding.

[0063] Step S3: Visual and linguistic feature extraction and fusion.

[0064] like Figure 3 The gating fusion mechanism shown utilizes a pre-trained multimodal coding network to extract visual feature vectors. V With text feature vectors L Visual feature vectors V Text feature vectors are extracted using the visual encoder CLIP.L The language model encoder BERT-base is used to extract and output a 512-dimensional embedding vector of the same dimension. To achieve semantic alignment between vision and language, an adaptive cross-modal gating fusion mechanism is adopted to generate fused multimodal features. F ; Step S4: Semantic Reasoning and Risk Assessment.

[0065] The multimodal features in step S3 F Input language large model, and security knowledge base Combined for identifying violations and quantifying risks; Knowledge Base K contains 83 chemical safety regulations, 56 typical violation examples, and 17 protection standards, covering three major categories: personal protective equipment, equipment operation, and work permits.

[0066] In this embodiment, the validation set test shows that the model of the present invention has an accuracy rate of 92.8% in recognizing the "not wearing a safety helmet" scenario and 91.6% in recognizing the "not wearing work clothes" scenario, with an overall average risk assessment error of less than 1.2%.

[0067] Step S5: Loss function and model optimization.

[0068] During model training, a multi-task joint loss function is defined. L To detect loss L det With semantic loss L sem The weighted sum, in this embodiment Set the balance coefficient to 0.5.

[0069] In this embodiment, the detection loss is calculated using the binary cross-entropy method. L det The sample weights are dynamically adjusted according to the category distribution (for example, the proportion of safety helmet violations is about 12%, so they are given a weight of 1.5 times) to reduce the training bias caused by category imbalance.

[0070] Semantic loss L sem In This refers to the semantic score, calculated by the semantic similarity model (Sentence-BERT); when the model's inference results closely match the safety rules, Automatic reduction encourages logical consistency and interpretability in model output. This is achieved by minimizing the total loss function. It can simultaneously optimize the accuracy of visual detection and semantic reasoning; after 200 training rounds, the overall loss drops to 0.07, and the model achieves an F1-score of 0.94 on the validation set.

[0071] Step S6: Automatic report generation and alarm output.

[0072] When the risk value of violation R When the threshold value reaches >τ (specifically set to 0.65 in this embodiment), the system automatically generates a security event report and triggers an alarm; the report content is as follows: I. Overview of the Violation (1) Type of violation: Not wearing a safety helmet (2) Time of occurrence: 10:42:15 on October 21, 2025 (3) Location of occurrence: Ethylene reaction zone C3 (Camera number: CAM-03) (4) Personnel involved: Employee A (Employee ID: GY-2107) (5) Risk level: serious violation (R = 0.82) II. Basis for Detection and Reasoning (1) Visual inspection results: 1) Personnel target detected (confidence level 0.92) 2) No helmet target detected (confidence level < 0.05) 3) The work clothes passed inspection (confidence level 0.88). (2) Semantic description text: "In the ethylene reaction zone C3, at 10:42 on October 21, 2025, it was detected that employee A was not wearing a safety helmet." (3) Rule matching status: 1) Match the safety knowledge base entry: KB-03-001 (Regulations on Mandatory Wearing of Safety Helmets in Chemical Industrial Parks) 2) Semantic matching score

[0073] 3) Calculation of violation risk:

[0074] III. Rectification Recommendations (1) Recommendations for immediate action: The employee was instructed to put on a safety helmet immediately via the on-site broadcast system.

[0075] If the issue is not rectified within 10 minutes, the system will automatically notify the on-duty safety officer.

[0076] (2) Long-term recommendations: It is recommended to add a helmet dispensing point in area C3.

[0077] It is recommended that the employee receive refresher training on safety procedures.

[0078] IV. System Response Log (1) Alarm method: sound and light alarm (already triggered), SMS notification (already sent to safety officer) (2) Processing status: Pending confirmation (3) Recorder: Intelligent Safety Monitoring System (Model Version: V2.1) In addition, the method of the present invention also supports audible and visual alarms, SMS reminders and cloud recording functions. In the actual test environment of a chemical plant, the system can process video streams in real time with an average frame latency of less than 280ms. A single GPU (NVIDIA RTX4090) can process 12 channels of 1080p video simultaneously, meeting the needs of real-time safety supervision.

[0079] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for monitoring industrial safety behavior based on the fusion of visual and language models, characterized in that, Includes the following steps: Step S1: Acquire real-time video streams from the industrial site and extract frames from the images to obtain the original images as the input data source. After preprocessing, a dataset is formed. The improved YOLOv8 model is used to perform visual inspection on the dataset. The improved YOLOv8 model is to embed a coordinate attention mechanism into the backbone network of the original YOLOv8 model. Step S2: Perform structured integration of the visual detection results from Step S1 and generate semantic description text. T ; Step S3: Extract features from the original image obtained in Step S1 and the semantic description text obtained in Step S2 to obtain visual feature vectors. V With text feature vectors L Visual feature vectors V With text feature vectors L Multimodal fusion is performed to obtain multimodal features. F ; Step S4: Combine the multimodal features fused in step S3. F The input is fed into the language large model reasoning module, which then utilizes the language large model reasoning module in conjunction with the security rule knowledge base. K To coordinate and calculate the risk value of violations. R ; Step S5: Define and calculate the joint loss function to optimize the model, and automatically output a decision report. The joint loss function is the detection loss. L det With semantic loss L sem The weighted sum.

2. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 1, characterized in that, In step S1, the acquired video stream or image is decoded and time-synchronized, key frames are extracted according to the set frame rate, and the extracted key frames are preprocessed. Image preprocessing includes noise reduction, brightness correction, motion compensation, and resolution adaptive adjustment.

3. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 1, characterized in that, In step S2, the visual detection results are structured and integrated, and then output as a structured set: in, D A collection of detection results containing multiple detection targets; For the first Coordinates of each detection box For category labels and tooling labels, To determine the confidence level, N represents the total number of targets detected in the current image frame. i It is the target index, from 1 to N.

4. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 3, characterized in that, In step S2, the scene environment metadata is combined. M Semantic description text generated using a large language model T as follows: 。 5. A method for monitoring industrial safety behavior based on the fusion of visual and language models according to any one of claims 1-4, characterized in that, In step S3, multimodal features are calculated. F The methods include: (1) Calculate the dynamic gating vector ; in,[ V , L The symbol ] represents the concatenation of extracted visual features and text features. , These are the parameters of the fully connected layer of the gated network. Use the Sigmoid activation function; The value changes dynamically between [0, 1]. (2) Generate fused multimodal features F ; Here, ⊙ represents element-wise multiplication.

6. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 5, characterized in that, In step S4, the calculated violation risk value R as follows: in, Indicates the risk value of violation. For the Sigmoid function, These are trainable inference parameters. This is the regular attention vector.

7. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 6, characterized in that, Based on calculations R To set a threshold for comprehensively judging the level of violation, the details are as follows: when R If the value is less than 0.3, then it is not in violation of the rules; When 0.3≤ R A value less than 0.7 indicates a minor violation. when R A value of ≥0.7 indicates a serious violation.

8. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 7, characterized in that, In step S5, the joint loss function L The details are as follows: in, L det To detect loss, it is used to measure the deviation between the detection result and the true label; L sem Semantic loss is used to constrain the consistency between the model's language reasoning and the knowledge base rules. This is the balance coefficient.

9. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 8, characterized in that, In step S5, the detection loss is calculated using binary cross-entropy. L det : in, To predict the output, For real labels, These are the sample weights.

10. The industrial safety behavior monitoring method based on the fusion of visual and language models according to claim 8, characterized in that, In step S5, semantic loss L sem The calculation formula is as follows: in, The semantic score for language reasoning outputs a match with the knowledge base, with a value range of [0,1]. R This indicates the risk value for violations.

Images