A method and device for detecting security violations based on a visual language model
By combining visual language models with data augmentation and LoRA fine-tuning technology, the problems of low efficiency and poor accuracy in industrial safety inspection have been solved, achieving efficient and accurate identification and report generation of violations, and adapting to complex and long-distance low-quality monitoring scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH BEIJING
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-26
Smart Images
Figure CN122290199A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of industrial safety monitoring technology, and in particular to a method and device for detecting safety violations based on a visual language model. Background Technology
[0002] Industrial production is a high-risk industry, and unsafe worker behavior is the main cause of industrial accidents. In order to effectively detect unsafe worker behavior, various industries have deployed a large number of cameras in the work area and assigned dedicated personnel to identify violations. Traditional manual inspection methods have the following problems: (1) they are labor-intensive and time-consuming, and difficult to cover large-scale work areas; (2) they are easily affected by human factors, resulting in inconsistent inspection results and a high omission rate; (3) they cannot monitor and warn of complex and dynamic work scenarios in real time.
[0003] Currently, the mainstream methods for industrial safety inspection fall into three categories: (1) Target detection-based methods: Early industrial safety monitoring mainly used target detection networks (such as Faster R-CNN and YOLO series), focusing on the detection of personal protective equipment, such as safety helmets, safety glasses, and protective gloves. The main limitations of this type of method include: narrow detection range, limited to specific objects, unable to identify complex unsafe behaviors; unable to understand the context of the work scene, for example, even if workers wear equipment, working in an unsafe location is still a violation; and difficulty in capturing the interaction and related behaviors between multiple workers. (2) Zero-shot semantic models, which embed images and text into a shared feature space for matching, matching images and safety rule text in the shared feature space. The shortcomings of this method include: the similarity score is unstable when faced with complex prompt words; the processing efficiency decreases when the number of safety rules is large; and it is difficult to generate an interpretable explanation of the reasons for the violation. (3) Visual-Language Model (VLM) can jointly process multimodal inputs to generate natural language descriptions, but the general model has not been fine-tuned for industrial applications and is not capable of identifying safety violations specific to the industry; it lacks high-quality labeled datasets for industrial applications, making it difficult to perform targeted model optimization; when faced with hundreds or even thousands of safety rules, the inference latency increases sharply, making it difficult to support real-time applications.
[0004] In addition, existing methods have the following shortcomings: first, they lack high-quality labeled datasets specific to the mining industry; second, they have limited ability to identify subtle violations in distant or low-quality monitoring footage. Summary of the Invention
[0005] To address the aforementioned problems, the present invention aims to provide a method and apparatus for detecting safety violations based on a visual language model, thereby improving the efficiency and accuracy of detecting and identifying violations by workers in the industrial field.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On the one hand, a method for detecting security violations based on a visual language model is provided, the method comprising the following steps: S1. Collect video data containing safety violations from work sites in multiple industrial sectors and preprocess it; S2. Employ complementary data augmentation strategies to enhance the preprocessed data; S3. After obtaining the augmented data, use the open-source model to automatically generate security analysis annotations; S4. Construct a standardized training dataset; S5. Use LoRA fine-tuning technology to fine-tune the visual language model; S6. Deploy the model online and use the behavior amplification module to process the image of the input model; S7. Input the processed image into the fine-tuned visual language model for security violation detection and inference. S8. After completing the reasoning, generate a structured violation report and issue a real-time alert.
[0007] Optionally, step S1 specifically includes: S11. Video Sampling and Keyframe Extraction: Extract keyframes from the acquired video data at a frequency of 1fps; S12. Obtaining the original image set; The extracted keyframes are filtered to obtain a raw image set; the raw image set covers a variety of safety violation scenarios in various industrial fields; S13, Data quality check; Perform quality checks on the original image set, including: image resolution, image sharpness, content completeness, and identification of the violation scene.
[0008] Optionally, step S2 specifically includes: S21: Horizontal flip; The preprocessed image is randomly horizontally flipped with a 50% probability to change the spatial position of key objects and increase the model's robustness to geometric transformations. S22, Low-light synthesis; The brightness of the preprocessed image is randomly and non-linearly reduced to simulate low-light industrial scenes, including mines, basements, and night shifts. S23, masking; Randomly select 10%-30% of non-critical areas in the image for occlusion, and fill the occluded areas with random colors or Gaussian noise to enhance the model's focus on safety-critical areas.
[0009] Optionally, step S3 specifically includes: S31. Specific content annotation; The annotation information includes scene description: describing the number and location of workers in the image, the activities the workers are doing, the surrounding equipment, materials and working environment, lighting conditions, and safety hazards; S32. Safety rules are checked item by item; Determine whether each safety rule is in violation and provide specific evidence for the determination. If a determination cannot be made, explain the reason.
[0010] Optionally, step S4 specifically includes: Based on the fine-tuning requirements of the visual language model, a standardized training dataset is constructed, which includes three core components: system prompts, user questions, and model answers, forming a complete "instruction triple". The system prompts define the role, task objectives, and operational principles of the visual language model; user questions provide specific guidance on how the visual language model analyzes the current image samples, using a structured questioning approach to guide the model in generating high-quality analysis; and the model responses are the results formed after standardizing and structuring the content automatically generated by the open-source model.
[0011] Optionally, step S5 specifically includes: S51, LoRA parameter injection location design; LoRA parameters are injected into the following network structures: Query(Q) and Value(V) projection layers in the self-attention module of the visual language model; image-text fusion linear layer in the cross-modal alignment module; and high-level semantic mapping linear layer before the output layer. S52, Low-rank adaptive update mechanism; For any original linear transformation matrix LoRA updates its parameters as follows: in , , For low-rank constrained dimensions; during training, the original Save frozen, only for low-rank matrices and Update the database so that the number of new parameters is reduced from [previous number]. Reduce to This reduces computational complexity; S53, Multimodal Joint Fine-tuning Strategy; The model training employs a standard language model loss function: the causal language modeling loss function. For each training sample, the model predicts the current token based on previous tokens, minimizing the prediction error. In image-text pairs... During training, the standard autoregressive cross-entropy loss is defined as follows: In the formula, Indicates the input image. Represented as a target text sequence, It is the first One token, This indicates the previous token history sequence. This indicates the frozen pre-training parameters. This represents the LoRA parameters that are trainable for all linear layers.
[0012] Optionally, step S6 specifically includes: Design a behavior amplification module to process the image of the input model: First, an open-source object detector is used to accurately locate the worker's position. For each detected worker with a confidence level of conf>0.7, a local magnification operation is performed. The magnification strategy is 2x magnification, which means expanding the clipping area around the worker to twice the size of the original bounding box. Secondly, the resolution of the magnified worker area is enhanced by applying Real-ESRGAN super-resolution technology to generate a high-resolution image of the worker's details. Finally, the enlarged and enhanced regions are seamlessly reinserted into the original image to form a hybrid image, while non-critical background areas retain their original resolution to optimize computational efficiency.
[0013] Optionally, step S7 specifically includes: After behavior amplification, the processed mixed image is input into the fine-tuned visual language model for security violation detection inference; the input during the inference stage includes the following components: Visual input: A composite image processed by the behavior magnification module, containing multiple magnified and enhanced worker areas; Text input: system prompts, user questions, and relevant security rules.
[0014] Optionally, step S8 specifically includes: Each violation report includes the following: the confirmed violation rule number and description, the risk level, an analysis of the visual language model, and a description of specific evidence.
[0015] On the other hand, a security violation detection device based on a visual language model is provided for implementing the method described in any of the above embodiments, the device comprising: The data acquisition and preprocessing module is used to collect video data containing safety violations from work sites in multiple industrial sectors and to perform preprocessing. The data augmentation module is used to enhance preprocessed data using complementary data augmentation strategies; The automatic annotation module is used to automatically generate security analysis annotations using open-source models after obtaining augmented data; The dataset building module is used to build standardized training datasets; The model fine-tuning module is used to fine-tune the visual language model using LoRA fine-tuning technology; The online deployment module is used to deploy models online, and the behavior amplification module processes the images of the input models. The detection and inference module is used to input the processed image into the fine-tuned visual language model for security violation detection and inference. The report generation module is used to generate structured violation reports and issue real-time alerts after reasoning is completed.
[0016] On the other hand, an electronic device is provided, the electronic device comprising: processor; The memory stores computer-readable instructions, which, when loaded and executed by the processor, implement the steps of the security violation detection method based on the visual language model described above.
[0017] On the other hand, a computer-readable storage medium is provided, wherein program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the steps of the security violation detection method based on the visual language model described above.
[0018] The beneficial effects of the technical solution provided by this invention include at least the following: This invention provides a method and apparatus for detecting safety violations based on a visual language model. A complete training image set is constructed using specific sampling and data augmentation strategies. A multimodal standardized dataset is developed according to the standard format of system prompts, user questions, and model responses to fine-tune the model. Utilizing LoRA fine-tuning technology, efficient domain adaptation can be achieved by adjusting only a few parameters, quickly adapting a general visual language model to industrial safety detection tasks. By applying a behavior magnification module and image super-resolution technology, the worker area is magnified and enhanced through super-resolution, effectively addressing long-distance and low-quality monitoring scenarios. This enables high-quality violation recognition by the model, automatically generating structured text descriptions and analysis reports, thus improving the efficiency and accuracy of detecting and recognizing violations by workers in the industrial field. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart of the security violation detection method based on a visual language model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the processing procedure of the security violation detection method based on a visual language model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the training dataset for the security violation detection method based on a visual language model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the training effect of the security violation detection method based on a visual language model provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the training effect of the dataset and test set of the security violation detection method based on visual language model provided in this embodiment of the invention; Figure 6 This is a schematic diagram of the structure of the security violation detection device based on a visual language model provided in an embodiment of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0022] To facilitate understanding, the relevant terminology of Visual Language Modeling (VLM) needs to be explained. A dataset is the foundation for training and testing a visual language model. It consists of a set of input data and corresponding target data. A loss function is a function used to measure the difference between the model's predictions and the actual target. The main purpose of the loss function is to quantify the model's performance and help the model learn to adjust its parameters to minimize prediction error.
[0023] In this embodiment of the invention, the training dataset is input into the visual language large model in the form of image-text pairs. The constructed standardized dataset includes system prompts, user questions, and model responses. During fine-tuning, the model's original pre-trained parameters remain frozen; only a trainable low-rank parameter matrix is introduced alongside its existing linear transformation structure to incrementally adapt to the original model. Through continuous optimization of the cross-entropy objective function, the model develops a bias towards specific semantic distributions in the industrial safety domain at the lexical generation level. This results in key semantic lexical units related to violations having a higher generation probability under the same visual input conditions, thereby achieving stable identification and consistent description of safety violations.
[0024] The goal of using a large visual language model to detect safety violations is to more conveniently and efficiently reduce the heavy workload of manual detection. Two important metrics are involved in the detection process: firstly, the accuracy of detecting violations, which measures the model's learning ability; and secondly, the readability of the generated report, which reduces the workload of staff and provides analytical evidence.
[0025] like Figure 1 As shown, this embodiment of the invention provides a method for detecting security violations based on a visual language model. The method includes the following steps: S1. Collect video data containing safety violations from work sites in multiple industrial sectors and preprocess it.
[0026] Over 100 hours of video data, including footage of safety violations, was collected from real-world work sites across multiple industrial sectors, particularly mining. To ensure the representativeness and diversity of the data, the collected video data covered: different time periods (day shift, night shift, morning, evening) to capture varying lighting conditions; different work types (routine operations, emergency response, maintenance, etc.); and comparisons between normal and non-compliant operations.
[0027] Step S1 specifically includes: S11. Video Sampling and Keyframe Extraction: Keyframes are extracted from the acquired video data at a frequency of 1 fps (1 frame per second). This sampling frequency was chosen based on the following considerations: sampling at 1 fps can capture the evolution of violations and avoid missing key violations.
[0028] S12. Obtaining the original image set; Keyframes extracted from over 100 hours of video data were filtered to remove frames of extremely poor quality or those without personnel, resulting in a set of 3,000 original images. This set of 3,000 original images covers various safety violation scenarios across different industrial sectors.
[0029] S13, Data quality check; The original image set is subjected to quality checks, including: image resolution (1280×720 pixels), image sharpness, content integrity, and clarity of violation scenes, to ensure the quality of data used for subsequent processing.
[0030] S2. Employ complementary data augmentation strategies to enhance the preprocessed data.
[0031] In response to the diversity of industrial environments and the limited availability of data, this invention designs a complementary data augmentation strategy to further enrich violation images and improve the model's generalization ability and robustness.
[0032] Step S2 specifically includes: S21: Horizontal flip; By randomly flipping the preprocessed image horizontally with a 50% probability, the spatial positions of key objects (workers and equipment) are changed, increasing the model's robustness to geometric transformations and particularly enhancing the model's ability to observe workers and equipment from multiple directions.
[0033] S22, Low-light synthesis; The brightness of the preprocessed images is randomly and non-linearly reduced (by 20%-50%) to simulate low-light industrial scenarios, including mines, basements, and night shifts. This enhances the model's recognition capabilities under harsh lighting conditions and helps the model learn to identify key safety features under low signal-to-noise ratio conditions.
[0034] S23, masking; Randomly select 10%-30% of non-critical areas in the image for occlusion, and fill the occluded areas with random colors or Gaussian noise to enhance the model's attention to safety-critical areas, learn to ignore irrelevant backgrounds, and enhance the model's robustness in partially occluded scenarios (such as workers being occluded by other workers or equipment).
[0035] The above augmentation strategies effectively alleviated the problem of data scarcity. The original 3000 images were expanded into a larger augmented dataset.
[0036] S3. After obtaining the enhanced data, use the open-source model to automatically generate security analysis annotations.
[0037] After obtaining the enhanced image data, detailed security analysis annotations are needed to provide supervisory signals for subsequent model fine-tuning. This invention employs an automatically generated annotation stream, providing complete dataset information for the model. For both the original and enhanced images, security analysis annotations for all image data can be generated in batches using the open-source Qwen model.
[0038] Step S3 specifically includes: S31. Specific content annotation; The annotation information includes scene description: describing the number and location of workers in the image, the activities the workers are doing, the surrounding equipment, materials and working environment, lighting conditions, and obvious safety hazards; S32. Safety rules are checked item by item; For each of the 40 high-frequency safety rules (covering three categories: personal protective equipment, unsafe behavior, and misuse of tools and equipment), determine whether the rule is in violation and provide specific evidence for the determination (which parts of the image indicate a violation). If a determination cannot be made, explain the reason (e.g., critical areas are obscured).
[0039] S4. Construct a standardized training dataset.
[0040] Based on the fine-tuning requirements of the visual language model, a standardized training dataset is constructed, which includes three core components: system prompts, user questions, and model responses, forming a complete "instruction triplet".
[0041] System prompt word design: System prompt words define the role, task objectives, and operational principles of the visual language model. For example, the role of VLM is defined as a professional industrial safety monitoring analyst, and its task objective is clearly defined as analyzing monitoring images and identifying violations of safety rules.
[0042] User question design: User questions specifically guide the visual language model on how to analyze the current image samples, employing a structured questioning approach to guide the model in generating high-quality analysis. For example, a structured questioning approach could be used to request analysis of workers, actions, and the environment, followed by a check of each safety rule, providing visual evidence and clarity assessments (clear, general, blurry, insufficient evidence) for each potential violation.
[0043] The formation of the model response: The model response is the result of normalizing and structuring the content automatically generated by the open-source model (Qwen model). For example, the model response is normalized and structured based on the content automatically generated by the Qwen model, ultimately forming a complete answer that includes a scenario description, relevant rules, step-by-step analysis chains, and overall conclusions.
[0044] S5. Use LoRA fine-tuning technology to fine-tune the visual language model.
[0045] After obtaining the standardized training dataset (containing 9000 samples, namely the original 3000 samples, their augmented samples, and object detection annotations), this method uses LoRA (Low-Rank Adaptation) fine-tuning technology to achieve efficient adaptation of the model to industrial safety violation detection tasks while maintaining the original knowledge structure of the general visual language model.
[0046] Qwen2.5-VL-Instruct was chosen as the backbone model, with LoRA parameters set to rank=16 and scaling factor α=32. The LoRA module was strategically injected into each linear layer of the visual encoder and language decoder, introducing a trainable low-rank matrix while keeping the pre-trained weights frozen.
[0047] Step S5 specifically includes: S51, LoRA parameter injection location design; LoRA parameters are injected into the following network structures: the Query(Q) and Value(V) projection layers of the self-attention module in the visual language model; the image-text fusion linear layer in the cross-modal alignment module; and the high-level semantic mapping linear layer before the output layer. By introducing trainable low-rank parameters only to the linear layers that are highly relevant to semantic understanding and cross-modal reasoning, this invention significantly reduces the size of training parameters while maximizing the preservation of the general visual and language understanding capabilities of the pre-trained model.
[0048] S52, Low-rank adaptive update mechanism; For any original linear transformation matrix LoRA updates its parameters as follows: in , , For low-rank constrained dimensions; during training, the original Save frozen, only for low-rank matrices and Update the database so that the number of new parameters is reduced from [previous number]. Reduce to This reduces computational complexity.
[0049] S53, Multimodal Joint Fine-tuning Strategy; The model training employs a standard language model loss function: the Causal Language Modeling (CLM) loss function. For each training sample, the model predicts the current token based on previous tokens, minimizing the prediction error. In image-text pairs... During training, the standard autoregressive cross-entropy loss is defined as follows:
[0050] In the formula, Indicates the input image. Represented as a target text sequence, It is the first Each token (word element) This indicates the previous token history sequence. This indicates the frozen pre-training parameters. This represents the LoRA parameters that are trainable across all linear layers. Gradient descent is used to optimize the low-rank matrix parameters, making the model's output terms approximate the standard answers in the dataset, thereby learning the output patterns of industrial safety violations.
[0051] S6. Deploy the model online and use the behavior amplification module to process the image of the input model.
[0052] When deployed online, the model needs to run in a real industrial environment. Industrial environments often present extreme visual conditions: the camera is far from the worker, lighting conditions are poor, and workers are partially obscured. These factors can significantly reduce the model's recognition performance.
[0053] To enhance the model's recognition capabilities under these harsh conditions, this invention designs a BehaviorMagnifier (BM) module to process the input model's image: First, an open-source object detector is used to accurately locate the worker's position. For each detected worker with a confidence level (conf>0.7), a local magnification operation is performed, with a magnification strategy of 2x magnification, i.e., expanding the cropped region around the worker to twice the size of the original bounding box. Second, Real-ESRGAN super-resolution technology is applied to the magnified worker region to enhance its resolution, generating a high-resolution image of the worker's details. Finally, the magnified and enhanced region is seamlessly reinserted into the original image to form a hybrid image, while non-critical background areas retain their original resolution to optimize computational efficiency. This processing significantly improves the accuracy of recognizing fine-grained human behavior.
[0054] S7. Input the processed image into the fine-tuned visual language model for security violation detection and inference.
[0055] Step S7 specifically includes: After behavior amplification, the processed mixed image is input into the fine-tuned visual language model for security violation detection inference; the input during the inference stage includes the following components: Visual input: A composite image processed by the behavior magnification module, containing multiple magnified and enhanced worker areas; Text input includes system prompts (defining model roles and tasks), user questions (requiring the model to perform structured analysis), and relevant security rules.
[0056] S8. After completing the reasoning, generate a structured violation report and issue a real-time alert.
[0057] After VLM completes its inference, the system automatically generates structured violation reports and issues real-time alerts to support rapid response from on-site management personnel. Each violation report includes the following core elements: the confirmed violation rule number and description (e.g., "Rule: Not wearing a safety helmet"), the risk level (high / medium / low), VLM's analysis explanation, and specific evidence description. The system can output these reports in a structured format (such as JSON) and push them to the monitoring platform in real time via a network interface. It also supports automatically generating timestamped violation reports from user-uploaded videos.
[0058] For a detailed description of the implementation process of this invention, please refer to [link / reference]. Figure 2 As shown, by collecting large-scale model data images for security violation detection and employing data augmentation techniques, a complete multimodal dataset was constructed, such as... Figure 3 As shown, the dataset contains various types of information; the training curve results fine-tuned using LoRA technology are as follows. Figure 4 As shown, the lexical accuracy of the model output ( Figure 4 (a) continues to rise, optimizing the target loss ( Figure 4 (b) The accuracy of the large-scale violation detection model trained on the training and test sets is continuously decreasing; Figure 5 As shown, the accuracy rate can reach 86.50%, effectively identifying the vast majority of security violations; the online-deployed model can perform a series of analyses on violations and generate corresponding violation reports.
[0059] Accordingly, embodiments of the present invention also provide a security violation detection device based on a visual language model, such as... Figure 6 As shown, the device includes: The data acquisition and preprocessing module 201 is used to acquire video data containing safety violations from work sites in multiple industrial fields and to perform preprocessing. Data augmentation module 202 is used to augment preprocessed data using complementary data augmentation strategies; The automatic annotation module 203 is used to automatically generate security analysis annotations using open-source models after obtaining augmented data; Dataset building module 204 is used to build standardized training datasets; The model fine-tuning module 205 is used to fine-tune the visual language model using LoRA fine-tuning technology; The online deployment module 206 is used for online model deployment and uses the behavior amplification module to process the image of the input model; The detection and inference module 207 is used to input the processed image into the fine-tuned visual language model for security violation detection and inference. The report generation module 208 is used to generate a structured violation report and issue real-time alerts after completing the reasoning.
[0060] For ease of explanation, Figure 6 Only the main components of the device are shown. The device of this embodiment can be used to perform... Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.
[0061] The safety violation detection method and apparatus based on a visual language model provided in this invention constructs a complete training image set using specific sampling and data augmentation strategies. A multimodal standardized dataset is developed according to the standard format of system prompts, user questions, and model responses to fine-tune the model. Utilizing LoRA fine-tuning technology, efficient domain adaptation can be achieved by adjusting only 0.1%-1% of the parameters, quickly adapting a general VLM to industrial safety detection tasks. By applying a behavior magnification module and image super-resolution technology, the worker area is magnified and enhanced with super-resolution, effectively addressing long-distance and low-quality monitoring scenarios. This enables high-quality violation recognition by the model, automatically generating structured text descriptions and analysis reports, thus improving the efficiency and accuracy of detecting and recognizing worker violations in the industrial field.
[0062] In an exemplary embodiment, the present invention also provides an electronic device, the electronic device comprising: processor; The memory stores computer-readable instructions, which, when loaded and executed by the processor, implement the steps of the security violation detection method based on the visual language model described above.
[0063] In an exemplary embodiment, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the steps of the security violation detection method based on the visual language model described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.
[0064] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0065] The use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.
[0066] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0067] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0069] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0071] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.
[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for detecting security violations based on a visual language model, characterized in that, Includes the following steps: S1. Collect video data containing safety violations from work sites in multiple industrial sectors and preprocess it; S2. Employ complementary data augmentation strategies to enhance the preprocessed data; S3. After obtaining the augmented data, use the open-source model to automatically generate security analysis annotations; S4. Construct a standardized training dataset; S5. Use LoRA fine-tuning technology to fine-tune the visual language model; S6. Deploy the model online and use the behavior amplification module to process the image of the input model; S7. Input the processed image into the fine-tuned visual language model for security violation detection and inference. S8. After completing the reasoning, generate a structured violation report and issue a real-time alert. 2.The method of claim 1, wherein, Step S1 specifically includes: S11. Video Sampling and Keyframe Extraction: Extract keyframes from the acquired video data at a frequency of 1fps; S12. Obtaining the original image set; The extracted keyframes are filtered to obtain a raw image set; the raw image set covers a variety of safety violation scenarios in various industrial fields; S13, Data quality check; Perform quality checks on the original image set, including: image resolution, image sharpness, content completeness, and identification of the violation scene. 3.The method of claim 1, wherein, Step S2 specifically includes: S21: Horizontal flip; The preprocessed image is randomly horizontally flipped with a 50% probability to change the spatial position of key objects and increase the model's robustness to geometric transformations. S22, Low-light synthesis; The brightness of the preprocessed image is randomly and non-linearly reduced to simulate low-light industrial scenes, including mines, basements, and night shifts. S23, masking; Randomly select 10%-30% of non-critical areas in the image for occlusion, and fill the occluded areas with random colors or Gaussian noise to enhance the model's focus on safety-critical areas. 4.The method of claim 1, wherein, Step S3 specifically includes: S31. Specific content annotation; The annotation information includes scene description: describing the number and location of workers in the image, the activities the workers are doing, the surrounding equipment, materials and working environment, lighting conditions, and safety hazards; S32. Safety rules are checked item by item; Determine whether each safety rule is in violation and provide specific evidence for the determination. If a determination cannot be made, explain the reason.
5. The method for detecting security violations based on a visual language model according to claim 1, characterized in that, Step S4 specifically includes: Based on the fine-tuning requirements of the visual language model, a standardized training dataset is constructed, which includes three core components: system prompts, user questions, and model answers, forming a complete "instruction triple". The system prompts define the role, task objectives, and operational principles of the visual language model; user questions provide specific guidance on how the visual language model analyzes the current image samples, using a structured questioning approach to guide the model in generating high-quality analysis; and the model responses are the results formed after standardizing and structuring the content automatically generated by the open-source model.
6. The method for detecting security violations based on a visual language model according to claim 1, characterized in that, Step S5 specifically includes: S51, LoRA parameter injection location design; LoRA parameters are injected into the following network structures: Query(Q) and Value(V) projection layers in the self-attention module of the visual language model; image-text fusion linear layer in the cross-modal alignment module; and high-level semantic mapping linear layer before the output layer. S52, Low-rank adaptive update mechanism; For any original linear transformation matrix LoRA updates its parameters as follows: in , , For low-rank constrained dimensions; during training, the original Save frozen, only for low-rank matrices and Update the database so that the number of new parameters is reduced from [previous data]. Reduce to This reduces computational complexity; S53, Multimodal Joint Fine-tuning Strategy; The model training employs a standard language model loss function: the causal language modeling loss function. For each training sample, the model predicts the current token based on previous tokens, minimizing the prediction error. In image-text pairs... During training, the standard autoregressive cross-entropy loss is defined as follows: In the formula, Indicates the input image. Represented as a target text sequence, It is the first One token, This indicates the previous token history sequence. This indicates the frozen pre-training parameters. This represents the LoRA parameters that are trainable for all linear layers.
7. The method for detecting security violations based on a visual language model according to claim 1, characterized in that, Step S6 specifically includes: Design a behavior amplification module to process the image of the input model: First, an open-source object detector is used to accurately locate the worker's position. For each detected worker with a confidence level of conf > 0.7, a local magnification operation is performed. The magnification strategy is 2x magnification, which means expanding the clipping area around the worker to twice the size of the original bounding box. Secondly, the resolution of the magnified worker area is enhanced by applying Real-ESRGAN super-resolution technology to generate a high-resolution image of the worker's details. Finally, the enlarged and enhanced regions are seamlessly reinserted into the original image to form a hybrid image, while non-critical background areas retain their original resolution to optimize computational efficiency.
8. The method for detecting security violations based on a visual language model according to claim 1, characterized in that, Step S7 specifically includes: After behavior amplification, the processed mixed image is input into the fine-tuned visual language model for security violation detection inference; the input during the inference stage includes the following components: Visual input: A composite image processed by the behavior magnification module, containing multiple magnified and enhanced worker areas; Text input: system prompts, user questions, and relevant security rules.
9. The method for detecting security violations based on a visual language model according to claim 1, characterized in that, Step S8 specifically includes: Each violation report includes the following: the confirmed violation rule number and description, the risk level, an analysis of the visual language model, and a description of specific evidence.
10. A security violation detection device based on a visual language model, the device being used to implement the method as described in any one of claims 1 to 9, characterized in that, The device includes: The data acquisition and preprocessing module is used to collect video data containing safety violations from work sites in multiple industrial sectors and to perform preprocessing. The data augmentation module is used to enhance preprocessed data using complementary data augmentation strategies; The automatic annotation module is used to automatically generate security analysis annotations using open-source models after obtaining augmented data; The dataset building module is used to build standardized training datasets; The model fine-tuning module is used to fine-tune the visual language model using LoRA fine-tuning technology; The online deployment module is used to deploy models online, and the behavior amplification module processes the images of the input models. The detection and inference module is used to input the processed image into the fine-tuned visual language model for security violation detection and inference. The report generation module is used to generate structured violation reports and issue real-time alerts after reasoning is completed.