Operation and construction illegal behavior identification method and system

Abstract

The invention discloses an operation and construction violation behavior identification method and system, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a plurality of video clips of operation and construction violation behaviors; performing semantic alignment between the video clips and the dangerous behavior text on each video clip by using a pre-trained video text alignment model to obtain a semantic alignment data set; outputting a window text similarity score according to the semantic alignment data set by utilizing a pre-trained video text alignment model; generating an enhanced training set comprising positive samples and difficult negative samples according to the window text similarity score and the semantic alignment data set; using an o-LoRA module, a large-scale video understanding data set and an enhanced training set to train a multi-modal large model; and inputting monitoring videos of operation and construction into the trained multi-modal large model, and identifying to obtain illegal behaviors. According to the method, the operation and construction violation behaviors can be efficiently and accurately identified.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for identifying operational and construction violations. Background Technology

[0002] With the advancement of informatization and intelligentization, oil companies have widely deployed a large number of monitoring cameras in well sites, refining and chemical plant areas, storage areas, and oil pipeline corridors to achieve 24 / 7 video monitoring and safety management. However, due to the large number of monitoring points and the massive amount of video data, traditional methods relying on manual inspections and spot checks are insufficient for comprehensive supervision. Manually reviewing videos is not only time-consuming and labor-intensive, but also prone to omissions and misjudgments due to fatigue, subjective judgment differences, or distraction, reducing the timeliness and accuracy of safety control. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method and system for identifying violations of work and construction regulations, so as to accurately and efficiently identify violations of work and construction regulations.

[0004] One aspect of this application provides a method for identifying operational and construction violations, the method comprising the following steps:

[0005] Multiple video clips of violations of work and construction regulations are obtained; wherein the text indicating the dangerous behavior and the timestamp of occurrence are consistent in each video clip;

[0006] A pre-trained video-text alignment model is used to perform semantic alignment between each video segment and the text of dangerous behavior to obtain a semantic alignment dataset.

[0007] The pre-trained video text alignment model is used to output a window text similarity score based on the semantic alignment dataset.

[0008] An augmented training set, including positive and hard negative samples, is generated based on the window text similarity scores and the semantic alignment dataset.

[0009] The initial multimodal large model is obtained by training the basic multimodal large model using the first o-LoRA module and a large-scale video understanding dataset;

[0010] The initial multimodal large model is trained using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model;

[0011] The monitoring videos of the operation and construction are input into the target multimodal large model to identify violations of operation and construction regulations.

[0012] In some embodiments, acquiring multiple video clips of operational and construction violations includes the following steps:

[0013] To obtain multiple surveillance videos of potentially dangerous or already dangerous behaviors;

[0014] Each of the surveillance videos is segmented using a sliding window with a fixed length and a fixed step size, resulting in multiple video segments.

[0015] The step of segmenting each of the surveillance videos includes:

[0016] Let the total duration of the surveillance video be T seconds, the window length be L seconds, and the step size be S seconds. Divide the surveillance video into several consecutive or partially overlapping time windows. , No. The time range corresponding to each window Follow the formula below:

[0017] ;

[0018] in, To find the minimum value function, the window's end time is... Not exceeding the total duration of the surveillance video. ;

[0019] A predetermined number of keyframes are extracted from each video segment using a uniform sampling method;

[0020] The annotation information in each video segment is formatted into text, and the formatted text is converted into corresponding word vectors. The keyframes are synchronized with the word vectors so that each video segment is consistent with the corresponding dangerous behavior text and timestamp. The annotation information includes the type of dangerous behavior and the time period in which it occurred.

[0021] In some embodiments, the pre-trained video text alignment model is trained through the following steps:

[0022] Construct a video text alignment model based on CLIP architecture;

[0023] During training, the image backbone and text encoding backbone of the CLIP architecture are frozen, and the video projection head is... Text projection head The system also trains and updates the learnable temperature parameter logit_scale. The video projector and the text projector use the same network structure, which includes LayerNorm, linear transformation, GELU activation function, Dropout, and output layer. Finally, the output features are L2 normalized to map the video and text features to a unified semantic space.

[0024] Construct a multi-objective loss function that combines global contrastive learning and local hard-negative sample mining; the multi-objective loss function includes information contrastive loss. and ranking loss of difficult samples Through weighting coefficients To achieve balance, the calculation formula is:

[0025] ;

[0026] in The multi-objective loss function described above; The weighting factor for ranking loss is used to balance... and The relative importance;

[0027] Constructing information contrast loss This is used to calculate the bidirectional contrast loss between video and text features; given a batch size of B training samples, it calculates the similarity matrix between video features and text features. The element in the i-th row and j-th column is defined as:

[0028] ;

[0029] in, The first in the batch Feature vectors of each video; The first in the batch Feature vectors of each text; Here is the temperature parameter; exp(·) is the exponential function; This represents the similarity between the i-th video feature and the j-th text feature;

[0030] Information comparison loss Defined as:

[0031] ;

[0032] Where CE(·) is the cross-entropy function; I is the label matrix; and S is the similarity matrix between video features and text features. This is the transpose of the similarity matrix;

[0033] Constructing ranking loss for hard-to-bear samples Designed for the similar negative example problem in dangerous behavior identification, for each positive sample pair and several negative sample pairs Defined as:

[0034] ;

[0035] ;

[0036] in, Indicates the first Ranking loss for each positive sample pair; This indicates the final ranking loss for the entire batch; Indicates batch size; m is the ranking interval constant; Indicates positive sample pairs The similarity between them; Indicates negative sample pairs The similarity between them;

[0037] Difficult to bear sample ranking loss The strategy employs the most difficult negative sample. Positive sample windows use the labeled time periods of dangerous behavior, while a portion of the negative sample windows come from high-scoring windows obtained in the sliding window retrieval within the same video but with low IoU with manually labeled windows. Another portion comes from windows other than positive samples in the same batch. The union of the two windows As the aforementioned hard-to-bear sample.

[0038] In some embodiments, the step of using the pre-trained video text alignment model to output window text similarity scores based on the semantic alignment dataset includes the following steps:

[0039] By utilizing the human-computer dialogue corresponding to each video-text-time monitoring video and the start and end times of the labeled dangerous behavior, for each window batch, the text similarity score sim of the window is trained based on the backbone network and projection head of the CLIP architecture, and finally concatenated on all windows to obtain a one-dimensional similarity vector sims.

[0040] In some embodiments, generating an augmented training set including positive samples and hard negative samples based on the window text similarity score and the semantic alignment dataset includes the following steps:

[0041] The positive samples are selected using two complementary selection modes, namely, a non-maximum suppression mode and a merging mode.

[0042] The non-maximum suppression mode is used to select behaviors that are less than a first set duration, discrete, and repetitive, in order to select multiple time-dispersed high-confidence windows.

[0043] The merging mode is used to achieve a second set duration and interval-type behavior. First, the high-scoring windows are merged in time, and then the segment length is controlled; the second set duration is greater than the first set duration.

[0044] Around the selected positive samples, windows that are correlated with the text to a set threshold but are temporally separated from both the positive samples and the labeled intervals are selected from the similarity space as the hard negative samples.

[0045] In some embodiments, training a basic multimodal large model using a first o-LoRA module and a large-scale video understanding dataset to obtain an initial multimodal large model includes the following steps:

[0046] InternVid-10M-FLT and ActivityNet Captions were selected as the large-scale video understanding dataset. The basic multimodal large model deployed locally was used to train the basic multimodal large model to obtain the initial multimodal large model.

[0047] In some embodiments, training the initial multimodal large model using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model includes the following steps:

[0048] The first o-LoRA module, the second o-LoRA module, and the enhanced training set are used to add a trainable low-rank branch to the target projection matrix in the attention layer of the initial multimodal large model, and then training is performed to obtain the target multimodal large model.

[0049] Another aspect of this application embodiment provides a system for identifying operational and construction violations, the system comprising:

[0050] The video acquisition unit is used to acquire multiple video clips of violations of work and construction regulations; wherein, the text indicating the dangerous behavior and the timestamp of occurrence are consistent in each video clip;

[0051] The semantic alignment unit is used to perform semantic alignment between video segments and dangerous behavior texts using a pre-trained video text alignment model to obtain a semantic alignment dataset.

[0052] The window text similarity determination unit is used to output a window text similarity score based on the semantic alignment dataset using the pre-trained video text alignment model.

[0053] The data augmentation unit is used to generate an augmented training set including positive samples and hard negative samples based on the window text similarity score and the semantic alignment dataset;

[0054] The first model training unit is used to train the basic multimodal large model using the first o-LoRA module and a large-scale video understanding dataset to obtain the initial multimodal large model;

[0055] The second model training unit is used to train the initial multimodal large model using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model;

[0056] The violation identification unit is used to input the monitoring video of the operation and construction into the target multimodal large model to identify the violation of the operation and construction.

[0057] Another aspect of this application embodiment provides an electronic device, including a processor and a memory;

[0058] The memory is used to store programs;

[0059] The processor executes the program to implement any of the methods described above.

[0060] Another aspect of this application provides a computer-readable storage medium storing a program that is executed by a processor to implement the method described in any of the above embodiments.

[0061] This application includes at least the following beneficial effects:

[0062] This application can acquire multiple video clips of operational and construction violations; in each video clip, the labeled dangerous behavior text and the occurrence timestamp are consistent; a pre-trained video text alignment model is used to perform semantic alignment between each video clip and the dangerous behavior text, resulting in a semantic alignment dataset; the pre-trained video text alignment model outputs window text similarity scores based on the semantic alignment dataset; an augmented training set including positive and hard negative samples is generated based on the window text similarity scores and the semantic alignment dataset; a basic multimodal large model is trained using a first o-LoRA module and a large-scale video understanding dataset, resulting in an initial multimodal large model; the initial multimodal large model is trained using the first o-LoRA module, a second o-LoRA module, and the augmented training set, resulting in a target multimodal large model; the monitoring video of the operation and construction is input into the target multimodal large model to identify operational and construction violations. This application, by organizing existing video clips and training a multimodal large model using the o-LoRA module, can efficiently and accurately identify operational and construction violations using the final trained target multimodal large model. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 A flowchart illustrating a method for identifying operational and construction violations provided in this application embodiment;

[0065] Figure 2 An example flowchart of a method for identifying operational and construction violations provided in this application embodiment;

[0066] Figure 3 This is a structural block diagram of a work and construction violation identification system provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be described first, as follows:

[0069] Terminology Explanation:

[0070] Large Language Models (LLMs) are deep learning models based on the Transformer architecture, pre-trained on massive amounts of text data, and possess powerful natural language understanding and generation capabilities. These models learn statistical patterns and semantic knowledge of language from trillions of words through self-supervised learning, enabling them to perform various tasks such as text completion, question answering, translation, and code generation. Their core advantages lie in their emerging context learning and thought chain reasoning capabilities, allowing them to adapt to new tasks with a small number of examples or instruction prompts without requiring additional parameter updates. As a core infrastructure in the current field of artificial intelligence, large language models are not only the foundation for processing and understanding policy texts and performing complex reasoning, but also provide a universal semantic foundation for the fusion with multimodal information such as vision and speech.

[0071] Video-Text Alignment (VMA) is a cross-modal learning task that aims to establish fine-grained semantic correspondences between video content and its corresponding text descriptions. In real-world scenarios, misalignment is common between videos and text; for example, a long video might only correspond to a single summary description, or the text might describe a brief action in detail while the video contains numerous irrelevant frames. This misalignment directly leads to incorrect associations learned by the model, severely limiting the performance of cross-modal understanding and generation. VMA technology aims to overcome this challenge by calculating the similarity or matching degree between keyframe sequences and natural language descriptions, achieving accurate semantic mapping between video and text segments. This technique effectively utilizes the complementary semantic information between video and text, improving the model's robustness and generalization ability under noise. Common applications include video retrieval, video description generation, and cross-modal pre-training.

[0072] High resolution refers to a system's comprehensive ability to describe and process data with fine detail across temporal or spatial dimensions. In the temporal dimension, high resolution means the ability to capture and predict short-cycle, high-frequency dynamic changes, such as energy demand fluctuations measured in minutes or seconds, or behavioral events at the keyframe level. In the spatial dimension, high resolution refers to the ability to identify and analyze detailed features of small geographical units or local areas of an image, such as traffic flow distribution at the city block level or the fine structure of organs in medical images. Traditional methods are often limited by coarse data granularity, making it difficult to accurately model local features and instantaneous changes in complex systems. High-resolution technology, by fusing multi-source fine-grained data and employing advanced spatiotemporal modeling methods, effectively improves the accuracy of describing and predicting complex phenomena, providing a crucial data foundation for resource optimization, behavioral analysis, and intelligent decision-making. This is particularly important in energy demand forecasting tasks, significantly enhancing the model's perception capabilities and scenario adaptability.

[0073] CLIP (Contrastive Language-Image Pre-training): CLIP is a multimodal pre-training model based on contrastive learning proposed by OpenAI. Its core idea is to learn the general association between visual concepts and natural language descriptions through training on massive amounts of image-text pairs. The model employs a dual-encoder structure, including an image encoder and a text encoder, used to extract feature representations of the input image and text respectively, and mapping these features to a shared semantic embedding space. Within this space, a contrastive learning objective function is used to narrow the feature distance between matching image-text pairs while distancing mismatched pairs. CLIP's core advantage lies in its powerful zero-shot generalization capability. It requires no specific fine-tuning for downstream tasks and can generalize to various visual tasks, such as image classification, cross-modal retrieval, and visual question answering, simply by providing natural language cues. This effectively overcomes the limitations of traditional visual models that rely on fixed category labels and have insufficient generalization ability, laying a solid foundation for building generalized multimodal understanding systems. It is widely used as a basic visual language model for tasks such as video understanding and image-text alignment.

[0074] Non-Maximum Suppression (NMS) is a widely used post-processing algorithm in computer vision and object detection tasks. It's used to select the best few results representing different objects from a series of overlapping candidate regions or detection boxes. Its core idea is "suppressing non-maximums," meaning retaining only the detection results with the highest confidence within a local range, while removing redundant detections with high overlap but low confidence. The algorithm first sorts all candidate boxes by confidence, then calculates the overlap between the highest-scoring box and the remaining boxes. If the overlap exceeds a preset threshold, it's considered that the same object has been detected, and these low-scoring boxes are suppressed and removed. Traditional methods, lacking this filtering mechanism, often result in repeated detections of the same target, severely impacting the clarity and usability of the results. NMS eliminates redundancy, ensuring the simplicity and accuracy of the final output, making it indispensable in tasks such as object detection, video action localization, and sliding window detection.

[0075] Instruction Tuning: Instruction tuning is a crucial stage in the training process of large language models. Its goal is to transform a general, pre-trained model into an interactive assistant capable of following human instructions. In this stage, the model undergoes supervised tuning using a large amount of paired data consisting of natural language instructions and desired outputs. This data typically covers diverse tasks and expressions, aiming to teach the model to understand the intent of the instructions and generate appropriate, useful, and safe responses accordingly. After instruction tuning, the model's output is no longer a simple text continuation but a task execution result highly aligned with human intent, thus significantly improving the model's practicality and user experience in open-domain dialogue, task-oriented dialogue, and other scenarios.

[0076] Low-Rank Adaptation (LoRA): LoRA is a technique for efficiently fine-tuning large language models. Its core idea is to add a low-rank adapter module alongside the model's existing parameters to learn task-related incremental updates, rather than directly updating all the original model weights. During fine-tuning, the original large model parameters are frozen, and only these newly added, minimal-parameter low-rank matrices are trained. This method significantly reduces the number of parameters that need to be trained and stored, lowering computational overhead and hardware requirements, while avoiding the catastrophic forgetting problem that can occur with full-parameter fine-tuning of the entire model. LoRA makes it possible to quickly adapt large models to specific downstream tasks such as video understanding with limited computing power.

[0077] The relevant technical details are as follows:

[0078] This application aims to construct a dataset of monitoring videos of violations in the energy industry and a large-scale model-based system for identifying violations in these videos. The dataset focuses on the needs of violation identification in energy industry monitoring scenarios, covering typical violation types such as sleeping in the duty room, failure to fully cover construction sites, and various risks posed by construction workers not following regulations. It adds bilingual description functionality, with each sample annotated with a precise time interval, corresponding human-computer question-and-answer interaction pair, and core metadata, enhancing its scenario-specificity and adaptability. The dataset has a standardized format, clearly linking video segments with bilingual violation descriptions. It is compatible with large-scale model RealESRGAN image quality enhancement and CLIP semantic alignment data preprocessing workflows, and can also provide high-quality bilingual training samples for Qwen2-VL two-stage o-LoRA fine-tuning, helping to solve the problems of insufficient efficiency, accuracy, and generalization ability in energy monitoring violation identification. Specific objectives include: overcoming recognition bottlenecks in complex scenarios, improving the model's accuracy in identifying diverse violations in the energy industry, effectively coping with interference factors such as lighting changes, equipment obstruction, and inclement weather, and reducing missed detections and false positives. Based on the acquired proprietary dataset and optimized large model, real-time identification and rapid early warning of violations can be achieved, providing efficient and intelligent technical support for the safety production management of energy enterprises, reducing the probability of safety accidents and economic losses caused by violations, and promoting the intelligent transformation of energy industry monitoring from "passive recording" to "proactive early warning".

[0079] With the increasing demand for safety management in the energy industry, some AI monitoring technologies have been initially applied to violation identification scenarios. However, existing solutions are constrained by technological limitations and industry characteristics, and still have many shortcomings that need to be addressed, specifically in the following three aspects:

[0080] The data processing system has significant shortcomings. The energy industry has a massive number of monitoring devices, generating terabytes of video data daily. Existing data processing technologies mostly employ a "centralized storage + batch analysis" model, which is prone to high storage costs, data transmission bandwidth bottlenecks, and analysis latency, failing to meet the business needs of real-time monitoring. Furthermore, because the data involves confidential corporate information, the industry lacks a secure sharing mechanism, preventing the formation of large-scale training datasets and limiting the performance improvement of AI models. Current solutions rely heavily on manual annotation for training data. However, violations in the energy industry are highly specialized (e.g., improper clothing, inadequate enclosure during hoisting operations), requiring annotators to possess both monitoring technology and industry safety knowledge, resulting in high labor costs. Moreover, manual annotation is susceptible to subjective judgment, directly impacting model training effectiveness.

[0081] The performance of existing models is ill-suited to industry demands. Most models are trained on general computer vision datasets and are not optimized for the specific characteristics of violations in the energy industry. For example, in night vision scenarios, models struggle to distinguish between "workers not wearing safety helmets" and "blurred head contours caused by lighting," primarily because they haven't learned the unique features specific to energy industry violations. Furthermore, many existing models are developed for specific scenarios, leading to widespread "failure when changing scenarios." Additionally, some solutions employ complex model structures to improve accuracy but lack inference optimization, resulting in excessively long single-frame video analysis times. This prevents a rapid "identification-warning-response" closed loop, and delayed warnings can have serious consequences for emergency violations such as "unauthorized hot work."

[0082] Deployment and maintenance costs are high. On the one hand, energy operations are mostly located in remote areas, such as mountain transmission towers and suburban wind farms. These areas have poor network coverage, and deploying a large-scale model recognition system requires supporting hardware such as edge computing devices and high-definition monitoring, resulting in high initial equipment procurement and installation costs. On the other hand, subsequent maintenance is difficult. Not only do professional personnel need to be assigned to regularly maintain the hardware, but continuous investment of manpower is also required to update the algorithm to adapt to new scenarios. Small and medium-sized energy companies cannot afford this long-term expenditure, which limits the widespread application of the technology.

[0083] Application scenarios of this application:

[0084] With the advancement of informatization and intelligentization, oil companies have widely deployed a large number of monitoring cameras in well sites, refining and chemical plant areas, storage areas, and oil pipeline corridors to achieve 24 / 7 video monitoring and safety management. However, due to the large number of monitoring points and the massive amount of video data, traditional methods relying on manual inspections and spot checks are insufficient for comprehensive supervision. Manually reviewing videos is not only time-consuming and labor-intensive, but also prone to omissions and misjudgments due to fatigue, subjective judgment differences, or distraction, reducing the timeliness and accuracy of safety control.

[0085] In actual production processes, employee violations such as operating outside designated areas, not wearing safety protective equipment, unauthorized dismantling of equipment, sleeping or using mobile phones during shifts frequently occur. If these behaviors are not detected and stopped in a timely manner, they can easily lead to safety accidents, equipment damage, or even personal injury. How to utilize intelligent methods to automate the analysis of massive amounts of surveillance video and quickly and accurately identify violations and non-standard operations has become a key requirement for the digital transformation of safety management in petroleum enterprises.

[0086] The application of multimodal models can effectively reduce the burden of manual supervision and significantly reduce omissions and misjudgments. Through automated intelligent identification and real-time early warning, the system greatly improves the efficiency and accuracy of safety management in petroleum enterprises.

[0087] The technical problem that this application aims to solve is:

[0088] Although modern oil companies have widely deployed video surveillance systems, manually reviewing surveillance videos is not only inefficient but also susceptible to subjective human factors, leading to omissions or misjudgments of critical violations and failing to meet the needs for real-time detection and response to security incidents. Furthermore, the monitoring environment in oil companies is significantly complex and diverse. Lighting conditions, monitoring angles, equipment distribution, and personnel attire vary considerably across different plant areas, and monitoring footage is often accompanied by noise, obstructions, and equipment flash, making the recognition results of traditional computer vision algorithms unstable. Enterprise monitoring data often involves sensitive information such as production processes, equipment layout, and personnel activities; data privacy and security issues pose significant obstacles to the implementation of intelligent analysis systems.

[0089] In the field of industrial intelligent monitoring and deep learning model training, numerous research findings have addressed aspects such as improving video recognition efficiency, enhancing model robustness, and protecting data security. Among these, some methods, such as video object detection and temporal modeling, are widely used in industrial monitoring tasks to address the analytical burden brought by multi-camera, multi-angle, and long-duration video data. Another type of research focuses on the stability and generalization ability of models in complex environments. By introducing multimodal learning and sliding window mechanisms, models can better understand the dynamic semantic relationships between keyframes, thereby improving recognition accuracy and reliability under conditions such as changes in lighting, occlusion, and angle shifts.

[0090] Meanwhile, in response to the large amount of sensitive information and privacy security issues present in enterprise-level surveillance videos, various efficient and secure model optimization strategies have emerged in recent years. For example, methods such as low-rank parameter fine-tuning, distributed training, and local fine-tuning enable models to update parameters without uploading the original video data, thus maintaining data security.

[0091] Therefore, this application proposes the goal of establishing a model: an intelligent video recognition system based on the Qwen2-VL multimodal model. This system achieves high-precision identification of violations in complex industrial scenarios by introducing a low-rank adaptation module for efficient fine-tuning and combining it with temporal feature modeling methods. Simultaneously, sensitive corporate information is removed through video editing, effectively protecting corporate privacy.

[0092] Reference Figure 1 This application provides a method for identifying violations of work and construction regulations, specifically including the following steps S100~S160:

[0093] S100: Obtain multiple video clips of violations of work and construction regulations; wherein, the text indicating the dangerous behavior and the timestamp of occurrence are consistent in each video clip;

[0094] S110: Use a pre-trained video-text alignment model to perform semantic alignment between each video segment and the text of dangerous behavior to obtain a semantic alignment dataset;

[0095] S120: Utilize the pre-trained video text alignment model to output a window text similarity score based on the semantic alignment dataset;

[0096] S130: Generate an enhanced training set including positive samples and hard negative samples based on the window text similarity score and the semantic alignment dataset;

[0097] S140: Train the basic multimodal large model using the first o-LoRA module and a large-scale video understanding dataset to obtain the initial multimodal large model;

[0098] S150: Train the initial multimodal large model using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model;

[0099] S160: Input the monitoring video of the operation and construction into the target multimodal large model to identify the violations of the operation and construction.

[0100] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0101] The overall process of this embodiment includes as follows: Figure 2 The three stages shown:

[0102] Training a video-to-text alignment model;

[0103] Video sliding window retrieval and high-quality sample generation;

[0104] Two-stage o-LoRA fine-tuning based on the Qwen2-VL model.

[0105] Specifically, this embodiment includes the following technical solutions:

[0106] 1. Collect data on violations in petrochemical operation areas or construction areas, and perform data preprocessing.

[0107] This embodiment collects surveillance footage from different oil depots showing potentially dangerous or already occurring dangerous behaviors. These footage are labeled with the start and end times of the specific dangerous behaviors by oil depot management personnel. From the approximately 1000+ collected surveillance footage clips, this embodiment randomly selects two groups: a training set and a training set. and test set Proportional division ,Right now ,in This represents the number of data entries in the set.

[0108] In the video segment processing stage, this embodiment uses a sliding window with a fixed length and step size to segment the input video. Specifically, assuming the total duration of the original video is T seconds, the window length is set to L seconds, and the step size is S seconds, the video is divided into several consecutive or partially overlapping time windows. For the first windows (of which) (integer), and its corresponding time range Follow the formula below:

[0109] ;

[0110] in: Total duration of the original video (in seconds); : The length of the sliding window, i.e. the duration of each video segment (unit: seconds); Step size of the sliding window, which is the interval between the start time points of adjacent windows (unit: seconds). The minimum value function sets the window's end time. It will not exceed the total length of the video. .

[0111] The advantage of this design lies in its ability to perform localized analysis of long-term videos while maintaining temporal continuity, avoiding semantic ambiguity and computational redundancy during feature extraction of the entire video segment in a single step. Simultaneously, the sliding window mechanism allows the model to capture event features across segments at boundaries, improving the completeness of dangerous behavior detection. For each segmented video segment, a uniform sampling method is used to extract a certain number of keyframes from each segment as representative frames.

[0112] To address the low resolution and blurry details characteristic of surveillance videos, this embodiment integrates the RealESRGAN super-resolution enhancement module. This module is based on the RRDBNet architecture and includes... Each of the residual dense blocks contains [number] blocks. Each feature growth channel, through complex residual connections and feature reuse mechanisms, achieves image detail restoration and enhancement. The super-resolution model employs a block-based processing strategy during inference, including edge filling. Using half-precision computing, it significantly improves the visual quality of low-resolution surveillance footage while maintaining processing efficiency. Keyframes are processed by this module to enhance the clarity of low-resolution surveillance camera video.

[0113] The annotation information (such as dangerous behavior type, time period, etc.) in each video segment is formatted into text. The text is converted into corresponding word vectors or embedding vectors through tokenization. The keyframes and text label data are synchronized to ensure that each video segment is consistent with its corresponding dangerous behavior text annotation and timestamp.

[0114] 2. Feature encoder and trainable projector design.

[0115] To achieve accurate semantic alignment between surveillance video and textual descriptions of dangerous behaviors, this embodiment constructs a video-text alignment model based on the CLIP (Contrastive Language-Image Pretraining) architecture. To maintain the stability of the multimodal semantic space, the image and text encoding backbone of CLIP is frozen during training, and only the video projection head is aligned. Text projection head The learnable temperature parameter `logit_scale` is used for training and updates. The video and text projection heads employ the same network structure, including LayerNorm layers, a 1024-dimensional linear transformation layer, a GELU activation function, a Dropout layer, and a 512-dimensional output layer. Finally, the output features are L2 normalized to ensure that video and text features are mapped to a unified semantic space.

[0116] To train the model to achieve semantic alignment in this scenario and improve its discrimination ability in dangerous behavior recognition tasks, this embodiment designs a multi-objective loss function that combines global contrastive learning and local hard negative sample mining. The loss function consists of two parts: information contrastive loss. and ranking loss of difficult samples Through weighting coefficients To achieve balance, that is:

[0117] ;

[0118] in: The total loss function for multiple objectives, which is the objective function for optimizing the model; Information contrast loss is used to achieve global semantic alignment. The hard-to-bear sample ranking loss is used to improve the model's ability to distinguish hard-to-bear samples. The weighting factor for ranking loss, used to balance and The relative importance;

[0119] The InfoNCE loss function is used to calculate the bidirectional contrast loss between video and text features. Given a batch size of B training samples, it calculates the similarity matrix between video and text features. The element in the i-th row and j-th column is defined as:

[0120] ;

[0121] in: : in the batch Feature vectors of each video; : in the batch Feature vectors of each text; Temperature parameter; exp(·): exponential function; This represents the similarity between the i-th video feature and the j-th text feature.

[0122] Subsequently, the cross-entropy loss is calculated in two directions: in the video-to-text direction, each row of the similarity matrix is ​​normalized, and the cross-entropy loss between each video and all texts is calculated; in the text-to-video direction, the transpose of the similarity matrix (i.e., each column) is normalized, and the cross-entropy loss between each text and all videos is calculated. Finally, the InfoNCE loss is defined as the average of the losses in both directions, i.e.:

[0123] ;

[0124] Where: CE(·): cross-entropy function; I: label matrix; S: similarity matrix between video features and text features; : The transpose of the similarity matrix; this loss function achieves semantic alignment between video and text by maximizing the similarity between matched videos and text and minimizing the similarity between unmatched samples.

[0125] The hard negative sample ranking loss is specifically designed to address the problem of similar negative examples in dangerous behavior identification. It is designed for each positive sample pair. and several negative sample pairs Defined as:

[0126] ;

[0127] ;

[0128] in: : indicates the first Ranking loss for each positive sample pair; : Indicates the final ranking loss for the entire batch; : Indicates batch size; m: ranking interval constant; : Indicates a positive sample pair The similarity between them; : indicates negative sample pairs The similarity between them.

[0129] The above formula employs the most difficult negative sample strategy. The positive sample window directly uses the time periods of dangerous behavior labeled above, while part of the negative sample window comes from windows in the same video that received high scores in the sliding window retrieval but had low IoU with the manually labeled ones. Another portion comes from windows other than positive samples in the same batch. The union of the two windows These are treated as difficult-to-bear samples, enabling mixed sampling. This design forces the model to learn to distinguish between semantically related but actually different video segments, improving fine-grained recognition capabilities.

[0130] 3. Sliding window retrieval and sample generation.

[0131] To overcome the problem of scarce labeled data for dangerous behaviors, this embodiment proposes an automatic dataset augmentation method based on semantic alignment.

[0132] For the limited amount of manually constructed "video-text-time" weakly labeled data (i.e., each segment of oil depot monitoring video corresponds to a human-computer dialogue and the start and end times of dangerous behaviors), for each window batch, the CLIP backbone and projection head are initialized based on the trained AlignModel. Using the same training method, the window-text similarity (sim) is obtained, and finally, a one-dimensional similarity vector (sims) is obtained by concatenating the similarity vectors across all windows. To overcome the scale differences in the original similarity distribution between different videos, this embodiment performs Min-Max normalization on sims within a single video.

[0133] For positive samples, this embodiment provides two complementary positive sample segment selection modes, which are adapted to different types of dangerous behaviors: (1) Non-maximum suppression (NMS) mode: suitable for short, discrete, and recurring behaviors (such as momentary violations, brief sparks, etc.), emphasizing the selection of multiple high-confidence windows that are dispersed in time. (2) Merging mode: suitable for continuous and interval-type behaviors (such as long-term open flame operations, continuous failure to wear safety equipment, etc.), first merging high-scoring windows in time, and then controlling the segment length. The former performs time NMS on the normalized score, and adaptively selects several high-confidence time segments that do not seriously overlap from the whole video as semantic positive samples under the premise of meeting the minimum duration constraint; the latter merges windows with scores exceeding the set threshold and time adjacent intervals not exceeding the given value into continuous segments in time order, and then segments the excessively long segments according to the maximum duration and step size, thereby obtaining positive sample intervals with both high scores and high overlap.

[0134] Following the same method as before, this embodiment selects windows from the similarity space around these positive samples that are highly relevant to the text but are temporally separated from both the positive and labeled intervals as semantically difficult negative samples. This achieves the filling of the dataset.

[0135] 4. Two-stage fine-tuning of o-LoRA.

[0136] This embodiment, after completing the automatic dataset expansion based on semantic alignment, further introduces a multimodal large model fine-tuning stage to customize the general visual language large model for specific scenarios, enabling it to possess temporal understanding capabilities and the ability to understand dangerous behaviors in oil depot monitoring videos. Overall, it is divided into two stages:

[0137] In the first stage, this embodiment utilizes the o-LoRA module, selecting large-scale video understanding datasets such as InternVid-10M-FLT and ActivityNet Captions. The locally deployed multimodal model Qwen2-VL-2B-Instruct is used as the base model, and fine-tuning is performed on this base model. Through training in this stage, the model learns the temporal logical relationships and semantic coherence across frames, thereby acquiring the ability to reason about events and understand temporal relationships at the video segment level, laying the foundation for subsequent dangerous behavior recognition.

[0138] In the second stage, this embodiment continues to use the o-LoRA module, combining it with a dangerous behavior monitoring video dataset for scenario-based training, enabling the model to identify and distinguish various specific types of dangerous behaviors. Based on the model from the previous stage, a training method combining o-LoRA and low-bit quantization is introduced: while keeping the original large-scale pre-trained parameters basically frozen, only a small number of trainable low-rank branches are added to the key projection matrix in the attention layer.

[0139] In terms of training data construction, this embodiment directly utilizes the automatically labeled dataset generated earlier, treating each record as a supervised sample of "multi-frame image-question-standard answer". For the monitoring video corresponding to each sample, the corresponding time segment is extracted from the video based on the time interval field in the record, and a fixed number of keyframes are uniformly extracted within this time segment as visual input. On the text side, the existing "prompts" in the dataset are concatenated with a uniformly designed task template: the prefix is ​​used to explain "this is a multi-frame capture of oil depot monitoring, and it is necessary to determine whether a certain type of dangerous behavior exists", and the suffix imposes hard constraints on the output format, such as requiring the model to only output the time period string "ZZ behavior was found from XX.XX to YY.YY" when a dangerous behavior is detected, and only output "the behavior was not found" when no dangerous behavior is detected, clearly conveying the task requirements to the model.

[0140] In terms of optimization strategy, this embodiment adopts a mini-batch and gradient accumulation training method based on the large parameter scale of the multimodal model and the difficulty of the oil depot scenario task, combined with cosine annealing learning rate and warm-up phase. During training, optimizers and weight decay strategies are only established for the o-LoRA branch parameters. By separating the normalization layer, bias term, and ordinary weights, the update scale is reasonably constrained, further reducing the perturbation to the original pre-trained representation space. During training, the average loss of each round is continuously monitored, and the current o-LoRA weights and corresponding processor configuration are overwritten and saved only when the loss is better than the historical best value, ensuring that the final set of adapted parameters that performs best on the overall data is retained.

[0141] Through the above-mentioned technical implementation scheme, this embodiment further completes the scenario-based fine-tuning of the multimodal large model based on the automatic data expansion driven by semantic alignment: on the one hand, it enables the model to not only determine whether there are specified dangerous behaviors in oil depot monitoring, but also to give the interval of behavior occurrence in the form of standardized time periods; on the other hand, without significantly increasing the training cost and deployment complexity, it effectively improves the robustness and practicality of the model to complex oil depot scenarios (multiple cameras, multiple time periods, strong background noise), and provides a core model support that can be implemented for the overall intelligent recognition system of dangerous behaviors.

[0142] 5. This embodiment includes the following key technical solutions:

[0143] (1) This embodiment employs a unique sample processing and annotation strategy. It extracts actual violation event segments from enterprise surveillance video and utilizes manual editing and time window slicing techniques to obtain stable keyframe sequences. For samples from different time periods and angles, this embodiment introduces a manual annotation mechanism to generate text description files corresponding to the videos. This manual annotation method ensures the accuracy and relevance of data annotation, effectively improving the model's generalization ability in real-world monitoring environments.

[0144] (2) This embodiment employs a unique sample generation method for the obtained dataset. RealESRGAN super-resolution preprocessing is used to enhance video detail quality. The model employs a joint optimization of bidirectional InfoNCE loss and hard-negative sample ranking loss, ensuring high alignment between video and text embeddings in the semantic space. A contrastive learning mechanism is introduced to enhance the model's discriminative ability.

[0145] (3) This embodiment proposes a video data augmentation strategy based on sliding windows. The video is divided into windows with a fixed step size. The similarity between the window and the searched text is scored using an alignment model. After normalization, the window with the highest score is selected. Overlapping segments are removed by non-maximum suppression to generate multiple high-quality new samples, thereby expanding the dataset and improving sample diversity.

[0146] (4) This embodiment proposes a training strategy based on CLIP training and two-stage fine-tuning, and introduces an anti-forgetting mechanism. The CLIP backbone network is frozen, and only video and text projection heads are trained to maintain semantic consistency; the first fine-tuning stage introduces time segmentation labels to enable the model to have time awareness and temporal understanding capabilities; the second fine-tuning stage fine-tunes on the expanded scene data, and through low-rank parameter updates and orthogonal constraint mechanisms, catastrophic forgetting is effectively prevented, ensuring that the model maintains its original recognition ability and stability while learning new scenes and new behavioral features.

[0147] 6. The beneficial effects of the technical solution in this embodiment:

[0148] Data privacy and security: The training and deployment strategy, which is based on local fine-tuning and low-rank adaptation, ensures that all original surveillance videos are retained in the enterprise's private environment or on edge nodes. This embodiment protects the privacy and security of enterprise production and personnel, and meets the enterprise's compliance and information protection requirements.

[0149] Data Collaboration and Scheduling: This embodiment utilizes a sliding window automatic retrieval, video-text alignment, and sample augmentation mechanism to transform scattered, long-duration video data into high-quality, semantically clear training samples, making data from different work scenarios and different factories easily standardized and usable. Enterprises can share sample features or low-rank adaptation weights based on this method, supporting collaborative training and model updates across work areas and work groups.

[0150] Robustness and Reliability: This embodiment significantly improves the model's robustness in complex industrial environments through multi-layered technical means. RealESRGAN super-resolution is employed to enhance video details and improve feature visibility in distant and low-quality frames; bidirectional InfoNCE and hard-negative sample ranking loss are used to improve the discriminative ability of multimodal embeddings. This significantly improves recognition accuracy and stability under non-ideal conditions, reduces the negative impact of false positives and false negatives on safety decisions, and enhances the system's on-site availability and reliability.

[0151] Balancing personalization and generalization performance: This embodiment proposes a three-stage fine-tuning strategy to balance global generalization and local personalization, taking into account the differences in scenarios across different well sites, refineries, or work teams. This phased approach ensures that the model maintains overall performance while also providing customized improvements for typical local violation patterns during cross-scenario migration, achieving a dynamic balance between personalization and generalization performance.

[0152] Improved burden of manual supervision and efficiency of early warning: Through automated video-text alignment, sliding window retrieval and sample expansion, this embodiment significantly reduces the amount of video that needs to be viewed manually; real-time inference can quickly locate high-risk segments in massive video streams and trigger low-latency alarms, reducing human fatigue and the probability of missed detection caused by too many monitoring probes and too long screens.

[0153] Reference Figure 3 This application provides a system for identifying violations of work and construction regulations, including:

[0154] The video acquisition unit is used to acquire multiple video clips of violations of work and construction regulations; wherein, the text indicating the dangerous behavior and the timestamp of occurrence are consistent in each video clip;

[0155] The semantic alignment unit is used to perform semantic alignment between video segments and dangerous behavior texts using a pre-trained video text alignment model to obtain a semantic alignment dataset.

[0156] The window text similarity determination unit is used to output a window text similarity score based on the semantic alignment dataset using the pre-trained video text alignment model.

[0157] The data augmentation unit is used to generate an augmented training set including positive samples and hard negative samples based on the window text similarity score and the semantic alignment dataset;

[0158] The first model training unit is used to train the basic multimodal large model using the first o-LoRA module and a large-scale video understanding dataset to obtain the initial multimodal large model;

[0159] The second model training unit is used to train the initial multimodal large model using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model;

[0160] The violation identification unit is used to input the monitoring video of the operation and construction into the target multimodal large model to identify the violation of the operation and construction.

[0161] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0162] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0163] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0164] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0166] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0167] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0168] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0169] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0170] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for identifying operational and construction violations, characterized in that, The method includes the following steps: Multiple video clips of violations of work and construction regulations are obtained; wherein the text indicating the dangerous behavior and the timestamp of occurrence are consistent in each video clip; A pre-trained video-text alignment model is used to perform semantic alignment between each video segment and the text of dangerous behavior to obtain a semantic alignment dataset. The pre-trained video text alignment model is used to output a window text similarity score based on the semantic alignment dataset. An augmented training set, including positive and hard negative samples, is generated based on the window text similarity scores and the semantic alignment dataset. The initial multimodal large model is obtained by training the basic multimodal large model using the first o-LoRA module and a large-scale video understanding dataset; The initial multimodal large model is trained using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model; The monitoring videos of the operation and construction are input into the target multimodal large model to identify violations of operation and construction regulations.

2. The method for identifying operational and construction violations according to claim 1, characterized in that, The acquisition of multiple video clips of operational and construction violations includes the following steps: To obtain multiple surveillance videos of potentially dangerous or already dangerous behaviors; Each of the surveillance videos is segmented using a sliding window with a fixed length and a fixed step size, resulting in multiple video segments. The step of segmenting each of the surveillance videos includes: Let the total duration of the surveillance video be T seconds, the window length be L seconds, and the step size be S seconds. Divide the surveillance video into several consecutive or partially overlapping time windows. , No. The time range corresponding to each window Follow the formula below: ； in, To find the minimum value function, the window's end time is... Not exceeding the total duration of the surveillance video. ; A predetermined number of keyframes are extracted from each video segment using a uniform sampling method; The annotation information in each video segment is formatted into text, and the formatted text is converted into corresponding word vectors. The keyframes are synchronized with the word vectors so that each video segment is consistent with the corresponding dangerous behavior text and timestamp. The annotation information includes the type of dangerous behavior and the time period in which it occurred.

3. The method for identifying operational and construction violations according to claim 1, characterized in that, The pre-trained video text alignment model is trained through the following steps: Construct a video text alignment model based on CLIP architecture; During training, the image backbone and text encoding backbone of the CLIP architecture are frozen, and the video projection head is... Text projection head The system also trains and updates the learnable temperature parameter logit_scale. The video projector and the text projector use the same network structure, which includes LayerNorm, linear transformation, GELU activation function, Dropout, and output layer. Finally, the output features are L2 normalized to map the video and text features to a unified semantic space. Construct a multi-objective loss function that combines global contrastive learning and local hard-negative sample mining; the multi-objective loss function includes information contrastive loss. and ranking loss of difficult samples Through weighting coefficients To achieve balance, the calculation formula is: ； in The multi-objective loss function described above; The weighting factor for ranking loss is used to balance... and The relative importance; Constructing information contrast loss This is used to calculate the bidirectional contrast loss between video and text features; given a batch size of B training samples, it calculates the similarity matrix between video features and text features. The element in the i-th row and j-th column is defined as: ； in, The first in the batch Feature vectors of each video; The first in the batch Feature vectors of each text; Here is the temperature parameter; exp(·) is the exponential function; This represents the similarity between the i-th video feature and the j-th text feature; Information comparison loss Defined as: ； Where CE(·) is the cross-entropy function; I is the label matrix; and S is the similarity matrix between video features and text features. This is the transpose of the similarity matrix; Constructing ranking loss for hard-to-bear samples Designed for the similar negative example problem in dangerous behavior identification, for each positive sample pair and several negative sample pairs Defined as: ； ； in, Indicates the first Ranking loss for each positive sample pair; This indicates the final ranking loss for the entire batch; Indicates batch size; m is the ranking interval constant; Indicates positive sample pairs The similarity between them; Indicates negative sample pairs The similarity between them; Difficult to bear sample ranking loss The strategy employs the most difficult negative sample. Positive sample windows use the labeled time periods of dangerous behavior, while a portion of the negative sample windows come from high-scoring windows obtained in the sliding window retrieval within the same video but with low IoU with manually labeled windows. Another portion comes from windows other than positive samples in the same batch. The union of the two windows As the aforementioned hard-to-bear sample.

4. The method for identifying operational and construction violations according to claim 3, characterized in that, The step of using the pre-trained video text alignment model to output window text similarity scores based on the semantic alignment dataset includes the following steps: By utilizing the human-computer dialogue corresponding to each video-text-time monitoring video and the start and end times of the labeled dangerous behavior, for each window batch, the text similarity score sim of the window is trained based on the backbone network and projection head of the CLIP architecture, and finally concatenated on all windows to obtain a one-dimensional similarity vector sims.

5. The method for identifying operational and construction violations according to claim 1, characterized in that, The step of generating an augmented training set including positive and hard negative samples based on the window text similarity score and the semantic alignment dataset includes the following steps: The positive samples are selected using two complementary selection modes, namely, a non-maximum suppression mode and a merging mode. The non-maximum suppression mode is used to select behaviors that are less than a first set duration, discrete, and repetitive, in order to select multiple time-dispersed high-confidence windows. The merging mode is used to achieve a second set duration and interval-type behavior. First, the high-scoring windows are merged in time, and then the segment length is controlled; the second set duration is greater than the first set duration. Around the selected positive samples, windows that are correlated with the text to a set threshold but are temporally separated from both the positive samples and the labeled intervals are selected from the similarity space as the hard negative samples.

6. The method for identifying operational and construction violations according to claim 1, characterized in that, The process of training a basic multimodal large model using the first o-LoRA module and a large-scale video understanding dataset to obtain an initial multimodal large model includes the following steps: InternVid-10M-FLT and ActivityNet Captions were selected as the large-scale video understanding dataset. The basic multimodal large model deployed locally was used to train the basic multimodal large model to obtain the initial multimodal large model.

7. The method for identifying operational and construction violations according to claim 1, characterized in that, The process of training the initial multimodal large model using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model includes the following steps: The first o-LoRA module, the second o-LoRA module, and the enhanced training set are used to add a trainable low-rank branch to the target projection matrix in the attention layer of the initial multimodal large model, and then training is performed to obtain the target multimodal large model.

8. A system for identifying violations of work and construction regulations, characterized in that, The system includes: The video acquisition unit is used to acquire multiple video clips of violations of work and construction regulations; wherein, the text indicating the dangerous behavior and the timestamp of occurrence are consistent in each video clip; The semantic alignment unit is used to perform semantic alignment between video segments and dangerous behavior texts using a pre-trained video text alignment model to obtain a semantic alignment dataset. The window text similarity determination unit is used to output a window text similarity score based on the semantic alignment dataset using the pre-trained video text alignment model. The data augmentation unit is used to generate an augmented training set including positive samples and hard negative samples based on the window text similarity score and the semantic alignment dataset; The first model training unit is used to train the basic multimodal large model using the first o-LoRA module and a large-scale video understanding dataset to obtain the initial multimodal large model; The second model training unit is used to train the initial multimodal large model using the first o-LoRA module, the second o-LoRA module, and the enhanced training set to obtain the target multimodal large model; The violation identification unit is used to input the monitoring video of the operation and construction into the target multimodal large model to identify the violation of the operation and construction.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images