A construction site multi-type safety violation detection method based on security problem guidance and parameter efficient fine-tuning

Abstract

The application discloses a construction site multi-type safety violation detection method based on safety problem guidance and parameter efficient fine-tuning, comprising: constructing a construction safety inspection problem set and converting it into a text prompt; obtaining original single-problem safety question and answer data, integrating multiple safety question answers corresponding to the same construction site image into picture-level multi-label training data; inputting the to-be-detected construction site image and the complete safety problem text prompt into a fine-tuned visual language model to generate a structured Q-label label sequence; performing legal label analysis on the label sequence to obtain a predicted label set, and outputting a construction site multi-type safety violation detection result based on the same. The application can detect multiple types of safety violations such as safety helmet wearing, reflective vest wearing, high-altitude operation safety belt, operation below the hoisted object, excavator operation area intrusion, etc. in one inference, and output a format specification, which is convenient for automatic analysis and deployment, while reducing the model fine-tuning calculation cost and training data requirement.

Description

Technical Field

[0001] This invention belongs to the field of construction site safety violation detection technology based on visual language models, and particularly relates to a method for detecting multiple types of safety violations at construction sites based on safety issues and efficient parameter fine-tuning. Background Technology

[0002] Construction sites are high-risk working environments, typically involving multiple safety risks simultaneously, including personnel movement, machinery operation, work at height, hoisting operations, openings near edges, welding and cutting, and temporary power supply. To reduce accidents, construction companies usually need to continuously inspect aspects such as the wearing of safety helmets, reflective clothing, safety belt use, personnel standing under suspended loads, personnel entering the excavator's operating area, electrical box protection, and ladder operation procedures. Traditional methods rely mainly on manual inspections and the experience of on-site management personnel, which suffers from low inspection efficiency, strong subjectivity, difficulty in continuous coverage, and delayed detection of potential hazards.

[0003] With the development of computer vision technology, existing methods have begun to employ deep learning models such as object detection, image classification, and pose estimation for safety inspection of construction site images or videos. For example, by detecting targets such as personnel, safety helmets, reflective vests, machinery, and guardrails, it is possible to determine whether there are any safety violations. In addition, some research has begun to explore the application of visual language models to construction hazard identification and safety inspection.

[0004] However, while the above methods can identify some safety risks to a certain extent, they still have the following shortcomings in various construction safety violation detection scenarios:

[0005] First, manual inspection is inefficient and makes it difficult to achieve continuous, comprehensive, and real-time safety checks. Construction sites are characterized by dispersed work areas, frequent personnel movement, and rapid changes in equipment operating status. Relying solely on safety management personnel for inspections is easily affected by factors such as experience, workload, inspection frequency, and subjective judgment, leading to missed inspections and inconsistent assessments.

[0006] Second, existing target detection methods typically only identify fixed targets such as personnel, safety helmets, reflective vests, machinery, and guardrails, making it difficult to directly identify complex safety violations. Many construction safety violations are not simply a matter of whether a target exists; rather, they require a comprehensive assessment of the relationships between personnel, equipment, hazardous areas, and operational status. For example, whether personnel are under suspended loads, within the operating range of excavators, or not wearing safety harnesses while working at heights. Traditional target detection methods lack the ability to understand these types of safety rules and spatial relationships.

[0007] Third, existing image classification or multi-label classification methods lack semantic guidance on safety issues, easily simplifying construction safety inspections into a direct mapping from image to label. When multiple safety elements exist in the same image, the model may over-predict, generate numerous false alarms, or miss key violations, making it difficult to accurately express the specific judgment conditions corresponding to each type of safety violation.

[0008] Fourth, although the general visual language model has image understanding and text interaction capabilities, when directly applied to the detection of safety violations at construction sites, it lacks knowledge of construction safety and stable safety inspection rules. It is prone to outputting open-ended natural language descriptions, resulting in problems such as inconsistent answer formats, non-standard labels, and difficulty in automatically parsing and deploying results.

[0009] Fifth, if full-parameter fine-tuning is used to train a large-scale visual language model, it requires high computing resources and a large amount of labeled data, resulting in high training costs. At the same time, it is difficult to obtain samples of construction safety violations, and the number of samples for some violation types is limited. Direct full-parameter fine-tuning is prone to overfitting and may damage the model's original general visual language understanding ability.

[0010] Therefore, there is an urgent need for a construction site safety violation detection method that can combine construction safety inspection rules, support unified detection of multiple types of safety violations, output stable and analyzable results, and has low training costs. Summary of the Invention

[0011] In view of this, the present invention discloses a method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning. It aims to solve the problems of low efficiency and strong subjectivity in existing manual inspections, overcome the shortcomings of existing target detection or image classification methods in simultaneously identifying multiple types of construction safety violations and lacking semantic guidance for safety rules, and improve the current situation of general visual language models lacking knowledge of the construction safety domain, having unstable output formats, and being difficult to automatically parse and deploy results. At the same time, it avoids the problems of high computational cost, large training data requirements, and susceptibility to overfitting in large-scale visual language models that require full parameter fine-tuning.

[0012] The technical solution provided by this invention is specifically a method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning, comprising the following steps:

[0013] S1: Constructing a set of construction safety inspection questions:

[0014]

[0015] In the formula, Represents a set of security issues. Indicates the first One security check issue, This indicates the total number of security issues;

[0016] Transform the complete set of security questions Q into text prompts. ;

[0017] The set of security inspection questions Each security issue is assigned a corresponding structured security label, forming a set of valid labels. ;

[0018] S2: Obtain the original single-question safety Q&A data, and integrate the answers to multiple safety questions corresponding to the same construction site image into an image-level structured label sequence to form image-level multi-label training data;

[0019] S3: Based on the image-level multi-label training data, the pre-trained visual language model is adapted to the construction safety domain using an efficient parameter fine-tuning method.

[0020] S4: Obtain the construction site image to be inspected, and combine the construction site image with a text prompt consisting of a complete set of safety questions. The visual language model, after being efficiently fine-tuned by parameters, is used as a common input to generate a structured Q-label sequence Ŝᵢ;

[0021]

[0022] In the formula, Represents the original parameters of the basic model. This refers to the adaptive parameters obtained through efficient fine-tuning of the parameters.

[0023] S5: For the structured Q-label sequence After parsing the valid labels, the set of predicted labels is as follows:

[0024]

[0025] In the formula, This represents the tag parsing function. This represents the final set of valid predicted labels. Represents a set of legal tags;

[0026] Based on the safety issues and judgment results corresponding to each label in the predicted label set, output the detection results of multiple types of safety violations at the construction site.

[0027] Preferably, the set of security inspection questions is... Each security issue in the document is assigned a corresponding structured security label:

[0028] Each security question corresponds to two judgment results: Yes and No. Yes indicates that the violation status corresponding to the security question exists, while No indicates that the violation status corresponding to the security question does not exist or has not been detected.

[0029] The structured security label set is represented as:

[0030]

[0031] In the formula, Represents the set of all candidate security labels. Indicates the first The corresponding violation status exists for each security issue. Indicates the first The violation status corresponding to the security issue does not exist or has not been discovered.

[0032] Preferably, S2 specifically includes:

[0033] The original single-question security Q&A data format is as follows:

[0034]

[0035] In the formula, Indicates the first Zhang's construction site photos, Indicates the first One security check issue, This indicates that the image represents a binary answer to the question;

[0036] For the same construction site image, the answers to multiple safety questions are integrated to form a structured tag set:

[0037]

[0038] Each reconstructed training sample consists of a construction site image, a safety issue prompt, and a target structured label sequence, which can be represented as:

[0039]

[0040] In the formula, Indicates the first training samples, Images representing the construction site. A security prompt is displayed. This represents the target structured label sequence.

[0041] Preferably, S3 specifically comprises:

[0042] A pre-trained visual language model is used as the base model, including a visual encoder, a multimodal projection layer, and a language decoder;

[0043] The prediction process of the visual language model is represented as follows:

[0044]

[0045] In the formula, The parameter is Visual language model, This represents the sequence of label text generated by the model;

[0046] The original weight matrix is ​​denoted as Using a low-rank increment matrix Adapt the original weight matrix:

[0047]

[0048] The low-rank increment matrix can be represented as:

[0049]

[0050] In the formula, and For trainable low-rank matrices, Denotes the rank of a low-rank matrix. Indicates the scaling factor;

[0051] Let the input features be Then the output after adding the low-rank adaptor is:

[0052]

[0053] In the formula, Keep the system frozen during training, only update. and Two low-rank matrices.

[0054] Preferably, the visual language model employs a supervised fine-tuning loss function for efficient parameter fine-tuning training, the loss function being expressed as:

[0055] The model input is a construction site image. Security Prompt The target output is a structured label sequence. Let the target output sequence be:

[0056]

[0057] In the formula, Indicates the first in the target output sequence One token, The target sequence length is represented by the supervised fine-tuning loss function, which is:

[0058]

[0059] In the formula, Indicates the first The previous output content for each token, Indicates the model in parameters The probability of generating the target token;

[0060] For including The dataset contains training samples, and the overall training objective is represented as:

[0061] .

[0062] Preferably, the safety inspection problem set Q includes at least the following safety problems corresponding to multiple inspection items: violation of wearing a safety helmet, violation of wearing a reflective vest, violation of wearing a safety belt for working at heights, violation of working under a suspended object, violation of intrusion into the excavator's work area, violation of ladder operation, violation of welding and cutting protection, and violation of electrical box protection.

[0063] This invention also provides a multi-type safety violation detection system for construction sites based on safety issue guidance and efficient parameter fine-tuning, including:

[0064] The safety issue guidance module is used to construct a set of construction safety inspection questions Q, and then convert the complete set of safety issues Q into text prompts. ;

[0065] The tag system construction module is used to assign a corresponding structured security tag to each security question in the security inspection question set Q;

[0066] The image-level multi-label data construction module is used to acquire raw single-question safety Q&A data and integrate multiple safety question answers corresponding to the same construction site image into an image-level structured label sequence to form image-level multi-label training data.

[0067] The parameter fine-tuning module is used to freeze the parameters of the visual encoder and multimodal projection module of the pre-trained visual language model, and only introduces a low-rank adaptation matrix in the language decoder for adaptation in the construction safety domain.

[0068] The construction site image acquisition module is used to acquire images of the construction site to be inspected.

[0069] The visual language model inference module is used to compare the image of the construction site to be detected with the text prompt. The visual language model, after being efficiently fine-tuned by parameters, is used as a common input to generate a structured label sequence;

[0070] The tag parsing output module is used to parse the generated structured tag sequence into valid tags, extract tags that match the structured safety tags, form a predicted tag set, and output the detection results of multiple types of safety violations at the construction site based on the safety issues and judgment results corresponding to each tag in the predicted tag set.

[0071] Preferably, the system outputs a set of structured tags for detecting various types of safety violations at the construction site. Each tag in the set corresponds to a judgment result for a safety inspection issue, which is used to support the automated safety management system in further processing the detection results.

[0072] The beneficial effects of this invention are:

[0073] This invention constructs a set of construction safety inspection questions, transforming various safety inspection rules into clear text prompts. This enables the visual language model to make unified judgments on multiple types of safety violations at the construction site under the guidance of safety questions, overcoming the shortcomings of existing target detection or image classification methods that are difficult to identify multiple types of construction safety violations simultaneously.

[0074] This invention adapts a visual language model to the construction safety domain through efficient parameter fine-tuning. By freezing most of the original parameters, only a small number of adapted parameters are trained. Under conditions of limited training data and low computational cost, it can learn the correspondence between construction site image evidence, semantics of safety issues, and structured safety labels, effectively improving the ability to identify complex safety violations. At the same time, it avoids the problems of high computational cost, large data requirements, and easy overfitting caused by full parameter fine-tuning.

[0075] This invention constrains the output format of the model through a structured security label system, making the detection results standardized and parsable. This facilitates subsequent automatic parsing, statistical analysis, alarm prompts, and deployment of security management systems, solving the problems of unstable output format and difficulty in practical application when general visual language models are directly applied to security detection.

[0076] This invention utilizes a core set of safety issues, image-level multi-label data reconstruction, efficient parameter fine-tuning, and structured label parsing as its key technologies. It differs from existing construction safety inspection methods based on knowledge base retrieval or safety specification text recall, as well as existing safety equipment wearing compliance detection methods based on open-vocabulary target detection and visual text similarity calculation. Furthermore, it differs from existing multi-agent collaborative risk warning methods and construction safety monitoring methods based on behavioral feature vector comparison. This invention can improve the coverage, detection stability, and result parsing of various types of safety violations detected at construction sites, reduce the cost of fine-tuning large models, and provide technical support for intelligent safety inspections and automated safety management at construction sites.

[0077] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the present invention. Attached Figure Description

[0078] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0079] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 A schematic diagram of a method for detecting multiple types of safety violations at construction sites based on safety issues and efficient parameter fine-tuning, provided by the present invention;

[0081] Figure 2 A schematic diagram of the security issue guidance prompt and structured output provided for this invention;

[0082] Figure 3 A schematic diagram illustrating the correspondence between construction site images and structured safety tags provided by this invention;

[0083] Figure 4 This is a schematic diagram of visual disturbance data of construction site images provided by the present invention. Detailed Implementation

[0084] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems consistent with some aspects of the invention as detailed in the appended claims.

[0085] This invention belongs to the field of construction site safety violation detection based on visual language models, specifically involving the identification of multiple types of safety violations at construction sites, the output of structured safety labels, and the technology for efficient fine-tuning of visual language model parameters.

[0086] Among existing methods for detecting safety violations at construction sites, those based on target detection or image classification are difficult to cover multiple types of violations simultaneously; traditional visual models lack semantic guidance for safety rules, making it difficult to understand the spatial relationships and operational status between personnel, equipment, and hazardous areas; general visual language models lack knowledge of the construction safety domain, resulting in unstable output formats, difficult structured analysis of results, and engineering deployment when applied directly; and if full parameter fine-tuning is used, problems such as high computational cost, large training data requirements, and susceptibility to overfitting are encountered.

[0087] To address the aforementioned issues, this invention provides a method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning. It constructs a set of construction safety inspection questions, transforming safety inspection rules into text prompts to guide a visual language model in directional judgment. A structured safety tagging system enables parsable output of detection results. Through efficient parameter fine-tuning, only a small number of adaptive parameters are trained while most original parameters are frozen, achieving adaptation to the construction safety domain. This invention can simultaneously detect multiple types of safety violations, including helmet wearing, reflective vest wearing, working at heights, hoisting operations, ladder operations, welding and cutting operations, electrical box protection, and edge protection.

[0088] The detection method provided by this invention includes constructing a set of construction safety issues, constructing a Q-label structured labeling system, reconstructing raw single-issue VQA data into image-level multi-label data, efficiently fine-tuning visual language model parameters, and parsing and outputting valid labels. Through the above process, this invention can perform multiple safety checks on construction site images based on a complete safety issue Prompt in a single inference, and output multiple standardized safety labels, thereby achieving unified detection of multiple types of construction safety violations.

[0089] Specifically, it includes:

[0090] 1) Construct a set of security issues:

[0091] A set of safety issues is constructed based on common safety inspection items at construction sites. This set of safety issues can be represented as follows:

[0092]

[0093] In the formula, Represents a set of security issues. Indicates the first One security check issue, This indicates the total number of security issues. Preferably, the present invention can be set to... This involves constructing 16 on-site safety inspection issues, from Q0 to Q15.

[0094] Examples of security issues are shown in Table 1.

[0095] Table 1: Examples of Security Issue Sets

[0096]

[0097] By using the aforementioned set of safety issues, this invention explicitly writes the safety inspection rules for construction sites into the model input, enabling the visual language model to make directional judgments on images based on specific safety rules, rather than making unconstrained open descriptions of image content.

[0098] 2) For each security issue, this invention sets a corresponding structured security label. Each security issue corresponds to two judgment results: Yes and No. Yes indicates that the violation status corresponding to the security issue exists, and No indicates that the violation status corresponding to the security issue does not exist or has not been detected. The set of structured security labels can be represented as:

[0099]

[0100] In the formula, Represents the set of all candidate security labels. Indicates the first The corresponding violation status exists for each security issue. Indicates the first The violation status corresponding to the security issue does not exist or has not been discovered.

[0101] For a construction site image Its true set of labels can be represented as:

[0102]

[0103] For example, when an image does not show any instances of not wearing a helmet, but does show instances of not wearing reflective clothing and not wearing a safety belt while working at height, its tag set can be represented as:

[0104]

[0105] Through the aforementioned labeling system, this invention restricts the output of the visual language model to a standardized Q-label format, avoiding the problems of inconsistent format, difficulty in parsing, and inconvenience in deployment that exist in natural language output.

[0106] Figure 3 This image provides an example of a construction site image and its structured safety labels. The image illustrates the structured safety labels corresponding to different construction site images, such as Q0-No, Q1-Yes, Q3-Yes, and Q8-Yes. This method allows each construction site image to correspond to one or more safety inspection labels, thus supporting a unified expression of multiple types of safety conditions or multiple types of safety violations within a single image. This step, by constructing a safety issue guidance mechanism, transforms construction site safety inspection rules into a clear set of safety issues, enabling the visual language model to perform image understanding and violation judgment around specific inspection items such as wearing safety helmets, wearing reflective clothing, working at heights, hoisting operations, ladder operations, welding operations, and electrical box protection. By constructing a structured safety label system, each safety inspection issue is mapped to two types of Q-labels: Qj-Yes and Qj-No, allowing the detection results of multiple types of safety violations at the construction site to be output in a unified format.

[0107] 3) Construct image-level multi-label training data:

[0108] The original single-question safety Q&A data format: Existing construction safety visual Q&A data typically uses a single-question binary classification format, where a construction site image and a safety question are input together, and the model outputs Yes or No. Its format can be represented as:

[0109]

[0110] In the formula, Indicates the first Zhang's construction site photos, Indicates the first One security check issue, This indicates that the image represents a binary answer to the question.

[0111] This format can only answer one safety question at a time, which is not suitable for the actual construction site's detection needs where "one image may contain multiple safety conditions or multiple violations at the same time".

[0112] Reconstructing Image-Level Multi-Label Data: This invention reconstructs original single-question safety Q&A data into image-level multi-label training data. For the same construction site image, the answers to multiple corresponding safety questions are integrated to form a structured label set.

[0113]

[0114] Each reconstructed training sample consists of a construction site image, a safety issue prompt, and a target structured label sequence, which can be represented as:

[0115]

[0116] In the formula, Indicates the first training samples, Images representing the construction site. A security prompt is displayed. This represents the target structured label sequence.

[0117] Examples of training sample construction are shown in Table 2.

[0118]

[0119] Through the above data reconstruction method, this invention transforms construction safety inspection from a single question-and-answer task into an image-level multi-label generation task, enabling the model to output multiple safety inspection results simultaneously in a single inference.

[0120] 4) The efficient fine-tuning method for visual language model parameters involved in this invention specifically includes:

[0121] This invention uses a pre-trained visual language model as the base model. This model includes a visual encoder, a multimodal projection layer, and a language decoder. The visual encoder extracts visual features from construction site images, the multimodal projection layer maps these visual features to a feature space understandable by the language model, and the language decoder generates a sequence of safety labels based on image features and a safety prompt.

[0122] The prediction process of a basic visual language model can be represented as follows:

[0123]

[0124] In the formula, The parameter is Visual language model, This represents the sequence of label text generated by the model.

[0125] Principle of Efficient Parameter Fine-Tuning: Due to the large number of parameters in a visual language model, directly performing full parameter fine-tuning would result in significant computational resource consumption and training costs. Therefore, this invention employs an efficient parameter fine-tuning method, freezing most of the original model's parameters and training only a small number of suitable parameters.

[0126] For a given linear transformation layer in the model, the original weight matrix is ​​denoted as... This invention employs a low-rank increment matrix. Adapt it to:

[0127]

[0128] The low-rank increment matrix can be represented as:

[0129]

[0130] In the formula, and For trainable low-rank matrices, Denotes the rank of a low-rank matrix. This represents the scaling factor. Because... The dimensions are much smaller than the original weight matrix, thus significantly reducing the number of training parameters.

[0131] Let the input features be Then the output after adding the low-rank adaptor is:

[0132]

[0133] In the formula, Keep the system frozen during training, only update. and Two low-rank matrices. This approach allows the model to learn construction safety rules and structured label output methods while maintaining its original ability to understand general visual language.

[0134] During training, the model input is images of the construction site. Security Prompt The target output is a structured label sequence. Let the target output sequence be:

[0135]

[0136] In the formula, Indicates the first in the target output sequence One token, This represents the length of the target sequence. The supervised fine-tuning loss function is expressed as:

[0137]

[0138] In the formula, Indicates the first The previous output content for each token, Indicates the model in parameters The probability of generating the target token.

[0139] For including The dataset of training samples can be represented by the overall training objective as follows:

[0140]

[0141] By minimizing the above loss function, the model can learn the correspondence between construction site image evidence, safety issue semantics, and structured safety labels.

[0142] 5) Structured tag generation and parsing: By designing a structured tag parsing method, the output results of the visual language model are processed to extract legal tags, filter duplicate tags, and remove illegal outputs, thereby converting the model-generated text into stable, statistical, and deployable construction safety violation detection results.

[0143] Generate structured tags:

[0144] During the inference phase, input images of the construction site. Security Prompt The visual language model, after efficient parameter fine-tuning, generates a sequence of label texts.

[0145]

[0146] In the formula, Represents the original parameters of the basic model. This indicates the adaptive parameters obtained through efficient fine-tuning of the parameters.

[0147] Ideally, the model output consists of several valid structured labels, for example:

[0148]

[0149] To ensure that the model output can be used for engineering deployment, this invention sets a set of legal labels. The model-generated text is then parsed. The parsed set of predicted labels can be represented as:

[0150]

[0151] In the formula, This represents the tag parsing function. This represents the final set of valid predicted labels.

[0152] The tag parsing process includes: segmenting the output text according to a preset delimiter, identifying tags that conform to the format of "problem number-judgment result", filtering out illegal outputs that do not belong to the legal tag set, retaining only the first occurrence of duplicate tags, and finally forming a standardized predicted tag set.

[0153] Examples of tag parsing are shown in Table 3.

[0154] Table 3: Examples of Tag Parsing and Violation Results

[0155]

[0156] The overall architecture of the model described in this invention is shown in Figure 1. It mainly includes a construction site image input module, a safety issue guidance module, an image-level multi-label data construction module, a visual language model module, a parameter efficient fine-tuning module, a structured label generation module, and a label parsing and output module. Figure 1 illustrates the complete process of this invention from construction site image input, safety issue set construction, image-level multi-label training sample construction, visual language model parameter efficient fine-tuning, to the final output of structured safety labels. Specifically, the model obtains the detection task requirements through the safety issue guidance prompt and learns the mapping relationship between construction site image evidence, safety issue semantics, and structured Q-labels through the parameter efficient fine-tuning module.

[0157] The system comprises the following modules: a construction site image input module for acquiring images or video frames of the construction site; a safety issue guidance module for constructing a set of construction safety inspection questions and converting safety inspection rules into text questions; an image-level multi-label data construction module for integrating raw single-question safety Q&A data into an image-level structured label sequence; a visual language model module for simultaneously understanding construction site image information and safety issue text information; a parameter efficient fine-tuning module for domain adaptation of the visual language model under limited construction safety data conditions; a structured label generation module for outputting a standardized safety label sequence; and a label parsing output module for extracting legal labels and outputting the final safety violation detection results.

[0158] This invention represents the various types of safety violation detection tasks at construction sites as follows:

[0159]

[0160] In the formula, Indicates the first Zhang's construction site photos, A text prompt indicating a set of security questions. This represents the set of structured security labels predicted by the model.

[0161] By combining safety issue guidance, image-level multi-label data reconstruction, efficient parameter fine-tuning, and structured label parsing, a complete technical process suitable for detecting various types of safety violations on construction sites is formed, improving detection accuracy, output stability, and engineering application value.

[0162] 6) Dataset: This invention uses a construction site safety violation detection dataset to validate the proposed method. The main dataset is derived from visual question-and-answer data on construction site safety compliance inspections. The original data contains 2386 construction site images, of which 2384 images have valid safety labels, and 2 images lacking valid labels were not included in the training and testing process. The data covers typical construction safety scenarios such as personnel operations, machinery operation, work at heights, hoisting operations, ladder operations, welding and cutting operations, edge protection, and electrical box protection.

[0163] This invention divides the dataset into training, validation, and test sets at the image level to avoid data leakage caused by the same image appearing in different data partitions simultaneously. The training set contains 1669 images and 5598 structured labels, the validation set contains 357 images and 1180 structured labels, and the test set contains 358 images and 1187 structured labels. Furthermore, to verify adaptability in complex visual environments, this invention constructs a self-built single-label test set and five types of visual perturbation test sets, including perturbation types such as backlighting, dust, fisheye distortion, rain interference, and localized strong glare.

[0164] The dataset is divided as shown in Table 4.

[0165] Table 4: Data Classification of Construction Site Safety Violation Detection Datasets

[0166]

[0167] By using the above data partitioning method, this invention can avoid data leakage during image-level training, verification, and testing, and can also verify the model's generalization ability and robustness in complex construction scenarios through independent test sets and visual perturbation test sets.

[0168] In the experiment, this invention selected Qwen2.5-VL-3B-Instruct as the basic visual language model and adopted the LoRA parameter efficient fine-tuning method for adaptation to the construction safety domain. The training samples consisted of construction site images, safety question prompts, and structured Q-label sequences. During training, the visual encoder and multimodal projection module were frozen to retain the general image understanding ability of the basic model; low-rank adaptation parameters were introduced into the attention module and feedforward network module of the language decoder to enable the model to learn the mapping relationship between construction site image evidence, safety question semantics, and structured safety labels.

[0169] Figure 2 illustrates the difference between security question-guided prompts and structured output. This figure demonstrates the impact of different prompt formats on the model's security detection task, including no prompt, category-level prompts, and complete Q0–Q15 security question prompts. It also shows the difference between natural language description output and Q-label structured label output. Compared to natural language descriptions, Q-label output is more standardized, facilitating subsequent label parsing, metric evaluation, and engineering deployment.

[0170] During the inference phase, the model input consists of images of the construction site and a complete set of safety questions, and the output is a structured Q-label sequence. Subsequently, the label parsing module performs legal label extraction, illegal label filtering, duplicate label removal, and violation result determination on the model output.

[0171] The experimental setup is shown in Table 5.

[0172] Table 5: Experimental Setup

[0173]

[0174] 7) Evaluation indicators:

[0175] This invention can use metrics such as Exact-Set-Match, Micro-F1, Macro-F1, and Yes-Recall to evaluate model performance. Among them, Exact-Set-Match measures whether the predicted label set is completely consistent with the true label set, and its calculation formula is as follows:

[0176]

[0177] In the formula, Indicates the number of test samples. Indicates the first The set of predicted labels for each sample. Represents the set of real labels. This indicates an indicator function.

[0178] Micro-Precision, Micro-Recall, and Micro-F1 are used to evaluate the model's predictive performance at the overall label level, and their calculation formulas are as follows:

[0179]

[0180]

[0181]

[0182] In the formula, This represents the number of correctly predicted labels. This indicates the number of incorrectly predicted labels. This indicates the number of labels that were missed during the detection process.

[0183] Since on-site safety violation detection focuses more on whether actual violations are detected in a timely manner, this invention further employs a Yes-Recall index model to evaluate the recall capability of violation-related tags:

[0184]

[0185] In the formula, This indicates the number of correctly identified violation tags. This indicates the number of non-compliant labels that were missed during inspection.

[0186] To verify the effectiveness of the proposed method, it was compared with an un-fine-tuned visual language model, a few-shot Prompt-assisted visual language model, a traditional multi-label classification model, and a visual-text matching model. Experimental results show that while the un-fine-tuned visual language model possesses some general image understanding capabilities, it struggles to consistently output legitimate structured labels in construction site safety violation detection tasks. The method relying solely on Prompt prompts, while guiding the model to output the target format to some extent, is prone to class bias and missed violation detections. Traditional multi-label classification and visual-text matching models, although capable of outputting multi-label results, lack semantic guidance regarding safety issues, making them susceptible to over-prediction or missed detections of complex violations.

[0187] In contrast, the method of this invention, through security-related guidance and efficient parameter fine-tuning, enables the visual language model to stably output structured security labels and achieves better overall performance in various security violation detection tasks. Table 6 shows a performance comparison of different methods on independent test sets.

[0188] Table 6: Performance Comparison of Different Methods in Construction Site Safety Violation Detection Tasks

[0189]

[0190] Table 6 shows that the untuned visual language model can hardly complete this task directly under strictly structured output conditions, indicating that the general visual language model lacks the ability to adapt to the construction safety domain and provide structured output. While the few-sample Prompt visual language model can output a large number of labels, its Yes-Recall is only 0.0091, indicating its weak ability to identify true violations and its tendency to bias towards outputting non-violation results. The traditional multi-label classification model has a Micro-F1 score of 0.6788, demonstrating some detection capability, but its Exact score is only 0.2101, indicating its difficulty in completely predicting all safety labels in the image.

[0191] The CLIP and SigLIP linear classification models have high Yes-Recall scores of 0.9472 and 0.9490, respectively, but their predicted label counts of 2951 and 2663 are significantly higher than the actual number of labels in the test set, indicating that these methods suffer from overprediction. In contrast, the method of this invention predicts 1187 labels, which is close to the actual number of labels in the test set. Furthermore, its Exact score reaches 0.5854, Micro-F1 score reaches 0.8678, Macro-F1 score reaches 0.8633, and Yes-Recall reaches 0.8798. This demonstrates that the method of this invention can effectively control the number of false positives while maintaining violation recall capabilities and improving overall multi-label detection performance.

[0192] Furthermore, to verify the adaptability of the method of this invention in complex construction site environments, this invention constructed five types of visual disturbance data—rainy weather, dust, backlight, localized strong glare, and fisheye distortion—on a self-built construction safety test set, and evaluated the robustness of the model. The robustness test results are shown in Table 7.

[0193] Table 7: Robustness test results under different visual perturbation conditions

[0194]

[0195] As shown in Table 7, on the original image, the Top-4 hit rate of the method of this invention is 0.7244, and the total output hit rate is 0.8297; under the overall conditions of five types of visual perturbations, the Top-4 hit rate is 0.7205, and the total output hit rate is 0.8171. Compared with the original image, the Top-4 hit rate of the five types of perturbations decreased by only 0.0039, and the total output hit rate decreased by 0.0126, indicating that the method of this invention can still maintain relatively stable detection capabilities under complex construction site visual conditions such as rain, dust, backlight, local strong glare, and fisheye distortion. Among them, rainy weather interference has a relatively large impact on the model, with the Top-4 hit rate decreasing to 0.7109, but the overall recognition effect is still good.

[0196] Figure 4 shows the original construction site image and the visual disturbances such as backlighting, dust, fisheye distortion, rain interference, and localized strong glare. By constructing the aforementioned disturbance data, the robustness and adaptability of the method of this invention in complex construction site environments can be verified.

[0197] In summary, the experimental results demonstrate that the method of this invention, through safety issue guidance and efficient parameter fine-tuning, can effectively improve the accuracy, stability, and robustness of detecting various types of safety violations at construction sites. Compared with untuned visual language models, prompt-only methods, traditional multi-label classification models, and visual text matching models, the method of this invention has significant advantages in structured label output, violation recall, false alarm control, and adaptability to complex environments.

[0198] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

Claims

1. A method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning, characterized in that, Includes the following steps: S1: Construct a set of construction safety inspection questions: ； In the formula, Represents a set of security issues. Indicates the first One security check issue, This indicates the total number of security issues; Transform the complete set of security questions Q into text prompts. ; The set of security inspection questions Each security issue is assigned a corresponding structured security label, forming a set of valid labels. ; S2: Obtain the original single-question safety Q&A data, and integrate the answers to multiple safety questions corresponding to the same construction site image into an image-level structured label sequence to form image-level multi-label training data; S3: Based on the image-level multi-label training data, the pre-trained visual language model is adapted to the construction safety domain using an efficient parameter fine-tuning method. S4: Obtain the construction site image to be inspected, and combine the construction site image with a text prompt consisting of a complete set of safety questions. The visual language model, after being efficiently fine-tuned by parameters, is used as a common input to generate a structured Q-label sequence Ŝᵢ; ； In the formula, Represents the original parameters of the basic model. This refers to the adaptive parameters obtained through efficient fine-tuning of the parameters. S5: For the structured Q-label sequence After parsing the valid labels, the set of predicted labels is as follows: ； In the formula, This represents the tag parsing function. This represents the final set of valid predicted labels. Represents a set of legal tags; Based on the safety issues and judgment results corresponding to each label in the predicted label set, output the detection results of multiple types of safety violations at the construction site.

2. The method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning as described in claim 1, characterized in that, The set of security inspection questions Each security issue in the document is assigned a corresponding structured security label: Each security question corresponds to two judgment results: Yes and No. Yes indicates that the violation status corresponding to the security question exists, while No indicates that the violation status corresponding to the security question does not exist or has not been detected. The structured security label set is represented as: ； In the formula, Represents the set of all candidate security labels. Indicates the first The corresponding violation status exists for each security issue. Indicates the first The violation status corresponding to the security issue does not exist or has not been discovered.

3. The method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning as described in claim 1, characterized in that, S2 specifically includes: The original single-question security Q&A data format is as follows: ； In the formula, Indicates the first Zhang's construction site photos, Indicates the first One security check issue, This indicates that the image represents a binary answer to the question; For the same construction site image, the answers to multiple safety questions are integrated to form a structured tag set: ； Each reconstructed training sample consists of a construction site image, a safety issue prompt, and a target structured label sequence, which can be represented as: ； In the formula, Indicates the first training samples, Images representing the construction site. A security prompt is displayed. This represents the target structured label sequence.

4. The method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning as described in claim 1, characterized in that, Specifically, S3 is: A pre-trained visual language model is used as the base model, including a visual encoder, a multimodal projection layer, and a language decoder; The prediction process of the visual language model is represented as follows: ； In the formula, The parameter is Visual language model, This represents the sequence of label text generated by the model; The original weight matrix is ​​denoted as Using a low-rank increment matrix Adapt the original weight matrix: ； The low-rank increment matrix can be represented as: ； In the formula, and For trainable low-rank matrices, Denotes the rank of a low-rank matrix. Indicates the scaling factor; Let the input features be Then the output after adding the low-rank adaptor is: ； In the formula, Keep frozen during training, only update. and Two low-rank matrices.

5. The method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning as described in claim 1, characterized in that, The visual language model employs a supervised fine-tuning loss function for efficient parameter fine-tuning training. The loss function is expressed as follows: The model input is a construction site image. Security issues Prompt The target output is a structured label sequence. Let the target output sequence be: ； In the formula, Indicates the first in the target output sequence One token, The target sequence length is represented by the supervised fine-tuning loss function, which is: ； In the formula, Indicates the first The previous output content for each token, Indicates the model in parameters The probability of generating the target token; For including The dataset contains training samples, and the overall training objective is represented as: 。 6. The method for detecting multiple types of safety violations at construction sites based on safety issue guidance and efficient parameter fine-tuning as described in claim 1, characterized in that, The safety inspection problem set Q includes at least the following types of safety issues: violation of helmet wearing, violation of reflective clothing wearing, violation of safety belt wearing for working at heights, violation of working under suspended objects, violation of intrusion into the excavator's work area, violation of ladder operation, violation of welding and cutting protection, and violation of electrical box protection.

7. A multi-type safety violation detection system for construction sites based on safety issue guidance and efficient parameter fine-tuning, characterized in that, include: The safety issue guidance module is used to construct a set of construction safety inspection questions Q, and then convert the complete set of safety issues Q into text prompts. ; The tag system construction module is used to assign a corresponding structured security tag to each security question in the security inspection question set Q; The image-level multi-label data construction module is used to acquire raw single-question safety Q&A data and integrate multiple safety question answers corresponding to the same construction site image into an image-level structured label sequence to form image-level multi-label training data. The parameter fine-tuning module is used to freeze the parameters of the visual encoder and multimodal projection module of the pre-trained visual language model, and only introduces a low-rank adaptation matrix in the language decoder for adaptation in the construction safety domain. The construction site image acquisition module is used to acquire images of the construction site to be inspected. The visual language model inference module is used to compare the image of the construction site to be detected with the text prompt. The visual language model, after being efficiently fine-tuned by parameters, is used as a common input to generate a structured label sequence; The tag parsing output module is used to parse the generated structured tag sequence into valid tags, extract tags that match the structured safety tags, form a predicted tag set, and output the detection results of multiple types of safety violations at the construction site based on the safety issues and judgment results corresponding to each tag in the predicted tag set.

8. The system according to claim 7, characterized in that, The system outputs a set of structured tags for detecting various types of safety violations at the construction site. Each tag in the set corresponds to a judgment result for a safety inspection issue, which is used to support the automated safety management system in further processing the detection results.

Images