End-to-end construction hazard identification and report generation system and method based on visual language models

Abstract

The present invention provides systems and methods for automated end-to-end construction hazard identification and report generation based on vision-language models (VLMs). The methods include: preparing a construction hazard dataset for training and evaluation; fine-tuning a pre-trained VLM with the prepared training dataset to obtain a VLM-enabled construction hazard identifier; identifying one or more hazards in a construction site by the VLM-enabled construction hazard identifier; and generating an end-to-end real-time construction hazard report based on the identified construction hazards by a large language model. Evaluation results show that the VLM-enabled construction hazard identifier outperforms advanced and traditional pre-trained VLMs while exhibiting strong robustness and few-shot learning capabilities. The application of the present invention in actual construction projects shows that it can generate well-organized and accurate hazard reports.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 782,616, filed April 3, 2025, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] This invention generally relates to engineering management technology, and more specifically, to an automated end-to-end method and system for identifying and generating reports on construction hazards based on a vision-language model (VLM). Background Technology

[0003] The construction industry has long been considered one of the high-risk industries [1, 2]. Statistics show that 21.16% of all fatal workplace accidents in the United States in 2020 occurred in the construction industry [3]. These accidents are mainly caused by a large number of inherent hazards in the construction environment [4]. Studies have shown that about 80% of construction-related accidents can be traced back to unsafe worker behaviors [5, 6]. To reduce such risks, the traditional behavior-based safety (BBS) approach emphasizes the combination of manual inspection and immediate corrective measures [7] and is considered an effective management practice [3]. However, such methods are inefficient, susceptible to human error, and lack continuous monitoring capabilities [8, 9]. In recent years, the advancement of computer vision (CV) technology and the deployment of complex construction site monitoring systems have alleviated some of the limitations of manual inspection [10-12]. However, existing solutions often only identify a limited number of common construction hazards and rely on complex rule-based decision-making processes. In addition, since the supervised learning paradigm relies entirely on labeled datasets in model training, its generalization ability is significantly limited, making it difficult to effectively cope with unseen complex and dynamic construction scenarios [9, 14]. Therefore, there is an urgent need for a framework that utilizes advanced scene understanding and perception technologies to achieve accurate and comprehensive identification of various construction hazards. Table 1 summarizes previous research on CV (Computer Vision) identification of different construction hazards.

[0004] Table 1. Previous studies on the use of CV technology for construction hazard identification.

[0005] The key to achieving these goals lies in a comprehensive and in-depth understanding of the construction context and in combining advanced reasoning capabilities to identify potential construction hazards. In recent years, research on visual language models (VLM)

[15] has made significant progress. VLM is a type of AI model that integrates computer vision and natural language processing technologies, and is capable of understanding visual and textual information as a whole. Therefore, VLM combines visual understanding, human-like knowledge, and powerful reasoning capabilities. Typical examples include CLIP

[16] and BLIP

[17] . Compared with traditional computer vision methods, VLM exhibits higher robustness and generalization ability when dealing with complex scenarios

[18] . In addition, VLM's self-supervised learning mechanism enables it to learn knowledge from unlabeled data, thereby reducing its dependence on manually labeled datasets[19,20]. In the field of construction, Liang et al.

[21] proposed a CLIP-based few-shot learning algorithm for identifying objects at temporary construction sites; Tsai et al.

[18] combined CLIP with GPT-2 for safety hazard detection; Chen et al.

[22] integrated CLIP with 3D-based ResNet for excavator activity recognition under limited datasets; Yong et al.

[23] used BLIP and data augmentation techniques to identify ergonomic problems. Despite these advances, existing methods still face several challenges in construction hazard identification. First, they are difficult to achieve end-to-end output of hazard categories, such as directly extracting hazards from construction videos and generating reports

[24] . Second, the types of hazards at construction sites are complex and diverse, including unattended rotating mechanical parts and randomly piled materials, which requires more refined hazard identification capabilities. Finally, traditional VLMs such as CLIP have limitations in semantic understanding of complex scenes and input image scale

[25] , which highlights the necessity of exploring and applying more advanced VLMs to address these challenges.

[0006] More advanced VLMs (such as GPT-4o

[26] , LlaVA

[27] and Qwen2-VL

[28] ) have demonstrated stronger image and text understanding capabilities. Meanwhile, the novel learning paradigm of "pre-training-fine-tuning-prediction" (i.e., transfer learning process) has shown significant performance improvements in a variety of domain-specific visual understanding tasks

[20] . For example, this framework has been successfully applied to real-time hazard description in autonomous driving systems

[29] and industrial anomaly detection

[30] .

[0007] Despite significant progress, applying advanced video hazard identification (VLM) to construction site hazard identification remains a major challenge. While VLM excels in understanding open worlds, the urgent need is to develop customized VLMs specifically designed for construction hazard identification to meet the stringent high-precision requirements of construction safety management. Specifically, although some tools, equipment, and their usage protocols in the construction environment may resemble those in general contexts, subtle differences can significantly impact a VLM's assessment of potential hazards. This underscores the necessity of developing dedicated VLMs for construction hazard identification. Furthermore, the development of carefully crafted cues for VLM fine-tuning and the utilization of multi-source video data for automated construction hazard reporting remain underexplored areas worthy of further research.

[0008] Research on comprehensive construction hazard identification and automated report generation remains limited. Analysis of existing literature reveals two significant technological gaps:

[0009] First, there is a lack of fine-grained, end-to-end construction hazard identification capabilities. Existing computer vision-based methods mainly focus on detecting specific hazards (such as PPE violations or worker behavior), but struggle to simultaneously identify multiple hazard categories in complex construction environments. Furthermore, previous research largely relies on rule-based decision-making and labeled datasets, limiting the model's adaptability to unknown or dynamic construction scenarios. Traditional VLMs also fall short in semantic understanding of specific construction contexts, making it difficult to directly output hazard categories from visual data. While advanced VLMs have shown some potential, their open-world training paradigm lacks the specialized capabilities tailored to construction safety, resulting in insufficient accuracy in identifying critical hazard types.

[0010] Secondly, the automated generation of hazard reports from multi-source video data remains quite limited in terms of practicality. Current methods for generating construction hazard reports often rely on manual verification of hazard categories or independent image analysis, which hinders real-time and scalable safety management. Furthermore, existing frameworks have not adequately addressed the processing of continuous multi-source video streams, the extraction of context-dependent hazards, and the generation of structured reports without human intervention. In addition, the integration of VLM (Visual Management Model) for construction hazard identification with alerting engineering techniques remains an underexplored area, particularly in practical applications that address the dynamic characteristics of construction sites and ensure output quality, requiring further in-depth research. Summary of the Invention

[0011] To address the above technical issues, this invention proposes an innovative framework: an advanced VLM-enabled construction hazard identifyer (hereinafter referred to as ChatCH) and an end-to-end method for generating construction hazard reports.

[0012] According to one aspect of the present invention, a system based on a Visual Language Model (VLM) for end-to-end construction hazard identification and reporting generation is provided, comprising: one or more processors; a VLM-enabled construction hazard identifyer supported by the one or more processors and specifically designed for construction hazard identification; a Large Language Model (LLM) supported by the one or more processors and specifically designed for construction hazard reporting generation; and one or more memory devices interoperably coupled to the one or more processors and storing one or more instructions, which, when executed by the one or more processors, cause: the VLM-enabled construction hazard identifyer to autonomously identify one or more construction hazards in a construction site; and the LLM to generate an end-to-end real-time construction hazard report based on the identified construction hazards; wherein the VLM-enabled construction hazard identifyer is obtained by: constructing multiple image-text pairs related to the construction hazards; designing multiple label alignment prompts based on the image-text pairs; establishing a construction hazard dataset by associating (or pairing) the image-text pairs with corresponding label alignment prompts; fine-tuning multiple general VLMs using the construction hazard dataset; and selecting the best-performing fine-tuned VLM as the VLM-enabled construction hazard identifyer.

[0013] Preferably, multiple image-text pairs related to construction hazards are constructed by: systematically classifying multiple common construction hazards based on at least one of past literature and practical experience to establish a construction hazard checklist; collecting multiple images with various hazard characteristics from the construction site; classifying the multiple collected images according to the construction hazard checklist so that each image is aligned with one or more construction hazard categories; and associating (or pairing) each classified image with a corresponding text description that confirms the image represents at least one construction hazard category to form multiple image-text pairs.

[0014] Preferably, several common construction hazards are classified into four main categories: personal protective equipment (PPE), worker behavior, tools and equipment, and environment.

[0015] Preferably, the multiple label alignment prompts are designed by: defining the VLM to be fine-tuned as an expert with expertise in identifying hazards at the construction site; specifying the input information and output requirements for the VLM; and listing multiple hazard categories based on classification images associated with multiple image-text pairs.

[0016] Preferably, the general visual language model (VLM) has different parameter sizes; and each of the general VLMs is fine-tuned using low-rank adaptive techniques without changing the parameters of the general VLM.

[0017] Preferably, each of the fine-tuned VLMs is evaluated by: using the fine-tuned VLM to infer the output on the evaluation dataset; obtaining the alignment result between the inferred output and the hazard label; and calculating the evaluation metric based on the obtained alignment result.

[0018] Preferably, one or more processors are configured to: receive real-time video footage from the construction site; extract multiple image frames from the real-time video footage; and input the multiple image frames into a VLM-enabled construction hazard identifyer; and the VLM-enabled construction hazard identifyer is configured to identify one or more construction hazards in the multiple image frames; the one or more processors are further configured to determine the identified hazards corresponding to each time point and location at the construction site; and the large language model is configured to: compile all identified hazards within a specific time period; and generate a real-time construction hazard report for the construction site within the specific time period based on the compiled identified hazards.

[0019] Preferably, multiple image frames are extracted from the real-time video frame by: capturing image frames from the real-time video frame at predetermined time intervals; and associating (or pairing) each captured image frame with a corresponding position and capture time.

[0020] Preferably, one or more construction hazards are identified in multiple image frames by the following operations: sequentially reading multiple image frames; recording multiple image frames in a database; and archiving a hazard record for each of the multiple image frames, wherein the hazard record includes the location and timestamp associated with the image frame.

[0021] Preferably, the construction hazard report is generated within a specific time period through the following operations: designing multiple report generation prompts to organize the construction hazard identification results across all locations within the specific time period, the report generation prompts including data format, source explanation, output format, and considerations for ignoring occasional hazards in the time series; using the report generation prompts to read construction hazard record data of a specified length from the database, the specified length being determined based on the maximum number of tokens that can be processed by the Large Language Model (LLM); storing the output results generated by the LLM in the database; repeating the read and store steps until all hazard records in the database have been organized; using the report generation prompts to read the segmented construction hazard organization results from the database; and generating a construction hazard report based on the segmented construction hazard organization results.

[0022] According to another aspect of the present invention, a method based on a visual language model (VLM) for end-to-end construction hazard identification and reporting generation is provided, comprising: identifying one or more hazards in a construction site using a VLM-enabled construction hazard identifyer; and generating an end-to-end real-time construction hazard report based on the identified construction hazards using a large language model; wherein the VLM-enabled construction hazard identifyer is obtained by: constructing multiple image-text pairs related to the construction hazard; designing multiple label alignment prompts based on the image-text pairs; establishing a construction hazard dataset by associating (or pairing) the image-text pairs with corresponding label alignment prompts; fine-tuning multiple general VLMs using the construction hazard dataset; and selecting the best-performing fine-tuned VLM as the VLM-enabled construction hazard identifyer.

[0023] The main contribution of this invention lies in proposing a construction hazard identifyer, ChatCH, driven by an advanced VLM (Visual Modeling). This identifyer achieves comprehensive and accurate identification of construction hazards by utilizing image-text pairs. Simultaneously, a Construction Hazard Dataset (CHD) is constructed, containing 1,308 images from actual construction sites, covering 32 fine-grained hazard categories, which can serve as a robust benchmark for training and evaluating the performance of the construction hazard identification model.

[0024] Comparisons between ChatCH and advanced pre-trained VLMs (Qwen2-VL, LLaVA1.5) and traditional pre-trained VLMs (CLIP, Blip) based on CHD results show that ChatCH outperforms both advanced and traditional pre-trained VLMs, demonstrating strong robustness and few-shot learning capabilities. Furthermore, the application of an end-to-end construction hazard report generation method incorporating ChatCH in a real-world construction project validates its ability to output high-quality, structured, and accurate hazard reports. Attached Figure Description

[0025] Embodiments of the invention are described in more detail below with reference to the accompanying drawings, in which:

[0026] Figure 1 The main steps for identifying construction hazards according to an embodiment of the present invention are shown.

[0027] Figure 2A A block diagram of an artificial intelligence (AI) system for identifying and generating reports of construction hazards according to an embodiment of the present invention is shown.

[0028] Figure 2B This illustrates the overall framework of the invention.

[0029] Figure 3 Low-rank adaptation (LoRA) within the converter architecture is shown.

[0030] Figure 4 The structured output parser hints and process are displayed.

[0031] Figure 5 The LLM prompts for generating construction hazard reports are displayed.

[0032] Figure 6 Examples of potential construction hazards and their corresponding labels are shown.

[0033] Figure 7 The count for each label in the test and training sets is displayed.

[0034] Figure 8 The hints for constructing the UPD and CPD datasets are shown.

[0035] Figure 9 The .json file of the UPD dataset is displayed.

[0036] Figure 10 The image shown is the original image and the image after fogging and low-light processing.

[0037] Figures 11A to 11F The performance and loss curves of ChatCH are shown. Figure 11A Results of ChatCH using Qwen2-VL-2B on CPD. Figure 11B Results of ChatCH using Qwen2-VL-7B on CPD. Figure 11C Results of ChatCH using LLaVA1.5-7B on CPD. Figure 11D Results of ChatCH using Qwen2-VI-2B on UPD. Figure 11E Results of ChatCH using Qwen2-VL-7B on UPD, and Figure 11F Results of ChatCH using LLaVA1.5-7B on UPD.

[0038] Figure 12A and Figure 12B The performance and loss curves for the finely tuned CLIP and Blip are shown respectively.

[0039] Figure 13A and Figure 13B The performance of each construction hazard category identified before and after fine-tuning is shown. Figure 13A The performance of the pre-trained Qwen2 VL7B model without fine-tuning is shown. Figure 13B The performance of the pre-trained Qwen2VL7B model after fine-tuning is shown (results from round 12 on the CPD dataset).

[0040] Figure 14 Examples of construction hazard identification generated by ChatCH output at different construction locations after label alignment (i.e., after the structured output profiler).

[0041] Figure 15 The results of the end-to-end generated construction hazard reports are displayed. Detailed Implementation

[0042] In the following description, details of the invention are set forth as preferred embodiments. It will be apparent to those skilled in the art that modifications, including additions and / or substitutions, can be made without departing from the scope and spirit of the invention. Specific details may be omitted to avoid obscuring the invention; however, this disclosure is prepared to enable those skilled in the art to practice the teachings herein without requiring extensive experimentation.

[0043] Figure 1 The main steps for construction hazard identification and report generation according to an embodiment of the present invention are described. Step 1 involves selecting a pre-trained VLM that has been trained on a large corpus of open-world image-text pairs, equipping it with strong semantic understanding of open-world images. Step 2 requires constructing specific image-text pairs related to construction hazards and designing datasets and prompts for fine-tuning the pre-trained VLM. Step 3 uses the pre-trained VLM from Step 1 and the construction hazard dataset from Step 2 for fine-tuning using Parametric Efficient Fine-Tuning (PEFT)

[31] techniques to obtain an advanced VLM-enabled construction hazard identifyr, ChatCH, which can effectively identify various construction hazards while maintaining the capabilities of the basic pre-trained VLM. Step 4 involves extracting images from multi-source construction site videos and using ChatCH to identify construction hazards, ultimately generating a detailed construction hazard report through prompting engineering techniques. This process facilitates specific tasks of construction hazard identification using VLMs and supports the automatic generation of construction hazard reports, thereby providing strong support for automated construction hazard inspection.

[0044] Figure 2AA block diagram of a Visual Language Model (VLM)-based system 100 for construction hazard identification and reporting generation according to an embodiment of the present invention is shown. The VLM-based system includes: one or more processors 110; a VLM-enabled construction hazard identifyer 120, supported by the one or more processors and dedicated to construction hazard identification; a Large Language Model (LLM) 130, supported by the one or more processors and dedicated to construction hazard report generation; and one or more memory devices 140 interoperably coupled to the one or more processors and storing one or more instructions, which, when executed by the one or more processors, cause: the VLM-enabled construction hazard identifyer 120 to autonomously identify one or more construction hazards in a construction site; and the LLM 130 to generate an end-to-end real-time construction hazard report based on the identified construction hazards.

[0045] Figure 2B This paper presents a framework for implementing a VLM-based system to monitor potential hazards on construction sites and generate hazard reports. The framework comprises three main phases: dataset preparation, fine-tuning, and practical application.

[0046] In the initial dataset preparation phase, the inventors identified typical construction site hazards and collected, filtered, and cleaned images containing these hazards. The inventors then labeled the images to create image hazard pairs, divided the dataset into training and validation sets, and designed prompts for training the ChatCH model.

[0047] During the fine-tuning phase, the inventors used fine-tuning data to specifically tune the VLM for identifying construction hazards using advanced pre-trained VLM and fine-tuning techniques. Designed LLM hints were used to align the output of the fine-tuned VLM with the labels to ensure accurate evaluation of the model's hazard identification performance on the assessment data. Subsequently, standard evaluation metrics were used to evaluate the model's effectiveness to identify the optimal ChatCH model for construction hazard identification.

[0048] During the practical application phase, the focus shifted to generating automated construction hazard reports. This involved extracting keyframes from multi-source video and inputting the resulting image sequences into ChatCH for hazard identification. Construction hazard reports were then automatically generated using designed LLM prompts.

[0049] In BBS construction risk assessment, hazards encompass not only worker use of PPE and unsafe behaviors, but also the condition of tools / equipment and environmental conditions. Based on the framework proposed by Chen et al. [3] and Lee et al.

[58] , combined with a review of CV-based hazard identification studies (see Table 1) and insights from site inspections, this study systematically classifies common construction hazards into four main types—PPE, worker behavior, tools / equipment, and environment. Specifically, all visually identifiable hazard categories belonging to these four main types can be included in the classification. Figure 2A As shown, a high-quality CHD is assembled by collecting images of various hazards from real construction sites. Manual classification ensures alignment with predefined hazard categories, guaranteeing that each image represents at least one hazard type. Subsequently, all hazard images are manually labeled to generate image-text pairs, resulting in the final CHD dataset. The performance of ChatCH is evaluated using randomly selected samples from the CHD dataset.

[0050] To fine-tune the VLM, a three-part cue structure was designed: System, Human, and Recognizer. The System component defines the model's role as an expert in construction hazard identification, ensuring a focused response. The Human component simulates user queries by providing input requirements and expected outputs. The Recognizer component generates a response listing the identified hazard categories based on the input image. This structured interactive process effectively fine-tunes the pre-trained VLM for construction hazard identification tasks. Using Python scripts, .json files for fine-tuning and evaluation are automatically generated based on the designed cue and image-text pairs from the CHD dataset.

[0051] Pre-trained VLMs serve as general models for vision and language understanding, initially pre-trained on a wide range of varied datasets. These models are then optimized with specialized labeled data relevant to a specific task or domain, thereby enhancing their specialized performance and adaptability to the target domain

[59] . This process aligns with the principles of transfer learning and offers significant advantages for domain-specific task optimization

[60] . Given the rather large number of parameters in pre-trained models, full fine-tuning requires substantial computational resources and memory

[61] . PEFT techniques mitigate the memory overhead associated with improving pre-trained VLMs, thereby facilitating adaptability in environments with limited resources. Furthermore, PEFT techniques freeze the weights of pre-trained VLMs during the training process, ensuring that they do not directly participate in model training, thus avoiding catastrophic forgetting

[62] . Notably, in PEFT techniques, low-rank adaptation (LoRA)

[31] provides an effective solution to challenges such as overfitting and catastrophic forgetting encountered in conventional fine-tuning methods. Moreover, LoRA maintains inference efficiency by avoiding additional latency during task-specific adaptation

[63] .

[0052] LoRA can be applied to different components of the transformer. Specifically, LoRA is mainly implemented on the query and key projection matrices within the multi-head attention layer of the transformer, such as... Figure 3 As explained in the document. Specifically, during LoRA-based fine-tuning, the pre-trained VLM weight matrix in the transformer architecture is... Adjusted to Where ΔW represents the low-rank matrix (using rank r) derived from the product of two matrices A and B. This allows Minimal changes are required, while supporting effective fine-tuning. The optimization objective throughout the fine-tuning process is to minimize the loss function L(θ, A, B) for the target task:

[0053] Where θ represents parameters other than A and B. W o Indicates all pre-trained VLM weights. x i and y i The input sample i and its label are indicated respectively. By integrating the low-rank matrix into the pre-trained VLM and training only these parameters, this method reduces computational requirements while preserving the knowledge of the original VLM. Therefore, through LoRA-based fine-tuning techniques, the model can achieve improved accuracy in understanding potential hazards at construction sites while maintaining strong visual and linguistic capabilities.

[0054] To address the issue of inaccurate model evaluation caused by VLM not strictly outputting text content in a given format, a structured output parser is provided to resolve this problem. Specifically, the inventors in this case designed as follows: Figure 4 The prompts shown in the example comprise four main parts: role and task definitions, output format, sample output, and real data. This structured prompt design enhances the consistency and reliability of the model output, making subsequent automated processing and evaluation more efficient and accurate. Therefore, by introducing a structured output parser, the inventors ensure that the content generated by VLM conforms to the expected format and accurately reflects VLM's performance in identifying construction hazards.

[0055] To enhance the efficiency of construction safety management, this section proposes an end-to-end method for automatically generating construction hazard reports using the ChatCH identifier and prompting engineering techniques. The system is deployed on a server architecture (…). Figure 2B The operational workflow is formalized in Algorithm 1. The input parameter batch size b is aligned with the maximum token length constraint of the LLM. To optimize processing efficiency and reduce computational overhead, keyframes are sampled at fixed time intervals Δt.

[0056] Develop a dedicated LLM hint, G1, specifically designed for construction hazard documentation, to transform sequential construction images into structured hazard reports. For example... Figure 5As shown, G1 contains three components: (1) role definitions for establishing the task context and output format requirements of LLM, (2) an exemplary output template containing an overview summary and a detailed enumeration list to standardize the report, and (3) a list of hazard labels with location tags extracted from the video stream via ChatCH.

[0057] The complete data processing workflow is as follows. Line 1 of Algorithm 1 initializes an empty set of reports R and real-time hazard records LH, as well as the timestamp of the last identified frame TL. Line 2 continuously reads the incoming video stream. Line 3 determines whether the current frame is the keyframe required for hazard identification. Lines 4-6 use ChatCH to identify hazards within the image by extracting visual features and semantic information related to the construction environment. Each image sequence is analyzed to determine if there are hazards that need to be recorded, and then the identified hazard r and associated (or paired) data are added to the set LH when TL is updated. Lines 7-10 describe the process when the hazard count in set LH exceeds b; a dedicated LLM hint G1 is applied to generate a report of the batch of hazards, the result r is added to set R, and then LH is cleared. Finally, the final construction hazard report CHR is generated using set R with LLM and design hint G2. Specifically, similar to the architecture of G1, G2 aggregates all batches of hazard reports in set R into a comprehensive output, which includes the hazard category and a description corresponding to its location.

[0058] This framework reduces the need for manual intervention while enabling real-time hazard monitoring and alerts. By automating hazard identification and documentation, the system provides timely and accurate safety insights, thereby improving the intelligence and effectiveness of construction site safety management. These advancements enhance behavior-based safety (BBS) management practices and support accident prevention, worker protection, and project continuity.

[0059] CHD dataset

[0060] As shown in Table 2, this invention covers 32 specific and common construction hazards. Subsequently, corresponding construction site images for these hazard categories were collected. These images were primarily derived from historical inspection records captured by safety managers, each record labeled with a detailed description of the associated hazard. Due to the limited availability of historical hazard images, the CHD dataset was augmented using publicly accessible datasets (e.g., scene graph datasets of CPPED[9] and construction sites

[53] ) and internet resources. A filtering process was applied to all images, involving verifying their relevance to the construction site context, removing low-quality or blurry images, and removing text from images using generative AI techniques. Image annotation was performed using AliCloud's iTag tool, combined with rigorous quality assurance steps (e.g., cross-review among annotators and final review by the construction manager) to ensure dataset integrity. The resulting CHD dataset contains images of 1,308 construction hazards. The CHD dataset contains instance images and their corresponding construction hazard tags, such as Figure 6 As shown in the image.

[0061] Table 2.32 Categories of Construction Hazards

[0062] When dividing the dataset into training and test sets, the inventors used a stratified sampling method to achieve an approximate 7:3 ratio. However, some hazard categories had very few instances. To ensure the effectiveness of the evaluation, the inventors adjusted the ratio of the sample data in this part and prioritized ensuring that there were enough instances in the test set for reliable performance evaluation. Figure 7 The final distribution of labels across the training and test sets is described in the diagram. Given the importance of cues for fine-tuning the VLM, appropriate cue design for VLM fine-tuning can significantly enhance model performance for a specific task

[64] . Therefore, two cues were designed for subsequent experimental comparison: one covering all category descriptions and the other excluding these specific descriptions, such as... Figure 8 As explained in the document, the corresponding hazard descriptions are derived from the "Description of Construction Hazard Category" column in Table 2. Specifically, the dataset built using "Human A" is an Unclassified Hazard Dataset (UPD), while the dataset developed using "Human B" is a Classified Hazard Dataset (CPD). The main difference between these two datasets lies in whether all hazard label descriptions are provided in the "Human" component. Figure 9 The structure of the UPD .json file is explained below.

[0063] Experiments and evaluations

[0064] Multiple experiments were designed to evaluate the provided method. To validate the provided method for generating end-to-end construction hazard reports, specific construction projects were selected, with actual construction site videos used for testing.

[0065] Based on the constructed datasets, CPD, and UPD, this invention employs the LoRA technique to fine-tune pre-trained VLMs. Specifically, Qwen2-VL (https: / / huggingface.co / Qwen) with parameter sizes of 2 billion and 7 billion, and LLaVA-1.5 (https: / / huggingface.co / llava-hf / llava-1.5-7b-hf) with a parameter size of 7 billion, were selected for experiments. Using training sets from the CPD and UPD datasets, these models were fine-tuned and validated using their corresponding test sets. To accurately track model performance and avoid underfitting or overfitting, the method involves saving and evaluating the model every 3 epochs. Notably, Figure 4 The same structured output parser hints shown in the figure are used for label alignment during the verification of VLM effects.

[0066] For effective comparison, the traditional VLM, CLIP, and Blip serve as baseline models. Specifically, the CLIP-based method utilizes the ViT-B / 32 transformer architecture as the image encoder and employs a masked self-attention transformer as the text encoder (https: / / huggingface.co / openai / clip-vit-base-patch32). Meanwhile, the Blip-based method employs the blip-image-captioning-large model (https: / / huggingface.co / Salesforce / blip-image-captioning-large). According to previous studies by Chen et al.

[54] and Tsai et al.

[18] , this process involves fine-tuning the CLIP and Blip models with images and labels. During validation, predictions are made by comparing the similarity scores between test set images and construction hazard labels. If the similarity score exceeds a preset threshold d, the label is considered a prediction.

[0067] Few-shot learning is a machine learning paradigm focused on developing algorithms that can quickly adapt to new tasks or categories with only a small number of samples

[65] . This capability is particularly important in construction scenarios where data scarcity is common. To evaluate the performance of ChatCH under limited data conditions, the experimental design used 2, 5, 10, and 20 samples randomly selected from the training portion of the CPD dataset for each construction hazard category. The corresponding full test set was then used for evaluation. In addition, to evaluate the robustness of the model, images from the test set of CHD were processed using a Python script developed for this purpose (see [link to Python script]). Figure 10 To simulate foggy and low-light conditions (available at https: / / github.com / QHCV / lowlight-foggy), the method described by Chen et al. [9] and Ferdous and Ahsan

[66] was followed. Finally, the method presented for generating automated construction hazard reports was validated in a real-world construction project using video footage from the study by Chen et al. [9].

[0068] Evaluation metrics and implementation details

[0069] To evaluate the model's performance, the inventors applied several widely accepted metrics. Precision (the proportion of correctly identified positive samples out of those predicted as positive), recall (the proportion of actually positive samples correctly identified), and F1 score (the harmonic mean of precision and recall) were calculated using the following formulas:

[0070]

[0071]

[0072]

[0073] Where TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative. Furthermore, to emphasize the importance of recall and minimize missed detections, especially considering the potential severity of construction site accidents, the inventors calculated the F2 score (p=2):

[0074]

[0075] Compared to precision, the F2 score assigns greater weight to recall, thus ensuring that potential construction hazards are detected as completely as possible. The Cohen's Kappa coefficient

[67] measures the consistency between observed results and expected results that exceed chance, and is calculated as follows:

[0076]

[0077] Where po is the observed consistency rate and pe is the accidental expected consistency rate. A Coen-Kappa coefficient close to 1 indicates near-perfect consistency, suggesting that the model significantly outperforms random guessing, thus enhancing confidence in the model's reliability. Conversely, values ​​close to or below 0 indicate that the consistency is no better than or may be worse than accidental expectations, highlighting the model's potential limitations.

[0078] During the experiment, the inventors used Llama-Factory

[68] , a high-efficiency fine-tuning framework. Specific parameter settings included a LoRA rank of 8, a learning rate of 0.00005, and a threshold d set to 0.95. The LLM used was a Qwen2.5-14B model deployed on a local server, capable of receiving up to 32,768 input tokens. In the application of the automatic construction hazard report generation method, the batch size b was set to 300, and Δt was set to 2 seconds. All experiments were conducted on a server equipped with a GPU (32GB of storage) running Ubuntu 22.04.

[0079] The results of VLM's fine-tuned construction hazard identification

[0080] Based on the experimental design above, Figure 11 illustrates the results using the CPD and UPD datasets of a pre-trained VLM with two parameter sizes, along with the corresponding loss curves from the training process. As shown in the figure, the loss curves for all training processes drop sharply in the third round and then gradually stabilize in subsequent rounds. Figure 11A ChatCH highlights the performance of the Qwen2-VL-2B model on the CPD dataset. Without fine-tuning, the model achieves a high recall of nearly 0.85 but suffers from an extremely low precision of 0.05, resulting in an F1 score of only 0.1. Fine-tuning significantly enhances the identification of construction hazards, increasing precision to 0.47 in the third round and reaching a peak precision of 0.825 in the 30th round. Figure 11B ChatCH demonstrates the performance of the Qwen2-VL-7B model on the same CPD dataset. Before fine-tuning, this model has an F1 score of 0.226, compared to... Figure 11A The Qwen2-VL-2B model performs slightly better. After the third round, the performance stabilizes and reaches a maximum accuracy of 0.894 in the 12th round, representing an improvement of 0.705 compared to the unadjusted state. Figure 11C The results of ChatCH using the LLaVA1.5-7B model on the CPD dataset are presented. The model shows improved stability during training, stabilizing after 18 epochs, with the F1 score consistently above 0.7, but the rest remaining below 0.7. Figure 11B The results in. Figure 11DThis presents the results of ChatCH using the Qwen2-VL-2B model on the UPD dataset. The training process stabilized after the ninth epoch, generating results similar to... Figure 11A The relatively low F1 score is attributed to the low recall despite high precision on the CPD dataset. Conversely, Figure 11E This describes ChatCH using the Qwen2-VL-7B model on the UPD dataset, where stabilization occurs after the third round. Figure 11B In comparison, the model performed better without fine-tuning; however, precision, recall, and F1 score were generally lower after fine-tuning. Figure 11B Precision, recall, and F1 score. Figure 11F The results for ChatCH, presented on the UPD dataset using the LLaVA1.5-7B model, show an overall upward trend with occasional fluctuations. The F1 score is close to 0.7, while the recall remains slightly below [previous value]. Figure 11C The recall rate is shown in the figure. This indicates that the CPD dataset is more helpful in enhancing the effectiveness of fine-tuning, especially in improving recall. In summary, larger pre-trained model parameters and including descriptive details of construction hazards in the fine-tuning cues help improve the performance of construction hazard identification.

[0081] The loss curve depicted in Figure 12 indicates the trend of the loss value over time during the fine-tuning process of the pre-trained CLIP and Blip models. Initially, the loss is relatively high but gradually decreases over time until it stabilizes. For construction hazard identification tasks, such as... Figure 12A As shown in the figure, the pre-trained CLIP model exhibits a recall close to 1, indicating near-perfect detection of positive samples. However, the model's precision consistently remains around 0.05 and does not significantly improve with fine-tuning. Furthermore, the recall tends to decrease as fine-tuning progresses, indicating a reduction in the model's ability to accurately identify positive samples. (As shown in...) Figure 12B As shown in the results, the Blip model exhibits similarly poor performance, with recall jumping from 0.037 (before fine-tuning) to 1 (after fine-tuning), while precision and F1 score remain unchanged at 0.053 and 0.096, respectively. Fine-tuning failed to produce significant improvements in precision or F1 score, highlighting the ongoing challenge of the model in distinguishing construction hazards. In summary, despite fine-tuning, both CLIP and Blip models struggle to adequately understand and distinguish features associated with construction hazards, limiting their application to this specific task. The capabilities of traditional pre-trained CLIP and Blip models are insufficient to effectively address the complexity of construction hazard identification.

[0082] Table 3 provides F2 scores, Cohen-Kappa coefficient values, and frames per second (FPS) data for model recognition on the test set. It compares two main models: CLIP-based and Qwen2-VL (2B and 7B), each evaluated with and without fine-tuning. For the CLIP-based model, fine-tuning did not lead to performance improvement in construction hazard identification. Both its F2 score and Cohen-Kappa coefficient decreased, indicating that fine-tuning was not beneficial for this model used for construction hazard identification. Conversely, the Qwen2-VL model showed significant improvement after fine-tuning. On the CPD dataset, the Qwen2-VL-2B model started with an F2 score of 22.46% and a Cohen-Kappa coefficient of -0.03%. The Qwen2-VL-7B model started with an F2 score of 36.5% and a Cohen-Kappa coefficient of 11.8%. After fine-tuning, the Qwen2-VL-7B model achieved an F2 score of 77.89% and a Coen-Kappa coefficient of 75.13%, representing significant improvements of 41.39% and 63.33%, respectively. These results highlight the effectiveness of fine-tuning in enhancing the performance of the Qwen2-VL model, particularly in balancing precision and recall. The fine-tuned Qwen2-VL model's overall performance on the UPD dataset is slightly lower than that on the CPD dataset, suggesting that including specific construction hazard category descriptions in the fine-tuning prompts helps improve identification performance.

[0083] Table 3. F2 scores and Cohen-Kappa coefficients for different pre-trained VLMs.

[0084] Regarding recognition efficiency, models fine-tuned using LoRA techniques typically exhibit faster recognition speeds. For example, under the same hardware conditions, the FPS of the Qwen2-VL-2B model increased from 0.280 without fine-tuning to 0.968 after fine-tuning. Fine-tuning with the CPD dataset also yielded higher speeds compared to using the UPD dataset; the Qwen2-VL-7B model achieved 1.269 FPS on the CPD dataset, compared to 0.949 on the UPD dataset. Although the CLIP-based model boasts a high FPS of 52.63, its low accuracy in identifying construction hazard means this speed advantage does not constitute a significant benefit. Therefore, adding specific construction hazard category descriptions during the fine-tuning process enhances both recognition accuracy and speed.

[0085] Figure 13 illustrates the performance of the Qwen2-VL-7B model in identifying each construction hazard category before and after fine-tuning. Figure 13AIn the example, the model's performance without fine-tuning is demonstrated, with two potential hazard categories (H22 and H32) completely unidentified. Furthermore, three categories (H01, H02, and H15) exhibit extremely low precision and recall, almost zero. While a few categories achieve high recall, such as H11 and H14 with recall close to 1, precision across all categories remains below 0.6. After fine-tuning, as shown... Figure 13B As described, the model's performance has significantly improved across each construction hazard category. Most categories now exhibit precision and recall above 0.8. Notably, previously unidentified categories, such as H22, are now successfully identified. For example, category H22 achieved a precision of 1.0 and a recall of 0.8 after fine-tuning, demonstrating a significant enhancement in the model's ability to identify this construction hazard category. These improvements highlight the effectiveness of fine-tuning in enhancing ChatCH's ability to accurately identify a wide range of construction hazards.

[0086] Few-shot learning results of fine-tuned VLM

[0087] Based on the experimental design, Table 4 presents the performance of the pre-trained Qwen2-VL-7B model fine-tuned with different numbers of samples for each construction hazard category. The results shown are the average of the best three rounds of evaluation. Overall, the model's performance shows an upward trend across several evaluation metrics as the number of training samples per category increases. When two samples per category are used for fine-tuning, there is a significant improvement in construction hazard identification compared to no fine-tuning: precision increases by 54.99% and recall increases by 14.81%. As the number of samples used for fine-tuning per category further increases, the model's performance continues to improve, albeit at a slower rate. For example, increasing the number of samples per category from 10 to 20 results in a slight decrease in recall of 1.21%, but precision reaches 80.63%, marking a 4.88% improvement. This indicates that even a small number of samples can significantly enhance the model's effectiveness.

[0088] Table 4. Performance of the Qwen2-VL-7B model under finite-sample fine-tuning

[0089] Results of robustness tests of fine-tuned VLM

[0090] To evaluate the robustness of the fine-tuned VLM in the construction hazard identification task, test set images from the CPD dataset were manipulated to simulate low-light and foggy scenarios. Table 5 shows the mean values ​​from six repeated experiments under these scenarios. The variance of the F1 score across all six trials remained below 0.000008, demonstrating excellent stability and reliability of the model's performance under different environmental conditions. The results indicate that the performance of ChatCH using the pre-trained Qwen2-VL-2B model degrades to varying degrees in both low-light and foggy scenarios. Specifically, under low-light conditions, precision decreases by 2.53%, and recall decreases by 3.48%.

[0091] Under foggy conditions, precision decreased by 6.29% and recall by 3.24%. Conversely, when ChatCH used the pre-trained Qwen2-VL-7B model, the performance reduction was minimal under both conditions, and some metrics even showed improvement. Under low light conditions, precision decreased by only 1.43%, while recall increased slightly by 0.01%. Under foggy conditions, precision decreased by only 0.64%, while recall improved by 0.14%. These findings demonstrate the strong robustness of finely tuned pre-trained VLMs for construction hazard identification, especially when utilizing pre-trained models with larger parameter sizes.

[0092] Table 5. Performance of pre-trained Qwen2 VL 2B and 7B models in low-light and foggy scenes.

[0093] End-to-end construction hazard report generation

[0094] To verify the effectiveness of the automatic construction hazard report generation method provided by this invention, the inventors of this case referred to Chen et al. [9] and selected videos from three different areas of the actual construction site: the material crane access road (CL1), the staff passage (CL2), and the material inlet and outlet (CL3). Videos were selected from the period in which hazards occurred more frequently to help evaluate the effectiveness of ChatCH. In the experiment, the ChatCH model (based on a pre-trained Qwen2-VL-7B fine-tuned on the CPD dataset) and specially designed prompts from LLM were used to execute Algorithm 1. Figure 14 This section presents examples of construction hazard identification results from ChatCH output (following the structured output parser) at different construction locations. It can even identify very detailed hazards, such as not wearing gloves during material handling.

[0095] Figure 15The final generated construction hazard report is displayed in the image. The results indicate that the LLM can generate the required construction hazard report content based on the pre-designed prompts, thus providing clear and concise information for the safety manager. The identified hazards are classified into four categories, with three, three, and two hazards identified at CL1, CL2, and CL3, respectively. This classification is aligned with the actual hazard types observed in the video, where the corresponding hazard records were derived from the study by Chen et al. [9]. Regarding the identification efficiency, the average time per image for CL1, CL2, and CL3 was 0.907, 0.935, and 0.846 seconds, respectively, meaning that each image is processed in less than one second while video frames are extracted at two-second intervals. Therefore, our method not only achieves fast video data processing but also maintains accuracy and real-time hazard identification. This efficiency and accuracy are crucial for construction site safety monitoring, allowing for timely detection and reporting of potential hazards, thus enabling rapid and appropriate preventative actions to reduce the accident rate. Furthermore, this automated method significantly enhances identification efficiency and reduces labor costs compared to traditional manual inspection and report generation.

[0096] The functional units and modules according to the embodiments disclosed herein may be implemented using computing devices, computer processors, or electronic circuit systems, including but not limited to application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of this disclosure. Based on the teachings of this disclosure, those skilled in the art of software or electronics can readily prepare computer instructions or software code to run in computing devices, computer processors, or programmable logic devices.

[0097] All or part of the methods according to the embodiments can be performed in one or more computing devices, including server computers, personal computers, laptop computers, mobile computing devices such as smartphones and tablet computers.

[0098] Embodiments may include computer storage media, temporary and non-temporary memory devices storing computer instructions or software code that can be used to program or configure a computing device, computer processor, or electronic circuit system to perform any of the processes of the present invention. Storage media, temporary and non-temporary memory devices may include, but are not limited to, floppy disks, optical disks, Blu-ray discs, DVDs, CD-ROMs and magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of media or device suitable for storing instructions, code, and / or data.

[0099] Each of the functional units and modules according to various embodiments may also be implemented in a distributed computing environment and / or cloud computing environment, wherein all or part of the machine instructions are executed in a distributed manner by one or more processing devices interconnected by a communication network, such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0100] While this disclosure has been described and illustrated with reference to specific embodiments thereof, such descriptions and illustrations are not limiting. Illustrations may not necessarily be drawn to scale. Due to manufacturing processes and tolerances, the descriptions in this disclosure may differ from actual equipment. Other embodiments of this disclosure may exist that are not specifically described. Modifications may be made to adapt particular circumstances, materials, material compositions, methods, or processes to the objectives and scope of this disclosure. All such modifications are intended to fall within the scope of the appended claims. Although the methods disclosed herein have been described with reference to specific operations performed in a particular order, it should be understood that these operations may be combined, subdivided, or reordered to form equivalent methods without departing from the teachings of this disclosure. Therefore, unless specifically indicated herein, the order and grouping of operations are not limiting.

[0101] References [1] B. Yuan, S. Xu, M. Niu, K. Guo. Will improved safety attitudes necessarily curb unsafe behavior? Hybrid method based on NCA and SEM. Journal of Environmental and Public Health. 2022(1): 9271690, 2022. https: / / doi.org / 10.1155 / 2022 / 9271690. [2] Y. Han, Y. Diao, Z. Yin, R. Jin, J. Kangwa, O. J. Ebohon. Immersive technology-driven investigations on influence factors of cognitive load incurred in construction site hazard recognition, analysis and decision making. Advanced Engineering Informatics. 48101298, 2021. https: / / doi.org / 10.1016 / j.aei.2021.101298. [3] Q. Chen, D. Long, C. Yang, H. Xu. Knowledge Graph Improved Dynamic Risk Analysis Method for Behavior-Based Safety Management on a Construction Site. Journal of Management in Engineering. 39(4): 04023023, 2023. https: / / doi.org / 10.1061 / JMENEA.MEENG-5306. [4] X. Zhang, D. Tian, ​​Q. Ren, M. Li, Y. Shen, S. Han. A hybrid deep semantic mining method considering fuzzy expressions for the automatic recognition of construction safety hazard information. Advanced Engineering Informatics. 61102507, 2024. https: / / doi.org / 10.1016 / j.aei.2024.102507. [5] Q. Xiang, G. Ye, Y. Liu, Y. Miang Goh, D. Wang, T. He. Cognitive mechanism of construction workers' unsafe behavior: A systematic review. Safety Science. 159106037, 2023. https: / / doi.org / 10.1016 / j.ssci.2022.106037. [6] Z. Hu, W. T. Han, H. Hu, F. Xu. Cognitive Factors Underlying Unsafe Behaviors of Construction Workers as a Tool in Safety Management: A Review. Journal of Construction Engineering and Management. 149(3): 03123001, 2023. https: / / doi.org / 10.1061 / JCEMD4.COENG-11820. [7] Y. Ouyang, X. Luo. Differences between inexperienced and experienced safety supervisors in identifying construction hazards: Seeking insights for training the inexperienced. Advanced Engineering Informatics. 52101602, 2022. https: / / doi.org / 10.1016 / j.aei.2022.101602. [8] Y. Wang, B. Xiao, A. Bouferguene, M. Al-Hussein. Proactive safety hazard identification using visual-text semantic similarity for construction safety management. Automation in Construction. 166 105602, 2024. https: / / doi.org / 10.1016 / j.autcon.2024.105602. [9] Q. Chen, D. Long, S. Wang, Q. Chen, B. Yuan. Real-Time Detection of Personal Protective Equipment Violations for Construction Workers Using Semisupervised Learning and Video Clips. Journal of Construction Engineering and Management. 151(3): 04024213, 2025. https: / / doi.org / 10.1061 / JCEMD4.COENG-15310.

[10] B. Xiao, X. Yin, S.-C. Kang. Vision-based method of automatically detecting construction video highlights by integrating machine tracking and CNN feature extraction. Construction Automation. 129103817, 2021. https: / / doi.org / 10.1016 / j.autcon.2021.103817.

[11] S. Chen, K. Demachi, F. Dong. Graph-based linguistic and visual information integration for on-site occupational hazards identification. Construction Automation. 137104191, 2022. https: / / doi.org / 10.1016 / j.autcon.2022.104191.

[12] I. Jeelani, K. Asadi, H. Ramshankar, K. Han, A. Albert. Real-time vision-based worker localization and hazard detection for construction. Construction Automation. 121 103448, 2021. https: / / doi.org / 10.1016 / j.autcon.2020.103448.

[13] Y. Wang, H. Luo, W. Fang. An integrated approach for automatic safety inspection inconstruction: Domain knowledge with multimodal large language model. Advanced Engineering Informatics. 65 103246, 2025. https: / / doi.org / 10.1016 / j.aei.2025.103246.

[14] B. Xiao, Y. Zhang, Y. Chen, X. Yin. A semi-supervised learning detection method for vision-based monitoring of construction sites by integrating teacher-student networks and data augmentation. Advanced Engineering Informatics. 50101372, 2021. https: / / doi.org / 10.1016 / j.aei.2021.101372.

[15] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, F. Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14455-14465. https: / / doi.org / 10.48550 / arXiv.2401.12168.

[16] A. Radford, J.W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark. Learning transferable visual models from natural language supervision. International conference on machine learning, PMLR, 2021, pp. 8748-8763. https: / / proceedings.mlr.press / v139 / radford21a.

[17] J. Li, D. Li, C. Xiong, S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. International Conference on Machine Learning, PMLR, 2022, pp. 12888-12900. https: / / proceedings.mlr.press / v162 / li22n.html.

[18] W.-L.Tsai, P.-L.Le, W.-F.Ho, N.-W.Chi, JJLin, S.Tang, S.-H.Hsieh. Construction safety inspection with contrastive language-image pre-training (CLIP) image captioning and attention. Construction Automation. 169 105863, 2025. https: / / doi.org / 10.1016 / j.autcon.2024.105863.

[19] D. Gil, G. Lee. Zero-shot monitoring of construction workers’ personal protective equipment based on image captioning. Construction Automation. 164105470, 2024. https: / / doi.org / 10.1016 / j.autcon.2024.105470.

[20] J. Zhang, J. Huang, S. Jin, S. Lu. Vision-Language Models for Vision Tasks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 46(8): 5625-5644, 2024. https: / / doi.org / 10.1109 / TPAMI.2024.3369699.

[21] Y. Liang, P. Vadakkepat, D. K. H. Hua, S. Wang, Z. Li, S. Zhang. Recognizing temporary construction site objects using CLIP-based few-shot learning and multi-modal prototypes. Construction Automation. 165 105542, 2024. https: / / doi.org / 10.1016 / j.autcon.2024.105542.

[22] C. Chen, B. Xiao, Y. Zhang, Z. Zhu. Automatic vision-based calculation of excavator earthmoving productivity using zero-shot learning activity recognition. Construction Automation. 146 104702, 2023. https: / / doi.org / 10.1016 / j.autcon.2022.104702.

[23] G. Yong, M. Liu, S. Lee. Explainable Image Captioning to Identify Ergonomic Problems and Solutions for Construction Workers. Journal of Computing in Civil Engineering. 38(4): 04024022. 2024. https: / / doi.org / 10.1061 / JCCEE5.CPENG-5744.

[24] B. Xiao, Y. Wang, Y. Zhang, C. Chen, A. Darko. Automated daily report generation from construction videos using ChatGPT and computer vision. Construction Automation. 168 105874, 2024. https: / / doi.org / 10.1016 / j.autcon.2024.105874.

[25] Y. Ouali, A. Bulat, A. Xenos, A. Zaganidis, IMMetaxas, G. Tzimiropoulos, B. Martinez. Discriminative Fine-tuning of LVLMs. arXiv preprint arXiv: 2412.04378.2024. https: / / doi.org / 10.48550 / arXiv.2412.04378.

[26] A. Hurst, A. Lerer, A. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford. Gpt-4o system card. arXiv preprint arXiv: 2410.21276.2024. https: / / doi.org / 10.48550 / arXiv.2410.21276.

[27] H. Liu, C. Li, Q. Wu, Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, Vol. 36, 2024. https: / / proceedings.neurips.cc / paper_files / paper / 2023 / hash / 6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html.

[28] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, WGe. Qwen2-v1: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv: 2409.12191.2024. https: / / doi.org / 10.48550 / arXiv.2409.12191.

[29] D. Xiao, M. Dianati, P. Jennings, R. Woodman. HazardVLM: A Video Language Model for Real-Time Hazard Description in Automated Driving Systems. IEEE Transactions on Intelligent Vehicles. 1-13, 2024. https: / / doi.org / 10.1109 / TIV.2024.3451350.

[30] Z.Gu, B.Zhu, G.Zhu, Y.Chen, M.Tang, J.Wang. Anomalygpt: Detecting industrial anomalies using large vision-language models. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, 2024, pp. 1932-1940. https: / / doi.org / 10.1609 / aaai.v38i3.27963.

[31] EJHu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations. https: / / openreview.net / forum?id=nZeVKeeFYf9.

[32] BHW Guo, YMGoh, K. Le Xin Wong. A system dynamics view of a behavior-based safety program in the construction industry. Safety Science. 104202-215, 2018. https: / / doi.org / 10.1016 / j.ssci.2018.01.014.

[33] W. Xu, T.-K. Wang. Dynamic safety prewarning mechanism of human-machine-environment using computer vision. Engineering, Construction and Architectural Management. 27(8): 1813-1833, 2020. https: / / doi.org / 10.1108 / ECAM-12-2019-0732.

[34] J. Al-Bayati Ahmed, R. Eiris, O. Hernandez Guillermo, A. Al-Bayati Miriam. Addressing Social Inequity in Construction Safety Research: Personal Protective Equipment Focus. Journal of Construction Engineering and Management. 149(12): 04023127, 2023. https: / / doi.org / 10.1061 / JCEMD4.COENG-13864.

[35] Z. Kolar, H. Chen, X. Luo. Transfer learning and deep convolutional neural networks for safetyguardrail detection in 2D images. Construction Automation. 89 58-70, 2018. https: / / doi.org / 10.1016 / j.autcon.2018.01.003.

[36] W. Fang, PEDLove, H. Luo, L. Ding. Computer vision for behavior-based safety in construction: A review and future directions. Advanced Engineering Information. 43 100980, 2020. https: / / doi.org / 10.1016 / j.aei.2019.100980.

[37] Z. Tian, ​​S. Zhou, A. Lee, Y. Shan, B. Detmann. How to identify earth pressures on in-service tunnel linings: Insights from Bayesian inversion to address non-uniqueness. Transportation Geotechnics. 48 101344, 2024. https: / / doi.org / 10.1016 / j.trgeo.2024.101344.

[38] Y. Jeon, D. Q. Tran, A. A. Kulinan, T. Kim, M. Park, S. Park. Vision-based motion prediction for construction workers safety in real-time multi-camera system. Advanced Engineering Information. 62102898, 2024. https: / / doi.org / 10.1016 / j.aei.2024.102898.

[39] J. Yin, X. Yin, M. Pan, L. Li. Scalable and transparent automated sewer defect detection using weakly supervised object localization. Construction Automation. 174 106152, 2025. https: / / doi.org / 10.1016 / j.autcon.2025.106152.

[40] L. Liu, Z. Guo, Z. Liu, Y. Zhang, R. Cai, X. Hu, R. Yang, G. Wang. Multi-Task Intelligent Monitoring of Construction Safety Based on Computer Vision. Buildings. 14(8): 2429, 2024. https: / / doi.org / 10.3390 / buildings14082429.

[41] AMVukicevic, M.Djapan, V.Isailovic, D.Milasinovic, M.Savkovic, P.Milosevic. Generic compliance of industrial PPE by using deep learning techniques. Safety Science. 148 105646, 2022. https: / / doi.org / 10.1016 / j.ssci.2021.105646.

[42] X.Mei, W.Ma, F.Xu, Z.Zhang. Vision-Based Detection of Unsafe Worker GuardrailClimbing Based on Posture and Instance Segmentation Data Fusion. Journal of Construction Engineering and Management. 150(11): 04024156, 2024. https: / / doi.org / 10.1061 / JCEMD4.COENG-14266.

[43] M. Yang, C. Wu, Y. Guo, R. Jiang, F. Zhou, J. Zhang, Z. Yang. Transformer-based deep learning model and video dataset for unsafe action identification in construction projects. Automation of Construction. 146 104703, 2023. https: / / doi.org / 10.1016 / j.autcon.2022.104703.

[44] M. Park, A.S. Kulinan, D.Q. Tran, J. Bak, S. Park. Preventing falls from floor openings using quadrilateral detection and construction worker pose-estimation. Construction Automation. 165 105536, 2024. https: / / doi.org / 10.1016 / j.autcon.2024.105536.

[45] Q. Fang, H. Li, X. Luo, L. Ding, H. Luo, TM. Rose, W. An. Detecting non-hardhat-use by a deep learning method from far-field surveillance videos. Construction Automation. 85 1-9, 2018. https: / / doi.org / 10.1016 / j.autcon.2017.09.018.

[46] J.Li, X.Zhao, G.Zhou, M.Zhang. Standardized use inspection of workers' personal protective equipment based on deep learning. Safety Science. 150 105689, 2022. https: / / doi.org / 10.1016 / j.ssci.2022.105689.

[47] H. Luo, J. Liu, W. Fang, PEDLove, Q. Yu, Z. Lu. Real-time smart video surveillance to manage safety: A case study of a transport mega-project. Advanced Engineering Information. 45 101100, 2020. https: / / doi.org / 10.1016 / j.aei.2020.101100.

[48] ​​Y. Su, C. Mao, R. Jiang, G. Liu, J. Wang. Data-Driven Fire Safety Management at Building Construction Sites: Leveraging CNN. Journal of Engineering Management. 37(2): 04020108, 2021. https: / / doi.org / 10.1061 / (ASCE)ME.1943-5479.0000877.

[49] YGLim, J. Wu, YMGoh, J. Tian, ​​V. Gan. Automated classification of “cluttered” construction housekeeping images through supervised and self-supervised feature representation learning. Automation of Construction. 156 105095, 2023. https: / / doi.org / 10.1016 / j.autcon.2023.105095.

[50] L. Tan, X. Hu, T. Tang, D. Yuan. A lightweight metro tunnel water leakage identification algorithm via machine vision. Engineering Failure Analysis. 150107327, 2023. https: / / doi.org / 10.1016 / j.engfailanal.2023.107327.

[51] KW Johansen, J. Teizer, C. Schultz. Automated rule-based safety inspection and compliance checking of temporary guardrail systems in construction. Construction Automation. 168105849, 2024. https: / / doi.org / 10.1016 / j.autcon.2024.105849.

[52] H. Zhang, D. Cheng, Q. Kou, M. Asad, H. Jiang. Indicative Vision Transformer for end-to-end zero-shot sketch-based image retrieval. Advanced Engineering Information. 60 102398, 2024. https: / / doi.org / 10.1016 / j.aei.2024.102398.

[53] L. Zhang, J. Wang, Y. Wang, H. Sun, X. Zhao. Automatic construction site hazard identification integrating construction scene graphs with BERT based domain knowledge. Construction Automation. 142104535, 2022. https: / / doi.org / 10.1016 / j.autcon.2022.104535.

[54] Z. Chen, H. Chen, M. Imani, R. Chen, F. Imani. Vision language model for interpretable and finae-grained detection of safety compliance in diverse workplaces. Expert Systems with Applications. 265125769, 2025. https: / / doi.org / 10.1016 / j.eswa.2024.125769.

[55] S. Wen, M. Park, DQTran, S. Lee, S. Park. Automated construction safety reporting system integrating deep learning-based real-time advanced detection and visual question answering. Advances in Engineering Software. 198 103779, 2024. https: / / doi.org / 10.1016 / j.advengsoft.2024.103779.

[56] H. Pu, X. Yang, J. Li, R. Guo. AutoRepo: A general framework for multimodal LLM-based automated construction reporting. Expert Systems Applications. 255124601, 2024. https: / / doi.org / 10.1016 / j.eswa.2024.124601.

[57] CY Chang, A. Khanum, HHHsu, MHTsai, WTChen, SGSu. Enhancing Daily Reports: Application of Multimodal GPT for Image Description Generation in Construction Site Quality Assurance. 2024 IEEE International Conference on Image Processing Challenges and Workshops (ICIPCW), 2024, pp. 4204-4208. https: / / doi.org / 10.1109 / ICIPCW64161.2024.10769129.

[58] P.-C. Lee, J. Wei, H.-I. Ting, T.-P. Lo, D. Long, L.-M. Chang. Dynamic analysis of construction safety risk and visual tracking of key factors based on behavior-based safety and building information modeling. KSCE Journal of Civil Engineering. 23(10): 4155-4167, 2019. https: / / doi.org / 10.1007 / s12205-019-0283-z.

[59] Y.Mao, Y.Ge, Y.Fan, W.Xu, Y.Mi, Z.Hu, Y.Gao. A survey on LoRA of large language models. Frontiers of Computer Science. 19(7): 197605, 2024. https: / / doi.org / 10.1007 / s11704-024-40663-9.

[60] C.-ZTXie, Q.Chen, B.Zhu, EWMLee, T.-Q.Tang, X.Yin, Z.Yuan, B.Zhang. Coordinating dynamic signage for evacuation guidance: A multi-agent reinforcement learning approach integrating mesoscopic crowd modeling and fire propagation. Chaos, Solitons & Fractals. 194116246, 2025. https: / / doi.org / 10.1016 / j.chaos.2025.116246.

[61] Y. Zhong, H. Jiang, L. Li, R. Nakada, T. Liu, L. Zhang, H. Yao, H. Wang. NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models. arXiv preprint arXiv: 2410.01870.2024. https: / / doi.org / 10.48550 / arXiv.2410.01870.

[62] S. Dou, E. Zhou, Y. Liu, S. Gao, W. Shen, L. Xiong, Y. Zhou, X. Wang, Z. Xi, X. Fan. LoRAMoE: Alleviating world knowledge forgetting in large language models via MoE-style plugin. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Paper), 2024, pp. 1932–1945. https: / / doi.org / 10.18653 / v1 / 2024.acl-long.106.

[63] S. Oh, H. Yang, E. Park. Parameter-Efficient Instance-Adaptive Neural Video Compression. Asian Conference on Computer Vision, 2024. https: / / doi.org / 10.48550 / arXiv.2405.08530.

[64] XL Li, P. Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol. 1: Long Paper), 2021, pp. 4582-4597. https: / / doi.org / 10.18653 / v1 / 2021.ac1-long.353.

[65] Y. Song, T. Wang, P. Cai, SK Mondal, JP Sahoo. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys. 55(13s): 1-40, 2023. https: / / doi.org / 10.1145 / 3582688.

[66] M. Ferrdous, SMMAhsan. PPE detector: a YOLO-based architecture to detect personal protective equipment (PPE) for construction sites. PeerJ Computer Science. 8e999, 2022. https: / / doi.org / 10.7717 / peerj-cs.999.

[67] J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement. 20 37-46, 1960. https: / / doi.org / 10.1177 / 001316446002000104.

[68] Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv: 2403.13372.2024. https: / / doi.org / 10.48550 / arXiv.2403.13372.

[69] B. Cao, J. Yuan, Y. Liu, J. Li, S. Sun, J. Liu, B. Zhao. SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model. ArXiv.abs / 2402.18068, 2024. https: / / doi.org / 10.48550 / arXiv.2402.18068.

[70] S. Poddar, P. Koley, J. Misra, N. Ganguly, S. Ghosh. Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models. ArXiv.abs / 2502.05610, 2025. https: / / doi.org / 10.48550 / arXiv.2502.05610.

[71] Y. Zhang, A. Unell, X. Wang, D. Ghosh, Y. Su, L. Schmidt, S. Yeung-Levy. Why are Visually-Grounded Language Models Bad at Image Classification? arXiv preprint arXiv: 2405.18415.2024. https: / / doi.org / 10.48550 / arXiv.2405.18415.

[72] S. Kumar, M. Poyyamozhi, B. Murugesan, N. Rajamanickam, R. Alroobaea, W. Nureldeen. Investigation of Unsafe Construction Site Conditions Using Deep Learning Algorithms Using Unmanned Aerial Vehicles. Sensors. 24(20): 6737, 2024. https: / / doi.org / 10.3390 / s24206737.

[0102] The above references are included in this paper in full.

Claims

1. A system based on a visual language model (VLM) for end-to-end construction hazard identification and reporting generation, comprising: One or more processors; A VLM-enabled construction hazard detector, supported by one or more processors and specifically designed for construction hazard identification; and Large Language Model (LLM), which is supported by one or more processors and is specifically designed for generating construction hazard reports; One or more memory devices, interoperably coupled to the one or more processors and storing one or more instructions, which, when executed by the one or more processors, cause: the VLM-enabled construction hazard identifyer to autonomously identify one or more construction hazards in the construction site; and the LLM to generate an end-to-end real-time construction hazard report based on the identified construction hazards; The VLM-enabled construction hazard identifyer is obtained through the following operations: constructing multiple image-text pairs related to construction hazards; designing multiple label alignment prompts based on the image-text pairs; establishing a construction hazard dataset by associating the image-text pairs with the corresponding label alignment prompts; using the construction hazard dataset to fine-tune multiple general VLMs; and selecting the best-performing fine-tuned VLM as the VLM-enabled construction hazard identifyer.

2. The system based on a Visual Language Model (VLM) according to claim 1, wherein the plurality of image-text pairs related to construction hazards are constructed by the following operations: Based on at least one of the past literature and practical experience, several common construction hazards are systematically classified to establish a construction hazard checklist; Collect multiple images from the construction site that exhibit various potential hazards; The collected images are categorized according to the construction hazard checklist so that each image is aligned with one or more construction hazard categories. as well as Each image category is associated with a corresponding text description that confirms the image represents at least one category of construction hazard, to form the plurality of image-text pairs.

3. The system based on a visual language model (VLM) according to claim 2, wherein the plurality of common construction hazards are classified into four main categories: personal protective equipment (PPE), worker behavior, tools and equipment, and environment.

4. The system based on a visual language model (VLM) according to claim 2, wherein the plurality of label alignment prompts are designed by: defining the VLM to be fine-tuned as an expert role and expertise for identifying potential hazards at a construction site; specifying input information and output requirements for the VLM; and listing a plurality of hazard categories based on classified images associated with a plurality of image-text pairs.

5. The system based on a visual language model (VLM) according to claim 1, wherein the general visual language model (VLM) has different parameter sizes; and each of the general VLMs is fine-tuned using a low-rank adaptive technique without changing the parameters of the general VLM.

6. The system based on a visual language model (VLM) according to claim 1, wherein each of the fine-tuned VLMs is evaluated by: The finely tuned VLM is used to infer the output on the evaluation dataset; Obtain the alignment result between the inferred output and the hazard label; as well as The evaluation metric is calculated based on the obtained alignment results.

7. The system based on a visual language model (VLM) according to claim 1, wherein: The one or more processors are configured to receive real-time video feeds from the construction site; Extract multiple image frames from the real-time video feed; and input the multiple image frames into the VLM-enabled construction hazard identifier; The VLM-enabled construction hazard identifyer is configured to identify one or more construction hazards in the plurality of image frames; The one or more processors are further configured to determine the identified hazards corresponding to each time point and location at the construction site; and The large language model is configured to: compile all identified hazards within a specific time period; and generate a real-time construction hazard report for the construction site within the specific time period based on the compiled identified hazards.

8. The system based on a visual language model (VLM) according to claim 7, wherein the plurality of image frames are extracted from the real-time video frame by: capturing image frames from the real-time video frame at predetermined time intervals; and associating each captured image frame with a corresponding position and capture time.

9. The system based on a visual language model (VLM) according to claim 8, wherein the one or more construction hazards are identified in the plurality of image frames by: sequentially reading the plurality of image frames; recording the plurality of image frames in a database; and archiving a hazard record for each of the plurality of image frames, wherein the hazard record includes a location and timestamp associated with the image frame.

10. The system based on a Visual Language Model (VLM) according to claim 9, wherein the construction hazard report is generated within the specific time period through the following operations: The design incorporates multiple report generation prompts to organize the construction hazard identification results across all locations within the specified time period. These prompts include data format, source explanation, output format, and considerations for ignoring incidental hazards in the time series. The report generation prompt is used to read a specified length of construction hazard record data from the database, the specified length being determined based on the maximum number of tokens that can be processed by the Large Language Model (LLM); The output results generated by the LLM are stored in the database; Repeat the read and store steps until all potential hazard records in the database have been processed; The report generation prompts the reader to retrieve the segmented construction hazard analysis results from the database; and The construction hazard report is generated based on the results of the segmented construction hazard analysis.

11. A method based on a visual language model (VLM) for end-to-end construction hazard identification and reporting generation, comprising: A VLM-enabled construction hazard detector identifies one or more hazards on the construction site; and The system generates end-to-end real-time construction hazard reports based on the identified construction hazards using a large language model. The VLM-enabled construction hazard identifyer is obtained through the following operations: constructing multiple image-text pairs related to construction hazards; designing multiple label alignment prompts based on the image-text pairs; establishing a construction hazard dataset by associating the image-text pairs with the corresponding label alignment prompts; using the construction hazard dataset to fine-tune multiple general VLMs; and selecting the best-performing fine-tuned VLM as the VLM-enabled construction hazard identifyer.

12. The method based on a visual language model (VLM) according to claim 11, wherein the plurality of image-text pairs related to construction hazards are constructed by the following operations: Based on at least one of the past literature and practical experience, several common construction hazards are systematically classified to establish a construction hazard checklist; Collect multiple images from the construction site that exhibit various potential hazards; The collected images are categorized according to the construction hazard checklist so that each image is aligned with one or more construction hazard categories. as well as Each image category is associated with a corresponding text description that confirms the image represents at least one category of construction hazard, to form the plurality of image-text pairs.

13. The method based on a visual language model (VLM) according to claim 12, wherein the plurality of common construction hazards are classified into four main categories: personal protective equipment (PPE), worker behavior, tools and equipment, and environment.

14. The method based on a visual language model (VLM) according to claim 12, wherein the plurality of label alignment prompts are designed by: defining the VLM to be fine-tuned as an expert role and expertise for identifying hazards at a construction site; specifying input information and output requirements for the VLM; and listing a plurality of hazard categories based on classified images associated with a plurality of image-text pairs.

15. The method based on a visual language model (VLM) according to claim 11, wherein the general visual language model (VLM) has different parameter sizes; and each of the general VLMs is fine-tuned using a low-rank adaptive technique without changing the parameters of the general VLM.

16. The method based on a visual language model (VLM) according to claim 11, wherein each of the fine-tuned visual language models (VLMs) is evaluated by: The finely tuned VLM is used to infer the output on the evaluation dataset; Obtain the alignment result between the inferred output and the hazard label; as well as The evaluation metric is calculated based on the obtained alignment results.

17. The method based on a visual language model (VLM) according to claim 11, further comprising: Receive real-time video feeds from the construction site; Extract multiple image frames from the real-time video footage; The multiple image frames are input into the VLM-enabled construction hazard identifier; Identify one or more potential construction hazards in the plurality of image frames; Identify the potential hazards corresponding to each time point and location at the construction site; Compile all identified vulnerabilities within a specific time period; as well as A real-time construction hazard report for the construction site during the specific time period is generated based on the identified hazards compiled.

18. The method based on a visual language model (VLM) according to claim 17, wherein the plurality of image frames are extracted from the real-time video frame by: capturing image frames from the real-time video frame at predetermined time intervals; and associating each captured image frame with a corresponding position and capture time.

19. The method based on a visual language model (VLM) according to claim 18, wherein the one or more construction hazards are identified in the plurality of image frames by: sequentially reading the plurality of image frames; recording the plurality of image frames in a database; and archiving a hazard record for each of the plurality of image frames, wherein the hazard record includes a location and timestamp associated with the image frame.

20. The method based on a Visual Language Model (VLM) according to claim 19, wherein the construction hazard report is generated within the specific time period by the following operations: The design incorporates multiple report generation prompts to organize the construction hazard identification results across all locations within the specified time period. These prompts include data format, source explanation, output format, and considerations for ignoring incidental hazards in the time series. The report generation prompt is used to read a specified length of construction hazard record data from the database, the specified length being determined based on the maximum number of tokens that can be processed by the Large Language Model (LLM); The output results generated by the LLM are stored in the database; Repeat the read and store steps until all potential hazard records in the database have been processed; The report generation prompts the reader to retrieve the segmented construction hazard analysis results from the database; and The construction hazard report is generated based on the results of the segmented construction hazard analysis.

Images